Target classification device, target classification method, and target classification system

Abstract

The target classification device generates region unit information indicating region information of a target, and classifies the target by determining the type or state of the target on the basis of a region unit feature amount extracted from the region unit information and a partial region unit feature amount extracted from a partial region of an input image. And the classification result of the target is prompted to the user. As a result, even if targets having various sizes, lengths, shapes and the like are mixed in the image, the types or states of the targets can be classified with high accuracy.

Description

[0001] This application claims the benefit of priority based on Japanese patent application No. 2022-128897 filed on August 12, 2022, the contents of which are hereby incorporated by reference into this application. Technical Field

[0002] The present invention relates to techniques for classifying objects within an image. Background Art

[0003] When inspecting cells, materials, etc., inspection images captured by optical microscopes, electron microscopes, CCD cameras, CMOS cameras, etc. are sometimes used. However, visually observing inspection images is very time-consuming and often requires professional knowledge. Therefore, in order to assist in the inspection of cells and materials using inspection images, technologies such as automatic classification of objects in images have been developed.

[0004] For example, Japanese Patent Application Laid-Open No. 2010-203949 discloses a method of performing global classification using a low-magnification image of a fixed size and then performing local classification using a high-magnification image. Summary of the invention

[0005] Problems to be solved by the invention

[0006] However, depending on the inspection object, the size and length of the objects contained in the image are sometimes extremely different. In the method using a fixed-size low-magnification image, it is sometimes impossible to accurately extract the regional information for all objects. For example, when the size of the low-magnification image is determined according to the small object, it is difficult to accurately extract the regional information of the large object because the entire image of the large object does not enter the patch. In addition, when the size of the low-magnification image is determined according to the large object, it is possible that many other objects and dust existing around the small object are included, which may affect the accurate extraction of regional information of the classified object.

[0007] An object of the present invention is to classify objects with high accuracy even when objects of various sizes, lengths, shapes, etc. are mixed in an image.

[0008] Means for solving problems

[0009] A target classification device according to one embodiment of the present invention includes: a computing device; and a storage device storing a program executed by the computing device, wherein the computing device determines a target area of ​​a target as a whole in an image, generates information of a target area unit based on the target area, extracts a target area unit feature value from the information of the target area unit, respectively extracts partial area unit feature values ​​from one or more partial areas within the target area of ​​the image, and classifies the target based on the target area unit feature value and the partial area unit feature value.

[0010] Effects of the Invention

[0011] According to one embodiment of the present invention, even when objects of various sizes, lengths, shapes, etc. are mixed in an image, objects can be classified with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a diagram showing an example of the hardware configuration of the object classification system of Example 1.

[0013] Figure 2 This is a diagram showing an example of a functional block diagram of the target classification device of Example 1.

[0014] Figure 3A This is a diagram showing an example of a method for calculating area unit information.

[0015] Figure 3B This is a diagram showing an example of a method for calculating area unit information.

[0016] Figure 4 This is a diagram showing an example of area unit information.

[0017] Figure 5A This is a diagram showing an example of a method for extracting a partial region unit feature value.

[0018] Figure 5B This is a diagram showing an example of a method for extracting a partial region unit feature value.

[0019] Figure 6 This is a diagram showing an example of a target classification method.

[0020] Figure 7 This is a diagram showing an example of the majority decision calculation result in the classification unit.

[0021] Fig. 8A This is a diagram showing an example of a method of presenting classification results.

[0022] Figure 8B This is a diagram showing an example of a method of presenting classification results.

[0023] Figure 8C This is a diagram showing an example of a method of presenting classification results.

[0024] Fig. 9A This is a diagram showing an example of a GUI for displaying the degree of contribution of each feature amount to a classification result.

[0025] Fig. 9B This is a diagram showing an example of a GUI for displaying the degree of contribution of each feature amount to a classification result.

[0026] Fig. 9C This is a diagram showing an example of a GUI for displaying the degree of contribution of each feature amount to a classification result.

[0027] Fig.9D This is a diagram showing an example of a GUI for displaying the degree of contribution of each feature amount to a classification result.

[0028] Fig.10 This is a diagram showing an example of the processing flow of the object classification method of Example 1.

[0029] Fig.11 This is a diagram showing an example of the hardware configuration of the object classification system of Example 2.

[0030] Fig.12 This is a diagram showing an example of evaluation rules for an image group according to the second embodiment.

[0031] Fig.13 This is a diagram showing an example of area unit information.

[0032] Fig.14 This is a diagram showing an example of area unit information. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the object classification device, method, and system of the present invention will be described according to the accompanying drawings. In addition, in the following description and the accompanying drawings, components having the same functional structure are denoted by the same reference numerals to omit repeated descriptions.

[0034] Example 1

[0035] use Figure 1 , an example of a target classification system in an image of Example 1 is described. The target classification system 100 includes a target classification device 101, a shooting device 120, and a display device 121. The target classification device 101 includes an interface 110, a computing device 111, a memory 112, and a bus 113. The interface 110, the computing device 111, and the memory 112 send and receive information via the bus 113. The target classification device 101 is connected to the shooting device 120 and the display device 121 via the interface 110.

[0036] The components of the object classification device 101 are described. The interface 110 is a communication device that transmits and receives signals with a device outside the object classification device 101. Examples of devices that communicate with the interface 110 include a camera 120 and a display device 121. The details of the camera 120 and the display device 121 will be described later.

[0037] The computing device 111 is a device that performs various processes in the target classification device 101, such as a CPU (Central Processing Unit) or an FPGA (Field-Programmable Gate Array). Figure 2 This will be described later.

[0038] The memory 112 is a device for storing programs, parameters, coefficients, processing results, etc. executed by the operation device 111, and is a HDD, SSD, RAM, ROM, flash memory, etc.

[0039] The target classification device 101 can be a physical computer system (one or more physical computers) or a system built on a computing resource group (multiple computing resources) such as a cloud platform. The computer system or computing resource group can include one or more interface devices (for example, including a communication device and an input / output device), one or more storage devices (for example, including a memory (main storage) and an auxiliary storage device) and one or more computing devices.

[0040] When a function is implemented by executing a program including command codes by a computing device, a storage device and / or an interface device are appropriately used to perform the determined processing, so that the function can be at least a part of the computing device. The processing described with the function as the subject can also be the processing performed by the computing device or a system having the computing device. The program can be installed from a program source.

[0041] The program source may be, for example, a program distribution computer or a computer-readable storage medium (eg, a computer-readable non-transitory storage medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0042] The photographing device 120 is a device for photographing an image of a target, such as a camera or a microscope. The photographing device 120 sends the photographed image to the target classification device 101. In addition, the photographing device 120 can be replaced by a communication device that receives an image via a network, or a recording device that receives an image by reading an image recorded in a recording medium.

[0043] The display device 121 is a device for displaying the object classification information output by the object classification device 101 , and is, for example, a display or a printer.

[0044] The object classification device 101 will be described in detail below.

[0045] Figure 2 This is an example of a functional block diagram of the first embodiment of the object classification device 101. In addition, these functions may be constituted by dedicated hardware, or the computing device 111 may operate according to software.

[0046] The object classification device 101 includes an image input unit 201, an object region calculation unit 202, an area unit information calculation unit 203, an area unit feature extraction unit 204, a partial area unit feature extraction unit 205, an object classification unit 206, and an output unit 207. Each unit will be described below.

[0047] The image input unit 201 receives an image including an object to be classified, which is input from the interface 110. The input image is input to the object region calculation unit 202, the partial region unit feature quantity extraction unit 205, and the output unit 207 via the memory 112.

[0048] The target region calculation unit 202 will be described. The target region calculation unit 202 extracts a target region to be classified using the input image received by the image input unit 201. As an example of a method of extracting a target region, a case of using a region segmenter will be described.

[0049] Region segmentation is a method of dividing an image into meaningful sets in units of pixels. For example, the pixels where the target to be classified is located are regarded as the foreground, and the other pixels are regarded as the background for binary classification. The combined area of ​​the areas classified as the foreground is detected, thereby detecting the area for each target.

[0050] As a method of region segmentation, there are methods using U-Net, SegNet, etc. In addition, since objects are densely present in the inspection image, it is possible to use instance segmentation or the like in which a region segmentation result is calculated for each object instead of the usual region segmentation.

[0051] The area unit information calculation unit 203 is described. Based on the target area information output by the target area calculation unit 202, the area unit information calculation unit 203 expresses the area information of each target such as size, length, deformation, brightness, color, etc. as one-dimensional or two-dimensional information of a fixed size and outputs it.

[0052] use Figure 3A and Figure 3B , which illustrates an example of area unit information. Figure 3A as well as Figure 3BThe calculation example of the area unit information shown is an example of expressing the shape of the area as waveform information (two-dimensional information) of a fixed size by plotting the distance from a point in the target area to each contour point of the area in polar coordinate format.

[0053] Figure 3A represents the region segmentation result 301, Figure 3B The area unit information 302 is shown. The area segmentation result 301 is included in the target area information output by the target area calculation unit 202. The target area 303 is the target area in the area segmentation result 301. The point 304 is a point in the target area 303.

[0054] Area unit information 302 is area unit information calculated based on target area 303. Point 305 is a point on the outline of target area 303. Deflection angle 306 is the deflection angle relative to point 305 when the vertical direction (downward direction in the figure) is set to 0 degrees from point 304. Distance 307 is the distance on the image to point 305 when point 304 is the origin. Figure 3B , the horizontal axis represents the deflection angle 306, and the vertical axis represents the distance 307. The deflection angle 306 increases clockwise. The vertical axis represents the distance 307 increasing toward the bottom of the figure.

[0055] First, the area unit information calculation unit 203 calculates a point 304 from the target area 303 included in the input area segmentation result 301. The calculation method of the point 304 may be, for example, the centroid or the heart of the target area 303. Thus, it is possible to obtain area unit information that can be classified with higher accuracy. Points in target areas different from these can also be used. Next, the area unit information calculation unit 203 scans the pixels in the area segmentation result 301 starting from the point 304 in the vertical direction, and measures the distance from the point 304 to the contour of the target area 303. The area unit information calculation unit 203 draws the measured distance in the area unit information 302.

[0056] The horizontal axis of the area unit information 302 represents the deflection angle when the vertical direction is set to 0 degrees from the point 304, and the vertical axis represents the distance from the point 304 to the contour. For example, with respect to the point 305 on the contour line existing in the direction of the deflection angle 306 from the point 304, when the distance from the point 304 to the point 305 is the distance 307, it is drawn as a point corresponding to the deflection angle 306 and the distance 307 in the area unit information 302. Thus, the shape of the target area 303 is expressed as waveform information of a fixed size like the area unit information 302.

[0057] Figure 44 shows examples of extraction of the above-mentioned area unit information for objects having various shapes. Object 401, object 402, object 403, and object 404 are examples of objects having different shapes. Area unit information 411, area unit information 412, area unit information 413, and area unit information 414 are examples of area unit information extracted from object 401, object 402, object 403, and object 404, respectively. Figure 4 In the example above, the centroid is used as the reference point within the target area. Figure 3B The area unit information 302 is the same.

[0058] The target 401 is a circular target, and the distance from a point in the target area to the outline is the same in any direction, so the area unit information 411 is represented as a straight line. The target 402 is also circular, but the size is larger than the target 401. Therefore, the area unit information 412 is a straight line like the area unit information 411, but is drawn at a lower position than the area unit information 411.

[0059] The target 403 is elongated, so the area unit information 413 is expressed as a periodic wave. The outline of the target 404 is not smooth but deformed. The corresponding area unit information 414 also reflects the deformation. As described above, the area unit information described here is information that can express the shape of targets of various sizes, lengths, and shapes as waveform information of a fixed size.

[0060] If the regional information of these objects is to be expressed by a low-magnification image of a fixed size, if the image size is made consistent with a small object, an object whose entire shape cannot be included in the image will be generated, and if the image size is made consistent with a large object, many areas irrelevant to the object to be classified will be included in the small object. In the case where the object to be classified exists in an area where other types of objects are densely packed or around non-classification objects such as garbage, the shape information may not be correctly extracted due to the influence of the surrounding objects or garbage.

[0061] In contrast, if the area unit information described in the embodiment of this specification is used, even when objects of various sizes, lengths, and shapes are mixed, the influence of other objects can be suppressed, and the shape of the target object can be expressed as information of a fixed size. As a result, through the cooperation with the area unit feature extraction unit 204, the partial area unit feature extraction unit 205, and the object classification unit 206 described later, the object can be classified with high classification accuracy.

[0062] In this description, only the area unit information of the target area information output by the target area calculation unit 202 is used. In another example, the input image may be received from the image input unit 201, and the area unit information may be extracted using both the input image and the target area information.

[0063] For example, in reference Figure 3A and Figure 3B In the example described, the area unit information calculation unit 203 can calculate the average value or variance value of the brightness of the input image on the line connecting the point 305 on the contour line from the point 304, and plot the deflection angle to extract the area unit information. In addition, the area unit information can also be one-dimensional information, for example, indicators such as the area, circularity, brightness, and hue statistics of the target area can be used. The area unit information calculation unit 203 can output multiple types of area unit information.

[0064] In the above, as an example of area unit information, a method of extracting information such as the distance from a point within the target to each contour point of the area in the form of polar coordinates is explained, but the method of generating area unit information is not limited to this. Any method can be used as long as the information of the area is extracted and expressed as one-dimensional or two-dimensional information of a fixed size.

[0065] As an example, use Fig.13 An example of obtaining area unit information using frequency conversion is described below. Image 1301 is an input image, including an object 1302. Image 1303 before frequency conversion is an image for frequency conversion in which only the area of ​​object 1302 is arranged at the center of the image, and image 1304 after frequency conversion is an image after frequency conversion of image 1303 before frequency conversion. Area unit information 1305 is an image obtained by extracting a part of image 1304 after frequency conversion.

[0066] First, the area unit information calculation unit 203 extracts the target area 1302 from the image 1301 based on the target area information output by the target area calculation unit 202. The extracted target area 1302 is arranged at the center of the image 1303 before frequency conversion. At this time, the brightness value of the area other than the target area 1302 of the image 1303 before frequency conversion is set to a fixed value (for example, 0). The target area 1302 may be an image obtained by extracting the target area from the image 1301, or may be the target area information output by the target area calculation unit 202 (an image in which the target area and other areas are represented by binary values).

[0067] The area unit information calculation unit 203 sets the size of the pre-frequency conversion image 1303 to the maximum size of the object included in the input image. Then, a frequency conversion image 1304 is generated by frequency conversion such as Fourier transformation. The frequency conversion image 1304 represents higher frequency components farther from the center, and represents larger components with higher brightness.

[0068] Since the low-frequency component is important for the overall shape information of the target area, the area unit information calculation unit 203 extracts the area representing the low-frequency component of the frequency-converted image 1304 as the area unit information 1305. Fig.13 In the example of , a low-frequency component appears in the center of the frequency-converted image 1304, so a partial image of the center is extracted with a predetermined fixed size.

[0069] In addition, the image before frequency conversion is also two-dimensional information of a fixed size, but in the case of a long and thin object, for example, most of the area is the above-mentioned fixed value, and the memory usage efficiency is poor. By using the low-frequency components of the image after frequency conversion, the shape of the object can be efficiently expressed as two-dimensional information of a fixed size.

[0070] In addition, when the object 1302 is arranged at the center of the image 1303 before frequency conversion, normalization processing such as rotating the object so that the long axis thereof becomes vertical may be performed. Fig.13 In the example of FIG. 1 , a low-frequency component appears in the center of the image 1304 after the frequency transformation, so a method of extracting a partial image in the center with a predetermined fixed size is described, but other frequency transformation methods may be used, and only information related to the required frequency component may be extracted according to the transformation method. For example, in the case of a frequency transformation method for extracting a plurality of frequency transformation results with different frequency components from an input image, only the frequency transformation result related to the required frequency component may be extracted as the area unit information.

[0071] In addition, as another example of area unit information, Fig.14 14. The example of drawing the angle of the tangent line at each point on the contour of the target and setting it as the area unit information is described below. The area segmentation result 1401 is the target area information output by the target area calculation unit 202. The target area 1402 is the target area in the area segmentation result 1401. The target contour image 1403 is the result of applying image processing to the area segmentation result 1401 to extract only the contour of the target.

[0072] The target contour line 1404 is the contour of the target area 1402. The point 1405 on the contour is a point on the target contour line 1404. The point 1409 on the contour is a point on the contour line 1404 adjacent to the point 1405 on the contour. The tangent line 1406 is the tangent line of the target contour line 1404 at the point 1405 on the contour. The angle 1407 is the angle between the tangent line 1406 and the horizontal line. The area unit information 1408 is the result of plotting the angle 1407 at the point 1405 on each contour.

[0073] First, the area unit information calculation unit 203 obtains a target contour image 1403 and a target contour line 1404 by applying an edge extraction process such as a Sobel filter to the area segmentation result 1401 output by the target area calculation unit 202. Next, points 1405 on the contour are arranged on the target contour line 1404. The number E of the points 1405 on the contour is set to a predetermined fixed value. For example, based on the value obtained by dividing the total length of the target contour line 1404 by E, a plurality of points 1405 on the contour are arranged at equal intervals.

[0074] Then, the area unit information calculation unit 203 calculates the tangent line 1406 for each point 1405 on the contour and obtains the angle 1407. Here, the horizontal line is assumed to be 0 degrees, and the angle 1407 is in the range of -90 degrees to 90 degrees. The area unit information 1408 is obtained by plotting each angle 1407 in a graph. The horizontal axis in the area unit information 1408 corresponds to the position of the point 1405 on each contour, and the vertical axis corresponds to the angle 1407. The value range of the horizontal axis is 1 to E, and the size of the area unit information 1408 is fixed regardless of the target size.

[0075] By using the above method, the shape information of the target's contour line can be expressed as a fixed-size graphic (two-dimensional information). As a difference from the polar coordinate format, it is not necessary to determine the center point and does not include information on the size of the target. Fig.14 In order to facilitate understanding, the area unit information 1408 is recorded as two-dimensional information, but it can also be expressed as one-dimensional information by setting the angle 1407 to the intensity of the signal (brightness in the case of an image). In addition, an example in which the horizontal direction is used as the reference of the angle is described here, but the tangent at the point 1409 on the contour adjacent to the point 1405 on a certain contour can also be used as the reference of the angle.

[0076] In this way, the area unit information calculation unit 203 calculates and outputs area unit information that expresses information on the entire target area as one-dimensional or two-dimensional information of a fixed size.

[0077] The regional unit feature extraction unit 204 is described. The regional unit feature extraction unit 204 extracts the regional feature of the entire classification target from the regional unit information output by the regional unit information calculation unit 203. As a feature extraction method, a machine learning method such as a convolutional neural network (CNN) and a perceptron can be used, or a manually designed feature extraction method such as a histogram of oriented gradients (HOG) can be used.

[0078] Furthermore, when there are a plurality of area unit information output from the area unit information calculation section 203 , feature amounts may be comprehensively extracted from the plurality of area unit information, or feature amounts may be individually extracted from each area unit information.

[0079] The partial region unit feature quantity extraction unit 205 is described. The partial region unit feature quantity extraction unit 205 extracts feature quantities related to texture, brightness, hue, etc. from the partial region of the input image output by the image input unit 201 based on the target region information output by the target region calculation unit 202. As a feature quantity extraction method, a machine learning method such as CNN and perceptron can be used, and a manually designed feature quantity such as HOG, or a quantitative value such as brightness and variance in a partial region can also be used as a feature quantity.

[0080] use Figure 5A and Figure 5B , which illustrates an example of extracting unit feature quantities of a partial area. Figure 5A An example showing target area information, Figure 5B Indicates that it is extracted Figure 5A An example of an input image with target region information is shown.

[0081] Target region information 501 is an example of target region information for a single target output by the target region calculation unit 202. Target region 503 is indicated in white, and background region is indicated in gray. Input image 502 is a region of the input image corresponding to target region information 501. Target 504 is a target corresponding to target region 503, and here, the entire duct structure in which cells are arranged relative to the cavity is taken as the target. ROIs 505 and 506 are regions of interest in target region information 501 and input image 502.

[0082] First, the partial region unit feature extraction unit 205 sets a region of interest (ROI 505) of the target region information 501 at the upper left of the target region information 501. In addition, the width and height of the region of interest are predetermined by the user. The partial region unit feature extraction unit 205 determines whether to extract partial region unit feature values ​​based on the target region information in the region of interest.

[0083] For example, when the pixels belonging to the target region 503 exist in the region of interest at a ratio greater than a predetermined threshold, the partial region unit feature quantity extraction unit 205 extracts the partial region unit feature quantity. For example, when the threshold is set to be greater than 50%, in the example of FIG. 5 , since there are no pixels belonging to the target region 503 in ROI 505 and the ratio is lower than the threshold, the partial region unit feature quantity is not extracted from the region of ROI 505. On the other hand, since there are more than 50% of the regions belonging to the target region 503 in ROI 506, the partial region unit feature quantity is extracted from the region of ROI 506.

[0084] After the partial area unit feature extraction unit 205 has completed the extraction process and whether the partial area unit feature of a certain area of ​​interest can be extracted, it performs the same process while changing the position of the area of ​​interest in a raster scan manner, for example. When the process of the entire input target area is completed, the partial area unit feature extraction unit 205 outputs the partial area unit feature.

[0085] The object classification unit 206 will be described. The object classification unit 206 classifies objects based on the area unit feature quantity output from the area unit feature quantity extraction unit 204 and the partial area unit feature quantity output from the partial area unit feature quantity extraction unit 205 .

[0086] use Figure 6 An example of the target classification method is described below. Input image 601 is an input image output by image input unit 201. Target region information 602 is target region information output by target region calculation unit 202. Partial region unit feature quantity 603 is partial region unit feature quantity output by partial region unit feature quantity extraction unit 205. Region unit feature quantity 604 is region unit feature quantity extracted by region unit information calculation unit 203 and region unit feature quantity extraction unit 204.

[0087] The classifier 605 is used to calculate the target classification result 606 of the partial area unit based on a part of the partial area unit feature 603 and the area unit feature 604. The classification result map 607 is an example of a result obtained by mapping the classification result 606 of the partial area unit according to the extraction position of the partial area unit feature 603 input to the classifier. In addition, the processing of the target classification unit 206 is Figure 6 The range within the dotted line.

[0088] The input to the object classification unit 206 is the area unit feature 604 and the partial area unit feature 603. The area unit feature 604 is represented by a feature vector for one object, and the partial area unit feature 603 is composed of feature vectors for each partial area. The object classification unit 206 has a classifier 605 inside, and the classifier 605 takes the feature vectors extracted from the partial areas in the partial area unit feature 603 and the area unit feature 604 as inputs, and outputs the classification result 606 of the partial area unit.

[0089] exist Figure 6 In the example, the type of the target (target A to C) is set as the classification result. As the classification method of the classifier 605, a machine learning method such as logistic regression and support vector machine (SVM) can be used, or a method using a manually designed classification formula can be used. According to the extraction position of the partial area unit feature quantity 603 input to the classifier 605, the classification result 606 of the partial area unit is recorded in the classification result map 607.

[0090] After calculating the classification results for all parts of the partial area unit feature 603 and recording them in the classification result map 607, the object classification unit 206 calculates the comprehensive classification result for the object based on the content of the classification result map 607. As a method for calculating the comprehensive classification result, there is a method of obtaining a majority decision of the classification results in the classification result map 607, for example.

[0091] Figure 7 An example of a majority decision calculation result calculated based on the classification results in the classification result map 607 is shown. Figure 7 In the example of , the classification results for object A are the largest, so the objects in the input image 601 are comprehensively determined to be object A. In this way, the object classification unit 206 calculates the classification results for each object in the image.

[0092] The output unit 207 is described below. The output unit 207 presents the classification result of the object classification unit 206 to the user. Fig. 8A , 8B 8C illustrates an example of a method for presenting classification results to a user. Fig. 8A An example of an input image is shown. The input image 801 is an example of an input image received by the image input unit 201 and is an inspection image obtained by imaging a granular material. Fig. 8A The granular raw material of the example is a raw material in which each particle is round and large, has no damage, etc., and has good quality. Here, particle 803 is a particle of good quality, and particle 804 and particle 805 are particles of poor quality.

[0093] Figure 8B An example of a classification result window is shown. Classification result window 802 is a window for presenting classification results to the user. Classification results 806, 807, and 808 respectively indicate classification results for particle 803, particle 804, and particle 805. Figure 8C An example of a quantitative value display window is shown. The quantitative value display window 809 is an example of a window that displays the ratio of each object in an image.

[0094] exist Fig. 8A , Figure 8B and Figure 8C In the example shown, the target classification unit 206 classifies each target (particle) into particles of good quality and particles of poor quality, classifies particle 803 as a particle of good quality, and classifies particles 804 and 805 as particles of poor quality. In the classification result window 802, for example, the classification result is visualized by color-coding the target area according to the classification result, or the visualization result is mixed with the input image for display. In this example, different textures are used to represent particles of good quality and particles of poor quality.

[0095] In addition, the output unit 207 may also calculate quantitative values ​​such as the number and ratio of various objects in the input image, and display the calculation results in the quantitative value display window 809 .

[0096] Furthermore, the output unit 207 may display evaluation values ​​for each target, such as the contribution and effectiveness of the area unit feature quantity and the partial area unit feature quantity to the classification result. Figures 9A to 9D This is an example of a GUI that displays the contribution of each feature value to the classification result.

[0097] Fig. 9A The GUI screen of FIG. 800 displays a pointer 901 on the classification result window 802. The pointer 901 is a pointer for the user to select a classification result of a target in the classification result window 802.

[0098] Fig. 9B An example of a contribution degree display window is shown. The contribution degree display window 902 is a window for displaying the contribution degree of area unit information and partial area unit information to the classification result of the target.

[0099] Fig. 9C This shows an example of the area unit information display window. Fig. 9C The area unit information display window 903 is a window for displaying area unit information. Fig.9D An example of a partial region unit contribution degree display window is shown. The partial region unit contribution degree display window 904 is a window that displays the contribution degree of each partial region.

[0100] First, after the target classification unit 206 calculates the classification result, the output unit 207 calculates the contribution of each feature quantity to the classification result. The contribution can be calculated using an analytical method such as GradCAM. The contribution of each feature quantity in the entire target area is calculated by obtaining the regional unit feature quantity and the contribution of the partial regional unit feature quantity for the classification result of each partial area and summing them up over the entire area. The calculated contribution is presented to the user via the contribution display window 902.

[0101] like Fig. 9A As shown in FIG. 8 , after prompting the user with the classification result window 802, the user's input is waited for. For example, if the user clicks a target in the classification result window 802 with the pointer 901, the user's input is waited for. Fig. 9B As shown, the contribution of each feature value to the classification result of the target is displayed in the contribution display window 902.

[0102] exist Fig. 9B In the example, it is shown that the contribution of "shape" in the area unit feature quantity is high, and then the contribution of the partial area unit feature quantity is high. The area unit information related to "shape" here is the information of the distance from the point in the target area to the contour line in each deflection direction described in the area unit information calculation unit 203. In addition, the information of the contribution of only one of the area unit information and the partial area unit information may be presented.

[0103] like Fig. 9C As shown, the output unit 207 automatically displays the area unit feature value of "shape" with the highest contribution, for example, in the area unit information display window 903. In addition, the display content of the area unit information display window 903 can be switched by the user's selection. For example, when the user clicks on the name of the area unit information item in the contribution display window 902, the clicked area unit information can be displayed in the area unit information display window 903.

[0104] like Fig.9D As shown, the partial area unit contribution display window 904 displays a graph of the contribution of each partial area. For example, in the example of the partial area unit contribution display window 904, the contribution is represented by a brightness value (white: low contribution, black: high contribution). As described above, by prompting information related to the classification result of the target, the user's understanding can be helped.

[0105] In the above, the details of the first embodiment are described for each functional block. However, the embodiments of the present invention do not necessarily have to be composed of Figure 2 The functional block structure only needs to realize the processing for implementing the actions of each functional block. Fig.10 An example of a processing flow chart of Example 1 is shown. Each step is Figure 2 The elements of the functional block diagram shown correspond to each other.

[0106] In the image input step 1001 , an image of a target is received from the interface 110 .

[0107] In the target region calculation step 1002, a target region to be classified is extracted using the input image received in the image input step 1001. The method of extracting the target region is as described above with respect to the target region calculation unit 202.

[0108] In the area unit information calculation step 1003, area information such as the size, length, and deformation status of each object is expressed as fixed-size information and output based on the target area information calculated in the target area calculation step 1002. The method of generating the area unit information is as described above for the area unit information calculation unit 203.

[0109] In the area unit feature extraction step 1004, the area feature of the classification target object is extracted from the area unit information calculated in the above-mentioned area unit information calculation step 1003. The method of extracting the area unit feature is as described above with respect to the area unit feature extraction unit 204.

[0110] In the partial region unit feature quantity extraction step 1005, feature quantities related to texture, brightness, hue, etc. are extracted from the partial region of the input image received in the image input step 1001 based on the target region information calculated in the target region calculation step 1002. The partial region unit feature quantity extraction method is as described above for the partial region unit feature quantity extraction unit 205.

[0111] In the object classification step 1006, objects are classified based on the area unit feature values ​​calculated in the area unit feature value extraction step 1004 and the partial area unit feature values ​​calculated in the partial area unit feature value extraction step 1005. The object classification method is as described above for the object classification unit 206.

[0112] In the output step 1007, the classification result calculated in the object classification step 1006 is presented to the user. The method of presenting the result to the user is the same as that described above with respect to the output unit 207.

[0113] As described above, according to the object classification device and method described in Example 1, even when objects of various sizes, lengths, shapes, textures, etc. are mixed in an image, the type or state of the object can be classified with high accuracy.

[0114] Example 2

[0115] Example 2 describes an object classification system that uses the object classification device described in Example 1 to comprehensively determine the type and state of an entire image or a group of multiple images.

[0116] Fig.11 The hardware structure diagram of Example 2 is shown. The target classification system 1100 includes a photographing device 120 such as a camera or a microscope, an input device 1101 for receiving operations from a user, and the target classification device 101 described in Example 1. The target classification system 1100 further includes: a quantitative calculation device 1102, which calculates a quantitative value such as the total number or ratio of each target based on the target classification result output by the target classification device 101, thereby calculating a discrimination result of the type or state of the entire image or image group; and a display device 121, which prompts the user with the discrimination result output by the quantitative calculation device 1102.

[0117] The imaging device 120 and the display device 121 are the same as the hardware components of the first embodiment, and thus the description thereof is omitted. Next, the input device 1101 and the quantitative calculation device 1102 are described.

[0118] The input device 1101 is a device such as a keyboard or a mouse for receiving operation signals from the user. The input device 1101 is mainly used to select an image group to be quantitatively calculated by the quantitative calculation device 1102 from the image group captured by the imaging device 120 .

[0119] The quantitative calculation device 1102 calculates a quantitative value for one or more image groups based on the target classification results output by the target classification device 101. The quantitative value is, for example, the total number or ratio of each target, the sum or average of the recognition scores, etc., and the quantitative value is calculated based on the overall target classification results for the image group selected by the user via the input device 1101. Then, the type and state of the image group are determined based on the calculated quantitative value.

[0120] The process of the target classification system processing of Example 2 is described by taking the quality evaluation of granular raw materials as an example. Quality refers to the state of the granular raw materials. First, the imaging device 120 captures images from multiple locations of the granular raw materials. The captured images are input to the target classification device 101, and each particle in each image is classified by the target classification device 101 into particles with good quality and particles with poor quality.

[0121] The user operates the quantitative calculation device 1102 via the input device 1101 to calculate the quantitative value from the image group obtained by photographing a raw material. The quantitative calculation device 1102 calculates the ratio of particles with good quality to particles with poor quality based on the target classification results of all the input image groups. The quantitative calculation device 1102 calculates the ratio of particles with good quality to particles with poor quality, for example, according to Fig.12 The evaluation result of the raw material is calculated based on the ratio of particles with good quality according to the rule shown. The calculated evaluation result of the raw material is presented to the user via the display device 121.

[0122] This allows the user to determine the type and state of the input image group based on the classification results of objects present in one or more input images.

[0123] Example 3

[0124] Other embodiments of the present specification are described. When determining points within the target area, the area unit information calculation unit 203 may use a method for suppressing point fluctuations. For example, a method may be used that uses only pixels on the target as point candidates, or calculates the distance from each pixel on the target to the background (distance image) and weights the calculated distance.

[0125] The area unit information calculation unit 203 is described as two-dimensional graph-like information, but it can also be expressed by other expression methods such as expressing the vertical axis with brightness or hue. In addition, in order to improve the robustness of the area unit information, the input image can be preprocessed or weighted when calculating the area unit information.

[0126] In addition, when calculating the area unit information in the form of polar coordinates or from each point on the contour line, in order to suppress the phase shift, the starting point for calculating the area unit information can be determined based on the target area information. For example, there is a method of calculating the area unit information from the point farthest from the center point in the target area or the point closest to the center point. In this way, the starting point can be determined based on the shape of the target area. Thus, a more appropriate starting point can be selected.

[0127] Regarding the target classification unit 206, a method of determining the classification result of the entire target by majority decision has been described, but the method of determining the classification result of the entire target is not limited to this. For example, it can be determined based on the sum of the classification scores in the target area, or a rule such as the presence of a certain number or a certain proportion of specific classification results in the target area can be used. In addition, weighting can also be performed when taking a score or majority decision based on the target area information.

[0128] In addition, the present invention is not limited to the above-mentioned embodiments, and includes various modified examples. For example, the above-mentioned embodiments are examples described in detail in order to easily explain the present invention, and are not necessarily limited to having all the structures described. In addition, a part of the structure of a certain embodiment can be replaced with the structure of another embodiment, and the structure of another embodiment can be added to the structure of a certain embodiment. In addition, with respect to a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0129] In addition, the above-mentioned structures, functions, processing units, etc. may also be implemented in part or in whole by hardware, for example, by designing with integrated circuits. In addition, the above-mentioned structures, functions, etc. may also be implemented by software by interpreting and executing programs for implementing the functions by a processor. Information such as programs, tables, and files that implement the functions may be placed in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card.

[0130] In addition, the control lines and information lines are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines on the product. In fact, it can be considered that almost all structures are connected to each other.

Claims

1. A target classification device, which is used to classify targets in an image, characterized in that: The target classification device comprises: computing device; as well as a storage device storing a program executed by the computing device, The computing device determines a target area of ​​an entire target within an image, generates information of a target area unit based on the target area, extracts a target area unit feature value from the target area unit information, respectively extracts partial area unit feature values ​​from more than one partial area within the target area, and classifies the target based on the target area unit feature value and the partial area unit feature value.

2. The target classification device according to claim 1, characterized in that: The target region unit information is a one-dimensional or two-dimensional signal of a fixed size representing the characteristics of the target region.

3. The target classification device according to claim 2, characterized in that: The computing device determines a point in the target area, and generates information of the target area unit in a polar coordinate format with the point as an origin.

4. The target classification device according to claim 1, characterized in that: The computing device generates information of the target area unit based on information of the target area and pixels of the image.

5. The target classification device according to claim 3, characterized in that: The computing device determines a starting point of a deflection angle when generating information of the target area unit in the polar coordinate format, based on the shape of the target area.

6. The target classification device according to claim 1, characterized in that: The computing device analyzes the target area unit feature value and / or the partial area unit feature value and the classification result of the target, determines the evaluation value of the target area unit information and / or the partial area unit feature value relative to the classification result, and prompts the user with the evaluation value.

7. The target classification device according to claim 1, characterized in that: The computing device determines a classification result of the partial area unit based on the target area unit feature value and the partial area unit feature values ​​of the plurality of partial areas, The computing device classifies the object based on the classification result of the partial area unit.

8. A target classification system, characterized in that: have: The target classification device according to any one of claims 1 to 7; a photographing device that photographs one or more images input to the object classification device; and a quantitative calculation device for calculating a quantitative value of each type of the object contained in the one or more images based on the classification result of the one or more images by the object classification device, The one or more images are classified according to the quantitative value.

9. A method for classifying objects in an image, characterized in that: The device determines a target area of ​​an entire target within the image; The device generates information of a target area unit based on the target area; The device extracts a target area unit feature quantity from the information of the target area unit; The device extracts partial area unit feature quantities from more than one partial area in the target area respectively; The device classifies the object based on the object area unit feature quantity and the partial area unit feature quantity.

10. The target classification method according to claim 9, characterized in that: The target region unit information is a one-dimensional or two-dimensional signal of a fixed size representing the characteristics of the target region.

11. The target classification method according to claim 10, characterized in that: The device determines a point in the target area, and generates information of the target area unit in a polar coordinate format with the point as an origin.

12. The target classification method according to claim 9, characterized in that: The device generates information of the target area unit according to information of the target area and pixels of the image.

13. The target classification method according to claim 11, characterized in that: The device determines a starting point of a deflection angle when generating information of the target area unit in the polar coordinate format according to a shape of the target area.

14. The target classification method according to claim 9, characterized in that: The device analyzes the target area unit feature value and / or the partial area unit feature value and the classification result of the target, determines the evaluation value of the target area unit information and / or the partial area unit feature value relative to the classification result, and prompts the user with the evaluation value.

15. The target classification method according to claim 9, characterized in that: The device determines a classification result of the partial area unit based on the target area unit feature value and the partial area unit feature values ​​of the plurality of partial areas, The device classifies the target based on the classification result of the partial area unit.

Images