Image processing system and method, program, and storage medium

Abstract

To balance between improving boundary accuracy using superpixels and reducing the load of segmentation work in image segmentation in annotation work.SOLUTION: An image processing system comprises: input means to which image data on an image is input; acquisition means that acquires the size of an object area including an object to be extracted included in the image; setting means for setting a division number by which to divide the object area; determination means that determines a superpixel segment size on the basis of the size of the object area and the division number; and generation means that uses the image data to generate superpixels having a predetermined range of sizes including the segment size determined by the determination means.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device and method, a program and a storage medium, and more particularly to an image segmentation technique. [Background technology]

[0002] Semantic segmentation is known in machine learning. Semantic segmentation is a task of dividing an image into multiple objects. To perform semantic segmentation, the image is refined into pixel units and output. In this case, the training data for semantic segmentation needs to be labeled for each pixel.

[0003] Therefore, in semantic segmentation, the annotation work (the work of labeling images and creating training data) is very burdensome, so superpixels are used to reduce the workload. A superpixel is a small region that groups pixels with similar colors and textures.

[0004] For example, Patent Document 1 discloses a method for segmenting an image using hybrid-scale superpixels. Specifically, a user replaces larger-scale superpixels in a region of interest (ROI) with smaller-scale superpixels (region size of the superpixel) to improve segmentation. This leads to a better depiction of boundaries.

[0005] Moreover, Patent Document 2 discloses a region discrimination device that discriminates regions based on a saliency map and superpixels. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2018-514024 [Patent Document 2] JP 2019-61658 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the technology described in Patent Document 1, when superpixels in an ROI are replaced with superpixels of a smaller scale, the scale is based on a user's instruction. Therefore, a good segmentation boundary is not always obtained. That is, if the scale is too large, the accuracy of the boundary deteriorates, and if the scale is too small, the workload of selecting superpixels when segmenting an image increases.

[0008] In addition, in Patent Document 2, the scale of the superpixels is not changed. Therefore, when an object of a different size than expected or multiple objects of different sizes exist in the same image, it is not possible to generate superpixels of an appropriate scale, and it is not possible to obtain good accuracy of the segmentation boundary.

[0009] The present invention has been made in consideration of the above problems, and aims to strike a balance between improving boundary accuracy using superpixels and reducing the load of segmentation work when segmenting images in annotation work. [Means for solving the problem]

[0010] In order to achieve the above-mentioned object, the image processing device of the present invention has an input means for inputting image data of an image, an acquisition means for acquiring the size of an object region containing an object to be extracted that is contained in the image, a setting means for setting the number of divisions into which the object region is divided, a determination means for determining a segment size of a superpixel based on the size of the object region and the number of divisions, and a generation means for using the image data to generate a superpixel having a size within a predetermined range that includes the segment size determined by the determination means. Effect of the Invention

[0011] According to the present invention, in image segmentation in annotation work, it is possible to balance between improving boundary accuracy using superpixels and reducing the load of the segmentation work. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing the functional configuration of an image processing system according to first and second embodiments of the present invention. [Diagram 2] FIG. 1 is a diagram showing an example of a hardware configuration of an image processing system according to an embodiment. [Diagram 3] FIG. 2 is a view showing an example of an input image to be analyzed in the first embodiment. [Figure 4] 5 is a flowchart showing the operation of the image processing system according to the first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram regarding designation of a region of a target object in the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of a GUI of application software according to the first embodiment. [Figure 7] 5A to 5C are diagrams showing examples of images during segmentation processing in the first embodiment. [Figure 8] FIG. 11 is a diagram showing an example of a portion of a GUI of application software according to the second embodiment. [Figure 9]FIG. 13 is a block diagram showing the functional configuration of an image processing system according to a third embodiment. [Figure 10] 10 is a flowchart showing the operation of an image processing system according to a third embodiment. [Figure 11] FIG. 13 is a block diagram showing the functional configuration of an image processing system according to a fourth embodiment. [Figure 12] 10 is a flowchart showing the operation of an image processing system according to a fourth embodiment. [Figure 13] FIG. 13 is a partial enlarged view showing an example of an input image in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] <First embodiment> Fig. 1 is a block diagram showing the functional configuration of an image processing system 100 according to a first embodiment of the present invention. As shown in Fig. 1, the image processing system 100 has an image input unit 101, an image processing unit 102, an operation input unit 103, a display unit 104, and a label output unit 105.

[0015] The image input unit 101 inputs an image 106 (image data) to be analyzed to the image processing system 100, and the image processing unit 102 executes application software to perform segmentation processing on the image 106 input to the image input unit 101. The operation input unit 103 is composed of a mouse, a keyboard, a tablet, etc., and an operator operates the operation input unit 103 to input information to the image processing unit 102. The display unit 104 interactively displays the image being processed by the image processing unit 102 and the operation results by the operation input unit 103, etc. The label output unit 105 outputs the segmentation result, which is the processing result of the image processing unit 102, as a label 107. The output label 107 is stored in a storage device (not shown).

[0016] Next, the functional configuration of the image processing unit 102 will be described. The target division number setting unit 121 sets the number of superpixels into which a target object in an image that is to be segmented (extracted) is to be divided (target division number) when the image 106 is subjected to superpixel processing. The condition determination unit 122 determines the superpixel generation condition based on the size of the area of ​​the target object (object area) and the set target division number. Here, the average segment size (average scale) of the superpixels is calculated as the superpixel generation condition. The superpixel generation unit 123 generates superpixels based on the average segment size.

[0017] The superpixel extraction unit 124 performs segmentation using superpixels corresponding to a target object selected by the GUI tool of the application software using the operation input unit 103. The segmentation correction unit 125 corrects errors in the segmentation results by the superpixel extraction unit 124. Specifically, among the superpixels selected by segmentation, a region of superpixels that protrudes from the target object and a region of superpixels that are missing or insufficient are corrected on a pixel-by-pixel basis using a pen tool or the like to bring them closer to the region of the target object.

[0018] Fig. 2 is a block diagram showing the configuration of a computer 200 as an example of the hardware configuration of the image processing system 100. Each function of the image processing system 100 can be realized by the computer 200. The computer 200 includes a central processing unit (CPU) 210, a storage unit 212, an operation input unit 103 (e.g., a mouse, a keyboard, etc.), and a display unit 104 (a display, etc.) shown in Fig. 2.

[0019] The storage unit 212 is composed of a main storage unit 215 (ROM or RAM, etc.) and an auxiliary storage unit 216 (magnetic disk device, SSD: Solid State Drive, etc.). The CPU 210 executes calculations and controls, and executes programs stored in the storage unit 212, thereby functioning as the image processing unit 102 of the image processing system 100 shown in FIG.

[0020] The computer 200 may have one or more CPUs 210 and storage units 212. That is, at least one processing device (CPU) and at least one storage device are connected, and when the at least one processing device executes a program stored in the at least one storage device, the computer 200 functions as the image processing unit 102. The configuration functioning as the image processing unit 102 is not limited to the CPU 210, and may be a field programmable gate array (FPGA) or an application specific integration circuit (ASIC), etc.

[0021] Next, the target division number setting unit 121 in this embodiment will be described in detail with reference to Fig. 3 showing an example of an image to be analyzed. In this embodiment, the image to be analyzed is a dental image, and a dental restoration is a target object for segmentation, but the types of images to be analyzed and target objects are not limited to this. In image 310, object 312 is a dental restoration that is the target object for segmentation.

[0022] The superpixel generation unit 123 executes a superpixel process on the entire image to be analyzed. Images 320 and 330 are images obtained by enlarging a region 311 near the target object from an image in which the result of the superpixel process is superimposed on the input image. When performing the superpixel process, an average segment size is set, which is a setting value indicating the size of the superpixels into which the image is divided. Image 320 is an example of a small average segment size, and image 330 is an example of a large average segment size.

[0023] As the average segment size increases, the size of each superpixel increases, and the extraction load on the superpixel extraction unit 124 decreases. On the other hand, as shown in image 330, superpixels are more likely to protrude from the target object 331, or to be missing or insufficient 332, resulting in lower segment accuracy. Also, the smaller the average segment size, the smaller the size of each superpixel, which increases the extraction load on the superpixel extractor 124, but as can be seen from image 320, the higher the segmentation accuracy for the target object. Thus, there is a trade-off between the workload of superpixel extraction and segmentation accuracy.

[0024] The target division number setting unit 121 sets how many superpixels to divide the target object, that is, in this embodiment, the restoration in the dental image, into (the target division number). For example, in the example of image 320 in Fig. 3, the restoration 312 is divided into about 9 superpixels, and the target division number is 9. On the other hand, in the example of image 330, the restoration 312 is divided into about 4 superpixels, and the target division number is 4.

[0025] Next, the operation of the image processing system 100 in the first embodiment will be described with reference to the flowchart shown in FIG.

[0026] First, in S400, the CPU 210 inputs an image to be analyzed through the image input unit 101.

[0027] In S401, the CPU 210 sets the target division number in the image processing unit 102 by the target division number setting unit 121 by the operator operating the operation input unit 103. As described above, there is a trade-off between the workload of superpixel extraction and the segmentation accuracy. The operator sets the target division number so that the workload of superpixel extraction is not large and the segmentation accuracy is sufficient. In many cases, a balance between the workload and the segmentation accuracy can be maintained by setting the target division number between 10 and 100. In this embodiment, the target division number is described as 50 as an example, but is not limited to this value. When the target division number is set to 50, the operator selects about 50 superpixels that constitute the target object.

[0028] In S402, the CPU 210 designates a target object through the condition determination unit 122 based on the operator's operation of the operation input unit 103. Fig. 5 shows an explanatory diagram relating to the designation of a target object according to the first embodiment.

[0029] As an example, the input image 500 to be analyzed is a dental image, and the target objects 511, 512, and 513 are dental restorations. The area of ​​the target object is specified by roughly surrounding the target object with a closed curve. As an example, the area of ​​the target object 511 may be specified as a circle 501 using a circle drawing tool, and as another example, the area of ​​the target object 512 may be specified as a rectangle 502 using a rectangle drawing tool. As yet another example, the area of ​​the target object 513 may be specified as a closed curve 503 using a freehand tool. When the freehand tool is used, the outline shape of the target object 513 can be obtained.

[0030] When the CPU 210 designates a target object by the above-mentioned method or the like, it acquires the number of pixels (size) of the area within the drawn closed curve as an approximate target size.

[0031] When target objects are the same size, the more complex the shape of the target object, the lower the accuracy of segmentation by superpixels. Therefore, the value of the target division number in S401 may be corrected based on the complexity of the shape of the target object obtained by using the freehand tool. The complexity is calculated using curvature entropy, circularity, etc., and the larger this value is, the more the value of the target division number is corrected. In this way, it is possible to improve the accuracy of segmentation by superpixels by taking the shape of the target object into consideration. Note that a default value may be set for the target division number set in S401, and the target division number may be re-determined based on the shape of the target object obtained in S402.

[0032] As another example, deep learning may be used to perform object detection of teeth and dental diseases, and the region of the target object may be specified by selecting one of the rectangular detection results obtained.

[0033] In S403, average segment size extraction is performed. Here, the CPU 210 obtains the approximate number of pixels (size) of the area of ​​the target object from the closed curve surrounding the target object specified in S402 by the condition determination unit 122, and divides it by the number of target divisions. In this way, the average segment size of the superpixels is calculated.

[0034] In S404, superpixels are generated. Here, the CPU 210 executes a superpixel generation process on the image to be analyzed using the average segment size calculated in S403, by using the superpixel generation unit 123. In this embodiment, LSC (Linear Spectral Clustering) is used as the superpixel algorithm. Other algorithms may be used. For example, the superpixel algorithm may be SEEDS (Superpixels Extracted via Energy-Driven Sampling) or SLIC (Simple Linear Iterative Clustering). Depending on the implementation method, the size of the image to be analyzed may be divided by the average segment size to calculate the number of superpixels in the entire image, and this may be used as the input for the superpixelization process. Furthermore, the segment size of each superpixel may be within a predetermined range including the average segment size calculated in S403.

[0035] In S405, superpixels corresponding to the target object are extracted. Here, the CPU 210 extracts superpixels corresponding to the target object using the superpixel extraction unit 124 based on the operator's work. In this process, the CPU 210 displays the GUI of the application software on the display unit 104, displays the input image on the GUI, and superimposes a layer of the segment map of the generated superpixels on the input image. Then, at a position on the image to be analyzed designated by the mouse or tablet, the superpixel corresponding to that position is displayed, and an area corresponding to the target object is extracted in superpixel units.

[0036] In S406, segment correction is performed. Here, the CPU 210 corrects the area extracted in S405 using the segmentation correction unit 125 based on the work of the operator. Specifically, in the area extracted in superpixel units, areas that protrude from the target object, are corrected in pixel units, and areas that are missing or insufficient are corrected. For the correction, an eraser tool, pen tool, etc. are used.

[0037] Then, if there are N types of approximate target sizes of target objects to be segmented in the input image, the above-mentioned processes from S402 to S406 are performed N times for each approximate target size. In other words, if there are multiple target objects in the input image, the same superpixelization process is performed for target objects having approximately the same approximate target size, and for target objects of different sizes, the area of ​​the target object is redesignated, the conditions for the superpixelization process are changed, and the superpixelization process is performed in a different cycle.

[0038] The types of approximate target sizes can be classified using multiple thresholds. As described above, when the area of ​​the target object is specified by a predetermined shape such as a circle or a rectangle, circles and rectangles of multiple sizes may be prepared in advance, and the approximate target size may be classified by using a circle or rectangle of an appropriate size from among them.

[0039] In S407, the processing result is output. Here, the segment map after correction in S406 is output in association with the input image and saved as a label. The label is saved in the IndexPNG format (also called the palette format), which is often used as training data for semantic segmentation. Furthermore, the label format may be a bitmap format or another format, and can be selected according to the application of the label.

[0040] Next, FIG. 6 shows an example of a GUI (Graphical User Interface) of the application software in this embodiment. A GUI 600 of the application software is displayed on the display unit 104, and various control areas are arranged to realize segmentation using superpixels.

[0041] By pressing the Open button 611, the directory of the image to be segmented can be selected. In ImageList 620, a list of images registered in the directory is displayed, and a selected image from among them is displayed in a picture box 630. In picture box 630, a pointer 631 that can be controlled by operation input unit 103 or a circular pointer 632 is displayed.

[0042] In TargetSettings 690, various tools used in specifying the area of ​​the target object in S402 are arranged. In targetSelectTool 691, which is a radio button group, tools for specifying a closed curve for roughly specifying the area of ​​the target object are arranged. In the present embodiment, as an example, rectangle for specifying a rectangular area, circle for specifying a circular area, FreehandTool for specifying a free area, and Disable are arranged. When Disable is selected, rough specification of the area of ​​the target object is disabled, and when a radio button other than Disable is selected, rough specification of the area of ​​the target object is enabled.

[0043] After obtaining the closed curve, the size of the closed curve area is displayed in targetPixNum693. Also, Nt 692 is an area for inputting the target division number in S401. Then, by pressing targetAreaSet 694, the average segment size calculated from the size of the closed curve and the target division number is displayed in AveSegSize 643.

[0044] In SuperPixelSettings 640, input boxes for setting values ​​of superpixel processing conditions are arranged. In the superpixel algorithm selection tab 641, a superpixel algorithm can be selected. In the example of FIG. 6, LSC is selected as the superpixel algorithm. In addition, in the LSC algorithm tab, NumIterations 642, which is a setting value of the number of iterations used in the clustering phase of the algorithm, AveSegSize (Average Segment Size) 643, which is a setting value of the square root of the average segment size (number of pixels) of superpixels, CFactor (Compactness Factor) 644, which is a setting value related to the shape of superpixels, and MinElementSize (Minimum Element Size) 645, which is a setting value of the minimum superpixel segment size, are arranged.

[0045] CFactor644 controls the shape of the superpixels; the higher the value, the more regular the shape of the superpixels. MinElementSize 645 represents the minimum segment size of a superpixel, and superpixels smaller than this size are absorbed into larger superpixels. Changing the MinElementSize setting affects the number of superpixel divisions, but the average segment size obtained in S403 is calculated taking this effect into account.

[0046] Note that these specific settings are merely examples and can be changed according to the characteristics of the image to be segmented. In addition, since there are multiple setting items in SuperPixelSettings 640, each setting value may be recorded in a json file and read out all at once by selecting the ReadParamFile 646 button or the like. Then, by pressing a CalculateSuperPixel button 647 , a superpixel process is executed on the image selected and displayed in the picture box 630 .

[0047] FIG. 7 shows an example of an image during the segmentation process. Image 710 shows the result of the superpixeling process. Image 720 shows the input image with a superpixel segment map superimposed. Also superimposed on the restoration 712 are the selected superpixels and the corrected portion 721. The corrected portion 721 is superimposed in the color corresponding to the selected index. Image 730 shows the superpixels selected by the user in image 720 and the correction results output as labels. When the CalculateSuperPixel button 647 is pressed, an image 720 is displayed in the picture box 630 .

[0048] In PaintTool 650 in FIG. 6, various tools (controls) used in the superpixel extraction process in S405 and the segment correction process in S406 are arranged.

[0049] SuperPixel 651 is a tool used in the superpixel extraction process in S405, and when you click on the picture box 630 with a mouse, tablet, or the like, a superpixel corresponding to the specified coordinates is extracted and the corresponding superpixel is superimposed and displayed. An example of the superimposed display is image 720 in FIG. 7. Pen 652 is a tool used in the segment correction process in S406, and when a mouse, tablet, or the like is dragged on the picture box 630, an area corresponding to the specified coordinates is extracted freehand. Note that the PaintTool is not limited to Pen 652 or SuperPixel 651, and may be a fill tool for filling in a closed area, or a tool for drawing shapes such as rectangles, circles, and triangles.

[0050] ColorIndex 660 is an area that specifies an index for labeling in the superpixel extraction in S405 and the segment correction in S406. The Eraser 661 performs labeling with Index 0. In this embodiment, Index 0 indicates a background label, and the Eraser 661 is a so-called eraser tool. When a color index is specified in the combo box 663, the color of the palette corresponding to the index is displayed in an area 664. Note that the index can be set to a value from 0 to 255, and for example, PascalVOC2012 may be the index value of the color map corresponding to the data set.

[0051] DisplaySettings 670 includes an area where labels and SuperPixel display can be turned on and off to facilitate labeling in the superpixel extraction process in S405 and the segment correction process in S406.

[0052] When the check box overlayLabel 671 is OFF, the label is not displayed over the picture box, and when it is ON, the label is displayed over the picture box with the transparency specified in the numeric setting box 672. This allows you to work while checking whether the labeling is working properly. When the check box OverlaySuperpixel 673 is ON, the superpixel is displayed, and when it is OFF, the superpixel is not displayed.

[0053] The Save button 680 converts the label, which is the processing result generated on the picture box 630, into a predetermined format based on the output in S407 and saves it.

[0054] These processes may be performed by resizing (reducing) the input image to a smaller size, and then resizing the output label to the original size. The resizing algorithm may use, for example, the nearest neighbors method, and after resizing, the boundaries may be smoothed by performing "erosion" and "dilation" processes, which are enlargement and reduction processes in morphology processing. In this way, by resizing to a smaller size once, the processing load of the superpixelization process can be reduced. However, if the image size is resized too small, the boundary accuracy deteriorates, so it is desirable to consider whether to perform resizing for each target object.

[0055] These configurations make it possible to balance the improvement of boundary accuracy using superpixels and the reduction of the load of segmentation work when segmenting images in annotation work.

[0056] <Second embodiment> Next, a second embodiment of the present invention will be described. Fig. 8 is a diagram showing a part of the GUI of the application software in the second embodiment, and shows targetSelectTool891 displayed in place of targetSelectTool691 in the GUI of the application software shown in Fig. 6. Other than this, the configuration of the GUI of the application software is the same as that shown in Fig. 6. Also, since the image processing system 100 in the second embodiment is the same as that shown in Figs. 1 and 2, a description thereof will be omitted.

[0057] In the second embodiment, as shown in targetSelectTool891, as a target region designation method used by the condition determination unit 122, SuperPixel805 is provided in addition to Disable, rectangle, circle, and FreeHand.

[0058] Next, the operation of the image processing system 100 in the second embodiment will be described, with reference to FIG. 4, focusing only on the processing that differs from the processing in the first embodiment. When SuperPixel 805 is selected in S402 of Fig. 4, the condition determination unit 122 uses the superpixel to specify the target object. First, the input image is subjected to superpixel processing using a first average segment size. The first average segment size is set by AveSegSize 806 of SuperPixel 805, and is a value larger than a second average segment size used in S404, which will be described later. Then, the target object is specified by selecting a target superpixel on the picture box 630 using the operation input unit 103.

[0059] By setting the first average segment size to a value larger than the second average segment size, the area of ​​the target object can be roughly specified with coarse superpixels, reducing the workload, and when generating superpixels in S404, high-precision processing can be achieved by using fine superpixels.

[0060] When the target object is specified by selecting a large superpixel, the approximate shape of the target object is known, so the value of the target division number in S401 may be corrected based on the complexity of the shape of the target object, as in the first embodiment. The complexity is calculated using curvature entropy, circularity, etc., and the larger this value is, the more the target division number value is corrected. In this way, the segmentation accuracy of the superpixel can be improved by taking the shape of the target object into consideration.

[0061] As described above, according to the second embodiment, the approximate size of the area of ​​the target object can be obtained with a smaller load, and the size can be obtained in a form that reflects the shape of the object more closely than tools such as a rectangle or a circle. This allows for superpixel processing with a more appropriate average segment size, and improves boundary accuracy.

[0062] <Third embodiment> Next, a third embodiment of the present invention will be described.

[0063] Fig. 9 is a block diagram showing the functional configuration of an image processing system 900 in the third embodiment. As shown in Fig. 9, the third embodiment differs from the image processing system 100 in the first embodiment shown in Fig. 1 in that a division number correction unit 901 is newly provided. The other configurations are similar to those described in the first embodiment, so the same reference numbers are used and the description will be omitted.

[0064] The division number correction unit 901 has a function of correcting the average segment size of a superpixel. Specifically, it takes statistics on the usage of the eraser tool Eraser 661 for each index (object), that is, the number of corrections, and generates a correction amount for the target division number specified in S401 based on the statistical value. Furthermore, it has a function of correcting the target division number Nt 692 based on the correction amount when the next superpixel of the same index (object) is generated. Here, Eraser 661 means labeling with the index value of the background.

[0065] Next, the operation of the image processing system 900 in the third embodiment will be described with reference to the flow chart shown in Fig. 10. As shown in Fig. 10, the third embodiment is different in that, in addition to the processing in the first embodiment shown in Fig. 4, a division number correction in S1001 and a division number correction amount acquisition in S1002 are newly added. Since the other processing is similar to the processing shown in Fig. 4, the same step numbers are used and the description will be omitted.

[0066] In the division number correction in S1001, after the region of the target object is specified, the CPU 210 specifies the index of the object to which the region of the target object belongs by the division number correction unit 901. The index is specified by a GUI of the application software (not shown). Then, if there is a division number correction amount obtained in S1002 for the specified index, the target division number Nt 692 is corrected. The division number correction amount is calculated based on usage statistics of the segmentation correction unit 125. When the correction amount of index i is Hi, the target division number is corrected by multiplying the target division number Nt by Hi.

[0067] In S1002, the division number correction amount is calculated based on usage statistics of the segmentation correction unit 125. Specifically, the CPU 210 uses the division number correction unit 901 to take statistics on the usage status of the Eraser 661, and generates a correction amount for the target division number specified in S401 for each index (object) based on the statistical value.

[0068] For example, when the average number of times the eraser tool is used per object for all indexes is Na, the average number of times the eraser tool is used per object for index i is Ni, and a coefficient controlling the magnitude of the correction amount is k, the correction amount Hi for index i may be calculated as shown in equation (1).

[0069] Hi = 1 + k (Ni / Na - 1) … (1) In this case, when the number of times the eraser tool is used for index i is average, that is, when Ni = Na, Hi = 1 and the target division number is not corrected. When the number of times the eraser tool is used for index i is higher than average, that is, when Ni > Na, Hi becomes larger than 1, the target division number Nt is corrected to a large value, and the image segmentation becomes more accurate.

[0070] Conversely, when the number of times the eraser tool for index i is used is less than the average, that is, when Ni < N, Hi becomes less than 1, the target number of divisions Nt is corrected to a small value, and the load of image segmentation is corrected in the direction of reduction.

[0071] In addition, when Ni < Na, that is, when Hi is less than 1, if the target number of divisions Nt is corrected, there is a concern that the target number of divisions is corrected to a small value and the image segmentation accuracy decreases. Therefore, it may be acceptable not to correct when Ni < Na.

[0072] In addition, for the acquisition of the division number correction amount in S1002, if effective correction parameters have been obtained, it is not necessary to perform it thereafter, and it may be sufficient to only use the obtained correction parameters in S1001.

[0073] As described above, according to the third embodiment, since the segmentation accuracy is considered to be low for an object with a high usage of the eraser tool, in the next segmentation, the target number of divisions of the same object is corrected to be larger. As a result, more accurate segmentation becomes possible, and in the next segmentation, it becomes possible to reduce the number of times the eraser tool is used.

[0074] <Fourth Embodiment> Next, a fourth embodiment of the present invention will be described.

[0075] FIG. 11 is a block diagram showing the functional configuration of an image processing system 1100 in the fourth embodiment. As shown in FIG. 11, in the fourth embodiment, in addition to the components of the image processing system 100 of the first embodiment shown in FIG. 1, a reflection region acquisition unit 1101 is newly provided, which is different. Since the other configurations are the same as those described in the first embodiment, the same reference numerals are given and the description is omitted.

[0076] In the following description, the case where the input image 106 is a dental image will be described. In dental images, saliva can cause specular reflections, which can cause objects in the intraoral photograph to appear different in color and texture. Figure 13 shows an example of a partial enlargement of an input image, showing an example of a specular reflection portion 1301 in the image. If the specular reflection portion 1301 is located on the boundary of a target object to be segmented, it can impair the accuracy of the segmentation.

[0077] The reflection area acquisition unit 1101 acquires a specular reflection area contained in a dental image, and highlights the reflection area in the picture box 630 displayed on the display unit 104 .

[0078] Fig. 12 is a flowchart showing the operation of the image processing system 1100 in the fourth embodiment. As shown in Fig. 12, the fourth embodiment is different in that, in addition to the processing in the first embodiment shown in Fig. 4, a reflection area highlighting display in S1201 is newly provided. Since the other processing is similar to the processing shown in Fig. 4, the same step numbers are assigned and the description will be omitted.

[0079] The reflective area highlighting display in S1201 judges pixels having a luminance equal to or greater than a certain threshold in the image histogram as reflective areas, and superimposes and highlights the reflective areas on the image in the picture box 630. Note that the reflective area may be judged based on the RGB gradation value.

[0080] As described above, according to the fourth embodiment, even if there is reflection in the intraoral photograph, the specular reflection area is highlighted, so that the superpixel extraction and segment correction in S405 and S406 can be performed while paying attention to the specular reflection part. This makes it possible to realize highly accurate segmentation.

[0081] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0082] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0083] <Summary> The disclosure of this embodiment includes the following configuration.

[0084] (Configuration 1) An input means for inputting image data of an image; An acquisition means for acquiring a size of an object region including an object to be extracted that is included in the image; A setting means for setting the number of divisions into which the object region is divided; A determining means for determining a segment size of a superpixel based on the size of the object region and the number of divisions; a generating means for generating superpixels having a size in a predetermined range including the segment size determined by the determining means, using the image data; 13. An image processing device comprising:

[0085] (Configuration 2) a selection means for selecting a superpixel corresponding to the object from among the generated superpixels; a correction means for correcting a region formed by the selected superpixels so as to approach a region of the object; an output means for outputting a label that associates an area formed by the corrected superpixels with the object; 2. The image processing device according to configuration 1, further comprising:

[0086] (Configuration 3) 3. The image processing apparatus according to configuration 2, wherein the setting means, the selection means, and the correction means are configured by a GUI (Graphical User Interface) and an operation member.

[0087] (Configuration 4) Further comprising a reduction means for reducing the image, The image processing device according to configuration 2 or 3, characterized in that the generating means, the selecting means, and the correcting means process the image reduced by the reducing means, and the outputting means resizes the area constituted by the corrected superpixels to the size before reduction, and then outputs the label.

[0088] (Configuration 5) The image processing device according to configuration 4, wherein the output means resizes the area formed by the corrected superpixels using a nearest neighbors method, performs a scaling process in morphological processing to correct the boundary of the resized area, and then outputs the label.

[0089] (Configuration 6) The image processing device according to any one of configurations 1 to 5, wherein the generating means generates superpixels using any one of the following algorithms: LS (Linear Spectral Clustering), SEEDS (Superpixels Extracted via Energy-Driven Sampling), and SLIC (Simple Linear Iterative Clustering).

[0090] (Configuration 7) The method further includes a designation means for designating the object region, the designation means designates the object region by any one of the following means: a means for a user to designate the object region with a rectangle, a means for designating the object region with a circle, a means for designating the object region freehand, or a means for selecting a rectangle from one of the detection results of object detection, or a combination thereof; 7. The image processing device according to claim 1, wherein the acquisition means acquires the size of the specified object region.

[0091] (Configuration 8) 8. The image processing device according to configuration 7, wherein the designation means is configured by a GUI (Graphical User Interface) and an operation member.

[0092] (Configuration 9) 9. The image processing device according to claim 7 or 8, wherein when the object region is specified freehand by the specifying means, the setting means sets a larger number of divisions as the shape of the object region becomes more complicated.

[0093] (Configuration 10) a designation unit that uses the image data to generate superpixels of a predetermined size and selects a superpixel corresponding to the object from among the generated superpixels to designate the object region; 10. The image processing device according to any one of claims 1 to 9, wherein the predetermined size is larger than the segment size determined by the determining means.

[0094] (Configuration 11) 11. The image processing device according to configuration 10, wherein the means for selecting a superpixel corresponding to the object among the designation means is configured by a GUI (Graphical User Interface) and an operation member.

[0095] (Configuration 12) 12. The image processing device according to claim 10, wherein the setting means sets a larger number of divisions as the complexity of the shape of the object region designated by the designation means increases.

[0096] (Configuration 13) The method further includes a division number correction means for correcting the division number set by the setting means, The division number correction means corrects the division number in accordance with an amount of correction by the correction means, 3. The image processing device according to claim 2, wherein, when the number of divisions has been corrected by the number of divisions correction means, the determination means determines the segment size using the corrected number of divisions.

[0097] (Configuration 14) The image processing device according to configuration 13, wherein the division number correction means increases the division number when the correction amount is greater than a predetermined threshold, and decreases the division number when the correction amount is smaller than a predetermined threshold.

[0098] (Configuration 15) 15. The image processing device according to any one of configurations 1 to 14, wherein the input image is a dental image.

[0099] (Configuration 16) A determination means for determining pixels having a luminance equal to or greater than a predetermined threshold value using the image data; and a control means for controlling the image to be displayed on a display means, 16. The image processing device according to any one of configurations 1 to 15, wherein the control means superimposes and displays the pixels determined by the determination means on the image displayed on the display means in an emphasized manner.

[0100] (Configuration 17) an input step of inputting image data of an image; obtaining a size of an object region including an object to be extracted that is included in the image; obtaining a number of divisions into which the object region is divided; determining a superpixel segment size based on the size of the object region and the number of divisions; using the image data to generate superpixels having a predetermined range of sizes that includes the determined segment size; 13. An image processing method comprising:

[0101] (Configuration 18) A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 16.

[0102] (Configuration 19) A computer-readable storage medium storing the program according to configuration 18.

[0103] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0104] 100, 900, 1100: image processing system, 101: image input unit, 102: image processing unit, 103: operation input unit, 104: display unit, 105: label output unit, 106: input image, 107: label, 121: target division number setting unit, 122: condition determination unit, 123: superpixel generation unit, 124: superpixel extraction unit, 125: segmentation correction unit, 901: division number correction unit, 1101: reflection area acquisition unit

Claims

1. An input means for inputting image data of an image; An acquisition means for acquiring a size of an object region including an object to be extracted that is included in the image; A setting means for setting the number of divisions into which the object region is divided; A determining means for determining a segment size of a superpixel based on the size of the object region and the number of divisions; a generating means for generating superpixels having a size in a predetermined range including the segment size determined by the determining means, using the image data; 13. An image processing device comprising:

2. a selection means for selecting a superpixel corresponding to the object from among the generated superpixels; a correction means for correcting a region formed by the selected superpixels so as to approach a region of the object; an output means for outputting a label that associates an area formed by the corrected superpixels with the object; 2. The image processing apparatus according to claim 1, further comprising:

3. 3. The image processing apparatus according to claim 2, wherein the setting means, the selection means, and the correction means are configured by a GUI (Graphical User Interface) and an operation member.

4. Further comprising a reduction means for reducing the image, 3. The image processing device according to claim 2, wherein the generating means, the selecting means, and the correcting means process the image reduced by the reducing means, and the outputting means resizes the area constituted by the corrected superpixels to the size before reduction, and then outputs the label.

5. 5. The image processing device according to claim 4, wherein the output means resizes the area constituted by the corrected superpixels using a nearest neighbors method, performs a scaling process in morphological processing to correct the boundary of the resized area, and then outputs the label.

6. 2. The image processing device according to claim 1, wherein the generating means generates superpixels using any one of the following algorithms: Linear Spectral Clustering (LS), Superpixels Extracted via Energy-Driven Sampling (SEEDS), and Simple Linear Iterative Clustering (SLIC).

7. The method further includes a designation means for designating the object region, the designation means designates the object region by any one of the following means: a means for a user to designate the object region with a rectangle, a means for designating the object region with a circle, a means for designating the object region freehand, or a means for selecting a rectangle from one of the detection results of object detection, or a combination thereof; 2. The image processing apparatus according to claim 1, wherein said acquisition means acquires a size of the specified object area.

8. 8. The image processing apparatus according to claim 7, wherein the designation means comprises a GUI (Graphical User Interface) and an operation member.

9. 8. The image processing device according to claim 7, wherein, when the object area is specified freehand by the specifying means, the setting means sets a larger number of divisions as the shape of the object area becomes more complicated.

10. a designation unit that uses the image data to generate superpixels of a predetermined size and selects a superpixel corresponding to the object from among the generated superpixels to designate the object region; 2. The image processing apparatus according to claim 1, wherein the predetermined size is larger than the segment size determined by the determining means.

11. 11. The image processing apparatus according to claim 10, wherein the means for selecting the superpixel corresponding to the object among the designation means is configured by a GUI (Graphical User Interface) and an operation member.

12. 11. The image processing apparatus according to claim 10, wherein the setting means sets a larger number of divisions as the degree of complexity of the shape of the object region designated by the designation means increases.

13. The method further includes a division number correction means for correcting the division number set by the setting means, The division number correction means corrects the division number in accordance with an amount of correction by the correction means, 3. The image processing apparatus according to claim 2, wherein, when the division number has been corrected by the division number correction means, the determination means determines the segment size using the corrected division number.

14. 14. The image processing device according to claim 13, wherein the division number correction means increases the division number when the correction amount is greater than a predetermined threshold value, and decreases the division number when the correction amount is smaller than a predetermined threshold value.

15. 2. The image processing apparatus according to claim 1, wherein the input image is a dental image.

16. A determination means for determining pixels having a luminance equal to or greater than a predetermined threshold value using the image data; and a control means for controlling the image to be displayed on a display means, 2. The image processing apparatus according to claim 1, wherein the control means superimposes and displays the pixels determined by the determining means on the image displayed on the display means in an emphasized manner.

17. an input step of inputting image data of an image; obtaining a size of an object region including an object to be extracted that is included in the image; obtaining a number of divisions into which the object region is divided; determining a superpixel segment size based on the size of the object region and the number of divisions; using the image data to generate superpixels having a predetermined range of sizes that includes the determined segment size; 13. An image processing method comprising:

18. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 16.

19. A computer-readable storage medium storing the program according to claim 18.

Images