Image processing apparatus and method, program and storage medium

Abstract

To balance between improving boundary accuracy using superpixels and reducing the load of segmentation work in image segmentation in annotation work.SOLUTION: An image processing system comprises: input means to which image data on an image is input; acquisition means that acquires the size of an object area including an object to be extracted included in the image; setting means for setting a division number by which to divide the object area; determination means that determines a superpixel segment size on the basis of the size of the object area and the division number; and generation means that uses the image data to generate superpixels having a predetermined range of sizes including the segment size determined by the determination means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0006] , , , , , ,

[0001] The present invention relates to an image processing apparatus and method, a program, and a storage medium, and particularly relates to an image segmentation technique.

Background Art

[0002] Conventionally, semantic segmentation in machine learning has been known. Semantic segmentation is a task of dividing an image into regions of a plurality of objects, and in order to perform semantic segmentation, an image is detailed and output in pixel units. At this time, the teacher data for semantic segmentation needs to be labeled for each pixel.

[0003] Therefore, in semantic segmentation, since the load of the annotation work (the work of labeling an image and creating teacher data) is large, superpixels are used to reduce the work load. A superpixel is a small region in which pixels with similar colors and textures are grouped.

[0004] For example, in Patent Document 1, a method of segmenting an image using superpixels of a hybrid scale is disclosed. Specifically, a user replaces superpixels of a larger scale with superpixels of a smaller scale (the region size of the superpixels) in a region of interest (ROI) for improving segmentation. Thereby, a method of achieving better boundary description is disclosed.

[0005] Further, in Patent Document 2, a region discrimination device that discriminates regions based on a saliency map and superpixels is disclosed.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Special Publication No. 2018-514024 [Patent Document 2] Japanese Patent Publication No. 2019-61658 [Overview of the project] [Problems that the invention aims to solve]

[0007] However, in the technology described in Patent Document 1, when replacing superpixels within an ROI with superpixels of a smaller scale, the size of the scale is based on user input. Therefore, a good segmentation boundary is not always obtained. In other words, if the scale is too large, the accuracy of the boundary deteriorates, and if the scale is too small, the workload of selecting superpixels when segmenting the image increases.

[0008] Furthermore, Patent Document 2 does not change the scale of the superpixels. Therefore, if there are objects of different sizes than expected, or multiple objects of different sizes, in the same image, it is not possible to generate superpixels of the appropriate scale, and good segmentation boundary accuracy cannot be obtained.

[0009] This invention was made in view of the above-mentioned problems, and aims to balance the improvement of boundary accuracy using super pixels with the reduction of the burden of segmentation work in image segmentation during annotation work. [Means for solving the problem]

[0010] To achieve the above objective, the image processing apparatus of the present invention includes an input means for inputting image data of an image; an acquisition means for acquiring the size of an object region containing an object to be extracted in the image; a setting means for setting the number of divisions for dividing the object region; a determination means for determining the segment size of a superpixel based on the size of the object region and the number of divisions; and a generation means for generating a superpixel having a predetermined size range including the segment size determined by the determination means, using the image data. [Effects of the Invention]

[0011] According to the present invention, it is possible to balance improved boundary accuracy using superpixels with a reduction in the workload of image segmentation during annotation work. [Brief explanation of the drawing]

[0012] [Figure 1] A block diagram showing the functional configuration of an image processing system according to the first and second embodiments of the present invention. [Figure 2] A diagram showing an example of the hardware configuration of an image processing system in an embodiment. [Figure 3] A figure showing an example of an input image to be analyzed in the first embodiment. [Figure 4] A flowchart illustrating the operation of the image processing system in the first embodiment. [Figure 5] An explanatory diagram illustrating the specification of the target object region in the first embodiment. [Figure 6] A diagram showing an example of the GUI of the application software in the first embodiment. [Figure 7] A diagram showing an example of an intermediate image during the segmentation process in the first embodiment. [Figure 8] A diagram showing an example of a part of the GUI of the application software in the second embodiment. [Figure 9]Block diagram showing the functional configuration of the image processing system in the third embodiment. [Figure 10] Flowchart showing the operation of the image processing system in the third embodiment. [Figure 11] Block diagram showing the functional configuration of the image processing system in the fourth embodiment. [Figure 12] Flowchart showing the operation of the image processing system in the fourth embodiment. [Figure 13] Partial enlarged view showing an example of an input image in the fourth embodiment.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant explanations are omitted.

[0014] <The First Embodiment> FIG. 1 is a block diagram showing the functional configuration of an image processing system 100 in the first embodiment of the present invention. As shown in FIG. 1, the image processing system 100 includes an image input unit 101, an image processing unit 102, an operation input unit 103, a display unit 104, and a label output unit 105.

[0015] The image input unit 101 inputs an image 106 (image data) to be analyzed into the image processing system 100. The image processing unit 102 executes application software on the image 106 input by the image input unit 101 to perform segmentation processing. The operation input unit 103 is composed of a mouse, a keyboard, a tablet, etc., and an operator operates the operation input unit 103 to input information to the image processing unit 102. The display unit 104 interactively displays the image being processed by the image processing unit 102 and the operation results by the operation input unit 103. The label output unit 105 outputs the segmentation result, which is the processing result of the image processing unit 102, as a label 107. The output label 107 is stored in a storage device (not shown).

[0016] Next, the functional configuration of the image processing unit 102 will be described. The target division number setting unit 121 sets the number of superpixels into which a target object (extraction target) to be segmented in the image is divided (target division number) when performing superpixelization processing on the image 106. The condition determination unit 122 determines superpixel generation conditions based on the size of the region of the target object (object region) and the set target division number. Here, as the superpixel generation condition, the average segment size (average scale) of the superpixels is calculated. The superpixel generation unit 123 generates superpixels based on the average segment size.

[0017] The superpixel extraction unit 124 performs segmentation using superpixels corresponding to the target object selected using the operation input unit 103 via the GUI tool of the application software. The segmentation correction unit 125 corrects errors in the segmentation results from the superpixel extraction unit 124. Specifically, it corrects areas of superpixels that extend beyond the target object, or areas of missing or insufficient superpixels, on a pixel-by-pixel basis using a pen tool or the like, bringing them closer to the area of ​​the target object.

[0018] Figure 2 is a block diagram showing the configuration of a computer 200 as an example of the hardware configuration of an image processing system 100. Each function of the image processing system 100 can be realized by the computer 200. The computer 200 comprises a central processing unit (CPU) 210, a storage unit 212, an operation input unit 103 (e.g., mouse, keyboard, etc.), and a display unit 104 (display, etc.) as shown in Figure 2.

[0019] The storage unit 212 consists of a main storage unit 215 (ROM or RAM, etc.) and an auxiliary storage unit 216 (magnetic disk drive, SSD: Solid State Drive, etc.). The CPU 210 performs calculations and control, and executes programs stored in the memory unit 212, thereby functioning as the image processing unit 102 of the image processing system 100 shown in Figure 1.

[0020] The computer 200 may have one CPU 210 and one memory unit 212. That is, at least one processing unit (CPU) and at least one memory unit are connected, and when at least one processing unit executes a program stored in at least one memory unit, the computer 200 functions as an image processing unit 102. The configuration for the image processing unit 102 is not limited to a CPU 210, but may also include an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integration Circuit), etc.

[0021] Next, the details of the target division number setting unit 121 in this embodiment will be explained using Figure 3, which shows an example of an image to be analyzed. In this embodiment, the image to be analyzed is a dental image and the dental restoration is used as the target object for segmentation, but the types of images to be analyzed and target objects are not limited to these. In image 310, object 312 is a dental restoration, which is the target object for segmentation.

[0022] The superpixel generation unit 123 performs superpixel processing on the entire image to be analyzed. Images 320 and 330 are images obtained by superimposing the results of the superpixel processing onto the input image and then zooming in on the region 311 near the target object. During the superpixel processing, an average segment size is set, which is a setting value that indicates how large the superpixels will be to divide the image. Image 320 is an example of a small average segment size, and image 330 is an example of a large average segment size.

[0023] The larger the average segment size, the larger the size of each superpixel, and the lower the workload on the superpixel extraction unit 124. On the other hand, as shown in image 330, superpixels are more likely to overflow 331 from the target object, or be missing or incomplete 332, resulting in lower segment accuracy. Furthermore, while a smaller average segment size results in smaller sizes for each superpixel, increasing the workload on the superpixel extraction unit 124, the segmentation accuracy for the target object remains high, as can be seen from image 320. Thus, there is a trade-off between the workload of superpixel extraction and segment accuracy.

[0024] The target division number setting unit 121 sets how many superpixels (target division number) to divide the target object, that is, in this embodiment, the restoration in the dental image. For example, in the example of image 320 in Figure 3, the restoration 312 is divided into approximately 9 superpixels, so the target division number is 9. On the other hand, in the example of image 330, the restoration 312 is divided into approximately 4 superpixels, so the target division number is 4.

[0025] Next, the operation of the image processing system 100 in the first embodiment will be explained using the flowchart shown in Figure 4.

[0026] First, in S400, the CPU 210 receives the image to be analyzed via the image input unit 101.

[0027] In S401, the CPU 210 is configured by the operator operating the operation input unit 103 to set the number of target divisions in the image processing unit 102 using the target division number setting unit 121. As mentioned above, there is a trade-off between the workload of superpixel extraction and segment accuracy. The operator sets the number of target divisions such that the workload of superpixel extraction is not too high and the segment accuracy is sufficient. In many cases, setting the number of target divisions between 10 and 100 allows for a good balance between workload and segment accuracy. In this embodiment, the number of target divisions is described as 50 as an example, but it is not limited to this value. When the number of target divisions is set to 50, the operator will select approximately 50 superpixels that make up the target object.

[0028] In S402, the CPU 210 specifies the target object using the condition determination unit 122 based on the operator's operation input unit 103. Figure 5 shows an explanatory diagram regarding the specification of the target object according to the first embodiment.

[0029] As an example, the input image 500 to be analyzed is a dental image, and the target objects 511, 512, and 513 are dental restorations. Specifying the area of ​​the target object is done by roughly enclosing the target object with a closed curve. For example, the area of ​​the target object 511 can be specified using the circle drawing tool, as shown in circle 501, or, as another example, the area of ​​the target object 512 can be specified using the rectangle drawing tool, as shown in rectangle 502. area You may also specify it. As yet another example, you may specify the area of ​​the target object 513, such as the closed curve 503, using the freehand tool. When using the freehand tool, you can obtain the approximate shape of the target object 513.

[0030] When a target object is specified using the method described above, CPU210 obtains the number of pixels (size) of the region within the drawn closed curve as the approximate target size.

[0031] If the target objects are the same size, the segmentation accuracy using superpixels decreases as the shape of the target object becomes more complex. Therefore, the value of the number of target divisions in S401 may be corrected based on the complexity of the shape of the target object obtained using the freehand tool. Complexity is calculated using curvature entropy, roundness, etc., and the larger this value, the greater the correction to the number of target divisions. In this way, the segmentation accuracy using superpixels can be improved by taking into account the shape of the target object. Alternatively, a default value may be set for the number of target divisions set in S401, and the number of target divisions may be re-determined in S402 based on the shape of the target object obtained.

[0032] Alternatively, as another example, one could use deep learning to perform object detection on teeth or dental diseases, and then select one of the resulting rectangle detection results to specify the region of the target object.

[0033] In S403, average segment size calculation The output is then performed. Here, the CPU 210 obtains the approximate number of pixels (size) of the target object's region from the closed curve surrounding the target object specified in S402 by the condition determination unit 122, and divides this by the target division number. This calculates the average segment size of the superpixels.

[0034] In S404, superpixels are generated. Here, the CPU 210, using the superpixel generation unit 123, performs a superpixelization process on the image to be analyzed using the average segment size calculated in S403. In this embodiment, LSC (Linear Spectal Clustering) is used as the superpixel algorithm. Other algorithms may also be used. For example, SEEDS (Superpixels Extracted via Energy-Driven Sampling) or SLIC (Simple Linear Iterative Clustering) may be used as superpixel algorithms. Depending on the implementation method, the number of superpixels in the entire image may be calculated by dividing the size of the image to be analyzed by the average segment size, and this can be used as input for the superpixelization process. Furthermore, the segment size of each superpixel only needs to fall within a predetermined range that includes the average segment size calculated in S403.

[0035] In S405, superpixels corresponding to the target object are extracted. Here, the CPU 210, using the superpixel extraction unit 124, extracts superpixels corresponding to the target object based on the operator's input. In this process, the CPU 210 displays the application software's GUI on the display unit 104, displays the input image on the GUI, and overlays a layer of the generated superpixel segment map onto the input image. Then, the superpixels corresponding to the positions specified by the mouse or tablet on the image to be analyzed are displayed, and the region corresponding to the target object is extracted on a superpixel-by-superpixel basis.

[0036] In S406, segment correction is performed. Here, the CPU 210 corrects the region extracted in S405 based on the operator's work using the segmentation correction unit 125. Specifically, it corrects areas that extend beyond the target object, or areas that are missing or insufficient, on a pixel-by-pixel basis within the region extracted in superpixel units. Tools such as the eraser tool and pen tool are used during the correction.

[0037] Then, the processes from S402 to S406 described above are performed N times for each approximate target size if there are N types of approximate target sizes for the target objects to be segmented in the input image. In other words, if there are multiple target objects in the input image, the same super-pixel processing is performed on target objects with approximately the same approximate target size for segmentation, and for target objects of different sizes, the region of the target object is re-specified, the conditions for the super-pixel processing are changed, and the super-pixel processing is performed in a separate cycle.

[0038] Furthermore, the approximate target size can be classified using multiple thresholds. Also, as mentioned above, when specifying the area of ​​a target object using a predetermined shape such as a circle or rectangle, it is possible to prepare circles and rectangles of multiple sizes in advance and classify the approximate target size by using a circle or rectangle of an appropriate size from among them.

[0039] In S407, the processing results are output. Here, the corrected segment map from S406 is output, associated with the input image, and saved as a label. The label is saved in IndexPNG format (also known as palette format), which is commonly used as training data for semantic segmentation. Furthermore, the label format can be bitmap or other formats, and can be selected according to the label's purpose.

[0040] Next, Figure 6 shows an example of the GUI (Graphical User Interface) of the application software in this embodiment. The GUI600 application software is displayed on the display unit 104 and contains various control areas for realizing segmentation using super pixels.

[0041] By pressing the Open button 611, you can select the directory of the images to be segmented. ImageList620 displays a list of images registered in the directory, and the selected image is displayed in PictureBox630. PictureBox630 displays a pointer 631 or a circular pointer 632 that can be controlled by the operation input unit 103.

[0042] TargetSetting g6 90 contains the tools used to specify the area of ​​the target object in S402. The radio button group is targetSelect t6 91 contains tools for specifying a closed curve to roughly define the area of ​​the target object. In this embodiment, as an example, there are rectangle for specifying a rectangular area, circle for specifying a circular area, FreehandTool for specifying a free area, and Disable. If Disable is selected, the rough definition of the target object's area is disabled, and if any radio button other than Disable is selected, the rough definition of the target object's area is enabled.

[0043] After obtaining the closed curve, the size of the closed curve region is displayed in targetPixNum693. Furthermore, Nt692 is the area in S401 for inputting the number of target divisions. Then, by pressing targetAreaSet694, the average segment size calculated from the closed curve size and the number of target divisions is displayed in AveSegSize643.

[0044] SuperPixelSetting g6Section 40 contains input boxes for setting the processing conditions for superpixels. The Superpixel Algorithm Selection tab 641 allows you to select a superpixel algorithm. In the example in Figure 6, LSC is selected as the superpixel algorithm. The LSC algorithm tab also contains settings for NumIterations 642, which is the number of iterations used in the clustering phase of the algorithm; AveSegSize (Average Segment Size) 643, which is the square root of the average segment size (number of pixels) of superpixels; CFactor (Compactness Factor) 644, which is a setting related to the shape of superpixels; and MinElementSize (Minimum Element Size) 645, which is the segment size of the smallest superpixel.

[0045] CFactor644 controls the shape of superpixels; the higher the value, the more regular the shape of the superpixels becomes. MinElementSize645 represents the minimum segment size of a superpixel; superpixels smaller than this size are absorbed into larger superpixels. Changing the MinElementSize setting affects the number of superpixel divisions, but the average segment size obtained in S403 takes this effect into account.

[0046] Note that these specific settings are merely examples and can be changed to suit the characteristics of the image being segmented. Also, SuperPixelSetting g6 Since there are multiple 40 setting items, you could record each setting value in a JSON file and allow it to be selected and read all at once using a button such as ReadParamFile646. Then, by pressing the CalculateSuperPixel button 647, the superpixelization process is executed on the image selected and displayed in the picture box 630.

[0047] Here, Figure 7 shows an example of an intermediate image during the segmentation process. Image 710 shows the result of the superpixel processing. Image 720 displays the input image with the superpixel segment map superimposed. Additionally, the selected superpixels and corrected areas 721 are superimposed on the restored object 712. The corrected areas 721 are superimposed in a color corresponding to the selected index. Image 730 is an image output with the superpixels selected by the user in Image 720 and the corrected results as labels. When you press the CalculateSuperPixel button 647, image 720 will be displayed in picture box 630.

[0048] The PaintTool650 in Figure 6 contains various tools (controls) used in the superpixel extraction process in S405 and the segment correction process in S406.

[0049] SuperPixel651 is a tool used in the superpixel extraction process in S405. When you click on the picture box 630 with a mouse, tablet, etc., the superpixel corresponding to the specified coordinate is extracted and the corresponding superpixel is displayed overlaid. An example of the overlaid display is shown in image 720 of Figure 7. Pen652 is a tool used in segment correction processing in S406. When dragged on the picture box 630 with a mouse, tablet, etc., the area corresponding to the specified coordinates is extracted freehand. Note that PaintTool is not limited to Pen652 or SuperPixel651; it may also be a fill tool for filling closed areas, a tool for drawing shapes such as rectangles, circles, and triangles, etc.

[0050] ColorIndex660 is the area that specifies the index used when labeling in superpixel extraction in S405 and segment correction in S406. Eraser661 is used for labeling at Index0. In this embodiment, Index0 represents the background label, and Eraser661 is the so-called eraser tool. When you specify a color index in combo box 663, the palette color corresponding to that index will be displayed in area 664. Note that the index can be set from 0 to 255, and for example, PascalVOC2012 may be the index value of the color map corresponding to the dataset.

[0051] DisplaySettings670 includes an area where you can turn on or off the display of labels and SuperPixels, making it easier to label them during the superpixel extraction process in S405 and the segment correction process in S406.

[0052] The checkbox `overlayLabel671`, when OFF, does not overlay the label on the picture box, and when ON, it overlays the label with the transparency specified in the numerical setting box 672. This allows you to check whether the labeling is working correctly while you work. Also, when the checkbox `OverlaySuperpixel673` is ON, super pixels are displayed, and when OFF, super pixels are not displayed.

[0053] The Save680 button converts the label, which is the processing result generated on the picture box 630, into a predetermined format and saves it via the output in S407.

[0054] These processes may be performed by resizing (reducing) the input image to a smaller size, and then resizing the output labels to their original size. The resizing algorithm may, for example, be the nearestneighbors method, and after resizing, the boundaries may be smoothed by performing erosion and dilation, which are scaling processes in morphology. Thus, resizing to a smaller size once can reduce the processing load of the superpixelization process. However, resizing to too small an image size can degrade boundary accuracy, so it is advisable to consider whether to resize each target object individually.

[0055] These configurations make it possible to balance improved boundary accuracy using superpixels with reduced workload during image segmentation in annotation work.

[0056] <Second Embodiment> Next, a second embodiment of the present invention will be described. Figure 8 shows a part of the GUI of the application software in the second embodiment, and shows targetSelectTool891 which is displayed instead of targetSelectTool691 in the GUI of the application software shown in Figure 6. The rest of the GUI configuration of the application software is the same as that shown in Figure 6. Furthermore, the image processing system 100 in the second embodiment is the same as that shown in Figures 1 and 2, so its description is omitted.

[0057] In the second embodiment, as a method for specifying the target area used in the condition determination unit 122, in addition to Disable, rectangle, circle, and FreeHand, SuperPixel 805 is provided, as shown in targetSelectTool 891.

[0058] Next, the operation of the image processing system 100 in the second embodiment will be described, but only the processes that differ from those in the first embodiment will be explained with reference to Figure 4. In step S402 of Figure 4, when SuperPixel 805 is selected, the condition determination unit 122 uses super pixels to specify the target object. Here, first, the input image is processed to create super pixels using a first average segment size. The first average segment size is set by AveSegSize 806 of SuperPixel 805 and is a larger value than the second average segment size used in S404, which will be described later. Then, the target object is specified by selecting the target super pixels using the operation input unit 103 on the picture box 630.

[0059] By setting the first average segment size to a value greater than the second average segment size, the target object's region can be roughly specified with coarse superpixels, reducing the workload. When generating superpixels in S404, finer superpixels can be used, enabling high-precision processing.

[0060] Furthermore, if the target object is specified by selecting a large superpixel, the approximate shape of the target object can be determined, and the value of the number of target divisions in S401 may be corrected based on the complexity of the target object's shape, as in the first embodiment. The complexity is calculated using curvature entropy, roundness, etc., and the larger this value, the greater the correction to the number of target divisions. In this way, the segment accuracy of the superpixel can be improved by taking the shape of the target object into consideration.

[0061] As described above, according to the second embodiment, the approximate size of the target object's region can be obtained with less overhead, and the size can be obtained in a way that better reflects the object's shape than tools such as rectangles or circles. This enables super-pixel processing with a more appropriate average segment size, further improving boundary accuracy.

[0062] <Third Embodiment> Next, a third embodiment of the present invention will be described.

[0063] Figure 9 is a block diagram showing the functional configuration of the image processing system 900 in the third embodiment. As shown in Figure 9, the third embodiment differs from the image processing system 100 of the first embodiment shown in Figure 1 in that a division number correction unit 901 is newly added. The other configurations are the same as those described in the first embodiment, so the same reference numerals are used and their description is omitted.

[0064] The division number correction unit 901 has the function of correcting the average segment size of the superpixel. Specifically, it takes statistics on the usage of the eraser tool, Eraser661, for each index (object), i.e., the number of corrections, and generates a correction amount for the target division number specified in S401 based on these statistics. Furthermore, it has the function of correcting the target division number Nt692 based on that correction amount when the next superpixel is generated for the same index (object). Here, Eraser661 refers to labeling by the index value of the background.

[0065] Next, the operation of the image processing system 900 in the third embodiment will be explained using the flowchart shown in Figure 10. As shown in Figure 10, the third embodiment differs from the first embodiment shown in Figure 4 in that it newly includes division number correction (S1001) and acquisition of division number correction amount (S1002). The other processes are the same as those shown in Figure 4, so the same step numbers are used and their explanation is omitted.

[0066] In the division number correction in S1001, after the region of the target object is specified, the CPU 210 uses the division number correction unit 901 to specify the index of the object to which the region of the target object belongs. The index is specified using the GUI of the application software (not shown). Then, if there is a division number correction amount to be obtained in S1002 for the specified index, the target division number Nt 692 is corrected. The division number correction amount is calculated based on the usage statistics of the segmentation correction unit 125. When the correction amount for index i is Hi, the target division number Nt is corrected by multiplying it by Hi.

[0067] In S1002, the division number correction amount is calculated based on the usage statistics of the segmentation correction unit 125. Specifically, the CPU 210 uses the division number correction unit 901 to collect statistics on the usage status of the Eraser 661, and based on these statistical values, it generates a correction amount for the target division number specified in S401 for each index (object).

[0068] For example, if Na is the average number of times the eraser tool is used per object across all indices, Ni is the average number of times the eraser tool is used per object at index i, and k is a coefficient that controls the magnitude of the correction amount, then the correction amount Hi for index i can be calculated using equation (1).

[0069] Hi = 1 + k(Ni / Na-1) …(1) In this case, when the number of times the eraser tool is used for index i is average, i.e., Ni=Na, Hi=1, and no correction is made to the target division number. When the number of times the eraser tool is used for index i is above average, i.e., Ni>Na, Hi becomes greater than 1, the target division number Nt is corrected to a larger value, and the image segmentation accuracy is improved.

[0070] Conversely, when the number of times the eraser tool at index i is used is less than the average, that is, when Ni < N, Hi becomes less than 1, the target segmentation number Nt is corrected to a small value, and the load of image segmentation is corrected in the direction of reduction.

[0071] In addition, when Ni < Na, that is, when Hi is less than 1, if the target segmentation number Nt is corrected, there is a concern that the target segmentation number will be corrected to a small value and the image segmentation accuracy will decrease. Therefore, it may be acceptable not to correct when Ni < Na.

[0072] In addition, for the acquisition of the segmentation number correction amount in S1002, if an effective correction parameter has been obtained, it is not necessary to perform it thereafter, and it may be sufficient to only use the obtained correction parameter in S1001.

[0073] As described above, according to the third embodiment, for an object with a high usage frequency of the eraser tool, the segmentation accuracy is considered to be low. Therefore, in the next segmentation, the target segmentation number of the same object is corrected to be larger. As a result, more accurate segmentation becomes possible, and in the next segmentation, it becomes possible to reduce the number of times the eraser tool is used.

[0074] <Fourth Embodiment> Next, a fourth embodiment of the present invention will be described.

[0075] FIG. 11 is a block diagram showing the functional configuration of an image processing system 1100 in the fourth embodiment. As shown in FIG. 11, in the fourth embodiment, in addition to the components of the image processing system 100 of the first embodiment shown in FIG. 1, a reflection region acquisition unit 1101 is newly provided, which is different. Since the other configurations are the same as those described in the first embodiment, the same reference numerals are given and the description is omitted.

[0076] <​​In dental imaging, saliva can cause specular reflection, which can alter the appearance of the color and texture of objects in intraoral photographs. Figure 13 is an example of a magnified portion of an input image, showing an example where a specular reflection 1301 is visible in the image. If the specular reflection 1301 is located at the boundary of the target object being segmented, the accuracy of the segmentation may be impaired.

[0077] The reflection area acquisition unit 1101 acquires specular reflection areas included in the dental image and highlights the reflection areas in the picture box 630 displayed on the display unit 104.

[0078] Figure 12 is a flowchart showing the operation of the image processing system 1100 in the fourth embodiment. As shown in Figure 12, the fourth embodiment differs from the first embodiment shown in Figure 4 in that it newly includes reflection area highlighting at S1201. The other processes are the same as those shown in Figure 4, so the same step numbers are used and their explanations are omitted.

[0079] In S1201, the reflection area highlighting function identifies pixels with a brightness above a certain threshold in the image histogram as reflection areas, and overlays these reflection areas onto the image in the picture box 630 to enhance the display. Note that the determination of reflection areas may also be based on RGB gradation values.

[0080] As described above, according to the fourth embodiment, even when there is reflection in the intraoral photograph, the specular reflection area is highlighted, allowing for superpixel extraction and segment correction in S405 and S406 while paying attention to the specular reflection. This enables highly accurate segmentation.

[0081] <Other Embodiments> Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device.

[0082] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0083] <Summary> This embodiment includes the following configuration.

[0084] (Composition 1) An input means for inputting image data, An acquisition means for obtaining the size of the object region containing the object to be extracted in the aforementioned image, A setting means for setting the number of divisions for dividing the object region, A determination means for determining the segment size of a superpixel based on the size of the object region and the number of divisions, A generation means that generates superpixels having a predetermined size range including the segment size determined by the determination means, using the image data. An image processing apparatus characterized by having

[0085] (Configuration 2) A selection means for selecting a superpixel corresponding to the object from among the generated superpixels, Correction means for correcting the region composed of the selected superpixels to approximate the region of the object, Output means for outputting a label that associates the region composed of the corrected superpixels with the object. The image processing apparatus according to configuration 1, further comprising the above.

[0086] (Composition 3) The image processing apparatus according to configuration 2, characterized in that the setting means, the selection means, and the correction means are configured by a GUI (Graphical User Interface) and an operating member.

[0087] (Composition 4) The system further includes a reduction means for reducing the size of the aforementioned image, The image processing apparatus according to configuration 2 or 3, characterized in that the generation means, the selection means, and the correction means perform processing on the image reduced by the reduction means, and the output means resizes the region composed of the corrected superpixels to its size before reduction, and then outputs the label.

[0088] (Composition 5) The image processing apparatus according to configuration 4, characterized in that the output means resizes the region composed of the corrected superpixels using the nearestneighbors method, corrects the boundary of the resized region by performing scaling in morphology processing, and then outputs the label.

[0089] (Composition 6) The image processing apparatus according to any one of configurations 1 to 5, characterized in that the generation means generates superpixels using one of the following algorithms: LS (Linear Spectal Clustering), SEEDS (Superpixels Extracted via Energy-Driven Sampling), or SLIC (Simple Linear Iterative Clustering).

[0090] (Composition 7) The system further includes a means for specifying the aforementioned object region, The designation means allows the user to designate the object area by one of the following means, or a combination thereof: designating the object area with a rectangle, designating the object area with a circle, designating the object area freehand, or selecting a rectangle from one of the object detection results. The image processing apparatus according to any one of configurations 1 to 6, characterized in that the acquisition means acquires the size of the specified object region.

[0091] (Composition 8) The image processing apparatus according to configuration 7, characterized in that the designation means is composed of a GUI (Graphical User Interface) and an operating member.

[0092] (Composition 9) The image processing apparatus according to configuration 7 or 8, characterized in that when the object region is specified freehand by the designation means, the setting means sets a larger number of divisions the greater the complexity of the shape of the object region.

[0093] (Composition 10) The system further includes a specifying means for specifying the object region by generating superpixels of a predetermined size using the aforementioned image data and selecting superpixels corresponding to the object from among the generated superpixels, The image processing apparatus according to any one of configurations 1 to 9, characterized in that the predetermined size is larger than the segment size determined by the determination means.

[0094] (Composition 11) The image processing apparatus according to configuration 10, characterized in that the means for selecting a superpixel corresponding to the object among the designation means is configured by a GUI (Graphical User Interface) and an operating member.

[0095] (Composition 12) The image processing apparatus according to configuration 10 or 11, characterized in that the setting means sets a larger number of divisions the greater the complexity of the shape of the object region specified by the designation means.

[0096] (Composition 13) The system further includes a division number correction means for correcting the number of divisions set by the setting means, The division number correction means corrects the division number according to the amount of correction by the correction means, The image processing apparatus according to configuration 2, characterized in that the determination means determines the segment size using the corrected number of divisions when the number of divisions has been corrected by the number of divisions correction means.

[0097] (Composition 14) The image processing apparatus according to configuration 13, characterized in that the division number correction means increases the division number when the correction amount is greater than a predetermined threshold, and decreases the division number when the correction amount is less than a predetermined threshold.

[0098] (Composition 15) An image processing apparatus according to any one of configurations 1 to 14, characterized in that the input image is a dental image.

[0099] (Composition 16) A determination means that uses the aforementioned image data to determine pixels having a brightness above a predetermined threshold, The system further includes a control means for controlling the display of the aforementioned image on a display means, The image processing apparatus according to any one of configurations 1 to 15, characterized in that the control means superimposes and displays pixels determined by the determination means onto the image displayed on the display means.

[0100] (Composition 17) The input process involves entering image data, The steps include obtaining the size of the object region containing the object to be extracted from the aforementioned image, A step of obtaining the number of divisions for dividing the object region, A step of determining the segment size of a superpixel based on the size of the object region and the number of divisions, A step of generating a superpixel having a predetermined size range including the determined segment size using the image data. An image processing method characterized by having the following features.

[0101] (Composition 18) A program for causing a computer to function as one of the means of an image processing apparatus described in any of configurations 1 to 16.

[0102] (Composition 19) A computer-readable storage medium containing the program described in Configuration 18.

[0103] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0104] 100, 900, 1100: Image processing system, 101: Image input unit, 102: Image processing unit, 103: Operation input unit, 104: Display unit, 105: Label output unit, 106: Input image, 107: Label, 121: Target division number setting unit, 122: Condition determination unit, 123: Super pixel generation unit, 124: Super pixel extraction unit, 125: Segmentation correction unit, 901: Division number correction unit, 1101: Reflection area acquisition unit

Claims

1. An input means for inputting image data, An acquisition means for obtaining the size of the object region containing the object to be extracted in the aforementioned image, A setting means for setting the number of divisions for dividing the object region, A determination means for determining the segment size of a superpixel based on the size of the object region and the number of divisions, A generation means that generates superpixels having a predetermined size range including the segment size determined by the determination means, using the image data. An image processing apparatus characterized by having

2. A selection means for selecting a superpixel corresponding to the object from among the generated superpixels, Correction means for correcting the region composed of the selected superpixels to approximate the region of the object, Output means for outputting a label that associates the region composed of the corrected superpixels with the object. The image processing apparatus according to claim 1, further comprising the above.

3. The image processing apparatus according to claim 2, characterized in that the setting means, the selection means, and the correction means are configured by a GUI (Graphical User Interface) and an operating member.

4. The system further includes a reduction means for reducing the size of the aforementioned image, The image processing apparatus according to claim 2, wherein the generation means, the selection means, and the correction means perform processing on the image reduced by the reduction means, and the output means resizes the region composed of the corrected superpixels to its size before reduction, and then outputs the label.

5. The image processing apparatus according to claim 4, wherein the output means resizes the region composed of the corrected superpixels using the nearestneighbors method, corrects the boundary of the resized region by performing scaling in morphology processing, and then outputs the label.

6. The image processing apparatus according to claim 1, characterized in that the generation means generates superpixels using one of the following algorithms: LS (Linear Spectal Clustering), SEEDS (Superpixels Extracted via Energy-Driven Sampling), or SLIC (Simple Linear Iterative Clustering).

7. The system further includes a means for specifying the aforementioned object region, The designation means allows the user to designate the object area by any of the following means: designating the object area with a rectangle, designating the object area with a circle, designating the object area freehand, or selecting a rectangle from one of the object detection results. The image processing apparatus according to claim 1, characterized in that the acquisition means acquires the size of the specified object region.

8. The image processing apparatus according to claim 7, characterized in that the designated means is composed of a GUI (Graphical User Interface) and an operating member.

9. The image processing apparatus according to claim 7, characterized in that when the object region is specified freehand by the designation means, the setting means sets a larger number of divisions the greater the complexity of the shape of the object region.

10. The system further includes a specifying means for specifying the object region by generating superpixels of a predetermined size using the aforementioned image data and selecting superpixels corresponding to the object from among the generated superpixels, The image processing apparatus according to claim 1, characterized in that the predetermined size is larger than the segment size determined by the determination means.

11. The image processing apparatus according to claim 10, characterized in that the means for selecting a superpixel corresponding to the object among the designation means is configured with a GUI (Graphical User Interface) and an operating member.

12. The image processing apparatus according to claim 10, characterized in that the setting means sets a larger number of divisions the greater the complexity of the shape of the object region specified by the designation means.

13. The system further includes a division number correction means for correcting the number of divisions set by the setting means, The division number correction means corrects the division number according to the amount of correction by the correction means, The image processing apparatus according to claim 2, wherein the determination means determines the segment size using the corrected number of divisions when the number of divisions has been corrected by the number of divisions correction means.

14. The image processing apparatus according to claim 13, wherein the division number correction means increases the division number when the correction amount is greater than a predetermined threshold, and decreases the division number when the correction amount is less than a predetermined threshold.

15. The image processing apparatus according to claim 1, characterized in that the input image is a dental image.

16. A determination means that uses the aforementioned image data to determine pixels having a brightness above a predetermined threshold, The system further includes a control means for controlling the display of the aforementioned image on a display means, The image processing apparatus according to claim 1, characterized in that the control means superimposes and displays pixels determined by the determination means onto the image displayed on the display means.

17. A method for controlling an image processing device, which is performed by the image processing device, The input process involves entering image data, The steps include obtaining the size of the object region containing the object to be extracted from the aforementioned image, A step of obtaining the number of divisions for dividing the object region, A step of determining the segment size of a superpixel based on the size of the object region and the number of divisions, A step of generating a superpixel having a predetermined size range including the determined segment size using the image data. A control method for an image processing apparatus, characterized by having the following features.

18. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 16.

19. A computer-readable storage medium storing the program described in claim 18.

Images