Image processing apparatus and method, program, and storage medium

Abstract

The image processing apparatus includes: an input section configured to input image data of an image; an acquisition section configured to acquire a size of an object region including an object to be extracted included in the image; a setting section configured to set a division number into which the target region is divided; a determination section configured to determine a segment size of a superpixel based on a size of the object region and the division number; and a generation section configured to generate superpixels, each of which has a size within a predetermined range including the segment size determined by the determination section, using the image data.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and method, a program and a storage medium, and more particularly to an image segmentation technology. Background Art

[0002] Traditionally, semantic segmentation is known in machine learning. Semantic segmentation is the task of dividing an image into multiple object regions. To perform semantic segmentation, the image is refined and output pixel by pixel. At this time, the training data used for semantic segmentation needs to be labeled for each pixel.

[0003] Therefore, in semantic segmentation, the labeling task (tagging images and creating training data) is very heavy, so in order to reduce the workload, superpixels are used. Superpixels are small areas formed by grouping pixels with similar colors and / or textures.

[0004] For example, Patent Document 1 discloses a method for segmenting an image using mixed-scale superpixels. Specifically, the user replaces larger-scale superpixels in a region of interest (ROI) with smaller-scale superpixels (region size of superpixels) to improve segmentation, thereby achieving better boundary delineation.

[0005] Furthermore, Patent Document 2 discloses a region discriminating device that discriminates a region based on a saliency map and superpixels.

[0006] Prior art literature

[0007] Patent Literature

[0008] Patent Document 1: Japanese Patent Application No. 2018-514024

[0009] Patent Document 2: Japanese Patent Application Laid-Open No. 2019-61658 Summary of the invention

[0010] Problem that the invention aims to solve

[0011] However, according to the technology described in Patent Document 1, when replacing a superpixel in an ROI with a superpixel of a smaller scale, the scale is based on a user's instruction, so a good segmentation boundary may not always be obtained. That is, if the scale is too large, the boundary accuracy deteriorates, and if the scale is too small, the workload of selecting superpixels when segmenting an image increases.

[0012] In addition, the scale of the superpixel does not change in Patent Document 2. Therefore, if an object having a size different from the expected size or multiple objects having different sizes exist in the same image, it is impossible to generate superpixels of appropriate scales and it is impossible to obtain segmentation boundaries with good accuracy.

[0013] The present invention has been made in view of the above-mentioned problems, and aims to strike a balance between improvement of boundary accuracy using superpixels and reduction of the load of the segmentation work when an image is segmented in a labeling work.

[0014] Solutions for solving problems

[0015] In order to achieve the above-mentioned purpose, an image processing device of the present invention includes: an input component, which is configured to input image data of an image; an acquisition component, which is configured to acquire the size of an object area included in the image and including an object to be extracted; a setting component, which is configured to set the number of divisions into which the object area is divided; a determination component, which is configured to determine the segment size of a superpixel based on the size of the object area and the number of divisions; and a generation component, which is configured to use the image data to generate superpixels, each of which has a size within a predetermined range including the segment size determined by the determination component.

[0016] Effects of the Invention

[0017] According to the present invention, when an image is segmented in a labeling task, a balance can be achieved between improving boundary accuracy using superpixels and reducing the load of the segmentation task.

[0018] Other features and advantages of the present invention will be apparent from the following description in conjunction with the accompanying drawings. Note that throughout the accompanying drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 is a block diagram illustrating a functional configuration of an image processing system according to a first embodiment and a second embodiment of the present invention.

[0021] Figure 2 is a diagram illustrating an example of a hardware configuration of an image processing system according to an embodiment.

[0022] Figure 3 is a diagram illustrating an example of an input image to be analyzed according to the first embodiment.

[0023] Figure 4 is a flowchart illustrating the operation of the image processing system according to the first embodiment.

[0024] Figure 5 1 is an explanatory diagram related to designation of a region of a target object according to the first embodiment.

[0025] Figure 6 is a diagram illustrating an example of a GUI of application software according to the first embodiment.

[0026] Figure 7 is a diagram illustrating an example of an intermediate image in the segmentation process according to the first embodiment.

[0027] Figure 8 is a diagram illustrating an example of a part of the GUI of application software according to the second embodiment.

[0028] Fig. 9 is a block diagram illustrating a functional configuration of an image processing system according to a third embodiment.

[0029] Fig.10 is a flowchart illustrating the operation of the image processing system according to the third embodiment.

[0030] Fig.11 is a block diagram illustrating a functional configuration of an image processing system according to a fourth embodiment.

[0031] Fig.12 is a flowchart illustrating the operation of the image processing system according to the fourth embodiment.

[0032] Fig.13 is a partially enlarged diagram illustrating an example of an input image according to the fourth embodiment. DETAILED DESCRIPTION

[0033] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to an invention requiring all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar structures, and redundant descriptions thereof are omitted.

[0034] <First embodiment>

[0035] Figure 1 1 is a block diagram illustrating the functional structure of the image processing system 100 according to the first embodiment of the present invention. Figure 1 As shown, the image processing system 100 has an image input unit 101 , an image processing unit 102 , an operation input unit 103 , a display unit 104 , and a label output unit 105 .

[0036] The image input unit 101 inputs an image 106 (image data) to be analyzed to the image processing system 100, and the image processing unit 102 executes application software to perform segmentation processing on the image 106 input to the image input unit 101. The operation input unit 103 is composed of a mouse, a keyboard, a tablet computer, etc., and an operator operates the operation input unit 103 to input information to the image processing unit 102. The display unit 104 interactively displays the image being processed by the image processing unit 102 and the operation result of the operation input unit 103, etc. The label output unit 105 outputs the segmentation result as the processing result of the image processing unit 102 as a label 107. The output label 107 is stored in a storage device (not shown).

[0037] Next, the functional structure of the image processing unit 102 will be described.

[0038] When dividing the image 106 into superpixels, the target division number setting unit 121 sets the number of superpixels (target division number) into which the target object to be segmented (extracted) in the image is to be divided. The condition determination unit 122 determines the superpixel generation condition based on the size of the area (object area) of the target object and the set target division number. Here, as the superpixel generation condition, the average segment size (average scale) of the superpixel is calculated. The superpixel generation unit 123 generates superpixels based on the average segment size.

[0039] The superpixel extraction unit 124 performs segmentation using superpixels corresponding to the target object selected by the GUI tool of the application software using the operation input unit 103. The segmentation correction unit 125 corrects errors in the segmentation result of the superpixel extraction unit 124. Specifically, among the superpixels selected by the segmentation, the area of ​​superpixels protruding from the target object or the area of ​​missing or insufficient superpixels is corrected pixel by pixel using a pen tool or the like to make the area closer to the area of ​​the target object.

[0040] Figure 2 1 is a block diagram illustrating a configuration of a computer 200 as an example of a hardware configuration of the image processing system 100. Each function of the image processing system 100 can be realized by the computer 200. The computer 200 includes Figure 2 Shown are a central processing unit (CPU) 210, a storage unit 212, an operation input unit 103 (eg, a mouse, a keyboard, etc.), and a display unit 104 (a display, etc.).

[0041] The storage unit 212 is composed of a main storage unit 215 (ROM, RAM, etc.) and an auxiliary storage unit 216 (magnetic disk device, SSD: Solid State Drive, etc.).

[0042] The CPU 210 performs calculations and controls, and executes programs stored in the storage unit 212, thereby serving as Figure 1 The image processing system 100 is shown with an image processing unit 102 .

[0043] The computer 200 may include one or more CPUs 210 and a storage unit 212. That is, if at least one or more processing devices (CPUs) are connected to at least one storage device, and if at least one or more processing devices execute programs stored in at least one or more storage devices, the computer 200 functions as the image processing unit 102. The structure used as the image processing unit 102 is not limited to the CPU 210, and may be an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit) or the like.

[0044] Next, reference will be made to an example of an image to be analyzed. Figure 3 The details of the target division number setting unit 121 in this embodiment are described below. In this embodiment, the image to be analyzed is a dental image, and a dental restoration is a target object for segmentation, but the types of the image to be analyzed and the target object are not limited thereto.

[0045] In the image 310 , the object 312 is a dental restoration which is a target object for segmentation.

[0046] The superpixel generation unit 123 performs superpixel processing on the entire image to be analyzed. Images 320 and 330 are enlarged images of a region 311 around a target object in an image in which the result of the superpixel processing is superimposed on the input image. In the superpixel processing, an average segment size is set as a setting value for indicating the size of superpixels into which the image is divided. Image 320 is an example in which the average segment size is small, and image 330 is an example in which the average segment size is large.

[0047] The larger the average segment size, the larger the size of each superpixel, and the lighter the extraction load on the superpixel extraction unit 124. On the other hand, as shown in the image 330, there is a greater possibility that the superpixel will protrude from the target object as a protruding portion 331, or that a part of the target object will not be included in the superpixel as a missing or insufficient portion 332, resulting in lower segmentation accuracy.

[0048] In contrast, the smaller the average segment size, the smaller the size of each superpixel, which increases the extraction workload on the superpixel extraction unit 124, but as can be seen in image 320, the segmentation accuracy for the target object is higher.

[0049] Therefore, there is a trade-off between the workload of superpixel extraction and the segmentation accuracy.

[0050] The target division number setting unit 121 sets how many superpixels (target division number) the target object (i.e., the restoration in the dental image in this embodiment) is to be divided into. Figure 3 In the example of the image 320 , the restoration 312 is divided into about 9 superpixels, and the target number of divisions is 9. On the other hand, in the example of the image 330 , the restoration 312 is divided into about 4 superpixels, and the target number of divisions is 4.

[0051] Next, refer to Figure 4 The flowchart shown is used to explain the operation of the image processing system 100 in the first embodiment.

[0052] First, in step S400 , the CPU 210 inputs an image to be analyzed via the image input unit 101 .

[0053] In step S401, the CPU 210 sets the target number of divisions in the image processing unit 102 using the target number of divisions setting unit 121 by the operator operating the operation input unit 103. As described above, there is a trade-off relationship between the workload of superpixel extraction and the segmentation accuracy. The operator sets a target number of divisions that does not impose a large workload of superpixel extraction and provides sufficient segmentation accuracy. In many cases, a balance can be achieved between the workload and the segmentation accuracy by setting the target number of divisions between 10 and 100. In the present embodiment, as an example, the target number of divisions is described as 50, but is not limited to this value. When the target number of divisions is set to 50, the operator selects approximately 50 superpixels that constitute the target object.

[0054] In step S402 , the CPU 210 specifies a target object using the condition determination unit 122 based on the operator's operation of the operation input unit 103 . Figure 5 An explanatory diagram related to the specification of a target object according to the first embodiment is shown.

[0055] As an example, the input image 500 to be analyzed is a dental image, and the target objects 511, 512, and 513 are dental restorations. The region of the target object is specified by roughly surrounding the target object with a closed curve. As an example, the region of the target object 511 can be specified with a circle 501 using a circle drawing tool, and as another example, the region of the target object 512 can be specified with a rectangle 502 using a rectangle drawing tool. As yet another example, the region of the target object 513 can be specified with a closed curve 503 using a free hand drawing tool. When using the free hand drawing tool, the approximate shape of the target object 513 can be obtained.

[0056] When the CPU 210 specifies the target object by the above-described method or the like, the CPU 210 obtains the number of pixels (size) of the area within the drawn closed curve as the approximate target size.

[0057] Considering that the target objects have the same size, the more complex the shape of the target object is, the lower the accuracy of the segmentation using superpixels is. Therefore, the value of the target division number set in step S401 can be corrected based on the complexity of the shape of the target object obtained by using the free hand drawing tool. The complexity is calculated using curvature entropy, roundness, etc., and the larger the value is, the more the value of the target division number is corrected. In this way, the accuracy of the segmentation using superpixels can be improved by considering the shape of the target object. Note that a default value can be set as the target division number in step S401, and the target division number can be re-determined based on the shape of the target object obtained in step S402.

[0058] As another example, deep learning may be used to perform object detection of teeth and dental diseases, and the region of the target object may be specified by selecting one of the obtained rectangular detection results.

[0059] In step S403, the average segment size is extracted. Here, the CPU 210 obtains the approximate number of pixels (size) of the area of ​​the target object based on the closed curve surrounding the target object specified by the condition determination unit 122 in step S402, and divides it by the target division number. In this way, the average segment size of the superpixel is calculated.

[0060] In step S404, superpixels are generated. Here, the CPU 210 uses the superpixel generation unit 123 to perform superpixel processing on the image to be analyzed using the average segment size calculated in step S403. In this embodiment, LSC (Linear Spectral Clustering) is used as the superpixel algorithm.

[0061] It should be noted that other algorithms may be used, such as SEEDS (Superpixel Extracted via Energy-Driven Sampling) or SLIC (Simple Linear Iterative Clustering), etc. Depending on the implementation, the size of the image to be analyzed may be divided by the average segment size to obtain the number of superpixels in the image as a whole, which may then be used as input for superpixel processing.

[0062] In addition, the segment size of each superpixel may be within a predetermined range including the average segment size calculated in step S403 .

[0063] In step S405, the superpixel corresponding to the target object is extracted. Here, the CPU 210 uses the superpixel extraction unit 124 to extract the superpixel corresponding to the target object based on the operation of the operator. In this process, the CPU 210 displays the GUI of the application software on the display unit 104, displays the input image on the GUI, and superimposes the layer of the segmented map of the generated superpixel on the input image. Then, at the position specified by the mouse or by the tablet computer on the image to be analyzed, the superpixel corresponding to the position is displayed, and the area corresponding to the target object is extracted in units of superpixels.

[0064] In step S406, segment correction is performed. Here, the CPU 210 uses the segment correction unit 125 to correct the region extracted in step S405 based on the operation of the operator. Specifically, in the region extracted in superpixel units, the portion protruding from the target object or the portion of the object not included in the superpixel is corrected in pixel units. The correction is performed using an eraser tool or a pen tool, etc.

[0065] If there are N types of approximate target sizes of the target object to be segmented in the input image, the above-mentioned processing from step S402 to step S406 is performed N times for each approximate target size. In other words, if there are multiple target objects in the input image, the same superpixel processing is performed on target objects with approximately the same approximate target size, and for target objects of different sizes, the area of ​​the target object is re-specified, the conditions used for superpixel processing are changed, and superpixel processing is performed in different cycles.

[0066] Multiple thresholds can be used to classify the types of approximate target sizes. In addition, as described above, in the case where the area of ​​the target object is specified by a predetermined shape such as a circle or a rectangle, multiple sizes of circles and rectangles can be prepared in advance, and the approximate target size can be classified by using appropriate sizes of circles and rectangles among the multiple sizes of circles and rectangles.

[0067] In step S407, the processing result is output. Here, the segmentation map corrected in step S406 is output in association with the input image and saved as a label. The label is saved in IndexPNG format (also called palette format) (which is often used as training data for semantic segmentation).

[0068] The format of the label may be a bitmap format or other formats, and may be selected according to the application of the label.

[0069] then, Figure 6 An example of a GUI (Graphical User Interface) of application software in this embodiment is shown.

[0070] A GUI 600 of the application software is displayed on the display unit 104 and has various control areas for implementing segmentation using superpixels.

[0071] By pressing the "Open" button 611, the directory of the image to be divided can be selected.

[0072] “ImageList” 620 displays a list of images registered in the directory, and an image selected from these images is displayed in a picture frame 630. In the picture frame 630, a pointer 631 or a circular pointer 632 that can be controlled using the operation input unit 103 is displayed.

[0073] In "TargetSettings" 690, various tools used when specifying the area of ​​the target object in step S402 are arranged. In "targetSelectTool" 691, which is a group of radio buttons, a tool for specifying a closed curve for roughly specifying the area of ​​the target object is arranged. In the present embodiment, as an example, "rectangle" for specifying a rectangular area, "circle" for specifying a circular area, "FreeHand" for specifying a free area, and "Disable" are arranged. When "Disable" is selected, the rough designation of the area of ​​the target object is invalid, and when a radio button other than "Disable" is selected, the rough designation of the area of ​​the target object is valid.

[0074] After the closed curve is obtained, the size of the area specified by the closed curve is displayed in “targetPixNum (target pixel number)” 693 .

[0075] "Nt" 692 is a region for inputting the target number of divisions in step S401. Then, by pressing "targetAreaSet" 694, the average segment size calculated from the size of the area specified by the closed curve and the target number of divisions is displayed in "AveSegSize" 643.

[0076] In "SuperPixel Settings" 640, an input box for setting the value of superpixel processing conditions is arranged. Using the superpixel algorithm selection tab 641, a superpixel algorithm can be selected. Figure 6In the example, LSC is selected as the superpixel algorithm. In addition, in the LSC algorithm tab, there are arranged: "NumIterations" 642, which is the setting value of the number of iterations used in the clustering stage of the algorithm; "AveSegSize" (average segment size) 643, which is the setting value of the square root of the average segment size (number of pixels) of the superpixel; "CFactor" (compactness factor) 644, which is a setting value related to the shape of the superpixel; and "MinElementSize" (minimum element size) 645, which is the setting value of the minimum superpixel segment size.

[0077] "CFactor" 644 controls the shape of the superpixel; the higher the value, the more regular the shape of the superpixel.

[0078] "MinElementSize" 645 represents the minimum segment size of a superpixel; superpixels smaller than this size are absorbed into larger superpixels. Changing the "MinElementSize" setting affects the number of superpixel divisions, so the average segment size obtained in step S403 is calculated taking this effect into account.

[0079] Note that these specific settings are merely examples and may be changed to suit the characteristics of the image to be segmented.

[0080] Since "SuperPixelSettings" 640 includes a plurality of setting items, each setting value may be recorded in a json file and then selected using a "ReadParamFile" 646 button or the like to read all of these setting values ​​at once.

[0081] Then, by pressing the “CalculateSuperPixel” button 647 , superpixel processing is performed on the image selected and displayed in the picture frame 630 .

[0082] Figure 7 An example of an intermediate image in the segmentation process is shown.

[0083] Image 710 shows the result of superpixel processing. Image 720 shows the input image superimposed with the superpixel segmentation map. The selected superpixel and the corrected portion 721 are superimposed on the restoration 712. The corrected portion 721 is superimposed and displayed in a color corresponding to the selected index. Image 730 is an image in which the superpixel selected by the user in image 720 and the correction result are output as labels.

[0084] When the “CalculateSuperPixel” button 647 is pressed, the image 720 is displayed in the picture frame 630 .

[0085] exist Figure 6 In the “Paint Tool” 650 in FIG. 4 , various tools (controls) used in the superpixel extraction process in step S405 and the segment correction process in step S406 are arranged.

[0086] "SuperPixel" 651 is a tool used in the superpixel extraction process in step S405; by clicking in the picture frame 630 using a mouse, tablet, etc., the superpixels corresponding to the specified coordinates are extracted and the corresponding superpixels are superimposed and displayed. An example of superimposed display is Figure 7 Image 720 in.

[0087] "Pen" 652 is a tool used in the segment correction process in step S406; a region corresponding to the designated coordinates is extracted by free hand drawing by dragging in the picture frame 630 using a mouse, a tablet, etc. Note that "PaintTool" is not limited to "Pen" 652 and "SuperPixel" 651, but may be a fill tool for filling in a closed area, or a tool for drawing shapes such as rectangles, circles, and triangles.

[0088] “ColorIndex” 660 is a field for specifying an index used in labeling in superpixel extraction in step S405 and segment correction in step S406 .

[0089] "Eraser" 661 is labeled with Index 0. In the present embodiment, Index 0 indicates a background label, and "Eraser" 661 is a so-called eraser tool.

[0090] When a color index is specified in the combo box 663, a color in the palette corresponding to the index is displayed in the area 664. Note that "Index" can be set to a value between 0 and 255, and for example, PascalVOC2012 can be an index value of a color map corresponding to a dataset.

[0091] "Display Settings" 670 includes an area in which the display of labels and superpixels can be turned ON / OFF to facilitate labeling in the superpixel extraction process in step S405 and the segment correction process in step S406.

[0092] When the check box of "OverlayLabel" 671 is OFF, the label is not displayed in the picture frame, and when the check box is ON, the label is displayed in the picture frame with the transparency specified in the value setting box 672. This makes it possible to work while confirming whether labeling is successful. When the check box "OverlaySuperpixel" 673 is ON, superpixels are displayed, and when the check box is OFF, superpixels are not displayed.

[0093] The “Save” button 680 converts the label as the processing result generated in the picture frame 630 into a predetermined format in the output in step S407 and saves it.

[0094] These processes can be performed by resizing (reducing) the input image to a smaller size and then resizing the output label to the original size. The resizing algorithm can use, for example, the nearest neighbor method, and after resizing, the boundaries can be smoothed by performing "Erosion" and "Dilation" processes (which are magnification and reduction processes in morphological processing).

[0095] In this way, by resizing the image to a smaller size once, the processing load of superpixel processing can be reduced. However, if the image is resized to an excessively small size, the boundary accuracy deteriorates, so it is desirable to consider whether to resize for each target object.

[0096] These structures make it possible to balance the improvement of boundary accuracy using superpixels and the reduction of the load of the segmentation task when segmenting an image in a labeling task.

[0097] <Second embodiment>

[0098] Next, a second embodiment of the present invention will be described.

[0099] Figure 8 is a diagram showing a portion of the GUI of the application software in the second embodiment, which shows a Figure 6 The rest of the structure of the application software GUI is similar to that of the targetSelectTool 691 in the application software GUI. Figure 6 In addition, the image processing system 100 in the second embodiment is the same as Figure 1 and Figure 2 The illustrated image processing system 100 is the same, and thus a description thereof will be omitted.

[0100] In the second embodiment, as shown in “targetSelectTool” 891 , in addition to “Disable,” “rectangle,” “circle,” and “FreeHand,” “SuperPixel” 805 is provided as a target area designation method used in the condition determination unit 122 .

[0101] Next, refer to Figure 4 The operation of the image processing system 100 in the second embodiment will be described, but only the processing different from that in the first embodiment will be described.

[0102] When Figure 4 When "SuperPixel" 805 is selected in step S402, the condition determination unit 122 uses the superpixel to specify the target object. Here, first, the input image is superpixeled using the first average segment size. The first average segment size is set using "AveSegSize" 806 of "SuperPixel" 805, and is a value larger than the second average segment size used in step S404 described later. Then, the target object is specified by selecting the superpixel of the target in the picture frame 630 using the operation input unit 103.

[0103] By setting the first average segment size to a value larger than the second average segment size, the region of the target object can be roughly specified using coarse superpixels, thereby reducing the workload, and by generating fine superpixels in step S404, high-precision processing can be achieved.

[0104] In addition, when specifying the target object by selecting a large superpixel, the approximate shape of the target object can be known, so as in the first embodiment, the value of the target division number set in step S401 can be corrected based on the complexity of the shape of the target object. The complexity is calculated using curvature entropy, roundness, etc., and the larger the value, the more the target division value is corrected. In this way, by taking into account the shape of the target object, the segmentation accuracy of the superpixel can be improved.

[0105] As described above, according to the second embodiment, the approximate size of the area of ​​the target object can be obtained with a smaller load, and the size can be obtained in a form that more closely reflects the shape of the object than using tools such as a rectangle or a circle, etc. This makes it possible to implement superpixel processing with a more appropriate average segment size, and makes it possible to further improve the boundary accuracy.

[0106] <Third embodiment>

[0107] Next, a third embodiment of the present invention will be described.

[0108] Fig. 9900 is a block diagram showing the functional structure of the image processing system 900 according to the third embodiment. Fig. 9 As shown, the third embodiment and Figure 1 The image processing system 100 according to the first embodiment shown is different in that a division number correction unit 901 is newly provided. The other structures are the same as those described in the first embodiment, so the same reference numerals are given and the description thereof will be omitted.

[0109] The division number correction unit 901 has a function of correcting the average segment size of the superpixel. Specifically, the division number correction unit 901 uses statistics (that is, the number of corrections) related to the use of the eraser tool "Eraser" 661 for each index (object), and generates a correction amount for the target division number specified in step S401 based on the statistics. In addition, the division number correction unit 901 has a function of correcting the target division number "Nt" 692 based on the correction amount when generating the next superpixel of the same index (object). Here, "Eraser" 661 represents labeling using the index value of the background.

[0110] Next, refer to Fig.10 The operation of the image processing system 900 in the third embodiment is described with reference to the flowchart shown in FIG. Fig.10 As shown, the third embodiment and Figure 4 The difference of the first embodiment shown is that the correction of the number of divisions in step S1001 and the acquisition of the correction amount of the number of divisions in step S1002 are newly performed. Figure 4 The processes shown are the same, so the same step numbers are given and their descriptions will be omitted.

[0111] In the division number correction in step S1001, after specifying the area of ​​the target object, the CPU 210 uses the division number correction unit 901 to specify the index of the object to which the area of ​​the target object belongs. The index is specified by the GUI (not shown) of the application software. Then, if the division number correction amount to be obtained in step S1002 exists for the specified index, the target division number "Nt" 692 is corrected. The division number correction amount is calculated based on the usage statistics obtained by the segmentation correction unit 125. When the correction amount of index i is Hi, the target division number "Nt" is corrected by multiplying the target division number by Hi.

[0112] In step S1002, the division number correction amount is calculated based on the usage statistics obtained by the segmentation correction unit 125. Specifically, the CPU 210 uses the division number correction unit 901 to obtain statistics related to the usage status of the "Eraser" 661, and generates a correction amount for the target division number specified in step S401 for each index (object) based on the statistical value.

[0113] For example, if the average number of times the eraser tool is used for an object for all indices is Na, the average number of times the eraser tool is used for an object for index i is Ni, and the coefficient for controlling the magnitude of the correction amount is k, then the correction amount Hi for index i can be calculated by Equation (1).

[0114] Hi = 1 + k (Ni / Na - 1) ... (1)

[0115] In this case, if the number of times the eraser tool is used for index i is the average value, that is, if Ni = Na, then Hi = 1 and the target number of divisions is not corrected. If the number of times the eraser tool is used for index i is higher than the average value, that is, if Ni > Na, then Hi becomes greater than 1, and the target number of divisions “Nt” is corrected to a larger value, and the image segmentation becomes more accurate.

[0116] Conversely, if the number of times the eraser tool is used for index i is less than the average value, that is, if Ni < N, then Hi becomes less than 1, and the target number of divisions “Nt” is corrected to a smaller value, and the load of the image segmentation is reduced.

[0117] In addition, if Ni < Na, that is, if Hi is less than 1, then in the case of correcting the target number of divisions “Nt”, the target number of divisions will be corrected to a small value, and there is a concern that the image segmentation accuracy will decrease. Therefore, if Ni < Na, the correction may not be performed.

[0118] Note that if a valid correction parameter is obtained, there is no need to obtain the division number correction amount in step S1002, and thereafter, only the correction parameter obtained in step S1001 can also be used.

[0119] As described above, according to the third embodiment, since it is considered that the segmentation accuracy is low for an object that frequently uses the eraser tool, in the next segmentation, the target number of divisions for the same object is corrected to be larger. This enables segmentation to be performed with higher accuracy and enables the number of times the eraser tool is used in the next segmentation to be reduced.

[0120] <Fourth Embodiment>

[0121] Next, a fourth embodiment of the present invention will be described.

[0122] Fig.11 is a block diagram showing the functional structure of the image processing system 1100 according to the fourth embodiment. As Fig.11 shown, the difference between the fourth embodiment and the first embodiment is that, in addition to Figure 1In addition to the components of the image processing system 100 according to the first embodiment shown, a reflection area acquisition unit 1101 is newly provided. The other components are the same as those described in the first embodiment, and therefore the same reference numerals are given and their description will be omitted.

[0123] In the following description, it is assumed that the input image 106 is a dental image.

[0124] In dental images, specular reflection occurs due to saliva, and the color and texture of objects shown in intraoral photographs may appear different. Fig.13 1301 is an example of a partially enlarged view of an input image, which shows an example in which a specular reflection portion 1301 is seen in the image. If any of the specular reflection portions 1301 is located on a boundary of a target object to be segmented, the accuracy of segmentation may be impaired.

[0125] The reflection region acquisition unit 1101 acquires the specular reflection region included in the dental image, and highlights the reflection region in the picture frame 630 displayed on the display unit 104 .

[0126] Fig.12 FIG. 1 is a flowchart showing the operation of the image processing system 1100 in the fourth embodiment. Fig.12 As shown, in the fourth embodiment, in addition to Figure 4 In addition to the processing in the first embodiment shown in the figure, the reflection area is newly highlighted in step S1201. Figure 4 The processes shown are the same, so the same step numbers are given and their descriptions will be omitted.

[0127] In the reflective region highlighting in step S1201, pixels having brightness equal to or greater than a certain threshold in the image histogram are determined to belong to the reflective region, and the reflective region is superimposed and displayed on the image in the picture frame 630 to highlight the reflective region. The reflective region may be determined based on RGB grayscale values.

[0128] As described above, according to the fourth embodiment, even if there is reflection in the intraoral photograph, by highlighting the specular reflection area, superpixel extraction and segment correction in steps S405 and S406 can be performed while paying attention to the specular reflection area. This makes it possible to achieve highly accurate segmentation.

[0129] <Other Examples>

[0130] The present invention can be applied to a system composed of a plurality of devices, or to an apparatus composed of a single device.

[0131] The present invention can be implemented by the following process: a program for implementing one or more functions of the above-mentioned embodiment is supplied to a system or device via a network or storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0132] The present invention is not limited to the above-described embodiments, and various changes and modifications may be made within the spirit and scope of the present invention. Therefore, in order to inform the public of the scope of the present invention, the following claims are attached.

[0133] This application claims priority from Japanese Patent Application No. 2022-175027, filed on October 31, 2022, which is hereby incorporated by reference herein.

Claims

1. An image processing device, include: an input component configured to input image data of an image; an acquisition component configured to acquire a size of an object region included in the image and including an object to be extracted; a setting component configured to set the number of divisions into which the object area is divided; A determination component configured to determine a segment size of a superpixel based on the size of the object area and the number of divisions; as well as A generating unit is configured to generate super pixels using the image data, each of the super pixels having a size within a predetermined range including the segment size determined by the determining unit.

2. The image processing device according to claim 1, further comprising: include: A selection component configured to select a superpixel corresponding to the object from the generated superpixels; a correction component configured to correct the area formed by the selected superpixels to approximate the area of ​​the object; as well as An output component is configured to output a label associating the corrected region formed by the superpixel with the object.

3. The image processing device according to claim 2, in, The setting means, the selecting means and the correcting means are configured using a graphical user interface (GUI) and operating means.

4. The image processing apparatus according to claim 2 or 3, further comprising a reduction unit configured to reduce the size of the image, in, The generating part, the selecting part and the correcting part process the image reduced by the reducing part, and the outputting part resizes the corrected region formed by the superpixel to the size before reduction, and then outputs the label.

5. The image processing device according to claim 4, in, The output component uses a nearest neighbor method to resize the corrected region formed by the superpixel, performs a scaling process in a morphological process to correct a boundary of the resized region, and then outputs the label.

6. The image processing device according to any one of claims 1 to 5, in, The generating component generates the superpixel using any one of the following algorithms: linear spectral clustering (LS), superpixel extraction via energy driven sampling (SEEDS), and simple linear iterative clustering (SLIC).

7. The image processing apparatus according to any one of claims 1 to 6, further comprising a specifying unit configured to specify the object area, in, The designating component designates the object area by using one or a combination of the following components: a component for allowing the user to designate the object area using a rectangle, a component for allowing the user to designate the object area using a circle, a component for allowing the user to designate the object area by free hand drawing, and a component for allowing the user to select a rectangle from one of the detection results of object detection, and The acquisition component acquires the size of the designated object area.

8. The image processing device according to claim 7, in, The designated part is constructed using a graphical user interface (GUI) and operating means.

9. The image processing device according to claim 7 or 8, in, In a case where the designating means designates the object region by free hand drawing, the setting means sets a larger number of divisions as the shape of the object region is more complicated.

10. The image processing apparatus according to any one of claims 1 to 9, further comprising a designating unit configured to designate the object area by generating superpixels of a predetermined size using the image data and selecting superpixels corresponding to the object from the generated superpixels, in, The predetermined size is greater than the segment size determined by the determining component.

11. The image processing device according to claim 10, in, In the specifying means, a means for selecting a superpixel corresponding to the object is constructed using a graphical user interface (GUI) and an operating member.

12. The image processing device according to claim 10 or 11, in, The setting section sets a larger number of divisions as the shape of the object area specified by the specifying section becomes more complicated.

13. The image processing apparatus according to claim 2, further comprising a division number correction unit configured to correct the division number set by the setting unit, in, The division number correction part corrects the division number according to the correction amount of the correction part, and Wherein, in a case where the division number correction component corrects the division number, the determination component uses the corrected division number to determine the segment size.

14. The image processing device according to claim 13, in, The division number correction section increases the division number if the correction amount is larger than a predetermined threshold value, and decreases the division number if the correction amount is smaller than the predetermined threshold value.

15. The image processing device according to any one of claims 1 to 14, in, The input image is a dental image.

16. The image processing apparatus according to any one of claims 1 to 15, further comprising: include: a determination component configured to determine pixels having a brightness equal to or greater than a predetermined threshold value using the image data; as well as a control component configured to control an image to be displayed on the display component, The control component superimposes the pixel determined by the determination component on the image displayed on the display component in an emphasized manner.

17. An image processing method, include: An input step for inputting image data of an image; An acquisition step for acquiring the size of an object region included in the image and including the object to be extracted; A setting step, used to set the number of divisions into which the object area is divided; A determination step for determining a segment size of a superpixel based on the size of the object area and the number of divisions; as well as A generating step for generating superpixels using the image data, each of the superpixels having a size within a predetermined range including the determined segment size.

18. A program for causing a computer to function as each component of the image processing apparatus according to any one of claims 1 to 16.

19. A computer-readable storage medium for storing the program according to claim 18.

Images