IMAGE EXTRACTION METHOD, IMAGE EXTRACTION DEVICE, IMAGE EXTRACTION PROGRAM, AND RECORDING MEDIUM

Abstract

To more accurately extract a partial area of an image.SOLUTION: An image extraction method includes probability distribution estimation steps (S8, S10) and adjacent pixel analysis steps (S12-S18). In the probability distribution estimation steps, a first probability distribution of a probability that a pixel on an image is included in a first area in coordinates on a color space is estimated on the basis of color information on the color space of a first reference pixel. In the adjacent pixel analysis steps, a first pixel is reset on the basis of color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image extraction method for generating an image by extracting a partial area included in an image, an image extraction device for implementing the image extraction method, an image extraction program, and a recording medium. [Background technology]

[0002] Techniques for cutting out any region from an image are used in photo and video processing, etc. For example, Patent Document 1 discloses a method for extracting a desired region using a graph cut method. Patent Document 2 discloses a method for sequentially adding pixels adjacent to a certain reference pixel to pixels contained in each region using a region growing method. In particular, Patent Document 2 discloses a method for dividing each axis (R axis, G axis, B axis) of the RGB color space into s grids and extracting a desired region using a foreground color cluster and a background color cluster. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-121901 A [Patent Document 2] JP 2010-79477 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 uses a graph cut method, but due to algorithm constraints, it is difficult to extract the regions of multiple objects or estimate a dynamic color model. In addition, the graph cut method requires that the boundaries of the region to be extracted be determined from a large number of combinations, which takes a long time to process.

[0005] Patent Document 2 uses a region growing method, but there are cases where similar colors are used in both the foreground color cluster and the background color cluster, i.e., color conflicts. For this reason, for pixels containing similar colors where color conflicts have occurred, it may be erroneously determined whether the pixel is included in the foreground or the background. In particular, when extracting the regions of multiple objects within one screen, a color cluster is generated for each region. For this reason, many color conflicts may occur, making it difficult to perform accurate cutouts.

[0006] Here, when extracting the regions of multiple objects, the image is not divided into two values, foreground and background, but divided into multiple values, region 1, region 2, ..., region N, and each region is extracted. [Means for solving the problem]

[0007] In order to solve the above problem, an image extraction method according to one embodiment of the present disclosure is an image extraction method for extracting a specific region from an image, the image extraction method including: a receiving step of receiving designation of a first reference pixel which is a pixel that is a part of a first region included in the image; a reference pixel setting step of setting the first reference pixel as a first pixel; a probability distribution estimation step of estimating a first probability distribution which is a probability that a pixel on the image is included in the first region at a coordinate in the color space based on color information of the first reference pixel in a color space; an adjacent pixel analysis step of determining whether the first adjacent pixel is a pixel included in the first region based on color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution, and if it is determined that the first adjacent pixel is a pixel included in the first region, adding the first adjacent pixel to the first pixel and resetting the first adjacent pixel and the first pixel as a new first pixel; and an extraction step of extracting the reset first pixel as a first extracted image.

[0008] In addition, an image extraction device according to another aspect of the present disclosure includes a reference pixel setting unit that accepts designation of a first reference pixel which is a pixel that is part of a first region included in an image, and sets the first reference pixel as a first pixel; a probability distribution estimation unit that estimates a first probability distribution which is a probability that a pixel on the image is included in the first region at coordinates in the color space based on color information of the first reference pixel in a color space; an adjacent pixel analysis unit that determines whether the first adjacent pixel is a pixel included in the first region based on color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution, and if it is determined that the first adjacent pixel is a pixel included in the first region, adds the first adjacent pixel to the first pixel and resets the first adjacent pixel and the first pixel as a new first pixel; and an extracted image generation unit that extracts the reset first pixel as a first extraction region.

[0009] The image extraction device according to each aspect of the present disclosure may be realized by a computer. In this case, an image extraction program that causes a computer to operate as each part (software element) of the image extraction device to realize the image extraction device on a computer, and a computer-readable recording medium on which the image extraction program is recorded, also fall within the scope of the present disclosure. Effect of the Invention

[0010] According to one aspect of the present disclosure, the accuracy of image extraction can be further improved. [Brief description of the drawings]

[0011] [Figure 1] 1 is a flowchart for explaining an image extraction method according to the first embodiment of the present disclosure. [Diagram 2] 1 is a block diagram of an image extraction device according to a first embodiment of the present disclosure. [Diagram 3] 1 is a schematic process diagram illustrating a method for specifying a reference pixel according to the first embodiment of the present disclosure. FIG. [Figure 4] FIG. 4 is a schematic diagram showing an example of color coordinates in an RGB color space of each reference pixel according to the first embodiment of the present disclosure. [Diagram 5] 3A to 3C are schematic diagrams illustrating examples of probability distributions in an RGB color space according to the first embodiment of the present disclosure. [Figure 6] FIG. 2 is a process schematic diagram showing a method for analyzing adjacent pixels according to the first embodiment of the present disclosure. [Figure 7] FIG. 11 is another schematic process diagram illustrating the method for analyzing adjacent pixels according to embodiment 1 of the present disclosure. [Figure 8] FIG. 4 is a schematic diagram showing how each probability distribution in an RGB color space is modified according to the first embodiment of the present disclosure. [Figure 9] 1A to 1C are diagrams illustrating an image according to an embodiment of the present disclosure, an extraction process for the image, and an extraction result for the image. [Figure 10] FIG. 13 is a schematic diagram showing an example of each color bucket in an RGB color space according to a comparative example. [Figure 11] 11A and 11B are diagrams illustrating a comparison between an image extraction result according to an embodiment of the present disclosure and an image extraction result according to a comparative example. [Figure 12] FIG. 11 is a process schematic diagram showing a method for specifying a reference pixel according to a second embodiment of the present disclosure. [Figure 13] FIG. 11 is a process schematic diagram showing a method for specifying a reference pixel according to a third embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] [Embodiment 1] <Image Extraction Device Overview> The image extraction method according to the present embodiment is a method for extracting a specific region from an image and generating an image including the specific region. In particular, the image extraction method according to the present embodiment is a method for extracting a first extraction region including the first region and a second extraction region including the second region from an image having a first region and a second region, and generating a first extraction image including the first extraction region and a second extraction image including the second extraction region. The image extraction method will be described with reference to Figs. 1 and 2. Fig. 1 is a flowchart showing the image extraction method according to the present embodiment. Fig. 2 is a block diagram showing an image extraction device 2 and a display device 4 for executing the image extraction method.

[0013] The image extraction device 2 according to this embodiment includes a control unit 6. The image extraction device 2 may further include a memory 8, a communication unit 10, an operation unit 12, an input / output interface 14, and a system bus 16. The control unit 6 includes a reference pixel setting unit 20, a probability distribution estimation unit 22, an adjacent pixel analysis unit 24, and an extracted image generation unit 26. The control unit 6 may further include a display control unit 18 and a probability distribution correction unit 28.

[0014] The image extraction device 2 according to this embodiment displays an image on the display device 4 by a method described in detail later, and generates an extracted image by extracting a partial area from the image in response to an operation from a user.

[0015] The display device 4 is a device that displays images, and includes, for example, a display, a smartphone, a tablet terminal, etc. In addition, although the image extracting device 2 and the display device 4 are illustrated as separate entities in Fig. 2, this is not limiting, and the image extracting device 2 may include the display device 4 as a display unit.

[0016] The control unit 6 generates an extracted image by extracting a partial area from the image based on the information input to the image extraction device 2. The detailed operation of each part of the control unit 6 will be described in more detail later together with the explanation of the image extraction method according to the present embodiment. The control unit 6 may include a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and each of these may operate as each part of the control unit 6.

[0017] The memory 8 at least temporarily stores data necessary for the operation of each unit included in the image extracting device 2 or data input from each unit. The data stored in the memory 8 is read out from each unit included in the image extracting device 2 as appropriate.

[0018] The communication unit 10 is a communication device for transmitting at least a part of the data read from the memory 8 to an external device of the image extracting device 2, receiving data transmitted from the external device, and recording the data in the memory 8. For example, in this embodiment, the communication unit 10 is capable of mutual data communication with the display device 4.

[0019] The operation unit 12 is operated, for example, by a user of the image extraction device 2. For example, the operation unit 12 may include a keyboard or the like used to input data from the outside for image extraction by the image extraction device 2. For example, the operation unit 12 may be used to input data for making settings for image extraction or the like.

[0020] The input / output interface 14 executes interface processing between the memory 8, the communication unit 10, and the operation unit 12 and each unit of the control unit 6. The each unit of the control unit 6 and the input / output interface 14 are connected via a system bus 16.

[0021] <Specify the reference pixel> In the image extraction method according to the present embodiment, first, a user designation of a reference pixel that is a reference for image extraction is accepted from an image to be extracted. The process of accepting the designation of the reference pixel will be described with reference to FIG.

[0022] In the image extraction method according to the present embodiment, for example, it is assumed that the image 30 shown in Fig. 3 is first displayed on the display device 4 (step S2). The image extraction device 2 may cause the display device 4 to display the image 30, for example, by controlling the display control unit 18 of the control unit 6.

[0023] For example, image 30 shown in (a) of Fig. 3 has a first object region 32 in which a specified object is depicted as a foreground region 31, which is a first region. Image 30 also has a background region 36 in which a specified background is depicted as a second region different from the first region. In this embodiment, a method for generating a first extracted image including first object region 32, which is the first region, and a second extracted image including background region 36, which is the second region, from image 30 will be described as an example.

[0024] Following step S2, a receiving step is performed in which, for example, from a user, a designation of a first reference pixel which is a pixel of a part of the first region and a second reference pixel which is a pixel of a part of the second region is received (step S4). Step S4 is executed, for example, by the reference pixel setting unit 20 receiving, from the user, a designation of a first reference region 38 and a second reference region 40 shown in Fig. 3(b). The first reference region 38 is designated as the first object region 32 which is the first region, and the second reference region 40 is designated as the background region 36 which is the second region.

[0025] In this embodiment, the image 30 may further have a second object region 34 depicting another specified object as a foreground region 31 that is a first region, in addition to the first object region 32, as shown in (c) of Fig. 3. In this case, in step S4, the first reference region 38 may be designated for each of the first object region 32 and the second object region 34 that are first regions, as shown in (d) of Fig. 3. In this way, in this embodiment, when a certain region is divided into a plurality of regions by other regions, each reference region is designated for each of the plurality of regions.

[0026] The first reference area 38 and the second reference area 40 are specified, for example, by the user using a mouse, a touch pen, or the like to directly write the first reference area 38 and the second reference area 40 on the image 30 displayed on the display device 4. Note that in this embodiment, both the first reference area 38 and the second reference area 40 are specified in a linear shape, but this is not limiting, and they may be specified as areas in a specified figure, such as a rectangle or a circle.

[0027] As a result, in step S4, the pixels in the first reference region 38 are set as first reference pixels. Also, in step S4, the pixels in the second reference region 40 are set as second reference pixels. The setting of the first and second reference pixels is performed by the reference pixel setting unit 20 analyzing the specified first reference region 38 and second reference region 40, and identifying which pixels in the image 30 each reference region contains.

[0028] Next, in this embodiment, the first reference pixel is set as the first pixel, and the second reference pixel is set as the second pixel (step S6). Step S6 is executed, for example, by the reference pixel setting unit 20 following the setting of the first reference pixel and the second reference pixel.

[0029] <Color coordinates and probability distribution> Next, the probability distribution estimation unit 22 analyzes the color information of each of the first and second reference pixels, and identifies the coordinates in the color space of each reference pixel (step S8). An example of the color coordinates of each of the first and second reference pixels will be described with reference to FIG. 4. FIG. 4 is a diagram showing the color coordinates of each of the first and second reference pixels in the RGB color space. In the RGB color space shown in FIG. 4, first color coordinates 42 obtained from the color information of each of the first reference pixels and second color coordinates 44 obtained from the color information of each of the second reference pixels are plotted.

[0030] Next, the probability distribution estimation unit 22 estimates a first probability distribution and a second probability distribution in the color space from the color coordinates of the first reference pixel and the second reference pixel (step S10). Here, the first probability distribution and the second probability distribution are distributions that express the probability that a pixel on the image 30 having a color corresponding to each coordinate on the color space is included in the color of the pixel included in the first region and the second region, respectively.

[0031] In other words, the first probability distribution is the probability that a pixel on the image 30 is included in the first region in the coordinates of the color space, estimated based on the color information of the first reference pixel in the color space, and the second probability distribution is the probability that a pixel on the image 30 is included in the second region in the coordinates of the color space, estimated based on the color information of the second reference pixel in the color space.

[0032] The first probability distribution and the second probability distribution, together with their estimation methods, will be described with reference to Fig. 5. Fig. 5 is a diagram showing a first probability distribution 46 and a second probability distribution 48 in an RGB color space. Note that in Fig. 5, for convenience of illustration, the first probability distribution 46 and the second probability distribution 48 are expressed as distributions on a certain plane in the RGB color space, but in reality, they are expressed as three-dimensional distributions in the RGB color space.

[0033] In general, for example, in an image having both a foreground region 31 and a background region 36, the color of a pixel located in the foreground region 31 is highly likely to be similar to the color of pixels located around that pixel. On the other hand, the colors of the pixels located in the foreground region 31 and the pixels located in the background region 36 are less likely to be similar to each other.

[0034] Generally, the distance between two color coordinates on the RGB color space can represent the similarity of the colors corresponding to the color coordinates. Specifically, the closer the distance between two color coordinates on the RGB color space, the more similar the colors corresponding to the color coordinates are to each other.

[0035] Therefore, for example, when many first color coordinates 42 are plotted around a certain color coordinate in the RGB color space, it can be said that there is a high probability that the color coordinates of the colors of the pixels included in the first region are located around the specific color coordinate. Therefore, the first probability distribution 46 is determined so that it is high around the coordinates where many first color coordinates 42 are plotted, and low around the coordinates where the number of plots of the first color coordinates 42 is low. For the same reason, the second probability distribution 48 is determined so that it is high around the coordinates where many second color coordinates 44 are plotted, and low around the coordinates where the number of plots of the second color coordinates 44 is low.

[0036] 5, the first probability distribution 46 has a first low probability region 46A, a first medium probability region 46B, and a first high probability region 46C in this order from the periphery toward the coordinates where at least one of the first probability distributions 46 has a high probability. Also, the second probability distribution 48 has a second low probability region 48A, a second medium probability region 48B, and a second high probability region 48C in this order from the periphery toward the coordinates where at least one of the second probability distributions 48 has a high probability.

[0037] The first probability distribution 46 and the second probability distribution 48 are determined, for example, by a Gaussian Mixture Model (GMM) in the RGB color space. For example, the first probability distribution 46 and the second probability distribution 48 have a distribution in which a plurality of spatial Gaussian distributions in the RGB color space are superimposed. However, this embodiment is not limited to this, and the first probability distribution 46 and the second probability distribution 48 are not particularly limited as long as they are distributions that express the probability of the color coordinates of the pixels in the first region and the pixels in the second region.

[0038] In the rest of this specification, when the term "probability distribution" is used, it refers to the distribution of the probability that a pixel in an image having a color corresponding to each coordinate in color space is included in the color of the pixels contained in a certain area.

[0039] <Adding pixels> Next, an adjacent pixel analysis step is performed in which one of the pixels adjacent to the pixel already set as a pixel of each region is added to a pixel of one of the regions. The adjacent pixel analysis step will be described with further reference to Figures 6 and 7. Figures 6 and 7 are schematic diagrams of the step that show, in an enlarged manner, pixels around the already set first pixel and second pixel among the pixels of image 30.

[0040] 6 and 7, the image 30 has a structure in which a plurality of substantially rectangular pixels 50 are arranged along the row and column directions. Here, the pixels 50 each include a first pixel 52 that is set as a pixel belonging to the first region, and a second pixel 54 that is set as a pixel belonging to the second region.

[0041] In the adjacent pixel analysis step, first, a first adjacent pixel 56, which is a pixel adjacent to the first pixel 52 in the row direction or column direction, is identified. In addition, a second adjacent pixel 58, which is a pixel adjacent to the second pixel 54 in the row direction or column direction, is identified (step S12). Depending on the positions of the first pixel 52 and the second pixel 54, a conflicting pixel 60 may be generated that is adjacent to both the first pixel 52 and the second pixel 54. The conflicting pixel 60 is treated as an adjacent pixel that is included in both the first adjacent pixel 56 and the second adjacent pixel 58. Next, color information of each of the first adjacent pixel 56 and the second adjacent pixel 58 is analyzed, and coordinates in the RGB color space are calculated (step S14).

[0042] Next, the pixel most likely to be included in the first pixel 52 is determined from the first adjacent pixels 56. Also, the pixel most likely to be included in the second pixel 54 is determined from the second adjacent pixels 58 (step S16). In this embodiment, the cost C N An example will be described in which step S16 is achieved by calculating (s, t) for each adjacent pixel and comparing them.

[0043]

number

[0044] In formula (1), one of the pixels in each region is pixel s, one of the pixels adjacent to pixel s is pixel t, and N represents region N including pixel s.

[0045] D(s,t) is the Euclidean distance in the RGB color space between the color of pixel s and the color of pixel t, and represents the similarity between the colors of pixel s and pixel t. The smaller D(s,t) is, the more similar the colors of pixel s and pixel t are, and the higher the probability that pixel t is included in region N. C(s) is the cost C of pixel s when pixel s is set in each region. N The value of (s,t). If pixel s is the reference pixel, C(s) is set to 0.

[0046] W 1 (s, t) is a weighting for D(s, t) and is defined by the following equation (2).

[0047]

number

[0048] In formula (2), l M (t) represents the probability that the color of pixel t is included in the colors of pixels contained in region M. M (t) is calculated, for example, by comparing the coordinates of the color of pixel t in the RGB color space with the probability distribution in region M. If the color of pixel t is highly likely to be included in the colors of region M, in other words, M If the value of (t) is high, the pixel t is likely to be included in the region M. 0 | is l M (t) is found by searching all pixels t, M (t) is the absolute minimum value of W 1 This is a term for normalizing (s,t) so that it does not become a negative value. From equations (1) and (2), the higher the similarity between the color of pixel s and the color of pixel t, and the higher the probability that the color of pixel t is included in the color of region N, the higher the cost C N (s,t) becomes lower.

[0049] W 2 (s, t) is a weighting for C(s) and is defined by the following equation (3).

[0050]

number

[0051] From equation (3), if the probability that the color of pixel t is included in the color of region N is higher than the probability that the color of pixel t is included in the color of any region other than region N, then W 2 The value of (s,t) is 0, otherwise, W 2 The value of (s,t) is 1. In other words, W 2 (s,t) is the cost C when pixel s is set as a pixel in a certain region when the color of pixel t is highly likely to be included in the colors of region N. N This term is provided to avoid taking (s,t) into account.

[0052] From the above, each pixel t is assigned a calculated cost C N The lower the (s, t), the higher the probability that the pixel s is included in the region N. Therefore, in step S16, for example, the cost C N (s, t) are calculated, and the lowest pixel is identified. As a result, the first adjacent pixel 56 and the second adjacent pixel 58 are identified as having the lowest cost C N A first pixel candidate 62, which is the pixel with the lowest cost (s, t), is identified. Here, for example, N If there are multiple pixels with the lowest (s,t), then multiple pixel candidates may be identified, for example a second pixel candidate 64 different from the first pixel candidate 62 may be further identified.

[0053] For example, if there are multiple first pixels 52 adjacent to a certain first adjacent pixel 56, in step S16, each of the first pixels 52 is treated as a pixel s, and the cost C N (s, t) may be calculated multiple times. In this case, the calculated multiple costs C NThe average value of (s, t) is calculated as the cost C N It may be considered as (s,t).

[0054] Next, the adjacent pixel is added to the pixels included in the region that is most likely to include the adjacent pixel, and the pixel is reset (step S18). Specifically, the first pixel candidate 62 is added to the pixels in the region that is most likely to include the first pixel candidate 62 and that is adjacent to the first pixel candidate 62. For example, as shown in FIG. 7, if the first pixel candidate 62 is adjacent only to the first pixel 52, the first pixel candidate 62 is added to the first pixel 52. Also, if the first pixel candidate 62 is a competitive pixel 60 that is adjacent to both the first pixel 52 and the second pixel 54, the first pixel candidate 62 is added to the first pixel 52, and the cost C is reduced. N The pixel is added to the pixel in the region where (s, t) is low. Accordingly, the first adjacent pixel 56 adjacent to the added first pixel 52 is added.

[0055] In other words, in the adjacent pixel analysis step in this embodiment, it is determined whether or not the first adjacent pixel 56 is a pixel included in the first region, based on the color information of the first adjacent pixel 56 and the first probability distribution 46. Furthermore, in the adjacent pixel analysis step in this embodiment, if it is determined that the first adjacent pixel 56 is a pixel included in the first region, the first adjacent pixel 56 is added to the first pixel 52, and the first adjacent pixel 56 and the first pixel 52 are reset as a new first pixel 52.

[0056] Moreover, in the adjacent pixel analysis step in this embodiment, it is determined whether or not the second adjacent pixel 58 is a pixel included in the second region, based on the color information of the second adjacent pixel 58 and the second probability distribution 48. Furthermore, in the adjacent pixel analysis step in this embodiment, if it is determined that the second adjacent pixel 58 is a pixel included in the second region, the second adjacent pixel 58 is added to the second pixel 54, and the second adjacent pixel 58 and the second pixel 54 are reset as a new second pixel 54.

[0057] If the second pixel candidate 64 is identified in step S16, then in step S18, the second pixel candidate 64 may also be added to a pixel in an area that is adjacent to the second pixel candidate 64 and is most likely to include the second pixel candidate 64. For example, as shown in Fig. 7, if the second pixel candidate 64 is adjacent only to the second pixel 54, the second pixel candidate 64 is added to the second pixel 54.

[0058] In this way, the adjacent pixel analysis step is executed, one of the adjacent pixels is added to the pixels included in each region, and the pixels included in each region are reset. Note that steps S12 to S18 of the adjacent pixel analysis step are executed by the adjacent pixel analysis unit 24 by analyzing each pixel 50.

[0059] As described above, the neighboring pixels to be added to the pixel of each region are determined based on the probability indicating which of the regions the neighboring pixels are included in. Therefore, the neighboring pixel analysis step can automatically determine which pixel the conflicting pixel 60 should be added to. In other words, in the neighboring pixel analysis step, when the first neighboring pixel 56 and the second neighboring pixel 58 are the same pixel, it is determined whether the conflicting pixel 60 should be added to the first pixel 52 or the second pixel 54 based on the first probability distribution 46 and the second probability distribution 48. Therefore, in this embodiment, the probability that the conflicting pixel 60 is set to a pixel included in an inappropriate region is more effectively reduced.

[0060] <Determining when to finish adding pixels and correcting the probability distribution> Next, it is determined whether or not there is any pixel among the pixels 50 that has not been set to any of the regions (step S20). If there is any pixel that has not been set to any of the regions in step S20, it is then determined whether or not the number of pixels that have been set to the region to which they belong has increased by a certain amount or more since the probability distribution in each region was determined (step S22).

[0061] When step S22 is executed for the first time, step S22 counts the number of pixels each region contains that has increased since step S10 was executed. Step S22 counts the increase in the number of pixels each region contains based on the number of pixels each region contains when step S10 was executed, until the number of pixels in each region increases to a certain number or more.

[0062] In step S22, if the increase in the number of pixels contained in each region is not equal to or greater than a certain level, step S12 is executed again. Therefore, each time steps S12 to S22 are repeated, the number of pixels contained in each region increases. In step S22, if the increase in the number of pixels contained in each region is equal to or greater than a certain level, a probability distribution correction step is executed to correct the probability distribution in the color space for each region (step S24).

[0063] In step S24, the probability distribution in each region is modified based on the color information of each pixel currently included in each region. In other words, in step S24, the first probability distribution 46 is modified based on the color information in the RGB color space of the reset first pixel 52. In step S24, the second probability distribution 48 is modified based on the color information in the RGB color space of the reset second pixel 54. The modification of each probability distribution in step S24 is performed by the same method as the estimation of each probability distribution in step S10.

[0064] Once step S24 has been executed, in the next and subsequent steps S22, the increase in the number of pixels contained in each region is counted based on the number of pixels contained in each region at the time step S24 was executed last time, and it is determined whether step S24 needs to be executed. In other words, step S24 is executed if a prescribed number or more of step S18 have been executed since the most recent execution of step S10 or step S24.

[0065] The correction of the probability distribution in each region in step S24 will be described in more detail with reference to FIG. 8. S24-A shown in FIG. 8(a) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 before step S24 is executed for the first time. S24-B shown in FIG. 8(b) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 after the correction of the probability distribution in step S24 is executed once. S24-C shown in FIG. 8(c) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 after the correction of the probability distribution in step S24 is executed twice. S24-D shown in FIG. 8(d) is an RGB color space showing the first probability distribution 46 and the second probability distribution 48 when the first probability distribution 46 and the second probability distribution 48 are estimated from all pixels of the image 30.

[0066] In step S10, a first probability distribution 46 and a second probability distribution 48 are estimated from color information of a first reference pixel that is part of the first region and a second reference pixel that is part of the second region. Therefore, the first probability distribution 46 and the second probability distribution 48 estimated in step S10 are likely to differ from the ideal first probability distribution 46 and the second probability distribution 48 estimated from all pixels of the image 30.

[0067] The estimation of the probability distribution in step S24 uses an increased number of reference pixels compared to the estimation of the probability distribution in step S10. Therefore, the probability distribution corrected in step S24 is more likely to approximate an ideal probability distribution estimated from all pixels of the image 30. Furthermore, the more times step S24 is performed, in other words, the more the probability distribution is corrected, the more likely the probability distribution will approximate the ideal probability distribution.

[0068] Therefore, each time the number of set pixels increases by a certain amount or more, step S24 is executed to correct the probability distribution, so that the analysis of each adjacent pixel is executed based on a probability distribution closer to an ideal probability distribution. Therefore, by executing the probability distribution correction process of step S24, the probability that adjacent pixels are added to pixels in an appropriate area is further improved. Note that step S24 is executed by the probability distribution correction unit 28, but is not limited to this, and may be executed by the probability distribution estimation unit 22.

[0069] In this embodiment, step S24 is performed after a specified number of pixels or more have been set since the previous estimation or modification of the probability distribution. If the difference in the number of color coordinates used to estimate the probability distribution is small, the probability distribution will fluctuate only slightly even if it is modified. For this reason, it is more effective to modify the probability distribution when the number of reference pixels has increased to a certain extent. Therefore, in this embodiment, the speed of image extraction can be improved while modifying the probability distribution more effectively than by performing step S24 every time the number of set pixels increases.

[0070] <Image Extraction> Steps S12 to S24 are repeated, and when all pixels 50 in the image 30 have been set as pixels in any of the regions, the process proceeds from step S20 to an extraction step (step S26).

[0071] In step S26, an image including the pixels included in each region is extracted as an extracted image. Specifically, in step S26, the first pixel 52 is extracted as a first extracted image, and the second pixel 54 is extracted as a second extracted image. The extraction step is performed by the extracted image generating unit 26 identifying the first pixel 52 and the second pixel 54 and extracting images including them.

[0072] As a result of the above, ideally, a first extracted image including pixels in the foreground region 31 and a second extracted image including pixels in the background region 36 are generated. Therefore, in this embodiment, an image in which the foreground region 31 is extracted and an image in which the background region 36 is extracted are obtained from the image 30.

[0073] In the image extraction method according to the present embodiment, the area in which each pixel is included is determined based on the probability distribution in each area. Therefore, the image extraction method according to the present embodiment reduces the probability that each pixel is set as a pixel of an inappropriate area, and improves the accuracy of extracting pixels included in each area from an image. The image extraction method can be realized by using the image extraction device 2 according to the present embodiment.

[0074] Furthermore, in this embodiment, all pixels 50 in the image 30 are set to either the first pixel 52 or the second pixel 54. This makes it possible to more reliably set which region each pixel is included in, compared to setting the pixels included in one region and then generating an extracted image. Therefore, in this embodiment, the accuracy of generating the first extracted image and the second extracted image is further improved.

[0075] <Evaluation of Examples and Comparative Examples> The effect of the image extraction method according to the present embodiment is evaluated by comparing an example carried out according to the present embodiment with a comparative example carried out according to the image extraction method according to the comparative embodiment. The process of generating an extracted image from a specific image in the example will be described with reference to FIG.

[0076] In the example, an extracted image was generated from an image 66 shown in Fig. 9(a) based on the image extraction method according to this embodiment. The image 66 is an image including an object region 68 and a background region 70. In the example, in step S4, a first reference region 38 and a second reference region 40 were specified as shown in Fig. 9(a).

[0077] Thereafter, based on the image extraction method according to this embodiment, an attempt was made to generate from image 66 a first extracted image including object region 68 as foreground region 31, and a second extracted image including background region 70. In the example, step S24 was executed when 40% and 80% of all pixels in image 66 were added to either region.

[0078] A first extracted image 72A shown in (b-1) of FIG. 9 is an extracted image including the first pixel 52 immediately before the first step S24, in other words, at the time when 40% of all pixels of the image 66 are set to the first pixel 52 or the second pixel 54. A second extracted image 74A shown in (c-1) of FIG. 9 is an extracted image including the second pixel 54 immediately before the first step S24. A first extracted image 72B shown in (b-2) of FIG. 9 is an extracted image including the first pixel 52 immediately before the second step S24, in other words, at the time when 80% of all pixels of the image 66 are set to the first pixel 52 or the second pixel 54. A second extracted image 74B shown in (c-2) of FIG. 9 is an extracted image including the second pixel 54 immediately before the second step S24. A first extracted image 72 shown in (b-3) of Fig. 9 is an extracted image generated in step S26 and including the first pixel 52. A second extracted image 74 shown in (c-3) of Fig. 9 is an extracted image generated in step S26 and including the second pixel 54.

[0079] 9 show how the first pixel 52 and the second pixel 54 are gradually added in sequence in this embodiment. Also, in this embodiment, the first extracted image 72 including the first pixel 76 at a position corresponding to the foreground region 31 and the second extracted image 74 including the second pixel 78 at a position corresponding to the background region 70 are obtained.

[0080] Next, an image extraction method according to a comparative embodiment will be described. The image extraction method according to the comparative embodiment differs from the image extraction method according to the present embodiment in that it includes a step of generating color buckets in a color space for each region instead of step S10. This step will be described with reference to FIG.

[0081] S10-C in Fig. 10 is a schematic diagram showing an example of color buckets in an RGB color space generated in a comparative embodiment. In the comparative embodiment, after identifying the color coordinates in the color space of each reference pixel, color buckets are generated in each region from the information of the color coordinates. In the comparative embodiment, the R axis, G axis, and B axis in the RGB color space are divided into a plurality of grids, the RGB color space is divided into a plurality of sections, and color buckets are generated in some of the sections. The color buckets in each region are generated in sections where the color coordinates of each reference pixel are mostly located.

[0082] In the comparative embodiment, for example, as shown in Fig. 10, a first color bucket 46D, which is a color bucket in the first region, is generated in a section that contains many first color coordinates 42 among some sections in the RGB color space. Also, in the comparative embodiment, for example, as shown in Fig. 10, a second color bucket 48D, which is a color bucket in the second region, is generated in a section that contains many second color coordinates 44 among some sections in the RGB color space. Here, in a section that has both the first color coordinates 42 and the second color coordinates 44, a competing color bucket CB that belongs to both the first color bucket 46D and the second color bucket 48D may be generated.

[0083] In the comparative example, the cost C N Calculate (s,t) and calculate the cost C N The pixel with the lowest (s,t) is added to one of the pixels contained in each region. However, the cost C N W of (s,t) 1 (s,t) and W 2 (s, t) are changed to the following equations (4) and (5), respectively.

[0084]

number

[0085]

number

[0086] In equations (4) and (5), q(t) represents the color of pixel t with a reduced quantum number, and corresponds to the color coordinate of pixel t. M represents the partition in the RGB color space where the color buckets in the area M are generated. Therefore, from equation (4), W 1 (s,t) is 1 if the color coordinates of pixel t are included only in the color bucket of region N, 4 if they are included only in the color buckets of regions other than region N, and 2 otherwise. Also, from equation (5), W 2 (s,t) is 0 if the color coordinates of pixel t are included in the color bucket of region N, and 1 otherwise.

[0087] Furthermore, in the comparative embodiment, steps S22 and S24 are not executed, and if there is a pixel that has not been set as one of the pixels included in any of the regions in step S20, step S12 is executed again.

[0088] Except for the above-mentioned points, the image extraction method according to the comparative embodiment is executed in the same manner as the image extraction method according to the present embodiment.

[0089] In the comparative example, each extracted image was generated from an image 66 by specifying the same reference areas as the first reference area 38 and the second reference area 40 shown in Fig. 9, using an image extraction method according to a comparative embodiment. The extracted images generated in the comparative example and the extracted images generated in the embodiment will be compared with reference to Fig. 11.

[0090] Fig. 11(a) is a diagram showing a first extracted image 72C generated in the comparative example. Fig. 11(b) is a diagram showing a first extracted image 72 generated in the example. In Fig. 11(a) and (b), in order to more clearly compare the accuracy of image extraction, second pixels 78C and 78, which correspond to the background region 70, are overlaid on the first extracted image 72C and the first extracted image 72, respectively, with reduced density.

[0091] The first extracted image 72C generated in the comparative example has a first pixel 76C that corresponds to the object region 68. However, the first extracted image 72C has an insufficient region E in which some of the pixels located in the object region 68 are not set as the first pixel 76C. It is presumed that the insufficient region E has occurred because the colors of the pixels around the insufficient region E are included in the competing color bucket CB, and therefore some of the pixels located in the object region 68 are erroneously set as the second pixel 78C.

[0092] On the other hand, the first extracted image 72 generated in the example does not have an insufficient area E, and the setting of the first pixel 76 is generally successful. Thus, the image extraction method according to the present embodiment can generate an extracted image with higher accuracy than the image extraction method according to the comparative example.

[0093] [Embodiment 2] <Generation of the third extracted image> Other embodiments of the present disclosure will be described below. For convenience of explanation, the same reference numerals will be given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0094] In this embodiment as well, a method for generating an extracted image from image 30 will be described as an example. However, this embodiment differs from the previous embodiment in that a third extracted image is generated in addition to the first extracted image and the second extracted image.

[0095] In the image extraction method according to this embodiment, for example, first, the image 30 shown in FIG. 12 is displayed on the display device 4 by the same method as step S2 in the previous embodiment. Note that the image 30 shown in FIG. 12(a) is the same as the image 30 shown in FIG. 3(c). However, in this embodiment, the first region has only the first object region 32, and the second object region 34 is included in the third region. Furthermore, in this embodiment, a method of generating a first extracted image including the first object region 32 as the first region, a second extracted image including the background region 36 as the second region, and a third extracted image including the second object region 34 as the third region from the image 30 will be described as an example. In other words, the image extraction method according to this embodiment is a method of further extracting a third extracted region including the third region from an image having the third region, and generating a third extracted image including the third extracted region.

[0096] In this embodiment, in addition to the designation of the first and second reference pixels in step S4 in the previous embodiment, the designation of a third reference pixel is accepted. The designation of the third reference pixel is performed by designating a third reference region 80 in the second object region 34, which is the third region, and designating a pixel included in the third reference region 80 as the third reference pixel, as shown in (b) of Fig. 12. The designation of the third reference pixel is performed by the same method as the designation of the first and second reference pixels.

[0097] Hereafter, in this embodiment, steps that are partially modified from the steps in the previous embodiment are executed in sequence. In step S6 in this embodiment, a third reference pixel is further set to the third pixel. In steps S8 and S10 in this embodiment, a third probability distribution is further estimated, which expresses the probability that a pixel on image 30 having a color corresponding to each coordinate in the color space is included in the color of the pixel included in the third region. In steps S12 to S18 in this embodiment, a cost C N(s, t) is calculated and each pixel is set. In step S24 according to this embodiment, the third probability distribution is further modified. In step S26 according to this embodiment, a third extracted image including the third pixel is further generated. Each of steps S6 to S26 according to this embodiment is performed by the same method as each of steps S6 to S26 according to the previous embodiment, except for the points described above. The image extraction method according to this embodiment is performed by the same device as the image extraction device 2 according to the previous embodiment.

[0098] In this embodiment, in addition to the first and second extracted images, a third extracted image including a third pixel can be generated. Therefore, according to the image extraction method according to this embodiment, for example, from an image having a plurality of object regions in a background region, a plurality of extracted images having different object regions and an extracted image having a background region can be obtained.

[0099] In this embodiment, the area in which each pixel is included is determined based on the probability distribution in each area. Therefore, according to the image extraction method of this embodiment, even if the number of types of areas to be extracted increases, the probability that each pixel is set as a pixel of an inappropriate area decreases, and the accuracy of extracting pixels included in each area from an image is improved.

[0100] In this embodiment, the image 30 may further have a region different from any of the first to third regions, including, for example, a fourth region, a fifth region, etc. In this case, in this embodiment, the image may be extracted using four or more types of reference pixels including not only the first reference pixel, the second reference pixel, and the third reference pixel, but also the fourth reference pixel, the fifth reference pixel, etc. Even in this case, according to the image extraction method of this embodiment, an extracted image including pixels of the region can be further generated.

[0101] In this embodiment, the first region and the third region include the first object region 32 and the second object region 34 of the image 30, respectively. Therefore, in the image extraction method according to this embodiment, it is possible to generate a plurality of extracted images in which a plurality of object regions are individually extracted from an image. However, this is not limited thereto, and in this embodiment, the third region may include a portion of the background region 36. In this case, in the image extraction method according to this embodiment, it is possible to generate a plurality of extracted images in which a portion of the background region is each extracted from an image.

[0102] [Embodiment 3] <Generation of the first extracted image only> Another embodiment of the present disclosure will be described below. In this embodiment, too, a method for generating an extracted image from an image 30 will be described as an example. However, this embodiment differs from the above-described embodiments in that only a first extracted image is generated.

[0103] In the image extraction method according to this embodiment, for example, first, an image 30 shown in Fig. 13(a) is displayed on the display device 4 by the same method as step S2 in each of the above-mentioned embodiments. Note that the image 30 shown in Fig. 13(a) is the same as the image 30 shown in Fig. 3(a). However, in this embodiment, a method of generating only a first extracted image including a first object region 32 as a foreground region 31, which is a first region, from the image 30 will be described as an example.

[0104] In this embodiment, only the designation of the first reference pixel is accepted in step S4 in the previous embodiment, so that in this embodiment, only the first reference region 38 is designated for the image 30, as shown in (b) of FIG.

[0105] From this point on, in this embodiment, steps that are partially modified from the steps in the previous embodiment are executed in sequence. In step S6 in this embodiment, only the first pixel 52 is set as the first reference pixel. In steps S8 and S10 in this embodiment, only the first probability distribution 46 is estimated. In steps S12 to S18 in this embodiment, only the first adjacent pixel 56 is identified, and a cost C N (s, t) is calculated, and only the first pixel 52 is reset. In step S24 according to the present embodiment, only the first probability distribution 46 is corrected. In step S26 according to the present embodiment, only the first extracted image including the first pixel 52 is generated.

[0106] Here, in the image extraction method according to the present embodiment, in order to generate only the first extracted image including the first region, not all pixels 50 of the image 30 are added to the first pixel 52, but only some of the pixels 50 are added to the first pixel 52. For this reason, it is considered that step S20 according to the present embodiment may be changed to, for example, the following process.

[0107] For example, in step S20 according to this embodiment, the cost C N It is determined whether (s, t) is equal to or greater than a certain specified value. In this case, in step S20, the cost C of all the first adjacent pixels 56 is calculated. N If (s, t) is equal to or greater than the specified value, the process proceeds to step S26. If the color difference between the first region and the second region of the image 30 is clear and the cost C N Step S20 is useful when it can be determined that all of the first pixels 52 have been identified if (s, t) is equal to or greater than a specified value.

[0108] Alternatively, for example, in step S20 according to the present embodiment, it is determined whether or not the total number of first pixels 52 is equal to or greater than a certain specified value. In this case, if it is determined in step S20 that the total number of first pixels 52 is equal to or greater than the specified value, the process proceeds to execution of step S26. Step S20 is useful when the ratio of the area of ​​the first region to the total area of ​​the image 30 is known at least approximately.

[0109] In the image extraction method according to the present embodiment, by performing step S20, only a part of the pixels 50 of the image 30 can be added to the first pixels 52. As a result, among the pixels 50, N Since the number of pixels 50 for which (s, t) is to be calculated is reduced, the image extraction method according to this embodiment can shorten the time required to generate the first extracted image.

[0110] Each of steps S6 to S26 according to the present embodiment is executed by the same method as each of steps S6 to S26 according to the previous embodiment, except for the points described above. The image extraction method according to the present embodiment is executed by the same device as the image extraction device 2 according to the previous embodiment.

[0111] In this embodiment, by simply specifying the first reference pixel, a first extracted image including at least the first pixel 52 can be generated. Therefore, according to the image extraction method according to this embodiment, the user does not need to specify the second reference pixel, and the first extracted image can be generated more simply.

[0112] Also in this embodiment, whether each pixel is included in the first pixel 52 is determined based on the first probability distribution 46 in the first region. Therefore, according to the image extraction method of this embodiment, the probability that a pixel 50 that should be added to the first pixel 52 is not added to the first pixel 52 decreases. In addition, according to the image extraction method of this embodiment, the probability that a pixel 50 that should not be added to the first pixel 52 is added to the first pixel 52 decreases. Therefore, according to the image extraction method of this embodiment, it is possible to generate a first extracted image including the first pixel 52 from the image 30 more simply and accurately.

[0113] In this embodiment, the first region includes the first object region 32. Therefore, in the image extraction method according to this embodiment, an extracted image in which only the object region is extracted from the image can be generated. However, the present disclosure is not limited to this, and in this embodiment, the first region may not include the first object region 32, but may include at least a part of the background region 36. In this case, in the image extraction method according to this embodiment, an extracted image in which only at least a part of the background region is extracted from the image can be generated. In addition, the first region may include the first object region 32 and the second object region 34.

[0114] <Example of software implementation> The functions of the image extraction device (hereinafter, referred to as the "device") according to each of the above-described embodiments can be realized by an image extraction program for causing a computer to function as the device. In this case, the functions of the device can be realized by a program for causing a computer to function as each unit included in the control unit 6, for example.

[0115] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0116] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0117] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0118] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may be executed by the control device or another device (for example, an edge computer or a cloud server).

[0119] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0120] 2. Image Extraction Device 6. Control Unit 20 Reference pixel setting section 22 Probability distribution estimation unit 24 Adjacent pixel analysis section 26 Extraction image generation unit 28 Probability distribution correction unit

Claims

1. An image extraction method for extracting a specific region from an image, comprising the steps of: a receiving step of receiving a designation of a first reference pixel which is a pixel that is part of a first region included in the image; a reference pixel setting step of setting the first reference pixel as a first pixel; a probability distribution estimation step of estimating a first probability distribution, which is a probability that a pixel on the image is included in the first region at a coordinate in the color space, based on color information in the color space of the first reference pixel; an adjacent pixel analysis step of determining whether the first adjacent pixel is a pixel included in the first region based on color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution, and adding the first adjacent pixel to the first pixel when it is determined that the first adjacent pixel is a pixel included in the first region, and resetting the first adjacent pixel and the first pixel as a new first pixel; an extraction step of extracting the reset first pixel as a first extracted image; and a probability distribution modifying step of modifying the first probability distribution based on the reset color information of the first pixel in the color space.

2. the image has a second region different from the first region; The receiving step further includes receiving a designation of a second reference pixel which is a pixel that is a part of the second region; The reference pixel setting step further includes setting the second reference pixel as a second pixel; the probability distribution estimation step further comprises estimating a second probability distribution, which is a probability that a pixel on the image is included in the second region at a coordinate in the color space, based on color information in the color space of the second reference pixel; the adjacent pixel analyzing step further includes judging whether the second adjacent pixel is a pixel included in the second region based on color information of a second adjacent pixel adjacent to the second pixel and the second probability distribution, and when it is determined that the second adjacent pixel is a pixel included in the second region, adding the second adjacent pixel to the second pixel, and resetting the second adjacent pixel and the second pixel as a new second pixel; 2. The image extraction method according to claim 1, wherein the extraction step further comprises extracting the reset second pixel as a second extracted region.

3. 3. The method of claim 2, further comprising repeatedly performing the adjacent pixel analysis step until all pixels in the image are set to either the first pixel or the second pixel.

4. 4. The image extraction method according to claim 2 or 3, wherein, in the adjacent pixel analysis step, when the first adjacent pixel and the second adjacent pixel are the same pixel, it is determined whether to add the pixel to the first pixel or the second pixel based on the first probability distribution and the second probability distribution.

5. the image has a third region that is different from both the first region and the second region; The receiving step further includes receiving a designation of a third reference pixel which is a pixel that is a part of the third region; The reference pixel setting step further includes setting the third reference pixel as a third pixel; the probability distribution estimation step further comprises estimating a third probability distribution, which is a probability that a pixel on the image is included in the third region at a coordinate in the color space, based on color information in the color space of the third reference pixel; the neighboring pixel analyzing step further comprises determining whether the third neighboring pixel is a pixel included in the third region based on color information of a third neighboring pixel adjacent to the third pixel and the third probability distribution, and when it is determined that the third neighboring pixel is a pixel included in the third region, adding the third neighboring pixel to the third pixel, and resetting the third neighboring pixel and the third pixel as a new third pixel; The image extraction method according to claim 2 , wherein the extraction step further comprises extracting the reset third pixel as a third extraction region.

6. The image extraction method according to claim 2 , further comprising a probability distribution correction step of correcting the second probability distribution based on the reset color information of the second pixel in the color space.

7. The image extraction method according to claim 5 , further comprising a probability distribution modification step of modifying the third probability distribution based on the reset color information of the third pixel in the color space.

8. The image extraction method according to claim 1 , wherein the probability distribution correction step is executed when the adjacent pixel analysis step has been executed a specified number of times or more since the most recent execution of the probability distribution estimation step or the probability distribution correction step.

9. a reference pixel setting unit that receives a designation of a first reference pixel that is a pixel of a part of a first region included in an image, and sets the first reference pixel as a first pixel; a probability distribution estimation unit that estimates a first probability distribution, which is a probability that a pixel on the image is included in the first region at a coordinate in the color space, based on color information on the color space of the first reference pixel; an adjacent pixel analysis unit that determines whether or not the first adjacent pixel is a pixel included in the first region based on color information of a first adjacent pixel adjacent to the first pixel and the first probability distribution, and adds the first adjacent pixel to the first pixel when it is determined that the first adjacent pixel is a pixel included in the first region, and resets the first adjacent pixel and the first pixel as a new first pixel; an extraction image generating unit that extracts the reset first pixel as a first extraction region; a probability distribution correction unit that corrects the first probability distribution based on the reset color information of the first pixel in the color space.

10. 10. An image extraction program for causing a computer to function as the image extraction device described in claim 9, the image extraction program for causing the computer to function as the reference pixel setting unit, the probability distribution estimation unit, the adjacent pixel analysis unit, the extracted image generation unit, and the probability distribution correction unit.

11. A computer-readable recording medium having the image extraction program according to claim 10 recorded thereon.

Images