Image processing device, image processing method, and program

Abstract

To enable acquisition of color information that enables extraction of a specific object area from a captured image with high accuracy, with less burden on a user in a short time.SOLUTION: An image processing apparatus acquires color information about an object and a background in a first image captured by an image capturing apparatus, and stores the acquired color information about the object and the background. Then, the stored color information is expanded in a three-dimensional color space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for processing captured images. [Background technology]

[0002] In recent years, research has been conducted on mixed reality, which presents information of a virtual space to a user by superimposing it on a real space in real time. An information processing device that realizes mixed reality generates an image of a virtual space according to the position and orientation of an imaging device by computer graphics (CG), and generates and displays a synthetic image by superimposing all or a part of an image of the real space captured by the imaging device on the CG image. When providing a user with a highly realistic experience by using mixed reality, it is important not only to simply superimpose a CG image on a video of the real space as the background, but also to have an interaction in which the user virtually operates a virtual object drawn by CG. In order to realize such an interaction, it is necessary to display the hand of the user that operates the virtual object (hereinafter referred to as the subject) in front of the virtual object (foreground side). This is because if the subject that should be in front of the virtual object is hidden by the virtual object, the sense of distance and reality from the virtual object will be destroyed, and the sense of realism will be lost.

[0003] In response to this, Patent Document 1 discloses a technique for preventing the image of a subject that should be in the foreground from being hidden by a CG image. In Patent Document 1, the subject area is separated and extracted from the background in a camera image, and the drawing of the CG image in the subject area is prohibited. To separate the subject, color information of the subject is registered in advance, and a process is used to determine whether it is the subject or background according to the color of the camera image. Furthermore, Patent Document 2 discloses a user interface that supports registration of subject regions and deletion of erroneously detected regions (called noise regions) that are not subjects but are erroneously detected as subject regions through simpler operations. In addition to the method of identifying the subject area by extracting a pre-registered color, there is also a method of extracting the foreground subject area by setting a depth value threshold for the depth image obtained by a depth sensor. However, this method requires a depth sensor separate from the camera to capture images simultaneously, which has disadvantages such as the cost of the device and the effort required to calibrate the sensor. Moreover, Patent Document 3 discloses a process for extracting the subject area in a short time by providing a user interface that allows the user to simultaneously specify the "subject area" and the "noise (background) area" of Patent Document 2. Additionally, Patent Document 4 discloses a method of storing color information of a subject and background as a color information table and automatically discriminating an area where the color information of the subject and background overlap. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-107967 A [Patent Document 2] JP 2005-228140 A [Patent Document 3] JP 2015-230695 A [Patent Document 4] JP 2011-018359 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method of Patent Document 1, when the color of the subject and the color of the background are close, or when there is a white or black area (area close to achromatic color) in the background, there may be a background area that is extracted as a subject area (falsely detected area). In addition, there may be a subject area that is not extracted as a subject area (undetected area). In this case, by using the user interface shown in Patent Document 2, the user can eliminate the falsely detected area and the undetected area. However, if color information is registered until there are no undetected areas of the subject, a falsely detected area may occur on the background side, and if an attempt is made to eliminate the falsely detected area on the background side, the undetected area on the subject side may increase. For this reason, it is necessary to repeatedly adjust the color information through trial and error so as to minimize the falsely detected area and the undetected area, and the adjustment takes time.

[0006] In addition, in Patent Document 3, a user interface for simultaneously inputting color information of the subject and background is provided, thereby enabling the subject to be extracted in a shorter time. However, since the main purpose of the method in Patent Document 3 is to increase speed by not requiring the user to make adjustments, the method does not allow for user adjustments when noise occurs.

[0007] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to make it possible to obtain color information that enables a specific subject area to be extracted with high accuracy from a captured image, in a short time and with less of a burden on the user. [Means for solving the problem]

[0008] The image processing device of the present invention includes a color information acquisition means for acquiring color information of a subject and a background in a first image captured by an imaging device, a storage means for storing the acquired color information of the subject and the background, and an expansion means for expanding the color information stored in the storage means in a three-dimensional color space. When the expansion means expands at least one of the color information of the subject and the background in the luminance direction and the chromaticity direction of the three-dimensional color space, the expansion number in the luminance direction and the expansion number in the chromaticity direction are different. It is characterized by the above. Effect of the Invention

[0009] According to the present invention, color information that enables a specific subject area to be extracted with high accuracy from a captured image can be acquired in a short time with less burden on the user. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image processing device according to a first embodiment. [Diagram 2] FIG. 11 is a diagram used to explain an undetected region and an erroneously detected region. [Diagram 3] FIG. 13 is a diagram showing expansion in the luminance direction and a region extracted as a subject. [Figure 4] FIG. 13 is a diagram illustrating an example of the configuration of a color information table. [Diagram 5] 13 is a diagram illustrating an example of a functional configuration of an expansion unit. FIG. [Figure 6] 11 is a flowchart of color information registration and expansion processing by the image processing device. [Figure 7] 11A and 11B are diagrams used to explain a process of expanding a background color. [Figure 8] 13 is a flowchart showing details of an expansion process. [Figure 9] 1A and 1B are diagrams used to explain a three-dimensional color space during expansion processing. [Figure 10] FIG. 1 is a diagram showing the direction of expansion in a three-dimensional color space. [Figure 11] FIG. 13 is a diagram showing conditions for determining attributes after expansion in the expansion process. [Figure 12] FIG. 11 is a diagram illustrating an example of the configuration of an image processing device according to a second embodiment. [Figure 13] 11 is an explanatory diagram of a process for correcting an erroneous input by a user based on depth information. FIG. [Figure 14] FIG. 1 is a diagram illustrating an example of a hardware configuration to which an image processing device is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. The configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings. Note that the same configurations or processes will be described with the same reference symbols. <First embodiment> In this embodiment, a case will be described in which information of a virtual space is superimposed on a real space in real time to provide a user with a mixed reality. For this reason, the image processing device according to this embodiment has a mixed reality generation processing function that generates an image of a virtual space according to the position and orientation of an imaging device by CG, and generates a synthetic image by superimposing the entire area or a part of an image captured by the imaging device on the CG image. In addition, the image processing device according to this embodiment not only superimposes a CG image on a video of the real space as a background, but also has a function of realizing an interaction such as a user virtually operating a virtual object drawn by CG. Note that since the processing technology for generating a synthetic image by superimposing a real image on a CG image of a virtual space is an existing technology, illustrations and detailed explanations of the configuration will be omitted.

[0012] FIG. 1 is a block diagram showing an example of the configuration of an image processing device 100 according to this embodiment. As shown in FIG. 1, an image processing device 100 is connected to an imaging unit 101, an input unit 150, and a display unit 270. The imaging unit 101 is a video camera capable of capturing color images, and captures an image including the subject by capturing an image of the subject. For example, assuming an application example to a video see-through type head mounted display, the imaging unit 101 is a stereo camera capable of capturing stereo images according to the visual fields of the left and right eyes of the user. In the case of a stereo camera, the images to be processed are two images from the left and right cameras, and each image is processed in sequence in a time-division manner. However, this embodiment is not limited to a stereo camera, and can be applied to a monocular camera as well.

[0013] The input unit 150 includes an operation device used when the user inputs operations such as instructions to the image processing device 100, and in this embodiment, a mouse is used as an example of the operation device. Although details will be described later, in this embodiment, the input unit 150 is operated when the user specifies color information of the subject and background, when adjusting the subject extraction process, etc.

[0014] The display unit 270 includes a display device that displays an image viewed by the user, and in this embodiment, the display device is, for example, a head-mounted display. The display device of the display unit 270 may be a large monitor such as a stationary type. When providing mixed reality, the display unit 270 displays a composite image of the result of extracting the subject and synthesizing it with a real image and provides feedback to the user, thereby realizing an interaction in which the user virtually operates a virtual object drawn by CG. In addition, although details will be described later, the display unit 270 displays a user interface image when the user specifies, via the input unit 150, an area that may become noise in the area of ​​the subject or other background areas other than the subject in the captured image of the imaging unit 101. In other words, the user can specify, via the input unit 150, an area that may become noise in the area of ​​the subject or other background areas other than the subject while viewing the display result of the display unit 270.

[0015] Before describing the details of the internal configuration of the image processing device 100 according to this embodiment, points to be considered when realizing mixed reality and interaction will be described with reference to Figs. 2(a) to 2(j) and Figs. 3(a) and 3(b). Here, in order to simplify the description, a case where the captured image input from the imaging unit 101 is a still image will be taken as an example. In the following description, a case where a subject area is separated and extracted from the background in the captured image and a CG image is prohibited from being drawn on the subject will be taken as an example. In the following description, a method of separating the subject area will be taken as an example of a method of registering color information of the subject in advance and determining whether the subject area is a subject area or a background according to the color of the captured image.

[0016] 2(a) is a diagram showing an example of registering the color of a subject appearing in a captured image, and is a diagram showing a schematic diagram of a state in which a hand 510 is taken as an example of a subject in an image and the color of the hand 510 is registered. Note that in video 500 in which a captured image is displayed, in addition to the subject hand 510, a box 520 of a color close to the color of the hand (e.g., yellow) is also shown at the same time. When registering the color of a subject, the user operates a mouse, which is an input device, to move the mouse cursor while viewing the image 500 presented on the user interface screen. In the example of FIG. 2(a), the user recognizes that the region is the hand 510 on the captured image, and drags and moves the mouse cursor from position 530 to position 540 to specify region 550. This allows the luminance and color information of the image included in region 550 specified by the movement of the cursor to be acquired. The luminance and color information is mapped, for example, to a YCbCr space to form a three-dimensional registered color region 560 in FIG. 2(c). As described in the above-mentioned Patent Documents 1 and 2, in order to eliminate the influence of color changes due to differences in lighting conditions, etc., a registered color region 570 obtained by projecting the three-dimensional registered color region 560 onto the CbCr space is used to extract colors from the captured image.

[0017] FIG. 2(b) is a diagram showing the result of extracting an area estimated to be the color of the subject from a captured image using the registered color area 570 in FIG. 2(c). In the example of FIG. 2(b), an image 500 showing the extraction result displays an undetected area 552 that could not be extracted inside a hand 510, and erroneously detected areas 555 and 557 that were erroneously extracted from a background area. Here, the undetected area and the erroneously detected area are noise areas when extracting a subject area, and therefore need to be removed. For this reason, the user performs an operation to remove the noise using a mouse on the displayed image 500.

[0018] 2(d) is a schematic diagram showing the state when specifying the exclusion of noise. First, the user specifies an area 550 by dragging the mouse cursor from position 530 to position 540 in order to exclude the false detection area 557 (the dark area of ​​the box 520) as noise. This causes the brightness and color of the captured image within the specified area 550 to be acquired, and these are registered as information on the non-registered color.

[0019] FIG. 2(e) is a diagram showing the state of the color space (YCbCr space) after the non-registered color is set, and an area 565 in the YCbCr space is set as a non-registered color area. The non-registered color area 565 is then projected onto the CbCr plane (area 575). Here, for areas that overlap with a registered color area 570 that has been set as a registered color in advance, the non-registered color area 575 is assumed to have priority (if there is an overlapping area, the area most recently specified by the user is assumed to have priority). At this time, the registered color area 570 is eroded by the non-registered color area 575, and the registered color considered as the subject is reduced. As a result, as shown in FIG. 2(f), the undetected area 552 in the area of ​​the hand 510 is enlarged compared to the state in FIG. 2(d). However, the erroneous detection area 557 is excluded as expected.

[0020] Next, the user specifies an area 550 by dragging the mouse cursor from position 530 to position 540 shown in Fig. 2(f) in order to exclude a false detection area 555, which is noise that appears in a bright area of ​​the yellow box 520. This causes the brightness and color on the captured image within the specified area 550 to be acquired, and this is registered as information on a non-registered color.

[0021] FIG. 2(g) shows the state of the color space (YCbCr space) after the non-registered color is set. As a result, a newly added non-registered color area 567 is set, and similarly projected onto the CbCr space (area 577). At this time, as described above, in the area where the registered color area 570 and the non-registered color area 577 overlap, the newly set non-registered color area 577 takes precedence, and noise of an undetected area occurs in the bright area portion 553 of the hand 510, as shown in FIG. 2(h). However, the erroneously detected area 555 is excluded as expected.

[0022] Next, in order to make undetected region 552 of hand 510 detectable, the user specifies and registers the colors of undetected regions 552 and 553. Fig. 2(i) is a diagram showing the state of the color space (YCbCr space) of the additionally set registered color area. Compared to the previous state of registered color area 560 in Fig. 2(g), registered color area 560 in Fig. 2(i) is larger. Registered color area 560 is then projected onto CbCr space to become registered color area 570 after projection. If registered color area 570 has an area that overlaps with non-registered color areas 575 and 577, it is overwritten as registered color area 570, and therefore erroneously detected areas 555 and 557 appear again, as shown in Fig. 2(j).

[0023] As described above, while repeatedly setting registered colors and non-registered colors, adjustment is made so that the false detection areas and undetected areas are minimized, and when the false detection areas and undetected areas become equal to or less than a predetermined threshold pixel, they are excluded as noise areas, and the expected subject area can be extracted. However, such settings and processing require a long time and a lot of effort for adjustment. Also, while the above example shows a still image, even more time and effort is required for videos. In other words, it is necessary to obtain multiple sample images from the video, repeat the above process, and adjust so that the false detection areas and undetected areas are minimized in all frames of the video, which requires even more time and effort.

[0024] In addition, in the above-mentioned Patent Document 2, the color is extracted using information in the YCbCr (three-dimensional color space) without projection onto the CbCr plane (two-dimensional color space), and since projection onto the CbCr plane is not performed, there is no need to determine overlapping areas on the projection plane. However, in order to suppress erroneous detection areas and undetected areas in all frames of the video, it is necessary to register in advance all luminance information and color information (YCbCr values) of the subject that may appear in the image. In other words, it is necessary to obtain color information of the subject of more images than when color is extracted using only CbCr values, and this requires more time for user operations.

[0025] For example, when the color of a part of the subject area is registered as YCbCr values ​​as shown in FIG. 2(a) based on the user's instruction for one captured image, the color information for extraction may be insufficient, and an undetected area may remain. This phenomenon will be explained using FIG. 3(a) and FIG. 3(b). FIG. 3(a) is a diagram showing the difference between a registered color area 560 and a correct answer area 595 in a three-dimensional color space when the color of the area shown in FIG. 2(a) is registered as YCbCr values. In this case, the difference between the registered color area 560 and the correct answer area 595 in FIG. 3(a) becomes an undetected area, and an undetected area 552 remains as shown in FIG. 3(b). When detecting from color data in a two-dimensional color space, detection is performed only with color information without referring to luminance information, so that bright and dark areas may be erroneously detected as the color of the subject even though they are not the subject area. When detecting in a three-dimensional color space, luminance information is also used for discrimination, and detection is performed using luminance information obtained from the subject, so there is an advantage in that erroneous detection due to luminance information is reduced. However, because the acquired color information includes luminance in addition to hue, the amount of input data required to minimize undetected and erroneously detected areas increases compared to detection in a two-dimensional color space.

[0026] Therefore, in the image processing device 100 of this embodiment, the registered color region 560 and the unregistered color region 565 are expanded in the luminance and hue directions in three-dimensional color space, thereby reducing the time and effort required for adjustment and achieving subject extraction processing that suppresses the occurrence of erroneous detected and undetected regions. As shown in FIG. 1, the image processing device 100 of this embodiment includes a storage unit 110, a color information registration unit 120, an extraction unit 140, an expansion unit 130, and an output unit 115.

[0027] The storage unit 110 is a temporary storage medium for storing or transmitting to another program data of the subject area on the captured image determined by the image processing device 100, and is composed of, for example, a memory or a hard disk. In this embodiment, the storage unit 110 stores information such as the captured image input from the imaging unit 101, a color information table used for subject extraction, an extracted area image showing the result extracted by the extraction unit 140, an attribute assignment condition table, the number of times of expansion in the luminance direction, and the number of times of expansion in the hue direction. Details of these pieces of information will be described later. In this embodiment, the storage unit 110 is not limited to storing only these pieces of data and information, and can also store other information necessary for processing.

[0028] FIG. 4 is a diagram showing an example of a color information table. In the color information table, data based on luminance information (Y) and color information (Cb, Cr) is stored. For example, 8-bit information from 0 to 255 is stored as an index for the luminance information (Y). Similarly, information from 0 to 255 is stored as an index for the color information (Cb, Cr). In the color information table, YCbCr values ​​are arranged as shown in FIG. 4, and attribute information and expansion information are associated with each YCbCr value. The attribute information is, for example, information of "registered color", "unregistered color", and "none", which can be selected. In addition, in the expansion information, for example, "0" is stored for a color that has not been expanded, and "1" is stored for a color that has been expanded and registered.

[0029] The number of expansion times in the luminance direction and the number of expansion times in the chromaticity direction are information that sets the number of times expansion processing is performed in the luminance direction and the chromaticity direction in the three-dimensional color space. The more the number of expansion times, the larger the area that is expanded. In this embodiment, the number of expansion times in the luminance direction and the number of expansion times in the chromaticity direction are each set to one.

[0030] The color information registration unit 120 is a module for the user to input the area of ​​the subject or background in the captured image via the input unit 150. In the case of this embodiment, the user specifies the area of ​​the subject or background in the captured image by, for example, dragging the mouse of the input unit 150. That is, the color information registration unit 120 has a color information acquisition function for acquiring color information of the area specified by the user from the first image captured by the imaging unit 101 and stored in the storage unit 110, and a color information registration function for registering the color information in a color information table of the storage unit 110. In this embodiment, values ​​in the YCbCr space are used as color information, but the present invention is not limited to using an expression in the YCbCr space, and any expression method of the color space (such as Lab or HSV) can be applied.

[0031] The expansion unit 130 reads out color information of the object and background regions stored in the storage unit 110, and performs processing to expand the color information of the object and background in a three-dimensional color space. In the present embodiment, the expansion unit 130 acquires a color information table from the storage unit 110, refers to data in which the attribute of the YCbCr value is "registered color" or "unregistered color," and performs processing to expand a color region of interest in the three-dimensional color space of YCbCr. By performing processing to expand color information in a three-dimensional color space, it is possible to reduce the undetected region 610 inside the registered color region 560 as shown in FIG. 3(a) and to reduce the gap between the registered color region 560 and the correct answer region 595. Similarly, the expansion unit 130 performs processing to expand the unregistered color region in a three-dimensional color space, estimates the color of the background that has not yet been acquired, and reduces the erroneous detection region. Furthermore, for the YCbCr values ​​that have been determined to be expanded by the expansion process, the expansion unit 130 changes the attribute value in the color information table of FIG. 4 in the storage unit 110 from "none" to "registered color" or "unregistered color." Furthermore, the expansion unit 130 changes the expansion value in the color information table from "0" to "1," and records that the color has been expanded and registered. Details of these expansion processes will be described later.

[0032] The extraction unit 140 extracts a subject area from the second image captured by the imaging unit 101 based on color information acquired from the subject and background areas included in the first image and color information expanded by the expansion unit 130 described above. In this embodiment, the extraction unit 140 refers to the YCbCr values ​​whose attribute is "registered color" in the color information table of the storage unit 110, and detects pixels whose colors match those of each pixel of the captured image recorded in the storage unit 110. The extraction unit 140 then outputs the detected pixel area to the storage unit 110 as an "extracted area image" representing the subject area.

[0033] The output unit 115 outputs the image of the subject region extracted by the extraction unit 140 to the display unit 270. In the present embodiment, the output unit 115 generates a display image by coloring the subject region generated by the extraction unit 140 and combining it with the captured image stored in the storage unit 110, and outputs the display image to the display unit 270. This allows the user to check, via the display on the display unit 270, any overdetection or non-detection that occurred during extraction of the subject region.

[0034] 5 is a block diagram showing a detailed configuration of the expansion section 130. The expansion section 130 has an expansion control section 210, a luminance expansion section 220, and a color expansion section 230. The dilation control unit 210 refers to the YCbCr values ​​of data for which the attributes of the color information table obtained from the storage unit 110 are set as "registered color" and "unregistered color," and maps the data to a three-dimensional color space in memory. The result of the mapping is assumed to be a three-dimensional color space state as shown in FIG. 3(a), for example. FIG. 3(a) shows the state when the user first registers a color by the color information registration unit 120 via the input unit 150.

[0035] If the extraction unit 140 were to extract color in the state shown in Fig. 3(a), the entire hand, which is the area to be extracted, would not be extracted, and some areas would remain as undetected areas, as shown in Fig. 3(b). This is because the registered color area 560 is smaller than the correct answer area 595 shown in Fig. 3(a). Therefore, the expansion control unit 210 executes control to expand the registered color area 560 so as to approach the correct answer area 595.

[0036] First, the expansion control unit 210 controls the luminance expansion unit 220 to perform expansion in the luminance (Y) direction, and further controls the color expansion unit 230 to perform expansion in the color (Cb, Cr) direction. However, in this embodiment, it is not limited to expanding the luminance direction and the color direction independently, and the luminance direction and the color direction may be expanded simultaneously. Note that the correct area 595 shown in FIG. 3(a) is an area set for explaining the process, and is not known information in the actual process.

[0037] The luminance expansion unit 220 refers to the color information mapped to the three-dimensional color space by the expansion control unit 210 and expands the registered color or the non-registered color in the luminance direction. The expansion is necessary, for example, to fill holes in the undetected area 610 shown in FIG. 3(a) by interpolating with the registered color or the non-registered color. Filling these holes contributes to reducing the area of ​​the undetected area 552 shown in FIG. 3(b). When the color (YCbCr value) currently being focused on meets the expansion conditions, the luminance expansion unit 220 updates the "attribute" information and "expansion" information in the color information table of the storage unit 110 corresponding to the YCbCr value of the color being focused on. The details of this process will be described later.

[0038] The color expansion unit 230 expands a registered color or a non-registered color in the color direction by referring to the color information mapped to the three-dimensional color space by the expansion control unit 210. When the color (YCbCr value) of interest matches the expansion condition, the color expansion unit 230 updates the "attribute" information and "expansion" information in the color information table of the storage unit 110, similar to the luminance expansion unit 220. Details of this process will be described later.

[0039] FIG. 6 is a flowchart showing the flow of color information registration and expansion processing in the image processing device 100 of this embodiment. Note that the processing of the flowchart in FIG. 6 is repeated every time the captured image from the imaging unit 101 is updated. In this embodiment, the processing of registering color information is performed before the user experiences mixed reality, and it is not necessary for the user to register colors during the experience. In this way, when the color information of the subject is registered in advance, processing including the processing of steps S110, S140, and S145 of the flowchart in FIG. 6 is performed. On the other hand, when only color extraction is performed when the user experiences mixed reality, the processing is performed by skipping the processing of steps S110, S140, and S145.

[0040] In step S100, the image processing device 100 acquires a captured image from the imaging unit 101, and the captured image is stored in the storage unit 110. Note that the image stored in the storage unit 110 is not limited to the captured image from the imaging unit 101, and a rendered CG image may be input and stored.

[0041] Next, in step S110, the color information registration unit 120 determines whether or not there is an input for image selection from the user via the input unit 150. If there is an input from the user, the color information registration unit 120 proceeds to step S120, whereas if there is no input, the color information registration unit 120 returns to step S100 and processes the captured image of the next frame.

[0042] When proceeding to step S120, the color information registration unit 120 acquires color information contained in the area 550 further specified by the user on the captured image selected in step S110, and registers it in the color information table of the storage unit 110. The area 550 is specified, for example, as shown in FIG. 2(a), by the user operating the mouse cursor of the input unit 150 to specify an area on the captured image. Note that this embodiment is not limited to the user individually registering the areas of the subject and background for the selected captured image, and can also be applied to a case where the color of an area previously fixed on the captured image is acquired as the color of the subject. In addition, the area of ​​the subject may be determined by a discriminator previously trained by machine learning, and color information of the area may be acquired.

[0043] Next, in step S130, the extraction unit 140 extracts the area of ​​the subject in the selected captured image based on the color information table of the storage unit 110. In the extraction process, the area of ​​the subject is extracted by referring to the YCbCr values ​​whose "attributes" in the color information table are set to registered colors. That is, since the registered colors are color information of the subject, the area of ​​the subject is extracted based on the registered colors.

[0044] Next, in step S140, the expansion unit 130 determines color information to be expanded as a registered color based on color information whose attribute information in the color information table is set to "registered color". Details of the process will be described later in the flowchart of FIG. 8. For example, if the registered color area 560 shown in FIG. 3(a) is registered in the color space by the color information registration unit 120, the registered color area 560 expands in the luminance direction and the color tone direction as shown in the area 561 in FIG. 3(e) by the process of step S140. As a result, the color extraction result in step S130 when only color extraction is executed later will be as shown in FIG. 3(b) to FIG. 3(f), and the undetected area will be reduced.

[0045] Next, in step S145, the expansion unit 130 determines color information to be expanded as a non-registered color based on color information whose attribute information in the color information table is set to "non-registered color". Details of the process will be described later with reference to the flowchart in FIG. 8. Here, for example, a case is assumed in which a registered color area 560 shown in FIG. 7(a), an expanded registered color area 561, and a non-registered color area 565 are registered. FIG. 7(a) shows an example in which a hole area 566 exists in the non-registered color area 565, and the hole area 566 is a registered color area. In this state, as shown in FIG. 7(b), a false detection area (area 577) occurs outside the subject area. By the process of this step S145, the non-registered color area 565 is expanded in the luminance direction and the color direction as in the area 599 in FIG. 7(c). As a result, the hole region 566 is filled as a non-registered color region, so that when only color extraction is performed later in step S130, the region 557 that was a false detection region disappears as shown in FIG. 7(d).

[0046] Next, in step S150, the color information registration unit 120 determines whether or not there is an end instruction from the user via the input unit 150, and if there is no end instruction, the process returns to step S100. On the other hand, if there is an end instruction, the image processing device 100 ends the process of the flowchart in FIG. The above explanation has mainly been about the process flow when registering color information of a subject. However, if only color extraction is performed when the user experiences mixed reality, the processes of steps S110, S140, and S145 are skipped as described above.

[0047] In this embodiment, the expansion processes for registered and non-registered colors are performed separately in steps S140 and S145, but this is not limiting and both expansion processes may be performed simultaneously. When performing expansion processes simultaneously, the registered and non-registered colors can be simultaneously mapped to the color information in the three-dimensional space handled by the luminance expansion unit 220 and the color expansion unit 230 described later and processed.

[0048] 8 is a flowchart showing details of the process of the expansion unit 130. The processes of steps S140 and S145 in FIG. In step S300, the expansion control unit 210 registers color information in a three-dimensional color space in memory based on the color information read from the storage unit 110.

[0049] Fig. 9 is a schematic diagram for explaining the expansion process. For the sake of simplicity, Fig. 9 shows a YCr space (two-dimensional color space) when the Cb value is fixed and cut out from the YCbCr space, which is originally three-dimensional. For the sake of simplicity, a specific range of Y values ​​and Cr values ​​is cut out and displayed. The symbols in the blocks in Fig. 9 are determined by a combination of attribute information and expansion information. In this embodiment, each symbol is defined as follows: If the attribute information is "Registered Color" and the expansion information is "0", it is "P". If the attribute information is "Registered Color" and the expansion information is "1", select "Q" If the attribute information is "unregistered color" and the expansion information is "0", then "N" If the attribute information is "unregistered color" and the expansion information is "1", it is "M"

[0050] Next, in step S310, the luminance expansion unit 220 selects as the color of interest from the first item in the color information table shown in Fig. 4, whose YCbCr values ​​are all 0, through to the last item whose YCbCr values ​​are all 254. If the color of interest is a "registered color" or an "unregistered color," the luminance expansion unit 220 skips processing and sets the next color in the color information table as the color of interest. If the attribute information of the color of interest is "none," the luminance expansion unit 220 refers to the attribute information of adjacent colors that have the same hue and a luminance of ±1 when the YCbCr values ​​of the color of interest are mapped into the three-dimensional color space in memory.

[0051] Fig. 10 is a diagram showing the direction of expansion in a three-dimensional color space, and in the color space shown in Fig. 10, for example, when the color of interest is color of interest 700, the luminance expansion unit 220 refers to the attributes of color 710 and color 720. The luminance expansion unit 220 determines the attributes of color of interest 700 based on the attribute assignment condition table of Fig. 11 registered in advance in storage unit 110, and records the attributes in the attribute information of the color information table of storage unit 110.

[0052] Next, since the attention color 700 is a color that has been expanded and registered, the luminance expansion unit 220 replaces the expansion information in the color information table of the storage unit 110 with "1". For example, if the attribute of the color 710 with the next higher luminance is "registered color" and the attribute of the color 720 with the next lower luminance is "none", the attribute value of the attention color 700 becomes "registered color". Figures 9(a) and 9(b) show an example of performing luminance expansion on a registered color first. As a result of expanding the registered color 1110 registered in the three-dimensional color space in the luminance direction first, an area 1130 indicated by symbol Q as shown in Figure 9(b) is registered as a registered color.

[0053] In step S315, the expansion control unit 210 acquires the number of expansions in the luminance direction from the storage unit 110, and determines whether the number of expansions has reached a specified number. If the number of expansions in the luminance direction has reached the specified number, the expansion control unit 210 proceeds to step S315, and if not, proceeds to step S310.

[0054] In step S320, the color expansion unit 230 selects as a color of interest the first item in the color information table shown in Fig. 4, in which all YCbCr values ​​are 0, through to the last item in which all YCbCr values ​​are 254. When the color of interest is a "registered color" or an "unregistered color," if the attribute information of the color of interest is "none," the color expansion unit 230 refers to the attribute information of adjacent colors that have the same luminance and are ±1 in the color Cb and Cr directions when the YCbCr values ​​of the color of interest are mapped into the three-dimensional color space in memory.

[0055] For example, in the color space shown in FIG. 10, when the color of interest is the color of interest 700, the color expansion unit 230 refers to the attributes of the colors 730, 740, 750, and 760. The color expansion unit 230 determines the attribute of the color of interest 700 based on the attribute assignment condition table of FIG. 11 registered in advance in the storage unit 110, and records it in the attribute of the color information table of the storage unit 110. For example, if the attribute of the color 730 is "registered color", the attribute of the color 740 is "none", the attribute of the color 750 is "non-registered color", and the attribute of the color 760 is "none", the attribute value of the color of interest 700 becomes "none". FIG. 9(c) shows an example of expanding the registered color in the color direction. When the expansion process is performed in the color direction in the state of FIG. 9(b), the area 1140 of the symbol Q is registered as a registered color as shown in FIG. 9(c).

[0056] In step S325, the expansion control unit 210 obtains the number of expansions in the color direction from the storage unit 110 and determines whether the number of expansions has reached a specified number. If the number of expansions in the color direction has reached the specified number, the expansion control unit 210 ends the process. If the number of expansions has not reached the specified number, the expansion control unit 210 proceeds to step S320.

[0057] Fig. 9(d) is a schematic diagram of the time when the expansion unit 130 first maps the unregistered color 1150 into the three-dimensional color space when expanding the unregistered color in step S145 in Fig. 6. In this case, expansion is performed in the luminance direction in step S310, but as a result of processing based on the attribute assignment condition table in Fig. 11, expansion is not performed in the displayed area, and no change occurs. This is because the attributes of registered and unregistered colors exist in adjacent blocks.

[0058] On the other hand, in the expansion process in the color direction in step S320, the attribute of a non-registered color is set to block 1160, as shown in Fig. 9(e). Also, block 1170 is not expanded because an adjacent block has a color with the attribute of a registered color. In this embodiment, the above process does not expand the registered color and the non-registered color independently, but controls the expansion so as not to intentionally expand the blocks where overlap occurs. That is, when expanding the color information of the subject in the three-dimensional color space, the expansion control unit 210 of the expansion unit 130 controls the expansion so as not to overlap by referring to the position in the three-dimensional color space of the color information of the background. As a result, according to this embodiment, it is possible to reduce the erroneous detection area 557 and the undetected area 552 of the extracted subject area without bias.

[0059] <Variation 1> As described above, in the first embodiment, the number of times of expansion in the luminance direction and the color direction is set to one, but the number of times of expansion is not limited to one, and the number of times of expansion in the luminance direction and the color direction is not limited to the same number. The number of times of expansion may be set in advance, or may be set to an arbitrary number by the user.

[0060] In Modification 1 of the first embodiment, a case where the expansion count is changed will be described. For example, a case where the expansion count in the color direction is set to two times will be described. Note that the configuration of the image processing device 100 in Modification 1 is the same as the configuration described above, and therefore illustration and description will be omitted. The process for implementing Modification 1 is also the same as the flowchart described above. However, while the processes of steps S315 and S325 in the flowchart shown in FIG. 8 were performed once in the above description, when the expansion count is set to two times as in Modification 1, the processes of steps S310 and S320 are performed twice.

[0061] As a result of the processing in the first modification, the state of expansion is such that the area of ​​the registered color is expanded as shown in the area of ​​the registered color 1180, as shown in Fig. 9(f). However, the block 1190 adjacent to the registered color and the non-registered color is not expanded. Note that if the number of dilations in the luminance direction is greater than the number of dilations in the hue direction, the width of the dilation will be longer in the luminance direction. In particular, in scenes where the subject moves between bright and dark areas, adjusting the dilation in the luminance direction to be more effective can reduce undetected areas. In addition, in a scene where the user moves through rooms with different lighting color temperatures, the number of expansions in the hue direction can be adjusted to be greater than that in the luminance direction, thereby reducing the number of undetected areas.

[0062] <Variation 2> In the above example, the number of expansions is stored in advance in the storage unit 110 and is not changed during processing. The number of expansions is not limited to being a fixed number during processing, and as a second modified example, after the user checks the subject extraction result on the display unit 270 for the current captured image, the user may instruct to change the number of expansions for the same image. To realize the processing of the modified example 2, the user needs to input the dilation count to be stored in the storage unit 110 via the input unit 150, change it via the color information registration unit 120, and execute the same processing.

[0063] <Second embodiment> In the first embodiment, when registering the color of the subject, the color information registration unit 120 acquires input from the user via the input unit 150. However, since the input from the user is not always correct, there is a possibility that a color that is not part of the subject area may be registered. In this case, for example, a background color may be registered as the registered color of the subject, and the erroneous detection area may increase in the extraction result. Furthermore, when the expansion process as described above is performed, the registered color is expanded, which may promote the increase in the erroneous detection area. In view of this, in the second embodiment, an image processing device that automatically prevents an incorrect area from being registered even if the user input is incorrect will be described.

[0064] Fig. 12 is a block diagram showing an example of the configuration of the image processing device 100 according to the second embodiment. In the configuration of Fig. 12, a depth measurement unit 1300 and a depth determination unit 1310 are further added to the configuration shown in Fig. 1.

[0065] The depth measurement unit 1300 is a distance acquisition unit that acquires distance information from the imaging unit 101 to an object or the like. In this embodiment, the depth measurement unit 1300 measures and assigns the distance from the imaging unit 101 to an object or the like as a depth value for each pixel of a captured image. In the second embodiment, for example, it is assumed that the imaging unit 101 is a stereo camera, and the depth value on the captured image is measured by a known stereo matching process. Note that the depth information may be acquired using another device such as a depth sensor.

[0066] The depth determination unit 1310 refers to the depth values ​​on the captured image obtained by the depth measurement unit 1300, and identifies pixel positions of depth values ​​within a predetermined distance range. The depth determination unit 1310 registers a value of 1 for pixels with depth values ​​within a predetermined range on the captured image, and a value of 0 for other pixels, to generate an image of an area at an appropriate distance. The generated image of the area at the appropriate distance is then input to the color information registration unit 120, which automatically excludes areas other than the subject that the user erroneously attempted to input.

[0067] To realize the processing of this embodiment, in step S120 of FIG. 6, only the color information of the area of ​​the subject specified by the user that has a value of 1 in the area of ​​the appropriate distance generated by the depth determination unit 1310 is recorded in the color information table of the storage unit 110.

[0068] For example, if a user mistakenly surrounds both the subject and the background, a registered color area 560 in a three-dimensional color space is generated as shown in Fig. 13(a). A part of the registered color area 560 deviates from the correct answer area 595, and if the subject is extracted using this color information table, erroneous detection areas 555 and 557 are detected as noise as shown in Fig. 13(b). If the distance in the depth direction from the imaging unit 101 at which the depth determination unit 1310 determines that there is a high probability that a subject exists is input as a predetermined distance range, an area image of an appropriate distance such as that shown in Fig. 13(c) can be obtained. In the second embodiment, this information is used to register color information, and the area can be automatically corrected to that shown in Fig. 13(d). Also in the second embodiment, by expanding the color information, an extracted image of the subject with few undetected areas and erroneous detection areas can be obtained.

[0069] Fig. 14 is a diagram showing an example of the hardware configuration of an information processing device to which the image processing device 100 in the first and second embodiments described above can be applied. The information processing device shown in Fig. 14 can be realized by a general computer or the like. The CPU 901 controls the entire device. The RAM 907 is used as a working area when the CPU 901 performs processing while controlling each unit. The CPU 901 also operates as a display control unit that controls the display unit 270. The ROM 902 stores a control program, various application programs, data, and the like. The CPU 901 deploys the control program stored in the ROM 902 to the RAM 907 and executes it, thereby realizing the image processing device 100 of FIG. 1 or FIG. 12. The I / F 903 is used for communication with an external device of the information processing device. For example, the I / F 903 converts a signal of an image captured by the imaging unit 101 into a format that can be processed by the image processing device 100 and acquires it. Note that when the information processing device is connected to an external device by wire, a communication cable is connected to the I / F 903. When the information processing device has a function of wireless communication with an external device, the I / F 903 is provided with an antenna.

[0070] The information processing device may have one or more dedicated hardware devices different from the CPU, and the dedicated hardware devices may execute at least a part of the processing performed by the CPU. Examples of the dedicated hardware devices include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The display unit 270 is composed of, for example, a liquid crystal display or an LED display, and displays a GUI (Graphical User Interface) for the user to operate the information processing device. For example, the GUI may be used when the user specifies the area on the subject or the area on the background, or when the user specifies the number of times of expansion. The mouse 908 and the keyboard 909 are operation devices operated by the user. The operation devices may include a joystick, a touch panel, and the like. The external storage device 906 is a storage device connected to the information processing device. The storage medium drive 905 is a drive device that drives a storage medium such as a memory card when the storage medium is attached to the information processing device. The bus 910 connects the various parts of the information processing device to transmit information. In this embodiment, the external device connected to the information processing device includes the imaging unit 101 as described above as well as other information processing devices, etc. Also, the mouse 908, keyboard 909, etc. may exist as separate devices outside the information processing device.

[0071] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics. [Explanation of symbols]

[0072] 100: image processing device, 101: imaging section, 110: storage section, 115: output section, 120: color information registration section, 130: expansion section, 140: extraction section, 210: expansion control section, 220: luminance expansion section, 230: color expansion section

Claims

1. a color information acquisition means for acquiring color information of a subject and a background in a first image captured by the imaging device; a storage means for storing the acquired color information of the subject and the background; an expansion means for expanding the color information stored in the storage means in a three-dimensional color space; having An image processing device characterized in that, when the expansion means expands at least one of the color information of the subject and the background in the luminance direction and the chromaticity direction of a three-dimensional color space, the number of expansions in the luminance direction and the number of expansions in the chromaticity direction are different.

2. A color information acquisition means for acquiring color information of a subject and a background in a first image captured by an imaging device; a storage means for storing the acquired color information of the subject and the background; an expansion means for expanding the color information stored in the storage means in a three-dimensional color space; having The image processing device is characterized in that, when expanding the color information of the subject in a three-dimensional color space, the expansion is controlled by referring to the position of the color information of the background in the three-dimensional color space so as not to overlap.

3. 3. The image processing device according to claim 1, further comprising an extraction means for extracting an area of ​​the subject from a second image captured by the imaging device based on the color information stored in the memory means and the color information expanded by the expansion means.

4. 4. The image processing apparatus according to claim 3, further comprising output means for outputting the area of ​​the subject extracted by said extraction means.

5. 5. The image processing apparatus according to claim 1, wherein the storage means stores a color information table in which the color information is divided into luminance information and color information.

6. 6. The image processing apparatus according to claim 1, wherein the expansion means expands at least one of the color information of the subject and the color information of the background in a luminance direction in a three-dimensional color space.

7. 7. The image processing apparatus according to claim 1, wherein the expansion means expands at least one of the color information of the subject and the background in a color direction in a three-dimensional color space.

8. 8. The image processing device according to claim 1, wherein the expansion means controls the expansion based on a number of expansions input by a user or a preset number of expansions.

9. The image processing device according to claim 1, characterized in that when the expansion means expands at least one of the color information of the subject and the background in the luminance direction and the chromaticity direction of a three-dimensional color space, the number of expansions in the luminance direction is made greater than the number of expansions in the chromaticity direction.

10. The image processing device according to claim 1, characterized in that when the expansion means expands at least one of the color information of the subject and the background in the luminance direction and the chromaticity direction of a three-dimensional color space, the number of expansions in the chromaticity direction is made greater than the number of expansions in the luminance direction.

11. a distance information acquisition unit that acquires distance information from the imaging device to the subject; 11. The image processing apparatus according to claim 1, wherein the storage means stores the color information when the distance information is within a specified range.

12. a color information acquisition step of acquiring color information of a subject and a background in a first image captured by the imaging device; a storage step of storing the acquired color information of the subject and background; an expansion step of expanding the color information stored in the storage step in a three-dimensional color space; having An image processing method characterized in that, when at least one of the color information of the subject and the background is expanded in the luminance direction and the chromaticity direction of a three-dimensional color space in the expansion step, the number of expansions in the luminance direction and the number of expansions in the chromaticity direction are different.

13. A color information acquisition step of acquiring color information of a subject and a background in a first image captured by an imaging device; a storage step of storing the acquired color information of the subject and background; an expansion step of expanding the color information stored in the storage step in a three-dimensional color space; having An image processing method characterized in that, in the expansion step, when expanding the color information of the subject in a three-dimensional color space, the expansion is controlled by referring to the position in the three-dimensional color space of the color information of the background so that there is no overlap.

14. A program for causing a computer to execute each step of the image processing method according to claim 12 or 13.

Images