Information processing device, information processing method, and program

Abstract

To provide an information processing device that enables a specific area to be extracted from an image with low delay and high accuracy.SOLUTION: An information processing device includes: extraction means for extracting a pixel of a non-registered color that is not a registered color held in holding means from a photographed image; correction means for correcting the pixel of the non-registered color so as to correspond to a value indicating a registered color when the other pixel group includes a specified number or more of pixels of the registered color as a result of referring to the other pixel group near the extracted pixel of the non-registered color; and update means for updating information on the registered color held in the holding means so that a non-registered color of the corrected pixel is included in the registered color.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing technique using images. [Background technology]

[0002] In recent years, work support systems using XR (cross reality) technology, which projects images that combine the virtual world and the real world onto a head-mounted display (HMD), have been developed and are being used in a variety of fields, including manufacturing, logistics, and business negotiations. For example, in the manufacturing field, work and training efficiency is improved by extracting hand regions from actual images of the hands, and then projecting an image of the work operation, which combines the extracted hand image with a CG image of a target device, onto the HMD in real time. Patent Document 1 discloses a technique for extracting specific regions, such as hand regions, from a captured image. The technique calculates an evaluation value representing the size of a skin-colored region, compares the difference between the evaluation value and a reference value indicating the size of the subject, and narrows the color range. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-184906 Summary of the Invention [Problem to be solved by the invention]

[0004] In the case of HMD projection in the aforementioned XR, it is necessary to project an image with low latency that fuses a specific area, such as a hand, extracted from an image taken of the real world with a CG image. However, in the case of the aforementioned technology, the number of pixels is used as an evaluation value that represents the size of the skin color area, so when the scene changes, for example, it becomes necessary to reset the reference image, which is time-consuming to follow the scene change and makes it difficult to achieve low latency.

[0005] Therefore, an object of the present invention is to enable extraction of a specific area from an image with low delay and high accuracy. [Means for solving the problem]

[0006] The information processing device of the present invention is characterized by having an extraction means for extracting pixels of non-registered colors that are not registered colors stored in a storage means from a captured image; a correction means for correcting, when a result of referring to other pixel groups close to the pixels of the non-registered colors extracted by the extraction means shows that the other pixel groups contain more than a specified number of pixels of the registered color, the pixels of the non-registered color so that they correspond to values ​​indicating the registered color; and an update means for updating the information of the registered color stored in the storage means so that the non-registered color of the pixel corrected by the correction means is included in the registered colors. [Effects of the Invention]

[0007] According to the present invention, it is possible to extract a specific area from an image with low delay and high accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device. [Figure 2] FIG. 10 is a diagram illustrating an example of a registered color table. [Figure 3] 10A and 10B are diagrams illustrating an example of noise generated in the hand region extraction process. [Figure 4] 10A and 10B are diagrams illustrating an example of correction processing and updating of a registered color table. [Figure 5] 10A and 10B are diagrams illustrating an example of reduction of noise generated in the hand region extraction process. [Figure 6] 4 is a flowchart of information processing according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing an example of noise reduction by dilation / erosion processing. [Figure 8] 10 is a flowchart of information processing according to the second embodiment. [Figure 9] FIG. 11 is a diagram showing an example of a registered color table according to the third embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, the present invention may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, redundant descriptions of the same or similar configurations and processing steps will be omitted.

[0010] In this embodiment, an example of application to a work support system using XR (cross reality) technology that projects an image that combines the virtual world and the real world onto an HMD (head-mounted display) is described. In this embodiment, as an example, an actual image (real world) captured of the user's hand and a CG image (virtual world) of a target device are precisely combined based on distance information to the target object from an imaging device mounted on the HMD or an imaging device used to create the CG image, and an image of the work operation is projected onto the HMD in real time.

[0011] Here, low-latency projection is required for HMD projection in XR to improve realism. For example, when using color information to extract hand regions, it is expected that hand color information will be registered in advance to achieve low latency. Furthermore, because hand color varies depending on the HMD wearer, preparing a color information table in which individual hand colors are registered in advance and using the color information table corresponding to the individual wearing the HMD can improve the accuracy of detecting hand regions for each individual. However, if the lighting environment when the color information table is generated differs from the lighting environment when the HMD is actually worn and used, parts of the hand in the image may have different colors. As a result, when detecting hand regions using the color information table, erroneous detection of parts with different colors may occur. Furthermore, even if the lighting environment is the same when the color information table is generated and when the HMD is used, parts of jewelry such as rings may not be detected as hand regions, resulting in an unnatural image when combined with a CG image.

[0012] The information processing device of this embodiment has the functional units described below, and by performing the following processing, it is possible to accurately detect specific areas such as hands from an actual captured image and generate a natural-looking image by combining an image of a hand with a CG image with low latency.

[0013] First Embodiment FIG. 1 is a functional block diagram showing an example of the functional configuration of an information processing apparatus according to the first embodiment. The image receiving unit 100 receives image data of the real world captured by an imaging device (not shown) mounted on the HMD. The imaging device is, for example, a camera that captures multiple images that can be used for distance calculation, and the image receiving unit 100 receives the multiple image data. In this embodiment, the imaging device is, for example, a stereo camera, and the image receiving unit 100 receives stereo image data from the stereo camera. In the following description, image data handled in the information processing device will be simply referred to as an image unless otherwise specified. The image received by the image receiving unit 100 is sent to a registered color designation unit 101, a registered color detection unit 103, a distance calculation unit 107, and a synthesis unit 109.

[0014] The registered color designation unit 101 extracts color information of a designated area from an image received by the image receiving unit 100 from the imaging device, and generates a registered color table with the extracted color information as the registered color. The designated area when extracting color information from an image may be, for example, an area of ​​a specific subject whose shape is detected from the image, or an area specified by a user via a user interface (not shown). The registered color table is a table for masking areas other than the registered color from an image captured by the imaging device and extracting only the area of ​​the registered color. In this embodiment, the registered color designation unit 101 designates an area of ​​a hand appearing in an image received by the image receiving unit 100 from the imaging device as the designated area, and generates a registered color table with color information extracted from the designated area as the registered color. In this embodiment, the registered color table is a table for performing hand masking processing, which masks areas of colors different from the registered color of the hand and extracts only the color area of ​​the hand.

[0015] FIG. 2 is a diagram showing an example of the registered color table. The registered color table includes an INDEX column in which an INDEX corresponding to each set of Y, U, and V values ​​is written; a YUV column in which the Y, U, and V values ​​are each written as an 8-bit value; and a Registered / Unregistered column in which a value indicating whether the color is registered or unregistered is written. The Registered / Unregistered column stores a "1" if the color is registered and a "0" if the color is unregistered. The registered color designation unit 101 acquires YUV values ​​as color information for a specified area of ​​an image received by the image receiving unit 100 and writes the YUV values ​​extracted from the specified area as registered colors. For example, the INDEX [9528979], Y=145, U=102, V=147, is color information extracted from a specified area of ​​an image received by the image receiving unit 100. Therefore, the Registered / Unregistered column stores a value of "1" and the color is registered as a registered color. For example, Y=145, U=102, V=148 in INDEX[9528980] is color information that has not been extracted from the specified area, so the value in the registered / unregistered column is "0" and the color is unregistered (i.e., it is registered as an unregistered color). Note that the registered color table in Figure 2 gives an example in which the color space is expressed in YUV, but this is not limited to YUV, and color spaces such as RGB, HSV, and Lab are also acceptable.

[0016] Table holding unit 102 holds the registered color table generated by registered color designation unit 101, and outputs the registered color table to registered color detection unit 103 in response to a request from registered color detection unit 103 to reference the registered color table. The registered color table held in table holding unit 102 is updated by correction unit 105, which will be described later. Table holding unit 102 can also be provided in an external device to the information processing device, and the information processing device can obtain the registered color table via a network.

[0017] The registered color detection unit 103 extracts color information (YUV values) for each pixel of the image received by the image receiving unit 100 from the imaging device and determines whether a pixel of interest in the image is a registered color or an unregistered color by referencing the registered color table in the table storage unit 102. The registered color detection unit 103 then generates a binarization map (hereinafter referred to as the registered color map) in which a value of "1" indicating a registered color or a value of "0" indicating an unregistered color is assigned to each pixel of the image. The registered color map is a binarization map in which a value indicating a registered color or an unregistered color is assigned to each pixel of the image. However, for convenience in the following explanation, the location in the registered color map corresponding to each pixel of the image will also be referred to as a pixel. Furthermore, within the registered color map, a location assigned a value indicating a registered color will be referred to as a registered color pixel, and a location assigned a value indicating an unregistered color will be referred to as an unregistered color pixel. Furthermore, as long as the locations of registered and unregistered colors can be distinguished, the registered color map can use identifiers other than the binary map's "0" and "1."

[0018] The correction target extraction unit 104 extracts pixels (hereinafter referred to as correction target pixels) from the registered color map generated by the registered color detection unit 103 that are to be corrected from non-registered color pixels (i.e., values ​​indicating non-registered colors) to registered color pixels (i.e., values ​​indicating registered colors). In this embodiment, the correction target pixels in the registered color map are pixels (also referred to as noise in this embodiment) that should be registered color pixels (values ​​indicating registered colors) but are non-registered color pixels (values ​​indicating non-registered colors). For example, if there are discrete non-registered pixels in an area that should be contiguous with registered color pixels, such as the hand area, these pixels become candidates for correction target pixels. When a pixel of interest in the registered color map is a non-registered color pixel, the correction target extraction unit 104 extracts the pixel of interest based on the number of registered color pixels in a group of eight neighboring pixels centered on the non-registered color pixel. When the number of registered color pixels in the group of eight neighboring pixels of the non-registered color pixel of the non-registered color pixel of the non-registered color pixel is equal to or exceeds a specified number, the pixel of interest is selected as a correction target pixel. Then, if the difference between the UV value of a pixel on the image corresponding to that correction target candidate and the average value of the UV values ​​of pixels on the image corresponding to a specified number or more of registered color pixels is less than a predetermined threshold, the correction target extraction unit 104 extracts that correction target candidate as a correction target pixel.

[0019] The correction unit 105 corrects the value in the registered color map corresponding to the pixel of interest that has been determined by the correction target extraction unit 104 to be the pixel to be corrected to "1," indicating a registered color, and updates the value in the corresponding registered / unregistered column in the registered color table. That is, the correction unit 105 replaces "0," which indicated the unregistered color of the pixel to be corrected in the registered color map, with "1," indicating a registered color. At the same time, the correction unit 105 updates the value in the registered / unregistered column in the registered color table of the table storage unit 102, which corresponds to the YUV value of the pixel in the image of the pixel to be corrected, to "1," indicating a registered color.

[0020] Map transmission unit 106 transmits the registered color map corrected by correction unit 105 to distance calculation unit 107. At this time, map transmission unit 106 transmits the registered color map as valid area information when distance calculation unit 107 calculates the distance.

[0021] The distance calculation unit 107 calculates the distance from the image capture device for each pixel in the stereo image received by the image receiving unit 100 while referring to the valid area information (registered color map), and generates a distance map in which a distance value is allocated for each pixel. That is, the distance calculation unit 107 generates the distance map by calculating the distance value for each pixel, with the area consisting of each registered color pixel in the registered color map as the valid area for distance calculation. In this way, the distance calculation unit 107 calculates the distance for the valid area, so that the dependency on the background (in this embodiment, the background other than the hand area) during distance calculation can be reduced, and it is expected that the accuracy of distance calculation will be improved and the load will be reduced (speed will be increased). Note that the distance calculation method can be a known method, such as a method of calculating parallax using template matching and converting the parallax into distance.

[0022] The CG generation unit 108 generates CG data of the virtual world. That is, the CG generation unit 108 generates CG data of the virtual world to be synthesized (fused) with the image received by the image receiving unit 100, i.e., the actual image of the real world captured by the imaging device. The CG generation unit 108 then transmits the generated CG data to the synthesis unit 109. The synthesis unit 109 generates a synthetic image by synthesizing the CG data generated by the CG generation unit 108 with the actual image based on the distance map from the imaging device sent from the distance calculation unit 107. For example, the synthesis unit 109 superimposes the area of ​​a hand located approximately 10 cm from the imaging device on the area of ​​a CG object located approximately 20 cm from the imaging device to generate a synthetic image that shows that the hand is closer to the imaging device than the CG object. Conversely, if the area of ​​the hand is 30 cm from the imaging device, the synthesis unit 109 superimposes the area of ​​the CG object on the area of ​​the hand to generate a synthetic image that shows that the hand is farther from the imaging device than the CG object. The projection unit 110 projects the composite image generated by the synthesis unit 109 onto a display device (display) included in the HMD. As a result, an image that precisely combines the hand shown in the actual image (real world) captured of the hand and the CG image (virtual world) of the target device or the like based on distance information is projected on the HMD display in real time.

[0023] FIG. 3 shows an example of an image obtained by hand region extraction processing (hand mask processing) using the information processing device of this embodiment. This image shows an example of noise caused by pixels within the hand region being erroneously determined to be a hand as a non-hand pixel. In the example of FIG. 3, a YUV registered color table is referenced for each pixel in a captured hand image, and registered color pixels are displayed as white and non-registered color pixels are displayed as black. While all pixels within the hand region should be determined to be registered color pixels, differences in lighting conditions, etc., result in multiple pixels, such as around the metacarpal bones, being erroneously determined to be non-registered color pixels, resulting in black pixels, creating noise areas within the hand region that are incorrectly recognized as not being a hand region. If image synthesis is performed in a state where black pixels within the hand region are erroneously determined to be non-registered color pixels during the hand region extraction processing, inconsistencies will occur in the distance map threshold determination process, resulting in an unnatural image being projected onto the HMD. For example, an unnatural synthesized image will be generated in which pixels determined to be non-registered color pixels are associated with pixels representing a CG object located deep within the hand.

[0024] FIG. 4 is a diagram used to explain a specific example of the correction process for the registered color map and the update process for the registered color table in the information processing device of this embodiment. Figure 4(a) shows an example of the initial state of the registered color table. In Figure 4(a), for example, the YUV values ​​Y=145, U=102, V=147 are registered colors, and Y=145, U=102, V=148 are unregistered colors. When applied to the example of noise occurring in the hand area shown in Figure 3, the registered color pixels are correctly extracted as part of the hand area, but the unregistered color pixels in the hand area are not correctly extracted as registered color pixels of the hand, and appear as black noise. FIG. 4(b) is a diagram showing UV values ​​of each pixel of 20 pixels in a 5×4 area of ​​the registered color map, extracted from the registered color table. Figure 4(c) is a diagram showing an example of a binary map consisting of a value of "1" indicating a registered color and a value of "0" indicating an unregistered color, generated by referring to a registered color table for each pixel of the YUV values ​​in the 5x4 region shown in Figure 4(b).

[0025] FIG. 4(d) is a diagram used to explain the correction determination of whether or not to correct a target pixel of interest, which is a non-registered color pixel, in the binarization map illustrated in FIG. 4(c). The correction target extraction unit 104 checks the number of registered color pixels in the group of eight pixels adjacent to the target pixel 404, which is a non-registered color pixel. If the number of registered color pixels in the group of eight pixels surrounding the target pixel 404 is equal to or greater than a specified number, the target pixel 404 is designated as a correction target candidate. In the example of FIG. 4(d), if the specified number is set to 3, the target pixel 404 is designated as a correction target candidate because there are three registered color pixels 405, 406, and 407 in the group of eight pixels surrounding the target pixel 404. Furthermore, the correction target extraction unit 104 compares the difference between the UV value of the pixel on the image corresponding to the target pixel 404, which is the correction target candidate, and the average UV value of each pixel on the image corresponding to the registered color pixels 405, 406, and 407 in the group of eight pixels surrounding the target pixel 404, with a predetermined UV threshold. For example, if the difference between the UV value of the correction candidate and the average UV value of a specified number or more of surrounding registered color pixels is within a predetermined UV threshold, the correction target extraction unit 104 selects the correction candidate as the correction target pixel. The correction unit 105 then corrects the value "0" indicating an unregistered color for the correction target pixel in the registered color map to "1" indicating a registered color, and simultaneously updates the value in the registered / unregistered column in the registered color table corresponding to the YUV value of the correction target pixel to "1" indicating a registered color. In other words, this updating process of the registered color table registers the YUV value of the unregistered color of the correction target pixel so that it is included in the registered colors.

[0026] In the example of FIG. 4(d), the group of eight adjacent pixels surrounding the target pixel 404, which is a non-registered color pixel, includes registered color pixels 405, 406, and 407. Among the UV values ​​of each pixel in FIG. 4(b), the UV values ​​corresponding to the target pixel 404 in FIG. 4(d) are U=102 and V=148. The average values ​​of the UV values ​​corresponding to the adjacent registered color pixels 405, 406, and 407 are U=(102+102+103+102+103) / 5=102 and V=(152+153+153+151+152) / 5=152. If the predetermined UV threshold is ±10, the difference between the UV value of the target pixel 404 and the average value of the UV values ​​of the adjacent registered color pixels 405, 406, and 407 is within the UV threshold, and therefore the target pixel 404 is the pixel to be corrected. Therefore, the correction unit 105 corrects the value "0" indicating an unregistered color for the pixel of interest 404 to the value "1" indicating a registered color, and at the same time updates the value in the registered / unregistered column of the registered color table to the value "1" indicating a registered color.

[0027] FIG. 4(e) shows the binarized map after the pixel of interest 404, which was a non-registered color pixel in FIG. 4(d), has been updated to "1", which is a value indicating a registered color. 4(f) shows an example of the registered color table after it has been updated by the correction unit 105. The YUV value (Y=145, U=102, V=148) written in the YUV column 410 corresponding to the pixel of interest 404 has been updated to "1" indicating a registered color in the registered color / registered color column 411. In other words, the YUV value (Y=145, U=102, V=148) that was the unregistered color of the pixel to be corrected has been added to the registered colors and registered.

[0028] FIG. 5 is a diagram used to explain an example in which noise generated during the hand region extraction process is reduced by the above-described correction. FIG. 5(a) shows an example of a binarized map generated by performing hand region extraction processing with reference to a registered color table before update, generated based on the image of the initial frame. FIG. 5(b) shows an example of a binarized map generated by performing hand region extraction processing with reference to a registered color table after the aforementioned correction of the registered color map and update of the registered color table. Compared to the binarized map shown in FIG. 5(a), the binarized map shown in FIG. 5(b) has less noise within the hand region (black areas determined to be non-registered colors). FIG. 5(c) shows an example of a binarized map generated by performing hand region extraction processing with reference to the updated registered color table after further repeated correction of the registered color map and update of the registered color table from the example shown in FIG. 5(b). The example shown in FIG. 5(c) shows a result of the hand region extraction processing in which the noise within the hand region is even less than that shown in FIG. 5(b).

[0029] 5(d) to 5(f) are explanatory diagrams illustrating an example of a change (improvement) in a binarization map resulting from a correction process for a registered color map. FIG. 5(d) shows a portion of a binarization map (a binarization map for 20 pixels in a 5 × 4 area) based on a registered color table before update, generated from an image of an initial frame. FIG. 5(e) shows an example of a portion of a binarization map based on a registered color table after some updating, and FIG. 5(f) shows an example of a portion of a binarization map based on a registered color table after updating is complete. As can be seen, even if noise due to non-registered color pixels was present when referencing the registered color table before updating, repeated updates to the registered color table reduce the noise, ultimately transforming the non-registered color pixels into registered color pixels. In this embodiment, an example was given in which target pixels are sequentially set and the registered color map is updated within a range of one pixel per target pixel. However, the range during the update is not limited to one pixel. For example, if latency is acceptable, the range can be expanded to speed up the convergence of the update of the registered color table.

[0030] FIG. 6 is a flowchart showing the flow of information processing according to the first embodiment, which is executed by the information processing device. First, in step S600, the information processing device sets various parameters necessary for correcting the registered color map and determining whether to update the registered color table, and then proceeds to step S601. The parameters set in step S600 include a designated number used in the determination made in subsequent step S605, a predetermined threshold value used in the determination made in step S607, and the like.

[0031] In step S601, the registered color detection unit 103 reads the registered color table stored in the table storage unit 102. At this point, the table storage unit 102 stores the initial registered color table created by the registered color designation unit 101 from the image of the initial frame, as described above in FIG. 4(a). In other words, the flowchart in FIG. 6 explains the flow of information processing when the initial registered color table is successively updated. Next, in step S602, the image receiving unit 100 sends the image received from the imaging device to the registered color detecting unit 103, the distance calculating unit 107, and the combining unit 109 as an input image.

[0032] Next, in step S603, the registered color detection unit 103 generates a registered color map based on the input image from the image receiving unit 100 and the registered color table in the table holding unit 102, as described above. Next, in step S604, the correction target extraction unit 104 determines whether the pixel of interest is a non-registered color pixel in the registered color map. If the correction target extraction unit 104 determines that the pixel of interest is not a non-registered color pixel, the information processing device proceeds to step S608. On the other hand, if the correction target extraction unit 104 determines in step S604 that the pixel of interest is a non-registered color pixel, the correction target extraction unit 104 proceeds to step S605.

[0033] In step S605, the correction target extraction unit 104 determines whether the eight neighboring pixels surrounding the target pixel, which is a non-registered color pixel, contain a specified number of registered color pixels or more. If the correction target extraction unit 104 determines that the eight neighboring pixels do not contain the specified number of registered color pixels or more, the processing of the information processing device proceeds to step S608. On the other hand, if the correction target extraction unit 104 determines that the eight neighboring pixels contain the specified number of registered color pixels or more, the processing proceeds to step S606.

[0034] In step S606, the correction target extraction unit 104 calculates the average value of the UV values ​​of each pixel on the image that corresponds to a specified number or more of registered color pixels adjacent to the pixel of interest. Next, in step S607, the correction target extraction unit 104 determines whether the difference between the UV value of the pixel in the image corresponding to the pixel of interest and the average UV value calculated for each registered color pixel in step S606 is within a predetermined threshold. If the correction target extraction unit 104 determines that the difference in UV value is within the predetermined threshold, the process proceeds to step S609. On the other hand, if the correction target extraction unit 104 determines that the difference in UV value is not within the predetermined threshold, the process of the information processing device proceeds to step S608.

[0035] In step S608, correction unit 105 outputs the registered color map sent from correction target extraction unit 104 as is to map transmission unit 106, and map transmission unit 106 outputs the registered color map to distance calculation unit 107. After step S608, the processing of the information processing device proceeds to step S611.

[0036] On the other hand, if the process proceeds to step S609, the correction unit 105 corrects the value of the pixel to be corrected in the registered color map to "1", a value indicating a registered color, and outputs the corrected registered color map to the map transmission unit 106. The corrected registered color map is then output from the map transmission unit 106 to the distance calculation unit 107. Furthermore, in the process of step S610, correction unit 105 updates the value of the registered / unregistered column corresponding to the YUV value of the pixel on the image of the pixel to be corrected in the registered color table held in table holding unit 102 to the registered color "1". As a result, the registered color table is updated so that the YUV value of the pixel on the image of the pixel to be corrected (i.e., the unregistered color) is included in the registered colors. Thereafter, the process of the information processing device proceeds to step S611.

[0037] In step S611, map transmission unit 106 transmits the registered color map to distance calculation unit 107 as effective area information for distance calculation, and distance calculation unit 107 calculates the distance for each pixel from the stereo image based on the effective area information to generate a distance map. Also in step S611, synthesis unit 109 generates a composite image by synthesizing the CG data generated by CG generation unit 108 with the actual image based on the distance map transmitted from distance calculation unit 107, and transmits the composite image to projection unit 110. Projection unit 110 projects the composite image on the display of the HMD.

[0038] Next, in the process of step S612, the information processing device determines whether or not the above-mentioned process has been performed on all pixels of the input image received by the image receiving unit 100. If not, the process returns to step S603, and the above-mentioned process is performed on unprocessed pixels. If it is determined that the process has been performed on all pixels, the information processing device proceeds to step S613.

[0039] In step S613, the information processing device determines whether the lighting conditions have changed. The determination of whether the lighting conditions have changed may be performed using various known methods, such as determining the lighting conditions based on the output of an environmental sensor or determining the brightness, hue, or color temperature of a captured image. If the information processing device determines that the lighting conditions have changed, the information processing device returns to step S601 and resets the registered color table to reflect changes in the registered color map caused by the change in lighting conditions. That is, the registered color designation unit 101 generates a new registered color table based on images captured by the image capture device after the change in lighting conditions. On the other hand, if the information processing device determines that the lighting conditions have not changed, the information processing device terminates the processing of the flowchart in FIG. 6.

[0040] In addition to changes in lighting conditions, the information processing device can also reset the registered color table when the hand region, which is the subject of region extraction, changes significantly, for example, when the hand moves back and forth in the depth direction relative to the imaging device. For example, the information processing device determines that the hand region has changed significantly when the difference in size between the hand region before and after the change is equal to or greater than a predetermined threshold, and resets the registered color table. In this case, the registered color designation unit 101 generates a new registered color table based on an image of the hand region after the significant change. Alternatively, the information processing device may count the number of pixels of non-registered colors and reset the registered color table when the count value changes significantly. For example, the information processing device determines that the count value of non-registered colors has changed significantly when the count value of the number of pixels of non-registered colors in the hand region changes and the difference between the count values ​​before and after the change is equal to or greater than a predetermined threshold, and resets the registered color table. In this case, the registered color designation unit 101 generates a new registered color table based on an image of the hand region after the significant change.

[0041] As described above, the information processing device of the first embodiment generates a binarization map (registered color map) for extracting a registered color region using a registered color table. The information processing device then corrects, within the binarization map, pixels to be corrected (non-registered color pixels) that are noise, to registered color pixels according to the conditions of neighboring pixels. At the same time, the information processing device updates the registered color table. By using the updated registered color table, the information processing device can accurately extract a hand region. Furthermore, even if lighting conditions, etc., change, the information processing device updates the registered color table, enabling region extraction that tracks the lighting conditions, etc. Furthermore, if updating of the registered color table stops, the information processing device can temporarily suspend table reference to reduce power consumption, for example, and resume reference to the registered color table when lighting conditions, etc. change. Thus, according to the information processing device of the first embodiment, the registered color table is updated according to the wearing environment of the HMD. This makes it possible to correct missing registered colors in the hand region, for example, with low latency, thereby enabling HMD projection with less discomfort. Furthermore, according to this embodiment, by storing color information of registered colors in a table format, it is possible to easily correct color loss other than the color of the hand, for example.

[0042] In this embodiment, an example has been given in which the registered color table for extracting registered color regions is updated to enable low-latency processing, but the method is not necessarily limited to updating the registered color table, for example, if a processing delay is acceptable. If a processing delay is acceptable, the information processing device can also use a method such as updating a mask image required for region extraction corresponding to the registered color table using the results of shape recognition, contour extraction, etc.

[0043] <Second embodiment> In the first embodiment, the registered color table is referenced, and if the difference between the UV value of a non-registered color pixel and the average UV value of nearby registered color pixels is within a threshold, the non-registered color pixel is corrected to a registered color pixel and the registered color table is updated to include the non-registered color in the registered colors. However, there are cases where the difference between the UV value of a non-registered color pixel and the average UV value of nearby registered pixels does not fall within the threshold. For example, if an accessory such as a ring of a different color from the hand is worn on the hand, the accessory may lose color during the hand region extraction process.

[0044] Fig. 7(a) shows an example of a binarized map (registered color map) generated by the registered color detection unit 103 from an image of a hand with a ring on the ring finger, by referencing a registered color table for hand colors. That is, Fig. 7(a) is a binarized map generated from an image of a hand with a ring on it, by referencing a registered color table (a registered color table in which the ring color is not registered) generated based on an image of the hand without a ring. Because the registered color table in which the ring color is not registered was referenced, the ring on the ring finger in the binarized map in Fig. 7(a) is determined to be a non-registered color, resulting in a colorless black horizontal line.

[0045] Figure 7(c) shows a portion of the registered color table for hand color only, assuming that the YUV values ​​Y=171-175, U=128, and V=128 represent the color information for the ring. In other words, in the registered color table, all values ​​in the registered / unregistered columns corresponding to the YUV values ​​Y=171-175, U=128, and V=128, which are the color information for the ring, are "0." If the color of such a ring is referenced to the registered color table for unregistered colors, the difference between the UV value of the target pixel of the unregistered color pixel and the average UV value of the specified number of nearby registered color pixels surrounding it will not fall within the threshold. As a result, even if the process of the flowchart in Figure 6 described above is repeated, the unregistered color pixels in the registered color map will not be corrected to registered color pixels, and the registered color table, which uses unregistered colors as registered colors, will not be updated sufficiently. Therefore, the phenomenon of color loss and other issues with the ornamental parts during hand region extraction will continue.

[0046] Therefore, the information processing device of the second embodiment repeats expansion and contraction processes a predetermined number of times on the registered color map (binarized map) that is the detection result by the registered color detection unit 103. The information processing device of the second embodiment then updates the registered color table based on the color registered map after the repeated expansion and contraction processes, thereby making it possible to prevent the occurrence of poor area extraction where parts such as ornaments are missing color. In this embodiment, an example will be described in which a correction target area 701 is set for the registered color map obtained from the registered color table generated when the HMD is started, and the expansion and contraction processes are repeated a predetermined number of times on the correction target area 701. Note that the expansion and contraction processes can be performed using, for example, known morphology processing and closing processing, and therefore detailed description thereof will be omitted.

[0047] In the second embodiment, the processing after updating the registered color table by performing expansion / contraction processing of the registered color map when the HMD is started is the same as that in the first embodiment described above. Therefore, the functional block configuration and registered color table of the information processing device according to the second embodiment are also generally the same as those of the first embodiment, and illustrations and descriptions thereof will be omitted. In the case of the second embodiment, the processing performed by the correction target extraction unit 104 and the correction unit 105 when the HMD is started is different from that of the first embodiment. The following description will focus on the differences from the first embodiment.

[0048] 7(b) shows an example of a binarized map (registered color map) after the correction target extraction unit 104 sets a correction target region 701 as shown in FIG. 7(a) for the registered color map, and the correction unit 105 repeatedly performs expansion and contraction processing on the correction target region 701. Note that it is also possible to simply perform only expansion processing on the registered color map, but in that case, a registered color map is generated in which the correction target region 701 expands toward the periphery. For this reason, the correction unit 105 suppresses the expansion toward the periphery by performing contraction processing on the correction target region 701 that has been expanded.

[0049] In the second embodiment, the correction unit 105 repeats the above-described dilation / erosion process a predetermined number of times to generate a registered color map in which pixels that were non-registered color pixels before the dilation / erosion process are replaced with registered color pixels. Furthermore, the correction unit 105 performs threshold comparison on the pixel values ​​of the non-registered color pixels in the image that have been replaced with registered color pixels in the registered color map, thereby determining whether to update the registered color table to include the color information of the pixels in the registered colors. As an example, the correction unit 105 determines a threshold range based on the pixel values ​​of the background region in the image that is not the hand region, and compares the pixel values ​​of the non-registered color pixels in the image that have been replaced with registered color pixels in the registered color map with the threshold range. Then, if the comparison with the threshold range shows that the pixel values ​​of the non-registered color pixels in the image that have been replaced with registered color pixels in the registered color map fall within the threshold range, the correction unit 105 does not update the registered color table to include color information based on the pixel values ​​in the registered colors. This prevents the registered color table from being updated with pixel values ​​of the background region as registered colors. On the other hand, if the comparison with the threshold range shows that the pixel value of the non-registered color pixel in the image that has been replaced with a registered color pixel on the registered color map does not fall within the threshold range, the correction unit 105 updates the registered color table to include color information based on that pixel value in the registered color. That is, the correction unit 105 updates the value of the registered / unregistered column in the registered color table for the color information of the pixel in the image of the non-registered color pixel that has been replaced with a registered color pixel on the registered color map to "1," which is the value of the registered color. As a result, the color information of the non-registered color pixel before the dilation / erosion processing is added to the registered color.

[0050] Figure 7(d) shows a portion of the registered color table corresponding to Figure 7(c), which has been updated based on the registered color map corrected by repeated expansion and contraction processes. As shown in Figure 7(d), the values ​​in the registered / unregistered column corresponding to the YUV values ​​Y=171-175, U=128, and V=128, which are the color information of the ring, are all set to "1." In other words, the registered color table has been updated so that the YUV values, which are the color information of the ring, are included in the registered colors.

[0051] According to the information processing device of the second embodiment, the hand area is extracted by referring to the registered color table that has been updated to include the colors of accessories that are different from the color of the hand as registered colors, as described above, and the accessory part is also extracted as part of the hand area.

[0052] FIG. 8 is a flowchart showing the flow of information processing according to the second embodiment, which is executed by the information processing device. First, in the process of step 800, the registered color detection unit 103 reads the registered color table stored in the table storage unit 102. In the case of the flowchart of Fig. 8, as in the example of the flowchart of Fig. 6, it is assumed that at this point in time, the table storage unit 102 stores a registered color table created from the image of the initial frame at the time of HMD startup, as described above in Fig. 4(a).

[0053] Next, in step S801, the correction target extraction unit 104 of the second embodiment sets a correction target region for updating the registered color table. As an example, the correction target extraction unit 104 sets the correction target region based on the region of the registered color in the registered color map. Note that the correction target region may be set in accordance with a user's specification via a user interface (not shown), for example.

[0054] Furthermore, in the process of step S802, the correction target extraction unit 104 sets a predetermined number of repetitions of the dilation / erosion process for the correction target region of the registered color map, which is performed by the subsequent correction unit 105. The number of repetitions is assumed to be a preset number, but may also be specified by the user via, for example, a user interface (not shown).

[0055] Next, in step S803, the image receiving unit 100 sends the input image received from the imaging device to the registered color detecting unit 103, the distance calculating unit 107, and the combining unit 109. Next, in step S804, the registered color detection unit 103 generates a registered color map based on the image input from the image receiving unit 100 and the registered color table read from the table holding unit 102, as in step S603 described above.

[0056] Next, in step S805, the correction unit 105 performs expansion processing on the registered color map. In this embodiment, the correction unit 105 performs expansion processing on the correction target region 701, but it may also perform expansion processing on the entire registered color map. In step S805, the correction unit 105 determines whether the number of times the expansion processing has been performed has reached a predetermined number, and if the number is less than the predetermined number, the process returns to step S805 and the expansion processing is repeated. Then, if the number of times the expansion processing has been repeated has reached the predetermined number, the process of the correction unit 105 proceeds to step S807.

[0057] In step S807, the correction unit 105 performs an erosion process on the correction target area of ​​the registered color map after the expansion process has been performed a predetermined number of times. At this time, in step S808, the correction unit 105 determines whether the erosion process has been performed a predetermined number of times, and if not, returns to step S808 to repeat the erosion process. Then, when the erosion process has been repeated a predetermined number of times, the correction unit 105 proceeds to step S809.

[0058] In step S809, the correction unit 105 performs the process of step S810 by referring to the registered color map after the above-described expansion / contraction process. In step S810, the correction unit 105 determines, for each pixel in the registered color map, whether a pixel that was a non-registered color pixel before the dilation / erosion processing has been replaced with a registered color pixel after the dilation / erosion processing. For example, if the correction unit 105 determines that the pixel has been replaced from a non-registered color pixel to a registered color pixel by the dilation / erosion processing and is not included in the threshold range based on the pixel value of the background region, the process proceeds to step S811. On the other hand, the correction unit 105 proceeds to step S812 for a pixel other than a pixel that has changed from a non-registered color pixel to a registered color pixel by the dilation / erosion processing, or a pixel that has been replaced from a non-registered color pixel to a registered color pixel by the dilation / erosion processing but is included in the threshold range.

[0059] In step S811, the correction unit 105 updates the registered color table so that the YUV values ​​in the image of the non-registered color pixels that have been replaced with registered color pixels in the registered color map by the dilation / erosion processing are added as registered colors. As a result, the registered color table is updated so that the YUV values, which are the color information of the pixels that were non-registered color pixels in the registered color map before the dilation / erosion processing, are added as new registered colors. After step S811, the processing of the information processing device proceeds to step S812.

[0060] In step S812, the information processing device determines whether steps S810 to S811 have been performed on all pixels in the registered color map after the dilation / erosion processing. If not, the information processing device returns to step S810 and performs the above-described processing on unprocessed pixels. If it determines that the processing has been performed on all pixels, the information processing device proceeds to step S813. In step S813, the information processing device determines that the initial update process of the registered color table at the time of HMD startup has been completed, and transitions to normal mode, terminating the process of the flowchart in Fig. 8. In normal mode, for example, the process described in the first embodiment is performed, and a composite image obtained by combining CG data and a real image is projected onto the HMD. Note that normal mode may be a mode in which the update process of the registered color table described in the first embodiment is not performed.

[0061] As explained above, in the second embodiment, by repeatedly performing expansion and contraction processing on the registered color map generated when the HMD is started and updating the registered color table, it becomes possible to extract the hand area with noise corrected even when ornaments or the like of a color different from the color of the hand are attached.

[0062] <Third embodiment> In the first embodiment, an example was given in which the registered color table in which the YUV values ​​of the registered colors shown in Figure 2 are written is referenced, but if only a specific registered color corresponding to the color of the hand is targeted, creating a registered color table that corresponds to all YUV spaces would be a waste of memory. This is also true for the second embodiment.

[0063] Therefore, in the third embodiment, an example of saving memory by reducing the registered color table will be described. Note that the functional block configuration and flowchart processing of the information processing device according to the third embodiment are generally similar to those of the first embodiment, and therefore illustrations and descriptions thereof will be omitted. However, in the third embodiment, the registered color table generated by the registered color designation unit 101 is different from that of the first embodiment. The following description will focus on the differences from the first embodiment, but the third embodiment can also be applied to the second embodiment.

[0064] Fig. 9 shows an example of a registered color table according to the third embodiment. As with the example shown in Fig. 2, the registered color table includes an INDEX column, a YUV column in which Y, U, and V values ​​are entered, and a registered / unregistered column in which a value indicating whether the color is registered or unregistered is entered. However, in the case of the registered color table shown in Fig. 9, the number of Y, U, and V values ​​has been reduced compared to the example shown in Fig. 2.

[0065] FIG. 9(a) shows an example of a 256×64×64 registered color table in which offsets are applied to UV values ​​in the YUV column. Assuming that UV values ​​are 8-bit, the expressible range is 0 to 255, but in the case of FIG. 9(a), only values ​​from 0 to 63 are registered. In other words, if the range of UV values ​​to be treated as registered colors is narrowed in advance, it is possible to conserve memory by registering them with offsets as in the example of FIG. 9(a). In the case of FIG. 9(a), the U offset is 80 and the V offset is 120, so U=80 to 143 and V=120 to 183 are registered in the registered color table. By notifying the registered color detection unit 103 of this offset information, the registered color detection unit 103 can take the offset into account when detecting registered colors and determine the desired YUV value as a registered color.

[0066] Figure 9(b) shows an example of a 128x64x64 registered color table in which UV values ​​are offset and Y values ​​are thinned out. Because Y values ​​are 8 bits, the range that can be expressed is 0 to 255, but in the case of Figure 9(b), only values ​​from 0 to 127 are registered. For example, if the accuracy of the Y value is not important, it is possible to save memory by thinning and registering the Y value as in the example of Figure 9(b). Figure 9(b) shows an example in which the Y value is thinned out to 1 / 2 and registered. Therefore, for example, if Y=72 is a registered color, the 8-bit Y values ​​Y=144 and Y=145 will also be treated as registered colors.

[0067] As described above, in the third embodiment, in cases where only some specific colors are targeted or where the resolution of YUV values ​​is not important, thinning processing with an offset is performed, thereby making it possible to save memory required to hold the registered color table.

[0068] <Hardware configuration of information processing device> FIG. 10 is a diagram showing an example of the hardware configuration of the information processing apparatus of this embodiment. The CPU 1000 is a central processing unit (Central Processing Unit) that performs calculations and logical decisions for various processes. The ROM (Read-Only Memory) 1010 stores a control program. The RAM (Random Access Memory) 1020 is used as a temporary storage area such as the CPU 1000's main memory and work area. The mass storage device 1030 is a recording device for storing the information processing program according to this embodiment and various data used for generating CG data, etc. An external storage device may also be used to perform a similar function to the mass storage device 1030. The external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for accessing the medium. Known examples of such media include a flexible disk (FD), CD-ROM, DVD, USB memory, MO, and flash memory. The external storage device may also be a server device connected via a network.

[0069] The input unit 1040 is configured with a keyboard, a touch panel, etc., and accepts input from the user. The display unit 1050 is configured with a liquid crystal display, etc., and can display various data, information processing results, etc. to the user. The present device can also communicate with an HMD or other devices via a communication unit 1060. The present device may receive user instructions from other devices via the communication unit 1060, or may output processing results to other devices. The information processing device of this embodiment can be realized by a general-purpose personal computer, tablet terminal, smartphone, etc., having the above-mentioned configuration.

[0070] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions. The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0071] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an extracting means for extracting pixels of a non-registered color that is not a registered color stored in the storing means from the captured image; a correction means for correcting the pixels of the non-registered color so that they correspond to values ​​indicating the registered color when, as a result of referring to other pixel groups adjacent to the pixels of the non-registered color extracted by the extraction means, the other pixel groups contain a specified number or more of pixels of the registered color; an updating means for updating the information on the registered colors stored in the storing means so that the non-registered color of the pixel corrected by the correcting means is included in the registered colors; An information processing device comprising: (Configuration 2) The information processing device described in configuration 1 is characterized in that the correction means corrects the non-registered color pixel so that it corresponds to a value indicating the registered color when the difference between the UV value of the non-registered color pixel extracted by the extraction means and the UV average value of each pixel of the specified number or more of the adjacent registered colors is within a predetermined threshold. (Configuration 3) the correction means has a processing means for performing expansion processing and contraction processing on the area in which the registered color is detected, The information processing device described in configuration 1 or 2, characterized in that the update means updates the information on the registered colors so that the non-registered colors of pixels in which the non-registered color pixels have been replaced with registered color pixels by the expansion processing and contraction processing are included in the registered colors. (Configuration 4) The information processing device described in configuration 3 is characterized in that the update means determines whether to update the information of the registered colors to include the non-registered colors in the registered colors based on a comparison between the color information of the pixels in which the non-registered color pixels have been replaced with registered color pixels by the expansion process and contraction process and a predetermined threshold range. (Configuration 5) the predetermined threshold range is a range based on color information of a background area relative to an area in which the registered color is detected from the captured image, The information processing device described in configuration 4 is characterized in that the update means updates the information of the registered colors so as to include the non-registered colors in the registered colors when the color information of the pixels in which the non-registered colors have been replaced with pixels of the registered colors by the expansion processing and contraction processing does not fall within the predetermined threshold range. (Configuration 6) The information processing device described in any one of configurations 3 to 5, characterized in that the correction means repeats the expansion process a predetermined number of times on a correction target area set for the area in which the registered color is detected, and then repeats the contraction process a predetermined number of times. (Configuration 7) 7. The information processing apparatus according to any one of configurations 1 to 6, wherein the storage means stores information on the registered colors only for a specific color space for which an offset of the color space is specified. (Configuration 8) a designation means for designating a registered color based on the captured image, 8. The information processing apparatus according to any one of configurations 1 to 7, wherein the storage means stores information about the registered color designated by the designation means. (Configuration 9) a determining means for determining whether the lighting conditions have been changed; When the determining means determines that the lighting conditions have been changed, the designation means designates the registered color based on the image captured under the changed lighting conditions; 9. The information processing apparatus according to configuration 8, wherein the storage means stores information about the registered color designated by the designation means after the lighting conditions are changed. (Configuration 10) a determining means for determining whether the size of the area of ​​the specific subject has changed; If the determining means determines that the size of the area of ​​the subject has changed, the designation means designates the registered color based on the image of the subject captured after the size has been changed; 9. The information processing apparatus according to configuration 8, wherein the storage means stores information about the registered color designated by the designation means after the size of the area of ​​the subject has changed. (Configuration 11) a determination means for determining whether the count value of pixels of a non-registered color has changed; If the determination means determines that the count value of the pixel of the non-registered color has changed, the designation means designates the registered color based on the image captured after the count value has changed; 9. The information processing apparatus according to configuration 8, wherein the storage means stores information about the registered color designated by the designation means after the count value has changed. (Configuration 12) a generating means for generating a binarized map that expresses each pixel of the captured image as a binary color of either a registered color or a non-registered color based on the information of the registered color; the extraction means extracts pixels having values ​​representing the unregistered color from a region of values ​​representing the registered color in the binarization map generated by the generation means; 12. The information processing device according to any one of configurations 1 to 11, wherein the correction means corrects the value representing the unregistered color extracted from the binarization map by the extraction means to a value representing the registered color. (Configuration 13) a distance calculation means for calculating a distance from an imaging device that captured the image for each pixel of the image by using the binarized map corrected by the correction means as an effective area for distance calculation; a synthesis means for synthesizing the captured image and a CG image based on the distance for each pixel calculated by the distance calculation means; and 13. The information processing apparatus according to configuration 12, wherein the image synthesized by the synthesis means is displayed on a predetermined display device. (Method 1) an extraction step of extracting pixels of non-registered colors that are not registered colors stored in the storage means from the captured image; a correction step of referring to another pixel group adjacent to the pixel of the non-registered color extracted in the extraction step, and correcting the pixel of the non-registered color so that the value indicating the registered color corresponds to the pixel of the non-registered color when the other pixel group contains a specified number of pixels of the registered color or more; an updating step of updating the information on the registered colors stored in the storage means so that the non-registered color of the pixel corrected in the correction step is included in the registered colors; An information processing method comprising: (Program 1) 14. A program for causing a computer to function as the information processing device according to any one of configurations 1 to 13. [Explanation of symbols]

[0072] 200: Learning device, 220: Data input unit, 230: Importance determination unit, 240: Registered color detection unit, 241: Detector, 242: Parameter storage unit, 250: Loss calculation unit, 260: Weight update unit, 270: Output unit

Claims

1. an extracting means for extracting pixels of a non-registered color that is not a registered color stored in the storing means from the captured image; a correction means for correcting the pixels of the non-registered color so that they correspond to values ​​indicating the registered color when, as a result of referring to other pixel groups adjacent to the pixels of the non-registered color extracted by the extraction means, the other pixel groups contain a specified number or more of pixels of the registered color; an updating means for updating the information on the registered colors stored in the storing means so that the non-registered color of the pixel corrected by the correcting means is included in the registered colors; An information processing device comprising:

2. The information processing device described in claim 1, characterized in that the correction means corrects the non-registered color pixel to correspond to a value indicating the registered color when the difference between the UV value of the non-registered color pixel extracted by the extraction means and the UV average value of each pixel of the specified number or more of the adjacent registered colors is within a predetermined threshold.

3. the correction means has a processing means for performing expansion processing and contraction processing on the area in which the registered color is detected, The information processing device according to claim 1, characterized in that the updating means updates the information on the registered colors so as to include the non-registered colors of pixels in which the non-registered color pixels have been replaced with registered color pixels by the expansion processing and contraction processing.

4. The information processing device described in claim 3, characterized in that the update means determines whether to update the information of the registered colors to include the non-registered colors in the registered colors based on a comparison between the color information of pixels in which the non-registered color pixels have been replaced with registered color pixels by the expansion processing and contraction processing and a predetermined threshold range.

5. the predetermined threshold range is a range based on color information of a background area relative to an area in which the registered color is detected from the captured image, The information processing device described in claim 4, characterized in that the update means updates the information of the registered colors so as to include the non-registered colors in the registered colors when the color information of pixels in which the non-registered color pixels have been replaced with registered color pixels by the expansion processing and contraction processing does not fall within the specified threshold range.

6. 6. The information processing apparatus according to claim 3, wherein the correction means repeats the expansion process a predetermined number of times on a correction target area set for an area in which the registered color is detected, and then repeats the contraction process a predetermined number of times.

7. 2. The information processing apparatus according to claim 1, wherein said storage means stores information on the registered colors only for a specific color space for which an offset of the color space is specified.

8. a designation means for designating a registered color based on the captured image, 2. The information processing apparatus according to claim 1, wherein said storage means stores information about the registered color designated by said designation means.

9. a determining means for determining whether the lighting conditions have been changed; When the determining means determines that the lighting conditions have been changed, the designation means designates the registered color based on the image captured under the changed lighting conditions; 9. The information processing apparatus according to claim 8, wherein said storage means stores information about the registered color designated by said designation means after the lighting conditions are changed.

10. a determining means for determining whether the size of the area of ​​the specific subject has changed; If the determining means determines that the size of the area of ​​the subject has changed, the designation means designates the registered color based on the image of the subject captured after the size has been changed; 9. The information processing apparatus according to claim 8, wherein said storage means stores information about the registered color designated by said designation means after the size of the area of ​​the subject has changed.

11. a determination means for determining whether the count value of pixels of a non-registered color has changed; If the determination means determines that the count value of the pixel of the non-registered color has changed, the designation means designates the registered color based on the image captured after the count value has changed; 9. The information processing apparatus according to claim 8, wherein said holding means holds information about the registered color designated by said designating means after the count value has changed.

12. a generating unit for generating a binarization map that expresses each pixel of the captured image as a binary color, either a registered color or a non-registered color, based on the information of the registered color; the extraction means extracts pixels having values ​​representing the unregistered color from a region of values ​​representing the registered color in the binarization map generated by the generation means; 2. The information processing apparatus according to claim 1, wherein the correction means corrects the value representing the unregistered color extracted from the binarization map by the extraction means to a value representing the registered color.

13. a distance calculation means for calculating a distance from an imaging device that captured the image for each pixel of the image by using the binarization map corrected by the correction means as an effective area for distance calculation; a synthesis means for synthesizing the captured image and a CG image based on the distance for each pixel calculated by the distance calculation means; and 13. The information processing apparatus according to claim 12, wherein the image synthesized by said synthesis means is displayed on a predetermined display device.

14. an extraction step of extracting pixels of non-registered colors that are not registered colors stored in the storage means from the captured image; a correction step of referring to another pixel group adjacent to the pixel of the non-registered color extracted in the extraction step, and correcting the pixel of the non-registered color so that the value indicating the registered color corresponds to the pixel of the non-registered color when the other pixel group contains a specified number of pixels of the registered color or more; an updating step of updating the information on the registered colors stored in the storage means so that the non-registered color of the pixel corrected in the correction step is included in the registered colors; An information processing method comprising:

15. Computer, an extracting means for extracting pixels of a non-registered color that is not a registered color stored in the storing means from the captured image; a correction means for correcting the pixels of the non-registered color so that they correspond to values ​​indicating the registered color when, as a result of referring to other pixel groups adjacent to the pixels of the non-registered color extracted by the extraction means, the other pixel groups contain a specified number or more of pixels of the registered color; an updating means for updating the information on the registered colors stored in the storing means so that the non-registered color of the pixel corrected by the correcting means is included in the registered colors; A program that causes the device to function as an information processing device having the above.

Images