Information processing apparatus, information processing method and program

Abstract

To reduce processing loads while enabling registration or updating of color information for extracting a prescribed subject area from an image.SOLUTION: An information processing apparatus includes recognition means for recognizing a prescribed subject area from a captured image, extraction means for extracting color information from the recognized subject area, registration means for registering the extracted color information as a registered color, and detection means for detecting a prescribed event. When the detection means detects the prescribed event, the information processing apparatus registers, as the registered color, the color information extracted by the extraction means from the subject area recognized by the recognition means, or the color information extracted by the extraction means from the subject area obtained by the recognition performed by the recognition means in response to the detection of the prescribed event by the detection means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing technique using images. [Background technology]

[0002] In recent years, work support systems using XR (cross reality) technology, which projects images that combine the virtual world and the real world onto an HMD (head-mounted display), have been developed and are being used in a variety of fields, including manufacturing, logistics, and business negotiations. For example, in the manufacturing field, an image of the hand area is extracted from a captured actual image, and the extracted image of the hand is then combined with a CG image of the target device to create an image of the work operation, which is then projected onto the HMD in real time, thereby improving work and training efficiency.

[0003] A known method for extracting a subject area, such as a hand, from a captured image is to extract a skin-colored area from the captured image based on pre-registered color information (e.g., skin color information). Other known methods for registering color information include extracting and registering color information for a manually specified area from the captured image using a GUI or the like, and registering color information for an area detected from the captured image using image recognition processing. Patent Document 1 also discloses a technique for identifying a person's hand area by registering the color of a face area detected by shape recognition from a captured image of the person as a reference color and comparing the colors of multiple hand areas detected by shape recognition from the same image with the reference color. In particular, the technique described in Patent Document 1 acquires the reference color using the most recent (latest) captured images (e.g., five frames). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-288951 Summary of the Invention [Problem to be solved by the invention]

[0005] In a work assistance system, multiple people may take turns wearing a single HMD. However, when the HMD user (wearer) changes, it may become impossible to accurately extract the subject area, such as the user's hands. Even when the same user wears a single HMD, if the environment or location during use changes, it may become impossible to accurately extract the subject area, such as the user's hands. This is thought to be because users' skin tones vary from person to person, and because the hue of the user's skin changes due to the influence of external light that changes depending on the location and time of day, making the registered color information incompatible. For this reason, it is desirable to update the registered color information. On the other hand, the technology described in Patent Document 1 acquires a reference color based on the most recent captured images, so the reference color is generally updated to the most recent color information. However, Patent Document 1 constantly acquires the reference color for each of the multiple images, which imposes a heavy processing load.

[0006] Therefore, an object of the present invention is to enable registration or updating of color information for extracting a predetermined subject area from an image while reducing the processing load. [Means for solving the problem]

[0007] The information processing device of the present invention comprises a recognition means for recognizing a predetermined subject area from a captured image, an extraction means for extracting color information from the subject area recognized by the recognition means, a registration means for registering the color information extracted by the extraction means as a registered color, and a detection means for detecting a predetermined event, and is characterized in that the registration means registers, as the registered color, the color information extracted by the extraction means from the subject area recognized by the recognition means when the detection means detects the predetermined event, or the color information extracted by the extraction means from the subject area obtained by recognition performed by the recognition means in response to the detection of the predetermined event by the detection means. [Effects of the Invention]

[0008] According to the present invention, it is possible to register or update color information for extracting a predetermined subject area from an image while reducing the processing load. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a functional configuration diagram of an information processing device according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of a registered color table. [Figure 3] 4 is a flowchart of information processing according to the first embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of a configuration in which processing is shared between an HMD processing unit and a PC processing unit. [Figure 5] FIG. 10 is a functional configuration diagram of an information processing device according to a second embodiment. [Figure 6] 10 is a flowchart of information processing according to the second embodiment. [Figure 7] FIG. 10 is a functional configuration diagram of an information processing device according to a third embodiment. [Figure 8] 10 is a flowchart of information processing according to the third embodiment. [Figure 9] FIG. 10 is a functional configuration diagram of an information processing device according to a fourth embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an image in which noise occurs. [Figure 11] 10 is a flowchart of information processing according to the fourth embodiment. [Figure 12] FIG. 10 is a functional configuration diagram of an information processing device according to a fifth embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a GUI for instructing a user to place a hand within a dotted frame. [Figure 14] FIG. 10 is a diagram showing an example of a GUI for notifying the user when extraction and registration are completed. [Figure 15] 13 is a flowchart of information processing according to the fifth embodiment. [Figure 16] FIG. 13 is a functional configuration diagram of an information processing device according to a sixth embodiment. [Figure 17]FIG. 10 is a diagram illustrating an example of a registered color table associated with each individual. [Figure 18] 13 is a flowchart of information processing according to the sixth embodiment. [Figure 19] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, the present invention may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, redundant descriptions of the same or similar configurations and processing steps will be omitted.

[0011] In the following embodiments, an example of application to a work support system using XR (cross reality) technology that projects an image that combines the virtual world and the real world onto an HMD (head-mounted display) will be described. In this embodiment, a case will be described in which an actual image (real world) captured of the hand and a CG image (virtual world) are precisely combined based on distance information from an imaging device mounted on the HMD to a target object, and an image of a work operation is projected onto the HMD in real time.

[0012] Here, low-latency projection is required for HMD projection in XR to improve realism. For example, when extracting hand regions using color information, it is expected that hand color information will be registered in advance to achieve low latency. On the other hand, as mentioned above, in work assistance systems, multiple people may take turns wearing a single HMD. When users (wearers) change, differences in the users' skin color may prevent accurate extraction of the user's hand region. Furthermore, even when the same user wears a single HMD, as mentioned above, the user's skin color may change due to changes in the environment or location during use, resulting in incorrect extraction of the user's hand region. For example, if the user's hand region cannot be accurately extracted and pixels within the hand region are erroneously determined to be outside the hand region, an unnatural composite image with noise in the hand region will be generated in the image projected by the HMD in XR. For this reason, it is desirable to appropriately update or re-register the registered color information.

[0013] Therefore, the information processing device according to each of the following embodiments has the following functional units and executes the following processes to enable updating and re-registration of registered colors in response to changes in the user, environment, or location during use. In each of the following embodiments, registration of color information includes new registration, updating of a registered color, and re-registration of a registered color.

[0014] First Embodiment FIG. 1 is a functional block diagram showing an example of the functional configuration of an information processing apparatus according to the first embodiment. The image receiving unit 100 receives image data of the real world captured by an imaging device (not shown) mounted on the HMD. The imaging device is, for example, a camera that captures multiple images that can be used for distance calculation, and the image receiving unit 100 receives the multiple image data. In this embodiment, the imaging device is, for example, a stereo camera, and the image receiving unit 100 receives stereo image data from the stereo camera. In the following description, image data handled in the information processing device will be simply referred to as an image unless otherwise specified. The captured image received by the image receiving unit 100 is sent to the shape recognition unit 101, the registered color detection unit 104, the distance calculation unit 106, and the synthesis unit 107.

[0015] The shape recognition unit 101 performs shape recognition processing to recognize the shape of a predetermined subject area (a human hand area in this embodiment) captured in an image input from the image receiving unit 100. In this embodiment, if the shape recognition unit 101 recognizes a predetermined subject area within a predetermined range in the image through shape recognition processing, it detects this as a predetermined event and transmits information about the recognized subject area to the registered color designation unit 102. In this embodiment, the predetermined subject area is a human hand area, and the predetermined range in the image is an area near the center of the image. Examples of the hand shape recognition processing include known shape recognition processing using bone information connecting the joints of the hand and shape recognition processing using a model previously trained to recognize the shape of the hand. Note that because the shape recognition processing is a relatively heavy process, when the shape recognition processing is performed, it is performed, for example, less frequently than the frequency at which the registered color detection unit 104, described below, performs registered color detection processing. The shape recognition unit 101 then transmits information indicating the hand area recognized as the predetermined subject area through shape recognition processing to the registered color designation unit 102 along with the captured image. That is, the shape recognition unit 101 detects, as a predetermined event, that a human hand has been recognized within a range near the center of the image through shape recognition processing, and sends information about the hand area recognized upon detection of that event and the image to the registered color designation unit 102.

[0016] When the registered color designation unit 102 receives the hand region information and the captured image from the shape recognition unit 101, it designates the hand region as a color extraction region and extracts color information from the image of the color extraction region. Then, the registered color designation unit 102 generates a registered color table with the extracted color information as the registered color. In this embodiment, the registered color table is a table for masking regions other than the registered color from the image captured by the imaging device and extracting only the registered color region. That is, in this embodiment, the registered color table is a table for performing hand masking processing, which masks regions of colors different from the registered color of the hand and extracts only the color region of the hand.

[0017] FIG. 2 is a diagram showing an example of the registered color table. The registered color table includes an INDEX column in which an INDEX corresponding to each set of Y, U, and V values ​​is written; a YUV column in which the Y, U, and V values ​​are each written as an 8-bit value; and a Registered / Unregistered column in which a value indicating whether the color is registered or unregistered is written. The Registered / Unregistered column stores a "1" if the color is registered and a "0" if the color is unregistered. The registered color designation unit 102 acquires YUV values ​​from the color extraction region as color information for the hand region and stores the acquired YUV values ​​as registered colors. For example, INDEX[9528979] (Y=145, U=102, V=147) is color information extracted from the hand region, so the Registered / Unregistered column stores a "1" and the color is registered as a registered color. For example, Y=145, U=102, V=148 in INDEX[9528980] is color information that has not been extracted from the hand area, so the value in the registration / non-registration column is "0" and it is an unregistered color (i.e., it is registered as an unregistered color). Note that the registered color table in Figure 2 shows an example in which the color space is expressed in YUV, but it is not limited to YUV and color spaces such as RGB, HSV, Lab, etc. can also be used.

[0018] The registered color storage unit 103 stores the registered color table generated by the registered color designation unit 102. In this embodiment, as described above, the registered color designation unit 102 outputs the registered color table, in which the color information extracted from the hand region is used as the registered color, to the registered color storage unit 103. Therefore, if the registered color storage unit 103 already stores a registered color table, the registered color table is registered (updated or re-registered) with color information generated in response to the detection of a predetermined event. Furthermore, in response to a registered color table reference request from the registered color detection unit 104, the registered color storage unit 103 outputs the registered color table to the registered color detection unit 104.

[0019] The registered color detection unit 104 extracts color information (YUV values) for each pixel of the image received by the image receiving unit 100 from the imaging device and determines whether a pixel of interest in the image is a registered color or an unregistered color by referencing the registered color table in the registered color storage unit 103. The registered color detection unit 104 then generates a binarized map (hereinafter referred to as the registered color map) in which a value of "1" indicating a registered color or a value of "0" indicating an unregistered color is assigned to each pixel of the image. The registered color map is thus a binarized map in which a value indicating a registered color or an unregistered color is assigned to each pixel of the image. However, for convenience in the following explanation, the location in the registered color map corresponding to each pixel of the image will also be referred to as a pixel. Furthermore, within the registered color map, a location assigned a value indicating a registered color will be referred to as a registered color pixel, and a location assigned a value indicating an unregistered color will be referred to as an unregistered color pixel. Note that the registered color map can use identifiers other than the binary map's "0" and "1" as long as the locations of registered and unregistered colors can be distinguished.

[0020] The registered color transmission unit 105 transmits the registered color map generated by the registered color detection unit 104 to the distance calculation unit 106. At this time, the registered color transmission unit 105 transmits the registered color map as valid area information when the distance calculation unit 106 calculates the distance.

[0021] The distance calculation unit 106 calculates the distance from the image capture device for each pixel in the stereo image received by the image receiving unit 100 while referring to the valid area information (registered color map), and generates a distance map in which a distance value is allocated for each pixel. That is, the distance calculation unit 106 generates the distance map by calculating the distance value for each pixel, with the area consisting of each registered color pixel in the registered color map as the valid area for distance calculation. In this way, the distance calculation unit 106 calculates the distance for the valid area, so that the dependency on the background (in this embodiment, the background other than the hand area) during distance calculation can be reduced, which can improve the accuracy of distance calculation and reduce the load (speed). Note that a known method can be used for distance calculation, such as a method of calculating parallax using template matching and converting the parallax into distance.

[0022] The CG generation unit 108 generates CG data of the virtual world. That is, the CG generation unit 108 generates CG data of the virtual world to be synthesized (fused) with the image received by the image receiving unit 100, i.e., the actual image of the real world captured by the imaging device. The CG generation unit 108 then transmits the generated CG data to the synthesis unit 107.

[0023] The synthesis unit 107 generates a synthetic image by synthesizing the CG data generated by the CG generation unit 108 with a real image based on the distance map generated by the distance calculation unit 106. Here, in order to avoid impairing the sense of realism in XR, for example, when an image of a hand is projected onto the HMD, it is necessary to synthesize and display the image so that a subject (the hand of the HMD wearer) in the foreground of the CG object is not hidden by the CG object. For this reason, for example, if the hand is approximately 10 cm away from the imaging device and the CG object is approximately 20 cm away from the imaging device, the synthesis unit 107 superimposes the image of the hand region on the image of the CG object. This generates a synthetic image that makes it clear that the hand is closer to the imaging device than the CG object. On the other hand, if the hand is 30 cm away from the imaging device and the CG object is 20 cm away from the imaging device, the synthesis unit 107 superimposes the image of the CG object on the image of the hand region. This generates a synthetic image that makes it clear that the hand is farther from the imaging device than the CG object.

[0024] The projection unit 109 projects the composite image generated by the synthesis unit 107 onto a display device (display) included in the HMD. As a result, an image that precisely combines the hand shown in the actual image (real world) captured of the hand and the CG image (virtual world) of the target device or the like based on distance information is projected on the HMD display in real time.

[0025] 3 is a flowchart showing the flow of information processing executed by the information processing device according to the first embodiment. In the information processing device of this embodiment, the processing from step S300 to step S303 and the processing from step S304 to step S307 are performed in parallel. First, in step S300, the image receiving unit 100 receives an image from the imaging device and transmits the image to the shape recognition unit 101. Next, in step S301, the shape recognition unit 101 executes hand shape recognition processing on the image received from the image receiving unit 100. That is, as described above, when the shape recognition unit 101 recognizes a human hand within a range near the center of the image through shape recognition processing, it detects this as a predetermined event and transmits information about the recognized hand area and the captured image to the registered color designation unit 102.

[0026] Next, in step S302, the registration color designation unit 102 designates the hand region recognized by shape recognition in the captured image as a color extraction region. Furthermore, in the processing of the next step S303, the registered color designation unit 102 extracts color information from the color extraction region and generates a registered color table with the extracted color information as the registered color. Then, the registered color holding unit 103 holds the registered color table generated by the registered color designation unit 102. After processing of this step S303, the processing of the information processing device returns to step S300.

[0027] Furthermore, in step S304, the image receiving unit 100 transmits the image received from the imaging device to the registered color detecting unit 104. Next, in step S305, image receiving unit 100 extracts color information for each pixel from the captured image received. Furthermore, registered color detection unit 104 determines whether a pixel of interest in the image is a registered color or a non-registered color by referencing the registered color table stored in registered color storage unit 103. Registered color detection unit 104 then generates a registered color map in which a value of "1" indicating a registered color or a value of "0" indicating a non-registered color is assigned to each pixel of the image. That is, registered color detection unit 104 detects each pixel of a color corresponding to a registered color from the input image, thereby detecting a hand region and generating a registered color map representing the detected hand region.

[0028] Next, in the process of step S306, distance calculation unit 106 generates a distance map in which distance values ​​are allocated for each pixel, based on the registered color map received via registered color transmission unit 105. Furthermore, synthesis unit 107 generates a composite image by synthesizing the CG data generated by CG generation unit 108 with the actual image, while referring to the distance map generated by distance calculation unit 106. Then, projection unit 109 projects the composite image onto a display device (display) provided in the HMD. Thereafter, in the process of step S307, the information processing device determines whether to end the series of processes in Fig. 3, and if the processes are to be continued without ending, the process returns to step S304 and performs the processes from step S304 onwards. On the other hand, if the information processing device determines that the series of processes are to end, it ends the process of the flowchart in Fig. 3. Note that the determination of whether to end the processes in step S307 is made, for example, by determining whether an instruction to end has been received from the user.

[0029] As described above, in the first embodiment, when a predetermined event is detected in the hand shape recognition process, color information is extracted from the hand region and registered. Furthermore, the hand region is detected from the captured image based on the registered color. That is, according to this embodiment, the shape of a predetermined subject, such as a hand, is recognized, and color information extracted from the region resulting from the shape recognition is registered as a registered color to be used for hand region detection. This reduces the occurrence of noise caused by pixels within the hand region that should be a hand being erroneously determined to be pixels outside the hand region.

[0030] Furthermore, according to this embodiment, the process of extracting and registering color information from a hand region and the process of detecting a hand region from a captured image based on the registered color are performed at different frequencies. That is, because the process of recognizing the shape of a hand or the like is generally complex and time-consuming, the process of extracting and registering color information from a hand region is performed less frequently than the process of detecting a hand region from a captured image based on the registered color.

[0031] Note that, in FIG. 1, an example is shown in which the processes from extracting color information based on the hand shape recognition results to registering it, and the processes from detecting the hand area from the captured image to combining it with a CG image are executed within a single information processing device, but this is not limiting. For example, as shown in the functional configuration diagram of FIG. 4 , the process from extracting color information based on the hand shape recognition results to registering it and the process from detecting the hand area from the captured image to combining it with a CG image may be performed by separate devices. Note that the functional units shown in FIG. 4 operate in the same manner as the corresponding functional units shown in the functional configuration of FIG. 1 described above, and therefore their description will be omitted. The configuration of FIG. 4 illustrates an example in which the functions of the shape recognition unit 101 and the registration color designation unit 102 are realized by a PC processing unit 401, and the other functions are realized by an HMD processing unit 400. The PC processing unit 401 is an information processing device such as a personal computer, and the HMD processing unit 400 is, for example, a processing device installed in an HMD. In other words, the configuration of FIG. 4 illustrates an example in which heavy processing, such as shape recognition processing, is performed by the PC processing unit 401, and processing requiring low latency processing is performed by the HMD processing unit 400. Note that the configuration in which the processes are separated as shown in FIG. 4 can also be applied to each of the embodiments described below.

[0032] <Second embodiment> Next, in a second embodiment, an example is given in which a predetermined event is detected based on the color difference between color information extracted based on the hand shape recognition result and a registered color registered in a registered color table. The information processing device of the second embodiment detects a predetermined event when the color difference between the color information extracted based on the hand shape recognition result and a registered color registered in the registered color table is equal to or greater than a predetermined difference threshold. When the predetermined event is detected, the information processing device of the second embodiment registers (updates or re-registers) the color information extracted from the hand region in the registered color table.

[0033] Fig. 5 is a functional block diagram showing an example of the functional configuration of an information processing device according to the second embodiment, and Fig. 6 is a flowchart showing the flow of information processing according to the second embodiment. In the functional configuration of Fig. 5, explanations of functional units that perform roughly the same operations as the functions in Fig. 1 will be omitted. In addition, in the flowchart of Fig. 6, explanations of steps that are the same as those in the flowchart of Fig. 3 will be omitted. The following explanation will focus on the differences from the first embodiment.

[0034] In the configuration of the second embodiment shown in Fig. 5, the registered color designation unit 102 transmits color information extracted from the hand region whose shape has been recognized by the shape recognition unit 101 to the color difference detection unit 500. Note that the shape recognition unit 101 may detect the recognition of the hand region as a predetermined event as in the first embodiment, but in the case of the second embodiment, it is the color difference detection unit 500 that detects the predetermined event.

[0035] The color difference detection unit 500 calculates an index representing the color difference between a registered color stored in the registered color storage unit 103 and color information extracted from the hand region by the registered color designation unit 102. For example, the color difference detection unit 500 calculates the difference in index representing the distance between two colors in the L*a*b* color space, such as ΔE, as an index representing the color difference. Furthermore, if ΔE, the index representing the color difference, is equal to or greater than a predetermined difference threshold, the color difference detection unit 500 detects this as a predetermined event. Then, when the color difference detection unit 500 detects a predetermined event, it registers the color information extracted from the hand region in a registered color table stored in the registered color storage unit 103. The color difference detection unit 500 may make a determination based on, for example, ΔE=2.5 (which allows colors to be recognized as nearly identical when determined at a distance).

[0036] In the flowchart shown in FIG. 6 of the second embodiment, the processes from step S600 to step S504 and the processes from step S304 to step S307 similar to those in FIG. 3 described above are performed in parallel. Steps S600 to S602 are generally similar to steps S300 to S302 in FIG. 3, but in the second embodiment, in step S602, the registered color designation unit 102 transmits color information extracted from the hand region to the color difference detection unit 500.

[0037] Next, in step S603, the color difference detection unit 500 calculates an index representing the color difference from the color information extracted from the hand region by the registered color designation unit 102 and the registered color already registered in the registered color storage unit 103, and compares the index with a predetermined difference threshold. If the index representing the color difference is less than the predetermined difference threshold, the color difference detection unit 500 returns to step S600, but if it is equal to or greater than the difference threshold, the process proceeds to step S604.

[0038] In step S604, the registered color holding unit 103 registers, in the registered color table, the color information extracted from the hand region by the registered color designation unit 102. After the process of step S604, the process of the information processing device returns to step S600.

[0039] As described above, in the second embodiment, when a predetermined event is detected in which the difference between a registered color and the color of the hand region extracted by shape recognition exceeds a predetermined difference threshold, the registered color table is updated or re-registered. This makes it possible to reduce noise caused by pixels in the hand region that should be hands being erroneously determined to be pixels outside the hand region.

[0040] <Third embodiment> Next, as a third embodiment, an example in which an environmental change is detected as a predetermined event will be described. In the third embodiment, when an environmental change is detected as a predetermined event, a hand shape recognition process is performed to acquire a hand region, and color information extracted from the hand region is registered in a registered color table.

[0041] Fig. 7 is a functional block diagram showing an example of the functional configuration of an information processing device according to the third embodiment, and Fig. 8 is a flowchart showing the flow of information processing according to the third embodiment. Note that in the functional configuration of Fig. 7, explanations of functional units that perform roughly the same operations as the respective functions in Fig. 1 will be omitted. The following explanation will focus on the differences from the first embodiment.

[0042] 7 of the third embodiment is provided with an environment detection unit 700. In the third embodiment, the shape recognition unit 101 may detect the recognition of a hand region as a predetermined event, as in the first embodiment, but in the third embodiment, the environment detection unit 700 detects the predetermined event.

[0043] The environment detection unit 700 receives environmental data acquired by an environmental sensor, analyzes the environmental data, and if an environmental change is detected, detects it as a predetermined event. In the third embodiment, the shape recognition unit 101 executes shape recognition processing in response to the detection of an environmental change as a predetermined event. Therefore, the registered color designation unit 102 in the third embodiment extracts color information from the hand region acquired by the shape recognition processing performed in response to the detection of the environmental change and generates a registered color table. That is, in the third embodiment, the shape recognition processing is performed in response to the detection of the environmental change, and the color information extracted from the hand region by the shape recognition processing is registered in the registered color table of the registered color storage unit 103.

[0044] The environmental sensor may be any of various optical sensors (such as an RGB sensor) that measure ambient light and detect the brightness (light intensity) and color tone (color temperature) of the ambient light. The environment detection unit 700 monitors the brightness (light intensity) and color tone (color temperature) of the ambient light measured by the environmental sensor as parameters that cause changes in the color of the hand. For example, the environment detection unit 700 monitors, at regular intervals, the average color temperature of the sky if the HMD is used outdoors, or the average color temperature of the ceiling lighting if the HMD is used indoors, for example, using an RGB sensor.

[0045] The environment detection unit 700 detects a change in the monitored environment as a predetermined event when the change is equal to or greater than a predetermined change threshold. As an example, the environment detection unit 700 detects a color temperature difference of 100 Kelvin or greater (or, expressed in color difference, ΔE 2.5 or greater), for example, as a predetermined event, and transmits the detection result to the shape recognition unit 101. Note that the environment sensor is not limited to an optical sensor that detects the brightness and color of the ambient light, but may also be a movement sensor that detects the movement of the HMD wearer. In this case, the environment detection unit 700 uses the movement sensor to detect the movement of the HMD wearer between indoors and outdoors as an environmental change that is likely to result in a change in the color of the hands, and transmits the detection result to the shape recognition unit 101.

[0046] When an environmental change is detected by the environment detection unit 700, the shape recognition unit 101 of the third embodiment executes hand shape recognition processing and transmits information indicating the hand region and the captured image to the registered color designation unit 102. The registered color designation unit 102 of the third embodiment extracts color information from the hand region acquired by the shape recognition processing performed in response to the detection of an environmental change, generates a registered color table, and causes the registered color storage unit 103 to store the table.

[0047] In this way, the information processing device of the third embodiment constantly monitors the situation of the surrounding environment in which the HMD is used based on the output of the environmental sensor, and when an environmental change is detected, processing from shape recognition onwards is carried out to register (update or re-register) the registered color. After the registered color is registered in response to the detection of an environmental change, as in the previous embodiment, the hand region corresponding to the registered color is detected based on the information in the registered color table, and a CG image is synthesized based on distance information and projected onto the HMD.

[0048] In the flowchart of the third embodiment shown in FIG. 8, first, in the process of step 800, the environment detection unit 700 detects environmental changes by monitoring the environment using the output of the environmental sensor as described above. Next, in step S801, environment detection unit 700 determines whether environmental fluctuations have made it necessary to register (update or re-register) the registered color table. That is, if the environmental fluctuations are equal to or greater than a predetermined fluctuation threshold, environment detection unit 700 determines that registration (update or re-registration) of the registered color table is necessary. If it is determined that registration of the registered color table is necessary, the process of the information processing device proceeds to step S802. On the other hand, if it is determined that registration (update or re-registration) of the registered color table is not necessary, the process of the information processing device proceeds to step S803.

[0049] If the process proceeds to step S802, the information processing apparatus executes registration (update or re-registration) of the registered color table in the same procedure as in the first embodiment, and then proceeds to the process of step S803. When the process proceeds from step S801 or step S802 to step S803, the image receiving unit 100 transmits to the registered color detecting unit 104 the captured image received from the imaging device. Next, in the process of step S804, the registered color detection unit 104, similar to the embodiment described above, references the registered color table to detect a hand region from the captured image and generates a registered color map. In the next step S805, distance calculation unit 106 generates a distance map based on the registered color map, and synthesis unit 107 generates a synthesized image by synthesizing the CG data and the actual image based on the distance map. Then, projection unit 109 projects the synthesized image onto a display device (display) provided in the HMD. Thereafter, in the process of step S806, the information processing device determines whether to end the series of processes in Fig. 8, and if the processes are to be continued without ending, the process returns to step S800 and the subsequent processes are performed. In other words, by returning to the process of step S800, it becomes possible to deal with further environmental changes. On the other hand, if the information processing device determines that the series of processes is to end, it ends the process of the flowchart in Fig. 8.

[0050] As described above, according to the third embodiment, when a predetermined event is detected based on an environmental change, the registered color table is registered (updated or re-registered) by processing after shape recognition. This makes it possible to reduce noise caused by pixels within the hand area being erroneously determined to be a hand as pixels outside the hand area, even in the third embodiment.

[0051] <Fourth embodiment> Next, in a fourth embodiment, an example will be described in which a predetermined event is detected based on a registered color map generated by referring to the registered colors in the registered color table (i.e., registered color information). That is, in the fourth embodiment, when a predetermined event is detected based on the registered color map generated by referring to the registered colors in the registered color table, the registered color table is registered (updated or re-registered).

[0052] Fig. 9 is a functional block diagram showing an example of the functional configuration of an information processing device according to the fourth embodiment. In the functional configuration of Fig. 9, explanations of functional units that perform roughly the same operations as the respective functions in Fig. 1 will be omitted. The following explanation will focus on the differences from the first embodiment.

[0053] 9 of the fourth embodiment is provided with a color loss extraction unit 900. In the fourth embodiment, the shape recognition unit 101 may detect the recognition of a hand region as a predetermined event, as in the first embodiment, but in the fourth embodiment, the color loss extraction unit 900 detects the predetermined event. In the fourth embodiment, the captured image received by the image receiving unit 100 is also sent to the color loss extraction unit 900 via the shape recognition unit 101 and the registered color designation unit 102.

[0054] The color loss extraction unit 900 uses the registered colors stored in the registered color storage unit 103 to detect a hand region from the captured image, generate a registered color map, and extract non-registered color pixels from the registered color map. Furthermore, the color loss extraction unit 900 calculates an evaluation index based on the number of non-registered color pixels or the area of ​​the region caused by non-registered color pixels in the registered color map, and compares the evaluation index with a predetermined evaluation threshold to determine whether registration (updating or re-registration) of the registered color table is necessary. That is, if the evaluation index is equal to or greater than the predetermined evaluation threshold, the color loss extraction unit 900 detects this as a predetermined event, and registers (updates or re-registers) the registered color table in the registered color storage unit 103 in response to the detection of the predetermined event.

[0055] For example, when the number of unregistered color pixels is used as the evaluation index, the decolorization extraction unit 900 determines whether the number of unregistered color pixels is equal to or greater than a predetermined evaluation threshold (e.g., 1%) relative to the number of pixels in the registered color map, i.e., whether the number of unregistered color pixels is equal to or greater than the predetermined evaluation threshold. If the decolorization extraction unit 900 determines that the number of unregistered color pixels is equal to or greater than the predetermined evaluation threshold, it detects this as a predetermined event and updates or re-registers the registered color table in the registered color storage unit 103 using a newly generated registered color table in the registered color designation unit 102. Similarly, when the area of ​​an unregistered color pixel region is used as the evaluation index, the decolorization extraction unit 900 determines whether the area of ​​the unregistered color pixel region is equal to or greater than a predetermined evaluation threshold relative to the area of ​​the registered color map. If the decolorization extraction unit 900 determines that the area of ​​the unregistered color pixel region is equal to or greater than the predetermined evaluation threshold, it detects this as a predetermined event and registers the registered color table generated by the registered color designation unit 102 in the registered color storage unit 103, as in the first embodiment.

[0056] FIG. 10 illustrates an example of an image obtained by masking a captured image using a registered color map generated based on the registered color table stored in the registered color storage unit 103. In an image obtained by masking a captured image using the registered color map, registered color pixels are displayed in white and non-registered color pixels are displayed in black. For example, if the skin color or lighting conditions have changed since the registered color table was generated, the registered color map generated using the registered color table may not be able to accommodate the changed skin color or lighting conditions. In other words, when a captured image is masked using the registered color map, multiple pixels in the hand area that should all be determined to be registered color pixels may be erroneously determined to be non-registered color pixels, resulting in noise represented by black pixels. If the registered color map generated using the registered color table is used in the subsequent distance calculation and image synthesis, inconsistencies may occur in the distance map threshold determination process, resulting in an unnatural composite image being projected onto the HMD. For example, an unnatural composite image may be generated by associating non-registered color pixels with pixels representing a CG object located behind the hand, resulting in an unnatural composite image being projected onto the HMD.

[0057] Therefore, in the fourth embodiment, if the number of non-registered color pixels or the area of ​​the non-registered color pixel area in the area corresponding to the hand in the registered color map generated based on the registered color table is equal to or greater than a predetermined evaluation threshold, the registered color table is registered (updated or re-registered).

[0058] FIG. 11 is a flowchart showing the flow of information processing according to the fourth embodiment. In the flowchart of FIG. 11, first, in step 1100, the image receiving unit 100 receives a captured image from the imaging device, and the captured image is sent to the color loss extraction unit 900 via the shape recognition unit 101 and the registered color designation unit . Next, in step S1101, the color loss extraction unit 900 uses the registered color table in the registered color holding unit 103 to detect a hand region from the captured image and generate a registered color map. Next, in step S1102, the color loss extraction unit 900 performs a color loss detection process to extract non-registered color pixels from the registered color map.

[0059] Next, in step S1103, the color loss extraction unit 900 calculates the evaluation index described above and, based on a comparison between the evaluation index and a predetermined evaluation threshold, determines whether the color information in the registered color table that has already been registered can no longer accommodate changes in skin color, lighting conditions, etc. That is, in step S1103, the color loss extraction unit 900 determines whether to register (update or re-register) the registered color table in the registered color holding unit 103 with a newly generated registered color table created by the registered color designation unit 102. If the color loss extraction unit 900 determines to register (update or re-register) the registered color table, the information processing device proceeds to step S1104. On the other hand, if it determines not to register (update or re-register) the registered color table, the information processing device proceeds to step S1109.

[0060] Steps S1104 to S1107 are the same as steps S300 to S303 in FIG. 3, and after step S1107, the process of the information processing device proceeds to step S1108. In step S1108, the registered color detection unit 104 refers to the registered color table registered in the registered color holding unit 103 in step S1107, and performs the process of detecting the hand region in the same manner as described above.

[0061] Proceeding to the next step S1109, distance calculation unit 106 generates a distance map based on the registered color map, and composition unit 107 generates a composite image by combining the CG data and the actual image based on the distance map. Then, projection unit 109 projects the composite image onto the display device provided in the HMD. Thereafter, in step S1110, the information processing device determines whether to terminate the series of processes in Fig. 11, and if the processes are to continue without terminating, the process returns to step S1100 and performs the processes from step S1100 onwards. In other words, by returning to the process of step S1100, if the skin color or lighting conditions, etc., change again and the registered color table can no longer accommodate those changes, the registered color table can be updated or re-registered. On the other hand, if the information processing device determines that the series of processes should terminate, the process of the flowchart in Fig. 11 ends.

[0062] As described above, in the fourth embodiment, if the number of unregistered color pixels or unregistered color pixel areas in a registered color map generated based on a registered registered color table exceeds a predetermined evaluation threshold, the registered color table in the registered color storage unit 103 is updated or re-registered. As a result, according to the fourth embodiment, it is possible to reduce the occurrence of noise caused by pixels within the hand area that should be a hand being erroneously determined to be pixels outside the hand area.

[0063] <Fifth embodiment> Next, in a fifth embodiment, an example will be described in which the arrival of a user wearing an HMD is detected as a predetermined event. That is, in the fifth embodiment, registration of the registered color table is performed in response to the detection of the predetermined event that a user wears an HMD.

[0064] Fig. 12 is a functional block diagram showing an example of the functional configuration of an information processing device according to the fifth embodiment. In the functional configuration of Fig. 12, explanations of functional units that perform roughly the same operations as the respective functions in Fig. 1 will be omitted. The following explanation will focus on the differences from the first embodiment.

[0065] In the configuration of the fifth embodiment shown in FIG. 12, a mounting detection unit 1200 and a hand position designation unit 1201 are provided. The wearing detection unit 1200 detects that the HMD is worn on the head of the user. For example, a method for detecting wearing of the HMD can be used, in which a pressure sensor provided on the head-mounted surface of the HMD detects a change in pressure when the HMD comes into contact with the head. Alternatively, an infrared light emitting unit can be provided on the head-mounted surface of the HMD, and the HMD can be detected as being worn when an infrared sensor detects infrared light emitted from the infrared light emitting unit and reflected by the user's head or the like.

[0066] When the wearing detection unit 1200 detects that the HMD is being worn, the hand position designation unit 1201 calculates the position where the user's hand should be placed within the projection space created by the HMD projection unit 109, and outputs the position information to the CG generation unit 108 and the shape recognition unit 101.

[0067] When CG generation unit 108 receives position information on where the hand should be placed from hand position specification unit 1201, it generates guidance text and a CG image informing the user to place their hand at that position and transmits them to synthesis unit 107. At this time, synthesis unit 107 superimposes the guidance text and CG image informing the user to place their hand on the captured image of real space received by image receiving unit 100. As a result, the guidance text and CG image informing the user to place their hand are displayed on the image projected by projection unit 109.

[0068] The shape recognition unit 101 determines a search range for the captured image received from the image receiving unit 100 based on the position information of the hand received from the hand position designation unit 1201, and acquires a hand region by performing shape recognition of the user's hand within the search range. Note that if the user's hand is not captured within the search range, the shape recognition unit 101 may send information to that effect to the hand position designation unit 1201. In this case, the hand position designation unit 1201 may recalculate the position where the user should place their hand, and send position information based on the recalculation to the CG generation unit 108 and the shape recognition unit 101.

[0069] The subsequent processing is the same as in the first embodiment described above, and the registered color designation unit 102 generates a registered color table based on the hand region acquired by the shape recognition unit 101, and the registered color table is registered in the registered color storage unit 103. In the fifth embodiment, when registration of the registered color table is complete, the CG generation unit 108 generates a graphic notifying the user that registration is complete, and the projection unit 109 projects this graphic. The subsequent processing is the same as in the first embodiment described above, and therefore a description thereof will be omitted.

[0070] 13 is a diagram showing an example of an image in which text and a CG image informing the user of the position where the hand should be placed, specified by the hand position specifying unit 1201, are superimposed on a captured image. As shown in the example of Fig. 13, the CG image indicating the position where the hand 1302 should be placed is displayed as an image of, for example, a dotted frame 1301 that resembles the shape of a hand, near the center of the image on the front side of the HMD within the captured image in real space. Furthermore, the text indicating the position where the hand 1302 should be placed is displayed as, for example, text informing the user to place the hand within the dotted frame 1301.

[0071] In the fifth embodiment, the shape recognition unit 101 sets the search range near the center of the screen, including the dotted-line frame 1301 illustrated in FIG. 13, and performs shape recognition processing. While the present embodiment illustrates an example of performing shape recognition on only one of the user's hands, it is also possible to recognize both the user's left and right hands and extract color information for both hands. Furthermore, the color of the palm side of the hand may be extracted in addition to the back side. In this case, the user may be notified of the orientation of the hand to be acquired each time, and shape recognition and color information extraction may be performed multiple times. Then, stable color information may be obtained by averaging the color information acquired multiple times, or a wider range of colors may be detected by acquiring color information for both the back side and palm side of the hand, which may differ slightly in skin color.

[0072] 14 is a diagram showing an example of shape recognition of a hand 1302 when the user places the hand 1302 within a dotted-line frame 1301 that resembles the shape of the hand, representing the position where the hand should be placed. The shape recognition unit 101 performs hand shape recognition with the user's hand 1302 placed within the dotted-line frame 1301, thereby detecting the approximate positions of the hand 1302 and each finger. The registered color designation unit 102 may set, for example, the center of the back of the hand as the extraction region for color information, or may set the entire region within the dotted-line frame as the extraction region for color information if the accuracy of shape recognition is high. Thereafter, the information processing device may notify the user, for example, after the extraction of color information and the registration of the registered color table are completed.

[0073] FIG. 15 is a flowchart showing the flow of information processing according to the fifth embodiment. 15, the wearing detection unit 1200 determines whether or not an HMD has been newly worn. If no new wearing is detected, the process of the information processing device proceeds to step S1501. On the other hand, if new wearing is detected, the process of the information processing device proceeds to step S1502.

[0074] When the process proceeds to step S1502, the hand position designation unit 1201 calculates the position where the user places their hand, the CG generation unit 108 generates text and a CG image to guide the user in placing their hand at that position, the synthesis unit 107 superimposes them on the captured image, and the projection unit 109 projects them. The following steps S1503 to S1506 are generally similar to steps S300 to S303 in Fig. 3, but in step S1504, shape recognition unit 101 performs shape recognition processing using the area around the center of the screen enclosed by the dotted line frame as described in Fig. 13 as the search range. After step S1506, the processing of the information processing device proceeds to step S1501.

[0075] 3, but if it is determined in step S1509 that the process should not end, the process of the information processing device returns to step S1500. In other words, by returning to the process of step S1500 and determining whether or not an HMD has been newly worn, for example, if the user is a different person, it becomes possible to re-register the registered color table corresponding to the new user.

[0076] As described above, in the fifth embodiment, when the user of the HMD changes, this is detected as a predetermined event, and a registered color table is registered in the registered color storage unit 103. Also, in the fifth embodiment, the user is notified of the position where to place their hands, and the shape of the hand can be recognized at that position, thereby enabling extraction of color information by specifying an accurate color extraction region. Also, in the fifth embodiment, color extraction is performed multiple times, thereby improving the accuracy of the registered color information.

[0077] Sixth Embodiment Next, in the sixth embodiment, an example will be described in which the identification of an individual user is detected as a predetermined event, a determination is made as to whether or not registration of the registered color table is necessary, and registration of the registered color table (new registration, update, or re-registration) is performed based on the determination result.

[0078] Fig. 16 is a functional block diagram showing an example of the functional configuration of an information processing device according to the sixth embodiment. In the functional configuration of Fig. 16, explanations of functional units that perform roughly the same operations as the respective functions in Fig. 1 will be omitted. The following explanation will focus on the differences from the first embodiment.

[0079] In the configuration of the sixth embodiment shown in FIG. 15, an individual identification unit 1600 is provided. The individual identification unit 1600 receives the individual recognition data of the user acquired by a personal recognition device (not shown). The individual identification unit 1600 determines whether or not registration of the registered color table for that user is necessary. Furthermore, if it is determined that registration of the registered color table is necessary, it detects this as a predetermined event. In the sixth embodiment, when it detects as a predetermined event that it has been determined that registration of the registered color table is necessary, the information processing device registers the registered color table (newly registers, updates, or re-registers).

[0080] Examples of a personal recognition method using a personal recognition device (not shown) include a method of detecting an iris pattern from an image of the eyes captured by a gaze input camera provided inside the HMD and using the iris pattern for personal recognition. Other personal recognition methods such as fingerprint recognition may also be used.

[0081] In the sixth embodiment, after the individual user has been identified, hand shape recognition processing is performed by the shape recognition unit 101, a color extraction area is specified by the registered color specification unit 102, and a registered color table is generated, similar to the first embodiment. Then, in the sixth embodiment, the registered color table is associated with the identification result of the individual recognition by the individual identification unit 1600 and registered in the registered color holding unit 103.

[0082] FIG. 17 shows an example of a registered color table associated with the identification result of personal recognition and registered in the registered color storage unit 103. The registered color table stores a set of tables, each of which sets registered colors and non-registered colors in the YUV color space as described in FIG. 2, in a list format linked to the identification result of personal recognition. That is, in the example of FIG. 17, table sets A to N, each of which sets registered colors and non-registered colors, are stored linked to the corresponding identification result (1 to n) of personal recognition. In the registered color table, as the number of HMD users increases, the number of table sets linked to each individual user increases. Note that if, as a result of identifying an individual user, the person wearing the HMD is already registered in the registered color table and managed as a list, the process up to generating the registered color table including the hand shape recognition by the shape recognition unit 101 need not be performed.

[0083] After determining whether or not to register the registered color table based on the result of identifying the individual user as described above, the same processing as in the first embodiment is performed. That is, the information processing device performs processing from detecting the hand area by the registered color using the registered registered color table, to synthesizing the CG image based on the distance information, and projecting it onto the HMD.

[0084] FIG. 18 is a flowchart showing the flow of information processing according to the sixth embodiment. In the flowchart shown in FIG. 18 of the sixth embodiment, first, in step 1800, the individual identification unit 1600 performs the process of identifying an individual based on the individual recognition data as described above. In the next step S1801, the individual identification unit 1600 determines whether the individual identified in step S1800 is already registered in the registered color storage unit 103. If it is determined in step S1801 that the individual is registered, the processing of the information processing device proceeds to step S1803, and on the other hand, if it is determined that the individual is not registered, the processing of the information processing device proceeds to step S1802.

[0085] When the process proceeds to step S1802, the information processing device generates a registered color table in the same manner as in the first embodiment, and the registered color table is linked to the identification result of the individual identified in step S1800 and registered in the registered color holding unit 103. After step S802, the process of the information processing device proceeds to step S1809.

[0086] When the process proceeds from step S1801 or step S1802 to step S1803, the subsequent steps S1803 to S1806 perform the same processing as steps S304 to S307 in Fig. 3 described above. However, if it is determined in step S1806 that the processing should not end, the processing of the information processing device returns to step S1800. That is, by returning to the processing of step S1800, if new personal recognition data has been acquired, the individual identification unit 1600 performs processing to identify the individual based on the personal recognition data, and further determines in step S1801 whether the person has already been registered. Note that if new personal recognition data has not been acquired, the processing of steps S1800, S1801, and S1802 is skipped.

[0087] As described above, in the sixth embodiment, whether or not registration of the registered color table is necessary is determined based on the result of identifying the individual user, and the determination that registration is necessary in the determination process is detected as a predetermined event. Then, in the information processing device, registration of the registered color table (update, re-registration, or new registration) is performed in response to the detection of the predetermined event. Thus, according to the sixth embodiment, registration of the registered color table is performed as necessary based on the result of identifying the individual user. After registering a registered color table for each individual as in the sixth embodiment, it is also possible to update or re-register the registered color table as described in the third, fourth, and fifth embodiments.

[0088] <Hardware configuration of information processing device> FIG. 19 is a diagram showing an example of the hardware configuration of an information processing apparatus according to this embodiment. The CPU 1000 is a central processing unit (Central Processing Unit) that performs calculations and logical decisions for various processes. The ROM (Read-Only Memory) 1010 stores a control program. The RAM (Random Access Memory) 1020 is used as a temporary storage area such as the CPU 1000's main memory and work area. The mass storage device 1030 is a recording device for storing the information processing program according to this embodiment and various data used for generating CG data, etc. An external storage device may also be used to perform a similar function to the mass storage device 1030. The external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for accessing the medium. Known examples of such media include a flexible disk (FD), CD-ROM, DVD, USB memory, MO, and flash memory. The external storage device may also be a server device connected via a network.

[0089] The input unit 1040 is composed of a keyboard, a touch panel, various buttons, etc., and accepts input from the user. The display unit 1050 is composed of a liquid crystal display, etc., and can display various data, information processing results, etc. to the user. In addition, the present device can communicate with an HMD or other devices via the communication unit 1060. The present device may receive user instructions from other devices via the communication unit 1060, or may output processing results to other devices. The information processing device of this embodiment can be realized by a general-purpose personal computer, tablet terminal, smartphone, etc., having the above-mentioned configuration.

[0090] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions. The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0091] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a recognition means for recognizing a predetermined subject area from the captured image; an extraction means for extracting color information from the subject area recognized by the recognition means; a registration means for registering the color information extracted by the extraction means as a registered color; a detection means for detecting a predetermined event; and an information processing device characterized in that the registration means registers, as the registered color, color information extracted by the extraction means from the subject area recognized by the recognition means when the detection means detects the specified event, or color information extracted by the extraction means from the subject area obtained by recognition performed by the recognition means in response to the detection of the specified event by the detection means. (Configuration 2) the detection means detects, as the predetermined event, that the subject area has been recognized by the recognition means within a predetermined range within the image; 2. The information processing device according to configuration 1, wherein the extraction means extracts the color information from the subject area recognized by the recognition means when the predetermined event is detected. (Configuration 3) the detecting means detects, as the predetermined event, that a difference between the color information extracted by the extracting means and the registered color is equal to or greater than a predetermined threshold; The information processing device according to configuration 1, wherein the registration means registers, as the registered color, color information extracted by the extraction means from the subject area recognized by the recognition means when the predetermined event is detected. (Configuration 4) the detecting means detects, as the predetermined event, that the environmental fluctuation is equal to or greater than a predetermined threshold; 2. The information processing apparatus according to configuration 1, wherein the recognition means recognizes the predetermined subject area from the image in response to the detection of the predetermined event. (Configuration 5) the detection means detects, as the predetermined event, that the number of pixels of a non-registered color that is not the registered color or the area of ​​a non-registered color is equal to or greater than a predetermined threshold value in a subject area detected from the captured image based on the registered color; 2. The information processing apparatus according to configuration 1, wherein the recognition means recognizes the predetermined subject area from the image in response to the detection of the predetermined event. (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, which is connected to a head-mounted display or includes a head-mounted display. (Configuration 7) the detection means detects that the user is wearing the head mounted display as the predetermined event; The information processing device according to configuration 6, wherein the registration means registers, in response to the detection of the predetermined event, the color information extracted by the extraction means from the subject area recognized by the recognition means as the registered color. (Configuration 8) the registration means registers the registered color in association with each user of the head mounted display, the detection means detects, as the predetermined event, a result of identifying a user associated with the registered color; 8. The information processing device according to configuration 6 or 7, wherein the recognition means recognizes the predetermined subject area from the image in response to the detection of the predetermined event. (Configuration 9) 9. The information processing device according to configuration 8, wherein the detection means detects, as the predetermined event, that a user associated with the registered color cannot be identified. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, wherein the recognition means recognizes a region of a human hand as the predetermined subject region from the captured image. (Configuration 11) a generating means for generating a binarized map that expresses each pixel of the captured image as a binary color, either a registered color or a non-registered color, based on the registered color information; a distance calculation means for calculating a distance from an imaging device that captured the image for each pixel of the image by using the binarized map generated by the generation means as an effective area for distance calculation; a synthesis means for synthesizing the captured image and a CG image based on the distance for each pixel calculated by the distance calculation means; and 11. The information processing apparatus according to any one of configurations 1 to 10, wherein the image synthesized by the synthesis means is displayed on a predetermined display device. (Method 1) a recognition step of recognizing a predetermined subject area from a captured image; an extraction step of extracting color information from the subject area recognized by the recognition step; a registration step of registering the color information extracted in the extraction step as a registered color; a detection step of detecting a predetermined event; and an information processing method characterized in that the color information extracted in the extraction step from the subject area recognized in the recognition step when the specified event is detected in the detection step, or the color information extracted in the extraction step from the subject area obtained by recognition performed in the recognition step in response to the detection of the specified event by the detection step, is registered as the registered color in the registration step. (Program 1) 12. A program for causing a computer to function as the information processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0092] 100: Image receiving unit, 101: Shape recognition unit, 102: Registered color designation unit, 103: Registered color storage unit, 104: Registered color detection unit, 105: Registered color transmission unit, 106: Distance calculation unit, 106: Synthesis unit, 108: CG generation unit, 109: Projection unit

Claims

1. a recognition means for recognizing a predetermined subject area from the captured image; an extraction means for extracting color information from the subject area recognized by the recognition means; a registration means for registering the color information extracted by the extraction means as a registered color; a detection means for detecting a predetermined event; and an information processing device characterized in that the registration means registers, as the registered color, color information extracted by the extraction means from the subject area recognized by the recognition means when the detection means detects the specified event, or color information extracted by the extraction means from the subject area obtained by recognition performed by the recognition means in response to the detection of the specified event by the detection means.

2. the detection means detects, as the predetermined event, that the subject area has been recognized by the recognition means within a predetermined range within the image; 2. The information processing apparatus according to claim 1, wherein the extraction means extracts the color information from the subject area recognized by the recognition means when the predetermined event is detected.

3. the detecting means detects, as the predetermined event, that a difference between the color information extracted by the extracting means and the registered color is equal to or greater than a predetermined threshold; 2. The information processing device according to claim 1, wherein the registration means registers, as the registered color, color information extracted by the extraction means from the subject area recognized by the recognition means when the predetermined event is detected.

4. the detecting means detects, as the predetermined event, that the environmental fluctuation is equal to or greater than a predetermined threshold; 2. The information processing apparatus according to claim 1, wherein the recognition means recognizes the predetermined subject area from the image in response to the detection of the predetermined event.

5. the detection means detects, as the predetermined event, that the number of pixels of a non-registered color that is not the registered color or the area of ​​a non-registered color is equal to or greater than a predetermined threshold value in a subject area detected from the captured image based on the registered color; 2. The information processing apparatus according to claim 1, wherein the recognition means recognizes the predetermined subject area from the image in response to the detection of the predetermined event.

6. 2. The information processing apparatus according to claim 1, wherein the information processing apparatus is connected to a head-mounted display or includes a head-mounted display.

7. the detection means detects that the user is wearing the head mounted display as the predetermined event; 7. The information processing apparatus according to claim 6, wherein the registration means registers, in response to the detection of the predetermined event, the color information extracted by the extraction means from the subject area recognized by the recognition means as the registered color.

8. the registration means registers the registered color in association with each user of the head mounted display, the detection means detects, as the predetermined event, a result of identifying a user associated with the registered color; 7. The information processing apparatus according to claim 6, wherein the recognition means recognizes the predetermined subject area from the image in response to the detection of the predetermined event.

9. 9. The information processing apparatus according to claim 8, wherein the detecting means detects, as the predetermined event, that a user associated with the registered color cannot be identified.

10. 10. The information processing apparatus according to claim 1, wherein the recognition means recognizes a region of a human hand as the predetermined subject region from the captured image.

11. a generating means for generating a binarization map that expresses each pixel of the captured image as a binary color, either a registered color or a non-registered color, based on the registered color information; a distance calculation means for calculating a distance from an imaging device that captured the image for each pixel of the image by using the binarized map generated by the generation means as an effective area for distance calculation; a synthesis means for synthesizing the captured image and a CG image based on the distance for each pixel calculated by the distance calculation means; and 2. The information processing apparatus according to claim 1, wherein the image synthesized by said synthesis means is displayed on a predetermined display device.

12. a recognition step of recognizing a predetermined subject area from a captured image; an extraction step of extracting color information from the subject area recognized by the recognition step; a registration step of registering the color information extracted in the extraction step as a registered color; a detection step of detecting a predetermined event; and an information processing method characterized in that the color information extracted in the extraction step from the subject area recognized in the recognition step when the specified event is detected in the detection step, or the color information extracted in the extraction step from the subject area obtained by recognition performed in the recognition step in response to the detection of the specified event by the detection step, is registered as the registered color in the registration step.

13. Computer, a recognition means for recognizing a predetermined subject area from the captured image; an extraction means for extracting color information from the subject area recognized by the recognition means; a registration means for registering the color information extracted by the extraction means as a registered color; a detection means for detecting a predetermined event; and A program that causes an information processing device to register, as the registered color, color information extracted by the extraction means from the subject area recognized by the recognition means when the detection means detects the specified event, or color information extracted by the extraction means from the subject area obtained by recognition performed by the recognition means in response to the detection of the specified event by the detection means.

Images