Information processing device, information processing system, information processing method, and recording medium

By dynamically determining the chroma key color based on real and CG image colors, the system effectively addresses the challenges of data compression and accurate superimposition in HMDs, ensuring seamless and high-quality CG image overlay on real images.

WO2024232242A1PCT designated stage expired Publication Date: 2024-11-14SONY GROUP CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/015583
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-04-19
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing display devices, such as HMDs, face challenges in accurately superimposing high-definition CG images onto real images due to data compression issues, where the chroma key color may change, causing transparency errors and potential visibility of the real image through CG areas, especially when the chroma key color is fixed or similar to the CG area's color.

Method used

The system determines a chroma key color dynamically based on the real image and CG image colors, using a color close to the real image and different from the CG image to ensure accurate separation and compression of the transparent area, preventing color changes and incorrect determinations during the superimposition process.

Benefits of technology

This approach allows for seamless and accurate superimposition of CG images onto real images without noticeable color changes in the transparent areas, preventing the real image from showing through and ensuring high-definition image quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024015583_14112024_PF_FP_ABST
    Figure JP2024015583_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present technology relates to an information processing device, an information processing system, an information processing method, and a recording medium that make it possible to suitably overlay a CG image. An information processing device according to the present technology comprises a chroma key color determination unit that, on the basis of a photographic image that is an image of a real space and a CG image that includes an object overlaid on the photographic image, determines a chroma key color for drawing a region other than the object within the CG image. The present technology can be applied to, for example, an information processing system having a VST function.
Need to check novelty before this filing date? Find Prior Art

Description

Information processing device, information processing system, information processing method, and recording medium

[0001] The present technology relates to an information processing device, an information processing system, an information processing method, and a recording medium, and more particularly to an information processing device, an information processing system, an information processing method, and a recording medium that are capable of suitably superimposing a CG image.

[0002] Some display devices, such as HMDs (Head Mounted Displays), have a VST (Video See-Through) function that superimposes virtual objects drawn with CG (Computer Graphics) on a portion of a live-action image of the external world (real space) of the display device. By viewing the image displayed on the display device, users can experience an XR (Extended Reality) space where real space and virtual space are fused together.

[0003] Generally, display devices such as HMDs support remote rendering, in which a device other than the display device renders a CG image including a virtual object superimposed on a live-action image. In remote rendering, a device other than the display device, such as a PC, renders the CG image, and the display device receives the CG image transmitted from the PC and displays an image obtained by combining the received CG image with the live-action image (see, for example, Patent Document 1).

[0004] Japanese Patent Application Laid-Open No. 2015-99562

[0005] A CG image is composed of, for example, a CG area including a virtual object superimposed on a live-action image, and a transparent area that is not superimposed on the live-action image and is an area other than the virtual object. The transparent area is drawn, for example, in a predetermined chromakey color. The display device can generate an image including CG and real objects by superimposing the CG image with the chromakey color area made transparent on the live-action image.

[0006] When CG images are rendered in high resolution, the amount of data required for the CG image becomes enormous, so the PC compresses the CG image before sending it to the display device. Compressing the CG image can cause the color of part of the transparent area in the CG image to change from the original chromakey color, and that part of the transparent area may not be transparent and may be superimposed on the live-action image.

[0007] Furthermore, since the chromakey color is determined in advance, if the CG area includes a portion of the same color as the chromakey color, the live-action image may be visible through that portion of the CG area.

[0008] The present technology has been made in consideration of such circumstances, and makes it possible to suitably superimpose CG images.

[0009] An information processing device according to a first aspect of the present technology includes a chromakey color determination unit that determines, based on a live-action image that is an image of real space and a CG image that includes an object superimposed on the live-action image, a chromakey color for drawing an area in the CG image other than the object.

[0010] In an information processing method according to a first aspect of the present technology, an information processing device determines a chromakey color for drawing an area other than an object in a CG image based on a live-action image, which is an image of real space, and a CG image including an object superimposed on the live-action image.

[0011] A recording medium according to a first aspect of the present technology records a program for executing a process for determining a chromakey color for drawing an area in a CG image other than an object, based on a live-action image, which is an image of real space, and a CG image including an object superimposed on the live-action image.

[0012] An information processing system according to a second aspect of the present technology includes an information processing device including: a live-action image, which is an image of real space, and a CG image including an object to be superimposed on the live-action image, and a chromakey color determination unit that determines a chromakey color for drawing an area other than the object in the CG image, based on the live-action image; and a drawing unit that draws the object in the CG image and draws the area other than the object in the CG image in the chromakey color; and a device that includes: an imaging unit that captures the live-action image; and a compositing unit that superimposes the CG image, with the area of ​​the chromakey color determined by the chromakey color determination unit made transparent, onto the live-action image.

[0013] In a first aspect of the present technology, a chromakey color for depicting an area other than the object in the CG image is determined based on a live-action image, which is an image of real space, and a CG image including an object superimposed on the live-action image.

[0014] In a second aspect of the present technology, an information processing device determines a chromakey color for drawing an area other than the object in a CG image based on a live-action image, which is an image of real space, and a CG image including an object to be superimposed on the live-action image, draws the object in the CG image, and draws the area other than the object in the CG image in the chromakey color. Also, a device captures the live-action image, and superimposes the CG image, with the area of ​​the chromakey color made transparent, on the live-action image.

[0015] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present technology. FIG. 2 is a diagram illustrating an example of a VST function. FIG. 3 is a diagram illustrating an example of a composite image generated when a CG area and a transparent area are distinguished based on a chromakey color. FIG. 4 is a first diagram illustrating an example of a method for determining a chromakey color according to the present technology. FIG. 5 is a second diagram illustrating an example of a method for determining a chromakey color according to the present technology. FIG. 6 is a block diagram illustrating example configurations of an HMD and an information processing device. FIG. 7 is a diagram illustrating an example of color information of a real-life image. FIG. 8 is a flowchart illustrating processing performed by an information processing system. FIG. 9 is a block diagram illustrating another example configuration of an HMD and an information processing device. FIG. 10 is a block diagram illustrating yet another example configuration of an HMD and an information processing device. FIG. 11 is a block diagram illustrating an example configuration of computer hardware.

[0016] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the present technology 2. Configuration and operation of each device 3. Modification

[0017] 1. Overview of the Present Technology FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present technology.

[0018] 1 is configured by connecting an HMD 1 and an information processing device 2 via a network 3 such as a LAN (Local Area Network) or the Internet. The HMD 1 and the information processing device 2 may also be connected by wire.

[0019] The HMD 1 is configured as a display device with a VST function, for example, and includes a camera and a non-transparent display. When wearing the HMD 1, the user's field of view is blocked by the display and the housing, so the user cannot directly see the outside world (real space). However, an image of the outside world captured by the camera is displayed on the display, allowing the user to see the outside world while wearing the HMD 1.

[0020] The field of view (image capture range) of the camera of the HMD 1 corresponds to the field of view (visible range) of the user when not wearing the HMD 1, and moves when the user turns their head, for example.

[0021] FIG. 2 is a diagram illustrating an example of the VST function.

[0022] 2, the VST function combines a real-life image P1, which is an image of the external world of the HMD 1, with a CG image P2, which is an image generated using virtual objects. The CG image P2 is composed of, for example, a CG area including virtual objects superimposed on the real-life image P1, and a transparent area A1, which is an area other than the virtual objects and is not superimposed on the real-life image P1.

[0023] The composite image P11, which is a combination of the real-life image P1 and the CG image P2, is an image in which the CG image P2, which has been made transparent through the transparent area A1, is superimposed on the real-life image P1, as shown in the lower part of Figure 2. In other words, the part of the real-life image P1 that corresponds to the transparent area A1 is placed as the background in the transparent area A1 in the CG image P2.

[0024] A composite image P11 is displayed on the display of the HMD 1. By viewing the composite image P11 displayed on the display of the HMD 1, the user can experience an XR space in which real space and virtual space are combined.

[0025] 1 , the information processing device 2 generates a CG image including a virtual object to be superimposed on a real-life image, and transmits the CG image to the HMD 1, thereby displaying the CG image on the HMD 1. The information processing device 2 is configured by, for example, a PC.

[0026] Generally, display devices such as HMDs support remote rendering, in which CG images are rendered by a device other than the display device. Some standalone HMDs equipped with mobile SoCs (System on a Chip) or batteries also support remote rendering as a solution to the following two problems:

[0027] ・HMDs are not suitable for rendering high-resolution CG images due to the power consumption and resource requirements for rendering. ・Existing PC applications for HMDs cannot be executed on HMDs.

[0028] Generally, in an HMD, an image is displayed on the display after going through two stages: the stage where the application draws it according to its logic, and the stage where a part called a compositor adjusts the drawing results from the application.

[0029] The process of superimposing virtual objects onto live-action images is often performed by a compositor. As mentioned above, a CG image, which corresponds to the rendering result of an application, is composed of CG areas and transparent areas. The compositor superimposes the virtual object onto the live-action image while referencing information in the CG image that distinguishes between CG areas and transparent areas.

[0030] Examples of information for distinguishing between CG areas and transparent areas within a CG image include an alpha channel that indicates the transparency of each pixel in the CG image, and a predetermined chromakey color. When CG areas and transparent areas are distinguished based on the alpha channel, for example, the maximum alpha value is stored in the pixels of the CG area, and the minimum alpha value is stored in the pixels of the transparent area. When CG areas and transparent areas are distinguished based on the chromakey color, for example, the transparent area is drawn in the chromakey color.

[0031] When CG images are rendered in high resolution, the amount of data required for the CG image becomes enormous. Therefore, when remote rendering is performed, the device that renders the CG image compresses the CG image before transmitting it to the display device. From the perspective of performance and versatility, CC images are compressed using formats such as H.264 and H.265.

[0032] Without using an extension for a specific platform, it is difficult to transmit four channels, including RGB channels and an alpha channel. Therefore, when CG areas and transparent areas are distinguished based on the alpha channel, the CG image is transmitted in two separate streams, one for the RGB channels and one for the alpha channel. When a CG image is transmitted in two separate streams, the resolution of the RGB channels is reduced, making it impossible to maintain the original high-resolution CG image.

[0033] Therefore, when prioritizing the maintenance of the detail of the CG image, the CG area and the transparent area are distinguished based on the chromakey color. When the CG area and the transparent area are distinguished based on the chromakey color, the information for distinguishing between the CG area and the transparent area is embedded in the RGB channel as the chromakey color, which can cause the following two problems in the composite image.

[0034] FIG. 3 is a diagram showing an example of a composite image generated when the CG area and the transparent area are distinguished based on the chromakey color.

[0035] The first problem is that compressing a CG image can cause the color of a portion of the transparent area in the CG image to change from the original chromakey color. In this case, as shown in gray in Figure 3A, the portion of the transparent area where the chromakey color has changed is not transparent and is superimposed on the actual image, and the area with the changed chromakey color is included in the composite image P11.

[0036] The second problem is that because the chromakey color is predetermined, if the CG area contains an area that is the same color as the chromakey color, the live-action image may be visible through that area, as shown in Figure 3B.

[0037] For example, Patent Document 1 describes a technology aimed at solving the first problem, which targets the area around the boundary between the CG area and the transparent area and extends the chromakey color, thereby enabling areas of the chromakey color that have changed due to compression of the CG image to be detected as transparent areas.

[0038] The technology described in Patent Document 1 aims to accurately separate CG areas from transparent areas when the chromakey color is fixed, so it does not solve the second problem. Because the chromakey color is extended, if a color close to the chromakey color is included in the edge portion of the CG area, there is a high possibility that the edge portion will be erroneously determined as a transparent area.

[0039] Therefore, the information processing system of the present technology determines a chromakey color for rendering a transparent area in a CG image based on a real-life image and a CG image.

[0040] FIG. 4 is a first diagram illustrating an example of a method for determining a chromakey color according to the present technology.

[0041] Regarding the first problem, if the chromakey color is different from the color of both the live-action image and the CG image, the area of ​​the chromakey color included in the composite image (part of the transparent area that is superimposed without being transparent) is easily visible.

[0042] Therefore, in the information processing system of the present technology, a color close to the color of the actual image is used as the chromakey color, as shown in FIG.

[0043] Specifically, as shown in A of Fig. 4, if the color of the entire live-action image P21 is, for example, a light color, a color similar to the color of the entire live-action image P21, such as a light color, is used as the chromakey color of the transparent area A11 in the CG image P31. Also, as shown in B of Fig. 4, if the color of the entire live-action image P22 is, for example, a dark color, a color similar to the color of the entire live-action image P22, such as a dark color, is used as the chromakey color of the transparent area A12 in the CG image P32.

[0044] FIG. 5 is a second diagram illustrating an example of a method for determining a chromakey color according to the present technology.

[0045] Regarding the second problem, when the chromakey color is close to the color of the CG area of ​​the CG image, it is more likely that a part of the CG area will be erroneously determined as a transparent area.

[0046] Therefore, in the information processing system of the present technology, as shown in FIG. 5, a color that is distant from the color of the CG area of ​​the CG image is used as the chromakey color.

[0047] Specifically, as shown in A of Fig. 5, if the color of the entire CG area (area other than transparent area A21) of CG image P41 is, for example, a light color, a color that is different from the color of the entire CG area, such as a dark color, is used as the chromakey color of transparent area A21. Also, as shown in B of Fig. 5, if the color of the entire CG area (area other than transparent area A22) of CG image P42 is, for example, a dark color, a color that is different from the color of the entire CG area, such as a light color, is used as the chromakey color of transparent area A22.

[0048] As described above, in the information processing system of the present technology, the chromakey color used to render transparent areas of a CG image is dynamically set based on the colors of the live-action image and the CG image. By using a color as the chromakey color that is close to the color of the live-action image but far from the color of the CG area in the CG image, the information processing system of the present technology can superimpose the changed chromakey color area on the live-action image without making it stand out, even if the color of the transparent area changes from the chromakey color due to compression of the CG image. Furthermore, the information processing system of the present technology can avoid including colors close to the chromakey color in the CG area, thereby preventing parts of the CG area from being erroneously determined to be transparent.

[0049] 2. Configuration and Operation of Each Device FIG. 6 is a block diagram showing an example of the configuration of each of the HMD 1 and the information processing device 2.

[0050] As shown in Figure 6, the HMD 1 is composed of a mono camera 11, an IMU 12, a self-position estimation unit 13, an RGB camera 14, a live image analysis unit 15, a live image delay compensation unit 16, a CG image delay compensation unit 17, a synthesis unit 18, a lens distortion correction unit 19, and a display unit 20.

[0051] The mono camera 11 captures an image of the outside world of the HMD 1 and supplies the captured image to the self-position estimation unit 13 .

[0052] The IMU (Inertial Measurement Unit) 12 is an inertial measurement device that integrates an acceleration sensor, an angular velocity sensor, etc. The IMU 12 supplies the measurement results of the acceleration, angular velocity, etc. to the self-position estimation unit 13.

[0053] The self-position estimation unit 13 estimates the self-position (position and orientation) of the HMD 1 based on the captured image captured by the mono camera 11 and the measurement results by the IMU 12. Note that the self-position of the HMD 1 may be estimated based not only on the captured image captured by the mono camera 11 and the measurement results by the IMU 12, but also on the captured image captured by a stereo camera and sensor data from various sensors such as a ToF (Time-of-Flight) sensor.

[0054] The self-position estimation unit 13 transmits self-position information indicating the self-position of the HMD 1 to the information processing device 2. The self-position estimation unit 13 also supplies the self-position information to the live-action image delay compensation unit 16 and the CG image delay compensation unit 17 via paths not shown.

[0055] The RGB camera 14 captures an image of the outside world of the HMD 1 and supplies the captured image to the captured image analysis unit 15 and the captured image delay compensation unit 16 .

[0056] The real-life image analysis unit 15 analyzes the color distribution in the entire real-life image supplied from the RGB camera 14, and acquires color information indicating the colors of the real-life image.

[0057] FIG. 7 is a diagram showing an example of color information of a real image.

[0058] The live-action image analysis unit 15 analyzes the live-action image and obtains a histogram of pixel values ​​expressed in, for example, the HSV color space, as shown in Fig. 7. In the example of Fig. 7, the solid line indicates the histogram of the hue component, the dashed line indicates the histogram of the saturation component, and the dashed-dotted line indicates the histogram of the lightness component. In Fig. 7, the horizontal axis indicates the value of each component, and the vertical axis indicates the frequency.

[0059] 6 , the real-life image analysis unit 15 transmits color information indicating the distribution of colors in the entire real-life image to the information processing device 2. Note that instead of analyzing the distribution of colors in the entire real-life image, it is also possible to analyze the distribution of colors in a boundary region including, among the multiple pixels constituting the real-life image, pixels that form the boundary between the real-life image and the CG image when the CG image is superimposed, and the surrounding pixels.

[0060] The live-action image delay compensation unit 16 predicts the self-position of the HMD 1 at the timing when the live-action image is displayed on the display unit 20, for example, based on the time series of self-position information supplied from the self-position estimation unit 13. The live-action image delay compensation unit 16 corrects the live-action image supplied from the RGB camera 14 to match the self-position of the HMD 1 at the timing when the live-action image is displayed on the display unit 20. The live-action image delay compensation unit 16 supplies the corrected live-action image to the synthesis unit 18.

[0061] The CG image delay compensation unit 17 predicts the self-position of the HMD 1 at the timing when the CG image is displayed on the display unit 20, for example, based on the time series of self-position information supplied from the self-position estimation unit 13. The CG image delay compensation unit 17 receives the CG image transmitted from the information processing device 2, and corrects the CG image to match the self-position of the HMD 1 at the timing when the CG image is displayed on the display unit 20. The CG image delay compensation unit 17 supplies the corrected CG image to the synthesis unit 18.

[0062] The composition unit 18 receives the chromakey information transmitted from the information processing device 2. The chromakey information is information indicating the chromakey color determined by the information processing device 2.

[0063] The composition unit 18 generates a composite image by combining the live-action image supplied from the live-action image delay compensation unit 16 and the CG image supplied from the CG image delay compensation unit 17 based on the chromakey information. Specifically, the composition unit 18 superimposes the CG image, which has a transparent chromakey color area, on the live-action image. The composition unit 18 supplies the generated composite image to the lens distortion correction unit 19.

[0064] The lens distortion correction unit 19 performs a correction process on the composite image supplied from the synthesis unit 18 to cancel out distortion caused by a lens placed between the display unit 20 and the user's eyes. The lens distortion correction unit 19 supplies the corrected composite image to the display unit 20.

[0065] The display unit 20 is configured by, for example, a display, and displays the composite image supplied from the lens distortion correction unit 19.

[0066] The information processing device 2 includes a chromakey color determination unit 31 , a drawing unit 32 , and a CG image analysis unit 33 .

[0067] The chromakey color determination unit 31 receives color information of the live-action image transmitted from the live-action image analysis unit 15 of the HMD 1. Here, the chromakey color determination unit 31 also functions as an acquisition unit that acquires color information of the live-action image from the HMD 1. Chromakey Color Determination Unit The chromakey color determination unit 31 determines a chromakey color based on the color information of the live-action image and the color information of the CG image supplied from the CG image analysis unit 33. The chromakey color determination unit 31 supplies chromakey information indicating the determined chromakey color to the drawing unit 32 and also transmits it to the composition unit 18 of the HMD 1.

[0068] The drawing unit 32 is made up of a chromakey color update unit 41 , an RGB drawing unit 42 , and an alpha drawing unit 43 .

[0069] The chromakey color update unit 41 controls the RGB drawing unit 42 so that the transparent area (background) is drawn in the chromakey color determined by the chromakey color determination unit 31 .

[0070] The RGB drawing unit 42 renders a CG image by drawing a CG area (virtual object) according to the self-position of the HMD 1 estimated by the self-position estimation unit 13 of the HMD 1 and drawing a transparent area in a chromakey color. The RGB drawing unit 42 supplies the rendered CG image to the CG image analysis unit 33 and also transmits it to the CG image delay compensation unit 17 of the HMD 1.

[0071] The alpha drawing unit 43 draws the alpha channel of the CG image and supplies it to the CG image analysis unit 33 .

[0072] The CG image analysis unit 33 analyzes the color distribution in the CG area within the CG image supplied from the RGB drawing unit 42, and obtains color information indicating the color of the CG image.

[0073] Specifically, the CG image analysis unit 33 first extracts a CG area from the CG image based on the alpha channel supplied from the alpha rendering unit 43. While extracting the CG area based on the alpha channel provides higher accuracy, it is also possible to extract the CG area from the CG image based on the chromakey color in order to reduce memory usage. Next, the CG image analysis unit 33 analyzes the CG area and obtains a histogram of pixel values ​​expressed in, for example, the HSV color space, in the same way as when analyzing a live-action image.

[0074] The CG image analysis unit 33 supplies color information indicating the distribution of colors in the CG area to the chromakey color determination unit 31 .

[0075] Next, the processing performed by the information processing system having the above configuration will be described with reference to the flowcharts of FIGS.

[0076] In step S1, the HMD 1 performs initialization of the XR service in general, such as initializing the camera and the recognition module.

[0077] In step S2, the self-position estimation unit 13 of the HMD 1 estimates the self-position of the HMD 1. Note that if the self-position of the HMD 1 is barely moving (if there is little change between frames in the real image and the CG image), the processes of steps S4 and S6 to S8 may be skipped.

[0078] In step S3, the RGB camera 14 of the HMD 1 captures a real image.

[0079] In step S4, the real-life image analysis unit 15 of the HMD 1 analyzes the real-life image and acquires color information of the real-life image.

[0080] In step S5, the HMD 1 transmits its own position information and color information of the real image to the information processing device 2, and the information processing device 2 receives the self-position information and color information of the real image transmitted from the HMD 1.

[0081] In step S6, the CG image analysis unit 33 of the information processing device 2 extracts a CG area from the CG image of the frame (previous frame) that is immediately before the frame (current frame) rendered by the rendering unit 32.

[0082] In step S7, the CG image analysis unit 33 analyzes the CG area of ​​the CG image of the previous frame and acquires color information of the CG image.

[0083] In step S8, the chromakey color determination unit 31 of the information processing device 2 determines a chromakey color based on the color information of the real image (first color information) and the color information of the CG image (second color information).

[0084] In step S9, the RGB drawing unit 42 of the information processing device 2 renders a CG image based on the self-position information.

[0085] In step S10, the information processing device 2 performs compression processing on the CG image for transmission over a network. Here, the CG image is compressed using a common format such as H.265. Note that before compression using a format such as H.265, the CG image may be subjected to spatial compression that takes into account lens distortion and the user's gaze area.

[0086] In step S11 , the information processing device 2 transmits the CG image and chromakey information to the HMD 1 .

[0087] In step S12, the HMD 1 receives the CG image and chromakey information transmitted from the information processing device 2, expands the compressed CG image, and restores a CG image equivalent to the CG image rendered by the information processing device 2.

[0088] In step S13 , the CG image delay compensation unit 17 of the HMD 1 corrects the CG image in accordance with the position of the HMD 1 at the time when the CG image is displayed on the display unit 20 .

[0089] In step S14, the live-action image delay compensation unit 16 of the HMD 1 performs correction on the live-action image to match the self-position of the HMD 1 at the timing when the live-action image is displayed on the display unit 20. Here, the correction to match the self-position of the HMD 1 is performed not on the live-action image (frame) captured in step S3 and analyzed in step S4, but on, for example, the live-action image (frame) captured immediately before the correction is performed.

[0090] In step S15, the composition unit 18 of the HMD 1 generates a composite image by combining the live image and the CG image based on the chromakey information. Note that, as a measure to prevent changes in the chromakey color due to compression of the CG image, the chromakey color may be expanded in advance (colors close to the chromakey color may also be determined as the chromakey color), or a conventional technique such as the technique described in Patent Document 1 may be used.

[0091] In step S16, the lens distortion correction unit 19 performs a correction process on the composite image to cancel out distortion caused by the lens placed between the display unit 20 and the user's eyes.

[0092] In step S17, the display unit 20 displays the composite image.

[0093] In step S18, the HMD 1 determines whether or not to terminate the app (application) that provides the XR service.

[0094] If it is determined in step S18 that the application should not be terminated, the process returns to step S2, and the subsequent processes are carried out. On the other hand, if it is determined in step S18 that the application should be terminated, the process ends.

[0095] Through the above processing, the information processing system of the present technology can superimpose a CG image onto a live-action image in an appropriate manner, even if the color of the transparent area changes from the chromakey color due to compression of the CG image, by superimposing the changed chromakey color area onto the live-action image without making the area with the changed chromakey color stand out, and by avoiding the inclusion of colors close to the chromakey color in the CG area.

[0096] 10 is a block diagram showing another example of the configuration of the HMD 1 and the information processing device 2. In Fig. 10, the same components as those in Fig. 6 are denoted by the same reference numerals. Duplicate descriptions will be omitted as appropriate.

[0097] The HMD 1 in Fig. 10 differs from the HMD 1 in Fig. 6 in that it does not include a real-image analysis unit 15. The information processing device 2 in Fig. 10 also differs from the information processing device 2 in Fig. 6 in that it includes a real-image analysis unit 101.

[0098] The HMD 1 transmits the actual image captured by the RGB camera 14 to the information processing device 2 .

[0099] The live-action image analysis unit 101 receives the live-action image transmitted from the HMD 1. Similar to the live-action image analysis unit 15 in Fig. 6, the live-action image analysis unit 101 analyzes the color distribution in the entire live-action image and supplies color information of the live-action image to the chromakey color determination unit 31. Here, the live-action image analysis unit 101 functions as an acquisition unit that acquires the live-action image from the HMD 1 and analyzes the live-action image to acquire color information.

[0100] If the communication environment in the network 3 is good (for example, the transfer bit rate is high) and the information processing device 2 has more resources than the HMD 1, analysis of the live image may be performed in the information processing device 2, as shown in Figure 10.

[0101] It is possible to simultaneously analyze live-action images, analyze CG images, and determine chromakey colors using a learning model trained using deep learning or other machine learning techniques. These processes use a learning model that takes live-action images and CG images as inputs and outputs chromakey information.

[0102] Fig. 11 is a block diagram showing yet another example of the configuration of the HMD 1 and the information processing device 2. In Fig. 11, the same components as those in Fig. 6 are denoted by the same reference numerals. Duplicate descriptions will be omitted where appropriate.

[0103] The HMD 1 in Fig. 11 is similar to the HMD 1 in Fig. 6. The information processing device 2 in Fig. 11 differs from the information processing device 2 in Fig. 6 in that a CG scene analysis unit 111 is provided instead of the CG image analysis unit 33.

[0104] The CG scene analysis unit 111 analyzes a virtual space (scene) used for rendering a CG image. For example, a large number of virtual objects are arranged in the virtual space. The CG scene analysis unit 111 analyzes a scene graph of the objects arranged in the virtual space and acquires information indicating the color distribution in the virtual space as color information of the CG image. The CG image analysis unit 33 supplies the color information of the CG image to the chromakey color determination unit 31.

[0105] Since the analysis is performed on the virtual space before rendering, rather than on the CG image after rendering, it is possible to determine the chromakey color based on the results of the analysis of the virtual space by analyzing the virtual space only once, without having to perform an analysis each time a CG image is rendered.

[0106] Regarding the computer, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0107] 12 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program. The information processing device 2 is, for example, a PC having a configuration similar to that shown in FIG.

[0108] A CPU (Central Processing Unit) 501 , a ROM (Read Only Memory) 502 , and a RAM (Random Access Memory) 503 are interconnected by a bus 504 .

[0109] An input / output interface 505 is also connected to the bus 504. An input unit 506 including a keyboard, a mouse, etc., and an output unit 507 including a display, a speaker, etc. are connected to the input / output interface 505. Also connected to the input / output interface 505 are a storage unit 508 including a hard disk, a nonvolatile memory, etc., a communication unit 509 including a network interface, etc., and a drive 510 that drives removable media 511.

[0110] In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executing it.

[0111] The program executed by the CPU 501 is installed in the storage unit 508 by being recorded on, for example, a removable medium 511 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0112] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0113] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0114] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0115] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0116] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0117] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0118] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0119] Example of configuration combinations The present technology can also be configured as follows.

[0120] (1) An information processing device comprising: a chromakey color determination unit that determines a chromakey color for drawing an area in a CG image other than an object, based on a live-action image, which is an image of real space, and a CG image including an object to be superimposed on the live-action image. (2) The information processing device described in (1), in which the chromakey color determination unit determines the chromakey color based on first color information indicating the color of the live-action image and second color information indicating the color of the CG image. (3) The information processing device described in (2), in which the chromakey color determination unit determines a color that is close to the color of the live-action image and far from the color of the CG image as the chromakey color. (4) The information processing device described in (2) or (3), in which the first color information is information indicating the distribution of colors in the entire live-action image. (5) The information processing device described in (2) or (3), in which the first color information is information indicating the distribution of colors in a boundary area including pixels that form a boundary between the object and the live-action image, among a plurality of pixels that constitute the live-action image. (6) The information processing device according to any one of (2) to (5), further comprising an acquisition unit that acquires the first color information. (7) The information processing device according to (6), wherein the acquisition unit acquires the first color information from an external device that captures the real-life image. (8) The information processing device according to (6), wherein the acquisition unit acquires the real-life image from an external device that captures the real-life image and analyzes the real-life image to acquire the first color information. (9) The information processing device according to any one of (2) to (8), wherein the second color information is information indicating a color distribution in a CG area that is a region of the object in the CG image. (10) The information processing device according to (9), further comprising an image analysis unit that extracts the CG area from the CG image and analyzes the CG area to acquire the second color information. (11) The information processing device according to (10), wherein the image analysis unit extracts the CG area based on an alpha channel of the CG image. (12) The information processing device according to (10), wherein the image analysis unit extracts the CG area based on the chromakey color.(13) The information processing device according to any one of (2) to (12), wherein the first color information and the second color information are histograms of pixel values ​​expressed in an HSV color space. (14) The information processing device according to any one of (2) to (8), wherein the second color information is information used for rendering the CG image and indicates a color distribution in a virtual space in which the object is placed. (15) The information processing device according to any one of (1) to (14), further comprising a drawing unit that captures the real-life image and draws the object in the CG image based on a self-position of an external device that displays the real-life image on which the object is superimposed, and draws an area in the CG image other than the object in the chromakey color. (16) An information processing system comprising: an information processing device comprising: a chromakey color determination unit that determines, based on a live-action image that is an image of real space and a CG image including an object to be superimposed on the live-action image, a chromakey color for drawing an area other than the object in the CG image, a drawing unit that draws the object in the CG image and draws an area other than the object in the CG image in the chromakey color, and a device that comprises: an imaging unit that captures the live-action image, and a compositing unit that superimposes the CG image, making the area of ​​the chromakey color determined by the chromakey color determination unit transparent, onto the live-action image. (17) An information processing method in which an information processing device determines, based on the live-action image that is an image of real space and a CG image including an object to be superimposed on the live-action image, a chromakey color for drawing an area other than the object in the CG image. (18) A computer-readable recording medium having recorded thereon a program for executing a process of determining a chromakey color for drawing an area other than an object in a CG image based on a live-action image, which is an image of real space, and a CG image including an object superimposed on the live-action image.

[0121] DESCRIPTION OF SYMBOLS 1 HMD, 2 Information processing device, 3 Network, 11 Mono camera, 12 IMU, 13 Self-position estimation unit, 14 RGB camera, 15 Live image analysis unit, 16 Live image delay compensation unit, 17 CG image delay compensation unit, 18 Composition unit, 19 Lens distortion correction unit, 20 Display unit, 31 Chromakey color determination unit, 32 Drawing unit, 33 CG image analysis unit, 41 Chromakey color update unit, 42 RGB drawing unit, 43 Alpha drawing unit, 101 Live image analysis unit, 111 CG scene analysis unit

Claims

1. An information processing device having a chromakey color determination unit that determines a chromakey color for drawing areas other than an object in a CG image based on a live-action image, which is an image of real space, and a CG image including an object to be superimposed on the live-action image.

2. The information processing device according to claim 1, wherein the chromakey color determination unit determines the chromakey color based on first color information indicating a color of the real-life image and second color information indicating a color of the CG image.

3. The information processing device according to claim 2, wherein the chromakey color determination unit determines, as the chromakey color, a color that is close to a color of the real-life image and far from a color of the CG image.

4. The information processing device according to claim 2, wherein the first color information is information indicating a distribution of colors in the entire real-life image.

5. An information processing device as described in claim 2, wherein the first color information is information indicating a color distribution in a boundary area including pixels that form the boundary between the object and the real-life image among the multiple pixels that constitute the real-life image.

6. The information processing device according to claim 2, further comprising an acquisition unit that acquires the first color information.

7. The information processing device according to claim 6, wherein the acquisition unit acquires the first color information from an external device that captures the real image.

8. The information processing device according to claim 6, wherein the acquisition unit acquires the real-life image from an external device that captures the real-life image, and analyzes the real-life image to acquire the first color information.

9. The information processing device according to claim 2, wherein the second color information is information indicating a color distribution in a CG area which is the area of ​​the object within the CG image.

10. The information processing device according to claim 9, further comprising an image analysis unit that extracts the CG area from the CG image and analyzes the CG area to obtain the second color information.

11. The information processing device according to claim 10, wherein the image analysis unit extracts the CG area based on an alpha channel of the CG image.

12. The information processing device according to claim 10, wherein the image analysis section extracts the CG area based on the chromakey color.

13. The information processing device according to claim 2, wherein the first color information and the second color information are histograms of pixel values ​​expressed in an HSV color space.

14. The information processing device according to claim 2, wherein the second color information is used in rendering the CG image and is information indicating a distribution of colors in a virtual space in which the object is placed.

15. The information processing device according to claim 1, further comprising a drawing unit that captures the real-life image, draws the object in the CG image based on a self-position of an external device that displays the real-life image on which the object is superimposed, and draws areas in the CG image other than the object in the chromakey color.

16. An information processing system having an information processing device including: a chromakey color determination unit that determines a chromakey color for drawing areas other than an object in a CG image based on a live-action image, which is an image of real space, and a CG image including an object to be superimposed on the live-action image; a drawing unit that draws the object in the CG image and draws areas other than the object in the CG image in the chromakey color; an imaging unit that captures the live-action image; and a compositing unit that superimposes the CG image, making the area of ​​the chromakey color determined by the chromakey color determination unit transparent, onto the live-action image.

17. An information processing method in which an information processing device determines, based on a live-action image, which is an image of real space, and a CG image including an object to be superimposed on the live-action image, a chromakey color for drawing areas in the CG image other than the object.

18. A computer-readable recording medium having recorded thereon a program for executing a process of determining a chromakey color for drawing areas other than an object in a CG image based on a live-action image, which is an image of real space, and a CG image including an object to be superimposed on the live-action image.

Citation Information

Patent Citations

  • Display controller and dynamic image / Graphics composite display method

    JP1999313339A

  • Device, system, and method for processing image, and storage medium

    JP2001242848A

  • Image display system, image display device, control method thereof, and computer program

    JP2009111511A

  • Image generator, image generation system, image generation method, and program

    JP2020064592A