Information processing device and information processing method

Abstract

The present disclosure relates to an information processing device and an information processing method that make it possible to sufficiently express the brightness of sunlight in IBL using an environment map. The information processing device comprises: a solar position calculation unit that calculates a solar position on an HDRI used as an environment map; a solar region determination unit that determines a solar region of the HDRI with the solar position as a starting point; and a solar region replacement unit that replaces the RGB value of each pixel in the solar region of the HDRI with the RGB value of a pixel adjacent to the solar region. The technology of the present disclosure can be applied to, for example, an information processing device that performs image-based lighting using an HDRI to perform live-action CG compositing.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method, and more particularly to an information processing device and an information processing method that are capable of sufficiently expressing the brightness of sunlight in IBL that uses an environment map.

[0002] Image-based lighting (IBL) is a well-known technique for setting the lighting for CG objects in live-action CG compositing, which combines live-action footage with CG objects as a background image. IBL uses a 360-degree panoramic image of the entire scene as an environment map, making it possible to easily and accurately reflect the lighting conditions of the scene on the CG object.

[0003] The 360-degree panoramic images used as environment maps in IBL generally use images with a wide dynamic range called HDRI (High Dynamic Range Image).

[0004] However, even with HDRI that has a wide dynamic range, there are cases where the high brightness of the sun cannot be fully reproduced. Non-Patent Document 1 describes that because HDRI cannot fully reproduce the high brightness of the sun, work is carried out to add lights in CG applications.

[0005] “Lighting Techniques Using HDRI to Recreate Filming Locations (Shirogumi x The Movies “Parasyte” & “Parasyte: The Final Chapter”)”, Internet <URL: https: / / cgworld.jp / feature / 201607-cgw215t1-kiseijyu.html>

[0006] The present disclosure has been made in view of the above circumstances, and aims to make it possible to sufficiently express the brightness of sunlight in IBL that uses an environment map.

[0007] An information processing device according to one aspect of the present disclosure includes: a sun position calculation unit that calculates the position of the sun on an image used as an environment map; a sun area determination unit that determines a sun area of ​​the image based on the sun position; and a sun area replacement unit that replaces the RGB values ​​of each pixel of the sun area of ​​the image with the RGB values ​​of pixels adjacent to the sun area.

[0008] An information processing method according to one aspect of the present disclosure includes an information processing device calculating a sun position on an image used as an environment map, determining a sun area of ​​the image based on the sun position, and replacing the RGB values ​​of each pixel of the sun area of ​​the image with the RGB values ​​of pixels adjacent to the sun area.

[0009] In one aspect of the present disclosure, the position of the sun on an image used as an environment map is calculated, a sun area on the image is determined starting from the sun position, and the RGB values ​​of the sun area on the image are replaced with the RGB values ​​of pixels adjacent to the sun area.

[0010] The information processing device according to one aspect of the present disclosure can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0011] The information processing device may be an independent device or an internal block constituting a single device.

[0012] FIG. 1 is a diagram illustrating live-action CG composition using IBL. FIG. 2 is a diagram illustrating a problem that may occur when the sun is reflected in an HDRI used as an environment map. FIG. 3 is a diagram illustrating a problem that may occur when the sun is reflected in an HDRI used as an environment map. FIG. 4 is a diagram illustrating an example configuration of a shooting camera. FIG. 5 is a block diagram illustrating an example configuration of an information processing system that is a first embodiment of the present disclosure. FIG. 6 is a diagram illustrating calculation of the sun position on an HDRI. FIG. 7 is a diagram illustrating examples of HDRI before and after replacing the RGB values ​​of the sun area. FIG. 8 is a flowchart illustrating processing for replacing the sun area of ​​an HDRI. FIG. 9 is a block diagram illustrating an example hardware configuration of a computer as an information processing device. FIG. 10 is a block diagram illustrating an example configuration of an information processing system that is a second embodiment of the present disclosure.

[0013] Hereinafter, with reference to the accompanying drawings, a description will be given of a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment). The description will be given in the following order: 1. Overview of live-action CG compositing using IBL 2. The problem of the sun in HDRI 3. Configuration of the shooting camera 4. Configuration example of the first embodiment 5. Processing flow of sun area replacement processing 6. Configuration example of a computer 7. Configuration example of the second embodiment 8. Real-time IBL

[0014] 1. Overview of Live-Action CG Composition Using IBL First, live-action CG composition using IBL (Image-Based Lighting) will be described with reference to FIG.

[0015] Live-action CG compositing is a technology that uses live-action footage, which is an image of a specific scene (real scene) that has been actually shot, as a background image and composites a 3DCG model object (hereinafter referred to as a CG object) onto the background image. When compositing a CG object onto a background image, it is necessary to adjust the lighting of the CG object to match the real scene. Image-based lighting (hereinafter referred to as IBL) is known as one method for adjusting the lighting of a CG object to match the real scene.

[0016] IBL uses a 360-degree panoramic image of the entire scene as an environment map, allowing the lighting conditions of the scene to be easily reflected in the CG object with high precision. For the 360-degree panoramic image used as the environment map, an image with a wide dynamic range, known as HDRI (High Dynamic Range Image), is used.

[0017] In live-action CG compositing, a CG application, which is application software for performing live-action CG compositing, first applies IBL using the desired HDRI to the CG object to be used in the live-action CG compositing, and a CG object with adjusted lighting is generated. The CG object with adjusted lighting is then composited with the live-action video that serves as the background image to generate a live-action CG composite image.

[0018] The HDRI to be used as the environment map may be selected from HDRIs that have been photographed in advance by the user, or an HDRI that is close to the actual scene may be selected from a publicly available HDRI database site.

[0019] 2. Problems with the Sun in HDRI> Problems that can occur when the sun appears in an HDRI used as an environment map will be described with reference to FIGS.

[0020] HDRI1 in Figure 2 shows an example of an HDRI taken outdoors, showing the sun. To the right of HDRI1 is a graph of the brightness values ​​of pixels along the horizontal line that passes through the sun in HDRI1. As can be seen from this brightness graph, the brightness value of the sun is limited to the upper limit of the HDRI's brightness values, resulting in a brightness value that is darker than actual sunlight. As such, even with an HDRI that has a wide dynamic range, there are cases where the sun's high brightness cannot be fully reproduced due to what is known as blown-out highlights.

[0021] Figure 3 shows an example of lighting a CG object using IBL using an HDRI with a sun that has blown out highlights.

[0022] Figure 3A shows an example of lighting a CG object using an HDRI in which the sun is not blown out.

[0023] In an HDRI where the sun is not blown out, the shadow of the CG object OBJ1 is clearly expressed under the influence of the bright sun.

[0024] Figure 3B shows an example of lighting a CG object using an HDRI with a blown-out sun.

[0025] In an HDRI where the sun has blown out highlights, the effects of lighting other than sunlight are relatively strong, so the shadow of the CG object OBJ1 is blurred and becomes a soft shadow. If a real-life object were to exist next to this CG object OBJ1, the shadow of that object would be strong due to the bright sunlight, and the intensity of the shadow would differ between the real object and the lit CG object, creating an unnatural feeling.

[0026] The technology disclosed herein solves the problem of HDRI not being able to fully reproduce the high brightness of the sun by processing automatically rather than manually. As will be explained in detail below, the information processing device disclosed herein identifies the sun area captured in the HDRI, replaces it with the RGB values ​​of the surrounding pixels, and performs processing to set sunlight for a CG application.

[0027] 3. Configuration of the Shooting Camera FIG. 4 is a diagram showing an example of the configuration of a shooting camera that can simultaneously shoot real-life footage and HDRI.

[0028] The photographing camera 10 has a main camera 21 and a wide-angle camera 22. The main camera 21 is a photographing camera with a predetermined angle of view (normal angle of view) that is not wide-angle, and is a camera that photographs a predetermined scene and generates the live-action video described in FIG. 1. The wide-angle camera 22 is a 360-degree camera that photographs the entire scene in a 360-degree range, and is a camera that generates the HDRI described in FIG. 1. The HDRI is an image in which the entire scene in a 360-degree range is developed into a plane as an equirectangular image, and is a two-dimensional image defined by longitude φ in the horizontal direction and latitude θ in the vertical direction.

[0029] The photographic camera 10 also includes a depth sensor 23, a sensor unit 24, and an interface device 25. The depth sensor 23 is a distance sensor that acquires distance information to a subject using a ToF method. The sensor unit 24 is a group of sensors for tracking the position and orientation of the main camera 21 and the wide-angle camera 22. The sensor unit 24 includes, for example, an inertial sensor that acquires inertial information such as the acceleration and angular velocity of the photographic camera 10, a GNSS sensor such as a GPS sensor that acquires latitude and longitude information, a direction sensor (geomagnetic sensor) that detects an angle relative to north, and a horizontal sensor that detects an elevation angle, which is the angle of the zenith direction relative to a horizontal plane. The sensor unit 24 may include at least one of the sensors listed above. The depth sensor 23 and the sensor unit 24 form a tracking system that tracks the position and movement of the photographic camera 10. The interface device 25 may be a device dedicated to the photographic camera 10, or it may be a device that can be linked to the photographic camera 10 using an information processing device such as a smartphone. For example, an image (through image) captured by the main line camera 21 is displayed on the display of the interface device 25. The display of the interface device 25 is a touch panel, and can also accept operation inputs from the user.

[0030] The wide-angle camera 22, depth sensor 23, sensor unit 24, and interface device 25 are attached to the main line camera 21 by a fixing jig such as a mounting unit 26. In a tracking system that tracks the position of the camera, the offset positions of the main line camera 21 and the wide-angle camera 22 are known.

[0031] 4. Configuration Example of First Embodiment FIG. 5 is a block diagram showing a configuration example of an information processing system according to a first embodiment of the present disclosure.

[0032] The information processing system 50 in FIG. 5 is made up of the photographing camera 10 and an information processing device 60 .

[0033] The photographic camera 10 has an image generation unit 71, a time information generation unit 72, a latitude and longitude information generation unit 73, a direction information generation unit 74, a metadata generation unit 75, and an output unit 76. Note that the configuration of the photographic camera 10 in Fig. 5 only shows the portion that solves the above-mentioned effect of sun blown-out highlights, and for example, the portion related to the main lane camera 21 is omitted.

[0034] The photographing camera 10 has a main camera 21 and a wide-angle camera 22 as shown in FIG. 4, generates an HDRI to be used as an environment map and metadata indicating the conditions at the time of photographing the HDRI, and outputs them to the information processing device 60.

[0035] The information processing device 60 includes an acquisition unit 81, a sun position calculation unit 82, a sun area determination unit 83, a sun area replacement unit 84, a sunlight setting unit 85, a CG synthesis unit 86, and a CG model storage unit 87. Note that, for convenience of explanation, the information processing device 60 in Fig. 5 is configured as being divided into a plurality of blocks, but each unit of the information processing device 60 can be realized by, for example, one CG application, and the information of each unit of the information processing device 60 can be shared.

[0036] The information processing device 60 identifies the sun area captured in the HDRI supplied from the photographing camera 10, replaces it with the RGB values ​​of the surrounding pixels, and performs processing to set sunlight for the CG application.

[0037] The image generation unit 71 of the photographing camera 10 is configured, for example, with a wide-angle camera 22, and generates an HDRI obtained by capturing an entire scene in a 360-degree range. The generated HDRI is supplied to the output unit 76.

[0038] The time information generating section 72 has, for example, a clock, and generates time information indicating the shooting time and outputs it to the metadata generating section 75 .

[0039] The latitude and longitude information generating unit 73 has a GNSS sensor such as a GPS sensor, generates latitude and longitude information indicating the latitude and longitude of the wide-angle camera 22 , and outputs it to the metadata generating unit 75 .

[0040] The orientation information generation unit 74 has, for example, a horizontal sensor and an orientation sensor, and detects an elevation angle that indicates the angle in the zenith direction of the wide-angle camera 22 with respect to a horizontal plane, and an azimuth angle that indicates the orientation of the wide-angle camera 22 with respect to north, and outputs these as orientation information to the metadata generation unit 75. Note that the orientation information generation unit 74 may have a GNSS sensor instead of the horizontal sensor and the orientation sensor, and detect the elevation angle and the azimuth angle using two GNSS sensors in addition to the GNSS sensor that the latitude and longitude information generation unit 73 has.

[0041] The metadata generating unit 75 acquires the data supplied from the time information generating unit 72, the latitude and longitude information generating unit 73, and the direction information generating unit 74, and compiles the data as metadata and supplies it to the output unit 76.

[0042] The output unit 76 links the HDRI supplied from the image generation unit 71 with the metadata from the metadata generation unit 75 , and outputs a set of the HDRI and the metadata to the information processing device 60 .

[0043] The acquisition unit 81 of the information processing device 60 acquires a set of HDRI and metadata supplied from the photographing camera 10 and supplies it to a solar position calculation unit 82 .

[0044] The solar position calculation unit 82 calculates the solar position P on the HDRI using the metadata of the HDRI. More specifically, the solar position calculation unit 82 first calculates the azimuth angle φ of the solar position P on the HDRI from the latitude and longitude information and the time information, as shown in FIG. sun and elevation angle θ sun Next, the solar position calculation unit 82 calculates the azimuth angle φ of the solar position P. sun and elevation angle θ sun , and the azimuth angle φ of the wide-angle camera 22 camera and elevation angle θ camera From the above, the azimuth angle φ of the sun position P in the camera coordinate system sun ' and elevation angle θ sun ' is converted to the azimuth angle φ of the sun position P in the camera coordinate system. sun ' and elevation angle θ sun ' is calculated using the following formula: φ sun '=φ sun -φ camera θ sun '=θ sun -θcamera

[0045] Next, the sun position calculation unit 82 calculates the azimuth angle φ of the sun position P in the camera coordinate system. sun ' and elevation angle θ sun ' is transformed into HDRI image coordinates using equirectangular transformation. The sun position P in camera coordinates can be transformed into coordinates on the 360-degree panoramic image of the HDRI, just as Earth coordinates are transformed into coordinates on a world map projected using the Mercator projection.

[0046] Returning to FIG. 5 , the sun area determination unit 83 determines whether the luminance value of the pixel at sun position P calculated by the sun position calculation unit 82 is the upper limit value. If the luminance value of the pixel at sun position P is the upper limit value, the sun area determination unit 83 searches for the luminance values ​​of surrounding pixels starting from sun position P and determines the sun area of ​​the HDRI. Specifically, the sun area determination unit 83 sequentially searches adjacent pixels starting from the pixel at sun position P as a search pixel, and if the luminance value of the search pixel is equal to or greater than a predetermined threshold value, determines that search pixel as the sun area. The process of sequentially searching adjacent pixels and determining whether or not a pixel is in the sun area is repeated until it is determined that there are no pixels with a luminance value equal to or greater than the predetermined threshold value in the vicinity of sun position P.

[0047] The sun area replacement unit 84 replaces the RGB values ​​of each pixel in the sun area determined by the sun area determination unit 83 with the RGB values ​​of pixels adjacent to the sun area. For example, the RGB values ​​of each pixel in the sun area are replaced with the RGB value that is most numerous among the RGB values ​​of all pixels adjacent to the sun area.

[0048] Figure 7 shows an example of HDRI1 before and after replacing the RGB values ​​of the sun area. HDRI1 in Figure 7A is HDRI1 before replacing the RGB values ​​of the sun area, and HDRI1 in Figure 7B is HDRI1 after replacing the RGB values ​​of the sun area. As shown in Figure 7B, HDRI1 after replacement may have discontinuities in brightness values ​​at the outline around the sun area, but because HDRI is an image used for lighting, the visual discomfort of the outline is acceptable.

[0049] The sunlight setting unit 85 in Fig. 5 sets sunlight to replace the deleted sun area based on the sun position P calculated by the sun position calculation unit 82. In a CG application, a light that simulates sunlight can be set as sunlight. The sunlight setting unit 85 sets the angle and intensity of sunlight to replace the deleted sun area. Since the sun position P as seen from the wide-angle camera 22 is known, the angle from the sun position P toward the wide-angle camera 22 is set as the sunlight parameter.

[0050] The CG composition unit 86 uses the HDRI with the corrected sun area as an environment map to render a predetermined CG object acquired from the CG model storage unit 87 using IBL, and generates a live-action CG composite image by combining it with the live-action video. The live-action video is supplied from the photographing camera 10. The generated live-action CG composite image is output to, for example, an external display device and displayed.

[0051] The CG model storage unit 87 is a database that stores a large number of CG objects created in advance, and supplies the CG objects stored therein to the CG synthesis unit 86 as needed.

[0052] The information processing system 50 is configured as described above. The CG model storage unit 87, which stores a large number of CG objects, may be located in a device separate from the information processing device 60, such as a local server or a cloud server connected via a network. In this case, the information processing device 60 connects via the network to the device that stores the CG objects and acquires the necessary CG objects.

[0053] 8, a description will be given of the HDRI sun area replacement process executed by the information processing device 60. This process is started, for example, when the acquisition unit 81 of the information processing device 60 acquires a set of HDRI captured by the wide-angle camera 22 and metadata from the capturing camera 10.

[0054] First, in step S1, the solar position calculation unit 82 calculates the solar position P on the HDRI using the metadata of the HDRI. Specifically, the solar position calculation unit 82 calculates the azimuth angle φ of the solar position P from the latitude and longitude information and the time information. sun and elevation angle θ sun Calculate the azimuth angle φ of the sun position P in the camera coordinate system. sun ' and elevation angle θ sun ' and then convert it to HDRI image coordinates to calculate the sun position P on the HDRI.

[0055] In step S2, the sun area determination unit 83 determines whether the luminance value of the pixel at sun position P calculated by the sun position calculation unit 82 is the upper limit value. If it is determined in step S2 that the luminance value of the pixel at sun position P is not the upper limit value, the processes after step S2 are skipped, and the sun area replacement process in FIG. 8 ends.

[0056] On the other hand, if it is determined in step S2 that the luminance value of the pixel at sun position P is the upper limit, the process proceeds to step S3, where the sun area determination unit 83 searches for the luminance values ​​of surrounding pixels starting from sun position P to determine whether or not they are in the sun area. Specifically, the sun area determination unit 83 sequentially searches for neighboring pixels starting from the pixel at sun position P, and if the luminance value of the search pixel is equal to or greater than a predetermined threshold, determines that search pixel to be in the sun area. Then, the process of step S3 is repeatedly executed until it is determined in step S4 that no pixel having a luminance value equal to or greater than the predetermined threshold exists in the vicinity of sun position P.

[0057] If it is determined in step S4 that no pixel having a brightness value equal to or greater than the predetermined threshold value exists around the sun position P, it is determined in step S4 that the search is complete, and the process proceeds to step S5.

[0058] In step S5, the sun area replacement unit 84 replaces the RGB values ​​of each pixel in the sun area determined by the sun area determination unit 83 with the RGB values ​​of pixels adjacent to the sun area. For example, the RGB values ​​of each pixel in the sun area are replaced with the RGB value that is most numerous among the RGB values ​​of all pixels adjacent to the sun area.

[0059] In step S6, the sunlight setting unit 85 sets sunlight to replace the deleted sun area, based on the sun position P calculated by the sun position calculation unit 82. Specifically, in the CG application, the angle and intensity of sunlight to replace the deleted sun area are set.

[0060] In step S7, the sunlight setting unit 85 outputs the HDRI after the sun area correction and the sunlight setting information to the CG synthesis unit 86. The CG synthesis unit 86 uses the HDRI with the sun area corrected as an environment map to render a predetermined CG object obtained from the CG model storage unit 87 using IBL. The sunlight setting information is also reflected in the rendering of the CG object. The CG synthesis unit 86 generates a live-action CG composite image by combining the rendered CG object with live-action video. The generated live-action CG composite image is output to and displayed on, for example, an external display device.

[0061] The process of replacing the sun area in the HDRI is executed as described above. When the high brightness of the sun cannot be adequately reproduced in the HDRI used as an environment map, the information processing system 50 according to the first embodiment identifies the sun area in the HDRI, replaces it with the RGB values ​​of the surrounding pixels, and sets sunlight at the sun position. This process can be performed automatically within the CG application. This allows the brightness of sunlight to be adequately represented in the IBL that uses the environment map.

[0062] It is not necessary to execute all of the above-described processes within the CG application, and some or all of the above-described processes may be executed in an application other than the CG application. The above-described sun area identification process, RGB value replacement process, sunlight setting process, etc. may be plug-ins for the CG application.

[0063] 6. Computer Configuration Example The series of processes executed by the information processing device 60 can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed in a computer. Here, the computer includes a microcomputer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0064] FIG. 9 is a block diagram showing an example of the hardware configuration of a computer as an information processing device 60 when the above-described series of processes are executed by a program.

[0065] The computer 200 includes a central processing unit (CPU) 201, a read-only memory (ROM) 202, and a random access memory (RAM) 203. The CPU 201, the ROM 202, and the RAM 203 are connected to one another by a bus 204.

[0066] An input / output interface 205 is also connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

[0067] The input unit 206 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 207 includes a display, speaker, output terminal, etc. The storage unit 208 includes a hard disk, SSD (Solid State Drive), RAM disk, non-volatile memory, etc. The communication unit 209 includes a network interface, etc. The drive 210 drives removable media 211 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0068] In the computer 200 configured as above, the CPU 201 performs the above-described series of processes by, for example, loading a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executing the program. The RAM 203 also stores data and the like necessary for the CPU 201 to execute various processes.

[0069] The program executed by the CPU 201 of the computer 200 can be provided by being recorded on a removable medium 211 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0070] In the computer 200, the program can be installed in the storage unit 208 via the input / output interface 205 by inserting the removable medium 211 into the drive 210. The program can also be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. Alternatively, the program can be installed in the ROM 202 or the storage unit 208 in advance.

[0071] 7. Configuration Example of Second Embodiment FIG. 10 is a block diagram showing a configuration example of an information processing system according to a second embodiment of the present disclosure.

[0072] In the second embodiment of FIG. 10, components having substantially the same functional configuration as those in the first embodiment described above are denoted by the same reference numerals, and redundant description will be omitted.

[0073] The information processing system 50 in Fig. 10 is composed of a photographic camera 10' and an information processing device 60'. Similar to the first embodiment shown in Fig. 5, only the part of the photographic camera 10' that solves the effect of sun blowout is shown.

[0074] Comparing the configuration of the second embodiment in FIG. 10 with the first embodiment in FIG. 5, the sun position calculation unit 82, the sun area determination unit 83, and the sun area replacement unit 84 are provided in the photographing camera 10′, not in the information processing device 60′.

[0075] In the information processing system 50 of the second embodiment, the photographing camera 10' executes the processes from step S1 to step S5 described in the flowchart of FIG. 8 , i.e., the processes up to and including calculating the sun position P on the HDRI photographed by the wide-angle camera 22 and generating an HDRI in which the RGB values ​​of each pixel in the sun area are replaced with the RGB values ​​of surrounding pixels. The output unit 76 of the photographing camera 10' outputs a set of the HDRI with the corrected sun area and metadata to the information processing device 60'. This metadata includes coordinate information of the sun position P on the HDRI in addition to the time information, latitude and longitude information, and orientation information similar to those in the first embodiment.

[0076] The sunlight setting unit 85 of the information processing device 60′ sets sunlight based on the coordinate information of the sun position P in the metadata. The CG synthesis unit 86 uses the HDRI with the corrected sun area as an environment map to render a predetermined CG object acquired from the CG model storage unit 87 using IBL, and generates a live-action CG composite image by synthesizing it with the live-action video.

[0077] In the information processing system 50 according to the second embodiment, if the high brightness of the sun cannot be adequately reproduced in the HDRI used as an environment map, the sun area in the HDRI is replaced with the RGB values ​​of the surrounding pixels, and sunlight is set at the sun position. This processing can be performed automatically within the CG application. This allows the brightness of sunlight to be adequately expressed in the IBL using the environment map.

[0078] 8. Real-time IBL In the first and second embodiments described above, the photographing camera 10 can simultaneously capture live-action footage from the main camera 21 and a 360-degree panoramic HDRI image from the wide-angle camera 22. This enables real-time IBL, in which the live-action footage from the main camera 21 is acquired as a moving image, and the 360-degree panoramic image as an environment map is also acquired as a moving image, to generate a live-action CG composite image that responds to lighting conditions that change from moment to moment. The sun region replacement process described above is extremely simple and can be processed at high speed, making it fully applicable to real-time IBL.

[0079] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0080] For example, in the first and second embodiments described above, an example was described in which the image used as the environment map was an HDRI with a wide dynamic range. However, the above-described series of processes, such as identifying the sun area and replacing RGB values, and setting sunlight, may also be applied to an image called an LDRI (Low Dynamic Range Image), which has a normal 256-level range. In other words, this technology can be applied to any image that can be used as an environment map when performing IBL, regardless of the gradation level of the image used as the environment map.

[0081] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0082] Each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0083] The steps described in the flowchart may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order and may be performed in parallel or at any time required, such as when a call is made.

[0084] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0085] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0086] The technology disclosed herein may employ the following configurations. (1) An information processing device comprising: a sun position calculation unit that calculates the position of the sun on an image used as an environment map; a sun area determination unit that determines a sun area on the image starting from the sun position; and a sun area replacement unit that replaces the RGB values ​​of each pixel in the sun area of ​​the image with the RGB values ​​of pixels adjacent to the sun area. (2) The information processing device described in (1), wherein the sun position calculation unit calculates the sun position on the image using metadata associated with the image. (3) The information processing device described in (2), wherein the metadata includes latitude and longitude information, time information, and orientation information of a camera that captured the image. (4) The information processing device described in (2) or (3), wherein the metadata includes coordinates of the sun position on the image. (5) The information processing device described in any of (1) to (4), wherein the sun area determination unit searches for pixels having a brightness value equal to or greater than a predetermined threshold starting from the sun position and determines the sun area. (6) The information processing device according to any one of (1) to (5), wherein the sun area replacement unit replaces the RGB value of each pixel in the sun area with the RGB value that is most numerous among the RGB values ​​of pixels adjacent to the sun area. (7) The information processing device according to any one of (1) to (6), further comprising a sunlight setting unit that sets sunlight in a CG application based on the position of the sun on the image. (8) The information processing device according to any one of (1) to (7), further comprising a CG synthesis unit that renders a CG object using, as the environment map, an image in which the RGB values ​​of each pixel in the sun area of ​​the image have been replaced with the RGB values ​​of the adjacent pixels. (9) An information processing method, including an information processing device calculating the position of the sun on an image to be used as the environment map, determining a sun area of ​​the image using the position of the sun as a starting point, and replacing the RGB value of each pixel in the sun area of ​​the image with the RGB value of a pixel adjacent to the sun area.

[0087] 10, 10' Shooting camera, 21 Main camera, 22 Wide-angle camera, 23 Depth sensor, 24 Sensor unit, 25 Interface device, 26 Mounting unit, 50 Information processing system, 60, 60' Information processing device, 71 Image generation unit, 72 Time information generation unit, 73 Latitude and longitude information generation unit, 74 Orientation information generation unit, 75 Metadata generation unit, 76 Output unit, 81 Acquisition unit, 82 Sun position calculation unit, 83 Sun area determination unit, 84 Sun area replacement unit, 85 Sunlight setting unit, 86 CG synthesis unit, 87 CG model storage unit, 200 Computer, 201 CPU, 202 ROM, 203 RAM, 208 Storage unit

Claims

1. An information processing device comprising: a sun position calculation unit that calculates the position of the sun on an image used as an environment map; a sun area determination unit that determines the sun area of ​​the image based on the sun position; and a sun area replacement unit that replaces the RGB values ​​of each pixel of the sun area of ​​the image with the RGB values ​​of pixels adjacent to the sun area.

2. The information processing device according to claim 1, wherein the solar position calculation unit calculates the solar position on the image using metadata associated with the image.

3. The information processing device according to claim 2, wherein the metadata includes latitude and longitude information, time information, and direction information of the camera that captured the image.

4. The information processing device according to claim 2, wherein the metadata includes coordinates of the position of the sun on the image.

5. The information processing device according to claim 1, wherein the sun area determination unit searches for pixels having a brightness value equal to or greater than a predetermined threshold value, starting from the sun position, and determines the sun area.

6. The information processing device according to claim 1, wherein the sun area replacement unit replaces the RGB value of each pixel in the sun area with the RGB value that is most numerous among the RGB values ​​of pixels adjacent to the sun area.

7. The information processing device according to claim 1, further comprising a sunlight setting unit that sets sunlight in a CG application based on the position of the sun on the image.

8. The information processing device according to claim 1, further comprising a CG synthesis unit that renders a CG object using, as the environment map, an image in which the RGB values ​​of each pixel in the sun area of ​​the image have been replaced with the RGB values ​​of the adjacent pixels.

9. An information processing method including: an information processing device calculating the position of the sun on an image used as an environment map; determining a sun area of ​​the image starting from the sun position; and replacing the RGB values ​​of each pixel of the sun area of ​​the image with the RGB values ​​of pixels adjacent to the sun area.

Images