Information processing device and information processing method

Abstract

The present invention relates to an information processing device and an information processing method that make it possible to more easily select an environment map suitable for a real scene. An information processing device according to the present invention is provided with: a comparison region detection unit that detects, for each of a plurality of HDRIs, a region corresponding to the angle of view of a captured image as a comparison region; a similarity calculation unit that calculates, for each of the plurality of HDRIs, the degree of similarity between a comparison region image and the captured image, the comparison region image being an image of the comparison region; and an HDRI determination unit that determines one or more HDRIs on the basis of the calculated degrees of similarity. The present invention can be applied to, for example, an information processing device for selecting an HDRI.

Description

Information processing device and information processing method

[0001] The present disclosure relates to an information processing device and an information processing method, and more particularly to an information processing device and an information processing method that enable easier selection of an environment map suitable for a real scene.

[0002] In live-action CG compositing technology, which composites CG objects using live-action images as background images, adjusting the lighting of the CG objects to match the real scene is difficult because it requires matching the light source information of the virtual environment (such as the position, color, and intensity of the light source) to the real environment.

[0003] Image-based lighting (IBL) is one of the well-known techniques for setting the lighting for CG objects. IBL uses a 360-degree panoramic image of the entire scene as an environment map, making it possible to easily and accurately reflect the lighting conditions of the scene on the CG object.

[0004] The 360-degree panoramic images used as environment maps in IBL generally have a wide dynamic range, known as HDRI (High Dynamic Range Image). However, HDRI requires multiple shots taken with a 360-degree camera while adjusting the exposure, and then composited, which makes the shooting costs high.

[0005] One way to address this issue is to select an HDRI that is close to the actual scene from a database site of HDRIs that have been photographed in advance and made publicly available, and use that as an environment map. For example, the application Unity allows you to switch between multiple HDRIs to compare lighting conditions and select the HDRI that is most suitable for the scene (see Non-Patent Document 1).

[0006] UnityDocumentation, HDRI View, <URL: https: / / docs.unity3d.com / ja / 2018.4 / Manual / LookDevHDRIView.html>

[0007] However, it is difficult and time-consuming for users to select an HDRI suitable for a real scene from a huge database of HDRIs. Therefore, an easier method for selecting an HDRI suitable for a real scene is desired.

[0008] The present disclosure has been made in light of these circumstances, and aims to make it easier to select an environment map that is suitable for a real scene.

[0009] An information processing device according to one aspect of the present disclosure includes: a comparison area detection unit that detects, for each of a plurality of images, an area corresponding to the angle of view of the live-action image as a comparison area; a similarity calculation unit that calculates, for each of the plurality of images, a similarity between a comparison area image that is an image of the comparison area and the live-action image; and a determination unit that determines one or more of the images based on the calculated similarity.

[0010] An information processing method according to one aspect of the present disclosure includes: detecting, for each of a plurality of images, an area corresponding to the angle of view of the real-life image as a comparison area; calculating, for each of the plurality of images, a similarity between a comparison area image, which is an image of the comparison area, and the real-life image; and determining one or more of the images based on the calculated similarity.

[0011] In one aspect of the present disclosure, for each of a plurality of images, an area corresponding to the angle of view of the live-action image is detected as a comparison area, and for each of the plurality of images, the similarity between the comparison area image, which is an image of the comparison area, and the live-action image is calculated, and one or more of the images are determined based on the calculated similarity.

[0012] The information processing device according to one aspect of the present disclosure can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting it via a transmission medium or by recording it on a recording medium.

[0013] The information processing device may be an independent device or an internal block constituting a single device.

[0014] FIG. 1 is a diagram illustrating live-action CG composition using IBL. FIG. 1 is a block diagram illustrating a configuration example of an information processing system according to a first embodiment of the present disclosure. FIG. 2 is a block diagram illustrating a detailed configuration example of an HDRI selection unit, which is a diagram illustrating an overview of processing performed by the HDRI selection unit. FIG. 3 is a diagram illustrating processing by a comparison region detection unit and a comparison region extraction unit. FIG. 4 is a diagram illustrating processing by a similarity calculation unit. FIG. 5 is a diagram illustrating similarity calculation when the index is the sun position. FIG. 6 is a flowchart illustrating HDRI selection processing performed by the HDRI selection unit. FIG. 7 is a diagram illustrating HDRI selection processing using a plurality of live-action images. FIG. 8 is a block diagram illustrating a configuration example when an information processing device is configured by a computer. FIG. 9 is a block diagram illustrating a configuration example of a camera according to a second embodiment of the present disclosure.

[0015] Hereinafter, a description will be given of a mode for carrying out the technology of the present disclosure (hereinafter referred to as an embodiment) with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration will be assigned the same reference numerals to avoid redundant description. The description will be given in the following order: 1. Overview of live-action CG composition using IBL 2. Configuration example of the first embodiment 3. Overview of processing performed by the HDRI selection unit 4. Detailed configuration example of the HDRI selection unit 5. Processing flow of HDRI selection processing 6. HDRI selection processing using multiple live-action images 7. Example of computer configuration 8. Example of configuration of the second embodiment

[0016] 1. Overview of Live-Action CG Composition Using IBL First, live-action CG composition using IBL (Image-Based Lighting) will be described with reference to FIG.

[0017] Live-action CG compositing is a technology that uses a real-life image, which is an actual photograph of a specific scene (real scene), as a background image and composites a 3DCG model object (hereinafter referred to as a CG object) onto the background image. When compositing a CG object onto a background image, it is necessary to adjust the lighting of the CG object to match the real scene. Adjusting the lighting of the CG object to match the real scene is difficult because it requires matching the light source information of the virtual environment (for example, the position, color, and intensity of the light source) to the real environment.

[0018] IBL is a technique that uses a 360-degree panoramic image of the entire scene as an environment map, allowing the lighting conditions of the scene to be easily reflected in CG objects with high precision. The 360-degree panoramic image used as the environment map is an image with a wide dynamic range called HDRI (High Dynamic Range Image).

[0019] In live-action CG compositing, first, in a 3DCG application, which is application software that performs live-action CG compositing, a CG object to be used in the live-action CG compositing is generated by applying IBL using the desired HDRI to the CG object to generate a CG object with adjusted lighting.The CG object with adjusted lighting is then composited with the live-action image that serves as the background image to generate a live-action CG composite image.

[0020] The HDRI used as the environment map may be selected from HDRIs previously captured by the user, or an HDRI that is similar to the actual scene may be selected from a publicly available HDRI database site. In either case, selecting an HDRI that is similar to the actual scene is difficult and time-consuming. The information processing device described below executes a process of selecting an HDRI suitable for an actual image captured by a camera from among multiple HDRIs stored in advance. This allows the user to more easily select an HDRI suitable for the actual scene.

[0021] 2. Configuration Example of First Embodiment FIG. 2 is a block diagram showing a configuration example of an information processing system according to a first embodiment of the present disclosure.

[0022] The information processing system 1 in FIG. 2 is made up of a camera 21 and an information processing device 22 .

[0023] The camera 21 is an imaging device that actually captures a predetermined scene (real scene), generates a real-life image that is an image obtained by capturing the image, and metadata that indicates the conditions at the time of capturing the image, and outputs the images to the information processing device 22. The camera 21 has an image generation unit 41, a lens data generation unit 42, an attitude data generation unit 43, a time information generation unit 44, a latitude / longitude / orientation information generation unit 45, and a metadata generation unit 46.

[0024] The information processing device 22 selects an HDRI that is optimal for the live-action image supplied from the camera 21, and generates and outputs a live-action CG composite image by combining the live-action image with a CG object. The information processing device 22 has an HDRI selection unit 61, an HDRI storage unit 62, a CG synthesis unit 63, and a CG model storage unit 64.

[0025] The image generating unit 41 of the camera 21 includes, for example, an imaging element such as a CMO image sensor or a CCD, an optical lens, etc., and generates a real-life image obtained by capturing a real scene as a subject. The generated real-life image is output to the information processing device 22 via an input / output unit (not shown).

[0026] The lens data generation unit 42 generates and stores data indicating optical lens characteristics such as the angle of view, focal length, F-number, and lens aperture as lens data. Some of the lens data may be data that has been stored in advance as fixed values. The lens data generation unit 42 outputs the lens data to the metadata generation unit 46.

[0027] The attitude data generation unit 43 has various sensors, such as a gyro sensor, a depth sensor, and an IMU sensor, and generates attitude data representing the camera attitude at the time of capturing an image based on sensor values ​​obtained from each sensor, and outputs the attitude data to the metadata generation unit 46. The attitude data includes, for example, (X, Y, Z) position coordinates and pan, tilt, and roll values ​​(angles). The attitude data generation unit 43 may estimate its own position using, for example, SLAM (Simultaneous Localization and Mapping). The various sensors, such as the gyro sensor and the depth sensor, may be provided inside the camera 21, or may be fixed integrally with the camera 21 as a tracking system that tracks the position and movement of the camera 21.

[0028] The time information generating section 44 has, for example, a clock or the like, generates time information indicating the shooting time, and outputs it to the metadata generating section 46 .

[0029] The latitude, longitude, and orientation information includes a GNSS sensor such as a GPS sensor, and generates latitude and longitude information indicating the latitude and longitude of the camera 21 and orientation information indicating the north direction, and outputs these to the metadata generation unit 46.

[0030] The metadata generation unit 46 acquires data supplied from the lens data generation unit 42, the attitude data generation unit 43, the time information generation unit 44, and the latitude / longitude / orientation information generation unit 45. The metadata generation unit 46 compiles the acquired data and outputs it to the information processing device 22 via an input / output unit (not shown) as metadata of the real image generated by the image generation unit 41. Hereinafter, the metadata of the real image will be referred to as camera metadata to distinguish it from HDRI metadata, which will be described later.

[0031] The HDRI selection unit 61 of the information processing device 22 acquires the actual image and camera metadata supplied from the camera 21. The HDRI selection unit 61 selects the HDRI that is most suitable for the acquired actual image from among the many HDRIs stored in the HDRI storage unit 62, and outputs it to the CG synthesis unit 63.

[0032] The HDRI storage unit 62 is a database that stores a large number of HDRIs created in advance, and supplies the HDRIs it holds to the HDRI selection unit 61 as needed.

[0033] The CG synthesis unit 63 is supplied with the live-action image and camera metadata from the camera 21, and also with an HDRI selected as the most suitable for the live-action image from the HDRI selection unit 61. The CG synthesis unit 63 uses the HDRI selected by the HDRI selection unit 61 as an environment map to render a predetermined CG object acquired from the CG model storage unit 64 using IBL, and generates a live-action CG composite image by combining the rendered CG object with the live-action image. The generated live-action CG composite image is output to, for example, an external display device and displayed.

[0034] The CG model storage unit 64 is a database that stores a large number of CG objects created in advance, and supplies the CG objects stored therein to the CG synthesis unit 63 as needed.

[0035] The information processing system 1 is configured as described above. Note that both or either one of the HDRI storage unit 62 that stores a large number of HDRIs and the CG model storage unit 64 that stores a large number of CG objects may be placed in a device separate from the information processing device 22, such as a local server or cloud server connected via a network. In this case, the information processing device 22 connects via the network to the device that stores the HDRI or CG object, and acquires the required HDRI or CG object.

[0036] As described above, the information processing device 22 selects an HDRI that is optimal for the live-action image captured by the camera 21, uses it as an environment map, and combines the live-action image with a rendered image obtained by rendering a CG object using IBL. Below, a detailed description will be given of the process performed by the HDRI selection unit 61 to select an HDRI that is optimal for the live-action image.

[0037] 3. Overview of Processing Performed by HDRI Selection Unit First, an overview of processing performed by the HDRI selection unit 61 will be described with reference to FIG.

[0038] The actual image 81 shown on the left side of Fig. 3 is an image captured by the camera 21. The three HDRIs 91 to 93 shown on the right side of Fig. 3 are HDRIs acquired from the HDRI storage unit 62. For simplicity, it is assumed in Fig. 3 that the HDRIs stored in the HDRI storage unit 62 are three, HDRIs 91 to 93.

[0039] The HDRI selection unit 61 calculates the similarity between the real-life image 81 captured by the camera 21 and each of the multiple HDRIs stored in the HDRI storage unit 62. In the example of FIG. 3 , the similarity between the real-life image 81 and HDRI 91 is calculated to be 0.8, the similarity between the real-life image 81 and HDRI 92 is calculated to be 0.4, and the similarity between the real-life image 81 and HDRI 93 is calculated to be 0.1. The HDRI selection unit 61 may calculate the similarity between the real-life image 81 and the entire HDRI image, but the HDRI is a 360-degree panoramic image and has a different angle of view from the real-life image 81. Therefore, the HDRI selection unit 61 sets a comparison region for the HDRI that corresponds to the angle of view of the real-life image 81, and calculates the similarity between the real-life image 81 and the comparison region of the HDRI. Note that, as a simpler method, a method of calculating the similarity between the real-life image 81 and the entire HDRI image may also be adopted.

[0040] Based on the similarity calculation results, the HDRI selection unit 61 selects the HDRI with the highest similarity as the HDRI optimal for the live-action image. Alternatively, the HDRI selection unit 61 selects a predetermined number of HDRIs with the highest similarity as the HDRI optimal for the live-action image. In the example of FIG. 3 , when selecting one HDRI, the HDRI 91 surrounded by a solid line, which is the HDRI with the highest similarity, is selected. When selecting two HDRIs as the predetermined number of HDRIs, the HDRI 91 and HDRI 92 surrounded by dotted lines are selected. When selecting multiple HDRIs, the number of HDRIs to select can be determined in advance using setting information, etc. Alternatively, an HDRI with a similarity equal to or greater than a predetermined value may be selected as the optimal HDRI.

[0041] 4. Detailed Configuration Example of HDRI Selection Unit FIG. 4 is a block diagram showing a detailed configuration example of the HDRI selection unit 61. As shown in FIG.

[0042] The HDRI selection unit 61 has a candidate search unit 101, a comparison area detection unit 102, a comparison area extraction unit 103, a similarity calculation unit 104, and an HDRI determination unit 105. Note that although the configuration of the HDRI selection unit 61 in Fig. 4 is divided into multiple blocks for convenience, it can also be configured as a single processing block, and information from each unit can be shared with each other.

[0043] The candidate search unit 101 searches for HDRIs to be compared from among the numerous HDRIs stored in the HDRI storage unit 62 based on the labels assigned to each HDRI. In other words, when the number of comparison candidates stored in the HDRI storage unit 62 is enormous, the amount of calculation increases if all HDRIs are compared. Therefore, the candidate search unit 101 narrows down the HDRIs to be compared to a certain extent. The HDRI storage unit 62 stores HDRIs together with metadata corresponding to the HDRIs (hereinafter referred to as HDRI metadata). The HDRI metadata includes the shooting time of the HDRI, orientation information indicating the north direction relative to the shooting camera, labels, etc. The labels indicate the shooting environment, such as outdoor, indoor, studio, sunrise, sunset, night, nature, or urban, and one or more labels appropriate for the image are assigned to the HDRI. The candidate search unit 101 searches for HDRIs that have been assigned the same label as the label corresponding to the real-life image specified by the user via an operation input unit (not shown), and outputs the HDRIs as comparison targets to the comparison area detection unit 102. HDRI metadata is also output to the comparison area detection unit 102.

[0044] The comparison area detection unit 102 detects a comparison area for each of the multiple HDRIs using the multiple HDRIs and HDRI metadata supplied as comparison targets from the candidate search unit 101, and the actual image and camera metadata supplied from the camera 21. As described above, since the HDRI and the actual image have different angles of view, the comparison area detection unit 102 sets a comparison area for each HDRI that corresponds to the angle of view of the actual image.

[0045] The comparison region extraction unit 103 extracts an image of the comparison region of the HDRI detected by the comparison region detection unit 102 for each of the HDRIs to be compared, generates a comparison region image, and outputs it to the similarity calculation unit 104.

[0046] The similarity calculation unit 104 calculates the similarity between the comparison region image extracted from the HDRI and the actual image for each HDRI to be compared. The calculation result is supplied to the HDRI determination unit 105.

[0047] The HDRI determination unit 105 determines one or more HDRIs based on the similarity calculated for each of the HDRIs to be compared, and outputs them as the HDRIs that are optimal for the live-action image.Whether to determine one HDRI that is optimal for the live-action image or to determine multiple HDRIs is determined in advance using setting information, etc.

[0048] <Processing of Comparison Area Detection Unit> Next, the processing of the comparison area detection unit 102 and the comparison area extraction unit 103 will be described with reference to Fig. 5. Fig. 5 shows an example in which the HDRI 91 shown in Fig. 3 is used as the comparison target.

[0049] The comparison area detection unit 102 detects a comparison area of ​​the HDRI 91 using the HDRI 91, its HDRI metadata, and the actual image 81, and its camera metadata. For example, the comparison area detection unit 102 detects a comparison area of ​​the HDRI 91 from the camera posture, orientation information, and angle of view in the metadata, using the light source position in the image as a reference. In the example of FIG. 5 , the light source position 82 in the actual image 81 and the light source position 92 in the HDRI 91 are detected and used as a reference for the longitude direction of the HDRI 91. Then, the comparison area detection unit 102 detects an area 91S of the HDRI 91 that corresponds to the angle of view of the actual image 81, as a comparison area, using the camera posture and angle of view in the camera metadata. The comparison area extraction unit 103 extracts an image 91S' of the area 91S of the HDRI 91 detected as the comparison area. If the light source position is unknown in the HDRI 91 and the actual image 81, the orientation information in the HDRI metadata and camera metadata is used to align the direction to north, and from the camera posture and angle of view, an area 91S of the HDRI 91 corresponding to the angle of view of the actual image 81 is detected as the comparison area.

[0050] The user may set the comparison region as desired, but setting it for each of the vast number of HDRIs to be compared is tedious. While there is also a method of setting the same region as the comparison region for all HDRIs, this would ignore the characteristics of the HDRI (e.g., light source position, orientation, etc.), and it may be impossible to set an appropriate comparison region. As described above, the comparison region detection unit 102 detects the comparison region based on the light source position and metadata, making it possible to automatically detect the optimal comparison region for each of the vast number of HDRIs.

[0051] <Processing of Similarity Calculation Unit> Next, the processing of the similarity calculation unit 104 will be described with reference to Fig. 6. Fig. 6 shows an example of similarity calculation between the real-life image 81 and an image 91S' of a region 91S detected as a comparison region in the HDRI 91 (hereinafter referred to as a comparison region image 91S').

[0052] First, the similarity calculation unit 104 calculates a probability based on multiple indices for each of the actual image 81 and the comparison region image 91S'. Examples of indices used to calculate the probability include location, time of day, season, weather, and color distribution. The location probability is divided into, for example, the probability of outdoors and the probability of indoors, with the sum of the two being 1. The time of day probability is divided into, for example, the probability of morning, afternoon, and night, with the sum of the three being 1. The season probability is divided into, for example, the probability of spring, summer, autumn, and winter, with the sum of the four being 1. The weather probability is divided into, for example, the probability of sunny, cloudy, and rainy, with the sum of the three being 1. The color distribution probability is divided into, for example, the probability of pixels with reddish colors (R), pixels with greenish colors (G), and pixels with blueish colors (B), with the sum of the three being 1. The number of indices used to calculate the probability may be one or more.

[0053] 6, the location probability of the actual image 81 is (outdoors, indoors) = (0.9, 0.1), the time period probability is (morning, afternoon, night) = (0.3, 0.5, 0.2), the season probability is (spring, summer, autumn, winter) = (0.3, 0.3, 0.2, 0.2), the weather probability is (sunny, cloudy, rainy) = (0.8, 0.1, 0.1), and the color distribution probability is (red, green, blue) = (0.1, 0.5, 0.4).

[0054] On the other hand, the location probability of the comparison area image 91S' extracted from the HDRI 91 is (outdoor, indoor) = (0.9, 0.1), the time of day probability is (morning, daytime, night) = (0.1, 0.8, 0.1), the season probability is (spring, summer, autumn, winter) = (0.3, 0.5, 0.1, 0.1), the weather probability is (sunny, cloudy, rainy) = (1.0, 0.0, 0.0), and the color distribution probability is (red color, green color, blue color) = (0.1, 0.4, 0.5).

[0055] The probability of each index can be calculated using, for example, image recognition AI or image processing. For example, a method for calculating the probability of a location (indoor, outdoor) using a DNN is disclosed in the non-patent document "Classification of Indoor-Outdoor Scene Using Deep Learning Techniques," <URL: https: / / link.springer.com / chapter / 10.1007 / 978-981-19-5868-7_38>. Also, a method for calculating the Cosine similarity of a color distribution by obtaining a color histogram is disclosed in "Calculating the distance between color distribution histograms," <URL: http: / / www.thothchildren.com / chapter / 5b01937eb8dc30181ec78917>.

[0056] The similarity calculation unit 104 uses the probability of each index as a vector to calculate the similarity between the actual image 81 and each index in the comparison region image 91S' using Cosine similarity. The Cosine similarity for location is 1.0, the Cosine similarity for time of day is 0.898, and the Cosine similarity for season is 0.915. The Cosine similarity for weather is 0.985, and the Cosine similarity for color distribution is 0.976.

[0057] Finally, the similarity calculation unit 104 calculates the final similarity between the actual image 81 and the comparison region image 91S' by multiplying the similarity (Cos similarity) of each index. In the example of FIG. 6, the similarity is calculated as follows: Similarity = 1.0 x 0.898 x 0.915 x 0.985 x 0.976 ≈ 0.790

[0058] In this manner, the similarity between the actual image 81 and the comparison area image 91S' is calculated to be 0.790.

[0059] In addition to the location, time period, season, weather, and color distribution described above, the position of the sun in the image may also be used as one of the indices for calculating the similarity. In the case of the sun position as an index, the similarity can be calculated using the similarity of the directional vector of the sun position as seen from the shooting position o as shown in Figure 7, rather than the probability.

[0060] Specifically, the similarity calculation unit 104 emits a ray of light from the optical center of the camera 21 in a specified camera orientation toward the sun's position 81A in the live-action image 81, and projects the sun's position 81A in the live-action image 81 onto a position 81B on the sphere. Because the reference north direction is determined from the orientation information in the camera metadata and the HDRI metadata, the sun's position 81B on the sphere of the live-action image 81 and the sun's position 91B in the HDRI 91 can be expressed by an azimuth angle and an altitude angle. By calculating the cosine similarity using the sun's position expressed by this (azimuth angle, altitude angle) as a vector, it is possible to calculate the similarity using the sun's position as an index. To estimate the position of the sun in a photographed image, a method such as that described in the non-patent document "Solar position algorithm for solar radiation applications," Solar Energy, Volume 76, Issue 5, 2004, Pages 577-589, <URL: https: / / www.sciencedirect.com / science / article / abs / pii / S0038092X0300450X?via%3Dihub> can be used. To estimate the position of the sun in a 360-degree panoramic image, methods such as those in the non-patent document "Generating 360 Outdoor Panorama Dataset with Reliable Sun Position Estimation", Shih-Hsiu Chang, Ching-Ya Chiu, Chia-Sheng Chang, Kuo-Wei Chen, Chih-Yuan Yao, Ruen-Rone Lee, Hung-Kuo Chu, National Tsing-Hua University, <URL: https: / / cgv.cs.nthu.edu.tw / projects / 360SP> and the non-patent document "PeterZhouSZ / 360-sun-detection, <URL: https: / / github.com / PeterZhouSZ / 360-sun-detection?tab=readme-ov-file>" can be used.

[0061] 5. Processing Flow of HDRI Selection Processing> Next, the HDRI selection processing executed by the HDRI selection unit 61 will be described with reference to the flowchart in Fig. 8. This processing starts when, for example, a real-life image and camera metadata are supplied from the camera 21.

[0062] First, in step S1, the candidate search unit 101 searches for HDRIs to be used as comparison targets from among the many HDRIs stored in the HDRI storage unit 62, based on the labels assigned to each HDRI. The candidate search unit 101 searches for HDRIs to which the same labels as those corresponding to the real-life image specified by the user via an operation input unit (not shown) are assigned, and outputs the HDRIs to the comparison area detection unit 102 as comparison targets. The candidate search unit 101 may itself assign labels to classify the real-life image by performing image recognition on the real-life image, and search for HDRIs to which the same labels as those corresponding to the real-life image are assigned. Alternatively, if the number of HDRIs stored in the HDRI storage unit 62 is small or if the labels are unknown, step S1 may be skipped (omitted).

[0063] In step S2, the comparison area detection unit 102 detects a comparison area for each of the multiple HDRIs using the multiple HDRIs and HDRI metadata supplied as comparison targets from the candidate search unit 101, and the actual image and camera metadata supplied from the camera 21. For example, a comparison area for the HDRI corresponding to the angle of view of the actual image is detected using the light source position in the image as a reference.

[0064] In step S3 , the comparison area extraction unit 103 extracts an image of the comparison area detected by the comparison area detection unit 102 for each of the HDRIs to be compared, generates a comparison area image, and outputs it to the similarity calculation unit 104 .

[0065] In step S4, the similarity calculation unit 104 calculates the similarity between the comparison region image extracted from the HDRI and the actual image for each HDRI to be compared. For example, as described with reference to FIG. 6 , the similarity calculation unit 104 calculates the probability of multiple indices, and calculates the similarity of each indices using Cos similarity, using the probability of each indices as a vector. Furthermore, the similarity calculation unit 104 calculates the final similarity between the actual image and the comparison region image by multiplying the similarities of the multiple indices. The calculation result is supplied to the HDRI determination unit 105.

[0066] In step S5, the HDRI determination unit 105 determines one or more most suitable HDRIs based on the final similarity calculated for each of the HDRIs to be compared. The determined one or more HDRIs are output as the HDRIs most suitable for the live-action image, and the HDRI selection process in Fig. 8 ends. If multiple HDRIs have been determined, the multiple HDRIs may be displayed on the display, and the user may select one HDRI to actually use in the rendering process of the CG synthesis unit 63 at the subsequent stage.

[0067] By performing the HDRI selection process as described above, the HDRI selection unit 61 can determine one or more HDRIs that are optimal for the real-life image captured by the camera 21. In other words, it is possible to more easily select an HDRI that is suitable for the real scene.

[0068] 6. HDRI Selection Process Using Multiple Real-Life Images In the above example, an example has been described in which one optimal HDRI is selected for one real-life image captured by the camera 21.

[0069] Next, referring to Figure 9, we will explain the process of selecting the optimal HDRI for multiple actual images when the camera 21 captures a moving image or when there are multiple actual images captured at a predetermined interval.

[0070] The upper part of Figure 9 shows the example described above, in which an optimal HDRI is selected for a single real-life image. Specifically, this shows an example in which HDRI 91 is selected as the optimal HDRI for a single real-life image 81. Region 91S represents the region within HDRI 91 that is detected as a comparison region.

[0071] The middle section of Figure 9 shows an example in which an optimal HDRI is selected from three real-life images captured in a predetermined order in the time direction. Specifically, this shows an example in which HDRI91 is selected as the optimal HDRI from three real-life images 141 captured in the order of real-life images 141-1, 141-2, and 141-3. Real-life image 141-1 is the same image as real-life image 81 in the upper section, and region 91S of HDRI91 is a region within HDRI91 detected as a comparison region for real-life image 141-1. Region 91X of HDRI91 is a region detected as a comparison region for real-life image 141-3, and region 91Y of HDRI91 is a region detected as a comparison region for real-life image 141-2.

[0072] When selecting one optimal HDRI using multiple real-life images, the HDRI selection unit 61 detects a comparison area in each of the multiple real-life images and calculates the similarity for each comparison area image in this way. The HDRI selection unit 61 then calculates an overall similarity by multiplying the similarities of the multiple real-life images, and the HDRI determination unit 105 can determine one or more HDRIs with the highest overall similarity as the optimal HDRI among the multiple HDRIs to be compared.

[0073] The lower part of Figure 9 shows another example in which an optimal HDRI is selected from three real-life images captured in a predetermined order in the time direction. In this example, the HDRI selection unit 61 generates a single composite real-life image 151 by combining three real-life images 141, 141-1, 141-2, and 141-3, using stitching or the like. The HDRI selection unit 61 then selects an optimal HDRI for the generated composite real-life image 151 using the same method as was used for the single real-life image 81 described above. The example in the lower part of Figure 9 shows an example in which an HDRI 91 is selected as the optimal HDRI for the single composite real-life image 151. A region 91Z of the HDRI 91 represents an area within the HDRI 91 detected as a comparison region corresponding to the angle of view of the composite real-life image 151.

[0074] As described above, the HDRI selection unit 61 can select the optimal HDRI for a plurality of real-life images. Even when selecting the optimal HDRI for a plurality of real-life images, it is possible to more easily select the HDRI that is suitable for the real scene.

[0075] 7. Computer Configuration Example The series of processes executed by the information processing device 22 can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed in a computer. Here, the computer includes a microcomputer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0076] FIG. 10 is a block diagram showing an example of the hardware configuration of a computer as the information processing device 22 when the above-described series of processes are executed by a program.

[0077] The computer 200 includes a central processing unit (CPU) 201, a read-only memory (ROM) 202, and a random access memory (RAM) 203. The CPU 201, the ROM 202, and the RAM 203 are connected to one another by a bus 204.

[0078] An input / output interface 205 is also connected to the bus 204. An input unit 206, an output unit 207, a storage unit 208, a communication unit 209, and a drive 210 are connected to the input / output interface 205.

[0079] The input unit 206 includes a keyboard, mouse, microphone, touch panel, input terminal, etc. The output unit 207 includes a display, speaker, output terminal, etc. The storage unit 208 includes a hard disk, SSD (Solid State Drive), RAM disk, non-volatile memory, etc. The communication unit 209 includes a network interface, etc. The drive 210 drives removable media 211 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0080] In the computer 200 configured as above, the CPU 201 performs the above-described series of processes by, for example, loading a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204 and executing the program. The RAM 203 also stores data and the like necessary for the CPU 201 to execute various processes.

[0081] The program executed by the CPU 201 of the computer 200 can be provided by being recorded on a removable medium 211 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0082] In the computer 200, the program can be installed in the storage unit 208 via the input / output interface 205 by inserting the removable medium 211 into the drive 210. The program can also be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. Alternatively, the program can be installed in the ROM 202 or the storage unit 208 in advance.

[0083] 8. Configuration Example of Second Embodiment FIG. 11 is a block diagram showing a configuration example of a camera according to a second embodiment of the present disclosure.

[0084] The camera 21EX in Fig. 11 has all the configurations of the camera 21 and the information processing device 22 shown in Fig. 2. The camera 21EX alone performs the above-mentioned generation of the real-life image, selection of the HDRI optimal for the real-life image, rendering of the CG object using the selected HDRI as an environment map, and generation of the real-life CG composite image by combining the real-life image and the CG object.

[0085] As described above, the camera 21EX includes the configuration of the information processing device 22 in addition to the configuration of the camera 21 shown in Fig. 2, and therefore can easily perform operations from capturing a real-life image to generating a real-life CG composite image, including selecting an HDRI that is optimal for the real-life image, using only the camera. Therefore, it is possible to more easily select an HDRI that is appropriate for the real scene.

[0086] It should be noted that both or either one of the HDRI storage unit 62 and the CG model storage unit 64 may be located not within the camera 21EX but in another device connected via a network, such as a local server or a cloud server.

[0087] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technology of the present disclosure.

[0088] For example, in the first and second embodiments described above, an example was described in which the image used as the environment map is an HDRI with a wide dynamic range, and an HDRI suitable for a real-life image captured of a real scene is selected. However, the image used as the environment map may also be an image called a Low Dynamic Range Image (LDRI), which has a normal 256-level range. In other words, this technology can be applied to any process of selecting an image to be used as an environment map from a large number of images, regardless of the gradation level of the environment map.

[0089] For example, the technology of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0090] Each step described in the above flowchart can be executed by one device or can be shared and executed by multiple devices. Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0091] The steps described in the flowchart may be performed in chronological order in the order described, but they do not necessarily have to be processed in chronological order and may be performed in parallel or at any time required, such as when a call is made.

[0092] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0093] The effects described in this specification are merely examples and are not intended to be limiting, and there may be effects other than those described in this specification.

[0094] The technology disclosed herein may employ the following configurations. (1) An information processing device comprising: a comparison area detection unit that detects, for each of a plurality of images, an area corresponding to an angle of view of a real-life image as a comparison area; a similarity calculation unit that calculates, for each of the plurality of images, a similarity between a comparison area image that is an image of the comparison area and the real-life image; and a determination unit that determines one or more of the images based on the calculated similarity. (2) The similarity calculation unit calculates the similarity between the comparison area image and the real-life image using a plurality of indicators, and calculates a final similarity between the comparison area image and the real-life image from the similarities of the plurality of indicators, and the determination unit determines one or more of the images based on the final similarity. (3) The information processing device described in (2) above, in which the similarity calculation unit calculates the final similarity by multiplying the similarities of the plurality of indicators. (4) The information processing device described in any of (2) to (3), in which the plurality of indicators include at least one of location, time period, season, weather, color distribution, or solar position. (5) The information processing device according to any one of (2) to (4), wherein the similarity calculation unit calculates the probability of each index for each of the comparison area image and the real-life image, and calculates the similarity of each index using Cosine similarity, where the probability of each index is a vector. (6) The information processing device according to any one of (1) to (5), wherein the comparison area detection unit detects the comparison area of ​​the image using metadata of the image and the real-life image. (7) The information processing device according to (6), wherein the comparison area detection unit detects the comparison area of ​​the image using orientation information and an angle of view of the metadata. (8) The information processing device according to any one of (1) to (7), wherein the comparison area detection unit detects the comparison area of ​​the image based on light source positions of the image and the real-life image. (9) The information processing device according to any one of (1) to (8), further comprising a candidate search unit that searches for an image to be compared from among the plurality of images based on labels assigned to the images, and the comparison area detection unit detects the comparison area for the image searched for by the candidate search unit.(10) The information processing device according to any of (1) to (9), wherein the similarity calculation unit calculates an overall similarity by multiplying the similarities of the plurality of real-life images, and the determination unit determines one or more of the images based on the overall similarity. (11) The information processing device according to any of (1) to (10), wherein the similarity calculation unit calculates a similarity between a composite real-life image obtained by combining the plurality of real-life images and the comparison region image corresponding to an angle of view of the composite real-life image, and the determination unit determines one or more of the images based on the similarity. (12) An information processing method comprising: for each of a plurality of images, detecting a region corresponding to an angle of view of the real-life image as a comparison region; calculating a similarity between the comparison region image that is an image of the comparison region and the real-life image for each of the plurality of images; and determining one or more of the images based on the calculated similarity.

[0095] 1 Information processing system, 21, 21EX camera, 22 Information processing device, 41 Image generation unit, 42 Lens data generation unit, 43 Attitude data generation unit, 44 Time information generation unit, 45 Latitude, longitude, and orientation information generation unit, 46 Metadata generation unit, 61 HDRI selection unit, 62 HDRI storage unit, 63 CG synthesis unit, 64 CG model storage unit, 101 Candidate search unit, 102 Comparison area detection unit, 103 Comparison area extraction unit, 104 Similarity calculation unit, 105 HDRI determination unit, 200 Computer, 201 CPU, 202 ROM, 203 RAM, 206 Input unit, 207 Output unit, 208 Storage unit, 209 Communication unit, 210 Drive, 211 Removable media

Claims

1. An information processing device comprising: a comparison area detection unit that detects an area corresponding to the angle of view of a real-life image as a comparison area for each of a plurality of images; a similarity calculation unit that calculates the similarity between a comparison area image, which is an image of the comparison area, and the real-life image for each of the plurality of images; and a determination unit that determines one or more of the images based on the calculated similarity.

2. The information processing device of claim 1, wherein the similarity calculation unit calculates the similarity between the comparison area image and the real-life image using a plurality of indices, and calculates a final similarity between the comparison area image and the real-life image from the similarity of the plurality of indices, and the determination unit determines one or more of the images based on the final similarity.

3. The information processing device according to claim 2, wherein the similarity calculation unit calculates the final similarity by multiplying the similarities of a plurality of indices.

4. The information processing device according to claim 2, wherein the plurality of indices include at least one of location, time period, season, weather, color distribution, and solar position.

5. The information processing device according to claim 2, wherein the similarity calculation unit calculates the probability of each index for each of the comparison area image and the actual image, and calculates the similarity of each index using Cosine similarity, where the probability of each index is used as a vector.

6. The information processing device according to claim 1, wherein the comparison area detection unit detects the comparison area of ​​the image using metadata of the image and the real image.

7. The information processing device according to claim 6, wherein the comparison area detection unit detects the comparison area of ​​the image using orientation information and angle of view of the metadata.

8. The information processing device according to claim 1, wherein the comparison area detection unit detects the comparison area of ​​the image based on light source positions of the image and the real image.

9. An information processing device as described in claim 1, further comprising a candidate search unit that searches for an image to be compared from among the plurality of images based on a label assigned to the image, and wherein the comparison area detection unit detects the comparison area for the image searched for by the candidate search unit.

10. The information processing device according to claim 1, wherein the similarity calculation unit calculates an overall similarity by multiplying the similarities of a plurality of the real-life images, and the determination unit determines one or more of the images based on the overall similarity.

11. The information processing device according to claim 1, wherein the similarity calculation unit calculates the similarity between a composite live-action image obtained by combining a plurality of the live-action images and the comparison area image corresponding to the angle of view of the composite live-action image, and the determination unit determines one or more of the images based on the similarity.

12. An information processing method comprising: detecting, for each of a plurality of images, an area corresponding to the angle of view of the actual image as a comparison area; calculating, for each of the plurality of images, a similarity between a comparison area image, which is an image of the comparison area, and the actual image; and determining one or more of the images based on the calculated similarity.

Images