Information processing device and information processing method

Abstract

The present technology pertains to an information processing device and an information processing method which, in imaging by virtual production, make it possible to more suitably reproduce a reflection on the surface of a subject. The information processing device using the technology comprises: an acquisition unit which acquires the view angle and the position and attitude of a camera for imaging a subject in a state in which a video displayed on a display is set as a background; an identification unit which identifies an inner frustum and an outer frustum in the video on the basis of the view angle and the position and attitude of the camera, and identifies, on the basis of reflection information prepared in advance, a reflection outer frustum region which includes a region to be reflected on the surface of the subject, in the outer frustum; and a rendering control unit which sets the resolution in the reflection outer frustum region to be higher than the resolution in a non-reflection outer frustum region, which is a region in the outer frustum excluding the reflection outer frustum region. The present technology can be applied to, for example, an imaging system for use in virtual production.

Description

Information processing device and information processing method

[0001] The present technology relates to an information processing device and an information processing method, and in particular to an information processing device and an information processing method that enable more suitable reproduction of reflections on the surface of a subject in virtual production shooting.

[0002] In recent years, a technique called virtual production (in-camera VFX) that uses a large LED (Light Emitting Diode) display has become known as a shooting technique for producing video content such as movies and dramas.

[0003] In virtual production, an image displayed on an LED display is used as the background, and a subject placed in front of the LED display is used as the foreground. In virtual production, an image is rendered based on the camera position and orientation, lens profile, etc., and displayed on the LED display, so that an image equivalent to that captured in a real landscape is captured as the background (see, for example, Patent Document 1).

[0004] In the image displayed on the LED display, the area including the part photographed by the camera is called the inner frustum, and the area other than the inner frustum is called the outer frustum.

[0005] Special Publication No. 2022-554415

[0006] Conventionally, outer frustum is not captured by a camera and is mainly used as part of the reflection or lighting on the surface of a real object, so it is rendered at a lower resolution than inner frustum.

[0007] Even if a scene is shot against a real landscape and the landscape is reflected in detail on the subject, when the virtual production is used, the outer frustum is rendered at a low resolution, causing the landscape (outer frustum) to be blurred and reflected on the subject, thereby compromising the realism of the footage shot using virtual production.

[0008] The present technology has been developed in light of these circumstances, and makes it possible to more appropriately reproduce reflections on the surface of a subject when shooting using virtual production.

[0009] An information processing device according to a first aspect of the present technology includes an acquisition unit that acquires the position, attitude, and angle of view of a camera that captures an image of a subject against a background of an image displayed on a display; an identification unit that identifies, based on the position, attitude, and angle of view of the camera, an inner frustum in the image that corresponds to the camera's imaging range and an outer frustum in the image that corresponds outside the camera's imaging range, and that identifies, based on pre-prepared reflection information, a reflection outer frustum region of the outer frustum that includes an area that is reflected on the surface of the subject; and a rendering control unit that sets the resolution of the reflection outer frustum region higher than the resolution of a non-reflection outer frustum region, which is the area of ​​the outer frustum other than the reflection outer frustum region.

[0010] An information processing method according to a first aspect of the present technology includes acquiring a position, attitude, and angle of view of a camera that captures an image of a subject against a background of an image displayed on a display; identifying, based on the position, attitude, and angle of view of the camera, an inner frustum within the image that corresponds to the camera's imaging range and an outer frustum within the image that corresponds outside the camera's imaging range; identifying, based on pre-prepared reflection information, a reflection outer frustum region of the outer frustum that includes an area that is reflected on the surface of the subject; and setting a resolution of the reflection outer frustum region higher than a resolution of a non-reflection outer frustum region, which is a region of the outer frustum other than the reflection outer frustum region.

[0011] An information processing method according to a second aspect of the present technology includes identifying a reflective outer frustum region, including a region that is reflected on the surface of the subject, from an outer frustum in an image that corresponds to outside the shooting range of a camera that shoots an image of the subject with the image displayed on a display as the background, and generating reflection information, which is information referenced by an information processing device that sets the resolution of the reflective outer frustum region to be higher than the resolution of a non-reflective outer frustum region, which is a region of the outer frustum other than the reflective outer frustum region, and is information for identifying the reflective outer frustum region.

[0012] In a first aspect of the present technology, the position, attitude, and angle of view of a camera that captures an image of a subject against the background of an image displayed on a display are acquired, and based on the position, attitude, and angle of view of the camera, an inner frustum in the image that corresponds to the camera's capture range and an outer frustum in the image that corresponds outside the camera's capture range are identified, and based on pre-prepared reflection information, a reflection outer frustum region of the outer frustum that includes an area that is reflected on the surface of the subject is identified, and the resolution of the reflection outer frustum region is set higher than the resolution of a non-reflection outer frustum region, which is the area of ​​the outer frustum other than the reflection outer frustum region.

[0013] In a second aspect of the present technology, a reflective outer frustum area including an area reflected on the surface of the subject is identified from an outer frustum in the image corresponding to an area outside the shooting range of a camera that shoots a subject with the image displayed on a display as the background, and reflective information is generated, which is information referenced by an information processing device that sets the resolution of the reflective outer frustum area to be higher than the resolution of a non-reflective outer frustum area, which is an area of ​​the outer frustum other than the reflective outer frustum area, and is information for identifying the reflective outer frustum area.

[0014] 1 is a diagram illustrating an overview of a photography system to which the present technology is applied. FIG. 1 is a diagram illustrating an example of a photographed image photographed by a camera. FIG. 2 is a diagram illustrating an example of a background image displayed on a display. FIG. 1 is a first diagram illustrating details of the background image. FIG. 2 is a second diagram illustrating details of the background image. FIG. 3 is a diagram comparing reflection of a scenery on the surface of a subject when photography is performed with an actual scenery as the background and when photography is performed using virtual production. FIG. 4 is a diagram illustrating examples of a reflection outer frustum region and a non-reflection outer frustum region. FIG. 4 is a block diagram illustrating an example configuration of a photography system. FIG. 5 is a flowchart illustrating photography preparation processing performed by the photography system of the present technology. FIG. 6 is a diagram illustrating an example of a test chart. FIG. 7 is a diagram illustrating an example display of a test chart. FIG. 8 is a diagram illustrating an example of a camera position during preliminary photography. FIG. 9 is a diagram illustrating a method for evaluating the degree of reflection on a vehicle. FIG. 10 is a diagram illustrating an example of a table in which the degree of reflection of each unit image is recorded. FIG. 11 is a diagram illustrating examples of rendering parameters according to the degree of reflection of the unit image. FIG. 12 is a diagram illustrating examples of rendering parameters set for each camera position. FIG. 13 is a flowchart illustrating actual photography processing performed by the photography system of the present technology. FIG. 14 is a diagram illustrating an example of the relationship between a unit image and a reflection outer frustum region. FIG. 10 is a diagram showing the relationship between unit videos and nodes.

[0015] Hereinafter, embodiments of the present technology will be described in the following order: 1. Overview of the imaging system 2. Configuration and operation of the imaging system 3. Modified examples

[0016] 1. Overview of Imaging System FIG. 1 is a diagram illustrating an overview of an imaging system 1 to which the present technology is applied.

[0017] 1 is a system used for shooting, for example, virtual production (in-camera VFX). The shooting system 1 is composed of a camera 11, a video storage device (not shown) that stores the video captured by the camera 11, a wall-mounted display 12, and a controller (not shown) that controls the display 12.

[0018] The display 12 is configured as an LED display in which display units each having LEDs (Light Emitting Diodes) are arranged in a tiled pattern, and is placed in a real space such as a studio. The display 12 displays, for example, an image of a virtual space created using CG (Computer Graphics). The photographer P1 uses the camera 11 to capture an image of the subject, the motorcycle M1, with the image displayed on the display 12 as the background. Hereinafter, the image displayed on the display 12 will be referred to as the background image.

[0019] In this disclosure, "video" includes both still images and videos. Furthermore, "video" does not only refer to the state in which it is displayed on a display, but may also refer comprehensively to video data that is not displayed on a display. For example, in the embodiments, background video before being displayed on a display and captured video captured by a camera are not actually displayed video but are video data, but for convenience they will be referred to as "background video" or "captured video."

[0020] FIG. 2 is a diagram showing an example of a video image captured by the camera 11. As shown in FIG.

[0021] 2, the captured image captured by the camera 11 shows the motorcycle M1 as if it were present in the virtual space shown in the background image. In this way, by using the imaging system 1, the photographer P1 can capture in the studio an image in which the space shown in the background image appears to extend behind the motorcycle M1.

[0022] FIG. 3 is a diagram showing an example of a background image displayed on the display 12. As shown in FIG.

[0023] The background image displayed on the display 12 is configured by superimposing an inner frustum on an outer frustum, for example, as shown in FIG.

[0024] The inner frustum is an image that represents a scene seen from a virtual camera in a virtual space shown by the background image, the position and orientation of which corresponds to the position and orientation of the camera 11 in real space. The outer frustum is an image that is arranged around the inner frustum, and is, for example, an image that represents a scene seen from a virtual camera in a virtual space shown by the background image, the position and orientation of which corresponds to the position and orientation of the camera 11 in real space.

[0025] 3, when the camera 11 captures an image from the right side of the motorcycle M1 placed in the center, the inner frustum is displayed in the image capture area A1, which is an area on the display 12 included in the image capture range of the camera 11. In the image capture area A1, for example, an image of the background portion is displayed when an image is captured from the right side of the motorcycle M1 in the virtual space shown by the background image.

[0026] 3, when the camera 11 captures an image from the left side of the motorcycle M1 located in the center, the inner frustum is displayed in the image capturing area A1. For example, the image capturing area A1 displays an image of the background portion when the image is captured from the left side of the motorcycle M1 in the virtual space shown by the background image.

[0027] In this way, when taking a photograph, the photographer P1 takes a photograph while successively changing the position and orientation of the camera 11, the angle of view, the focus position, the aperture value, and the like.

[0028] The controller tracks the camera work (position and orientation) of the camera 11 and controls the position on the display 12 where the inner frustum is displayed in accordance with the camera work of the camera 11 .

[0029] In FIG. 3, for ease of explanation, the imaging area A1 is shown surrounded by a thick line, but in reality, the line surrounding the imaging area is not displayed.

[0030] Here, the background image will be described in detail with reference to FIGS.

[0031] 4 shows a schematic view of a studio in which displays 12A to 12C are installed so as to surround the left, right, and rear sides of a vehicle C1 as a subject. In the example of FIG. 4, each of the displays 12A to 12C is configured with four display units 21.

[0032] 4, the background image displayed on display 12A is divided into unit images UV0 to UV3 that are displayed on each of the four display units 21 that make up display 12A. The background image displayed on display 12B is divided into unit images UV4 to UV7 that are displayed on each of the four display units 21 that make up display 12A. The background image displayed on display 12C is divided into unit images UV8 to UV11 that are displayed on each of the four display units 21 that make up display 12A.

[0033] If the controller is configured as a network to which multiple nodes (e.g., PCs) are connected, the 12 unit images UV0 to UV11 are rendered, for example, by different nodes. Because multiple nodes share the rendering of the large-screen background image, the processing load per node for rendering is reduced, enabling real-time rendering of the background image. The number of unit images that make up the background image and the method for dividing the background image can be determined arbitrarily depending on the shape of the display 12 and the specifications of the controller.

[0034] Fig. 5 shows an example of division of a background image displayed on one display 12. In the example of Fig. 5, the background image is divided into 6 x 6 unit images.

[0035] 5, the inner frustum is composed of, for example, unit image UV4-3 in the third row and fourth column, unit image UV5-3 in the third row and fifth column, unit image UV4-4 in the fourth row and fourth column, unit image UV5-4 in the fourth row and fifth column, unit image UV5-4 in the fifth row and fourth column, and unit image UV5-5 in the fifth row and fifth column of the 6×6 unit images. The outer frustum is composed of the other unit images of the 6×6 unit images.

[0036] In order to obtain a video that looks as if it were shot in real space through virtual production, it is necessary to render the inner frustum photographed by the camera 11 at high resolution.

[0037] On the other hand, the outer frustum is not captured by a camera and is mainly used as a part of the illumination or reflection on the surface of the real subject, so it is rendered at a lower resolution than the inner frustum. Rendering the outer frustum at a lower resolution than the inner frustum reduces the processing load involved in rendering the entire background video.

[0038] FIG. 6 is a diagram comparing the reflection of scenery on the surface of a subject when shooting is performed using a real landscape as the background and when shooting is performed using virtual production.

[0039] 6A shows an example of a scene reflected in the vehicle C1 when an image is captured against a real landscape. When an image of the vehicle C1 is captured against a real landscape, the scene is reflected in detail, for example, on the side window of the vehicle C1, as shown in FIG. 6. The scene is also reflected in a blurred manner, for example, on the door of the vehicle C1.

[0040] An example of a scene reflected in the vehicle C1 when filming is performed using virtual production is shown in B of Fig. 6. When filming is performed using virtual production, the outer frustum is rendered at a low resolution, so that the scene displayed on the display 12A (the outer frustum) is reflected blurred on both the side window and the door of the vehicle C1, as shown in B of Fig. 6.

[0041] In this way, rendering the entire outer frustum at a uniformly low resolution would compromise the realism of the footage obtained through virtual production (in other words, whether the footage appears to be footage that was actually shot in the virtual space).

[0042] One embodiment of the present technology was conceived with the above points in mind, and proposes a technology that can reduce the processing load associated with rendering the entire background image while maintaining the realism of the captured image.

[0043] Specifically, the controller acquires the position, orientation, and angle of view of the camera 11 that captures an image of a subject against the background image displayed on the display, and, based on the position, orientation, and angle of view of the camera 11, identifies an inner frustum in the background image that corresponds to the imaging range of the camera 11 and an outer frustum in the background image that corresponds outside the imaging range of the camera 11. The controller identifies a reflection outer frustum region based on reflection information prepared in advance, and sets the resolution of the reflection outer frustum region higher than the resolution of the non-reflection outer frustum region.

[0044] The glare outer frustum region is a region of the outer frustum that includes a region that is glare on the surface of the subject, and the non-glare outer frustum region is a region of the outer frustum other than the glare outer frustum region.

[0045] FIG. 7 is a diagram showing an example of a glare outer frustum region and a non-glare outer frustum region.

[0046] 7, of the unit images that make up the outer frustum, the unit image UV2-4 in the fourth row and second column is set to be a reflection outer frustum region, and the other unit images are set to be non-reflection outer frustum regions. In other words, when viewed from the camera 11, only at least a part of the unit image UV2-4, of the unit images that make up the outer frustum, appears to be reflected on the surface of the subject.

[0047] In this case, the controller renders the unit image UV2-4, which is the reflection outer frustum region, at a higher resolution than the other unit images, which are the non-reflection outer frustum region. By rendering the unit image UV2-4 at a higher resolution, the controller can make the scenery shown in the unit image UV2-4 appear in the subject in detail, while rendering the other unit images that make up the outer frustum at a lower resolution, thereby reducing the processing load associated with rendering the entire background image.

[0048] 2. Configuration and Operation of the Photography System Configuration of the Photography System FIG. 8 is a block diagram showing an example of the configuration of the photography system 1. As shown in FIG.

[0049] 8 performs preliminary shooting before the actual shooting, performs shooting at various positions, orientations, and angles of view, checks the reflection outer frustum area, and generates a database indicating the relationship between the position, orientation, and angle of view and the reflection outer frustum area as reflection information. Then, in the actual shooting, the shooting system 1 renders the inner frustum and also renders the outer frustum based on the database generated in advance in the preliminary shooting.

[0050] As shown in FIG. 8, the photographing system 1 includes a camera 11, a display 12, a tracking unit 51, and a controller 52.

[0051] The camera 11 includes a photographing unit 61 and a camera setting value transmitting unit 62 .

[0052] The photographing unit 61 photographs a subject against the background image displayed on the display 12 to obtain a photographed image. The photographing unit 61 transmits the photographed image to the controller 52.

[0053] The camera setting value transmission unit 62 acquires the current camera setting values ​​from the photographing unit 61, and transmits information indicating the current camera setting values ​​to the controller 52. The camera setting values ​​include the angle of view, focus, focal length, and the like.

[0054] The tracking unit 51 includes a camera position estimation unit 71 and a camera position transmission unit 72 .

[0055] The camera position estimation unit 71 estimates the current position and orientation of the camera 11. In the present disclosure, the position and orientation of the camera 11 may be referred to as the camera position. The current camera position is estimated based on the results of tracking using, for example, an IR (Infrared) camera and markers formed of a retroreflective material. The camera position estimation unit 71 supplies information indicating the current camera position to the camera position transmission unit 72.

[0056] The camera position transmitting unit 72 transmits the information indicating the current camera position supplied from the camera position estimating unit 71 to the controller 52 .

[0057] The controller 52 is configured by an information processing device such as a PC, and includes a test chart display control unit 81, a camera position acquisition unit 82, a camera setting value acquisition unit 83, a rendering parameter setting unit 84, a rendering parameter recording unit 85, a rendering control unit 86, and a rendering unit 87.

[0058] The test chart display control unit 81 supplies a test chart to the display 12. The test chart is an image that is displayed on the display 12 during the preliminary shooting.

[0059] The camera position acquisition unit 82 supplies the information indicating the camera position supplied from the tracking unit 51 to the rendering parameter setting unit 84 or the rendering control unit 86 .

[0060] The camera setting value acquisition unit 83 supplies information indicating the camera setting values ​​supplied from the camera 11 to the rendering parameter setting unit 84 or the rendering control unit 86 .

[0061] The rendering parameter setting unit 84 acquires a captured image obtained by the camera 11 capturing an image of a subject with the test chart displayed on the display 12. The rendering parameter setting unit 84 identifies unit images that make up the reflection outer frustum region based on the captured image, and generates a database that indicates the relationship between the position and orientation and angle of view of the camera 11 and the reflection outer frustum region.

[0062] Specifically, the rendering parameter setting unit 84 detects the degree to which each unit image constituting the background image is reflected on the surface of the subject based on the captured image, and sets rendering parameters for each unit image based on the degree to which each unit image is reflected on the surface of the subject.

[0063] The rendering parameters indicate the ratio of the resolution of each unit image to a reference resolution (standard resolution). For example, a high value is set as the rendering parameter for a unit image that is a reflection outer frustum region, and a low value is set as the rendering parameter for a unit image that is a non-reflection outer frustum region.

[0064] The rendering parameter setting unit 84 associates the rendering parameters for each unit image with information indicating the current camera position supplied from the camera position acquisition unit 82 and information indicating the current camera setting values ​​supplied from the camera setting value acquisition unit 83, and records them in the rendering parameter recording unit 85.

[0065] The rendering parameter recording unit 85 records the rendering parameters in the form of a database, in association with the camera positions and camera setting values.

[0066] The rendering control unit 86 functions as an identification unit that identifies the unit images that make up the inner frustum and the unit images that make up the outer frustum within the background image based on information indicating the current camera position supplied from the camera position acquisition unit 82 and information indicating the current camera setting values ​​supplied from the camera setting value acquisition unit 83.

[0067] The rendering control unit 86 controls the rendering unit 87 to set the resolution of each unit image that constitutes the inner frustum higher than the resolution of each unit image that constitutes the outer frustum.

[0068] The rendering control unit 86 identifies unit images that make up the specular outer frustum region based on the current camera position, the current camera setting values, and the database recorded in the rendering parameter recording unit 85. The rendering control unit 86 controls the rendering unit 87 to set the resolution of each unit image that makes up the specular outer frustum region higher than the resolution of each unit image that makes up the non-specular outer frustum region.

[0069] Specifically, the rendering control unit 86 refers to the database recorded in the rendering parameter recording unit 85 to obtain rendering parameters for each unit video that are associated with the current camera position and the current camera setting values.

[0070] The rendering control unit 86 sets the resolution of each unit image constituting the outer frustum based on the rendering parameters acquired from the database. For example, the rendering unit 87 sets the resolution of each unit image constituting the outer frustum to a value obtained by multiplying the reference resolution by a ratio indicated by the rendering parameters.

[0071] The rendering unit 87 is configured as a processor such as a GPU (Graphics Processing Unit), and corresponds to a node that renders at least a part of the background image (unit image). If the controller 52 is configured as a network to which multiple nodes are connected, the background image is shared and rendered by the multiple nodes (rendering units).

[0072] The rendering unit 87 places a 3D model representing the three-dimensional shape of the background in a three-dimensional space. A virtual display 12 is placed in the three-dimensional space. The rendering unit 87 places a virtual camera in the three-dimensional space based on the current camera position and camera setting values, and renders the inner frustum and the outer frustum by projecting the 3D model as seen from the virtual camera onto the display surface of the virtual display 12.

[0073] Here, the rendering unit 87 renders each unit image constituting the inner frustum and each unit image constituting the outer frustum at the resolution set by the rendering control unit 86 .

[0074] The rendering unit 87 generates a background image by superimposing the inner frustum on the outer frustum, and supplies the background image to the display 12. The background image is displayed on the display 12 during actual shooting.

[0075] The display 12 displays a test chart or background image supplied from the controller 52 .

[0076] Operation of the photography system 1 The processing performed by the photography system 1 of the present technology includes a photography preparation process in which, as preliminary photography, a test chart is displayed on the display 12, photography is performed, and a database is created, and an actual photography process in which, as actual photography, a background image is displayed on the display 12 and photography is performed.

[0077] The photographing preparation process performed by the photographing system 1 of the present technology will be described with reference to the flowchart of FIG.

[0078] In step S1, the display 12 displays a test chart.

[0079] FIG. 10 is a diagram showing an example of a test chart.

[0080] The test chart shown in FIG. 10 is configured by drawing one or more unit numbers Un that can identify unit images (display units that display unit images) on a resolution chart defined by, for example, ISO12233.

[0081] 10, five vertical lines are drawn at the top of the resolution chart, with the thickness and spacing between the lines decreasing toward the center, nine vertical lines are drawn at the bottom of the resolution chart, five horizontal lines are drawn at the left of the resolution chart, and nine horizontal lines are drawn at the right of the resolution chart, with the thickness and spacing between the lines decreasing toward the center.

[0082] FIG. 11 is a diagram showing an example of a test chart display.

[0083] 11, test charts on which unit numbers Un are drawn are displayed as unit images UV0 to UV11 on displays 12A to 12C. For example, a test chart on which 0 is drawn is displayed as unit image UV0, and a test chart on which 1 is drawn is displayed as unit image UV1.

[0084] Note that the test chart need not necessarily include a unit number as long as it can identify the unit video. For example, a figure that can identify the unit video may be drawn on the test chart, or the same test chart may be displayed sequentially on each display unit.

[0085] 9 , in step S2, the photographing unit 61 of the camera 11 photographs the subject with the test chart displayed on the display 12, and acquires the photographed image. The camera position acquisition unit 82 and the camera setting value acquisition unit 83 of the controller 52 acquire information indicating the current camera position and information indicating the current camera setting values.

[0086] In step S3, the rendering parameter setting unit 84 of the controller 52 detects the degree of reflection of each unit image on the surface of the subject based on the captured image captured by the imaging unit 61, and records the degree of reflection of each unit image in the form of a table.

[0087] In step S4, the rendering parameter setting unit 84 sets rendering parameters for each unit video based on the degree of reflection of each unit video.

[0088] In step S5, the rendering parameter recording unit 85 records the rendering parameters for each unit video in association with information indicating the current camera position and information indicating the current camera setting values ​​in the database.

[0089] In step S6, the controller 52 determines whether or not to end the preliminary photographing.

[0090] If it is determined in step S6 that the preliminary photographing is to be completed, the photographing preparation process is completed.

[0091] On the other hand, if it is determined in step S6 that the preliminary photographing should not be ended, the camera position and camera setting values ​​are changed, and then the process returns to step S2, and the subsequent processes are carried out.

[0092] For example, as a preliminary image capture, images are captured at three camera positions Pos1 to Pos3 as shown in Fig. 12. Camera position Pos1 is a camera position for capturing an image of a vehicle C1 as a subject from a diagonally forward right direction, camera position Pos2 is a camera position for capturing an image of the vehicle C1 from a right direction, and camera position Pos3 is a camera position for capturing an image of the vehicle C1 from a diagonally rear right direction.

[0093] In Figure 12, dashed lines indicate the shooting range of the camera 11 at each camera position, and dashed lines indicate the trajectory of the camera 11 during preliminary shooting. In the following, it is assumed that the focus of the camera 11 is always set to the vehicle C1, and the angle of view of the camera 11 is constant. In Figure 12, display 12A is disposed so as to face the left side of the vehicle C1, and display 12B is disposed so as to face the front of the vehicle C1.

[0094] The rendering parameter setting unit 84 detects the degree of reflection of each unit image on the surface of the vehicle C1 based on the captured images captured at the camera positions Pos1 to Pos3. The degree of reflection on the surface of the vehicle C1 is evaluated on a scale of 0 to 9, for example.

[0095] The camera positions at which the reflection degree of each unit image is detected may be determined in advance by the photographer P1, or may be determined during preliminary shooting. The camera positions at which the reflection degree of each unit image is detected may be sampled at predetermined intervals. The number of camera positions at which the reflection degree of each unit image is detected may be determined based on the number of unit images.

[0096] FIG. 13 is a diagram illustrating a method for evaluating the degree of reflection on the surface of the vehicle C1.

[0097] The rendering parameter setting unit 84 detects the number of vertical and horizontal lines in the test chart that are reflected on the surface of the vehicle C1 in the captured image, and uses the number of vertical and horizontal lines detected in the test chart as a measure of the degree of reflection of the test chart (unit image). For example, if the rendering parameter setting unit 84 can detect, in the captured image, nine vertical or horizontal lines drawn in the test chart within the area surrounded by the dashed line in Figure 13 on the test chart, the degree of reflection of the unit image corresponding to that test chart is set to 9.

[0098] The degree of reflection of the unit image may be evaluated based on the definition of the test chart reflected on the surface of the vehicle C1 in the captured image. The degree of reflection of the unit image may also be evaluated by the photographer P1.

[0099] FIG. 14 is a diagram showing an example of a table in which the degree of reflection of each unit image is recorded.

[0100] 14, the image displayed on display 12A is divided into nine unit images, and the image displayed on display 12B is divided into four unit images. Here, the unit images constituting the image displayed on display 12A are assigned unit numbers from 0 to 8, and the unit images constituting the image displayed on display 12B are assigned unit numbers from 9 to 12.

[0101] In FIG. 14A, the unit numbers 8, 11, and 12 are shown in gray, which indicates that the unit images assigned with the unit numbers 8, 11, and 12 are reflected in detail on the hood and windshield of the vehicle C1.

[0102] The rendering parameter setting unit 84 records the degree of reflection of the vehicle C1 in each unit image detected based on the images captured at the camera positions Pos1 to Pos3 in the table shown in FIG. 14B.

[0103] The reflection degree of each unit image is recorded for each camera position in the table in Fig. 14B. In the table in Fig. 14B, for camera positions Pos1 and Pos2, the reflection degree of unit images assigned unit numbers 8, 9, and 10 is set to 9, and the reflection degree of unit images assigned other unit numbers is set to 0.

[0104] Since the hood and windshield of vehicle C1 are hardly visible in the image captured at camera position Pos3, in the table B of Figure 14, the degree of reflection of unit images assigned unit numbers 0 to 11 for camera position Pos3 is set to 0.

[0105] The rendering parameter setting unit 84 sets appropriate rendering parameters according to the degree of reflection of each unit video, which are recorded in the table of FIG. 14B.

[0106] FIG. 15 is a diagram showing an example of rendering parameters according to the degree of reflection of a unit video.

[0107] As shown in the upper left of Figure 15, if the test chart is reflected in detail on the surface of the subject, for example, if nine to eight vertical or horizontal lines can be detected in the test chart reflected on the surface of vehicle C1 in the captured image, the rendering parameter setting unit 84 sets the value of the rendering parameter to 1.0, as shown on the right side of Figure 15.

[0108] If six vertical or horizontal lines out of seven in the test chart reflected on the surface of the vehicle C1 in the captured video are detected, the rendering parameter setting unit 84 sets the value of the rendering parameter to 0.8. If three vertical or horizontal lines out of five in the test chart reflected on the vehicle C1 in the captured video are detected, the rendering parameter setting unit 84 sets the value of the rendering parameter to 0.4.

[0109] As shown in the lower left of Figure 15, if the test chart is blurred and reflected in the subject, for example, if two to zero vertical or horizontal lines can be detected in the test chart reflected in the vehicle C1 in the captured image, the rendering parameter setting unit 84 sets the value of the rendering parameter to 0.2, as shown on the right side of Figure 15.

[0110] In this way, the higher the degree of reflection of the unit video, the higher the value set as the rendering parameter.

[0111] FIG. 16 is a diagram showing an example of rendering parameters set for each camera position.

[0112] The rendering parameter setting unit 84 sets rendering parameters for each unit image shown in the lower part of Figure 16 for each camera position based on a table in which the degree of reflection of each unit image shown in the upper part of Figure 16 is recorded for each camera position.

[0113] In the example at the bottom of Fig. 16, for camera positions Pos1 and Pos2, the rendering parameter values ​​for unit videos assigned unit numbers 8, 9, and 10 are set to 1.0, and the rendering parameter values ​​for unit videos assigned other unit numbers are set to 0.2. Also, in the example at the bottom of Fig. 16, for camera position Pos3, the rendering parameter values ​​for unit videos assigned unit numbers 0 to 11 are set to 0.2.

[0114] As described above, the rendering parameters for each unit video set for each camera position are recorded in the database, and thus the reflection information to be used during actual shooting is prepared.

[0115] Next, the actual shooting process performed by the shooting system 1 of the present technology will be described with reference to the flowchart of FIG.

[0116] In step S21, the camera position acquisition unit 82 and the camera setting value acquisition unit 83 of the controller 52 acquire information indicating the current camera position and information indicating the current camera setting values.

[0117] In step S22, the rendering control unit 86 of the controller 52 identifies the unit images that make up the inner frustum and the unit images that make up the outer frustum based on the current camera position and current camera setting values. The rendering control unit 86 controls the rendering unit 87 to set the resolution of the inner frustum.

[0118] In step S23, the rendering control unit 86 identifies the unit images that make up the reflected outer frustum region based on the current camera position, the current camera setting values, and the database recorded in the rendering parameter recording unit 85.

[0119] In step S24, the rendering control unit 86 controls the rendering unit 87 to set the resolution of the reflection outer frustum region and the non-reflection outer frustum region.

[0120] In step S25, the rendering unit 87 renders a background image. Specifically, the rendering unit 87 renders each unit image that constitutes the inner frustum at a higher resolution than each unit image that constitutes the outer frustum. The rendering unit 87 also renders each unit image that constitutes the specular outer frustum region at a higher resolution than each unit image that constitutes the non-specular outer frustum region. The rendering unit 87 generates a background image by superimposing the inner frustum on the outer frustum.

[0121] In step S26, the display 12 displays the background image.

[0122] In step S27, the photographing section 61 of the camera 11 photographs the subject while the background image is displayed on the display, and acquires the photographed image.

[0123] In step S28, the controller 52 determines whether or not to end the actual shooting.

[0124] If it is determined in step S28 that the actual shooting is not to be ended, the process returns to step S21, and the subsequent processes are carried out.

[0125] On the other hand, if it is determined in step S28 that the actual shooting is to be ended, the actual shooting process is ended.

[0126] As described above, in the imaging system 1 of the present technology, the position, orientation, and angle of view of the camera 11 that images a subject against the background video displayed on the display 12 are acquired, and an inner frustum in the background video that corresponds to the imaging range of the camera and an outer frustum in the background video that corresponds to outside the imaging range of the camera 11 are identified based on the position, orientation, and angle of view of the camera 11. In the imaging system 1 of the present technology, the reflection outer frustum region is identified based on reflection information prepared in advance, and the resolution of the reflection outer frustum region is set higher than the resolution of the non-reflection outer frustum region.

[0127] By rendering the non-reflecting outer frustum region in the background video at a low resolution while allowing the scenery, etc., indicated by the reflecting outer frustum region in the background video to be reflected in detail onto the subject, it is possible to reduce the processing load associated with rendering the entire background video. Note that because the reflecting outer frustum region is identified based on prepared reflecting information, the rendering control unit 86 can identify the reflecting outer frustum region in a short time compared to when the reflecting outer frustum region is identified in real time based on the captured video.

[0128] 3. Modifications FIG. 18 is a diagram showing an example of the relationship between a unit image and a glare outer frustum region.

[0129] When a reflection outer frustum region spans multiple unit images, only the reflection outer frustum region in each unit image may be rendered at high resolution. In the example of Figure 18, the lower right part of unit image UV1-3, the lower left part of unit image UV2-3, the upper right part of unit image UV1-4, and the upper left part of unit image UV2-4 are rendered at high resolution, and the other parts of unit image UV1-3, unit image UV2-3, unit image UV1-4, and unit image UV2-4 are rendered at low resolution.

[0130] Similarly, if the reflection outer frustum region is narrower than the unit image, only the reflection outer frustum region within the unit image may be rendered at high resolution. In the example of Figure 18, the central part of the unit image UV2-5 is rendered at high resolution, and the other parts of the unit image UV2-5 are rendered at low resolution.

[0131] FIG. 19 is a diagram showing an example of the relationship between a unit image and an inner frustum and an outer frustum.

[0132] As shown in Fig. 19, one unit image may include an inner frustum and an outer frustum. In the example of Fig. 19, the right portions of unit images UV4-3 to UV4-5, the entire unit images UV5-3 to UV5-5, and the left portions of unit images UV6-3 to UV6-5, which constitute the inner frustum, are rendered at high resolution. On the other hand, the left portions of unit images UV4-3 to UV4-5 and the right portions of unit images UV6-3 to UV6-5, which constitute the outer frustum, are rendered at low resolution.

[0133] In the example of Figure 19, the lower part of unit image UV2-3 and the upper part of unit image UV2-4, which constitute the reflective outer frustum area, are rendered at high resolution, and the upper part of unit image UV2-3 and the lower part of unit image UV2-4, which constitute the non-reflective outer frustum area, are rendered at low resolution.

[0134] Also, in the example of Figure 19, the right central portion of unit image UV2-5 and the left central portion of unit image UV3-5, which constitute the reflective outer frustum region, are rendered at high resolution, and the other portions of unit image UV2-5 and unit image UV3-5, which constitute the non-reflective outer frustum region, are rendered at low resolution.

[0135] As described above, the inner frustum and outer frustum and the reflection outer frustum area and non-reflection outer frustum area may be specified in units of pixels rather than in units of unit images.

[0136] FIG. 20 is a diagram showing the relationship between unit videos and nodes.

[0137] As shown in FIG. 20, one unit video may be rendered by multiple nodes, or one node may render multiple unit videos.

[0138] In the example of Figure 20, the unit image LT that constitutes the upper left part of the background image is generated by combining viewport VPLT_0 rendered at node 0, viewport VPLT_2 rendered at node 2, and viewport VPLT_3 rendered at node 3.

[0139] The unit video RT that constitutes the upper right portion of the background video is generated by combining the viewport VPRT_1 rendered at node 1 and the viewport VPRT_3 rendered at node 3.

[0140] The unit image LB that constitutes the lower left part of the background image is generated by combining viewport VPLB_1 rendered at node 1, viewport VPLB_2 rendered at node 3, and viewport VPLB_3 rendered at node 3.

[0141] The unit video RB that forms the upper right portion of the background video is composed only of viewport VPRB_3 rendered by node 3.

[0142] For example, one unit image constituting a non-reflecting outer frustum region may be rendered by one node, and one unit image constituting a reflecting outer frustum region may be shared and rendered by multiple nodes. In other words, there is a one-to-many relationship between the nodes and the reflecting outer frustum region to be rendered. This configuration makes it possible to shorten the rendering time for the entire outer frustum. It should be noted that in this disclosure, the expressions "one unit image" or "one reflecting outer frustum" are used to express a one-to-many relationship with the node, and are not used to limit the expression to "only one unit image" or "only one reflecting outer frustum."

[0143] The database recording the rendering parameters may be generated not by performing preliminary shooting, but by simulation such as ray tracing using a 3D model of the studio. In this case, it becomes unnecessary to use the studio for preliminary shooting, thereby improving the efficiency of studio utilization.

[0144] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0145] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0146] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0147] For example, the present technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by a plurality of devices via a network.

[0148] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0149] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0150] <Examples of Combinations of Configurations> The present technology can also have the following configurations.

[0151] (1) An information processing device comprising: an acquisition unit that acquires a position, orientation, and angle of view of a camera that captures an image of a subject against a background of an image displayed on a display; an identification unit that identifies, based on the position, orientation, and angle of view of the camera, an inner frustum in the image that corresponds to the camera's imaging range and an outer frustum in the image that corresponds outside the camera's imaging range, and that identifies, based on pre-prepared reflection information, a reflection outer frustum region of the outer frustum that includes a region that is reflected on the surface of the subject; and a rendering control unit that sets a resolution of the reflection outer frustum region higher than a resolution of a non-reflection outer frustum region of the outer frustum that is a region other than the reflection outer frustum region. (2) The information processing device described in (1), wherein the reflection information is information indicating a relationship between the position, orientation, and angle of view of the camera and the reflection outer frustum region. (3) The information processing device described in (1) or (2), wherein the reflection information is information based on a degree of reflection on the surface of the subject for each of a plurality of regions into which the outer frustum is divided. (4) The information processing device according to (3), wherein the rendering control unit sets a higher resolution for the regions into which the outer frustum is divided as the degree of reflection on the surface of the subject increases. (5) The information processing device according to (4), wherein the reflection information is information indicating a ratio of the resolution of each of the plurality of regions into which the outer frustum is divided to a reference resolution. (6) The information processing device according to any one of (1) to (5), further comprising a rendering unit that renders at least a portion of the image in accordance with control by the rendering control unit. (7) The information processing device according to (6), wherein one reflection outer frustum region is shared and rendered by a plurality of the rendering units.(8) An information processing method including: acquiring the position, attitude, and angle of view of a camera that captures an image of a subject against a background of an image displayed on a display; identifying, based on the position, attitude, and angle of view of the camera, an inner frustum in the image that corresponds to the camera's imaging range and an outer frustum in the image that corresponds outside the camera's imaging range; identifying, based on pre-prepared reflection information, a reflection outer frustum region of the outer frustum that includes an area that is reflected on the surface of the subject; and setting the resolution of the reflection outer frustum region higher than the resolution of a non-reflection outer frustum region, which is the area of ​​the outer frustum other than the reflection outer frustum region. (9) An information processing method comprising: identifying a reflection outer frustum region, the reflection outer frustum region including a region reflected on the surface of the subject, from an outer frustum in an image corresponding to an area outside the imaging range of a camera that captures an image displayed on a display as a background; and generating reflection information, the reflection outer frustum region being information referenced by an information processing device that sets a resolution of the reflection outer frustum region higher than a resolution of a non-reflection outer frustum region, which is an area of ​​the outer frustum other than the reflection outer frustum region. (10) The information processing method according to (9), in which the reflection outer frustum region is identified based on an image captured by the camera capturing the subject with the image displayed on the display. (11) The information processing method according to (10), further comprising acquiring a position and orientation and an angle of view of the camera, in which the reflection information is generated to indicate a relationship between the position and orientation and the angle of view of the camera and the reflection outer frustum region. (12) The information processing method according to (10) or (11), wherein the reflection outer frustum region is identified by detecting, based on the captured video, the degree of reflection on the surface of the subject for each of a plurality of regions into which the outer frustum is divided, and the reflection information is generated based on the degree of reflection.(13) The information processing method according to (12), wherein the image displayed on the display is an image in which regions into which the outer frustum is divided can be identified. (14) The information processing method according to (9), wherein the reflected outer frustum region is identified by simulation.

[0152] REFERENCE SIGNS LIST 1 Photography system, 11 Camera, 12 Display, 21 Display unit, 51 Tracking unit, 52 Controller, 61 Photography unit, 62 Camera setting value transmission unit, 71 Camera position estimation unit, 72 Camera position transmission unit, 81 Test chart display control unit, 82 Camera position acquisition unit, 83 Camera setting value acquisition unit, 84 Rendering parameter setting unit, 85 Rendering parameter recording unit, 86 Rendering control unit, 87 Rendering unit

Claims

1. An information processing device comprising: an acquisition unit that acquires the position, attitude, and angle of view of a camera that captures an image of a subject against the background of an image displayed on a display; an identification unit that, based on the position, attitude, and angle of view of the camera, identifies an inner frustum in the image that corresponds to the camera's shooting range and an outer frustum in the image that corresponds outside the camera's shooting range, and, based on pre-prepared reflection information, identifies a reflection outer frustum region of the outer frustum that includes an area that is reflected on the surface of the subject; and a rendering control unit that sets the resolution of the reflection outer frustum region higher than the resolution of a non-reflection outer frustum region, which is the area of ​​the outer frustum other than the reflection outer frustum region.

2. The information processing device according to claim 1, wherein the reflection information is information indicating the relationship between the position and orientation and angle of view of the camera and the reflection outer frustum region.

3. The information processing device according to claim 1, wherein the glare information is information based on the degree of glare on the surface of the subject for each of a plurality of regions into which the outer frustum is divided.

4. The information processing device according to claim 3, wherein the rendering control unit sets a higher resolution for the regions into which the outer frustum is divided as the degree of reflection on the surface of the subject increases.

5. The information processing device according to claim 4, wherein the glare information is information indicating the ratio of the resolution of each of the multiple regions into which the outer frustum is divided to a reference resolution.

6. The information processing device according to claim 1, further comprising a rendering unit that renders at least a part of the video in accordance with control by the rendering control unit.

7. The information processing device according to claim 6, wherein one of the reflection outer frustum regions is rendered by a plurality of the rendering units in a shared manner.

8. An information processing method comprising: acquiring the position, attitude, and angle of view of a camera that captures an image of a subject against a background of an image displayed on a display; identifying, based on the position, attitude, and angle of view of the camera, an inner frustum in the image that corresponds to the camera's imaging range and an outer frustum in the image that corresponds outside the camera's imaging range; identifying, based on pre-prepared reflection information, a reflection outer frustum region of the outer frustum that includes an area that is reflected on the surface of the subject; and setting the resolution of the reflection outer frustum region higher than the resolution of a non-reflection outer frustum region, which is the area of ​​the outer frustum other than the reflection outer frustum region.

9. An information processing method comprising: identifying a reflection outer frustum area, including an area reflected on the surface of the subject, from an outer frustum in an image corresponding to an area outside the shooting range of a camera that shoots a subject with an image displayed on a display as the background; and generating reflection information, which is information referenced by an information processing device that sets the resolution of the reflection outer frustum area higher than the resolution of a non-reflection outer frustum area, which is an area of ​​the outer frustum other than the reflection outer frustum area, and is information for identifying the reflection outer frustum area.

10. The information processing method according to claim 9, wherein the reflected outer frustum region is identified based on a captured image obtained by the camera capturing an image of the subject while the image is displayed on the display.

11. The information processing method according to claim 10, further comprising acquiring the position and orientation and angle of view of the camera, and generating, as the reflection information, information indicating the relationship between the position and orientation and angle of view of the camera and the reflection outer frustum region.

12. The information processing method described in claim 10, wherein the reflection outer frustum area is identified by detecting the degree of reflection on the surface of the subject for each of multiple areas into which the outer frustum is divided based on the captured image, and the reflection information is generated based on the degree of reflection.

13. The information processing method according to claim 12, wherein the image displayed on the display is an image in which the divided regions of the outer frustum can be identified.

14. The information processing method according to claim 9, wherein the reflected outer frustum region is identified by simulation.

Images