Information processing device, information processing method, and program

Abstract

[Problem] To enable reproduction of illumination in a virtual space corresponding to the position of a subject. [Solution] An information processing device comprising a control unit that performs: a process for controlling the display of a video of a virtual space to a display area target disposed in a real space; a process for acquiring illumination information, which is information on illumination with respect to a virtual position of the subject in the virtual space corresponding to the position of the subject present in the real space; and a process for controlling, on the basis of the illumination information, the illumination by one or more lighting devices disposed in the real space.

Description

Information processing device, information processing method, and program

[0001] The present disclosure relates to an information processing device, an information processing method, and a program.

[0002] A technique called VFX (Visual Effects) that adds new visual effects to live-action video has been known. There are various types of VFX, and in recent years, there has been an in-camera VFX technique in which a virtual background image is displayed on a display device such as a large LED (Light Emitting Diode) panel while a subject in the foreground is photographed by a camera.

[0003] In-camera VFX technology synchronizes a camera in virtual space (virtual camera) with the position and orientation of a camera in real space, and the image captured by the virtual camera is displayed on a display device as background image, while the subject is photographed with the camera in real space. When images are photographed in advance, the position of the camera in real space is limited, but with in-camera VFX technology, the angle of the camera in real space can be freely selected when shooting.

[0004] Japanese Patent Application Laid-Open No. 2018-196577

[0005] The task of illuminating a subject using a lighting device placed in real space has traditionally been performed manually. However, it has been difficult to manually reproduce the lighting in a background image, which can change depending on the position and orientation of a real camera, using a lighting device placed in real space. If the lighting in the real space does not match the background image, the subject will not blend into the background, which can create an unnatural appearance.

[0006] The above-mentioned Patent Document 1 discloses an image display device arranged on a side wall of a room as a virtual window, and a technology for reproducing light streaming in from outdoors inside the room in accordance with an image of a virtual three-dimensional space outside the window displayed on the image display device. When reproducing the light streaming in from outdoors, the above-mentioned Patent Document 1 expresses shadows cast by obstructing objects such as furniture and furnishings placed inside the room in accordance with the light streaming in from the window, but does not consider lighting control for blending the subject into the background.

[0007] Therefore, the present disclosure proposes an information processing device, an information processing method, and a program capable of reproducing lighting in a virtual space corresponding to the position of a subject.

[0008] According to the present disclosure, there is provided an information processing device including a control unit that performs the following processes: controlling the display of an image of a virtual space to a display area target placed in a real space; acquiring lighting information that is information about lighting for a virtual position of a subject in the virtual space that corresponds to the position of the subject existing in the real space; and controlling lighting by one or more lighting devices placed in the real space based on the lighting information.

[0009] Furthermore, according to the present disclosure, an information processing method is provided, including: a processor controlling the display of an image of a virtual space to a display area target placed in a real space; acquiring lighting information that is information about lighting for a virtual position of a subject in the virtual space that corresponds to the position of the subject existing in the real space; and controlling lighting by one or more lighting devices placed in the real space based on the lighting information.

[0010] Furthermore, according to the present disclosure, a program is provided that causes a computer to function as a control unit that performs the following processes: controlling the display of an image of a virtual space on a display area target placed in real space; acquiring lighting information that is information about lighting for a virtual position of a subject in the virtual space that corresponds to the position of the subject existing in the real space; and controlling lighting by one or more lighting devices placed in the real space based on the lighting information.

[0011] 1 is a diagram simply illustrating an imaging environment to which an information processing system according to an embodiment of the present disclosure is applied. FIG. 1 is a diagram illustrating an overall configuration of an information processing system according to this embodiment. FIG. 2 is a block diagram illustrating an example of the configuration of an information processing device 20 according to this embodiment. FIG. 3 is a flowchart illustrating a flow of background video display processing by the information processing device 20 according to this embodiment. FIG. 4 is a diagram illustrating the necessity of lighting when imaging a subject U with a screen S as a background. FIG. 5 is a diagram illustrating a virtual position P of the subject U assumed in a virtual space V. FIG. 6 is a diagram illustrating acquisition of ambient light according to this embodiment. FIG. 7 is a flowchart illustrating an example of the flow of ambient light reproduction processing according to this embodiment. FIG. 7 is a diagram illustrating acquisition of direct light according to this embodiment. FIG. 8 is a flowchart illustrating an example of the flow of direct light reproduction processing according to this embodiment. FIG. 9 is a diagram illustrating a case where ambient light and direct light are reproduced according to this embodiment. FIG. 10 is a diagram illustrating camera angle of view determination according to a second modified example of this embodiment. FIG. 11 is a diagram illustrating lighting control for eliminating a backlit state according to a third modified example of this embodiment. FIG. 12 is a diagram illustrating an example of another backlit state according to the third modified example of this embodiment. FIG. 13 is a diagram illustrating lighting control according to the angle of view according to a fourth modified example of this embodiment.

[0012] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0013] The description will be given in the following order: 1. Configuration 1-1. System configuration 1-2. Configuration of information processing device 20 2. Display of background image 3. Lighting control 4. Modifications 4-1. First modification 4-2. Second modification 4-3. Third modification 4-4. Fourth modification 4-5. Fifth modification 4-6. Sixth modification 4-7. Seventh modification 5. Supplementary notes

[0014] <1. Configuration> <<1-1. System Configuration>> An information processing system according to an embodiment of the present disclosure can reproduce lighting in a virtual space that corresponds to the position of a subject.

[0015] 1 is a simplified diagram illustrating a shooting environment to which an information processing system according to an embodiment of the present disclosure is applied. In the example illustrated in FIG. 1 , a background projector 10 that projects a background image onto a screen S is used as an example of a display device that displays a background image. Note that the display device is not limited to a projector and may be, for example, an LED panel (an example of a display panel). The screen S, the LED panel, or the like corresponds to a display area target arranged in real space.

[0016] A person (hereinafter referred to as subject U) is positioned in front of the screen S, and the camera 30 captures the subject U together with the background image projected onto the screen S.

[0017] An illumination device is arranged around the screen S. As an example of the illumination device, a light-emitting device having a light source and a projector that projects an image are used. The light-emitting device is a device that can control color and brightness. In this embodiment, an LED illumination device 12 having an LED bulb is used as an example of the light-emitting device. Also, an illumination projector 14 is used as an example of the projector.

[0018] There may be one or more LED illumination devices 12 and one or more illumination projectors 14. In the example shown in Fig. 1, the LED illumination devices 12 (12a to 12c) and the illumination projectors 14 (14a to 14c) are arranged in a line above the subject U along the screen S, but the locations where the LED illumination devices 12 and the illumination projectors 14 are arranged are not particularly limited.

[0019] In this embodiment, in such a shooting environment, shooting can be performed using in-camera VFX technology. Specifically, the position and orientation of the virtual camera are synchronized with the position and orientation of the camera 30, an image of the virtual space acquired by the virtual camera is projected onto the screen S as a background image, and the subject is photographed by the camera 30 together with the background image projected onto the screen S. At this time, the LED lighting device 12 and the lighting projector 14 can reproduce lighting in the virtual space corresponding to the position of the subject U.

[0020] The overall configuration of the information processing system according to this embodiment will be described below.

[0021] 2 is a diagram showing the overall configuration of an information processing system 1 according to this embodiment. The information processing system 1 according to this embodiment includes a camera 30, a camera tracker 32, a background projector 10, an LED lighting device 12, a lighting projector 14, and an information processing device 20.

[0022] The camera 30 outputs captured images to the information processing device 20. The camera tracker 32 has a function of tracking the position and orientation of the camera 30. The camera tracker 32 may be mounted on the camera 30, for example. The tracking method is not particularly limited, and for example, the camera tracker 32 attached to the camera 30 may detect the movement of the camera 30 by reading reflective markers or infrared markers attached to a ceiling or floor. The camera tracker 32 may also be provided with an IMU (Inertial Measurement Unit), and tracking by reading the markers and inertial tracking using the IMU may be used in combination. The camera tracker 32 outputs information indicating the detected movement of the camera 30 (position and orientation information of the camera 30) to the information processing device 20.

[0023] The information processing device 20 determines the position and orientation of the virtual camera based on the position and orientation information of the camera 30 acquired from the camera tracker 32, generates a background image, and outputs it to the background projector 10. This allows the information processing system 1 according to this embodiment to display a background image synchronized with the position and orientation of the camera 30 in real time.

[0024] Furthermore, the information processing device 20 can acquire lighting information (lighting information) for the virtual position of the subject in the virtual space that corresponds to the position of the subject in the real space, and control the LED lighting device 12 and the lighting projector 14 based on the acquired lighting information to reproduce the lighting in the virtual space. The position of the subject can be acquired, for example, by analyzing captured images input from the camera 30. The information processing device 20 continuously acquires the position of the subject in the virtual space, acquires lighting information in the virtual space each time the subject moves, and reproduces the lighting in the virtual space appropriately in real time using the LED lighting device 12 and the lighting projector 14.

[0025] In this way, in this embodiment, an image of the virtual space is displayed as a background image, and the lighting for the position of the subject in the virtual space is reproduced in the real space. By matching the lighting in the real space with the lighting in the virtual space, the subject blends seamlessly into the background, giving the viewer a greater sense of realism, as if the subject were actually in the virtual space.

[0026] The overview of the information processing system 1 according to an embodiment of the present disclosure has been described above. Next, a specific configuration of the information processing device 20 according to this embodiment will be described with reference to the drawings.

[0027] 3 is a block diagram showing an example of the configuration of the information processing device 20 according to this embodiment. As shown in FIG. 3, the information processing device 20 includes a communication unit 210, a control unit 220, and a storage unit 230.

[0028] (Communication Unit 210) The communication unit 210 has a transmission unit that transmits data to an external device and a reception unit that receives data from an external device. The communication unit 210 according to this embodiment may be communicatively connected to an external device or the Internet using, for example, a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system)), or the like.

[0029] (Control Unit 220) The control unit 220 functions as an arithmetic processing unit and a control unit, and controls the overall operation of the information processing device 20 in accordance with various programs. The control unit 220 is realized by an electronic circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a microprocessor. The control unit 220 may also include a ROM (Read Only Memory) that stores programs to be used, arithmetic parameters, etc., and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

[0030] The control unit 220 in this embodiment can also function as a camera position and orientation information acquisition unit 221, a virtual camera position and orientation determination unit 222, a background image generation unit 223, a display control unit 224, a subject virtual position acquisition unit 225, an ambient light information acquisition unit 226, an LED lighting device control unit 227, a direct light information acquisition unit 228, and a lighting projector control unit 229.

[0031] The camera position and orientation information acquisition unit 221 acquires the position and orientation information of the camera 30 from the camera tracker 32 and outputs it to the virtual camera position and orientation determination unit 222 .

[0032] The virtual camera position and orientation determination unit 222 determines the position and orientation of the virtual camera in the virtual space based on the position and orientation information of the camera 30 output from the camera position and orientation information acquisition unit 221. Specifically, the virtual camera position and orientation determination unit 222 synchronizes the position and orientation of the virtual camera with the position and orientation of the camera 30.

[0033] The background image generation unit 223 generates a background image to be displayed behind the subject. Specifically, the background image generation unit 223 acquires an image of the virtual space captured by the virtual camera as the background image to be projected onto the screen S. The angle of view of the virtual camera can be set in advance according to the dimensions and placement information of the screen S. The background image generation unit 223 outputs the background image to the display control unit 224.

[0034] The display control unit 224 controls the display of the background image generated by the background image generation unit 223. For example, the display control unit 224 transmits the background image to the background projector 10 via the communication unit 210, and causes the background projector 10 to project the background image onto the screen S.

[0035] The subject virtual position acquisition unit 225 acquires the position of the subject in the virtual space. For example, the subject virtual position acquisition unit 225 may analyze an image captured by the camera 30 to calculate the position of the subject relative to the camera 30, and determine the position of the subject in the virtual space based on the position of the virtual camera corresponding to the camera 30. Note that the method of acquiring the subject's position is not limited to this. Furthermore, the subject virtual position acquisition unit 225 continues to acquire the position of the subject in real time while the camera 30 is capturing an image.

[0036] The ambient light information acquisition unit 226 acquires ambient light illumination information (also referred to as ambient light information) as one piece of information about the lighting environment (also referred to as illumination information) at the position of the subject in the virtual space. The ambient light information acquisition unit 226 acquires color and brightness information values ​​at the position of the subject as the ambient light information. The data format of the color and brightness information is not particularly limited, and may be, for example, RGB (Red, Green, Blue) format, HSV (Hue, Saturation, Value-Brightness) format, or another unique format. Details of the method for acquiring ambient light illumination information will be described later with reference to FIGS. 7 and 8 . The ambient light information acquisition unit 226 outputs the acquired ambient light information to the LED illumination device control unit 227.

[0037] The LED illumination device control unit 227 controls the illumination of the LED illumination device 12 based on the ambient light information. Specifically, the LED illumination device control unit 227 sets dimming and color-tuning parameters for the LED illumination device 12 based on the values ​​of color and brightness information acquired as ambient light information. The dimming and color-tuning parameters may be the values ​​of the color and brightness information acquired as ambient light information, or may be values ​​obtained by performing some conversion on the acquired color and brightness information. For example, the LED illumination device control unit 227 may use data obtained by performing processes such as gamma correction and color space conversion on the values ​​of the color and brightness information acquired as ambient light information as dimming and color-tuning parameters to control the parameters of the LED illumination device 12. The data format of the dimming and color-tuning parameters in this embodiment is not particularly limited and may be, for example, RGB format, HSV format, or another unique format. This makes it possible to reproduce in real space the same illuminance and chromaticity as the ambient light at the position of the subject in virtual space.

[0038] The direct light information acquisition unit 228 acquires lighting information of direct light (also referred to as direct light information) as one piece of information (lighting information) about the lighting environment at the position of the subject in the virtual space. The direct light information acquisition unit 228 acquires lighting texture at the position of the subject as the direct light information. More specifically, the direct light information acquisition unit 228 acquires an image as the lighting texture. For example, an image of light filtering through the trees may be acquired. Details of the method for acquiring direct light information will be described later with reference to FIGS. 9 and 10 . The direct light information acquisition unit 228 outputs the acquired direct light information to the lighting projector control unit 229.

[0039] The lighting projector control unit 229 controls the lighting projector 14 based on the direct light information. Specifically, the lighting projector control unit 229 controls the projection from the lighting projector 14 based on the image acquired as the direct light information. The lighting projector control unit 229 may control the lighting projector 14 to project an image based on the image acquired as the direct light information after performing color adjustment, filter processing, geometric correction, or the like on the image acquired as the direct light information. In this way, the lighting projector control unit 229 can control the lighting projector 14 to project an image based on the image acquired as the direct light information. This allows a direct light environment at the position of the subject in the virtual space to be reproduced in the real space. For example, a direct light environment can be reproduced by projecting an image (which may be appropriately corrected, etc.) including light filtering through trees caused by direct light using the lighting projector 14.

[0040] (Storage Unit 230) The storage unit 230 is realized by a ROM that stores programs and calculation parameters used in the processing of the control unit 220, and a RAM that temporarily stores parameters that change as needed.

[0041] The storage unit 230 stores a screen information DB 231, a camera information DB 232, an LED lighting device information DB 233, and a lighting projector information DB 234. The screen information DB 231 stores information such as the size and placement of the screen S. The camera information DB 232 stores information such as the angle of view of the camera 30. The LED lighting device information DB 233 stores information such as placement of the LED lighting devices 12. The lighting projector information DB 234 stores information such as placement of the lighting projector 14. The placement information of each device may be three-dimensional position information.

[0042] Although the configuration of the information processing device 20 has been specifically described above, the configuration of the information processing device 20 according to the present disclosure is not limited to the example shown in Fig. 3. For example, the information processing device 20 does not necessarily have all of the components shown in Fig. 3. Furthermore, the information processing device 20 may be realized by multiple devices.

[0043] 2. Display of Background Image Next, display of a background image according to this embodiment will be described. In this embodiment, a background projector 10 that projects a background image onto a screen S is used as an example of a display device that displays a background image.

[0044] FIG. 4 is a flowchart showing the flow of background video display processing by the information processing device 20 according to this embodiment.

[0045] 4, first, the information processing device 20 reads information about a background image (step S103). The information about the background image may be read from another device. The background image may be an image of a virtual space. For example, the information processing device 20 acquires information about a virtual space from a virtual space providing server (not shown) connected via the communication unit 210 in accordance with a user operation.

[0046] Next, the information processing device 20 acquires position and orientation information of the camera 30, which exists in the real space and captures an image of a subject with the screen S as the background (step S106). Specifically, the information processing device 20 acquires the position and orientation information of the camera 30 from the camera tracker 32 that tracks the camera 30.

[0047] Next, the information processing device 20 determines the position and orientation of the virtual camera in the virtual space based on the position and orientation information of the camera 30 (step S109). The information processing device 20 controls the position and orientation of the virtual camera so that it is synchronized with the position and orientation of the camera 30.

[0048] Next, the information processing device 20 generates a background image (step S112). Specifically, the information processing device 20 acquires an image captured by a virtual camera as the background image. The angle of view of the virtual camera may be the same as that of the camera 30 in the real space, or may be adjusted appropriately depending on the size and placement information of the screen S.

[0049] Next, the information processing device 20 projects the background image (step S115). Specifically, the information processing device 20 transmits the background image from the communication unit 210 to the background projector 10, and causes the communication unit 210 to project the background image onto the screen S.

[0050] The above-described steps S106 to S115 are repeated until shooting is completed (step S118). The position and orientation information of the camera 30 is continuously acquired, and the image of the virtual space displayed in the background is updated in real time in accordance with the movement of the camera 30 (movement of the line of sight), allowing the user to shoot at any angle.

[0051] 3. Lighting Control Next, lighting control according to this embodiment will be described.

[0052] FIG. 5 is a diagram illustrating the need for lighting when photographing a subject U with a screen S as a background. As shown in FIG. 5, in this embodiment, a screen S is placed behind the subject U, and an image of a virtual space V is projected onto the screen. LED lighting devices 12a to 12c and lighting projectors 14a to 14c that illuminate the subject U are placed around the subject U. Note that the placement locations and number of the LED lighting devices 12 and lighting projectors 14 are not particularly limited. Normally, a photography studio is expected to be dark when no lighting is being used, and as shown in FIG. 5, the subject U is in a dark state.

[0053] Here, various lighting settings can be configured in the virtual space V. Specifically, settings include ambient light that illuminates the entire space and one or more light sources that can be positioned at any location. These settings adjust the brightness of the entire virtual space, and shadows of virtual objects may also be cast depending on the positional relationship between the light source and the virtual object. In the example shown in FIG. 5, sunlight filtering through the trees Vs can be seen between the shadows of trees in the virtual space V.

[0054] In the shooting, a camera 30 existing in the real space captures a subject U together with a background image V projected on a screen S, and it is desirable to obtain an image that makes the subject U appear as if it actually exists in the virtual space. However, in the absence of any illumination, the subject U appears dark and does not blend into the background image, as shown in Figure 5. Furthermore, even if illumination is applied to the subject, if it does not match the virtual space, the subject U will not blend into the background image, resulting in an unnatural image.

[0055] 6 is a diagram illustrating a virtual position P of a subject U assumed in a virtual space V. The virtual position P of the subject U corresponds to the position of the subject U in real space. As shown in FIG. 6, ambient light A and light sources L (L1, L2) may be set in the virtual space V. Note that the ambient light A shown in FIG. 6 indicates the light of the entire space.

[0056] Assuming that a subject U exists in a virtual space V, the subject U may be illuminated by ambient light A and a light source L within the virtual space V. Therefore, for example, it can be considered that the subject U standing at a position P as shown in FIG. 6 is illuminated by sunlight Vs filtering through the leaves caused by the light source L and a virtual tree (not shown) placed in the virtual space V.

[0057] 5 , if a subject U standing in front of a screen S is photographed without the light of sunlight filtering through the trees Vs, the subject U will not blend into the background virtual space V and will not appear to exist in the virtual space V. Furthermore, not only when the sunlight filtering through the trees Vs is different from the color of the ambient light in the virtual space V and the color of the light illuminating the subject U (i.e., the ambient light in the photography studio) but also when the color of the ambient light in the virtual space V differs, the subject U will not blend into the background virtual space V and will not appear to exist in the virtual space V. In in-camera VFX, in which photography is performed while a background image is displayed, it is conceivable to manually adjust the lighting for the subject U in real space in accordance with a prepared background image, but it is extremely difficult to manually adjust the lighting to match the background image, which changes arbitrarily depending on the position and orientation of the camera 30.

[0058] In this embodiment, lighting control is performed to reproduce the lighting environment in the virtual space V using at least one of an LED lighting device 12 and a lighting projector 14 arranged in the real space, making it possible to blend the subject U into the background image (virtual space V).

[0059] The reproduction of ambient light and the reproduction of direct light will be described below in order. The information processing device 20 according to this embodiment can reproduce ambient light and direct light in a virtual space.

[0060] <<3-1. Reproduction of Ambient Light>> Reproduction of ambient light can be performed by the LED illumination device 12. One or more LED illumination devices 12 may be installed, and the ambient light information acquisition unit 226 of the information processing device 20 acquires, for each LED illumination device 12, information on the ambient light reproduced by each LED illumination device 12.

[0061] 7 is a diagram for explaining acquisition of ambient light according to this embodiment. Here, acquisition of information on ambient light reproduced by the LED illumination device 12c among the LED illumination devices 12a to 12c will be first explained.

[0062] As shown in Fig. 7 , the ambient light information acquisition unit 226 places a white plate object O, which is a plate-shaped white object, at a virtual position P in the virtual space V that corresponds to the position of the subject U in the real space. At this time, the ambient light information acquisition unit 226 faces the surface of the white plate object O toward a virtual position that corresponds to the position of the LED illumination device 12c placed in the real space. The position of the LED illumination device 12c is acquired from the LED illumination device information DB 233. The ambient light information acquisition unit 226 may orient the surface of the white plate object O directly toward the position of the LED illumination device 12c. It is assumed that the white plate object O placed in the virtual space V is not reflected in the image acquired by the virtual camera (the background image projected on the screen S).

[0063] Next, the ambient light information acquisition unit 226 acquires ambient light information by color picking the surface of the whiteboard object O, i.e., acquiring values ​​of color and brightness information (e.g., RGB values) of the surface of the whiteboard object O. The acquired values ​​of color and brightness information of the surface may be average values ​​of the surface. The acquired values ​​of color and brightness information are information indicating the hue of the ambient light at position P in the virtual space V. In this way, the ambient light information acquisition unit 226 acquires ambient light information at the virtual position P of the subject U in the virtual space V, i.e., information about the ambient light that would hit the subject U if it were assumed that the subject U actually exists at the virtual position P.

[0064] When acquiring the ambient light information, the ambient light information acquisition unit 226 enables the ambient light A and the light sources L1 and L2 of the virtual space V. Alternatively, the ambient light information acquisition unit 226 may enable the ambient light A and the light sources L1 and L2 of the virtual space V in sequence, acquire the ambient light information for each state, and acquire the ambient light information by adding them together. The ambient light information for each state includes the values ​​of color and brightness information of the surface of the white plate object O when only the ambient light A is enabled, the values ​​of color and brightness information of the surface of the white plate object O when only the light source L1 is enabled, and the values ​​of color and brightness information of the surface of the white plate object O when only the light source L2 is enabled. This is because the ambient light of the virtual space V may include not only the light set as ambient light A but also diffusely reflected light from the light sources L1 and L2.

[0065] The above-described process for acquiring the ambient light information is performed for each of the LED illumination devices 12, such as the LED illumination devices 12b, 12a, etc. In this manner, the ambient light information for each of the LED illumination devices 12 is acquired.

[0066] The LED illumination device control unit 227 then controls the corresponding LED illumination devices 12 based on the ambient light information acquired by the ambient light information acquisition unit 226, thereby reproducing the ambient light. For example, the LED illumination device control unit 227 can reproduce the ambient light of the virtual space V in the real space by controlling the illumination (illuminance, chromaticity, etc.) of the LED illumination device 12c based on the ambient light information corresponding to the LED illumination device 12c acquired by the ambient light information acquisition unit 226. The illumination of the LED illumination device 12c can be controlled by controlling the dimming and color adjustment parameters of the LED illumination device 12c. As described above, the dimming and color adjustment parameters may be, for example, RGB values ​​or HSV values.

[0067] FIG. 8 is a flowchart showing an example of the flow of the ambient light reproduction process according to this embodiment.

[0068] 8, first, the ambient light information acquisition unit 226 of the information processing device 20 reads the layout information of the LED illumination devices 12 (step S203). The layout information of the LED illumination devices 12 can be acquired from the LED illumination device information DB 233.

[0069] Next, the ambient light information acquisition unit 226 acquires the position of the subject U in the virtual space V (step S206).

[0070] Next, the ambient light information acquisition unit 226 acquires ambient light information (e.g., RGB values) at the position of the subject U in the virtual space V (step S209). Specifically, the ambient light information acquisition unit 226 places a whiteboard object O, which is not reflected in the background image, at the position of the subject U in the virtual space V, and, for each LED illumination device 12, orients the surface of the whiteboard object O toward the position of the LED illumination device 12 to acquire the RGB values ​​(an example of color and brightness information) on the surface as ambient light information to be used for controlling the corresponding LED illumination device 12.

[0071] Next, the LED illumination device control unit 227 controls, for example, the RGB values ​​of each LED illumination device 12 as dimming and color-adjusting parameters of each LED illumination device 12 based on the ambient light information (step S212). Specifically, the LED illumination device control unit 227 controls, for example, the RGB values ​​(an example of dimming and color-adjusting parameters) of the LED illumination device 12 based on the RGB values ​​(an example of color and brightness information) included in the ambient light information.

[0072] The above-described steps S206 to S212 are repeated until the image capture is completed (step S215). The position of the subject in the virtual space tracks the position of the subject in the real space, and ambient light information at that position is continuously acquired. The information processing device 20 acquires the ambient light information at the subject's position in real time and uses the ambient light information to control the LED lighting device 12, thereby reproducing the ambient light (illuminance and chromaticity) of the virtual space V in the real space.

[0073] <<3-2. Reproduction of Direct Light>> Next, reproduction of direct light in a virtual space will be described. Reproduction of direct light can be performed by the lighting projector 14. One or more lighting projectors 14 may be installed, and the direct light information acquisition unit 228 of the information processing device 20 acquires, for each lighting projector 14, information on the direct light reproduced by each lighting projector 14.

[0074] FIG. 9 is a diagram illustrating the acquisition of direct light according to this embodiment. Here, we will first describe the acquisition of information about direct light reproduced by the lighting projector 14c, which is located in the direction closest to the direction in which the light source L2 is located relative to the subject, among the lighting projectors 14a to 14c. In the virtual space, the three-dimensional positions of the light sources L1 and L2 can be set. Note that, depending on the light source, only the light direction (irradiation direction) may be set. For example, sunlight may be represented in the virtual space by only the direction. Therefore, for example, if the light source L2 is a light source whose direction is set, the lighting projector 14c that can project light onto the subject from the direction closest to the irradiation direction of the light source L2 relative to the subject may be selected as the lighting projector 14 that reproduces the direct light from that light source. The positions of the lighting projectors 14a to 14c shown in FIG. 9 are virtual positions corresponding to the positions of the lighting projectors 14a to 14c arranged in real space.

[0075] 9 , the direct light information acquisition unit 228 places a whiteboard object O at a virtual position P in the virtual space V that corresponds to the position of the subject U in the real space. Furthermore, the direct light information acquisition unit 228 faces the surface of the whiteboard object O toward the position of the lighting projector 14c that can project from a direction closest to the irradiation direction of the light source L2 toward the position P. The position of the lighting projector 14c is acquired from the lighting projector information DB 234. The direct light information acquisition unit 228 may cause the surface of the whiteboard object O to directly face the position of the lighting projector 14c. It is assumed that the whiteboard object O placed in the virtual space V is not reflected in the image acquired by the virtual camera (the background image projected on the screen S).

[0076] Next, the direct light information acquisition unit 228 acquires the texture on the surface of the whiteboard object O, i.e., image information reflected on the surface of the whiteboard object O, as the acquisition of direct light information. Such image information may be, for example, an image of light filtering through the trees. As a result, the direct light information acquisition unit 228 acquires direct light information at the virtual position P of the subject U in the virtual space V, i.e., information on the direct light that would hit the subject U if it were assumed that the subject U actually existed at the virtual position P (e.g., an image of light filtering through the trees). Note that in the example shown in FIG. 9 , an image of surrounding light other than the light filtering through the trees (light between leaves) can also be acquired on the surface of the whiteboard object O. When acquiring direct light information from the light source L2, the direct light information acquisition unit 228 enables only the light source L2 in the virtual space V.

[0077] Next, the direct light information acquisition unit 228 enables only the light source L1 and similarly acquires information about direct light reproduced by the lighting projector 14a that can project from the direction closest to the irradiation direction of the light source L1. Specifically, the direct light information acquisition unit 228 directs the surface of the white board object O placed at position P corresponding to the position of the subject U toward the position of the lighting projector 14a that can project from the direction closest to the irradiation direction of the light source L1 relative to position P, and acquires the texture on the surface of the white board object O, i.e., image information reflected on the surface of the white board object O, as the acquisition of direct light information.

[0078] The lighting projector control unit 229 then controls the corresponding lighting projector 14 based on each piece of direct light information acquired by the direct light information acquisition unit 228, thereby reproducing the direct light. For example, the lighting projector control unit 229 can reproduce the direct light of the virtual space V as illumination for the subject U by projecting, from the lighting projector 14c, an image (e.g., an image of sunlight filtering through the trees) that is the direct light information corresponding to the lighting projector 14c acquired by the direct light information acquisition unit 228. The image projected from the lighting projector 14c may be the image acquired by the direct light information acquisition unit 228 as is, or may be an image that has been subjected to appropriate color adjustment or the like.

[0079] It is also possible that there are overlapping lighting projectors 14 that can project from the direction closest to the irradiation direction of each of the multiple light sources L. In this case, the direct light information acquisition unit 228 may enable the multiple light sources L, point the whiteboard object O in the direction where one of the overlapping lighting projectors 14 is located, and acquire image information reflected on the surface of the whiteboard object O. Alternatively, the multiple light sources L may be enabled in sequence, and image information reflected on the surface of the whiteboard object O that is facing the direction where one of the lighting projectors 14 is located may be acquired for each light source L, and image information obtained by adding together the acquired multiple pieces of image information may be acquired.

[0080] FIG. 10 is a flowchart showing an example of the flow of direct light reproduction processing according to this embodiment.

[0081] 10 , first, the direct light information acquisition unit 228 of the information processing device 20 reads the layout information of the lighting projector 14 (step S353). The layout information of the lighting projector 14 can be acquired from the lighting projector information DB 234.

[0082] Next, the direct light information acquisition unit 228 acquires the position of the subject U in the virtual space V (step S356).

[0083] Next, the direct light information acquisition unit 228 acquires direct light information at the position of the subject U in the virtual space V (step S359). Specifically, the direct light information acquisition unit 228 places a whiteboard object O, which is not reflected in the background image, at the position of the subject U in the virtual space V. Next, the direct light information acquisition unit 228 assigns, for each light source L, a lighting projector 14 that can project light onto the subject from a direction closest to the irradiation direction of each light source L, and faces the surface of the whiteboard object O toward the position of the assigned lighting projector 14. Then, the direct light information acquisition unit 228 acquires the texture on the surface, i.e., image information reflected on the surface, as direct light information corresponding to the assigned lighting projector 14.

[0084] Next, the lighting projector control section 229 controls the projection of each lighting projector 14 based on each piece of direct light information (step S362). Specifically, the lighting projector control section 229 controls the lighting projector 14 to project an image based on the image acquired as the direct light information.

[0085] The above-described steps S356 to S362 are repeated until the image capture is completed (step S365). The position of the subject in the virtual space tracks the position of the subject in the real space, and direct light information at that position is continuously acquired. The information processing device 20 acquires the direct light information at the subject's position in real time and controls the lighting projector 14 to project the direct light information, thereby reproducing the direct light of the virtual space V in the real space.

[0086] <<3-3. Reproduction of Ambient Light and Direct Light>> The reproduction of ambient light and direct light has been specifically described above. In this embodiment, in addition to reproducing ambient light using the LED lighting device 12, direct light is also reproduced using the lighting projector 14, thereby providing a greater sense of realism and immersion.

[0087] Fig. 11 is a diagram illustrating a case where ambient light and direct light are reproduced according to this embodiment. In the example shown in Fig. 11, ambient light is reproduced by illumination from the LED lighting device 12, and direct light is reproduced by projection from the lighting projector 14. As shown in Fig. 11, the subject U, which was in a dark state with no illumination as shown in Fig. 5, is now illuminated with ambient light and direct light (light from sunlight filtering through the trees and surrounding light other than sunlight filtering through the trees) similar to that of the virtual space V reflected in the background. This allows the subject U to blend into the virtual space V reflected in the background.

[0088] The reproduction of ambient light and direct light has been specifically described above. In this embodiment, when reproducing ambient light and direct light in a virtual space, lighting information is acquired and controlled individually in consideration of the placement of the LED lighting device 12 and the lighting projector 14, so that the directionality of lighting relative to the position of the subject (similar lighting from the same direction as in the virtual space) can be reproduced.

[0089] 4. Modifications Next, modifications of this embodiment will be described.

[0090] <<4-1. First Modification>> In the above-described embodiment, a whiteboard object is used to acquire lighting information at the position of the subject U in the virtual space V, but the shape of the white object used to acquire lighting information is not limited to this. For example, the information processing device 20 may use a virtual object having a shape corresponding to the shape of the subject. If the subject is a human, a virtual object in the shape of the human is assumed; if the subject is an animal, a virtual object in the shape of the animal is assumed; and if the subject is an object, a virtual object in the shape of the object is assumed. Here, as an example, a case will be described in which the subject is a human and a human-shaped white object (hereinafter also referred to as a mannequin object) is used.

[0091] The information processing device 20 may place a mannequin object at the position of the subject U in the virtual space V and acquire information about the lighting shining on the mannequin object (RGB values ​​and image information on the surface of the mannequin object). The information processing device 20 may also set the body shape of the mannequin object to be similar to that of the subject U. The information processing device 20 may also change the pose (posture) of the mannequin object to synchronize with that of the subject U in real time.

[0092] Furthermore, the information processing device 20 may acquire illumination information from a predetermined area including the position of the subject U in the virtual space V without placing a white object.

[0093] <<4-2. Second Modification>> The information processing device 20 may change the illumination or the background image when the screen S is not within the angle of view of the camera capturing the image in real space. The case where the screen S is not within the angle of view of the camera is assumed to be, for example, a case where the angle of view of the camera is outside the screen S and a part outside the screen S (a part where the background image is not displayed) is reflected.

[0094] 12 is a diagram illustrating a camera angle of view determination according to a second modified example of this embodiment. As shown on the left side of Fig. 12, assume that when capturing an image with a camera 34 implemented by, for example, a smartphone, the screen S does not fall within the angle of view 341 of the camera 34. The information processing device 20 acquires position and orientation information and angle of view information of the camera 34, as well as size and arrangement information of the screen S, and determines whether the angle of view (imaging range) of the camera 34 falls outside the screen S.

[0095] When the screen S is not within the angle of view 341 of the camera 34, the information processing device 20 changes the illumination or the background image as shown in the right of Fig. 12. Specifically, the information processing device 20 may provide red illumination or the like using the LED illumination device 12 as shown in the upper right of Fig. 12, may turn off the LED illumination device 12 as shown in the center right of Fig. 12, or may turn off the LED illumination device 12 and not display the background image as shown in the lower right of Fig. 12.

[0096] By changing the lighting or background image in this way, the photographer or an editor who edits the footage after shooting can easily notice that the screen S is not within the angle of view 341 of the camera 34. The photographer can adjust the position and orientation of the camera 34 so that the screen S is within the angle of view. In addition, the editor can easily perform post-shooting editing, such as cutting out the footage in which the screen S is not within the angle of view.

[0097] <<4-3. Third Modification>> The information processing device 20 may adjust the illumination as appropriate when the illumination causes a backlit state. When the illumination causes a backlit state, the subject appears dark, so the information processing device 20 can eliminate the backlit state by turning off or dimming the illumination that causes the backlit state.

[0098] 13 is a diagram illustrating lighting control for eliminating backlit conditions according to a third modified example of this embodiment. As shown on the left side of FIG. 13, when a screen S, lighting devices (here, LED lighting devices 12d to 12g), a subject U, and a camera 30 are arranged, the illumination from LED lighting devices 12e and 12f creates backlight, causing the subject U to appear dark. Information processing device 20 can determine whether a backlit condition exists based on the position and orientation information and angle of view information of camera 30 and the positions and light distribution information of the lighting devices.

[0099] In the case of a backlit condition, the information processing device 20 can eliminate the backlit condition by turning off or reducing the brightness of the lighting devices (here, LED lighting devices 12e and 12f) causing the backlight, as shown on the right side of Figure 13.

[0100] Fig. 14 is a diagram illustrating another example of a backlit state according to the third modified example of this embodiment. When the position and orientation of the camera 30 is as shown on the left side of Fig. 14, the illumination by the LED illumination devices 12f and 12g creates backlighting, causing the subject U to appear dark. As shown on the right side of Fig. 14, the information processing device 20 can eliminate the backlit state by turning off or reducing the brightness of the LED illumination devices 12f and 12g that are causing the backlighting.

[0101] The information processing device 20 may determine that a backlight state exists when a certain amount or more of illumination light overlaps in the backlight direction within the camera's angle of view. As another method for eliminating the backlight state, the information processing device 20 may also perform control to increase illumination in directions other than the backlight direction.

[0102] <<4-4. Fourth Modification>> The information processing device 20 may be configured to illuminate at least the range (angle of view) captured by the camera.

[0103] 15 is a diagram illustrating lighting control according to the angle of view according to a fourth modified example of this embodiment. As shown on the left side of FIG. 15, when camera 32 captures an image of subject U with its angle of view including mainly the entire body, information processing device 20 turns on LED lighting devices 12h and 12i that illuminate the upper half of subject U's body from the left and right, respectively, and LED lighting devices 12j and 12k that illuminate the lower half of subject U's body from the left and right, respectively. Information processing device 20 determines which lighting devices will not affect the image capture based on the position and orientation information and angle of view information of camera 30, the positions and light distribution information of the lighting devices, and the position of the subject.

[0104] 15 right, when the camera 32 captures an image of the subject U, mainly including the upper body in its angle of view, the information processing device 20 determines that the LED lighting devices 12j and 12k that illuminate the lower body of the subject U from the left and right are lighting devices that will not affect the image capture even if they are turned off. In this case, the information processing device 20 can reduce power consumption by turning off the lighting devices that do not affect the image capture.

[0105] <<4-5. Fifth Modification>> When similar subjects (people, props, etc.) exist in virtual space and real space, the colors of the subjects in the virtual space and real space may appear different depending on the relationship between the modeling or rendering settings of the virtual space, the color reproducibility of the background image display device (for example, background projector 10), and the color rendering properties of the illumination light in the illumination device (LED illumination device 12 or illumination projector 14).

[0106] To prepare for such a case, the information processing device 20 may adjust the object color (hue) of the subject by further controlling the spectral characteristics (wavelength spectrum) of the illumination light when controlling the illumination device. The spectral characteristics may be controlled by controlling the output of each wavelength band (e.g., red, blue, green) of the illumination device, or by placing a color filter in front of the illumination device to weaken the intensity of a specific wavelength band.

[0107] This reduces the difference in color between similar subjects that exist in virtual and real space, and by matching the color rendering of the lighting in the background image with the color rendering of the actual lighting, it becomes possible to capture subjects that blend more seamlessly into the virtual space.

[0108] <<4-6. Sixth Modification>> In the above-described embodiment, a background video is displayed during shooting, but the present invention is not limited to this. A green screen may be used as the background, and the information processing device 20 may superimpose a background video generated by the background video generation unit 223 on a captured image (video) captured by a camera using a chroma key. The superimposition of the background video may be performed in real time, or may be performed in post-processing after shooting.

[0109] Even in such cases, lighting control linked to the virtual space as described above can be performed during shooting, resulting in the subject blending seamlessly into the background image.

[0110] <<4-7. Seventh Modification>> The lighting control linked to a virtual space according to the above-described embodiment is not limited to professional photography studios that perform in-camera VFX, and may be applied to lighting control when an in-camera VFX technique is used in VP (Virtual Production) for LBE (Location Based Entertainment), for example.

[0111] Furthermore, the lighting control linked to the virtual space according to the above-described embodiment is not limited to the time of shooting, but may also be applied to lighting control in LBE games, home games, or home environment reproductions. During games or home environment reproductions, lighting that reproduces the lighting of the virtual space may be applied to the floor of the real space (reproduction of lighting at a virtual floor position in the virtual space that corresponds to the floor position in the real space). This can enhance the sense of immersion in the game and background images.

[0112] 5. Supplementary Information Although preferred embodiments of the present disclosure have been described above in detail with reference to the accompanying drawings, the present technology is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0113] For example, one or more computer programs can be created for hardware such as a CPU, ROM, and RAM built into the information processing device 20 to perform the functions of the information processing device 20. Also provided is a computer-readable storage medium storing the one or more computer programs.

[0114] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0115] The present technology may also be configured as follows. (1) An information processing device including a control unit that performs the following processes: controlling display of an image of a virtual space in a display area target arranged in a real space; acquiring lighting information that is information about lighting for a virtual position of the subject in the virtual space corresponding to the position of the subject existing in the real space; and controlling lighting by one or more lighting devices arranged in the real space based on the lighting information. (2) The information processing device according to (1), wherein the control unit acquires placement information of the one or more lighting devices and acquires the lighting information corresponding to each of the one or more lighting devices in consideration of the placement information. (3) The information processing device according to (1) or (2), wherein the control unit acquires the lighting information from a region including the virtual position of the subject in the virtual space. (4) The information processing device according to (1) or (2), wherein the control unit arranges a virtual object having a shape corresponding to the shape of the subject, the virtual object not being drawn in the image displayed in the display area target, at the virtual position of the subject in the virtual space, and acquires the lighting information from the virtual object. (5) The information processing device according to (1) or (2), wherein the control unit places a whiteboard object, which is a plate-shaped object not drawn in the image to be displayed in the display area target, at a virtual position of the subject in the virtual space, and acquires the lighting information from the whiteboard object. (6) The information processing device according to (5), wherein the lighting device is a light-emitting device having a light source, and the control unit sequentially faces a surface of the whiteboard object toward the positions of each light-emitting device and acquires values ​​of color and brightness information of the surface as the lighting information corresponding to each light-emitting device. (7) The information processing device according to (6), wherein the control unit enables all ambient light settings and light source settings of the virtual space when acquiring the lighting information. (8) The information processing device according to (6) or (7), wherein the lighting information is lighting information for reproducing ambient light of the virtual space by the light-emitting device, and the control unit controls dimming and color adjustment parameters of the light-emitting device based on values ​​of color and brightness information, which are the acquired lighting information.(9) The information processing device according to (5), wherein the lighting device is a lighting projector, and the control unit faces the surface of the whiteboard object toward the position of the lighting projector and acquires an image of the surface as the lighting information corresponding to the lighting projector. (10) The information processing device according to (9), wherein the control unit sequentially enables one or more light sources arranged in the virtual space, selects, for each light source, the lighting projector that can project from a direction closest to the irradiation direction of the light source toward the subject, faces the surface of the whiteboard object toward the selected lighting projector, and acquires the lighting information corresponding to the lighting projector. (11) The information processing device according to (9) or (10), wherein the lighting information is lighting information for reproducing direct light in the virtual space by the lighting projector, and the control unit controls the lighting projector to project an image based on an image that is the acquired lighting information onto the subject. (12) The information processing device according to any one of (1) to (11), wherein the control unit acquires position and orientation information of a camera that captures the subject with the display area target as a background, synchronizes a virtual camera in the virtual space with the position and orientation of the camera, and controls to display an image of the virtual space captured by the virtual camera as a background image on the display area target. (13) The information processing device according to (12), wherein the display area target is a screen, and the control unit controls to project the image of the virtual space by a background projector arranged in real space. (14) The information processing device according to (12), wherein the display area target is a display panel, and the control unit controls to display the image of the virtual space on the display panel. (15) The information processing device according to (12), wherein the display area target is a green screen, and the control unit controls to superimpose the image of the virtual space on a captured image acquired by the camera by chromakey.(16) An information processing method, including: a processor controlling display of an image of a virtual space on a display area target placed in a real space, acquiring lighting information that is information about lighting for a virtual position of the subject in the virtual space corresponding to the position of the subject existing in the real space, and controlling lighting by one or more lighting devices placed in the real space based on the lighting information. (17) A program that causes a computer to function as a control unit that performs the following processes: controlling display of an image of a virtual space on a display area target placed in a real space, acquiring lighting information that is information about lighting for a virtual position of the subject in the virtual space corresponding to the position of the subject existing in the real space, and controlling lighting by one or more lighting devices placed in the real space based on the lighting information.

[0116] REFERENCE SIGNS LIST 1 Information processing system 10 Background projector 12 LED lighting device 14 Lighting projector 20 Information processing device 210 Communication unit 220 Control unit 221 Camera position and orientation information acquisition unit 222 Virtual camera position and orientation determination unit 223 Background image generation unit 224 Display control unit 225 Subject virtual position acquisition unit 226 Ambient light information acquisition unit 227 LED lighting device control unit 228 Direct light information acquisition unit 229 Lighting projector control unit 230 Storage unit 231 Screen information DB 232 Camera information DB 233 LED lighting device information DB 234 Lighting projector information DB 30 Camera 32 Camera tracker

Claims

1. An information processing device comprising a control unit that performs the following processes: controlling the display of an image of a virtual space on a display area target placed in a real space; acquiring lighting information that is information about lighting for a virtual position of a subject in the virtual space that corresponds to the position of the subject existing in the real space; and controlling lighting by one or more lighting devices placed in the real space based on the lighting information.

2. The information processing device according to claim 1, wherein the control unit acquires layout information of the one or more lighting devices, and acquires the lighting information corresponding to each of the one or more lighting devices by taking the layout information into consideration.

3. The information processing device according to claim 1, wherein the control unit acquires the illumination information from an area including a virtual position of the subject in the virtual space.

4. The information processing device according to claim 1, wherein the control unit places a virtual object having a shape corresponding to the shape of the subject at a virtual position of the subject in the virtual space, the virtual object not being drawn in the image to be displayed in the display area target, and acquires the lighting information from the virtual object.

5. The information processing device according to claim 1, wherein the control unit places a whiteboard object, which is a plate-shaped object that is not drawn in the image to be displayed in the display area target, at the virtual position of the subject in the virtual space, and acquires the lighting information from the whiteboard object.

6. The information processing device according to claim 5, wherein the lighting device is a light-emitting device having a light source, and the control unit sequentially directs the surface of the whiteboard object toward the position of each light-emitting device, and acquires the values ​​of color and brightness information of the surface as the lighting information corresponding to each light-emitting device.

7. The information processing device according to claim 6, wherein the control unit enables all of the ambient light settings and light source settings of the virtual space when acquiring the lighting information.

8. The information processing device according to claim 6, wherein the lighting information is lighting information for reproducing the ambient light of the virtual space by the light-emitting device, and the control unit controls dimming and color adjustment parameters of the light-emitting device based on values ​​of color and brightness information which are the acquired lighting information.

9. The information processing device according to claim 5, wherein the lighting device is a lighting projector, and the control unit directs a surface of the whiteboard object toward the position of the lighting projector and acquires an image of the surface as the lighting information corresponding to the lighting projector.

10. The information processing device according to claim 9, wherein the control unit sequentially enables one or more light sources arranged in the virtual space, and for each light source, selects the lighting projector capable of projecting from a direction closest to the direction of irradiation of the light source onto the subject, faces the surface of the whiteboard object toward the selected lighting projector, and acquires the lighting information corresponding to the lighting projector.

11. The information processing device of claim 9, wherein the lighting information is lighting information for reproducing direct light in the virtual space by the lighting projector, and the control unit controls the lighting projector to project an image based on an image that is the acquired lighting information onto the subject.

12. The information processing device according to claim 1, wherein the control unit acquires position and orientation information of a camera that captures the subject with the display area target as the background, synchronizes a virtual camera in the virtual space with the position and orientation of the camera, and controls the image of the virtual space captured by the virtual camera to be displayed on the display area target as a background image.

13. The information processing device according to claim 12, wherein the display area target is a screen, and the control unit controls the projection of the image of the virtual space by a background projector arranged in real space.

14. The information processing device according to claim 12, wherein the display area target is a display panel, and the control unit performs control to display an image of the virtual space on the display panel.

15. The information processing device according to claim 12, wherein the display area target is a green screen, and the control unit performs control to superimpose an image of the virtual space on an image captured by the camera using chromakey.

16. An information processing method comprising: a processor controlling the display of an image of a virtual space to a display area target placed in a real space; acquiring lighting information which is information about lighting for a virtual position of a subject in the virtual space corresponding to the position of the subject existing in the real space; and controlling lighting by one or more lighting devices placed in the real space based on the lighting information.

17. A program that causes a computer to function as a control unit that performs the following processes: controlling the display of an image of a virtual space on a display area target placed in real space; acquiring lighting information that is information about lighting for a virtual position of a subject in the virtual space that corresponds to the position of the subject existing in the real space; and controlling lighting by one or more lighting devices placed in the real space based on the lighting information.

Images