Information processing device, information processing method, program, and information processing system

Abstract

The present invention makes it possible to confirm the hue and the like of a captured video at a point in time before a virtual video on a display is captured by a camera. An information processing apparatus includes: a rendering unit (520a) that generates a virtual video by rendering using a 3D model used in a photographing system that photographs a video on a display (505) by a camera (502), the display (505) displaying the virtual video obtained by rendering using the 3D model; and a video processing unit (33) that performs, on the virtual video generated by the rendering unit, live-shot video conversion processing for generating an analog video by using a processing parameter that achieves luminance or color characteristics at the time of imaging by a camera used in the imaging system.

Description

Technical Field

[0001] This technology relates to information processing devices, information processing methods, programs, and information processing systems, and in particular to video production technology using virtual video. Background Technology

[0002] As a photographic method used to create video content such as movies, there is a known technique where actors perform in front of a so-called green screen and then composite the background video.

[0003] In addition, in recent years, the following shooting system has been developed as an alternative to green screen shooting: in a studio equipped with a large monitor, the background video is displayed on the monitor, and the actors perform in front of it, thereby making it possible to shoot the actors and the background. This shooting system is called virtual production, in-camera VFX (in-camera VFX), or LED wall virtual production.

[0004] The following patent document 1 discloses a technology for a system for filming actors or objects performing in front of a background video.

[0005] Existing technical documents Patent documents Patent Document 1: U.S. Patent Application Publication No. 2020 / 0145644A Summary of the Invention The technical problem that the invention aims to solve By displaying background video on a large display and then filming the actors and background video with a camera, cast and crew can visually understand the scene to perform and judge the quality of the performance, without the need for separate compositing of the background video after filming. This is superior to green screen photography.

[0006] However, the appearance of the background depends on the camera's position relative to the display and its imaging direction. When simply projecting background video, the background will not change even with different camera positions and imaging directions, resulting in an unnatural video. Therefore, by altering the background video (at least the video within the camera's field of view on the display) according to the camera's position and imaging direction, the background video can be made to have an appearance equivalent to actual three-dimensional space, thus enabling the shooting of videos that are equivalent to shooting videos against a real scene as a background.

[0007] When using a virtual background video shooting system, the color tone and other characteristics of the captured video may differ from what the director or crew envisioned during the actual shooting phase. In such cases, various settings adjustments are required in the studio or other locations used for filming to correct the color tone, which is an extremely cumbersome and time-consuming task. This reduces the efficiency of video production.

[0008] Therefore, this publication proposes a technology that enables more efficient video production.

[0009] Technical solutions for solving technical problems The information processing apparatus according to this technology includes: a rendering unit that generates a virtual video by rendering a 3D model used in a photography system, the photography system capturing video on a display using a camera, the display showing the virtual video obtained by rendering the 3D model; and a video processing unit that performs a live-action video conversion process on the virtual video generated by the rendering unit, the live-action video conversion process being used to generate a simulated video by using processing parameters that realize the brightness or color characteristics when captured by the camera used in the photography system.

[0010] Before filming, virtual videos, such as background videos, can be rendered using 3D models used in the photography system, and these videos can be monitored. In this case, video processing is performed on the virtual video to reproduce the brightness and color characteristics of the camera at the time of filming. Attached Figure Description

[0011] Figure 1 This is an explanatory diagram of a photographic system according to an embodiment of the present technology.

[0012] Figure 2 An explanatory diagram of the background video corresponding to the camera position of the photography system in the embodiment.

[0013] Figure 3 An explanatory diagram of the background video corresponding to the camera position of the photography system in the embodiment.

[0014] Figure 4 This is an explanatory diagram illustrating the video content production steps of an embodiment.

[0015] Figure 5 A block diagram of the photographic system as an example.

[0016] Figure 6 A flowchart illustrating the background video generation process of the photography system used in this embodiment.

[0017] Figure 7 This is a block diagram of a photography system using multiple cameras according to an embodiment.

[0018] Figure 8 This is a block diagram of an information processing apparatus according to an embodiment.

[0019] Figure 9 The illustration is a pre-visualization of an example.

[0020] Figure 10 This is a block diagram of a system configuration example for the first embodiment.

[0021] Figure 11 An explanatory diagram for setting the cropping range of an image sensor.

[0022] Figure 12 This is a flowchart of the real-shot video conversion process in the first embodiment.

[0023] Figure 13 This is a block diagram of a system configuration example for the second embodiment.

[0024] Figure 14 This is a flowchart of the color conversion information generation process in the second embodiment.

[0025] Figure 15 This is a block diagram of a system configuration example for the third embodiment.

[0026] Figure 16 Explanatory diagrams showing the storage settings for cameras in various scenarios of the third embodiment.

[0027] Figure 17 A flowchart for the processing of camera settings in the third embodiment.

[0028] Figure 18 This is a block diagram of a system configuration example for the fourth embodiment.

[0029] Figure 19 This is an explanatory diagram of the camera position operation video in the fourth embodiment.

[0030] Figure 20 This is an explanatory diagram of the moiré alarm in the fourth embodiment.

[0031] Figure 21 This is an explanatory diagram of the operation in the camera position operation video of the fourth embodiment.

[0032] Figure 22 This is a flowchart of the moiré alarm processing in the fourth embodiment.

[0033] Figure 23 This is a flowchart of the moiré alarm processing during shooting in the fourth embodiment.

[0034] Figure Labels 31: Rendering engine; 32: Rendering unit; 33: Video processing unit; 35: Adjustment operation unit; 36: Moiré prediction unit; 37: Position operation unit; 40: Monitor; 41: Cloud server; 70: Information processing device; 500: Photography system; 502: Camera; 503: Output monitor; 505: LED wall; 506: LED panel; 515: Camera signal processing unit; 520: Rendering engine; 520a: Rendering unit; 520b: Color calibration unit; 520c: CG camera setting unit; 700: Pre-visualization system; vB, vBP: Background video; vC: Shooting video; vM: Monitor video; vSM: Analog video; vP: Camera position operation video; Ic: Color conversion information Specific Implementation The embodiments will be described in the following order.

[0035] <1. Photography System and Content Production> <2. Configuration of Information Processing Devices> <3. First Embodiment> <4. Second Embodiment> <5. Third Embodiment> <6. Fourth Embodiment> <7. Imaging Data Stream> <8. Summary and Variations> Note that in this disclosure, "video" or "image" includes both still images and moving images. Furthermore, "video" refers not only to video data displayed on a display, but also to video data not displayed on a display.

[0036] For example, in this embodiment, the background video before it is displayed on the display, the video captured by the camera, the background video switched by the switcher, or the captured video is not the actual video displayed but video data, but for convenience, it is referred to as "background video," "captured video," etc.

[0037] <1. Camera System and Video Content Production> The present invention describes a photographic system and the production of video content that can utilize the technology disclosed herein.

[0038] Figure 1 The photographic system 500 is shown schematically. The photographic system 500 is a system for performing shooting as a virtual production, and some of the equipment arranged in the photographic studio is shown in the figure.

[0039] In the photography studio, a performance area 501 is set up for actors 510 to perform, such as acting. Large display devices are arranged at least at the back, and further to the left and right sides, and above the performance area 501. While the type of display device is not limited, the figure shows an example of using an LED wall 505 as a large display device.

[0040] An LED wall 505 is formed into a large panel by connecting and arranging multiple LED panels 506 vertically and horizontally. The size of the LED wall 505 mentioned here is not particularly limited, as long as it is the necessary or sufficient size for displaying the background when filming the actor 510.

[0041] A necessary number of lights 580 are installed above or to the side of the performance area 501 to illuminate the performance area 501.

[0042] Near performance area 501, a camera 502 for shooting movies and other video content may be configured. The cameraman 512 can move the camera 502 and manipulate its imaging direction, angle, etc. Remote operation of the camera 502's movement and angle manipulation is also possible. Furthermore, the camera 502 can move or change its angle automatically or autonomously. Therefore, the camera 502 can be mounted on a camera platform or a moving object.

[0043] Camera 502 captures both the actor 510 in the performance area 501 and the video displayed on the LED wall 505. For example, by displaying a landscape as background video vB on the LED wall 505, it is possible to capture a video similar to the actual performance of the actor 510 at that scenic spot.

[0044] An output monitor 503 is installed near performance area 501. The video captured by camera 502 is displayed in real-time on this output monitor 503 as a monitor video (VM). As a result, the director and staff producing the video content are able to review the captured video.

[0045] As described above, the photography system 500, which shoots the performance of actors 510 against the backdrop of LED wall 505 in a photography studio, has various advantages over green screen photography.

[0046] For example, during green screen filming, actors may find it difficult to visualize the background and scene conditions, which can affect their performance. However, by displaying background video (vB), actors can perform more easily, improving performance quality. Furthermore, directors and other staff can more easily determine whether an actor's performance matches the background and scene conditions.

[0047] Furthermore, post-production is more efficient than with green screen photography. This is because chroma keying may not be necessary, or color correction or reflection composition may not be required. Additionally, even if chroma keying is needed during shooting, simply displaying green or blue video eliminates the need for an additional physical background screen, further improving efficiency.

[0048] When shooting against a green screen, a green tint is added to the actors' bodies, costumes, and props, so this needs to be corrected. Additionally, when shooting against a green screen, if there are objects that reflect the surrounding scenery, such as glass, mirrors, or snowdomes, the reflected images need to be generated and combined, which is a time-consuming task.

[0049] However, in passing Figure 1 When shooting with the 500 camera system, no correction is needed because no green hue is added. Furthermore, by displaying background video vB, reflections on actual objects such as glass are naturally captured and filmed, eliminating the need to combine the reflected video.

[0050] Here, refer to Figure 2 and Figure 3 Let's explain the background video VB. Even if the background video VB is displayed on the LED wall 505 and filmed alongside the actor 510, simply displaying the background video VB alone will result in an unnatural background in the filmed video. This is because a three-dimensional and deep background is being used as the background video VB in a flat manner.

[0051] For example, camera 502 can film the actor 510 in performance area 501 from various directions and can perform zoom operations. The actor 510 also does not remain in one place. Then, the actual appearance of the background of the actor 510 should change according to the position, imaging direction, and viewing angle of camera 502, but this change cannot be obtained in the background video vB, which is a flat video. Therefore, the background video vB is changed to make the background resemble the actual appearance, including parallax.

[0052] Figure 2 The image shows camera 502 capturing the state of actor 510 from the left side of the diagram. Figure 3 The image shows camera 502 filming actor 510 from the right side of the diagram. In each diagram, the background video vB shows the filming area video vBC.

[0053] Note that the portion of the background video vB excluding the shooting area video vBC is called the "outer frustum", and the shooting area video vBC is called the "inner frustum".

[0054] The background video vB described here refers to the entire video displayed as the background, including the video of the shooting area vBC (internal view).

[0055] The range of the shooting area video vBC (internal view) is equivalent to the actual range captured by camera 502 within the display surface of LED wall 505. Then, the shooting area video vBC is a video representing the scene actually seen based on the position of camera 502, imaging direction, and viewing angle, when the position of camera 502 is set as the viewpoint.

[0056] Specifically, a 3D model, i.e. 3D background data, is prepared as the background. Based on the viewpoint position of the camera 502, the video vBC of the shooting area is rendered in real time and sequentially.

[0057] Note that the range of the video vBC in the shooting area is actually slightly wider than the range captured by camera 502 at that point in time. This is to prevent the video of the external view from being mapped due to drawing delay when the imaging range changes slightly due to panning, tilting, zooming, etc. of camera 502, and to avoid the influence of diffracted light from the video of the external view.

[0058] The video of the shooting area video vBC (internal view) is rendered in real time and composited with the video of the external view. The video of the external view used in the background video vB can be pre-rendered based on 3D background data, or it can be rendered in real time for each frame or each interval frame. The video of the shooting area video vBC (internal view) is incorporated into a part of the video of the external view to generate the entire background video vB.

[0059] Note that there are cases where the video of the external view is rendered frame by frame in a similar manner to the video of the internal view, but this example uses static video. The following explanation will mainly focus on the case where only the first frame of the video of the external view is rendered.

[0060] As a result, even when the camera 502 is moved forward or backward or left or right, or when a zoom operation is performed, the background of the area imaged along with the actor 510 is captured as video corresponding to the change in viewpoint position or field of view (FOV) that accompanies the actual movement of the camera 502.

[0061] like Figure 2 and Figure 3 As shown, the monitor video vM, including actor 510 and background, is displayed on the output monitor 503. This is the captured video. The background of the monitor video vM is the video vBC of the shooting area. That is, the background contained in the captured video is a real-time rendered video.

[0062] As described above, in the photography system 500 of the embodiment, not only is the background video vB simply displayed in a planar manner, but the background video vB, including the shooting area video vBC, is also changed in real time so that it is possible to shoot a video similar to that when actually shooting a landscape.

[0063] Note that the following concept can also be adopted: reduce the processing burden on the system by rendering only the video vBC of the shooting area mapped by the camera 502 in real time, rather than the entire background video vB displayed on the LED wall 505.

[0064] Here, the steps for creating video content as a virtual production, which is imaged by the camera system 500, will be explained. For example... Figure 4 As shown, the video content production process can be roughly divided into three stages: resource creation (ST1), production (ST2), and post-production (ST3).

[0065] Resource creation ST1 is the step of creating 3D background data for displaying the background video vB. As described above, the background video vB is generated by real-time rendering using 3D background data during shooting. For this purpose, 3D background data as a 3D model is created in advance.

[0066] Examples of methods for creating 3D background data include full computer graphics (CG), point cloud data scanning, and photogrammetry.

[0067] Full CG is a method of creating 3D models using computer graphics. Of the three methods, this one requires the most time and effort, but it's suitable for situations where you want to use non-realistic videos or videos that are difficult to film in real life as background video content.

[0068] Point cloud data scanning is performed as follows: distance is measured from a location using, for example, a LiDAR (Light Detection and Ranging) sensor; a 360-degree image is captured from the same location using a camera; and the color data captured by the camera is then applied to the points measured by the LiDAR sensor, thereby generating a 3D model based on the point cloud data. Compared to full CG (Graphics over Image), this method allows for the creation of 3D models in a shorter time. Furthermore, it facilitates the creation of 3D models with higher precision than photogrammetry.

[0069] Photogrammetry is a photographic measurement technique that analyzes parallax information from two-dimensional images of an object taken from multiple viewpoints to determine its size and shape. It can create 3D models in a short time.

[0070] Note that point cloud information obtained using lidar can be used to generate 3D data through photogrammetry.

[0071] In resource creation ST1, these methods can be used, for example, to create 3D models that serve as 3D background data. Of course, these methods can also be combined. For example, a portion of a 3D model created through point cloud data scanning or photogrammetry can be created and composited using CG.

[0072] Create ST2 for Figure 1 The imaging process is performed in the photography studio shown. Key technical elements in this scenario include real-time rendering, background display, camera tracking, and lighting control.

[0073] Real-time rendering is as follows: Figure 2 and Figure 3 As described, this is used to obtain the rendering process of the video vBC of the shooting area at each time point (each frame of the background video vB). This is to render the 3D background data created in resource creation ST1 from the viewpoint corresponding to the position of the camera 502 at each time point.

[0074] Real-time rendering is performed in this way to generate background video vB, which includes frames of the video vBC of the shooting area, and the background video vB is displayed on LED wall 505.

[0075] Camera tracking is performed to obtain imaging information from camera 502, tracking the position, imaging direction, and viewpoint of camera 502 at each time point. By providing this imaging information, along with each frame, to the rendering engine, real-time rendering corresponding to the viewpoint position of camera 502 can be performed.

[0076] Imaging information is information that is linked to or associated with the video as metadata.

[0077] Assume that the imaging information includes the position information of the camera 502 at each frame timing, camera orientation, angle of view, focal length, F-number (aperture value), shutter speed, lens information, etc.

[0078] Lighting control refers to controlling the lighting status in the camera system 500. Specifically, it involves controlling the light intensity, color, and direction of the lights 580. For example, lighting control is performed based on the time and location settings of the scene to be filmed.

[0079] Post-production (ST3) refers to various processing steps performed after filming. These include video correction, video adjustments, editing, and video effects.

[0080] As a video correction tool, it can perform color gamut conversion and color matching between the camera and the footage.

[0081] As a video adjustment tool, it allows for adjustments to color, brightness, contrast, and more.

[0082] As an editor, you can perform editing tasks such as cutting, reordering, and adjusting duration.

[0083] As a video effect, it can be used to composite CG videos or special effects videos.

[0084] The following will explain the configuration of the 500 camera system used in the creation of ST2.

[0085] Figure 5 To show that references have been made Figure 1 , Figure 2 and Figure 3 A block diagram illustrating the configuration of the 500 camera system, which has been described in detail.

[0086] Figure 5 The illustrated photography system 500 includes the aforementioned LED wall 505 comprising multiple LED panels 506, a camera 502, an output monitor 503, and lights 580. (As...) Figure 5 As shown, the photography system 500 also includes a rendering engine 520, a resource server 530, a synchronization signal generator 540, an operation monitor 550, a camera tracker 560, an LED processor 570, a lighting controller 581, and a display controller 590.

[0087] Each LED processor 570 is configured to correspond to one or more LED panels 506 and to drive the video display of the corresponding one or more LED panels 506.

[0088] Synchronization signal generator 540 generates a synchronization signal to synchronize the frame timing of the display video of LED panel 506 with the frame timing of imaging by camera 502, and supplies the synchronization signal to the corresponding LED processor 570, camera 502 and rendering engine 520.

[0089] Camera tracker 560 generates imaging information of camera 502 at each frame timing and supplies it to rendering engine 520. For example, camera tracker 560 detects the relative position information of camera 502 with respect to LED wall 505 or a predetermined reference position, and the imaging direction of camera 502 as one of the imaging information, and supplies this information to rendering engine 520.

[0090] One specific detection method for the camera tracker 560 is as follows: A reflector is randomly placed on the ceiling, and the position is detected based on the reflected light from the infrared light emitted from the camera tracker 560 mounted on the camera 502 and directed to the reflector. Alternatively, another detection method is to estimate the position of the camera 502 based on information from a gyroscope mounted on the gimbal or body of the camera 502, or through image recognition of video footage captured by the camera 502.

[0091] In addition, the camera 502 can provide the rendering engine 520 with information such as field of view, focal length, F-number, shutter speed, and lens information as imaging information.

[0092] Resource server 530 is a server capable of storing the 3D model, i.e., 3D background data, created in resource creation ST1 on a recording medium and retrieving the 3D model as needed. That is, this server serves as a database (DB) for 3D background data.

[0093] The rendering engine 520 processes the generation of the background video vB to be displayed on the LED wall 505. Therefore, the rendering engine 520 reads the required 3D background data from the resource server 530. Then, the rendering engine 520 generates a video of the external view to be used in the background video vB, as a video obtained by rendering the 3D background data in a form viewed from pre-specified spatial coordinates.

[0094] In addition, the rendering engine 520 renders the video vBC (interior view) of the shooting area by using imaging information supplied from the camera tracker 560 or the camera 502 to specify the viewpoint position relative to the 3D background data.

[0095] Then, the rendering engine 520 combines the dynamically changing video vBC of the shooting area, which changes according to the movement of the camera 502, with the external view to generate a background video vB as a 1-frame video data. The rendering engine 520 then sends the generated 1-frame video data to the display controller 590.

[0096] The display controller 590 generates a segmented video signal nD by dividing one frame of video data into video portions to be displayed on each LED panel 506, and sends the segmented video signal nD to the corresponding LED panel 506. At this time, the display controller 590 can perform calibration based on individual differences such as color rendering between the display units and manufacturing errors.

[0097] Note that the display controller 590 can be omitted, and the rendering engine 520 can perform these processes. That is, the rendering engine 520 can generate a segmented video signal nD, perform calibration, and send the segmented video signal nD to the corresponding LED panel 506.

[0098] The LED processor 570 drives the LED panels 506 based on the received segmented video signals nD, thereby displaying the entire background video vB on the LED wall 505. The background video vB includes a video vBC of the shooting area rendered according to the position of the camera 502 at that point in time.

[0099] Camera 502 is capable of capturing the performance of actors 510, including background video vB displayed on LED wall 505, in this manner. The video captured by camera 502 is recorded on a recording medium of camera 502 or an external recording device (not shown) and is supplied in real time to output monitor 503 and displayed as monitor video vM.

[0100] The operation monitor 550 displays the operation image vOP used to control the rendering engine 520. Engineers 511 can perform necessary settings and operations related to the rendering of the background video vB while viewing the operation image vOP.

[0101] The lighting controller 581 controls the luminous intensity, color, and direction of illumination of the lamp 580. For example, the lighting controller 581 can control the lamp 580 asynchronously with the rendering engine 520, or synchronously with imaging information and rendering processing. Therefore, the lighting controller 581 can perform luminous control based on instructions from the rendering engine 520 or a main controller (not shown). Additionally, the rendering engine 520 can control the lamp 580.

[0102] Figure 6 An example of the processing of the rendering engine 520 in the photography system 500 with this configuration is shown.

[0103] In step S10, the rendering engine 520 reads the 3D background data to be used from the resource server 530 and expands the 3D background data into the internal workspace.

[0104] This stage can generate video that can be used as an external view.

[0105] Afterwards, the rendering engine 520 repeats the processing steps S30 to S60 until the display of the background video vB based on the read 3D background data is determined to be finished in step S20.

[0106] In step S30, the rendering engine 520 acquires imaging information from the camera tracker 560 or the camera 502. As a result, the position and state of the camera 502 to be reflected in the current frame are confirmed.

[0107] In step S40, the rendering engine 520 performs rendering based on imaging information. That is, based on the position, imaging direction, and viewing angle of the camera 502 reflected in the current frame, the viewpoint position relative to the 3D background data is determined and rendered. At this time, video processing reflecting focal length, F-number, shutter speed, lens information, etc., can also be performed. Through this rendering, video data as the shooting area video vBC (internal view) can be obtained. The external view can be pre-generated as a fixed video in step S10, or it can be generated for each frame in step S40.

[0108] In step S50, the rendering engine 520 performs a process of compositing the entire background video, i.e., the external view, with a video reflecting the viewpoint position of the camera 502, i.e., the shooting area video vBC. For example, this process is to composite the video generated by reflecting the viewpoint of the camera 502 with the video of the entire background rendered at a specific reference viewpoint. As a result, a 1-frame background video vB is generated that is displayed on the LED wall 505, i.e., a background video vB including the shooting area video vBC.

[0109] The processing in step S60 is performed by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates a segmented video signal nD obtained by dividing a 1-frame background video vB into videos displayed on separate LED panels 506. Calibration can be performed. Then, the segmented video signal nD is sent to the corresponding LED processor 570.

[0110] Using the above processing, at each frame timing, a background video vB, including the shooting area video vBC to be captured by camera 502, is displayed on LED wall 505.

[0111] By the way, in Figure 5 Only one camera 502 is shown, but it is possible to take pictures using multiple cameras 502. Figure 7 An example configuration using multiple cameras 502a and 502b is shown. Cameras 502a and 502b are capable of shooting independently within the performance area 501. Furthermore, synchronization between cameras 502a and 502b and the LED processor 570 is maintained via a synchronization signal generator 540.

[0112] Output monitors 503a and 503b are set up corresponding to cameras 502a and 502b and are configured to display the videos captured by the corresponding cameras 502a and 502b as monitor videos vMa and vMb, respectively.

[0113] In addition, camera trackers 560a and 560b are respectively configured to detect the position and imaging direction of the corresponding cameras 502a and 502b. Imaging information from camera 502a and camera tracker 560a, as well as imaging information from camera 502b and camera tracker 560b, are sent to rendering engine 520.

[0114] The rendering engine 520 is able to render background video vB for each frame by using imaging information from one or both of the camera 502a side or camera 502b side.

[0115] Notice, Figure 7 An example using two cameras 502a and 502b is shown, but it is also possible to take pictures using three or more cameras 502.

[0116] However, when multiple cameras 502 are used and corresponding imaging information is used to render and display the shooting area video vBC (internal view) corresponding to each camera 502, there is a situation where the shooting area videos vBC interfere with each other. For example, in situations such as Figure 7 In the example shown using two cameras 502a and 502b, the shooting area video vBC corresponding to camera 502a is displayed. However, when using the video from camera 502b, a shooting area video vBC corresponding to camera 502b is also required. In this case, simply displaying the shooting area video vBCs corresponding to the respective cameras 502a and 502b will cause them to interfere with each other. Therefore, the display of the shooting area video vBCs needs to be designed.

[0117] <2. Configuration of Information Processing Devices> Next, refer to Figure 8 This document describes a configuration example of the information processing device 70 that can be used in resource creation ST1, production ST2, and post-production ST3.

[0118] The information processing device 70 is a device capable of information processing, especially video processing, such as a computer. Specifically, it is assumed that the information processing device 70 includes personal computers, workstations, portable terminal devices such as smartphones and tablets, and video editing devices. Alternatively, the information processing device 70 can also be a computer device configured as a server device or computing device in cloud computing.

[0119] In this embodiment, specifically, the information processing device 70 can be used as a 3D model making device for creating 3D models in resource creation ST1.

[0120] In addition, the information processing device 70 can also be used as a rendering engine 520 or a resource server 530 that constitutes the photography system 500 used in the production of ST2.

[0121] In addition, the information processing device 70 can also be used as a video editing device for various video processing tasks in post-production ST3.

[0122] Furthermore, the information processing device 70 can also be used for later reference. Figure 10 Information processing device (rendering engine 31) that generates simulated video vSM in the pre-visualization as described above.

[0123] Figure 8The CPU 71 of the information processing device 70 shown performs various processes based on programs stored in non-volatile memory units 74, such as ROM 72 or electrically erasable programmable read-only memory (EEP-ROM), or programs loaded from storage unit 79 into RAM 73. RAM 73 also stores, as appropriate, data required by the CPU 71 to perform various processes.

[0124] The video processing unit 85 is configured as a processor to perform various types of video processing. This processor is capable of performing one or more of the following: 3D model generation, rendering, database processing, video processing including color / brightness adjustment, video editing, and video parsing / detection.

[0125] The video processing unit 85 can be implemented by, for example, a CPU separate from the CPU 71, a graphics processing unit (GPU), a general-purpose computing (GPGPU) on the graphics processing unit, an artificial intelligence (AI) processor, etc.

[0126] Note that the video processing unit 85 can be configured as a function within the CPU 71.

[0127] The CPU 71, ROM 72, RAM 73, non-volatile memory unit 74, and video processing unit 85 are interconnected via bus 83. Input / output interface 75 is also connected to bus 83.

[0128] An input unit 76, including an operating element or operating device, is connected to an input / output interface 75. For example, as an input unit 76, various operating elements and operating devices such as keyboards, mice, buttons, trackballs, dials, touch panels, touchpads, raster panels, and remote controls are assumed.

[0129] Input unit 76 detects user operations, and CPU 71 interprets the signals corresponding to the input operations.

[0130] The microphone is also assumed to be input unit 76. It can also input user voice as operation information.

[0131] In addition, a display unit 77, including a liquid crystal display (LCD) or an organic electroluminescent (EL) panel, and a sound output unit 78, including a speaker, are integrated or separately connected to the input / output interface 75.

[0132] Display unit 77 is a display unit that performs various displays, including, for example, a display device disposed in the housing of information processing device 70, or a separate display device connected to information processing device 70.

[0133] The display unit 77 displays various images, operation menus, icons, messages, etc. on the display screen based on instructions from the CPU 71, that is, it performs the display as a graphical user interface (GUI).

[0134] Sometimes, storage units 79, including hard disk drives (HDDs) or solid-state storage, or communication units 80, are connected to input / output interfaces 75.

[0135] Storage unit 79 can store various types of data and programs. A database (DB) can also be configured in storage unit 79.

[0136] For example, when the information processing device 70 is used as a resource server 530 or a rendering engine 31 as described later, the storage unit 79 can be used to construct a database that stores 3D background data sets.

[0137] The communication unit 80 performs communication processing via transmission paths such as the Internet, and communicates with various devices such as external databases, editing devices, and information processing devices via wired / wireless communication, bus communication, etc.

[0138] For example, when the information processing device 70 is used as the rendering engine 520 or the rendering engine 31 described later, it can access the DB, which is the resource server 530, through the communication unit 80 and receive imaging information from the camera 502 or the camera tracker 560.

[0139] In addition, when using the information processing device 70 for post-production ST3, it is also possible to access the DB, which serves as the resource server 530, through the communication unit 80.

[0140] The drive 81 is also connected to the input / output interface 75 as needed, and a removable recording medium 82 such as a disk, optical disk, optical disc, or semiconductor memory is installed as appropriate.

[0141] Video data, various computer programs, etc., can be read from the removable recording medium 82 via the driver 81. The read data is stored in the storage unit 79, and the video and audio contained in the data are output through the display unit 77 and the audio output unit 78. In addition, computer programs, etc., read from the removable recording medium 82 can be installed in the storage unit 79 as needed.

[0142] In the information processing apparatus 70, for example, software for processing in this embodiment can be installed via network communication using the communication unit 80 or the removable recording medium 82. Alternatively, the software can be pre-stored in the ROM 72 or the storage unit 79, etc.

[0143] <3. First Embodiment> In the following embodiments, the focus is on the pre-visualization process. First, refer to... Figure 9This explains the positioning of pre-visualization during the video production process.

[0144] Figure 9 Previous references are shown Figure 4 The process of resource creation (ST1), production (ST2), and post-production (ST3) has been explained.

[0145] Production (ST2) is the actual shooting stage in the studio, which is divided into shooting preparation and shooting.

[0146] On the other hand, it is assumed that the pre-visualization, which is of interest in this embodiment, is mainly performed in the stage prior to shooting preparation. Note that the order of the time series does not need to be restricted, as long as the pre-visualization is performed after the 3D model used to render the background video vB is created in resource creation (ST1).

[0147] In this sense, pre-visualization can be considered as taking place in the final stage of resource creation (ST1), or as taking place in the stage prior to shooting preparation in production (ST2).

[0148] In pre-visualization, processes known as video verification and moiré alerts are performed. While these processes will be described in detail as examples, video verification is the process that allows the director and crew to review the captured video (vC) before filming. Verification of the tones of the captured video (vC) is primarily achieved using analog video.

[0149] Moiré pattern alerts are handled as follows: while staff virtually manipulate the camera position during video verification, moiré pattern prediction is performed, and a warning is issued if moiré patterns are predicted to appear. This helps prevent artifacts from appearing in subsequent shots.

[0150] During the shooting preparation phase, color calibration is performed along with various settings for the LEDs, cameras, etc., in the photography system 500. Specifically, a calibration lookup table (LUT) is generated to eliminate color changes that occur when the camera 502 photographs the LED wall 505. The calibration LUT is, for example, a 3D-LUT.

[0151] Note that instead of 3D-LUT, you can perform a combination of matrix and 1D-LUT, or other preparations for calibration that are performed in video signal processing.

[0152] During the shooting phase, moiré alerts are performed, LUTs are calibrated, and on-set camera settings are configured.

[0153] In this case, the moiré alert is a real-time output of the alert to the cameraman and others based on the moiré determination of the rendering engine 520.

[0154] The calibration LUT is a process that eliminates color variations associated with the background video display of LED wall 505, camera 502, etc. by applying the LUT generated during color calibration during shooting preparation.

[0155] The on-site camera is configured to retrieve camera settings set in pre-visualization from, for example, a cloud server and reflect those camera settings in camera 502 in the studio.

[0156] The following is for reference Figure 10 The following describes a configuration example as a first embodiment. Note that in the following embodiments, the configuration of the photography system 500 and the configuration for pre-visualization will be described in parallel. Figure 10 , Figure 13 , Figure 15 and Figure 18 The upper part of the attached diagram shows Figure 5 The simplified configuration of the photography system 500 described above is shown in the lower part of the attached figure, which is the configuration used in pre-visualization (pre-visualization system 700).

[0157] Regarding the camera system 500, Figure 10 The rendering engine 520, LED processor 570, LED wall 505, camera 502, camera signal processing unit 515, and operation monitor 550 or output monitor 503 are shown.

[0158] The operation monitor 550 or output monitor 503 in this figure generally refers to a monitor that displays the captured video vC and the monitor video vM in the studio, which serves as the shooting location. For ease of explanation, it will be abbreviated as "monitor 550, etc."

[0159] The rendering engine 520 is shown as including a rendering unit 520a and a color calibration unit 520b.

[0160] As described above, rendering unit 520a represents the function of rendering the internal view (video of the shooting area) vBC based on the external view and the position and imaging direction of camera 502, and generating the background video vB to be displayed on LED wall 505. That is, rendering unit 520a is responsible for... Figure 6 The processing function within.

[0161] The rendering engine 520 can function as a color calibration unit 520b. The color calibration unit 520b refers to the LED panel 506 constituting the LED wall 505, which is used to eliminate the processing of the LED processor 570. Figure 10 The function of calibration processing for color changes caused by factors such as LED light emission characteristics (not shown in the figure), camera light reception characteristics, and processing.

[0162] That is, the color calibration unit 520b performs color transformation on the rendered background video vB by using the LUT generated by the color calibration in the shooting preparation stage.

[0163] When performing LUT calibration in the rendering engine 520 as a function of color calibration unit 520b, it is possible to prevent color changes on the display side of the background video vB.

[0164] Note that the color changes mentioned here refer to the color changes caused by the processing, light emission characteristics, camera light reception characteristics, and processing, up to the point where the rendered background video vB is displayed on the LED wall 505.

[0165] When the rendering engine 520 does not have the function of color calibration unit 520b, or even if the rendering engine has the function but it is not enabled, sometimes there are changes in the color of the state after rendering from the 3D model and the color of the state displayed on the LED wall 505 due to hue, i.e. color difference, RGB balance, etc.

[0166] Figure 10 A camera signal processing unit 515 is shown that processes the video signal captured by camera 502. Although in Figure 5 and in Figure 7 The details are omitted, but the camera signal processing unit 515 may be formed by a processor or the like within the camera 502, or it may be a device that is set as a unit separate from the camera 502.

[0167] The video signal captured by the camera 502 is processed by the camera signal processing unit 515, including brightness processing, color processing, development processing, size adjustment processing, etc., and is output as captured video vC or monitor video vM.

[0168] The captured video (vC) is recorded on a recording medium as the main video feed, or sent to other devices for broadcasting or distribution. Additionally, the captured video (vC) is used as a monitor video (vM) and displayed on monitors such as the 550.

[0169] For the aforementioned photography system 500, for example, the pre-visualization system 700 is configured as a system outside the photography studio. The pre-visualization system 700 includes at least an information processing device 70 serving as the rendering engine 31 and a monitor 40.

[0170] Rendering engine 31 may include, for example, an information processing unit 70 separate from rendering engine 520.

[0171] The rendering engine 31 includes a rendering unit 32 and a video processing unit 33.

[0172] The rendering unit 32 has similar rendering capabilities to the rendering unit 520a in the rendering engine 520 described above, and can generate background video vBP based on the 3D model.

[0173] Rendering unit 32 is capable of rendering background video VB using at least the 3D model created in the resource creation (ST1) step. That is, for example, from resource server 530 (see reference...) Figure 5 and Figure 7 Obtain the same 3D model used in the production (ST2) step and use that 3D model for rendering.

[0174] Then, the rendering unit 32 can render, for example, an interior view using a 3D model. Note that when generating an interior view (video capture area vBC), an arbitrary camera position and orientation are input, and rendering is performed at a viewpoint position relative to the 3D model corresponding to that arbitrary camera position and orientation.

[0175] For ease of explanation, the background video generated by the rendering unit 32 will be referred to as "background video vBP" in contrast to the shooting area video vBC generated by the rendering engine 520 during shooting.

[0176] The video processing unit 33 represents the function of performing various video processing on the background video vBP.

[0177] The video processing unit 33 performs live-shot video conversion processing as one type of video processing. Live-shot video conversion processing refers to the process of generating a simulated video vSM that mimics the camera 502 used for actual shooting. In live-shot video conversion processing, at least processing parameters are used to generate the simulated video vSM for the background video vBP. These processing parameters realize the brightness or color characteristics when the video was shot by the camera 502 used by the photography system 500.

[0178] By setting the parameters related to brightness or color to be the same as those of the camera 502 at the time of shooting, an analog video vSM with the same hue as the shot video vC is generated, and the analog video vSM is displayed on the monitor 40.

[0179] Monitor 40 refers to a monitor device that displays the video output from the information processing unit 70, which is the rendering engine 31. It can be, for example, a monitor device located in a different location from the photography studio, or a monitor device in the photography studio. Monitor 40 can take any form as long as it displays the analog video vSM at least during the pre-visualization stage.

[0180] In rendering engine 31, rendering unit 32 generates background video vBP, and video processing unit 33 performs live-shot video conversion processing to generate analog video vSM. By displaying this on monitor 40, the operator can confirm that the analog video vSM has the same tone as the background video contained in the live-shot video vC. As a result, color confirmation can be performed before shooting, and the colors in camera 502 or camera signal processing unit 515 can be adjusted as needed during the shooting preparation stage.

[0181] As part of the live-shot video conversion processing of this video processing unit 33, the hue of the background video vBP rendered by the rendering unit 32 is set to be equivalent to the background video contained in the live-shot video vC.

[0182] Therefore, specifically, the video processing unit 33 adjusts parameters such as RGB gain, white balance value, gamma value and offset value to the parameters of the camera 502.

[0183] For example, the video processing unit 33 processes the background video vBP to give R, G, and B gains similar to the gain values ​​for the pixel signals of the corresponding colors read from the image sensor in the camera 502.

[0184] In addition, the video processing unit 33 performs white balance processing on the background video vBP, using the same white balance value as the white balance value when the camera 502 was shooting, in the real-shot video conversion processing.

[0185] The result is that an analog video vSM, which serves as the background, has the same color tone as the background in the captured video vC. As a result, on monitor 40, it is possible to confirm the video with the same color tone as when the captured video vC was viewed on monitor 550, etc., during filming.

[0186] Furthermore, as a real-shot video conversion process, the video processing unit 33 can perform processing such as cropping range and zoom ratio settings from the image sensor of the camera 502 to make the viewing angle the same as the background video vBP.

[0187] For example, Figure 11 An example of the readout range of the image sensor 30 of the camera 502 is shown. It is assumed that the solid line represents the entire pixel area of ​​the image sensor 30. The readout of the photoelectric conversion signal, which is the imaging signal from the image sensor 30, can sometimes be performed over the entire pixel area represented by the solid line, or it can be performed under various conditions depending on the readout mode, such as a range represented by a dotted line, a range represented by a dashed line, a range represented by a single-dot-dash line, etc. The viewing angle of the video vC varies depending on these differences in readout range.

[0188] Therefore, in the real-shot video conversion processing of video processing unit 33, the perspective is also made equivalent to the shooting video vC.

[0189] In addition, as part of the live-action video conversion process, the video processing unit 33 can process the background video vBP to reflect various characteristics of the lens mounted on the camera 502 (such as lens type (spherical or anamorphic), color, transmittance, flare, distortion aberration, and peripheral light loss). As a result, the analog video vSM is made closer to the live-action video vC.

[0190] Furthermore, as part of the live-action video conversion process, the video processing unit 33 can perform processing that reflects the characteristics of filters, such as the circularly polarized light filter installed on the camera 502, onto the background video vBP. As a result, even when the filter is installed and shooting is performed, the analog video vSM is made closer to the live-action video vC.

[0191] Simultaneously, during filming, color calibration processing in the rendering engine 520 eliminates color changes up to the LED wall 505, and the live-action video conversion processing can reflect color changes at the camera side. This is because it is assumed that the background video vB displayed on the LED wall 505 has no color change compared to its rendered state.

[0192] On the other hand, without the aforementioned color calibration, color changes sometimes occur in the background video vB displayed on the LED wall 505. In this case, during the live video conversion processing in the video processing unit 33, processing is also performed to reflect the luminance (intensity) characteristics of the corresponding color on the LED side. As a result, the analog video vSM can be made closer to the captured video vC.

[0193] Note that, as a prerequisite for the above-mentioned real-shot video conversion process, when rendering engine 31 renders the simulated video vSM, the assumed camera settings of the virtual camera, such as frame rate, shutter speed, and exposure value, are similar to those of the actual camera 502. That is, after setting the background video vBP to have RGB values ​​corresponding to various camera settings such as frame rate, shutter speed, exposure value, and ND setting, video processing unit 33 performs color processing such as RGB gain and white balance value to generate a simulated video vSM that mimics the tone of the real-shot video vC.

[0194] It can reflect the brightness of the LED wall 505 in a similar way to camera settings.

[0195] Figure 12 An example of the processing of rendering engine 31 during pre-visualization is shown.

[0196] First, in step S100, the rendering engine 31 acquires the camera settings and white balance of the camera 502 during shooting, and sets the processing parameters accordingly. Basic parameters such as frame rate, shutter speed, and exposure value are set. Additionally, in this step S100, the rendering engine 31 can acquire various characteristics of the lens mounted on the camera 502 during shooting and the characteristics of the filters, and reflect them in the processing parameters.

[0197] In step S101, the rendering engine 31 determines the end of monitoring the simulated video vSM in the pre-visualization. For example, until the end, the rendering engine 31 performs the processing in step S102 and thereafter for each frame.

[0198] In step S102, the camera position is set. This is a process of virtually setting the relative position and orientation (imaging direction) of the camera 502 relative to the LED wall 505. That is, this is a process of setting the viewpoint position relative to the 3D model during rendering.

[0199] If the position and path (camera movement trajectory) of camera 502 during actual shooting are set in the pre-visualization stage, then only this information needs to be used to set the camera position for each frame. Alternatively, staff can input arbitrary camera position settings during the pre-visualization stage.

[0200] In step S103, after setting the viewpoint position according to the set camera position, the rendering engine 31 uses a 3D model to render the background video vBP. For example, it generates a background video that is equivalent to the internal view (the video vBC of the shooting area).

[0201] In step S104, the rendering engine 31 determines whether to eliminate the color change up to the LED wall 505 through color calibration processing during shooting.

[0202] The rendering engine 31 can determine whether color calibration processing was performed during shooting and the content of such processing through user input, or it can determine this through communication with the rendering engine 520 of the camera system 500, provided that communication and collaboration with the camera system 500 are possible. When the rendering engine 31 is also used as a processing system for the rendering engine 520, it can know whether color calibration processing was performed without communication.

[0203] When no color calibration is performed during shooting, the rendering engine 31 performs step S105 as part of the live-action video conversion process. That is, it performs luminance and color processing, ensuring that the luminance or color variations generated in the background video vB displayed on the LED wall 505 are also reflected in the background video vBP. For example, color correction is performed on the background video vBP by setting gain parameters corresponding to the luminance balance of the corresponding colors in the RGB of the LED panel 506.

[0204] During shooting, color calibration processing eliminates the color change until the LED wall 505 changes. Since the background video vB displayed on the LED wall 505 does not produce a color change, the processing in step S105 is skipped.

[0205] That is, when color calibration is performed during shooting, the changes in brightness or color in the background video vB displayed on the LED wall 505 are eliminated. However, when color calibration is not performed during shooting, the changes in brightness or color in the background video vB displayed on the LED wall 505 affect the color of the shot video vC. Therefore, when color calibration is not performed during shooting, the process in step S105 is performed until the color changes of the LED wall 505 are also reflected in the analog video vSM.

[0206] Next, rendering engine 31 performs step S106 as part of the live-action video conversion process. This involves brightness and color processing, ensuring that color changes generated on the camera 502 side are also reflected in the background video vBP. For example, as described above, processing is performed to set the white balance value and RGB gain to be similar to those on camera 502.

[0207] In addition, note that in the real-shot video conversion process in step S106, the rendering engine 31 can perform processing that reflects the perspective, lens characteristics, filter characteristics, etc.

[0208] The background video vBP that has undergone the processing in step S106 or steps S105 and S106 becomes the analog video vSM.

[0209] In step S107, the rendering engine 31 outputs one frame as an analog video vSM. For example, the frame is output to the monitor 40 and displayed.

[0210] By having the above processing performed by the rendering engine 31, staff members who use the camera system 500 to shoot can check the color tone of the video VC to be shot in advance through the monitor 40.

[0211] <4. Second Embodiment> Figure 13 A configuration example of the second embodiment is shown.

[0212] In the second embodiment, staff can actively adjust the colors. On the monitor 40, an analog video (vSM) obtained through the conversion processing of the live video by the video processing unit 33 is displayed as described above, allowing staff to confirm the color tone.

[0213] In this case, the rendering engine 31 is equipped with the function of the adjustment operation unit 35, which allows users such as staff to perform color adjustment operations. For example, a GUI is provided to the user using the display unit 77 or input unit 76 of the information processing device 70 that serves as the rendering engine 31. As a result, the user can manipulate the color tone of the video while viewing the monitor 40.

[0214] That is, when staff watch the simulated video vSM obtained by converting and processing the live video and feel that the color tone is different from the expected color tone, the staff will make color adjustments to obtain the desired color.

[0215] In addition, by providing the adjusted content as color conversion information Ic to the camera 502 or the camera signal processing unit 515, the adjusted color can be reflected during shooting.

[0216] Specifically, in the above Figure 12 In addition to its own processing, the rendering engine 31 also utilizes the functions of the video processing unit 33 to perform... Figure 14 The processing is shown.

[0217] In step S201, the rendering engine 31 monitors the color adjustment operations performed by the user using the GUI through the adjustment operation unit 35.

[0218] When a color adjustment operation is detected, the rendering engine 31 proceeds to step S202 to perform color adjustment processing corresponding to the color adjustment operation.

[0219] Additionally, in step S203, the rendering engine 31 monitors the decision operation of the adjustment operation unit 35. This decision operation is the operation of generating instructions for color conversion information Ic, which is used to enable the reproduction of the adjusted color state later.

[0220] Specifically, the color conversion information Ic is the following LUT or color determination table (CDL), etc.

[0221] When the decision operation is performed, the rendering engine 31 proceeds to step S204 to generate a LUT or CDL as color conversion information Ic.

[0222] In this case, the LUT is, for example, a three-dimensional LUT (3D LUT), which uses R, G, B as three axes and stores the adjusted R, G, B values ​​for the original color represented by grid points on the three axes.

[0223] CDL has parameters for adjusting the slope, offset, and power of R, G, and B respectively, as well as parameters for adjusting saturation.

[0224] Color conversion information Ic, as used in LUTs or CDLs, represents, for example, color changes caused by user adjustments based on the colors of a simulated video vSM generated during live-action video conversion processing.

[0225] In step S205, the rendering engine 31 performs the process of storing the generated LUT or CDL and outputting the LUT or CDL to a predetermined device.

[0226] For example, as the output destination of a LUT or CDL, one might think of using, such as Figure 13 The camera 502 or camera signal processing unit 515 shown.

[0227] In other words, in the second embodiment, when the staff can pre-confirm the tone and other properties of the captured video vC through the simulated video vSM, the staff can actively adjust the colors. Then, a LUT or CDL corresponding to the adjusted content is generated, making it possible to use the LUT or CDL during subsequent shooting.

[0228] By importing a LUT or CDL into the camera 502 or the camera signal processing unit 515, color adjustment can be performed by applying the LUT or CDL to the captured video vC.

[0229] In other words, staff can pre-determine the color tone of the captured video (vC) and adjust it as needed. By storing / outputting it as a LUT or CDL, these color adjustments can be applied to the video processing during filming. The result is that the captured video (vC) undergoes color adjustments similar to those in the pre-visualization. In other words, the hue of the captured video (vC) can be pre-determined and even adjusted.

[0230] <5. Third Embodiment> The third embodiment is an example of how the video processing unit 33 of the rendering engine 31 processes camera setting information that is to be reflected in the analog video vSM according to the input and can output the camera setting information.

[0231] The camera settings information mentioned here refers to, for example, readout size settings, shutter speed settings, exposure settings, ND settings, lens characteristics, focus / aperture / zoom operation settings, white balance values, and RGB gain from the image sensor 30.

[0232] In the above-mentioned real-shot video conversion process, these settings are basically adjusted to the settings of the camera 502 used for shooting, thereby simulating real-shot video vC.

[0233] Then, in the third embodiment, by adjusting these camera settings, it is possible to prepare various camera settings in advance.

[0234] Figure 15The video processing unit 33 is shown performing camera preset processing.

[0235] Camera preset settings are, for example, processes where an operator changes the camera settings by operating the adjustment unit 35 while viewing an analog video vSM on monitor 40. The video processing unit 33 performs video processing, such as transformations of RGB luminance values, and virtually reflects changes in exposure settings in the analog video vSM based on operations such as shutter speed or exposure settings.

[0236] Furthermore, the video processing unit 33 enables the storage or transmission of camera settings that change according to the operation. For example, when shooting later, the camera settings corresponding to the adjustment operation can be reflected in the camera 502, etc.

[0237] Figure 15 The diagram illustrates sending camera settings to cloud server 41. It is assumed that cloud server 41 is capable of storing camera settings originating from rendering engine 31 and subsequently providing these camera settings to photography system 500.

[0238] For example, while confirming the simulated video vSM in the pre-visualization, the user can adjust various camera settings, such as adjusting the camera settings for each scene. These are uploaded to cloud server 41, so cloud server 41, for example... Figure 16 The camera settings are stored according to each scene SC1, SC2, and SC3, as shown.

[0239] This camera setting can be downloaded from the cloud server 41 to the camera 502 or the camera signal processing unit 515, thereby enabling the use of the camera setting during actual shooting.

[0240] Note that, in addition to camera settings, the LED side settings can be similarly adjusted, enabling the setting information to be downloaded to the LED processor 570.

[0241] In addition to the information mentioned above, camera settings may also include LUTs or CDLs as described in the second embodiment.

[0242] Figure 17 An example of rendering engine 31 using the functions of video processing unit 33 is shown. For example, rendering engine 31, in addition to... Figure 12 processing and Figure 14 In addition to processing, it can also perform Figure 17 The processing.

[0243] In step S301, the rendering engine 31 monitors whether the adjustment operation unit 35 has performed an adjustment operation related to the camera settings.

[0244] During the adjustment operation, in step S302, the rendering engine 31 changes the camera settings. Note that since there is no actual camera, the setting change modifies the necessary video processing parameters to achieve video processing based on the camera settings.

[0245] In this state, users can watch simulated video vSM with the camera settings changed.

[0246] In step S303, the rendering engine 31 detects a setting storage command. When the user provides a setting storage command through operations such as adjusting the operation unit 35, the rendering engine 31 proceeds to step S304 to store or send the camera setting information at that point in time. For example, the rendering engine 31 performs processing to store the information in an internal storage medium or send it to an external device such as a cloud server 41.

[0247] The result is that the camera settings adjusted by the staff during the pre-visualization stage can be used in the camera system 500 during subsequent shooting.

[0248] <6. Fourth Embodiment> Reference Figure 18 The fourth embodiment is described below. The fourth embodiment is an example of being able to output moiré alerts during the pre-visualization stage.

[0249] Figure 18 The rendering engine 31 shows the moiré prediction unit 36 ​​and the position manipulation unit 37. The position manipulation unit 37 is an interface function that allows the user to arbitrarily manipulate the position of the virtual camera.

[0250] The moiré prediction unit 36 ​​is a function that uses a specific algorithm to predict whether moiré patterns will appear under the state of the set camera position.

[0251] A specific algorithm is one that determines, for example, artifacts such as moiré patterns or scan lines that will occur when actually taking a picture in this state, based on information such as: • Camera position, angle, aperture / focus / zoom status • The position and angle of the monitor (the virtual LED wall set up by rendering engine 31) When moiré patterns are predicted to appear under conditions such as camera position, the moiré pattern prediction unit 36 ​​notifies the video processing unit 33 of the information on the generation of moiré patterns and the information on the generation location.

[0252] The video processing unit 33 causes the monitor 40 to display video, for example, a warning indicating a moiré pattern or the location of a moiré pattern.

[0253] For example, Figure 19The video shown includes a background video vBP and an image of a virtual LED wall 54 and a virtual camera 51.

[0254] The video processing unit 33 can not only simply display the analog video vSM of the background video vBP, but also display, for example, videos with the viewpoint set at a certain location in the studio.

[0255] exist Figure 19 The video also shows the location information 52 of the LED wall 54 and the camera 52.

[0256] Additionally, the origin information 50 shows the origin position, which serves as a reference for each location in space, as well as the directions of the three axes: X, Y, and Z.

[0257] Figure 20 An example is shown to display moiré alarm 53 and to show moiré position 57.

[0258] For example, the position of camera 51 is from Figure 19 The position in the image is changed, assuming that moiré pattern generation prediction is performed under this condition. In this case, moiré pattern alarm 53 is displayed, and the location where the moiré pattern is generated is displayed as moiré pattern position 57.

[0259] Note that the user can arbitrarily change the position of camera 51 on the video. For example, when the user performs an operation to specify camera 51, such as... Figure 21 As shown, the X, Y, and Z axes of movement 55 are displayed. When the user aligns the cursor 56 with a certain axis of movement 55 and performs operations such as dragging, the position of the camera 51 in the virtual space is changed. For example, the user interface described above is provided by the function of the position operation unit 37.

[0260] Therefore, users can check whether moiré patterns are generated while moving the virtual camera 51 at will.

[0261] As a result, it is also possible to set the camera position and camera path during actual shooting to avoid moiré patterns.

[0262] Figure 22 This illustrates the moiré-related processing performed by rendering engine 31. In addition to, for example... Figure 12 processing, Figure 14 processing, Figure 17 In addition to its processing, rendering engine 31 also performs Figure 22 The processing.

[0263] In step S401, the rendering engine 31 monitors whether camera position operation has been performed.

[0264] For example, when the user performs camera position operation through the function of position operation unit 37, rendering engine 31 proceeds to step S402 and uses the information of the changed camera position and imaging direction to perform moiré prediction.

[0265] Note that the process can proceed to step S402 not only when camera position is adjusted, but also when zooming, exposure adjustment, or other similar operations are performed.

[0266] In step S403, the rendering engine 31 confirms the result of the moiré prediction. When the moiré is predicted to occur, the engine outputs a moiré alarm in step S404.

[0267] As shown above, the display is as follows Figure 20 The moiré pattern alarm 53 shown allows staff to be aware of potential moiré patterns before shooting. This allows them to consider camera positions and paths that will not cause moiré patterns before actual shooting.

[0268] On the other hand, the rendering engine 520 is also able to determine moiré patterns during actual shooting.

[0269] Figure 18 The rendering engine 520 shows the CG camera setting unit 520c and the moiré determination unit 520d, in addition to the rendering unit 520a.

[0270] The CG camera setting unit 520c receives imaging information from the camera tracker 560, such as the position of the camera 502, imaging direction, viewing angle, focal length, F-number, shutter speed, lens information, etc., and causes the rendering unit 520a to perform rendering corresponding to this imaging information.

[0271] The moiré pattern determination unit 520d uses information obtained from the CG camera setting unit 520c to determine whether moiré patterns are currently generated in the rendered video vBC of the shooting area. In other words, it uses an algorithm similar to that of the moiré pattern prediction unit 36 ​​to determine whether moiré patterns are generated.

[0272] Then, when it is determined that moiré patterns have been generated, the information is sent to the camera 502, for example, so that a moiré pattern alarm is displayed on the viewfinder or the like.

[0273] Figure 23 This shows an example of how to process such a real-time moiré alert. Note that in Figure 23 In addition to Figure 6 In addition to steps S10 to S60 as described above, step S70 and subsequent steps are also performed.

[0274] As described above, the rendering engine 520 performs steps S30 to S60 at regular intervals for each frame. The result is that the background video vB is displayed on the LED wall 505.

[0275] The rendering engine 520 performs moiré pattern determination in step S70.

[0276] In step S71, the rendering engine 520 confirms the result of the moiré pattern determination. If moiré patterns are determined to have occurred, the rendering engine 520 processes the output of a moiré pattern alarm in step S72. For example, the rendering engine 520 sends the moiré pattern alarm information to the camera 502, causing the moiré pattern alarm to be displayed on the viewfinder or the like. Alternatively, the moiré pattern alarm can be displayed on the monitor 550 or the like.

[0277] As mentioned above, the photographer can identify and address moiré patterns that appear during shooting. For example, they can adjust the camera position or focus to counteract the moiré patterns.

[0278] <7. Imaging Data Stream> Next, we will explain the format in which the captured video vC obtained in the production using the camera system 500 (ST2) is provided to the post-production (ST3).

[0279] The video footage (vC) captured by camera 502 can be sent to a distribution server and streamed in real time. As a result, some production staff can remotely monitor the captured video footage (vC) on an external monitor.

[0280] In addition, the video footage can be converted into motion image files and forwarded to a cloud server, where it can be saved in a shareable manner by post-production staff, enabling them to perform various post-production tasks.

[0281] At this point, the imaging information used during shooting, such as the camera 502's position, camera orientation, angle of view, focal length, aperture value, shutter speed, lens information, and the LUT or CDL used, is converted into metadata and included in a file. Alternatively, the imaging information, LUT, CDL, etc., can be viewed in at least association with the moving image. As a result, editing using the information from the shooting process becomes possible.

[0282] Motion picture files can be any format, such as RAW motion picture files, Moving Picture Experts Group (MPEG)-4, Audio Video Interleaved (AVI), Quick File Format (MOV), or Windows Media Video (WMV), Mezzanine Files, or proxy motion picture files.

[0283] As described above, the confirmation or adjustment work in the pre-visualization can be performed by the rendering engine 31 of the information processing device 70 outside the studio.

[0284] Furthermore, as mentioned above, the real-time monitoring or sharing of video footage via cloud servers enables post-production work without relying on a photography studio. In other words, it facilitates remote work and efficiency across a range of video production tasks.

[0285] <8. Summary and Variations> The following effects can be obtained based on the above embodiments.

[0286] The rendering engine 31 of the embodiment implemented by the information processing device 70 includes a rendering unit 32 that renders a 3D model used during filming by the camera system 500 to generate a background video vBP as a virtual video. Additionally, the rendering engine 31 includes a video processing unit 33 that performs live-action video conversion processing on the background video vBP generated by the rendering unit 32, for use in generating a simulated video vSM by using processing parameters that implement the luminance or color characteristics captured by the camera 502 used in the camera system 500.

[0287] The result is a monitor video that displays the brightness and color conditions of the captured video (vc) before shooting. Consequently, the director or crew can confirm the brightness and color conditions of the captured video (vc) before filming.

[0288] During the actual filming phase, the resulting video VC (video file) may differ from the color tone and other aspects envisioned by the director and others. In such cases, color adjustments in the on-set studio are time-consuming and reduce production efficiency. Therefore, it is crucial to confirm the color tone and other aspects of the video VC before filming.

[0289] Additionally, during virtual production, a 3D model for the background video VB is created in the resource creation (ST1) phase. That is, in the stage before filming, an environment capable of rendering the background video VB is established. Therefore, as... Figure 10 As shown, the video processing unit 33 adds real-shot video conversion processing, thereby enabling pre-confirmation of luminance and color.

[0290] Note that in the embodiment, it has been stated that rendering engine 31 is implemented by an information processing device 70 that is different from rendering engine 520, but rendering engines 31 and 520 may also be implemented by a single information processing device 70.

[0291] In the embodiment, the following situation is described: the video processing unit 33 performs display video processing using processing parameters that realize the brightness or color change characteristics of the video on the LED wall 505. Figure 12 The video processing in step S105 is then performed using the processing parameters that realize the brightness or color characteristics of the camera 502 during shooting. Figure 12 Step S106 (video processing) to generate a simulated video vSM.

[0292] When there is a change in color or brightness on the LED wall 505 side, that is, when a change in color or brightness occurs during the processing of the LED panel 506 or the LED processor 570, this change can also be taken into account to simulate the captured video vC during shooting. As a result, the simulation accuracy can be improved, which is suitable for video verification.

[0293] Note that, as referenced Figure 10 As described above, through processing performed by the color calibration unit 520b in the rendering engine 520, color changes caused by the device or processing can be eliminated in the background video vB on the LED wall 505 side. In the case of such a photography system 500, since there is no reflection of the characteristics on the LED wall 505 side, real-shot video conversion processing simulating camera characteristics can be performed.

[0294] In the second embodiment, an example is described whereby the video processing unit 33 performs color adjustment processing on the analog video vSM according to the control signal based on the user operation, and generates color conversion information Ic based on the color adjustment processing.

[0295] For reference Figure 13 The video processing unit 33 generates a LUT or CDL as color conversion information Ic. Operators can perform color adjustment operations while monitoring the monitor 40, and a LUT or CDL is generated accordingly. In other words, by creating the desired color during the pre-visualization stage, color conversion information Ic can be generated to achieve that color creation.

[0296] As described in the second embodiment, the video processing unit 33 is capable of outputting color conversion information Ic.

[0297] When generating a LUT or CDL as color conversion information Ic, the video processing unit 33 outputs the LUT or CDL, enabling the LUT or CDL to be used by the camera 502, camera signal processing unit 515, etc. As a result, color adjustment using the LUT, etc., set in the pre-visualization stage can be easily achieved during actual shooting.

[0298] In the third embodiment, an example is described whereby the video processing unit 33 can process camera setting information to be reflected in the analog video vSM according to control signals based on user operation, and output the changed camera setting information.

[0299] For reference Figure 15 The video processing unit 33 maintains the camera 502's settings during shooting as camera presets. These settings include, for example, readout size settings from the image sensor 30, shutter speed settings, exposure settings, ND settings, lens characteristics, focus / aperture / zoom operation settings, white balance settings, etc. Note that settings for LUTs, etc., to be applied are also included.

[0300] For example, these settings can be changed from the same state as camera 502 through operator intervention. Then, the adjusted camera setting information can be output and provided to camera 502, camera signal processing unit 515, or LED processor 570. As a result, the video can be confirmed during the pre-visualization stage, while simultaneously adjusting the camera settings to their optimal state for shooting. Actual shooting can then begin from the state confirmed during pre-visualization.

[0301] In this embodiment, a white balance value is included as a processing parameter to realize the brightness or color characteristics of the camera 502 used in the photography system 500 during shooting.

[0302] It is important to be able to confirm the color tone of the video vC to be shot before shooting. Therefore, in the pre-visualization, it is best to generate a simulated video vSM that has undergone real-shot video conversion processing, including the white balance value applied by the camera 502 at the time of shooting.

[0303] Additionally, in the embodiments, gain values ​​for pixel signals of each color read from the image sensor in the camera 502 are included as processing parameters to realize the brightness or color characteristics of the camera 502 used in the photography system 500 during shooting.

[0304] As elements used to confirm the tone of the captured video vC, there is, for example, color balance determined by the gain of each pixel signal (R, G, and B). Therefore, in pre-visualization, it is best to perform a live video conversion process that includes the gain (color balance) of the R, G, and B signals applied in the camera 502 during shooting.

[0305] Additionally, in the embodiments, an example is described whereby the video processing unit 33 generates a simulated video vSM by utilizing the perspective settings based on the perspective information of the camera 502 used in the photography system 500 during shooting in the live-action video conversion process.

[0306] Analog video (vSM) is a video with the same perspective as the recorded video (vC) at the time of recording. This is because, for video verification, it is preferable that the analog video (vSM) has the same tone and the same perspective as the recorded video (vC).

[0307] Note that the viewpoint information includes viewpoint information based on zoom settings and viewpoint information representing the setting of the clipping range from image sensor 30.

[0308] Additionally, in the embodiments, an example is described whereby the video processing unit 33 generates a simulated video vSM that reflects the lens characteristics of the camera 502 used in the photography system 500 during the live video conversion process.

[0309] Since the preferred analog video (vSM) is as close as possible to the actual video (vC) captured during filming, it also exhibits characteristics such as lens distortion.

[0310] Additionally, in the embodiments, an example has been described whereby the video processing unit 33 generates an analog video vSM that reflects the characteristics of the filter installed on the camera 502 used in the photography system 500 during the live video conversion process.

[0311] When the filter is installed on the camera 502, by reflecting the characteristics of the filter, it is possible to make the analog video vSM approximate the captured video vC during shooting.

[0312] In the fourth embodiment, an example of performing moiré warning output processing based on the moiré prediction information determined by the background video vBP rendered by the rendering unit 32 is described.

[0313] For reference Figure 18 The process described above involves displaying a moiré alarm 53 on a monitor 40 or similar device based on the moiré prediction information from the moiré prediction unit 36. As a result, it is possible to predict the moiré pattern during the pre-visualization stage.

[0314] Additionally, in the fourth embodiment, an example is described of preparing a user interface for arbitrarily changing the camera position on the video and performing moiré prediction based on the camera position changed through the user interface (see reference). Figure 19 , Figure 20 , Figure 21 and Figure 22 ).

[0315] When enabling staff to simulate various camera positions in video footage, the generation of moiré patterns can be predicted based on the camera position. This also aids in the conceptualization of camera work, such as selecting camera positions where moiré patterns do not appear.

[0316] Furthermore, in the fourth embodiment, an example of processing to display a moiré alarm on the viewfinder of the camera 502, etc., based on the moiré determination information of the moiré determination unit 520d is also described (see reference). Figure 18 and Figure 23 As a result, photographers and others are able to be aware of and respond to moiré patterns during filming.

[0317] Note that the moiré prediction unit 36 ​​and the moiré determination unit 520d can use the same or different algorithms for moiré prediction. Additionally, when using the same algorithm, the prediction / determination thresholds can be the same or different.

[0318] The information processing device 70, which will be used as the rendering engine 31 in the embodiments, will be described as a device separate from the device constituting the photography system 500.

[0319] The information processing device 70 for pre-visualization is a device different from the equipment constituting the photography system 500 built in the photography studio. For example, staff use devices other than studio equipment, such as PCs or tablets, as the information processing device 70 in this embodiment. As a result, staff can verify the simulated video vSM in a location other than the studio, which is beneficial for improving the efficiency of video content production.

[0320] Note that the rendering engine 520 in the camera system 500 can also be used to generate the analog video vSM described in the embodiments. That is, if the rendering engine 520 has the same functions as the video processing unit 33 and performs live-action video conversion processing, the analog video vSM can also be confirmed before shooting begins, for example, by using the camera system 500.

[0321] The program of the embodiment is for causing a processor, such as a CPU, a DSP, or a device including such a processor, to execute as described above. Figure 12 , Figure 14 , Figure 17 and Figure 22 The procedure shown is for processing.

[0322] That is, the program of the embodiment is a program for causing the information processing device to perform the following processes: processing to generate a virtual video (e.g., background video vBP) by rendering using a 3D model used in the photography system 500; and real-shot video conversion processing, which generates a simulated video vSM for the generated virtual video using processing parameters that implement the brightness or color characteristics of the camera 502 used in the photography system 500 at the time of shooting.

[0323] Using such a program, an information processing device 70 that generates analog video vSM in the above-mentioned pre-visualization can be realized through various computer devices.

[0324] Such programs can be pre-recorded on HDDs, ROMs within microcomputers including CPUs, or other recording media built into computer devices. Furthermore, such programs can be temporarily or permanently stored (recorded) on removable recording media such as floppy disks, CD-ROMs, magneto-optical (MO) discs, DVDs, Blu-ray discs (registered trademark), magnetic disks, semiconductor memory, or memory cards. These removable recording media can be provided as software packages.

[0325] In addition, such programs can be installed on personal computers from removable recording media, or downloaded from download sites via networks such as local area networks (LANs) or the Internet.

[0326] Furthermore, such a program is suitable for information processing apparatus 70, which is widely provided as the rendering engine 31 in the embodiment. For example, by downloading the program to personal computers, communication devices, portable terminal devices such as smartphones or tablets, mobile phones, gaming devices, video devices, personal digital assistants (PDAs), etc., these devices can be used as the information processing apparatus 70 (rendering engine 31) of this disclosure.

[0327] Note that the effects described in this instruction manual are illustrative only and not limiting; other effects may also be present.

[0328] Note that this technology can also be configured as follows.

[0329] (1) An information processing device, comprising: A rendering unit generates virtual video by rendering a 3D model used in a camera system that captures video on a display screen using a camera, the display showing the virtual video obtained by rendering the 3D model; and The video processing unit performs real-shot video conversion processing on the virtual video generated by the rendering unit. This real-shot video conversion processing is used to generate a simulated video by using processing parameters that realize the brightness or color characteristics when the camera used in the photography system takes pictures.

[0330] (2) The information processing apparatus according to (1), wherein, The video processing unit performs display video processing using processing parameters that realize the brightness or color change characteristics of the video on the display, and then performs camera video processing using processing parameters that realize the brightness or color characteristics when the camera is shooting, in order to generate the simulated video.

[0331] (3) The information processing apparatus according to (1) or (2), wherein, The video processing unit performs color adjustment processing on the simulated video based on control signals based on user operation, and generates color conversion information based on the color adjustment processing.

[0332] (4) The information processing apparatus according to (3), wherein, The video processing unit can output the color conversion information.

[0333] (5) The information processing apparatus according to (4), wherein, The color conversion information is output as information for signal processing of the video captured by the camera.

[0334] (6) The information processing apparatus according to any one of (1) to (5), wherein, The video processing unit can process and change the camera setting information to be reflected in the simulated video according to the control signals based on user operation, and output the changed camera setting information.

[0335] (7) The information processing apparatus according to (6), wherein, The camera setting information is output as information for setting the camera.

[0336] (8) The information processing apparatus according to any one of (1) to (7), wherein, White balance value is included as a processing parameter to achieve the brightness or color characteristics when shooting by the camera used in the photographic system.

[0337] (9) The information processing apparatus according to any one of (1) to (8), wherein, Gain values ​​for pixel signals of each color read from the image sensor in the camera are included as processing parameters to achieve luminance or color characteristics when the camera is used in the photographic system.

[0338] (10) The information processing apparatus according to any one of (1) to (9), wherein, In the real-shot video conversion process, the video processing unit generates the simulated video by using a perspective setting based on the perspective information of the camera used in the photography system when shooting.

[0339] (11) The information processing apparatus according to any one of (1) to (10), wherein, In the real-shot video conversion process, the video processing unit generates the simulated video that reflects the lens characteristics of the camera used in the photography system.

[0340] (12) The information processing apparatus according to any one of (1) to (11), wherein, In the live-shot video conversion process, the video processing unit generates the simulated video that reflects the characteristics of the filter installed on the camera used in the photography system.

[0341] (13) The information processing apparatus according to any one of (1) to (12), wherein, Based on the moiré prediction information determined from the video rendered by the rendering unit, moiré warning output processing is performed.

[0342] (14) The information processing apparatus according to (13), wherein, Prepare a user interface for arbitrarily changing the camera position on the video, and perform moiré prediction based on the camera position changed through the user interface.

[0343] (15) The information processing apparatus according to any one of (1) to (14), wherein, The information processing device is separate from the equipment that constitutes the photographic system.

[0344] (16) An information processing method, comprising: Virtual video is generated by rendering using a 3D model used in a camera system, which captures video on a display screen, showing the virtual video rendered using the 3D model; and The real-shot video is converted into a simulated video by using processing parameters on the generated virtual video. These processing parameters reflect the brightness or color characteristics of the video when it was shot by the camera used in the photography system.

[0345] (17) A program that causes an information processing device to perform the following processing: The process of generating virtual video by rendering using a 3D model employed in a photographic system, which captures video with a camera and displays the virtual video rendered using the 3D model; and The real-shot video conversion process generates a simulated video by applying processing parameters to the generated virtual video. These processing parameters reflect the brightness or color characteristics captured by the camera used in the photography system.

[0346] (18) An information processing system, comprising: A photography system that uses a camera to capture video on a monitor, which displays virtual video rendered using 3D models; and Information processing device, using the 3D model, The information processing device includes: The rendering unit generates virtual video by rendering the 3D model; and The video processing unit performs a real-shot video conversion process on the virtual video generated by the rendering unit. This real-shot video conversion process is used to generate a simulated video by using processing parameters that realize the brightness or color characteristics when the camera used in the photography system takes the picture.

[0347] (19) The information processing system according to (18), wherein, When the video on the display is captured by the camera, the photography system performs moiré warning output processing based on moiré determination information.

[0348] (20) The information processing system according to (18) or (19), wherein, The video processing unit performs color adjustment processing on the simulated video based on control signals based on user operation, and generates and outputs color conversion information based on the color adjustment processing. The photography system uses the color conversion information to perform signal processing on the video captured by the camera.

[0349] (21) The information processing system according to any one of (18) to (20), wherein, The video processing unit processes the camera setting information to be reflected in the simulated video according to the control signals based on user operation, and outputs the changed camera setting information. The photography system configures the camera based on the camera setting information.

Claims

1. An information processing device, comprising: A rendering unit, which generates a virtual video by rendering using a 3D model used in a photographic system, the photographic system uses a camera to shoot a video on a display, and the display displays the virtual video obtained by rendering using the 3D model; as well as A video processing unit performs real-shot video conversion processing on the virtual video generated by the rendering unit, wherein the real-shot video conversion processing is used to generate a simulated video by using processing parameters, and the processing parameters realize the brightness or color characteristics when the camera used in the photography system is used for shooting.

2. The information processing device according to claim 1, wherein: The video processing unit performs display video processing using processing parameters that realize the brightness or color change characteristics of the video on the display, and then performs camera video processing using processing parameters that realize the brightness or color characteristics when photographed by the camera to generate the simulated video.

3. The information processing device according to claim 1, wherein: The video processing unit performs a color adjustment process on the analog video according to a control signal based on a user operation, and generates color conversion information based on the color adjustment process.

4. The information processing device according to claim 3, wherein: The video processing unit is capable of outputting the color conversion information.

5. The information processing device according to claim 4, wherein: The color conversion information is output as information used for signal processing of a video captured by the camera.

6. The information processing device according to claim 1, wherein: The video processing unit is capable of performing a process of changing the camera setting information to be reflected in the analog video according to a control signal based on a user operation, and outputting the changed camera setting information.

7. The information processing device according to claim 6, wherein: The camera setting information is output as information for setting of the camera.

8. The information processing device according to claim 1, wherein: A white balance value is included as a processing parameter for realizing luminance or color characteristics when photographing by the camera used in the photographing system.

9. The information processing device according to claim 1, wherein: A gain value for a pixel signal of each color read from an image sensor in the camera is included as a processing parameter for realizing a luminance or color characteristic when photographing by the camera used in the photography system.

10. The information processing device according to claim 1, wherein: The video processing unit generates the simulated video using the angle of view setting based on the angle of view information when shooting by the camera used in the photography system in the real-shot video conversion process.

11. The information processing device according to claim 1, wherein: The video processing unit generates the simulated video reflecting the lens characteristics of the camera used in the photography system in the real-shot video conversion process.

12. The information processing device according to claim 1, wherein: The video processing unit generates the simulated video reflecting the characteristics of the filter attached to the camera used in the photography system in the real-shot video conversion process.

13. The information processing device according to claim 1, wherein: Moire warning output processing is performed according to moire prediction information determined for the video rendered by the rendering unit.

14. The information processing device according to claim 13, wherein: A user interface for arbitrarily changing the camera position on the video is prepared, and moiré prediction is performed based on the camera position changed by the user interface.

15. The information processing device according to claim 1, wherein: The information processing device is a device separate from the devices constituting the photographing system.

16. An information processing method, comprising: Generate a virtual video by rendering using a 3D model used in a photographic system, the photographic system using a camera to shoot a video on a display, the display showing the virtual video rendered by using the 3D model; as well as The real-shot video conversion process is performed to generate a simulated video by using processing parameters on the generated virtual video, wherein the processing parameters realize the brightness or color characteristics when the camera used in the photography system is used for shooting.

17. A program causing an information processing device to execute the following processing: A process of generating a virtual video by rendering using a 3D model used in a photographic system, the photographic system using a camera to shoot a video on a display, the display showing the virtual video rendered by using the 3D model; and The real-shot video conversion process generates a simulated video by using processing parameters on the generated virtual video, and the processing parameters realize the brightness or color characteristics when the camera used in the photography system takes the picture.

18. An information processing system comprising: A camera system for capturing video on a display using a camera, the display showing a virtual video rendered by using a 3D model; as well as An information processing device, using the 3D model, Wherein, the information processing device includes: A rendering unit, generating a virtual video by rendering using the 3D model; and A video processing unit performs real-shot video conversion processing on the virtual video generated by the rendering unit, wherein the real-shot video conversion processing is used to generate a simulated video by using processing parameters, and the processing parameters realize the brightness or color characteristics when the camera used in the photography system takes pictures.

19. The information processing system according to claim 18, wherein: In the case where the video on the display is captured by the camera, the photography system performs moiré warning output processing according to the moiré determination information.

20. The information processing system according to claim 18, wherein: The video processing unit performs color adjustment processing on the analog video according to a control signal based on a user operation, and generates and outputs color conversion information based on the color adjustment processing, and The photographing system performs signal processing on a video photographed by the camera by using the color conversion information.

21. The information processing system according to claim 18, wherein: The video processing unit performs processing for changing the camera setting information to be reflected in the analog video according to a control signal based on a user operation, and outputs the changed camera setting information, and The photographing system performs settings of the camera based on the camera setting information.

Images