Information processing device, information processing method, and program

Abstract

The information processing device comprises a captured video processing unit. The captured video processing unit performs a determination process of determining, with respect to a captured video, a position of a boundary mark displayed at a peripheral portion in a display area of the display in a target frame of the captured video obtained by imaging the display by a camera, the display displays a video including the boundary marker.

Description

Technical Field

[0001] The present technology relates to an information processing device, an information processing method, and a program, and to a technology that can be used in the field of video production, for example. Background Art

[0002] As an imaging method for producing video contents such as movies, a technique is known in which performers perform in front of a so-called green screen and then a background video is composited.

[0003] Furthermore, in recent years, instead of green screen imaging, the following imaging system has been developed: in a studio equipped with a large monitor, a background video is displayed on the monitor, and performers perform in front of it, whereby the performers and the background can be imaged, and this imaging system is called so-called virtual production, in-camera VFX, or light emitting diode (LED) curtain wall virtual production.

[0004] The following Patent Document 1 discloses a technology of a system that captures an image of a performer or an object performing in front of a background video.

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: U.S. Patent Application Publication No. 2020 / 0145644A Summary of the invention

[0008] Problems to be solved by the present invention

[0009] When a background video is displayed on a large monitor and then the performer and the background video are imaged by a camera, there are many advantages over green screen imaging, so that the background video does not have to be synthesized separately after imaging, and the performers and staff can visually understand the scene and practice the performance or determine whether the performance is good or bad.

[0010] However, the display range of the background video or the like is within the frame of the display and is limited. Depending on the position, imaging direction or zoom angle of the camera, the imaging range may be outside the frame of the display.

[0011] In such a case, in order not to waste imaging, it is necessary to detect the other parts of the protrusion later and correct the video through synthesis processing, etc., but this is extremely troublesome work. In addition, in order to avoid this, the camera operator needs to carefully capture the range of the display, and the framing is limited.

[0012] Therefore, the present disclosure proposes a technology that does not reduce the efficiency of video production even when an image outside the frame of a display is captured.

[0013] Solution to the problem

[0014] According to the present technology, an information processing device includes a captured video processing unit, which performs the following determination processing: determining the position of a boundary mark displayed at a peripheral portion in a display area of ​​a target frame of a captured video obtained by imaging a display by a camera relative to the captured video, and the display displays a video including the boundary mark.

[0015] Furthermore, the information processing device according to the present technology includes a marking processing unit that performs adding a boundary marker at a peripheral portion to be displayed in a display area of ​​a display to an image to be captured by a camera to be displayed on the display.

[0016] For example, a boundary marker displayed at a peripheral portion of the display is added to a background video to be displayed on the display. The boundary marker enables accurate detection of the range of a frame of the display on the captured video. The video processing unit detects the boundary marker relative to the captured video. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is an explanatory diagram of an imaging system of an embodiment of the present technology.

[0018] Figure 2 is an illustrative diagram of a background video of a camera position of an imaging system according to an embodiment.

[0019] Figure 3 is an illustrative diagram of a background video of a camera position of an imaging system according to an embodiment.

[0020] Figure 4 It is an explanatory diagram of the video content production steps of the embodiment.

[0021] Figure 5 is a block diagram of an imaging system of an embodiment.

[0022] Figure 6 is a flow chart of background video generation of the imaging system of an embodiment.

[0023] Figure 7 is a block diagram of an imaging system using multiple cameras of an embodiment.

[0024] Figure 8 It is a block diagram of the information processing device of the embodiment.

[0025] Fig. 9 is an explanatory diagram of a marking process and a captured video process of an embodiment.

[0026] Fig.10 is an explanatory diagram of a background video displayed in the embodiment.

[0027] Fig.11 is an illustrative diagram of boundary marking of an embodiment.

[0028] Fig.12 is an illustrative diagram of a captured video including boundary markers of an embodiment.

[0029] Fig.13 is an explanatory diagram of an extracted marker image of an embodiment.

[0030] Fig.14 is an illustrative diagram of a marker image whose position has been estimated according to an embodiment.

[0031] Fig.15 is an explanatory diagram of a mask of the embodiment.

[0032] Fig.16 is an illustrative diagram of the final output video of the embodiment.

[0033] Fig.17 is a flow diagram of an embodiment including a markup process rendering.

[0034] Fig.18 is a flow chart of the capture video process of an embodiment.

[0035] Fig.19 1 is an explanatory diagram of calculation of the peripheral light amount drop amount and the distortion amount of the embodiment.

[0036] Fig. 20 1 is an explanatory diagram for comparing the calculation of the peripheral light amount drop amount and the distortion amount of the embodiment.

[0037] Fig.21 is an explanatory diagram of mask generation of the embodiment.

[0038] Fig. 22 is an explanatory diagram of mixing of masks of the embodiment.

[0039] Fig.23 is an explanatory diagram of the synthesis process of the embodiment.

[0040] Fig.24 is an illustrative diagram of a pre-visualization of an embodiment.

[0041] Fig.25 is a block diagram of a configuration example of a pre-visualization of an embodiment.

[0042] Fig.26 is a flowchart of an example of processing in pre-visualization of an embodiment. DETAILED DESCRIPTION

[0043] Hereinafter, the embodiments will be described in the following order.

[0044] <1. Imaging system and content production>

[0045] <2. Configuration of information processing device>

[0046] <3. Marker Processing and Capture Video Processing>

[0047] <4. Processing example>

[0048] <5. Processing in Pre-Visualization>

[0049] <6. Summary and Modification Examples>

[0050] Note that in the present disclosure, "video" or "image" includes both still images and moving images. In addition, "video" refers not only to the state displayed on the display, but also to video data that is not displayed on the display, which can be collectively referred to as "video". The same applies to "image".

[0051] For example, in an embodiment, the background video before being displayed on the display, the video captured by the camera, and the background video or captured video switched by the switch are not the actual displayed videos but video data, but for convenience, are referred to as "background video", "captured video", etc.

[0052] <1. Imaging system and video content production>

[0053] An imaging system and production of video content to which the techniques of this disclosure may be applied will be described.

[0054] Figure 1 An imaging system 500 is schematically illustrated. The imaging system 500 is a system that performs imaging as virtual production, and some of the devices arranged in an imaging studio are illustrated in the figure.

[0055] In the imaging studio, a performance area 501 is provided for a performer 510 to perform a performance such as a play. A large display device is arranged on at least the back surface, left and right side surfaces, and the upper surface of the performance area 501. Although the device type of the display device is not limited, the figure illustrates an example in which an LED curtain wall 505 is used as an example of a large display device.

[0056] One LED curtain wall 505 is formed by vertically and horizontally connecting and arranging a plurality of LED panels 506 to form a large panel. The size of the LED curtain wall 505 is not particularly limited, but only needs to be a size that is necessary or sufficient for displaying a background when imaging the performer 510.

[0057] A necessary number of lamps 580 are arranged at necessary positions such as above or at the side of the performance area 501 to illuminate the performance area 501 .

[0058] Near the performance area 501, for example, a camera 502 for imaging a movie or other video content is set. A camera operator 512 can move the position of the camera 502 and can perform operations of the imaging direction, viewing angle, etc. Of course, it is also conceivable that the movement of the camera 502, the viewing angle operation, etc. are performed by remote control. In addition, the camera 502 can automatically or autonomously move or change the viewing angle. For this purpose, the camera 502 can be installed on a camera platform or a mobile body.

[0059] Camera 502 captures performers 510 in performance area 501 and video displayed on LED curtain wall 505 together. For example, by displaying a scene as background video vB on LED curtain wall 505, it is possible to capture a video similar to the video in the case where performers 510 actually exist and perform at the location of the scene.

[0060] An output monitor 503 is provided near the performance area 501. The video captured by the camera 502 is displayed in real time as a monitor video vM on the output monitor 503. Therefore, the director and the staff who produce the video content can confirm the captured video.

[0061] As described above, the imaging system 500 for imaging a performance of a performer 510 against a background of an LED wall 505 in an imaging studio has various advantages over green screen imaging.

[0062] For example, in the case of green screen imaging, it is difficult for the performer to imagine the background and situation of the scene, which may affect the performance. However, by displaying the background video vB, the performer 510 can easily perform the performance, and the quality of the performance is improved. In addition, the director and other staff can easily determine whether the performance of the performer 510 matches the background and situation of the scene.

[0063] Furthermore, post-production after imaging is more efficient than post-production in the case of green screen imaging. This is because so-called chroma key compositing may not be necessary, or color correction or projection compositing may not be necessary. Furthermore, even in the case where chroma key compositing is required at the time of imaging, it is only necessary to display green or blue video, so not having to add a physical background screen also contributes to efficiency.

[0064] In the case of green screen imaging, a green tint is added to the performer's body, costumes, and objects, so it is necessary to correct it. In addition, in the case of green screen imaging, if there is an object such as glass, mirror, or snow top that is reflected in the surrounding scene, the reflected image must be generated and synthesized, but this is troublesome work.

[0065] However, in Figure 1In the case of imaging by the imaging system 500 in , the green hue is not increased, so correction is unnecessary. In addition, by displaying the background video vB, the reflection on the actual product such as glass is naturally obtained and captured, so there is no need to synthesize the reflection video.

[0066] Here, we will refer to Figure 2 and Figure 3 To describe the background video vB. Even if the background video vB is displayed on the LED curtain wall 505 and imaged together with the performer 510, the background of the imaged video is unnatural by simply displaying the background video vB. This is because a three-dimensional background with depth is actually used as the background video vB in a planar manner.

[0067] For example, the camera 502 can image the performer 510 in the performance area 501 from various directions, and can also perform a zoom operation. The performer 510 does not stay in one place. Therefore, the actual appearance of the background of the performer 510 should change according to the position, imaging direction, viewing angle, etc. of the camera 502, but such a change cannot be obtained in the background video vB which is a flat video. Therefore, the background video vB is changed so that the background is similar to the actual appearance including parallax.

[0068] Figure 2 The figure shows a state where the camera 502 is imaging the performer 510 from the position on the left side of the figure, and Figure 3 The figure shows a state where the camera 502 is imaging the performer 510 from the right side of the figure. In each figure, the capture area video vBC is shown in the background video vB.

[0069] Note that a portion of the background video vB other than the capture area video vBC is referred to as an “outer frustum”, and the capture area video vBC is referred to as an “inner frustum”.

[0070] The background video vB described here refers to the entire video displayed as the background including the capture area video vBC (inner frustum).

[0071] The range of the captured area video vBC (inner frustum) corresponds to the range actually imaged by the camera 502 in the display surface of the LED curtain wall 505. Thus, the captured area video vBC is a video representing the scene actually seen when the position of the camera 502 is set as the viewpoint according to the position of the camera 502, the imaging direction, the viewing angle, etc.

[0072] Specifically, a 3D (three-dimensional) model as a background, that is, 3D background data is prepared, and the capture area video vBC is sequentially rendered in real time based on the viewpoint position of the camera 502 relative to the 3D background data.

[0073] Note that the range of the capture area video vBC is actually a slightly wider range than the range imaged by the camera 502 at that point in time. This is to prevent the video of the outer frustum from being reflected due to drawing delay when the imaging range is slightly changed by panning, tilting, zooming, etc. of the camera 502, and to avoid the influence of diffracted light from the video of the outer frustum.

[0074] The video of the capture area video vBC rendered in real time in this way is synthesized with the video of the outer frustum. The video of the outer frustum used in the background video vB can be pre-rendered based on the 3D background data, or can be rendered in real time for each frame or each intermittent frame, and the video of the capture area video vBC (inner frustum) is incorporated into part of the video of the outer frustum to generate the entire background video vB.

[0075] Note that there is a case where the video of the outer frustum is also rendered for each frame similar to the inner frustum, but here, a static video is taken as an example, and in the following description, the case of rendering only the head frame of the video of the outer frustum is mainly described as an example.

[0076] Therefore, even when the camera 502 moves forward and backward or left and right or performs a zoom operation, the background of the range imaged together with the performer 510 is imaged as a video based on the changes in the viewpoint position and field of view (FOV) accompanying the actual movement of the camera 502.

[0077] like Figure 2 and Figure 3 As shown in FIG. 5 , a monitor video vM including a performer 510 and a background is displayed on an output monitor 503, and this is a capture video. The background of the monitor video vM is the capture area video vBC. That is, the background included in the imaging video is a real-time rendering video.

[0078] As described above, in the imaging system 500 of the embodiment, the background video vB including the capture area video vBC is changed in real time, so that not only the background video vB is simply displayed in a flat manner, but also a video similar to the video in the case of actual scenery imaging can be imaged.

[0079] Note that by rendering only the capture area video vBC as the range projected by the camera 502 in real time, rather than the entire background video vB displayed on the LED curtain wall 505, the device can reduce the processing load of the system.

[0080] Here, a video content production step as virtual production in which imaging is performed by the imaging system 500 will be described. Figure 4As shown in the figure, the video content production steps are roughly divided into three stages. These stages are asset creation ST1, production ST2 and post-production ST3.

[0081] Asset creation ST1 is a step of making 3D background data for displaying background video vB. As described above, background video vB is generated by performing rendering in real time using 3D background data at the time of imaging. For this purpose, 3D background data as a 3D model is made in advance.

[0082] Examples of methods of producing 3D background data include full computer graphics (CG), point cloud data scanning, and photogrammetry.

[0083] Full CG is a method of making a 3D model with computer graphics. Among the three methods, this method requires the most man-hours and time, but is preferably used when it is desired to use unrealistic videos, videos that are difficult to capture in practice, etc. as background videos vB.

[0084] Point cloud data scanning is a method of generating a 3D model based on point cloud data by performing distance measurement from a certain position using, for example, LiDAR, capturing a 360-degree image from the same position with a camera, and laying out the color data captured by the camera at the points measured by LiDAR. Compared with full CG, a 3D model can be produced in a short time. In addition, it is easy to produce a 3D model with higher definition than that obtained by photogrammetry.

[0085] Photogrammetry is a photogrammetry technique for analyzing parallax information from two-dimensional images obtained by imaging an object from multiple viewpoints to obtain size and shape. 3D model production can be performed in a short time.

[0086] Note that the point cloud information acquired by LiDAR can be used for 3D data generation through photogrammetry.

[0087] In asset creation ST1, for example, a 3D model to be 3D background data is produced by using these methods. Of course, the above methods can be used in combination. For example, part of a 3D model produced by point cloud data scanning or photogrammetry is produced by CG and synthesized.

[0088] ST2 is made in Figure 1 The steps of imaging are performed in the imaging studio shown in Figure 1. In this case, the essential technologies include real-time rendering, background display, camera tracking, lighting control, etc.

[0089] Real-time rendering is as shown in the reference Figure 2 and Figure 3The rendering process for obtaining the capture area video vBC at each time point (each frame of the background video vB) is described. This is to render the 3D background data produced in the asset creation ST1 from a viewpoint according to the position of the camera 502 at each time point, etc.

[0090] In this manner, real-time rendering is performed to generate a background video vB including each frame of the capture area video vBC, and the background video vB is displayed on the LED curtain wall 505 .

[0091] Camera tracking is performed to obtain imaging information of the camera 502, and the position information, imaging direction, viewing angle, etc. of the camera 502 at each time point are tracked. By providing the imaging information including these to the rendering engine in association with each frame, real-time rendering can be performed according to the viewpoint position of the camera 502, etc.

[0092] Imaging information is information linked or associated with the video as metadata.

[0093] It is assumed that the imaging information includes the position information of the camera 502 at each frame timing, the direction of the camera, the viewing angle, the focal length, the F value (aperture value), the shutter speed, the lens information, etc.

[0094] The illumination control is used to control the illumination state in the imaging system 500, and specifically, to control the light amount, emission color, illumination direction, etc. of the lamp 580. For example, the illumination control is performed according to the time setting, location setting, etc. of the scene to be imaged.

[0095] Post-production ST3 indicates various types of processing performed after imaging, such as video correction, video adjustment, clip editing, video effects, etc.

[0096] As video correction, color gamut conversion, color matching between camera and material, etc. can be performed.

[0097] As video adjustment, color adjustment, brightness adjustment, contrast adjustment, etc. can be performed.

[0098] As clip editing, cutting of clips, adjustment of order, adjustment of time length, etc. can be performed.

[0099] As a video effect, synthesis of CG video, special effect video, etc. can be performed.

[0100] Next, the configuration of the imaging system 500 used in producing ST2 will be described.

[0101] Figure 5 The diagram has been referenced Figure 1 , Figure 2 and Figure 3 A block diagram of the configuration of the imaging system 500 is described, an overview of which is shown.

[0102] Figure 5 The imaging system 500 shown in FIG. 5 includes the above-mentioned LED curtain wall 505 including a plurality of LED panels 506, a camera 502, an output monitor 503, and a light 580. Figure 5 As illustrated in , the imaging system 500 also includes a rendering engine 520 , an asset server 530 , a synchronization generator 540 , an operation monitor 550 , a camera tracker 560 , an LED processor 570 , a lighting controller 581 , and a display controller 590 .

[0103] Each of the LED processors 570 is provided corresponding to one or more LED panels 506 , and performs video display driving of the corresponding one or more LED panels 506 .

[0104] The synchronization generator 540 generates a synchronization signal for synchronizing the frame timing of the display video of the LED panel 506 with the frame timing of the imaging of the camera 502 , and supplies the synchronization signal to each LED processor 570 , the camera 502 , and the rendering engine 520 .

[0105] The camera tracker 560 generates imaging information of the camera 502 at each frame timing, and supplies the imaging information to the rendering engine 520. For example, the camera tracker 560 detects position information of the position of the camera 502 relative to the LED curtain wall 505 or a predetermined reference position and imaging direction of the camera 502 as one of the multiple imaging information, and supplies them to the rendering engine 520.

[0106] As a specific detection method of the camera tracker 560, there is a method of randomly arranging reflectors on the ceiling and detecting the position based on the reflected light of the reflectors with respect to the infrared light emitted from the camera tracker 560 assembled to the camera 502. In addition, as a detection method, there is also a method of estimating the own position of the camera 502 through information of a gyroscope mounted on the platform of the camera 502 or the body of the camera 502 or image recognition of a captured video of the camera 502.

[0107] In addition, the angle of view, focal length, F number, shutter speed, lens information, etc. can be supplied from the camera 502 to the rendering engine 520 as imaging information.

[0108] The asset server 530 is a server that can store the 3D model (ie, 3D background data) made in the asset creation ST1 on a recording medium and read the 3D model when necessary. That is, the asset server 530 serves as a database (DB) of 3D background data.

[0109] The rendering engine 520 performs a process of generating a background video vB to be displayed on the LED curtain wall 505. For this purpose, the rendering engine 520 reads necessary 3D background data from the asset server 530. Then, the rendering engine 520 generates a video of an outer frustum used in the background video vB as a video obtained by rendering the 3D background data in a form observed from pre-specified space coordinates.

[0110] Furthermore, the rendering engine 520 specifies the viewpoint position and the like relative to the 3D background data using the imaging information supplied from the camera tracker 560 or the camera 502 , and renders the capture area video vBC (inner frustum).

[0111] In addition, the rendering engine 520 synthesizes the capture area video vBC that dynamically changes according to the movement of the camera 502 relative to the outer frustum to generate the background video vB as video data of one frame. Then, the rendering engine 520 transmits the generated video data of one frame to the display controller 590.

[0112] The display controller 590 generates a divided video signal nD obtained by dividing the video data of one frame into video parts to be displayed on the corresponding LED panel 506, and transmits the divided video signal nD to the corresponding LED panel 506. At this time, the display controller 590 may perform calibration according to individual differences in color rendering, etc. between display units, manufacturing errors, etc.

[0113] Note that the display controller 590 may not be provided, and the rendering engine 520 may perform these types of processing. That is, the rendering engine 520 may generate the divided video signal nD, perform calibration, and transmit the divided video signal nD to the corresponding LED panel 506.

[0114] The LED processor 570 drives the corresponding LED panel 506 based on the received corresponding divided video signal nD, and the entire background video vB is displayed on the LED curtain wall 505. The background video vB includes the capture area video vBC rendered according to the position of the camera 502 at that point in time, etc.

[0115] In this way, the camera 502 can image the performance of the performer 510 (including the background video vB displayed on the LED curtain wall 505). The video obtained by imaging by the camera 502 is recorded on a recording medium in the camera 502 or an external recording device (not shown), and is supplied to the output monitor 503 in real time and displayed as a monitor video vM.

[0116] The operation monitor 550 displays an operation image vOP for controlling the rendering engine 520. The engineer 511 can perform necessary settings and operations regarding rendering of the background video vB while viewing the operation image vOP.

[0117] The lighting controller 581 controls the emission intensity, emission color, irradiation direction, etc. of the lamp 580. For example, the lighting controller 581 may control the lamp 580 asynchronously with the rendering engine 520, or may perform control in synchronization with the imaging information and rendering process. For this purpose, the lighting controller 581 may perform light emission control according to instructions from the rendering engine 520, a main controller (not shown), etc. In addition, the rendering engine 520 may control the lamp 580.

[0118] Figure 6 An example of processing of the rendering engine 520 in the imaging system 500 having such a configuration is illustrated.

[0119] In step S10 , the rendering engine 520 reads the 3D background data to be used this time from the asset server 530 , and deploys the 3D background data to the internal work area.

[0120] At this stage, a video can be generated that serves as the outer frustum.

[0121] Thereafter, the rendering engine 520 repeats the process of steps S30 to S60 until it is determined in step S20 that the display of the background video vB based on the read 3D background data is finished.

[0122] In step S30, the rendering engine 520 acquires imaging information from the camera tracker 560 or the camera 502. Thus, the position and state of the camera 502 to be reflected in the current frame are confirmed.

[0123] In step S40, the rendering engine 520 performs rendering based on the imaging information. That is, the viewpoint position relative to the 3D background data is specified based on the position of the camera 502 to be reflected in the current frame, the imaging direction, the viewing angle, etc., and rendering is performed. At this time, video processing reflecting the focal length, F number, shutter speed, lens information, etc. may also be performed. Through this rendering, video data as a capture area video vBC (inner frustum) may be obtained. The outer frustum is generated in advance as a fixed video in step S10, and may be generated for each frame in step S40.

[0124] In step S50, the rendering engine 520 performs a process of synthesizing the outer frustum as the entire background video with the video reflecting the viewpoint position of the camera 502 (i.e., the capture area video vBC). For example, the process is used to synthesize the video generated by reflecting the viewpoint of the camera 502 with the video of the entire background rendered at a specific reference viewpoint. Thus, a background video vB of one frame displayed on the LED curtain wall 505 is generated, that is, the background video vB including the capture area video vBC.

[0125] The processing in step S60 is performed by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates divided video signals nD obtained by dividing the background video vB of one frame into videos to be displayed on the respective LED panels 506. Calibration may be performed. Then, the divided video signals nD are transmitted to the LED processors 570, respectively.

[0126] Through the above processing, the background video vB including the capture area video vBC imaged by the camera 502 is displayed on the LED curtain wall 505 at each frame timing.

[0127] By the way, in Figure 5 Only one camera 502 is illustrated in FIG. 5 , but imaging may be performed by a plurality of cameras 502 . Figure 7 The configuration example in the case of using a plurality of cameras 502a and 502b is illustrated. The cameras 502a and 502b are enabled to independently perform imaging in the performance area 501. In addition, synchronization between the cameras 502a and 502b and the LED processor 570 is maintained by the synchronization generator 540.

[0128] The output monitors 503a and 503b are provided corresponding to the cameras 502a and 502b, respectively, and are configured to display the videos imaged by the corresponding cameras 502a and 502b as monitor videos vMa and vMb, respectively.

[0129] In addition, camera trackers 560a and 560b are respectively set corresponding to cameras 502a and 502b, and respectively detect the positions and imaging directions of corresponding cameras 502a and 502b. Imaging information from camera 502a and camera tracker 560a and imaging information from camera 502b and camera tracker 560b are transmitted to rendering engine 520.

[0130] The rendering engine 520 may perform rendering by using imaging information on the camera 502 a side or the camera 502 b side to obtain a background video vB for each frame.

[0131] Note that although Figure 7An example using two cameras 502 a and 502 b is illustrated, but imaging may also be performed by using three or more cameras 502 .

[0132] However, when a capture area video vBC (inner frustum) corresponding to each camera 502 is rendered and displayed using each piece of imaging information using a plurality of cameras 502, there is a case where the corresponding capture area videos vBC interfere with each other. Figure 7 In the example of using two cameras 502a and 502b illustrated in FIG, the capture area video vBC corresponding to the camera 502a is illustrated, but in the case of using the video of the camera 502b, the capture area video vBC corresponding to the camera 502b is also necessary. In this case, when the respective capture area videos vBC corresponding to the cameras 502a and 502b are simply displayed, they interfere with each other. For this reason, it is necessary to somehow achieve the display of the capture area video vBC.

[0133] <2. Configuration of information processing device>

[0134] Next, we will refer to Figure 8 A configuration example of the information processing apparatus 70 that can be used in asset creation ST1, production ST2, and post-production ST3 is described.

[0135] The information processing device 70 is a device such as a computer device that can perform information processing, especially video processing. Specifically, a personal computer, a workstation, a portable terminal device such as a smartphone or a tablet, a video editing device, etc. are assumed as the information processing device 70. In addition, the information processing device 70 may be a computer device configured as a server device or a computing device in cloud computing.

[0136] In the case of the present embodiment, specifically, the information processing apparatus 70 can be used as a 3D model making apparatus that makes a 3D model in the asset creation ST1.

[0137] Furthermore, the information processing apparatus 70 can also function as the rendering engine 520 and the asset server 530 constituting the imaging system 500 used in producing ST2.

[0138] Furthermore, the information processing device 70 can also be used as a video editing device that performs various types of video processing in post-production ST3.

[0139] In addition, the information processing device 70 can also be used as an information processing device (rendering engine 31) for generating a simulation video vSM in pre-visualization, as will be described later. Fig.25 etc. described.

[0140] Figure 8The CPU 71 of the information processing device 70 illustrated in FIG. 1 performs various types of processing according to a program stored in a ROM 72 or a nonvolatile storage unit 74 such as, for example, an electrically erasable programmable read-only memory (EEP-ROM) or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to perform various types of processing.

[0141] The video processing unit 85 is configured as a processor that performs various types of video processing. For example, the processor is a processor that can perform any one or more types of processing in 3D model generation processing, rendering, DB processing, video processing including color / brightness adjustment processing, video editing processing, video analysis / detection processing, etc.

[0142] Specifically, Figure 8 A captured video processing unit 85 a and a marking processing unit 85 b are illustrated as processing functions in the video processing unit 85 .

[0143] For example, there is an example in which the captured video processing unit 85 a and the marking processing unit 85 b are provided in the information processing device 70 as the rendering engine 520 .

[0144] The labeling processing unit 85b is a unit that performs the boundary labeling 60 (see Fig. 9 The function of adding the processing of the rendered background video vB to the background video vB is thus added. Thus, the background video vB displayed on the LED curtain wall 505 includes the image of the boundary marker 60.

[0145] The captured video processing unit 85a is a function of performing a determination process for determining the position of the boundary marker 60 in the target frame of the captured video vC obtained by imaging the background video vB including the boundary marker 60 displayed on the LED curtain wall 505. The target frame refers to a frame that is a processing target of the captured video process at a certain point in time. For example, each frame constituting the captured video vC as a moving image is sequentially set as a target frame.

[0146] Note that the position of the captured video vC in the target frame (eg, the position of the boundary marker 60) is the position of that frame in the image and does not include the positions of temporally different frames.

[0147] Details of the processing performed by the video processing unit 85 through the functions of the capture video processing unit 85a and the marking processing unit 85b will be described later.

[0148] Note that as the video processing unit 85 in the information processing device 70, an example including the captured video processing unit 85a but not including the labeling processing unit 85b and an example including the labeling processing unit 85b but not including the captured video processing unit 85a are also assumed.

[0149] The video processing unit 85 may be implemented by, for example, a CPU separate from the CPU 71, a graphics processing unit (GPU), general purpose computing on a graphics processing unit (GPGPU), an artificial intelligence (AI) processor, or the like.

[0150] Note that the video processing unit 85 may be provided as a function in the CPU 71 .

[0151] The CPU 71, the ROM 72, the RAM 73, the nonvolatile storage unit 74, and the video processing unit 85 are connected to one another via a bus 83. Furthermore, an input / output interface 75 is also connected to the bus 83.

[0152] An input unit 76 including operating elements and operating devices is connected to the input / output interface 75. As the input unit 76, for example, various operators and operating devices such as a keyboard, mouse, key, trackball, dial, touch panel, touch pad, and remote controller are assumed.

[0153] A user operation is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .

[0154] A microphone is also assumed as the input unit 76. A voice uttered by a user may also be input as the operation information.

[0155] Furthermore, a display unit 77 including a liquid crystal display (LCD), an organic electroluminescence (EL) panel, or the like, and an audio output unit 78 including a speaker or the like are connected to the input / output interface 75 integrally or separately.

[0156] The display unit 77 is a display unit that performs various displays, and includes, for example, a display device provided in the casing of the information processing apparatus 70 , a separate display device connected to the information processing apparatus 70 , and the like.

[0157] The display unit 77 performs display of various images, operation menus, icons, messages, and the like on a display screen (ie, display as a graphical user interface (GUI)) based on instructions from the CPU 71 .

[0158] In some cases, a storage unit 79 including a hard disk drive (HDD), a solid-state memory, or the like, or a communication unit 80 is connected to the input / output interface 75 .

[0159] The storage unit 79 can store various data and programs. A DB can also be configured in the storage unit 79.

[0160] For example, in the case where the information processing apparatus 70 functions as the asset server 530 or the rendering engine 31 to be described later, the storage unit 79 may be used to construct a DB storing a 3D background data group.

[0161] The communication unit 80 performs communication processing via a transmission line such as the Internet, wired / wireless communication with various devices such as an external DB, an editing apparatus, and an information processing apparatus, communication by bus communication, and the like.

[0162] For example, in the case where the information processing device 70 functions as the rendering engine 520 or the rendering engine 31 to be described later, the communication unit 80 may access a DB as the asset server 530 , or receive imaging information from the camera 502 or the camera tracker 560 .

[0163] Furthermore, also in the case of the information processing device 70 used in the post-production ST3 , the communication unit 80 can access the DB as the asset server 530 .

[0164] A drive 81 is also connected to the input / output interface 75 as necessary, and a removable recording medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted.

[0165] The drive 81 can read video data, various computer programs, and the like from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio included in the data are output by the display unit 77 and the audio output unit 78. In addition, the computer program and the like read from the removable recording medium 82 are installed in the storage unit 79 as necessary.

[0166] In the information processing apparatus 70, for example, software for the processing in the present embodiment can be installed via network communication through the communication unit 80 or the removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0167] <3. Marker Processing and Capture Video Processing>

[0168] A labeling process and a captured video process according to the embodiment will be described.

[0169] Fig. 9 The marking process and the captured video process in the virtual production process are illustrated.

[0170] For example, by Figure 8 The function of the marking processing unit 85b in the video processing unit 85 is to perform Fig. 9 The tag generation process P2 and the tag adding process P3 in .

[0171] The marker generation process P2 and the marker addition process P3 are executed as processes in the rendering engine 520 when imaging is performed in virtual production (ie, when the background video vB is displayed on the LED curtain wall 505), for example.

[0172] In addition, through Figure 8 The captured video processing (P11 to P19) is performed by using the function of the captured video processing unit 85a in the video processing unit 85.

[0173] The capture video processing (P11 to P19) may be performed in real time at the time of imaging (eg, in the rendering engine 520 to which the capture video vC is input or in another device), or may be performed at the stage of post-production ST3.

[0174] Here, as an example, a description is given on the assumption that the marker generation process and the captured video process are performed in real time by the rendering engine 520 at the time of imaging.

[0175] For example, as Fig. 9 In the rendering P1, the video processing unit 85 of the information processing device 70 as the rendering engine 520 generates a background video vB.

[0176] Fig.10 FIG. A illustrates an example of a background video vB. Note that the rendering engine 520 generates the background video vB, such as Fig.10 , but this example is a video with a wider range than the display area of ​​the LED curtain wall 505. For example, the range of the actual display area 100 indicated by the dotted line is the video actually displayed on the LED curtain wall 505. The rendering engine 520 can generate a background video vB within a range equivalent to the display range of the LED curtain wall 505, but by Fig.10 A diagrammatically illustrates the generation of a background video vB within a wider range, and the background video vB can be used for synthesis in subsequent captured video processing.

[0177] Furthermore, the background video vB also includes the capture area video vBC (inner frustum).

[0178] The video processing unit 85 of the rendering engine 520 performs the function of the marking processing unit 85b. Fig. 9 Mark generation processing P2 shown in the figure.

[0179] In the marker generation process P2, the rendered background video vB and the information of the actual display area 100 are input to generate a marker image vMK.

[0180] Fig.11 A and Fig.11 FIG. 1B illustrates an example of a marked image vMK.

[0181] Fig.11 The example of the marker image vMK in A is an example in which a pattern of the boundary marker 60 surrounds the peripheral portion.

[0182] Fig.11 The example of the marker image vMK in B is an example in which a pattern as the boundary marker 60 is arranged at the left, right, and upper peripheral portions.

[0183] The boundary marker 60 enables detection of the imaging range of the camera 502 that images the LED curtain wall 505 protruding from the LED curtain wall 505. Fig.11 As shown in FIG. 1A , it is preferred that the boundary mark 60 is formed on the entire circumference of the peripheral portion. Figure 1 In the performance area 501 of the imaging system 500 shown in FIG. 1 , a set of real scenes related to the content of the background video vB can be provided, and in this case, even if the imaging range of the camera 502 protrudes below the LED curtain wall 505, there is no problem. In this case, the LED curtain wall 505 can be used as shown in FIG. Fig.11 The boundary marker 60 is illustrated in B.

[0184] like Fig.11 A and Fig.11 The boundary marker 60 illustrated in FIG. 1B includes a boundary line 61 and a detection pattern 62 .

[0185] The boundary line 61 is a line formed in, for example, a frame shape, for clearly indicating the edge of the LED curtain wall 505 .

[0186] The detection pattern 62 is a vertical and horizontal grid-like pattern provided for determining the amount of peripheral light drop and the amount of spatial distortion caused by the lens of the camera 502. Note that the pattern shape is an example, and the pattern shape is not limited thereto.

[0187] The pattern shape of such boundary mark 60 only needs to be maintained in the system in advance.

[0188] Then, in Fig. 9 In the marker generation process P2 illustrated in FIG. 5 , the rendering engine 520 sets the brightness, color, and size of the boundary marker 60 to generate a marker image vMK.

[0189] The brightness and color of the boundary marker 60 are set to be conspicuous in the background video vB with reference to the brightness and color of the background video vB. For example, if the background video vB indicates a bright scene, the brightness of the boundary marker 60 is reduced, and if the background video vB indicates a dark scene, the brightness of the boundary marker 60 is increased. In addition, the color is also set according to the color of the background video vB.

[0190] Furthermore, the rendering engine 520 sets the size of the boundary marker 60 so that the boundary marker 60 is displayed at the peripheral portion within the range of the actual display area 100. Note that the line of the actual display area 100 is added for description, and the background video vB does not actually include a line indicating such an actual display area 100.

[0191] Next, the video processing unit 85 of the rendering engine 520 performs a mark adding process P3 through the function of the mark processing unit 85b.

[0192] As the marker adding process P3 , the rendering engine 520 performs a process of synthesizing the marker image vMK with the rendered background video vB and adding the boundary marker 60 .

[0193] That is, the pixel value at the position corresponding to the boundary marker 60 in the original background video vB is replaced with the pixel value of the boundary marker 60 .

[0194] Fig.10 FIG. B illustrates a background video vB to which a boundary marker 60 is added. The boundary marker 60 is added to the background video vB. Fig.10 The position of the peripheral portion within the range of the background video vB of A is used as the actual display area 100.

[0195] The background video vB with such a boundary mark 60 added is displayed on the LED curtain wall 505. Fig. 9 As shown in FIG. 5 , boundary markers 60 are displayed on a peripheral portion of the LED curtain wall 505 .

[0196] The background video vB of such LED curtain wall 505 and performer 510 are captured by camera 502. Note that the background video vB captured by camera 502 is the range of the capture area video vBC. Fig. 9 , the capture area video vBC is within the LED curtain wall 505, and the imaging range of the camera 502 does not protrude from the LED curtain wall 505. In addition, in the example of the figure, the capture area video vBC does not overlap with the boundary marker 60.

[0197] Since the capture area video vBC substantially matches the imaging range of the camera 502 , when the imaging range of the camera 502 protrudes from the LED curtain wall 505 , part of the capture area video vBC protrudes from the LED curtain wall 505 .

[0198] Next, captured video processing on the captured video vC of the camera 502 will be described.

[0199] For example, it is assumed that the rendering engine 520 receives an input of a captured video vC of the camera 502, performs a captured video process on each frame of the captured video vC, and outputs a final output video vCF as a final video obtained by imaging. That is, the captured video process is sequentially performed with each frame of the captured video vC as a target frame.

[0200] Specifically, by capturing the functions of the video processing unit 85a, the rendering engine 520 performs Fig. 9 The mark extraction processing P11, mark position estimation processing P12, peripheral light drop amount / distortion detection processing P13, mask generation processing P14, peripheral light drop amount / distortion addition processing P15, peripheral light drop amount / distortion removal processing P16, selection processing P17 and P18 and synthesis processing P19.

[0201] Note that the processing of the rendering engine 520 is an example, and the following captured video processing may be performed as processing of another device or processor inside the camera 502 .

[0202] like Fig. 9 As shown in the figure, the captured video vC is output by the camera 502 (output process P10). The camera 502 simultaneously captures the captured area video vBC in the background video vB displayed on the LED curtain wall 505 and the actual scene of the performer 510, etc. At this time, depending on the position, imaging direction and viewing angle of the camera 502, the outside of the LED curtain wall 505 may be included in the captured video vC. In this case, the scene outside the frame of the LED curtain wall 505 and the boundary marker 60 displayed at the peripheral portion of the LED curtain wall 505 are also imaged together.

[0203] Alternatively, even if the imaging range of the camera 502 does not reach outside the frame of the LED curtain wall 505 , imaging may be performed including a portion of the boundary marker 60 .

[0204] Fig.12 4 shows an example of capturing video vC. This is a case where the imaging range of the camera 502 protrudes toward the upper side, left side, and right side of the LED curtain wall 505. For example, in a case where the camera 502 has a wide angle exceeding the width of the LED curtain wall 505 and the imaging direction is too upward, a video such as Fig.12 The captured video vC is shown in FIG.

[0205] In addition to the capture area video vBC and the performer 510 in the background video vB, the capture video vC also includes the boundary marker 60 and the scene around the LED curtain wall 505 (black in the figure).

[0206] For example, the rendering engine 520 performs the following operations on the target frame of the captured video vC: Fig. 9Mark extraction process in P11.

[0207] For marker extraction, the rendering engine 520 refers to the information (marker generation information Imk) at the time of the marker generation process P2. That is, it acquires the brightness and color information of the boundary marker 60. The marker generation information Imk is information such as the brightness, color, pattern shape and size of the boundary marker 60 set in the marker generation process P2.

[0208] The rendering engine 520 sets the threshold levels of brightness and color based on the marker generation information Imk, compares the pixel values ​​of the captured video vC, and extracts the boundary marker 60 from the captured video vC. Fig.12 The captured video vC in is generated as Fig.13 The marker extracted image vMKa is shown in FIG.

[0209] In addition, the rendering engine 520 performs Fig. 9 The marker position estimation process P12 is shown in FIG. This is a process of estimating at which position of the target frame in the captured video vC the boundary marker 60 is displayed in the calculation.

[0210] If the relative position relationship between the LED curtain wall 505 and the camera 502 for the target frame of the current captured video vC, the viewing angle, and the size of the image sensor are known, the position of the boundary marker 60 in the captured video vC can be calculated.

[0211] The information of the arrangement position and angle of the LED curtain wall 505 is known in advance in the system. For example, the rendering engine 520 can be stored as information of the LED curtain wall 505. The size of the image sensor of the camera 502 is also known and set in advance for calculation. The position, imaging direction and zoom angle of the camera 502 are obtained as imaging information from the camera tracker 560 (see Figure 5 )enter.

[0212] Therefore, the rendering engine 520 can estimate at which position in the captured video vC the boundary marker 60 is displayed by performing real-time calculation by acquiring imaging information from the camera tracker 560 for each frame of the captured video vC.

[0213] Then, for example, the rendering engine 520 generates an indication such as Fig.14 The estimated marker position estimated image vMKb of the boundary marker 60 is illustrated in FIG.

[0214] here, Fig.14 The position of the boundary marker 60 in the marker position estimation image vMKb is an ideal position without the influence of lens distortion. On the other hand, Fig.13The position of the boundary marker 60 in the marker extraction image vMKa is a position affected by the lens distortion of the camera 502 .

[0215] By using the marker extraction image vMKa and the marker position estimation image vMKb, the amount of distortion and the amount of peripheral light drop due to the lens can be detected.

[0216] The rendering engine 520 executes the peripheral light drop amount / distortion amount detection process P13.

[0217] In this process, the rendering engine 520 calculates the peripheral light drop amount and the distortion amount using the marker extraction image vMKa and the marker position estimation image vMKb. As the calculation process, for example, as the peripheral light drop amount, a plurality of images in which the peripheral light drop amount is added to the marker position estimation image vMKb by different amounts are prepared, and the difference between the marker extraction image vMKa and the marker position estimation image vMKb is taken to select the peripheral light drop amount that minimizes the sum of the absolute values ​​of the differences of the corresponding pixels. For the distortion amount, the same method can be used. As will be described in detail later, through this process, the peripheral light drop amount and the distortion amount in the captured video vC are obtained.

[0218] The rendering engine 520 performs a mask generation process P14.

[0219] The rendering engine 520 generates a marker extraction image vMKa or an image obtained by adding a distortion amount to the marker position estimation image vMKb. Fig.15 The mask vK is information indicating that a first area 111 (shaded portion) of the captured video vC and a second area 112 of the synthesized CG video (e.g., background video vB) are used in the synthesis process P19. The first area 111 is an area inside the boundary marker 60 in the target frame of the captured video vC. The second area 112 is an area including the boundary marker 60 and located outside the first area 111.

[0220] Note that, in the mask vK, all values ​​corresponding to the corresponding pixels in the first region 111 are set to “1”, and all values ​​corresponding to the corresponding pixels in the second region 112 are set to “0”.

[0221] Alternatively, the mask vK may generate an alpha channel image having a value gradually transitioning from "1" to "0" by setting a boundary region (a mixed region 111a to be described later) with the second region 112 in the first region 111. The details will be described later.

[0222] The rendering engine 520 performs the peripheral light drop / distortion adding process P15 or the peripheral light drop / distortion removing process P16.

[0223] The peripheral light drop / distortion addition processing P15 uses the peripheral light drop amount and distortion amount obtained in the peripheral light drop amount / distortion amount detection processing P13 to add the peripheral light drop amount and distortion amount similar to the peripheral light drop amount and distortion amount of the captured video vC of the camera 502 to the background video vB.

[0224] The peripheral light drop / distortion removal process P16 is a process of applying opposite characteristics to the captured video vC and removing the peripheral light drop and distortion from the captured video vC using the peripheral light drop amount and distortion amount obtained in the peripheral light drop amount / distortion amount detection process P13.

[0225] Which of these processes is to be executed is determined by selecting the processes P17 and P18.

[0226] The selection process P17 selects the case where the captured video vC is output as is (referred to as "A process" for description) and the case where the peripheral light drop / distortion removal process P16 is performed on the captured video vC and the captured video vC is output (referred to as "B process" for description).

[0227] The selection process P18 selects the case where the peripheral light drop / distortion adding process P15 is performed on the background video vB and the background video vB is output (A process) and the case where the background video vB is output as it is (B process).

[0228] In the case where the A process is selected in the selection processes P17 and P18, a synthesis process P19 is performed on the captured video vC and the background video vB to which the peripheral light drop amount and distortion amount are added. In this case, the video using the distortion or light drop caused by the camera 502 is synthesized.

[0229] In the case where the B process is selected in the selection processes P17 and P18, the synthesis process P19 is performed on the captured video vC and the background video vB from which the peripheral light drop and distortion have been removed. In this case, a video without distortion or light drop caused by the camera 502 is synthesized.

[0230] It is only necessary to appropriately select the A process and the B process according to the production intention of the producer.

[0231] In the synthesis process P19, the rendering engine 520 uses the mask vK to synthesize the captured video vC and the background video vB selected in the A process or the B process, and generates Fig.16 The final output video vCF is shown in the figure.

[0232] The final output video vCF becomes a video in which a portion of the background video vB corresponding to the second area 112 is added as the out-of-frame added video vOT to the outside of the dotted line BD relative to the captured video vC corresponding to the first area 111 of the mask vK indicated as the inside of the dotted line BD. That is, a portion of the CG video generated as the background video vB is added as the out-of-frame added video vOT to the in-plane area outside the boundary marker 60 including the boundary marker 60 in the captured video vC.

[0233] By performing the captured video processing as described above, for example, through the rendering engine 520, even if the imaging range protrudes from the LED curtain wall 505, the background video vB is added to the protruding area, so that the protrusion can be prevented.

[0234] Note that the above captured video processing ( P11 to P19 ) is performed in real time upon imaging by a device having a configuration of the information processing apparatus 70 as the rendering engine 520 , the camera 502 , or another device.

[0235] Alternatively, the above captured video processing (P11 to P19) may also be performed in the information processing device 70 serving as a video editing device in post-production ST3.

[0236] Furthermore, part of the capture video processing (P11 to P19) may be performed in real time at the time of imaging, and part thereof may be performed in post-production ST3.

[0237] For example, the marker extraction process P11 and the marker position estimation process P12 may be performed at the time of imaging, and the other processes may be performed in post-production ST3. Also, for example, the marker extraction process P11 to the selection processes P17 and P18 may be performed at the time of imaging, and the synthesis process P19 may be performed in post-production ST3. Other examples are contemplated.

[0238] In the case where part or all of the captured video processing (P11 to P19) is performed in the post-production ST3, it is sufficient to store information necessary for the post-production ST3.

[0239] For example, in the case where all of the captured video processing (P11 to P19) is performed in post-production ST3, it is sufficient if the captured video vC, background video vB, marker generation information Imk, imaging information (the position, imaging direction and viewing angle of the camera 502 in each frame) and layout information of the LED curtain wall 505 are stored and the information processing device 70 that performs post-production ST3 can obtain the information.

[0240] Furthermore, for example, in the case where only the synthesis process P19 is performed in the post-production ST3, it is sufficient if the information processing device 70 that performs the post-production ST3 can acquire the captured video vC and the background video vB of the A process or the B process and the mask vK.

[0241] In addition, Fig. 9 In the example of , the background video vB is used as a source of the out-of-frame added video vOT, but the CG video used as the out-of-frame added video vOT may be generated separately from the background video vB.

[0242] also, Fig.10 An example of generating a background video vB including the outside of the actual display area 100 is illustrated. In this case, the background video vB can be used for the imaging range protruding from the LED curtain wall 505, but in the case where the background video vB protrudes more, or in the case where the background video vB is only a video of the actual display area 100, it is conceivable to generate a CG video used as an out-of-frame added video vOT separately from the background video vB.

[0243] <4. Processing Examples>

[0244] Include the description Fig. 9 Here, for example, it is assumed that the rendering engine 520 performs the labeling process and the captured video process in real time at the time of imaging.

[0245] Fig.17 is to add tag handling to Figure 6 An example of the processing at the time of rendering described in FIG. Figure 6 The steps in are similar, so redundant description is avoided.

[0246] In the process up to step S50, a frame of the background video vB including the capture area video vBC (inner frustum) is generated.

[0247] The rendering engine 520 performs the marking process from step S51 to step S55 on the background video vB. Note that "P2" and "P3" are added to indicate the Fig. 9 The correspondence between the mark processing in .

[0248] In step S51, the rendering engine 520 calculates feature quantities of the background video vB. The feature quantities include brightness, color, edge, etc.

[0249] In step S52, the rendering engine 520 sets the brightness and color of the boundary marker 60 that can be easily distinguished from the background according to the feature amount of the background video vB, and generates a marker image vMK having the boundary marker 60 according to the setting in step S53.

[0250] For example, if the background video vB indicates a bright scene, the brightness of the boundary marker 60 decreases, and if the background video vB indicates a dark scene, the brightness of the boundary marker 60 increases.

[0251] In addition, for example, in the case of a blue background, the brightness is set to a comparable and conspicuous color. This can prevent glare caused by using bright boundary markers 60 in dark scenes.

[0252] Furthermore, for example, in the case where the background video vB has many vertical lines as edges, the boundary marker 60 is set to have a pattern consisting of many horizontal lines.

[0253] The boundary marker 60 may be generated by synthesizing the features of brightness, color, and edge.

[0254] In addition, the boundary marker 60 according to the feature amount may be non-uniform, such as making the right half bright and the left half dark. That is, it may be changed according to the feature of the background of the portion where the boundary marker 60 is displayed.

[0255] In step S54, the rendering engine 520 stores the marker generation information Imk of the boundary marker 60 of the current frame in the storage medium. That is, the marker generation information Imk associated with the current frame can be read out at a later point in time.

[0256] In step S55 , the rendering engine 520 performs a process of adding the boundary marker 60 to the background video vB.

[0257] Then, in step S60, the rendering engine 520 outputs Fig.10 The background video vB illustrated in B includes the boundary mark 60, and the LED curtain wall 505 displays the background video vB.

[0258] Fig.18 Figure 1 shows an example of captured video processing. Added "P11" to "P19" to indicate Fig. 9 The corresponding relationship of the captured video processing.

[0259] For example, the rendering engine 520 acquires the information of the LED curtain wall 505 in step S100 before imaging. That is, it is the information of the arrangement position and angle of the LED curtain wall 505 used for the marker position estimation process P12.

[0260] After imaging starts, the rendering engine 520 performs the processing of steps S101 to S113 for each frame of the captured video vC.

[0261] In step S101 , the rendering engine 520 acquires marker generation information Imk of a frame that is a current processing target.

[0262] In step S102 , the rendering engine 520 obtains imaging information of the frame that is currently being processed through the camera tracker 560 , in particular, information on the position, imaging direction, and viewing angle of the camera 502 .

[0263] In step S103 , the rendering engine 520 performs a marker position estimation process P12 using the information of the LED curtain wall 505 , the imaging information, and the marker generation information Imk (pattern shape and size) to generate a marker position estimation image vMKb.

[0264] Furthermore, in step S104 , the rendering engine 520 sets a threshold value based on the marker generation information Imk (brightness and color), and performs a marker extraction process P11 from the captured video vC to generate a marker extraction image vMKa.

[0265] Note that the order of steps S103 and S104 may be reversed.

[0266] Furthermore, in order to reduce the processing load in real time, step S103 may be executed only when the processing proceeds from step S105 to step S106 .

[0267] In step S105 , the rendering engine 520 determines whether part or all of the boundary marker 60 has been captured for the current frame.

[0268] Through the processing of step S104 , it can be determined whether part or all of the boundary marker 60 is mapped into the current frame.

[0269] If the boundary marker 60 is not mapped into the current frame, it is not necessary to perform the captured video processing according to the boundary marker 60. Therefore, the rendering engine 520 proceeds from step S105 to step S113, and outputs the frame of the captured video vC as it is as the final output video vCF obtained by imaging. For example, it is stored in a storage medium as a frame of the output video obtained by imaging, or is transmitted to an external device.

[0270] On the other hand, when the boundary marker 60 is mapped in the current frame, the captured video process requires the synthesis process P19 for replacing the external video including the boundary marker 60. Therefore, the rendering engine 520 proceeds from step S105 to step S106 and executes the peripheral light drop amount / distortion amount detection process P13.

[0271] Fig.19 A specific example of the peripheral light drop amount / distortion amount detection process P13 is illustrated.

[0272] The rendering engine 520 calculates the peripheral light amount drop amount and the distortion amount using the marker extraction image vMKa and the marker position estimation image vMKb.

[0273] First, the rendering engine 520 performs a binarization process P21 on the mark extraction image vMKa. Since the mark extraction image vMKa is an image obtained by extracting the boundary marker 60 from the captured video vC, the mark extraction image vMKa is not in a binary state. It is binarized to remove the peripheral light drop.

[0274] Furthermore, as the lens distortion adding process P22, the rendering engine 520 generates an image to which lens distortion of multiple levels from small to large is added based on the marker position estimation image vMKb. Fig. 20 As shown in FIG. 1 , n images are generated as images vMKb(n) starting from image vMKb1 . The distortion amounts of images vMKb1 to vMKb(n) are made different in stages.

[0275] Next, the rendering engine 520 performs difference calculation P23. This is a process of calculating a difference between each of the images vMKb1 to vMKb(n) and the binarized marker extraction image vMKa.

[0276] Then, the rendering engine 520 performs a distortion amount determination process P24. This is a process of determining an image having the smallest difference from the binarized marker extraction image vMKa among the images vMKb1 to vMKb(n), and determining the distortion amount added to the image as the distortion amount in the captured video vC of the current frame. For example, in the case where the image vMKb2 is the image having the smallest difference from the binarized marker extraction image vMKa, the distortion amount added to the image vMKb2 is the distortion amount to be detected.

[0277] Therefore, the amount of distortion in the captured video vC is detected.

[0278] Next, in order to detect the peripheral light amount drop amount, the rendering engine 520 performs a lens distortion adding process P25 on the marker position estimation image vMKb. That is, it is assumed that the marker position estimation image vMKb has lens distortion similarly to the captured video vC.

[0279] Note that the image selected in the distortion amount determination process P24 (eg, image vMKb2) can be used as it is without performing the lens distortion addition process P25. This is because the same lens distortion amount is added to the original marker position estimated image vMKb.

[0280] In the peripheral light drop addition process P26, based on the marker position estimation image vMKb (or similar image vMKb2) to which the detected distortion amount is added, an image to which multiple levels of peripheral light drop amounts from small to large are added is generated. Fig. 20As shown in FIG. 1 , m images are generated as images vMKb(m) starting from image vMKb11 . The peripheral light amount reduction amounts of images vMKb11 to vMKb(m) are made different in stages.

[0281] Next, the rendering engine 520 performs difference calculation P27. This is a process of calculating a difference between each of the images vMKb(m) from the image vMKb11 and the marker extraction image vMKa.

[0282] Then, the rendering engine 520 performs peripheral light drop amount determination processing P28. This is a process of determining the image with the smallest difference from the marker extraction image vMKa among the images vMKb11 to vMKb(m), and determining the peripheral light drop amount added to the image as the peripheral light drop amount in the captured video vC of the current frame. For example, in the case where the image vMKb12 is the image with the smallest difference from the marker extraction image vMKa, the peripheral light drop amount added to the image vMKb12 is the peripheral light drop amount to be detected. As described above, the peripheral light drop amount in the captured video vC is detected.

[0283] In executing Fig.19 After the peripheral light drop amount / distortion amount detection process P13 shown in FIG. Fig.18 In step S107, the processing is branched by selecting which of the A process and the B process is set by the processing P17 and P18.

[0284] In the case of the A process, the rendering engine 520 proceeds to step S107, performs the mask generation process P14, and performs the peripheral light drop / distortion addition process P15 on the background video vB in step S110.

[0285] In the case of the B process, the rendering engine 520 proceeds to step S109, executes the mask generation process P14, and executes the peripheral light drop / distortion removal process P16 on the captured video vC in step S111.

[0286] Will refer to Fig.21 The mask generation process P14 in step S108 is described.

[0287] As for the mask vK, as described above, the first region 111 may be "1" and the second region 112 may be "0", but Fig.21 An example of processing for blending at a boundary portion between the first region 111 and the second region 112 is illustrated. Note that Fig.21 An example of processing in step S108 (ie, when the A process is selected) is illustrated.

[0288] The mask vK may be generated based on the marker extraction image vMKa or the marker position estimation image vMKb.

[0289] As Fig.21 In the selection process P33 , the rendering engine 520 selects which of the marker extraction image vMKa and the marker position estimation image vMKb is to be used as the basis for generating the mask vK.

[0290] In the case where the marker position estimated image vMKb is selected, the rendering engine 520 adds the distortion amount detected in step S106 to the marker position estimated image vMKb through the distortion adding process P31 .

[0291] When the marker extraction image vMKa is selected, the rendering engine 520 binarizes the marker extraction image vMKa through the binarization process P32.

[0292] The rendering engine 520 performs the value setting process P34 on the distortion-added marker position estimation image vMKb or the binarized marker extraction image vMKa.

[0293] In the value setting process P34, if Fig. 22 As illustrated in A of , a mask vK is generated in which the inner first region 111 has α channel=1 and the outer second region 112 has channel=0.

[0294] Note that although the boundary marker 60 is Fig. 22 A is indicated by a dotted line, but this is added to indicate that the first region 111 is a region inside the boundary marker 60, and the boundary marker 60 is not added to the mask vK.

[0295] Next, the rendering engine 520 performs a tilt adding process P35.

[0296] like Fig. 22 As illustrated in FIG. 1B , a mixed region 111a is provided at a boundary portion between the first region 111 and the second region 112. In the mixed region 111a, the α channel is inclined so that the α channel changes between “0” and “1” in a direction away from a position in contact with the second region 112.

[0297] For example, the value of the alpha channel changes from the outside to the inside to "0.1", "0.2", ..., "0.9".

[0298] The rendering engine 520 outputs a mask vK to which the tilt is added in this manner.

[0299] After the mask generation process P14 is performed in step S108 as described above, the peripheral light amount drop / distortion addition process P15 is performed on the background video vB in step S110 .

[0300] Since this is the case where the A process is selected, the synthesis process P19 is performed by capturing the video vC, the background video vB to which the peripheral light drop / distortion is added, and the mask vK generated in step S108.

[0301] Note that in the mask generation process P14 in the case where the B process is selected and the process advances from step S107 to step S109, Fig.21 The processing of providing the mixed area 111a is performed similarly. However, in this case, when the mask vK is generated based on the marker position estimated image vMKb, it is not necessary to add distortion to the marker position estimated image vMKb. The value setting processing P34 and the tilt adding processing P35 only need to be performed using the marker position estimated image vMKb as it is.

[0302] Furthermore, in the case of generating the mask vK based on the marker extraction image vMKa, the marker extraction image vMKa is binarized, further distortion removal processing is performed, and then the value setting processing P34 and the tilt adding processing P35 are performed.

[0303] After the mask generation process P14 is performed in step S109 as described above, the peripheral light drop / distortion removal process P16 is performed on the captured video vC in step S111 .

[0304] Since this is the case where the B process is selected, the synthesis process P19 is performed by the captured video vC from which the peripheral light drop / distortion has been removed, the background video vB, and the mask vK generated in step S109.

[0305] In step S112, the rendering engine 520 performs a synthesis process P19 of the captured video vC and the background video vB. Fig.23 An example of the synthesis process P19 is illustrated.

[0306] In the synthesis process P19, the rendering engine 520 performs a multiplication process P41 of multiplying the pixel value of the frame of the captured video vC by the α value of the mask vK. As described above, since the first area 111 has an α channel = "1", the pixel value in the captured video vC is maintained as it is. However, the pixels of the mixed area 111a are multiplied by coefficients set in an inclined shape from "1" to "0" from the inside to the outside according to the position of the pixel. Since the second area 112 has an α channel = "0", the pixel value in the captured video vC is "0".

[0307] Furthermore, the rendering engine 520 performs an α value inversion process P42 on the mask vK, and then performs a multiplication process P43 of multiplying the mask vK with the background video vB.

[0308] By the alpha value inversion process P42 , the first region 111 has an alpha channel = “0”, and the second region 112 has an alpha channel = “1.” The mixed region 111 a has a slope value from “1” to “0” from the second region 112 inward.

[0309] Then, in the multiplication process P43, since the second region 112 has α channel = "1", the pixel value in the background video vB is kept as it is. Since the first region 111 has α channel = "0", the pixel value in the background video vB is "0". However, the pixels of the mixed region 111a are multiplied by coefficients set in an inclined shape from "1" to "0" from the outside to the inside according to the position of the pixel.

[0310] The rendering engine 520 performs an addition synthesis process P44 on the outputs of the multiplication processes P41 and P43.

[0311] Therefore, the inside of the mixed area 111a in the first area 111 is the captured video vC, the second area 112 is the background video vB, and the mixed area 111a gradually changes from a value with high relative importance of the captured video vC to a value with high relative importance of the background video vB from the inside to the outside, and the following is obtained: Fig.16 The final output video vCF is shown in the figure.

[0312] Note that although Fig.23 An example of performing so-called α blending is illustrated, but even in a case where the blending region 111a is not set in the mask vK and the α value is set to “1” for the first region 111 and to “0” for the second region 112, Fig.23 The processing performs synthesis.

[0313] In the case of virtual production, the LED curtain wall 505 will not be focused, so that there is no problem such as moire. In this case, the position, brightness and color of the background video vB and the captured video vC to be synthesized do not need to be accurately matched in pixels.

[0314] This is the end of the processing for one frame. In step S114, the rendering engine 520 determines whether to end the processing, and if not, returns to step S101. That is, similar processing is continuously performed corresponding to the next frame.

[0315] For example, when imaging is finished, the rendering engine 520 determines the end in step S114, and ends Fig.18 processing.

[0316] <5. Processing in Pre-Visualization>

[0317] In the above captured video processing, the marker extraction processing P11, the marker position estimation processing P12, the peripheral light drop amount / distortion amount detection processing P13, the mask generation processing P14, etc. are performed, but some of these can be performed before imaging. Therefore, the burden of real-time captured video processing during imaging can be reduced.

[0318] Therefore, pre-visualization prior to imaging will be described.

[0319] exist Fig.24 FIG. 4 illustrates the positioning of pre-visualization in the video production process.

[0320] Fig.24 The above reference is shown Figure 4 Describes the process of asset creation ST1, production ST2 and post-production ST3.

[0321] Production ST2 is the stage where imaging is actually performed in the studio, but this is divided into imaging preparation and imaging.

[0322] On the other hand, it is assumed that pre-visualization is mainly performed at a stage before imaging preparation. Note that the time sequence is not necessarily limited, but pre-visualization only needs to be performed at a stage after a 3D model for rendering background video vB is produced in asset production ST1.

[0323] In this sense, pre-visualization may be considered to be performed at the final stage of asset creation ST1 , or may be considered to be performed at a stage prior to imaging preparation in production ST2 .

[0324] In pre-visualization, a process called video confirmation and moiré alarm is performed. Video confirmation is a process for enabling a director or a crew member to confirm a captured video vC obtained by imaging before imaging. Confirmation mainly including the color of the captured video vC can be performed by simulating video.

[0325] For example, the moiré alarm is a process in which a worker performs moiré prediction while virtually manipulating the camera position while viewing a video, and issues a warning if moiré is predicted to occur. This makes it possible to avoid artifacts in subsequent imaging.

[0326] In the imaging preparation stage, color calibration is performed together with various settings such as LEDs and cameras in the imaging system 500. Specifically, a calibration lookup table (LUT) is generated that eliminates color variations caused when the LED curtain wall 505 is imaged by the camera 502. For example, the calibration LUT is a 3D-LUT.

[0327] Note that, instead of the 3D-LUT, synthesis of the matrix with the 1D-LUT may be performed or prepared for calibration performed in other video signal processing.

[0328] During the imaging phase, moiré alarm, calibration LUT and camera setup are performed.

[0329] The moiré alarm in this case is a process of outputting a moiré alarm to a camera operator or the like in real time based on the moiré determination of the rendering engine 520 .

[0330] Calibration LUT is the process of eliminating color variations associated with background video display of LED wall 505, camera 502, etc. by applying a LUT generated by color calibration in preparation for imaging.

[0331] Turning on the camera setting is a process of acquiring the camera setting set in the pre-visualization from, for example, a cloud server or the like and reflecting the camera setting in the camera 502 of the studio.

[0332] Fig.25 The configuration of the imaging system 500 and the configuration of the pre-visualization are illustrated in parallel. The upper portion of the figure illustrates Figure 5 et al., and the lower portion of the figure illustrates a simplified configuration of the imaging system 500 described in , and a configuration used in pre-visualization (pre-visualization system 700 ).

[0333] exist Fig.25 , regarding the imaging system 500, a rendering engine 520, an LED processor 570, an LED curtain wall 505, a camera 502, a capture video processing unit 85a, an operation monitor 550 or an output monitor 503 are illustrated.

[0334] The rendering engine 520 includes a rendering unit 520 a and a marking processing unit 85 b. Therefore, the background video vB with the boundary marking 60 is displayed on the LED curtain wall 505.

[0335] For the camera 502, a captured video processing unit 85a that performs processing on the captured video vC is illustrated. The captured video processing unit 85a is provided in the rendering engine 520 or the information processing apparatus 70 constituting another device.

[0336] The output final output video vCF is recorded on a recording medium as a main line video, or is transmitted to another device for broadcasting or distribution.

[0337] For the imaging system 500 as described above, for example, the pre-visualization system 700 is configured as a system outside the studio. The pre-visualization system 700 includes at least an information processing device 70 serving as a rendering engine 31 and a monitor 40 .

[0338] For example, the rendering engine 31 includes an information processing device 70 separate from the rendering engine 520 .

[0339] The rendering engine 31 includes a rendering unit 32 and a video processing unit 33 .

[0340] The rendering unit 32 has a rendering function similar to that of the rendering unit 520 a in the above-described rendering engine 520 , and can generate a background video vBP based on a 3D model.

[0341] The rendering unit 32 may perform rendering of the background video vB using at least the 3D model produced in the process of asset creation ST1. Figure 5 and Figure 7 ) obtains the same 3D model as the 3D model used in the process of making ST2, and uses the 3D model to perform rendering.

[0342] Then, the rendering unit 32 may perform rendering of the background video vB including the capture area video vBC (inner frustum) using the 3D model.

[0343] Note that in order to generate the captured area video vBC, the camera position, imaging direction and viewing angle are input to the rendering unit 32, and rendering is performed at the viewpoint position relative to the 3D model corresponding thereto. Therefore, it is assumed that the camera technique at the time of actual imaging (such as the position and imaging direction of the camera 502, the camera path and the change of viewing angle) has been determined. By obtaining information on the camera technique at the time of actual imaging, a captured area video vBC similar to the captured area video at the time of imaging can be obtained.

[0344] For the sake of description, the background video generated by the rendering unit 32 is referred to as “background video vBP” in the sense of being distinguished from the capture area video vBC generated by the rendering engine 520 at the time of imaging.

[0345] The video processing unit 33 indicates a function of performing various types of video processing on the background video vBP. The video processing unit 33 has functions as a marking processing unit 85b, a captured video processing unit 85a, and an actual captured video conversion processing unit 33a.

[0346] The video processing unit 33 performs the marking process by the function of the marking processing unit 85b. That is, in the background video vBP, the boundary marker 60 is added to a part that is to become the peripheral part when displayed on the LED curtain wall 505.

[0347] In addition, the video processing unit 33 performs actual captured video conversion processing through the function of the actual captured video conversion processing unit 33a. The actual captured video conversion processing is a process of generating a simulated video vSM that simulates the camera 502 actually used for imaging. In the actual captured video conversion processing, the simulated video vSM is generated for the background video vBP using at least a processing parameter that realizes the brightness or color characteristics of the camera 502 used in the imaging system 500 when imaging. Specifically, the video processing unit 33 adjusts parameters such as RGB gain, white balance value, gamma value, and offset value for the camera 502.

[0348] By setting the parameters related to brightness or color to be the same as those of the camera 502 during imaging, an analog video vSM having the same color as that of the captured video vC is generated and displayed on the monitor 40.

[0349] The monitor 40 indicates a monitor device that displays a video output from the information processing apparatus 70 as the rendering engine 31, and may be, for example, a monitor device in a place different from the imaging studio or a monitor device in the imaging studio. The monitor 40 may have any form as long as it displays the simulation video vSM at least in the pre-visualization stage.

[0350] In the rendering engine 31, the rendering unit 32 generates a background video vBP, and the video processing unit 33 performs actual captured video conversion processing to generate a simulated video vSM. By displaying it on the monitor 40, the staff can confirm the simulated video vSM having the same color as the background video included in the captured video vC at the time of imaging. Therefore, the color can be checked before imaging, and if necessary, color adjustment can be performed in the camera 502 or the camera signal processing unit 515, etc. at the imaging preparation stage.

[0351] Here, the video processing unit 33 performs a part of the captured video processing on the analog video vSM through the function of the captured video processing unit 85a.

[0352] For example, Fig.26 An example of processing by the video processing unit 33 is illustrated.

[0353] In the video processing unit step S200, the video processing unit 33 obtains information on the arrangement position and angle of the LED curtain wall 505 during imaging.

[0354] After starting to generate the simulation video vSM, the video processing unit 33 performs the processing of steps S201 to S208 for each frame of the captured video vC.

[0355] In step S201 , the video processing unit 33 acquires marker generation information Imk of a frame which is a current processing target of the simulation video vSM.

[0356] In step S202, the video processing unit 33 obtains information on the position, imaging direction, and viewing angle of the camera 502 as imaging information during actual imaging of the frame that is the current processing target. When the camera technique is predetermined, the information on the position, imaging direction, and viewing angle of the camera 502 corresponding to the frame of the simulated video vSM can be obtained.

[0357] In step S203 , the video processing unit 33 performs a marker position estimation process P12 using the information of the LED curtain wall 505 , the imaging information, and the marker generation information Imk (pattern shape and size) to generate a marker position estimation image vMKb.

[0358] Furthermore, in step S204 , the video processing unit 33 sets a threshold value based on the marker generation information Imk (brightness and color), and performs a marker extraction process P11 from the simulation video vSM to generate a marker extraction image vMKa.

[0359] In step S205 , the video processing unit 33 determines whether part or all of the boundary marker 60 is included in the current frame.

[0360] When the boundary marker 60 is not included in the current frame, the process proceeds to step S208.

[0361] On the other hand, if the boundary marker 60 is included in the current frame, the video processing unit 33 proceeds to step S206 and executes the peripheral light amount drop amount / distortion amount detection process P13.

[0362] Then, in step S207, the peripheral light drop amount and the distortion amount are stored as information of the current frame. For example, it is stored in Fig.25 The DB 42 is stored in the image processing module so that it can be referred to when the captured video processing is performed at the time of imaging.

[0363] The video processing unit 33 repeats the above process for each frame until determining in step S208 that the process is finished.

[0364] Through this process, the DB 42 detects the peripheral light amount drop amount and the distortion amount for each frame of the captured video vC before imaging.

[0365] Therefore, in Fig.18 When the captured video is processed, step S106 does not need to be executed. In addition, it is sufficient if either step S103 or step S104 is executed. This is because a mask can be generated from the marker extraction image vMKa or the marker position estimation image vMKb.

[0366] Note that the processing until the mask vK is generated may be performed in the pre-visualization stage so that in the captured video processing at the time of imaging, the mask vK is read for each frame and the synthesis processing P19 is performed.

[0367] <6. Summary and Modification Examples>

[0368] According to the above-described embodiment, the following effects can be obtained.

[0369] The information processing device 70 according to the embodiment performs the captured video processing and the marking processing as, for example, the rendering engine 520, the processor in the camera 502, the device that receives the input of the captured video vC from the camera 502 at the time of imaging, etc. Alternatively, the information processing device 70 according to the embodiment is a device that performs video editing in post-production ST3 and performs the captured video processing. In addition, the information processing device 70 according to the embodiment is used as the rendering engine 31 used in the pre-visualization stage, and performs the marking processing and the captured video processing.

[0370] Such an information processing device 70 includes a captured video processing unit 85a, which performs a determination process for determining the position of a boundary marker in a target frame of a captured video vC obtained by imaging an LED curtain wall 505 through a camera 502, wherein the LED curtain wall displays a video in which a boundary marker 60 is added to a virtual video such as a background video vB.

[0371] By detecting the boundary marker 60 in the captured video vC, even when the imaging range of the camera 502 protrudes beyond the display range of the LED curtain wall 505, the protruding position and amount can be accurately detected and the handling process can be performed. For example, compensation of the video such as the protruding part can be accurately performed, and waste of imaging can be prevented.

[0372] Furthermore, as a result, the camera operator 512 does not have to be clearly conspicuous when imaging so that the imaging range falls within the range of the LED curtain wall 505, and the degree of freedom of framing is expanded.

[0373] In the embodiment, as the determination process, the captured video processing unit 85a performs a process of extracting the boundary marker 60 from the target frame of the captured video vC based on the marker generation information Imk. That is, in the marker extraction process P11, the captured video processing unit 85a in the information processing device 70 such as the rendering engine 520 obtains the brightness and color information of the boundary marker 60 from the marker generation information Imk, and accordingly extracts the boundary marker 60 in the captured video vC.

[0374] In the case where clear brightness or color is selected for the color or background content of the background video vB and the boundary marker 60 is generated, the boundary marker 60 can be accurately extracted from the captured video vC by obtaining brightness or color information.

[0375] In addition, in other words, the brightness and color of the boundary marker 60 can be set according to the content of the background video vB. Therefore, the boundary marker 60 can be made to stand out in the background video vB.

[0376] In an embodiment, as a determination process, the captured video processing unit 85a performs a marker position estimation process P12 for estimating the position of the boundary marker 60 in the target frame of the captured video vC based on imaging information including the position information of the camera 502 during imaging, imaging direction and viewing angle, and position information of the LED curtain wall 505.

[0377] Therefore, the position of the boundary marker 60 in the captured video vC can be accurately determined in a state where there is no lens distortion or the like.

[0378] In the embodiment, the captured video processing unit 85a performs a first determination process (marker extraction process P11) of extracting the boundary marker 60 from the target frame of the captured video vC and a second determination process (marker position estimation process P12) of estimating the position of the boundary marker 60 in the target frame of the captured video vC. Then, the captured video processing unit 85a performs a process of calculating the distortion amount of the captured video vC using the marker extraction image vMKa and the marker position estimation image vMKb as the above results (see FIG. Fig.18 and Fig.19 , or step S106 in the above procedure.

[0379] By obtaining the distortion amount, adding distortion to the background video vB to be the frame-outside added video vOT and removing distortion from the captured video vC can be performed correctly, and the frame-outside added video vOT can be synthesized without discomfort. Therefore, the quality of the final output video vCF can be improved.

[0380] Furthermore, in the present embodiment, the captured video processing unit 85a performs processing for calculating the peripheral light amount drop amount of the captured video vC using the marker extraction image vMKa and the marker position estimation image vMKb in the first determination processing (marker extraction processing P11) and the second determination processing (marker position estimation processing P12) (see Fig.18 and Fig.19 , or step S106 in the above procedure.

[0381] By obtaining the peripheral light drop amount, it is possible to accurately add the peripheral light drop to the frame-outside added video vOT and remove the peripheral light drop from the captured video vC, and synthesize the frame-outside added video vOT without discomfort. Therefore, the quality of the final output video vCF can be improved.

[0382] In the embodiment, the captured video processing unit 85a performs a process of generating a mask vK for distinguishing the first region 111 inside the boundary marker 60 and the second region 112 outside the first region 111 in the target frame of the captured video vC, and a synthesis process of synthesizing the out-of-frame added video vOT, which is the video corresponding to the second region 112, with the video of the first region 111 in the captured video vC using the mask vK. That is, the captured video processing unit 85a performs the synthesis process P19 (see FIG. 14 ) using the mask vK generated in the mask generation process P14. Fig.18 , Fig.21 and Fig.23 . , steps S107 and S111 in FIG.

[0383] Therefore, a synthesis process is performed to replace the second area 112 including the boundary mark 60 in the captured video vC with the out-of-frame added video vOT, and a final output video vCF that is a portion protruding from the LED curtain wall 505 and is also a proper background can be output.

[0384] Note that, as described above, such mask generation processing P14 and synthesis processing P19 can be executed in real time at the time of imaging, but one or both may be executed at the stage of post-production ST3.

[0385] For example, if imaging information including the position information of the camera 502 during imaging, the imaging direction and the viewing angle, the position information of the LED curtain wall 505, and the marker generation information Imk are stored in association with each frame of the captured video vC, then the post-production ST3 can be performed. Fig.18 processing.

[0386] Furthermore, if the background video vB is stored in association with each frame of the captured video vC, the background video vB may be used as it is in the processes of steps S109 and S110 .

[0387] At the time of imaging, the mask vK may be generated in real time and stored in association with each frame of the captured video vC, and the mask vK may be read and used at the time of the compositing process in post-production ST3.

[0388] In the example of the embodiment, in the synthesis process P19, the captured video processing unit 85a may synthesize the video of the first area 111 in the captured video vC with the out-of-frame added video vOT to which the amount of distortion or the amount of peripheral light drop generated in the shot video vC is added. Fig. 9 The selection handles the case where process A is selected in P17 and P18.

[0389] Therefore, the frame-outside added video vOT to which the approximate distortion or the peripheral light amount drop is applied is synthesized with the captured video vC of the first area 111 in the second area 112. Therefore, the frame-outside added video vOT is synthesized with the captured video vC without discomfort. In particular, in this case, the final output video vCF in a state of lens distortion or the like of the camera 502 can be obtained.

[0390] Note that the distortion can be calculated for each frame during the pre-visualization phase (see Fig.26 ). Therefore, step S106 is used as the Fig.18 The processing of is unnecessary, and the processing load for adding the distortion amount to the background video vB to be the frame-outside added video vOT, etc. is reduced. Therefore, for real-time execution at the time of imaging Fig.18 It is preferred for the treatment situation.

[0391] Furthermore, in the example of the embodiment, there is a case where, in the synthesis process P19, the captured video processing unit 85a performs a process of synthesizing the video of the first area 111 in the captured video vC from which the distortion amount or the peripheral light amount drop amount has been removed with the frame-outside added video vOT (background video vB, etc.). That is, this is in Fig. 9 The selection handles the case where process B is selected in P17 and P18.

[0392] Therefore, the frame-outside added video vOT to which no distortion or peripheral light drop is applied is synthesized with the captured video vC from which the distortion or peripheral light drop of the first area 111 has been removed. Therefore, the frame-outside added video vOT is synthesized with the captured video vC without discomfort. In particular, in this case, the final output video vCF in a state where the lens distortion of the camera 502 is offset can be obtained.

[0393] Note that the amount of peripheral light drop is calculated for each frame during the pre-visualization phase (see Fig.26 ), when imaging Fig.18 In the process of step S106 is not necessary, and the processing load for removing the peripheral light drop amount of the captured video vC is reduced. Therefore, the processing is performed in real time during imaging. Fig.18 The processing case is preferred.

[0394] Furthermore, although the A process and the B process are selected, it is also conceivable to execute only the A process or only the B process without executing the selection processes P17 and P18.

[0395] That is, in the synthesis process P19, the video of the first area 111 in the captured video vC and the out-of-frame added video vOT (background video vB, etc.) to which the distortion amount or peripheral light drop amount generated in the captured video vC is added can be always synthesized.

[0396] Alternatively, in the synthesis process P19, the video of the first area 111 in the captured video vC from which the distortion amount or the peripheral light amount drop amount has been removed and the frame-outside added video vOT (background video vB, etc.) may be synthesized at all times.

[0397] In the embodiment, the following example is described: in the synthesis process P19, the captured video processing unit 85a performs synthesis by changing the video ratio of the captured video vC and the out-of-frame added video vOT based on the background video vB, etc. in stages near the boundary between the first area 111 and the second area 112.

[0398] As reference Fig. 22 and Fig.23 As described, the vicinity of the boundary of the first region 111 is set as the mixed region 111a, and the α value of the mask vK is gradually changed between "0" and "1". Therefore, the boundary between the captured video vC and the out-of-frame added video vOT can be made inconspicuous, and the quality of the final output video vCF can be improved.

[0399] In the embodiment, it has been described that the rendered background video vB to be displayed on the LED curtain wall 505 is used as an example of the out-of-frame added video vOT.

[0400] For example, the background video vB is directly used as the frame-outside added video vOT. Therefore, it is not necessary to separately produce the frame-outside added video vOT. In addition, for this reason, it is also suitable for real-time synthesis processing at the time of imaging.

[0401] However, the video of the second area 112 may be generated separately from the background video vB to be the frame-outside added video vOT.

[0402] In the embodiment, the example has been described in which the information processing device 70 includes the marking processing unit 85 b that performs a process of adding the boundary marker 60 to the background video vB (virtual video) to be displayed on the LED curtain wall 505 .

[0403] Therefore, the boundary marker 60 is displayed on the LED curtain wall 505. Therefore, the processing of the above-mentioned captured video processing unit 85a can be performed.

[0404] Note that both the functions of the capture video processing unit 85 a and the marking processing unit 85 b may be included in the information processing device 70 such as the rendering engine 520 , for example, but may be provided by a separate information processing device 70 .

[0405] In the embodiment, the following example has been described: the marker processing unit 85b generates a boundary marker 60 indicating the display area boundary of the LED curtain wall 505 and having a pattern that enables detection of the amount of distortion or the amount of peripheral light drop caused by imaging by the camera 502, and adds the boundary marker to the background video vB.

[0406] For example, the boundary marker 60 has a boundary line 61 and a detection pattern 62. Thus, the detection processing of the peripheral light amount drop amount and the distortion amount in the captured video processing unit 85a and the removal and addition processing thereof can be performed.

[0407] Note that the boundary mark 60 may include at least the boundary line 61, and it is sufficient if it can be detected that the imaging range protrudes outside the frame of the LED curtain wall 505. That is, there is an example in which the detection pattern 62 is not provided.

[0408] In the embodiment, the marking processing unit 85 b performs a process of adding the boundary marker 60 to a position corresponding to the peripheral portion of the LED curtain wall 505 in the background video vB displayed on the LED curtain wall 505 .

[0409] The entire area of ​​the background video vB generated by rendering is not necessarily displayed on the LED curtain wall 505. Fig.10 As shown in FIG. 1A , a background video vB having a wider range than the actual display area 100 can be generated. Fig.10 As shown in FIG. 1B , the boundary marker 60 is added to the peripheral portion of the actual display area 100, which is the range actually displayed on the LED curtain wall 505. Fig. 9 As illustrated in FIG. 5 , the boundary marker 60 is displayed on the peripheral portion of the LED curtain wall 505 , and the processing of the captured video processing unit 85 a can be appropriately performed.

[0410] Note that, for example, when the LED wall 505 displays Fig.10 When the entire background video vB is shown in FIG. A, it is appropriate to add the boundary marker 60 to the peripheral portion of the entire background video vB.

[0411] In the embodiment, the mark processing unit 85b performs processing of recording the mark generation information Imk (see Fig.17 , or step S54 in the above procedure.

[0412] For example, marker generation information Imk such as brightness, color, pattern shape and size of the boundary marker 60 is stored in the recording medium. Thus, even at a timing after adding a marker, such as at the stage of post-production ST3, marker extraction can be performed from the captured video vC.

[0413] Note that, Figure 8 The information processing device 70 in FIG. 1 is an example in which both the captured video processing unit 85a as a function of performing captured video processing and the labeling processing unit 85b as a function of performing labeling processing are included as the video processing unit 33, but the present invention is not limited thereto. For example, it is also possible to assume that the information processing device 70 includes the labeling processing unit 85b but does not include the captured video processing unit 85a, and the information processing device 70 includes the captured video processing unit 85a but does not include the labeling processing unit 85b.

[0414] In addition, in the embodiment, the background video vB, which is the video to be displayed on the LED curtain wall 505, is a virtual video obtained by rendering using a 3D model, but for example, even in the case where a real video image is displayed on the LED curtain wall 505, the marking processing for adding the boundary marker 60 and the captured video processing for the captured video vC are effective.

[0415] The program according to the embodiment is a program for causing a processor (eg, a CPU or a DSP, etc.) or a device including the processor to execute Fig.17 , Fig.18 and Fig.26 The processing procedure is shown in the figure.

[0416] That is, the program of the present embodiment is a program for causing the information processing device 70 to perform the following processing: determining the position of the boundary marker 60 in the target frame of the captured video vC relative to the captured video vC obtained by imaging a display by a camera, wherein the display displays a video (for example, a virtual video such as the background video vB) including the boundary marker 60 to be displayed on a peripheral portion in a display area of ​​a display such as an LED curtain wall 505.

[0417] With such a program, the information processing device 70 operating in the production ST2 such as the above-mentioned rendering engine 520, the information processing device 70 used in the post-production ST3, or the information processing device 70 used in pre-visualization can be realized by various computer devices.

[0418] Such a program may be recorded in advance in a HDD as a recording medium built into a device such as a computer apparatus, a ROM in a microcomputer having a CPU, or the like. In addition, such a program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium may be provided as so-called packaged software.

[0419] Furthermore, such a program may be installed from a removable recording medium into a personal computer or the like, or may be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0420] In addition, such a program is suitable for providing the information processing device 70 of the embodiment in a wide range. For example, by downloading the program to a personal computer, a communication device, a portable terminal device such as a smartphone or a tablet, a mobile phone, a game device, a video device, a personal digital assistant (PDA), etc., these devices can be used as the information processing device 70 of the present disclosure.

[0421] Note that the effects described in this specification are merely examples and not limitations, and other effects may be provided.

[0422] Note that the present technology may also have the following configurations.

[0423] (1) An information processing device comprising:

[0424] A captured video processing unit that performs the following determination processing: determining the position of a boundary mark displayed at a peripheral portion in a display area of ​​a target frame of a captured video obtained by imaging a display by a camera relative to the captured video, the display displaying a video including the boundary mark.

[0425] (2) The information processing device according to (1) above, wherein:

[0426] As the determination processing, the captured video processing unit performs a process of extracting the boundary marker from the target frame of the captured video based on the generation information of the boundary marker.

[0427] (3) The information processing device according to (1) or (2) above, wherein:

[0428] As the determination processing, the captured video processing unit performs processing for estimating the position of the boundary marker in the target frame of the captured video based on imaging information including position information of the camera at the time of imaging, imaging direction and viewing angle, and position information of the display.

[0429] (4) The information processing device according to any one of (1) to (3) above, wherein:

[0430] The capture video processing unit performs:

[0431] a first determination process of extracting the boundary marker from the target frame of the captured video;

[0432] a second determination process of estimating a position of the boundary marker in the target frame of the captured video; and

[0433] A process of calculating the distortion amount of the captured video based on a determination result of the first determination process and a determination result of the second determination process.

[0434] (5) The information processing device according to any one of (1) to (4) above, wherein:

[0435] The capture video processing unit performs:

[0436] a first determination process of extracting the boundary marker from the target frame of the captured video;

[0437] a second determination process of estimating a position of the boundary marker in the target frame of the captured video; and

[0438] A process of calculating an amount of decrease in the peripheral light amount of the captured video based on a determination result of the first determination process and a determination result of the second determination process.

[0439] (6) The information processing device according to any one of (1) to (5) above, wherein:

[0440] The capture video processing unit performs:

[0441] a process of generating a mask for distinguishing a first area within the boundary marker and a second area outside the first area in the target frame of the captured video; and

[0442] A synthesis process of synthesizing an additional video, which is a video corresponding to the second area, with the video of the first area in the captured video using the mask.

[0443] (7) The information processing device according to (6) above, wherein:

[0444] The capture video processing unit performs:

[0445] In the synthesis process, the video of the first area in the captured video is synthesized with the additional video to which the amount of distortion or the amount of peripheral light drop generated in the captured video is added.

[0446] (8) The information processing device according to (6) above, wherein:

[0447] The capture video processing unit performs:

[0448] In the synthesis process, the video of the first area in the captured video from which the distortion amount or the peripheral light amount drop amount is removed is synthesized with the additional video.

[0449] (9) The information processing device according to any one of (6) to (8) above, wherein:

[0450] The capture video processing unit performs:

[0451] In the synthesis process, synthesis is performed by changing a video ratio of the captured video to the additional video in stages in the boundary area between the first area and the second area.

[0452] (10) The information processing device according to any one of (6) to (9) above, wherein:

[0453] A rendered virtual video to be displayed on the display is used as the additional video.

[0454] (11) The information processing device according to any one of (1) to (10) above, comprising:

[0455] A marking processing unit performs a process of adding the boundary marker to the video to be displayed on the display.

[0456] (12) The information processing device according to (11) above, wherein:

[0457] The marker processing unit generates the boundary marker having a pattern that enables detection of an amount of distortion or an amount of peripheral light drop due to imaging by the camera, and adds the boundary marker to a video to be displayed on the display.

[0458] (13) The information processing device according to (11) or (12) above, wherein:

[0459] The marker processing unit performs a process of adding the boundary marker to a position corresponding to a peripheral portion of the display in a video to be displayed on the display.

[0460] (14) The information processing device according to any one of (11) to (13) above, wherein:

[0461] The label processing unit performs a process of recording generation information of the boundary label.

[0462] (15) The information processing device according to any one of (1) to (14) above, wherein:

[0463] The video displayed on the display is a virtual video obtained by rendering using a 3D model.

[0464] (16) An information processing device comprising:

[0465] A marking processing unit that performs a process of adding a boundary marker at a peripheral portion to be displayed in a display area of ​​a display to a video to be displayed on the display to be imaged by a camera.

[0466] (17) An information processing method comprising:

[0467] The position of a boundary marker displayed at a peripheral portion in a display area of ​​a display in a target frame of a captured video obtained by imaging a display by a camera relative to the captured video is determined. The display displays a video including the boundary marker.

[0468] (18) A program for causing an information processing device to execute the following steps:

[0469] The position of a boundary marker displayed at a peripheral portion in a display area of ​​a display in a target frame of a captured video obtained by imaging a display by a camera relative to the captured video is determined. The display displays a video including the boundary marker.

[0470] Reference Symbols List

[0471] 60 Boundary Marker

[0472] 70 Information processing device

[0473] 71CPU

[0474] 85 Video Processing Unit

[0475] 85a Output Video Processing Unit

[0476] 85b Marker Processing Unit

[0477] 111 First Area

[0478] 112 Second Area

[0479] 502 Camera

[0480] 505 LED Curtain Wall

[0481] 520 Rendering Engine

[0482] vB Background Video

[0483] vBC capture area video (inner frustum)

[0484] vC Capture Video

[0485] vCF final output video

[0486] vMK tagged images

[0487] vMKa Marker Extraction Image

[0488] vMKb marker position estimation image

[0489] vOT Add video outside the frame

[0490] vK Mask

[0491] Imk tag generation information

Claims

1. An information processing device, comprising: A captured video processing unit that performs the following determination processing: determining the position of a boundary mark displayed at a peripheral portion in a display area of ​​a target frame of a captured video obtained by imaging a display by a camera relative to the captured video, the display displaying a video including the boundary mark.

2. The information processing device according to claim 1, wherein: As the determination processing, the captured video processing unit performs a process of extracting the boundary marker from the target frame of the captured video based on the generation information of the boundary marker.

3. The information processing device according to claim 1, wherein: As the determination processing, the captured video processing unit performs processing for estimating the position of the boundary marker in the target frame of the captured video based on imaging information including position information of the camera at the time of imaging, imaging direction and viewing angle, and position information of the display.

4. The information processing device according to claim 1, wherein: The capture video processing unit performs: a first determination process of extracting the boundary marker from the target frame of the captured video; a second determination process of estimating a position of the boundary marker in the target frame of the captured video; and A process of calculating the distortion amount of the captured video based on a determination result of the first determination process and a determination result of the second determination process.

5. The information processing device according to claim 1, wherein: The capture video processing unit performs: a first determination process of extracting the boundary marker from the target frame of the captured video; a second determination process of estimating a position of the boundary marker in the target frame of the captured video; and A process of calculating an amount of decrease in the peripheral light amount of the captured video based on a determination result of the first determination process and a determination result of the second determination process.

6. The information processing device according to claim 1, wherein: The capture video processing unit performs: a process of generating a mask for distinguishing a first region within the target frame of the captured video and a second region outside the first region; and A synthesis process of synthesizing an additional video, which is a video corresponding to the second area, with the video of the first area in the captured video using the mask.

7. The information processing device according to claim 6, wherein: The capture video processing unit performs: In the synthesis process, the video of the first area in the captured video is synthesized with the additional video to which the amount of distortion or the amount of peripheral light drop generated in the captured video is added.

8. The information processing device according to claim 6, wherein: The capture video processing unit performs: In the synthesis process, the video of the first area in the captured video from which the distortion amount or the peripheral light amount drop amount is removed is synthesized with the additional video.

9. The information processing device according to claim 6, wherein: The capture video processing unit performs: In the synthesis process, synthesis is performed by changing a video ratio of the captured video to the additional video in stages in the boundary area between the first area and the second area.

10. The information processing device according to claim 6, wherein: A rendered virtual video to be displayed on the display is used as the additional video.

11. The information processing device according to claim 1, comprising: A marking processing unit performs a process of adding the boundary marker to the video to be displayed on the display.

12. The information processing device according to claim 11, wherein: The marker processing unit generates the boundary marker having a pattern that enables detection of the amount of distortion or the amount of peripheral light drop due to imaging by the camera, and adds the boundary marker to the video to be displayed on the display.

13. The information processing device according to claim 11, wherein: The marker processing unit performs a process of adding the boundary marker to a position corresponding to a peripheral portion of the display in a video to be displayed on the display.

14. The information processing device according to claim 11, wherein: The label processing unit performs a process of recording generation information of the boundary label.

15. The information processing device according to claim 1, wherein: The video displayed on the display is a virtual video obtained by rendering using a 3D model.

16. An information processing device, comprising: A marking processing unit that performs a process of adding a boundary marker at a peripheral portion to be displayed in a display area of ​​a display to a video to be displayed on the display to be imaged by a camera.

17. An information processing method performed by an information processing device, comprising: The position of a boundary marker displayed at a peripheral portion in a display area of ​​a display in a target frame of a captured video obtained by imaging a display by a camera, with respect to the captured video, is determined. The display displays a video including the boundary marker.

18. A program for causing an information processing device to execute the following processing: The position of a boundary marker displayed at a peripheral portion in a display area of ​​a display in a target frame of a captured video obtained by imaging a display by a camera, with respect to the captured video, is determined. The display displays a video including the boundary marker.

Images