Information processing device, information processing method, and program

Abstract

To provide an information processing device capable of dealing with even when a camera shooting range extends beyond a display when capturing a virtual image of the display with a camera.SOLUTION: The information processing device includes a shooting video processing unit that performs a series of determination processing to determine the position of a boundary marker in a target frame of the captured video on the video footage, which includes a boundary marker to be displayed at the periphery of the display area of the display photographed with a camera.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and relates to technology that can be used in the field of video production, for example. [Background technology]

[0002] 2. Description of the Related Art A known filming technique for producing video content such as movies is to have actors perform in front of a so-called green screen, and then later compose the performance with a background image. In recent years, instead of green screen filming, a filming system has been developed in which a background image is displayed on a large display in a studio, and the actors perform in front of it, allowing the actors and the background to be filmed together; this is known as Virtual Production, In-Camera VFX, or LED (Light Emitting Diode) Wall Virtual Production. Patent Document 1 below discloses a technology for a system that captures images of a performer or object acting in front of a background image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2020 / 0145644 Summary of the Invention [Problem to be solved by the invention]

[0004] Displaying the background image on a large display and then filming the performers and background image with a camera has many advantages over green screen filming, such as eliminating the need to separately composite the background image after filming and allowing performers and staff to visually understand the scene and judge the quality of their acting.

[0005] However, the range in which the background image is displayed is limited to the display frame. Depending on the camera position, shooting direction, or zoom angle, the captured range may extend outside the display frame. In such cases, in order to avoid wasting the footage, it is necessary to later detect the excess parts and correct the footage through compositing, etc., but this is an extremely time-consuming task. In addition, to avoid this, the cameraman must carefully capture the range of the display, which places restrictions on framing.

[0006] Therefore, the present disclosure proposes a technique that does not reduce the efficiency of video production even when shooting outside the frame of the display. [Means for solving the problem]

[0007] The information processing device according to the present technology includes a captured image processing unit that performs a determination process for determining the position of the boundary marker in a target frame of a captured image of a display that displays an image including a boundary marker on the periphery of the display area of ​​the display, the captured image being captured by a camera. The information processing device according to the present technology also includes a marker processing unit that performs processing to add a boundary marker to be displayed on the periphery of a display area of ​​the display to an image to be displayed on the display that is the subject of an image capture by a camera.

[0008] For example, a background image to be displayed on a display may be provided with a border marker that is displayed on the periphery of the display. The border marker allows the display frame to be accurately detected on the captured image. The image processing unit detects the border marker on the captured image. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is an explanatory diagram of an imaging system according to an embodiment of the present technology; [Figure 2]10A and 10B are explanatory diagrams of background images according to camera positions of the imaging system of the embodiment. [Figure 3] 10A and 10B are explanatory diagrams of background images according to camera positions of the imaging system of the embodiment. [Figure 4] FIG. 2 is an explanatory diagram of a video content production process according to an embodiment. [Figure 5] FIG. 1 is a block diagram of an imaging system according to an embodiment. [Figure 6] 10 is a flowchart of background image generation in the imaging system according to the embodiment. [Figure 7] FIG. 1 is a block diagram of an imaging system using multiple cameras according to an embodiment. [Figure 8] FIG. 1 is a block diagram of an information processing apparatus according to an embodiment. [Figure 9] 10A and 10B are explanatory diagrams of marker processing and captured video processing according to an embodiment. [Figure 10] FIG. 10 is an explanatory diagram of a background image displayed in the embodiment. [Figure 11] FIG. 2 is an explanatory diagram of a boundary marker according to an embodiment. [Figure 12] FIG. 10 is an explanatory diagram of a captured image including a boundary marker according to an embodiment. [Figure 13] FIG. 10 is an explanatory diagram of an extracted marker image according to the embodiment. [Figure 14] FIG. 10 is an explanatory diagram of a marker image whose position is estimated according to the embodiment. [Figure 15] FIG. 2 is an explanatory diagram of a mask according to an embodiment. [Figure 16] FIG. 10 is an explanatory diagram of a final output video according to the embodiment. [Figure 17] 10 is a flowchart of rendering including marker processing according to an embodiment. [Figure 18] 10 is a flowchart of a captured image process according to an embodiment. [Figure 19] 10A and 10B are diagrams illustrating calculation of a peripheral light falloff amount and a distortion amount according to an embodiment. [Figure 20] 10A and 10B are explanatory diagrams for comparison purposes for calculating the amount of peripheral light falloff and the amount of distortion according to the embodiment; [Figure 21]FIG. 10 is an explanatory diagram of mask generation according to an embodiment. [Figure 22] 10A and 10B are explanatory diagrams of mask blending according to an embodiment. [Figure 23] FIG. 10 is an explanatory diagram of a synthesis process according to an embodiment. [Figure 24] FIG. 1 is an explanatory diagram of pre-visualization according to an embodiment. [Figure 25] FIG. 1 is a block diagram of a configuration example of pre-visualization according to an embodiment. [Figure 26] 10 is a flowchart of an example of a process in pre-visualization according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The embodiments will be described below in the following order. <1. Filming system and content production> <2. Configuration of information processing device> <3. Marker processing and captured image processing> <4. Processing example> <5. Pre-visualization processing> <6. Summary and variations>

[0011] In this disclosure, "video" or "image" includes both still images and moving images. Furthermore, "video" does not only refer to the state in which it is displayed on a display, but may also refer comprehensively to video data that is not displayed on a display. The same applies to "image." For example, in the embodiments, the background image before being displayed on the display, the image captured by the camera, and the background image and captured image switched by the switcher are not actually displayed images but are video data, but for convenience they will be referred to as "background image," "captured image," etc.

[0012] <1. Filming system and video content production> A shooting system and video content production to which the technology of the present disclosure can be applied will be described. 1 is a schematic diagram of a photography system 500. This photography system 500 is a system for performing photography as a virtual production, and the figure shows some of the equipment arranged in a photography studio.

[0013] The filming studio is provided with a performance area 501 where performers 510 perform and other performances. Large display devices are placed at least on the back, left and right sides, and top of this performance area 501. The type of display device is not limited, but the figure shows an example in which an LED wall 505 is used as a large display device.

[0014] A single LED wall 505 is formed by arranging a plurality of LED panels 506 connected vertically and horizontally to form a large panel. The size of the LED wall 505 is not particularly limited, but it may be large enough or necessary to display a background when filming the performer 510.

[0015] A required number of lights 580 are placed at required positions, such as above or to the side of the performance area 501, to illuminate the performance area 501.

[0016] A camera 502 is placed near the performance area 501 for shooting, for example, movies or other video content. The camera 502 can be moved by a cameraman 512, and the camera's shooting direction, angle of view, etc. can also be controlled. Of course, it is also conceivable that the movement of the camera 502, the angle of view, etc. can be controlled by remote control. The camera 502 may also be designed to move and change the angle of view automatically or autonomously. For this reason, the camera 502 may be mounted on a platform or a moving body.

[0017] Camera 502 captures both performer 510 in performance area 501 and the image displayed on LED wall 505. For example, by displaying a landscape as background image vB on LED wall 505, it becomes possible to capture an image that is similar to the image of performer 510 actually being in the location of that landscape and performing.

[0018] An output monitor 503 is placed near the performance area 501. The video being shot by the camera 502 is displayed in real time on this output monitor 503 as a monitor video vM, allowing the director and staff producing the video content to check the video being shot.

[0019] In this way, the imaging system 500 that films the performance of the performer 510 in a film studio with the LED wall 505 as a backdrop has various advantages over green screen filming.

[0020] For example, when shooting on a green screen, it can be difficult for the actor to imagine the background or the situation of the scene, which can affect their performance. In contrast, by displaying the background image vB, it becomes easier for the actor 510 to act, improving the quality of their performance. It also makes it easier for the director and other staff to determine whether the actor 510's performance matches the background and the situation of the scene.

[0021] In addition, post-production is more efficient than with green screen shooting. This is because in some cases, so-called chromakey compositing is not necessary, and color correction and reflection compositing are not required. Even if chromakey compositing is required during shooting, it is only necessary to display green and blue images, so there is no need to add a physical background screen, which also contributes to efficiency.

[0022] When shooting on a green screen, the actors' bodies, costumes, and objects tend to have a green tint, which needs to be corrected. Also, when shooting on a green screen, if there are objects that reflect the surrounding scenery, such as glass, mirrors, or snow globes, it is necessary to generate and composite images of the reflections, which is a time-consuming process.

[0023] In contrast, when shooting with the shooting system 500 in Fig. 1, the green tint does not increase, so correction for this is not necessary. Also, by displaying the background image vB, reflections on actual objects such as glass are captured naturally and are shot, so there is no need to synthesize the reflected image.

[0024] Here, background image vB will be explained with reference to Figures 2 and 3. Even if background image vB is displayed on LED wall 505 and filming is performed together with performer 510, simply displaying background image vB will result in an unnatural background in the filmed video. This is because the background, which is actually three-dimensional and has depth, is displayed as two-dimensional background image vB.

[0025] For example, camera 502 can capture images of performer 510 in performance area 501 from a variety of directions and can also zoom. Performer 510 does not stand still in one place. Therefore, the actual appearance of the background of performer 510 should change depending on the position, shooting direction, and angle of view of camera 502, but such changes cannot be obtained with background image vB, which is a two-dimensional image. Therefore, background image vB is changed so that the background appears similar to how it actually appears, including parallax.

[0026] Figure 2 shows a camera 502 shooting a performer 510 from a position on the left side of the figure, and Figure 3 shows a camera 502 shooting a performer 510 from a position on the right side of the figure. In each figure, a shooting area image vBC is shown within a background image vB. The portion of the background image vB excluding the shooting area image vBC is called the "outer frustum," and the shooting area image vBC is called the "inner frustum." The background image vB described here refers to the entire image displayed as the background, including the shooting area image vBC (inner frustum).

[0027] The range of this shooting area image vBC (inner frustum) corresponds to the range actually captured by the camera 502 within the display surface of the LED wall 505. The shooting area image vBC is an image that expresses the scene that can be seen when the viewpoint is actually the position of the camera 502, depending on the position, shooting direction, angle of view, etc. of the camera 502.

[0028] Specifically, the shooting area video vBC is prepared using 3D background data, which is a 3D (three dimensions) model as the background, and is rendered sequentially in real time based on the viewpoint position of the camera 502 for the 3D background data. In reality, the range of the shooting area image vBC is set to be slightly wider than the range currently being shot by the camera 502. This is to prevent the image of the outer frustum from being captured due to a drawing delay when the shooting range changes slightly due to panning, tilting, zooming, etc. of the camera 502, and to avoid the effects of diffracted light from the image of the outer frustum. The image of the shooting area image vBC rendered in real time in this way is composited with the image of the outer frustum. The image of the outer frustum used in the background image vB may be rendered in advance based on 3D background data, or may be rendered in real time for each frame or intermittently, and the image of the shooting area image vBC (inner frustum) is incorporated into part of the image of the outer frustum to generate the overall background image vB. In some cases, the outer frustum image is also rendered every frame, just like the inner frustum image. However, here we will use a still image as an example, and in the following explanation we will mainly use the example of rendering only the first frame of the outer frustum image.

[0029] As a result, even if the camera 502 is moved back and forth or left and right, or zoomed, the background of the area being photographed along with the performer 510 will be captured as an image that corresponds to the changes in the viewpoint position and FOV (Field of View) that accompany the actual movement of the camera 502.

[0030] 2 and 3, the output monitor 503 displays a monitor image vM including a performer 510 and the background, which is the captured image. The background in this monitor image vM is the shooting area image vBC. In other words, the background included in the captured image is a real-time rendered image.

[0031] In this way, in the embodiment of the shooting system 500, the background image vB is not simply displayed two-dimensionally, but the background image vB including the shooting area image vBC is changed in real time so that an image similar to that obtained when an actual landscape is shot can be captured.

[0032] In addition, it is also possible to reduce the processing load on the system by rendering in real time only the shooting area image vBC, which is the range captured by the camera 502, rather than the entire background image vB displayed on the LED wall 505.

[0033] Here, we will explain the process of creating video content as a virtual production using the imaging system 500. As shown in Figure 4, the video content production process is roughly divided into three stages: asset creation ST1, production ST2, and post-production ST3.

[0034] Asset creation ST1 is the process of creating 3D background data for displaying background video vB. As mentioned above, background video vB is generated by rendering in real time using 3D background data during filming. For this purpose, 3D background data is created in advance as a 3D model.

[0035] Examples of methods for creating 3D background data include full computer graphics (CG), point cloud scanning, and photogrammetry.

[0036] Full CG is a method of creating 3D models using computer graphics. Of the three methods, it is the most labor-intensive and time-consuming, but it is best used when you want to use unrealistic images or images that are difficult to shoot in real life as background images (vB).

[0037] Point cloud data scanning is a method of generating a 3D model from point cloud data by measuring distance from a certain position using, for example, LiDAR, while also taking 360-degree images from the same position using a camera, and then overlaying the color data captured by the camera on top of the points measured by the LiDAR. 3D models can be created in a shorter time than with full CG. It is also easier to create high-resolution 3D models than with photogrammetry.

[0038] Photogrammetry is a photogrammetry technique that analyzes parallax information from 2D images of an object taken from multiple viewpoints to determine its dimensions and shape. It allows for the creation of 3D models in a short amount of time. In addition, when generating 3D data using photogrammetry, point cloud information obtained by LIDAR may also be used.

[0039] In Asset Creation ST1, for example, these methods are used to create 3D models that will serve as 3D background data. Of course, the above methods can also be used in combination. For example, parts of 3D models created using point cloud data scanning or photogrammetry can be created using CG and then combined.

[0040] Production ST2 is the process of filming in a film studio as shown in Figure 1. The elemental technologies used in this case include real-time rendering, background display, camera tracking, and lighting control.

[0041] Real-time rendering is a rendering process for obtaining the shooting area video vBC at each point in time (each frame of the background video vB) as explained in Figures 2 and 3. This involves rendering the 3D background data created in asset creation ST1 from a viewpoint that corresponds to the position of the camera 502 at each point in time.

[0042] In this way, real-time rendering is performed to generate the background image vB for each frame including the shooting area image vBC, and the generated background image vB is displayed on the LED wall 505.

[0043] Camera tracking is performed to obtain shooting information from the camera 502, and tracks the position information, shooting direction, angle of view, etc. of the camera 502 at each point in time. By providing this shooting information to the rendering engine in association with each frame, real-time rendering can be performed according to the viewpoint position of the camera 502, etc.

[0044] The shooting information is information that is linked or associated with the video as metadata. The shooting information is expected to include position information of the camera 502 at each frame timing, camera direction, angle of view, focal length, F-number (aperture value), shutter speed, lens information, and the like.

[0045] Lighting control refers to controlling the lighting conditions in the photography system 500, specifically controlling the light intensity, light color, lighting direction, etc. of the light 580. For example, lighting control is performed according to the time setting and location setting of the scene to be photographed.

[0046] Post-production ST3 indicates various processes that are carried out after shooting, such as image correction, image adjustment, clip editing, and image effects.

[0047] Image correction may involve color gamut conversion or color matching between cameras or materials. Image adjustments may include color adjustment, brightness adjustment, contrast adjustment, and the like. Clip editing may involve cutting clips, adjusting the order, adjusting the length of time, and so on. As a visual effect, CG images and special effect images may be combined.

[0048] Next, the configuration of the imaging system 500 used in production ST2 will be described. FIG. 5 is a block diagram showing the configuration of the imaging system 500 outlined in FIGS.

[0049] 5 includes the above-mentioned LED wall 505 made up of multiple LED panels 506, a camera 502, an output monitor 503, and a light 580. The photography system 500 further includes a rendering engine 520, an asset server 530, a sync generator 540, an operation monitor 550, a camera tracker 560, an LED processor 570, a lighting controller 581, and a display controller 590, as shown in FIG.

[0050] Each of the LED processors 570 is provided corresponding to one or more LED panels 506, and drives the corresponding one or more LED panels 506 to display images.

[0051] The sync generator 540 generates a synchronization signal for synchronizing the frame timing of the image displayed by the LED panel 506 with the frame timing of the image captured by the camera 502 , and supplies this to each LED processor 570 , the camera 502 , and the rendering engine 520 .

[0052] The camera tracker 560 generates shooting information by the camera 502 at each frame timing and supplies it to the rendering engine 520. For example, the camera tracker 560 detects, as one piece of shooting information, the position of the LED wall 505 or position information of the camera 502 relative to a predetermined reference position, and the shooting direction of the camera 502, and supplies this to the rendering engine 520. A specific detection method using the camera tracker 560 is to randomly place reflectors on the ceiling and detect the position from the reflected light of infrared light emitted from the camera tracker 560 attached to the camera 502. Another detection method is to estimate the self-position of the camera 502 by using gyro information mounted on the pan head or the body of the camera 502, or by image recognition of the video captured by the camera 502.

[0053] Furthermore, the camera 502 may supply the rendering engine 520 with shooting information such as the angle of view, focal length, F-number, shutter speed, and lens information.

[0054] The asset server 530 is a server that stores the 3D models created by the asset creation ST1, i.e., the 3D background data, on a recording medium and can read out the 3D models as needed. In other words, it functions as a DB (database) for the 3D background data.

[0055] The rendering engine 520 performs processing to generate the background image vB to be displayed on the LED wall 505. To do this, the rendering engine 520 reads out the necessary 3D background data from the asset server 530. The rendering engine 520 then generates an image of the outer frustum used in the background image vB by rendering the 3D background data as viewed from pre-specified spatial coordinates. The rendering engine 520 also uses the imaging information supplied from the camera tracker 560 and the camera 502 to identify the viewpoint position and the like relative to the 3D background data, and renders the imaging area image vBC (inner frustum).

[0056] Furthermore, the rendering engine 520 generates a background image vB as one frame of video data by combining the outer frustum with the shooting area image vBC, which dynamically changes in accordance with the movement of the camera 502. The rendering engine 520 then transmits the generated one frame of video data to the display controller 590.

[0057] The display controller 590 generates divided video signals nD by dividing one frame of video data into video portions to be displayed on each LED panel 506, and transmits the divided video signals nD to each LED panel 506. At this time, the display controller 590 may perform calibration according to individual differences / manufacturing errors in color development and the like between display sections. Note that these processes may be performed by the rendering engine 520 without providing the display controller 590. In other words, the rendering engine 520 may generate the divided video signal nD, perform calibration, and transmit the divided video signal nD to each LED panel 506.

[0058] Each LED processor 570 drives the LED panel 506 based on the divided video signal nD received, thereby displaying the entire background image vB on the LED wall 505. The background image vB includes the shooting area image vBC that is rendered according to the position of the camera 502 at that time, etc.

[0059] In this way, camera 502 can capture the performance of performer 510, including the background image vB displayed on LED wall 505. The image captured by camera 502 is recorded on a recording medium inside camera 502 or in an external recording device (not shown), and is also supplied in real time to output monitor 503 and displayed as monitor image vM.

[0060] The operation monitor 550 displays an operation image vOP for controlling the rendering engine 520. The engineer 511 can perform the necessary settings and operations related to the rendering of the background video vB while viewing the operation image vOP.

[0061] The lighting controller 581 controls the light emission intensity, light emission color, irradiation direction, etc. of the light 580. The lighting controller 581 may control the light 580 asynchronously with the rendering engine 520, for example, or may control the light 580 in synchronization with the shooting information and rendering processing. Therefore, the lighting controller 581 may control the light emission in response to instructions from the rendering engine 520 or a master controller (not shown). The light 580 may also be controlled by the rendering engine 520.

[0062] FIG. 6 shows an example of processing by the rendering engine 520 in the imaging system 500 configured as described above.

[0063] In step S10, the rendering engine 520 reads the 3D background data to be used this time from the asset server 530 and develops it in an internal work area. At this stage, an image to be used as the outer frustum may be generated.

[0064] Thereafter, the rendering engine 520 repeats the processes of steps S30 to S60 until it determines in step S20 that the display of the background video vB based on the read 3D background data has ended.

[0065] In step S30, the rendering engine 520 acquires shooting information from the camera tracker 560 and the camera 502. This allows the position and state of the camera 502 to be reflected in the current frame to be confirmed.

[0066] In step S40, the rendering engine 520 performs rendering based on the shooting information. That is, rendering is performed by identifying the viewpoint position relative to the 3D background data based on the position, shooting direction, or angle of view of the camera 502 to be reflected in the current frame. At this time, image processing that reflects the focal length, F-number, shutter speed, lens information, etc. can also be performed. This rendering makes it possible to obtain image data as the shooting area image vBC (inner frustum). The outer frustum can be generated in advance as a fixed image in step S10, or it can be generated for each frame in step S40.

[0067] In step S50, the rendering engine 520 performs processing to synthesize the outer frustum, which is the overall background image, with an image that reflects the viewpoint position of the camera 502, i.e., the shooting area image vBC. For example, this is processing to synthesize an image of the entire background rendered from a specific reference viewpoint with an image generated by reflecting the viewpoint of the camera 502. As a result, one frame of background image vB to be displayed on the LED wall 505, i.e., the background image vB including the shooting area image vBC, is generated.

[0068] The processing of step S60 is performed by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates divided video signals nD by dividing one frame of background video vB into videos to be displayed on the individual LED panels 506. Calibration may also be performed. Then, each divided video signal nD is sent to each LED processor 570.

[0069] By the above processing, the background image vB including the shooting area image vBC captured by the camera 502 is displayed on the LED wall 505 at each frame timing.

[0070] Although Fig. 5 shows only one camera 502, it is also possible to use multiple cameras 502 for filming. Fig. 7 shows an example of a configuration using multiple cameras 502a and 502b. The cameras 502a and 502b are each designed to be able to independently film the performance area 501. Furthermore, synchronization between the cameras 502a and 502b and the LED processors 570 is maintained by a sync generator 540.

[0071] Output monitors 503a and 503b are provided corresponding to the cameras 502a and 502b, and are configured to display the images captured by the corresponding cameras 502a and 502b as monitor images vMa and vMb, respectively.

[0072] Camera trackers 560a and 560b are provided corresponding to the cameras 502a and 502b, and detect the positions and shooting directions of the corresponding cameras 502a and 502b. Shooting information from the camera 502a and camera tracker 560a, and shooting information from the camera 502b and camera tracker 560b are transmitted to the rendering engine 520.

[0073] The rendering engine 520 can perform rendering to obtain the background image vB for each frame using the shooting information from either or both of the camera 502a and camera 502b.

[0074] Although FIG. 7 shows an example in which two cameras 502a and 502b are used, it is also possible to use three or more cameras 502 for image capture. However, when multiple cameras 502 are used and the shooting information of each camera is used to render and display the shooting area images vBC (inner frustum) corresponding to each camera 502, the shooting area images vBC may interfere with each other. For example, in an example using two cameras 502a and 502b as shown in FIG. 7, the shooting area image vBC corresponding to camera 502a is shown. However, when the image of camera 502b is used, the shooting area image vBC corresponding to camera 502b is also required. In this case, simply displaying the shooting area images vBC corresponding to each camera 502a and 502b will result in interference with each other. For this reason, some ingenuity is required regarding the display of the shooting area images vBC.

[0075] <2. Configuration of information processing device> Next, an example of the configuration of an information processing device 70 that can be used in asset creation ST1, production ST2, and post-production ST3 will be described with reference to FIG. The information processing device 70 is a device such as a computer that is capable of information processing, particularly video processing. Specific examples of the information processing device 70 include personal computers, workstations, mobile terminal devices such as smartphones and tablets, and video editing devices. The information processing device 70 may also be a computer configured as a server device or a computing device in cloud computing.

[0076] In the case of this embodiment, specifically, the information processing device 70 can function as a 3D model creation device that creates a 3D model in the asset creation ST1. The information processing device 70 can also function as a rendering engine 520 and an asset server 530 that constitute the imaging system 500 used in production ST2. The information processing device 70 can also function as a video editing device that performs various video processes in post-production ST3.

[0077] Furthermore, the information processing device 70 can also function as an information processing device (rendering engine 31) that generates a simulation video vSM in pre-visualization, which will be described later with reference to FIG. 25 and the like.

[0078] 8 executes various processes in accordance with programs stored in a ROM 72 or a nonvolatile memory unit 74 such as an EEPROM (Electrically Erasable Programmable Read-Only Memory), or programs loaded from a storage unit 79 to a RAM 73. The RAM 73 also stores data necessary for the CPU 71 to execute various processes as appropriate.

[0079] The video processing unit 85 is configured as a processor that performs various types of video processing, such as 3D model generation processing, rendering, DB processing, video processing including color and brightness adjustment processing, video editing processing, video analysis and detection processing, or a combination of these.

[0080] In particular, FIG. 8 shows a captured image processing section 85a and a marker processing section 85b as processing functions in the image processing section 85. The captured image processing unit 85a and the marker processing unit 85b may be provided in the information processing device 70 as the rendering engine 520, for example. The marker processing unit 85b is a function that performs processing to add boundary markers 60 (see FIG. 9, etc.), which will be described later, to the rendered background image vB. As a result, the background image vB displayed on the LED wall 505 includes an image of the boundary marker 60. The captured video processing unit 85a has a function for performing a determination process to determine the position of the boundary marker 60 in a target frame of the captured video vC, which is a captured background video vB including the boundary marker 60 and displayed on the LED wall 505. The target frame refers to a frame that is the target of captured video processing at a certain point in time. For example, each frame that makes up the captured video vC as a moving image is sequentially set as the target frame. Note that the position in the target frame of the captured video vC (for example, the position of the boundary marker 60) refers to the position in the image of that frame, and does not include positions in frames that are different in time.

[0081] The processing of the video processing unit 85 by the functions of the captured video processing unit 85a and the marker processing unit 85b will be described in detail later. Note that the image processing unit 85 in the information processing device 70 may have a captured image processing unit 85a but not a marker processing unit 85b, or may have a marker processing unit 85b but not a captured image processing unit 85a.

[0082] This video processing unit 85 can be realized by, for example, a CPU separate from the CPU 71, a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), an AI (Artificial Intelligence) processor, or the like. The video processing unit 85 may be provided as a function within the CPU 71 .

[0083] The CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74, and video processing unit 85 are interconnected via a bus 83. To this bus 83, the input / output interface 75 is also connected.

[0084] An input unit 76 consisting of operators and operation devices is connected to the input / output interface 75. For example, the input unit 76 may be various operators and operation devices such as a keyboard, a mouse, keys, a trackball, a dial, a touch panel, a touchpad, or a remote controller. An operation by the user is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71. A microphone may also be used as the input unit 76. Voice uttered by the user may also be input as operation information.

[0085] Furthermore, the input / output interface 75 is connected integrally or separately to a display unit 77 made up of an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, etc., and an audio output unit 78 made up of a speaker, etc. The display unit 77 is a display unit that displays various types of information, and is configured by, for example, a display device provided in the housing of the information processing device 70, or a separate display device connected to the information processing device 70, or the like. The display unit 77 displays various images, operation menus, icons, messages, etc., on the display screen based on instructions from the CPU 71, that is, displays a GUI (Graphical User Interface).

[0086] The input / output interface 75 may be connected to a storage unit 79 and a communication unit 80 that are configured from a hard disk drive (HDD) or solid-state memory.

[0087] The storage unit 79 can store various data and programs. A database can also be configured in the storage unit 79. For example, when the information processing device 70 functions as an asset server 530 or a rendering engine 31 described later, the storage unit 79 can be used to create a DB for storing a group of 3D background data.

[0088] The communication unit 80 performs communication processing via a transmission path such as the Internet, and communication with various devices such as an external DB, editing device, and information processing device via wired / wireless communication, bus communication, and the like. For example, when the information processing device 70 functions as a rendering engine 520 or the rendering engine 31 described below, the communication unit 80 can access a DB serving as an asset server 530 and receive shooting information from the camera 502 and the camera tracker 560. In addition, in the case of the information processing device 70 used in post-production ST3, it is also possible to access the DB serving as the asset server 530 via the communication unit 80.

[0089] A drive 81 is also connected to the input / output interface 75 as required, and a removable recording medium 82 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately mounted thereon. Drive 81 allows video data, various computer programs, and the like to be read from removable recording medium 82. The read data is stored in storage unit 79, and the video and audio contained in the data are output on display unit 77 and audio output unit 78. Furthermore, computer programs and the like read from removable recording medium 82 are installed in storage unit 79 as needed.

[0090] In this information processing device 70, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 80 or via a removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, etc.

[0091] <3. Marker processing and captured image processing> The marker processing and captured image processing according to the embodiment will be described. Figure 9 shows the marker processing and captured footage processing during the virtual production process.

[0092] For example, the marker generation process P2 and the marker addition process P3 in FIG. 9 are performed by the function of the marker processing unit 85b in the video processing unit 85 in FIG. The marker generation process P2 and the marker addition process P3 are executed as processes within the rendering engine 520, for example, during shooting in virtual production, that is, when the background video vB is displayed on the LED wall 505.

[0093] Furthermore, the captured image processing (P11 to P19) is performed by the function of the captured image processing unit 85a in the image processing unit 85 of FIG. The captured video processing (P11 to P19) may also be performed in real time during shooting, for example, within the rendering engine 520 that inputs the captured video vC, or in another device, or may be performed at the post-production stage ST3.

[0094] Here, as an example, the following description will be given assuming that the rendering engine 520 performs marker generation processing and captured image processing in real time during shooting.

[0095] For example, the video processing unit 85 of the information processing device 70 as the rendering engine 520 generates the background video vB as the rendering P1 in FIG.

[0096] An example of background image vB is shown in Fig. 10A. Note that the rendering engine 520 generates background image vB as shown in Fig. 10A, but in this example, the image is a range wider than the display area of ​​the LED wall 505. For example, the range of actual display area 100 indicated by the dashed line is the image that is actually displayed on the LED wall 505. The rendering engine 520 may generate background image vB with a range equivalent to the display area of ​​the LED wall 505, but by generating background image vB with a wider range as shown in Fig. 10A, this background image vB can be used for compositing in subsequent captured image processing. The background image vB also includes the shooting area image vBC (inner frustum).

[0097] The video processing unit 85 of the rendering engine 520 performs the marker generation process P2 shown in FIG. 9 using the function of the marker processing unit 85b. In the marker generation process P2, the rendered background video vB and information on its actual display area 100 are input to generate a marker image vMK.

[0098] 11A and 11B show examples of the marker image vMK. The example of the marker image vMK in FIG. 11A is an example in which a pattern as the boundary marker 60 surrounds the periphery. The example of the marker image vMK in FIG. 11B is an example in which patterns as boundary markers 60 are arranged on the left, right, and top peripheral edges.

[0099] The boundary marker 60 is intended to enable detection of any protrusion of the imaging range of the camera 502 capturing the LED wall 505 beyond the LED wall 505. For this reason, it is preferable to form boundary markers 60 around the entire periphery as shown in FIG. 11A. However, the performance area 501 of the imaging system 500 shown in FIG. 1 may be provided with a set of real scenes that are connected to the content of the background video vB, in which case it does not matter if the imaging range of the camera 502 extends below the LED wall 505. In such cases, a boundary marker 60 such as that shown in FIG. 11B may be used.

[0100] The boundary marker 60 as shown in FIGS. 11A and 11B has a boundary line 61 and a detection pattern 62. The boundary line 61 is a line formed in, for example, a frame shape to clearly indicate the edge of the LED wall 505 . The detection pattern 62 is a vertical and horizontal grid pattern provided for determining the amount of peripheral light falloff and spatial distortion caused by the lens of the camera 502. Note that the pattern shape is an example and is not limited to this. The pattern shape of such boundary markers 60 itself may be stored in advance in the system.

[0101] Then, in the marker generation process P2 shown in FIG. 9, the rendering engine 520 sets the brightness, color, and size of the boundary marker 60 to generate a marker image vMK. The brightness and color of the boundary marker 60 are set with reference to the brightness and color of the background video vB so that the boundary marker 60 stands out within the background video vB. For example, if the background video vB shows a bright scene, the brightness of the boundary marker 60 is lowered, and if the background video vB shows a dark scene, the brightness of the boundary marker 60 is increased. The color is also set according to the color of the background video vB. Furthermore, the rendering engine 520 sets the size of the boundary marker 60 so that the boundary marker 60 is displayed on the periphery of the range of the actual display area 100. Note that the lines of the actual display area 100 are added for the purpose of explanation, and the background image vB does not actually include such lines indicating the actual display area 100.

[0102] Next, the video processing unit 85 of the rendering engine 520 performs a marker addition process P3 using the function of the marker processing unit 85b. As the marker addition process P3, the rendering engine 520 performs a process of adding a boundary marker 60 by combining the marker image vMK with the rendered background video vB. That is, the pixel value at the position corresponding to the boundary marker 60 in the original background image vB is replaced with the pixel value of the boundary marker 60.

[0103] 10B shows background image vB to which boundary markers 60 have been added. Boundary markers 60 have been added to positions that will become the periphery of the range of actual display area 100 in background image vB in FIG. 10A.

[0104] The background image vB to which such boundary markers 60 have been added is displayed on the LED wall 505. Therefore, on the LED wall 505, boundary markers 60 are displayed on the periphery as shown in FIG.

[0105] Camera 502 captures background image vB of LED wall 505 and performer 510. The background image vB captured by camera 502 is within the range of shooting area image vBC. In FIG. 9, the shooting area image vBC is within the LED wall 505, and the shooting range of camera 502 does not extend beyond the LED wall 505. Furthermore, in the illustrated example, shooting area image vBC does not overlap with boundary marker 60.

[0106] Since the shooting area image vBC roughly matches the shooting range of the camera 502, if the shooting range of the camera 502 extends beyond the LED wall 505, part of the shooting area image vBC will extend beyond the LED wall 505.

[0107] Next, the captured image processing for the image vC captured by the camera 502 will be described. For example, the rendering engine 520 receives the captured video vC captured by the camera 502, performs captured video processing on each frame of the captured video vC, and outputs the final output video vCF as the final captured video. In other words, the rendering engine 520 sequentially performs captured video processing on each frame of the captured video vC as the target frame.

[0108] Specifically, the rendering engine 520, using the function of the captured image processing unit 85a, performs marker extraction processing P11, marker position estimation processing P12, peripheral light falloff and distortion detection processing P13, mask generation processing P14, peripheral light falloff and distortion addition processing P15, peripheral light falloff and distortion removal processing P16, selection processing P17 and P18, and synthesis processing P19 shown in Figure 9 on the target frame.

[0109] Note that the processing by the rendering engine 520 is just an example, and the following captured image processing may be performed by another device or by a processor inside the camera 502.

[0110] As shown in Fig. 9, the camera 502 outputs the captured video vC (output process P10). The camera 502 simultaneously captures the captured area video vBC in the background video vB displayed on the LED wall 505, and the actual scenery of the performer 510 and the like. Depending on the position, shooting direction, and angle of view of the camera 502, the captured video vC may also include the area outside the LED wall 505. In this case, the view outside the frame of the LED wall 505 and the boundary markers 60 displayed on the periphery of the LED wall 505 are also captured. Alternatively, even if the photographing range of the camera 502 does not extend outside the frame of the LED wall 505, the area including the boundary marker 60 may be photographed.

[0111] An example of the captured image vC is shown in Fig. 12. This is the case when the capture range of camera 502 extends above, to the left, and to the right of LED wall 505. For example, if camera 502 is set to a wide angle that exceeds the width of LED wall 505 and the capture direction is set too far upward, the captured image vC shown in Fig. 12 is obtained. The shot image vC includes the shot area image vBC in the background image vB, the performer 510, the boundary marker 60, and the surrounding scenery of the LED wall 505 (shown in black in the figure).

[0112] For example, the rendering engine 520 performs the marker extraction process P11 of FIG. 9 on the target frame of such a captured video vC.

[0113] To extract the markers, the rendering engine 520 refers to the information (marker generation information Imk) from the marker generation process P2. That is, it acquires information on the brightness and color of the boundary marker 60. The marker generation information Imk is information on the brightness, color, pattern shape, size, etc. of the boundary marker 60 that was set during the marker generation process P2. The rendering engine 520 sets brightness and color threshold levels based on this marker generation information Imk, compares each pixel value of the captured video vC, and extracts the boundary marker 60 from the captured video vC. For example, a marker-extracted image vMKa such as that shown in Figure 13 is generated from the captured video vC of Figure 12.

[0114] The rendering engine 520 also performs a marker position estimation process P12 shown in Fig. 9. This is a process for estimating, by calculation, the position in the target frame within the captured video vC at which the boundary marker 60 will be displayed. If the relative positional relationship between the LED wall 505 and the camera 502, the angle of view, and the size of the image sensor for the target frame of the current captured video vC are known, the position of the boundary marker 60 within the captured video vC can be calculated.

[0115] Information about the placement position and angle of the LED wall 505 is known in advance within the system; for example, the rendering engine 520 can store this information as information about the LED wall 505. The size of the image sensor of the camera 502 is also known and set in advance for calculation. The position, shooting direction, and zoom angle of the camera 502 are input as shooting information from the camera tracker 560 (see FIG. 5). Therefore, by acquiring shooting information from the camera tracker 560 for each frame of the captured video vC, the rendering engine 520 can estimate, through real-time calculation, where in the captured video vC the boundary marker 60 will be displayed. The rendering engine 520 then generates a marker position estimation image vMKb showing the estimated boundary marker 60, as shown in FIG.

[0116] Here, the position of the boundary marker 60 in the marker position estimation image vMKb in Fig. 14 is an ideal position without the influence of lens distortion, whereas the position of the boundary marker 60 in the marker extraction image vMKa in Fig. 13 is a position affected by the lens distortion of the camera 502. By using this marker extraction image vMKa and the marker position estimation image vMKb, the amount of distortion caused by the lens and the amount of peripheral light falloff can be detected.

[0117] The rendering engine 520 performs peripheral light falloff and distortion detection processing P13. In this process, the rendering engine 520 calculates the amount of vignetting and distortion using the marker extraction image vMKa and the marker position estimation image vMKb. For example, the calculation process involves preparing multiple images in which different amounts of vignetting have been added to the marker position estimation image vMKb, calculating the difference between the marker extraction image vMKa and the marker position estimation image vMKb, and selecting the amount of vignetting that minimizes the sum of the absolute values ​​of the differences between each pixel. A similar method can also be used for the amount of distortion. This process, which will be described in more detail later, determines the amount of vignetting and distortion in the captured video vC.

[0118] The rendering engine 520 performs a mask generation process P14. The rendering engine 520 generates the mask vK in FIG. 15 using an image obtained by adding a distortion amount to the marker extraction image vMKa or the marker position estimation image vMKb. The mask vK is information indicating a first area 111 (hatched area) that uses the captured video vC in the composition process P19, and a second area 112 that combines a CG image (for example, background video vB). The first area 111 is an area inside the boundary marker 60 in the target frame of the captured video vC. The second area 112 is an area outside the first area 111, including the boundary marker 60.

[0119] In this mask vK, all values ​​corresponding to the pixels in the first region 111 are set to "1" and all values ​​corresponding to the pixels in the second region 112 are set to "0". Alternatively, the mask vK may be formed by providing a boundary area (a blend area 111a described later) between the first area 111 and the second area 112, and generating an α channel image with values ​​that gradually transition from "1" to "0." Details will be described later.

[0120] The rendering engine 520 performs peripheral light falloff / distortion addition processing P15 or peripheral light falloff / distortion removal processing P16.

[0121] The peripheral light falloff / distortion addition process P15 is a process that adds peripheral light falloff and distortion amounts similar to those of the image vC captured by camera 502 to the background image vB using the peripheral light falloff and distortion amounts calculated in the peripheral light falloff / distortion detection process P13. The vignetting and distortion removal process P16 uses the vignetting and distortion amounts found in the vignetting and distortion detection process P13 to apply the inverse characteristics to the captured image vC, thereby removing vignetting and distortion from the captured image vC.

[0122] Which of these processes is performed is determined by selection processes P17 and P18. The selection process P17 selects whether to output the captured image vC as is (referred to as "Course A" for the sake of explanation) or to output the captured image vC after performing the peripheral light falloff and distortion removal process P16 (referred to as "Course B" for the sake of explanation).

[0123] Selection process P18 selects between performing peripheral light falloff and distortion addition process P15 on the background image vB and outputting it (course A), and outputting the background image vB as is (course B).

[0124] If Course A is selected in selection processes P17 and P18, a synthesis process P19 is performed on the captured image vC and the background image vB to which peripheral light falloff and distortion have been added. In this case, the images are synthesized, taking advantage of the distortion and light falloff caused by the camera 502. If Course B is selected in selection processes P17 and P18, a synthesis process P19 is performed on the captured image vC from which peripheral light falloff and distortion have been removed and the background image vB. In this case, images without distortion or light falloff caused by the camera 502 are synthesized. Course A or Course B can be selected based on the creator's creative intent.

[0125] In the synthesis process P19, the rendering engine 520 uses the mask vK to synthesize the captured image vC selected in course A or course B with the background image vB, and generates the final output image vCF as shown in Figure 16.

[0126] This final output video vCF is an image in which the portion of the background video vB corresponding to the second region 112 outside the dashed line BD is added as the outside-frame added video vOT to the photographed video vC corresponding to the first region 111 of the mask vK shown inside the dashed line BD. In other words, a part of the CG video generated as the background video vB is added as the outside-frame added video vOT to the in-plane region outside the boundary marker 60 including the boundary marker 60 in the photographed video vC.

[0127] By performing the above-described captured image processing, for example, by the rendering engine 520, even if the captured range extends beyond the LED wall 505, the background image vB can be added to the extending area to prevent the extension from occurring.

[0128] The above captured image processing (P11 to P19) is executed in real time during image capture by the rendering engine 520, the camera 502, or another device having the configuration of the information processing device 70. Alternatively, the above-described captured video processing (P11 to P19) may be executed in the information processing device 70, which serves as a video editing device, in post-production ST3.

[0129] Furthermore, some of the captured image processing (P11 to P19) may be performed in real time during shooting, and some may be performed in post-production ST3. For example, the marker extraction process P11 and the marker position estimation process P12 may be performed during shooting, and the others may be performed in post-production ST3. Alternatively, the marker extraction process P11 through the selection processes P17 and P18 may be performed during shooting, and the synthesis process P19 may be performed in post-production ST3. Other examples are also possible.

[0130] If part or all of the captured video processing (P11 to P19) is performed in post-production ST3, it is sufficient that the information required in post-production ST3 is stored. For example, if all of the captured video processing (P11 to P19) is performed in post-production ST3, the captured video vC, background video vB, marker generation information Imk, shooting information (position of camera 502 for each frame, shooting direction, angle of view), and placement information of LED wall 505 are stored and can be acquired by the information processing device 70 that performs post-production ST3.

[0131] Furthermore, for example, when only the compositing process P19 is performed in post-production ST3, it is sufficient that the information processing device 70 performing post-production ST3 can acquire the shot video vC of course A or course B, the background video vB, and the mask vK.

[0132] In addition, in the example of FIG. 9, the background image vB is used as the source of the outside-frame additional image vOT, but the CG image that becomes the outside-frame additional image vOT may be generated separately from the background image vB.

[0133] 10 shows an example in which background image vB is generated to include the area outside actual display area 100. In this case, background image vB can be used for the shooting range that extends beyond LED wall 505, but if it extends even further, or if background image vB is an image of only actual display area 100, it is possible to generate a CG image that will be an outside-frame additional image vOT separately from background image vB.

[0134] <4. Processing example> An example of processing by the information processing device 70 including marker processing and captured video processing as shown in Fig. 9 will be described below. Here, the description will be made assuming that the rendering engine 520 performs marker processing and captured video processing in real time during shooting, for example.

[0135] Fig. 17 shows an example in which marker processing is added to the processing during rendering explained in Fig. 6. Steps S10 to S50 are the same as those in Fig. 6, so redundant explanations will be avoided.

[0136] In the processing up to step S50, a frame of the background image vB including the shooting area image vBC (inner frustum) is generated. The rendering engine 520 performs marker processing on this background video vB from step S51 to step S55. Note that "P2" and "P3" are added to indicate correspondence with the marker processing in FIG.

[0137] In step S51, the rendering engine 520 calculates the feature amount of the background image vB, such as brightness, color, and edge.

[0138] In step S52, the rendering engine 520 sets the brightness and color of the boundary marker 60 so that it can be easily distinguished from the background according to the features of the background image vB, and in step S53 generates a marker image vMK having the boundary marker 60 according to the settings.

[0139] For example, if the background image vB shows a bright scene, the brightness of the boundary marker 60 is reduced, and if the background image vB shows a dark scene, the brightness of the boundary marker 60 is increased. For example, if the background is blue, use a color with equal brightness that stands out. This will prevent flare caused by using a bright border marker 60 in a dark scene. Also, for example, if the background image vB has many vertical lines as edges, the boundary marker 60 is made to have a pattern with many horizontal line components.

[0140] The boundary markers 60 may be generated by combining brightness, color, and edge features. Furthermore, the boundary marker 60 according to the feature amount may be non-uniform, for example, the right half being bright and the left half being dark. In other words, the boundary marker 60 may change according to the features of the background of the area where the boundary marker 60 is displayed.

[0141] In step S54, the rendering engine 520 stores the marker generation information Imk for the boundary marker 60 of the current frame in the storage medium, so that the marker generation information Imk can be read out in relation to the current frame at a later time.

[0142] In step S55, the rendering engine 520 performs processing to add the boundary marker 60 to the background image vB.

[0143] Then, in step S60, the rendering engine 520 outputs the background image vB including the boundary marker 60 as shown in FIG. 10B, and displays it on the LED wall 505.

[0144] 18 shows an example of captured image processing. "P11" to "P19" are added to indicate correspondence with the captured image processing in FIG.

[0145] The rendering engine 520 acquires information about the LED wall 505 in step S100, for example, prior to shooting. That is, the information is about the arrangement position and angle of the LED wall 505 for the marker position estimation process P12.

[0146] After the start of shooting, the rendering engine 520 performs the processes of steps S101 to S113 for each frame of the shot video vC.

[0147] In step S101, the rendering engine 520 obtains the marker generation information Imk for the currently processed frame. In step S102, the rendering engine 520 acquires shooting information for the current frame to be processed, in particular, information on the position, shooting direction, and angle of view of the camera 502 obtained by the camera tracker 560.

[0148] In step S103, the rendering engine 520 performs the marker position estimation process P12 using the information on the LED wall 505, the shooting information, and the marker generation information Imk (pattern shape, size), and generates a marker position estimation image vMKb.

[0149] Furthermore, in step S104, the rendering engine 520 sets a threshold based on the marker generation information Imk (brightness, color), and performs a marker extraction process P11 from the captured video vC to generate a marker-extracted image vMKa.

[0150] The order of steps S103 and S104 may be reversed. In order to reduce the real-time processing load, step S103 may be executed only when proceeding from step S105 to step S106.

[0151] In step S105, the rendering engine 520 determines whether or not part or all of the boundary marker 60 has been photographed for the current frame. By the process of step S104, it is possible to determine whether or not a part or all of the boundary marker 60 is reflected in the current frame.

[0152] If the boundary marker 60 is not reflected in the current frame, there is no need to perform captured image processing according to the boundary marker 60. Therefore, the rendering engine 520 proceeds from step S105 to step S113, and outputs the frame of the captured image vC as the final captured output image vCF. For example, it may be stored in a storage medium as one frame of the captured output image, or transmitted to an external device.

[0153] On the other hand, if the boundary marker 60 is captured in the current frame, the captured image processing requires a synthesis process P19 to replace the outer image including the boundary marker 60. Therefore, the rendering engine 520 proceeds from step S105 to step S106, and performs a peripheral light falloff and distortion detection process P13.

[0154] FIG. 19 shows a specific example of the peripheral light falloff and distortion detection process P13. The rendering engine 520 calculates the amount of peripheral light falloff and the amount of distortion using the marker extraction image vMKa and the marker position estimation image vMKb.

[0155] First, the rendering engine 520 performs binarization processing P21 on the marker-extracted image vMKa. The marker-extracted image vMKa is not in a binary state because it is an image in which the boundary markers 60 are extracted from the captured video vC. This image is binarized to remove peripheral light loss.

[0156] Furthermore, as a lens distortion addition process P22, the rendering engine 520 generates an image based on the marker position estimation image vMKb, to which several levels of lens distortion, from small to large, have been added. For example, as shown in Fig. 20, n images are generated, image vMKb1 to image vMKb(n). The amount of distortion of these images vMKb1 to vMKb(n) varies in stages.

[0157] Next, the rendering engine 520 performs a difference calculation P23. This is a process of calculating the difference between each of the images vMKb1 to vMKb(n) and the binarized marker-extracted image vMKa.

[0158] The rendering engine 520 then performs distortion amount determination processing P24. This is processing in which the image vMKb1 to vMKb(n) that has the smallest difference from the binarized marker-extracted image vMKa is determined, and the amount of distortion added to that image is determined as the amount of distortion in the captured video vC of the current frame. For example, if image vMKb2 is the image that has the smallest difference from the binarized marker-extracted image vMKa, the amount of distortion added to image vMKb2 becomes the amount of distortion to be detected. In this way, the amount of distortion in the captured image vC is detected.

[0159] Next, the rendering engine 520 performs lens distortion addition processing P25 on the marker position estimation image vMKb to detect the amount of peripheral light falloff. That is, the marker position estimation image vMKb is made to have lens distortion similar to the captured video vC. Note that the image selected in the distortion amount determination process P24 (e.g., image vMKb2) may be used as is without performing this lens distortion addition process P25, because it is an image to which the same lens distortion amount has been added to the original marker position estimation image vMKb.

[0160] In the vignetting addition process P26, an image with vignetting added at various levels, from small to large, is generated based on the marker position estimation image vMKb (or a similar image vMKb2) to which the detected distortion has been added. For example, as shown in FIG. 20, m images are generated, designated as images vMKb11 to vMKb(m). These images vMKb11 to vMKb(m) each have a different level of vignetting.

[0161] Next, the rendering engine 520 performs a difference calculation P27. This is a process of calculating the difference between each of the images vMKb11 to vMKb(m) and the marker-extracted image vMKa.

[0162] The rendering engine 520 then performs peripheral light falloff determination processing P28. This is a process in which the image with the smallest difference from the marker-extracted image vMKa is determined from among images vMKb11 to vMKb(m), and the amount of peripheral light falloff added to that image is determined as the amount of peripheral light falloff in the captured video vC of the current frame. For example, if image vMKb12 is the image with the smallest difference from the marker-extracted image vMKa, the amount of peripheral light falloff added to image vMKb12 becomes the amount of peripheral light falloff to be detected. This completes the detection of the amount of peripheral light falloff in the captured video vC.

[0163] After performing the peripheral light falloff and distortion detection process P13 as shown in Figure 19, the rendering engine 520 branches the process in step S107 of Figure 18 depending on whether Course A or Course B is set by the selection processes P17 and P18.

[0164] In the case of Course A, the rendering engine 520 proceeds to step S107 to perform mask generation processing P14, and also performs peripheral light falloff and distortion addition processing P15 on the background image vB in step S110. In the case of Course B, the rendering engine 520 proceeds to step S109 to perform mask generation processing P14, and also performs peripheral light falloff and distortion removal processing P16 on the captured video vC in step S111.

[0165] The mask generation process P14 in step S108 will be described with reference to FIG. As for the mask vK, as described above, the first region 111 may be set to "1" and the second region 112 to "0", but Fig. 21 shows a processing example for blending at the boundary between the first region 111 and the second region 112. Note that Fig. 21 shows a processing example when step S108, that is, course A, is selected.

[0166] The mask vK can be generated based on either the marker extraction image vMKa or the marker position estimation image vMKb. In the selection process P33 in FIG. 21, the rendering engine 520 selects whether the mask vK is to be generated based on the marker extraction image vMKa or the marker position estimation image vMKb.

[0167] When the marker position estimation image vMKb is selected, the rendering engine 520 adds the amount of distortion detected in step S106 to the marker position estimation image vMKb through distortion addition processing P31. When the marker extraction image vMKa is selected, the rendering engine 520 binarizes the marker extraction image vMKa through binarization processing P32.

[0168] The rendering engine 520 performs a value setting process P34 on the marker position estimated image vMKb to which this distortion has been added or the binarized marker extraction image vMKa. In the value setting process P34, a mask vK is generated in which the inner first region 111 has α channel=1 and the outer second region 112 has channel=0, as shown in FIG. 22A. In Figure 22A, the boundary marker 60 is shown with a dashed line, but this is added to indicate that the first region 111 is an area inside the boundary marker 60, and the boundary marker 60 is not added to the mask vK.

[0169] Next, the rendering engine 520 performs a slope addition process P35. 22B, a blending area 111a is set in the boundary between the first area 111 and the second area 112. In this blending area 111a, a slope is added to the alpha channel so that the alpha channel changes between "0" and "1" in the direction away from the position where it contacts the second area 112. For example, the value of the alpha channel can be changed from the outside to the inside as "0.1", "0.2", "0.9", etc. The rendering engine 520 outputs the mask vK with the slope added in this way.

[0170] After the mask generation process P14 is performed in step S108 as described above, the background image vB is subjected to peripheral light falloff and distortion addition process P15 in step S110. Since Course A is selected, a synthesis process P19 is performed using the captured image vC, the background image vB with peripheral light falloff and distortion added, and the mask vK generated in step S108.

[0171] Note that when course B is selected and the process proceeds from step S107 to step S109, the process of providing the blend region 111a is also performed in the mask generation process P14, as in Figure 21. However, in this case, when generating the mask vK based on the marker position estimation image vMKb, it is not necessary to add distortion to the marker position estimation image vMKb. The value setting process P34 and slope addition process P35 can be performed using the marker position estimation image vMKb as is. Furthermore, when generating a mask vK based on the marker-extracted image vMKa, the marker-extracted image vMKa is binarized and then subjected to a process of removing distortion, followed by a value setting process P34 and a slope addition process P35.

[0172] After the mask generation process P14 is performed in step S109 as described above, the captured image vC is subjected to peripheral light falloff and distortion removal process P16 in step S111. Since Course B is selected, a synthesis process P19 is performed using the captured image vC from which peripheral light falloff and distortion have been removed, the background image vB, and the mask vK generated in step S109.

[0173] In step S112, the rendering engine 520 performs a synthesis process P19 of the shot video vC and the background video vB. An example of the synthesis process P19 is shown in FIG.

[0174] In the synthesis process P19, the rendering engine 520 performs a multiplication process P41 in which pixel values ​​of a frame of the captured video vC are multiplied by the α value of the mask vK. As described above, since the α channel of the first region 111 is "1," the pixel values ​​in the captured video vC are maintained as they are. However, the pixels in the blending region 111a are multiplied by a coefficient that is set in a slope from "1" to "0" from the inside to the outside depending on the position of the pixel. Since the α channel of the second region 112 is "0," the pixel values ​​in the captured video vC are set to "0."

[0175] The rendering engine 520 also performs an α value inversion process P42 on the mask vK, and then performs a multiplication process P43 on the background video vB. As a result of the alpha value inversion process P42, the alpha channel in the first region 111 becomes "0" and the alpha channel in the second region 112 becomes "1." In the blend region 111a, the value slopes from "1" to "0" going inward from the second region 112. In the multiplication process P43, the second region 112 has an α channel of "1," so the pixel values ​​in the background image vB are maintained as they are. The first region 111 has an α channel of "0," so the pixel values ​​in the background image vB are set to "0." However, the pixels in the blending region 111a are multiplied by coefficients that slope from "1" to "0" from the outside to the inside depending on the position of the pixel.

[0176] The rendering engine 520 performs an additive synthesis process P44 on the outputs of the multiplication processes P41 and P43. As a result, in the first region 111, the area inside the blending region 111a contains the shot image vC, the second region 112 contains the background image vB, and from the inside to the outside of the blending region 111a, the weight of the shot image vC gradually changes to a weight of the background image vB, resulting in the final output image vCF shown in Figure 16.

[0177] Although Figure 23 shows an example of so-called alpha blending, even if the blending area 111a is not provided in the mask vK and the alpha value is set to "1" for the first area 111 and "0" for the second area 112, compositing can be performed using the processing in Figure 23.

[0178] In the case of virtual production, the LED wall 505 may not be in focus to prevent problems such as moire. In this case, the position, brightness, and color of the background image vB and the shot image vC to be combined do not need to match precisely on a pixel-by-pixel basis.

[0179] This completes the processing for one frame. In step S114, the rendering engine 520 determines whether or not to end the processing, and if not, returns to step S101. In other words, the same processing continues for the next frame. For example, when the image capture is completed, the rendering engine 520 determines that the process is finished in step S114, and ends the process of FIG.

[0180] <5. Pre-visualization processing> The captured image processing described above includes marker extraction processing P11, marker position estimation processing P12, peripheral light falloff and distortion detection processing P13, and mask generation processing P14, but some of these processes may be performed before shooting, which reduces the burden of real-time captured image processing during shooting. So, I will explain pre-visualization before shooting. Figure 24 shows the position of previsualization in the video production process.

[0181] FIG. 24 shows the flow of asset creation ST1, production ST2, and post-production ST3, which were previously explained in FIG. Production ST2 is the stage where actual shooting takes place in the studio, and is divided into pre-shooting and shooting.

[0182] In contrast, previsualization is expected to take place mainly at a stage before filming preparations. Although the chronological order is not necessarily limited, previsualization can be carried out at a stage after the 3D model for rendering the background video vB in asset creation ST1 has been created. In that sense, previsualization can be considered to take place in the final stage of asset creation ST1, or in the stage before filming preparation in production ST2.

[0183] Previsualization involves processes such as video confirmation and moiré alert. Video confirmation is a process that allows the director and staff to check the resulting video (vC) prior to filming. It primarily enables confirmation of the color tone of the video (vC) using simulation video.

[0184] Moiré Alert is a process that predicts moiré when, for example, a staff member is virtually adjusting the camera position while checking the video, and issues a warning if moiré is predicted to occur, thereby helping to prevent artifacts from occurring during subsequent filming.

[0185] In the preparation stage for shooting, color calibration is performed along with various settings for the LEDs, cameras, etc. in the shooting system 500. Specifically, a calibration LUT (Look Up Table) is generated to cancel color changes that occur when the LED wall 505 is shot with the camera 502. The calibration LUT is, for example, a 3D-LUT. Instead of the 3D-LUT, a combination of a matrix and a 1D-LUT or preparation for calibration performed by other video signal processing may be performed.

[0186] During the shooting stage, moiré alerts, calibration LUTs, and on-set camera settings are implemented. The moiré alert in this case is a process of outputting a moiré alert to a cameraman or the like in real time in response to a moiré determination made by the rendering engine 520. The calibration LUT is a process of canceling color changes related to the background image displayed by the LED wall 505, the camera 502, etc. by applying an LUT generated during color calibration during preparation for shooting. On-set camera setting is a process of obtaining the camera settings set in pre-visualization from, for example, a cloud server and reflecting them on the camera 502 in the studio.

[0187] 25 shows the configuration of the imaging system 500 and the configuration for previsualization side by side. The upper part of the drawing shows a simplified version of the configuration of the imaging system 500 described in FIG. 5 and other drawings, and the lower part of the drawing shows the configuration used for previsualization (previsualization system 700).

[0188] 25, the imaging system 500 includes a rendering engine 520, an LED processor 570, an LED wall 505, a camera 502, a captured image processing unit 85a, an operation monitor 550, and an output monitor 503.

[0189] The rendering engine 520 includes a rendering unit 520a and a marker processing unit 85b, which allows the background image vB with the boundary markers 60 added to be displayed on the LED wall 505.

[0190] A captured image processing unit 85a that processes the captured image vC is shown for the camera 502. The captured image processing unit 85a is provided in the rendering engine 520 or the information processing device 70 that constitutes another device.

[0191] The final output video vCF is treated as main video and is recorded on a recording medium or transmitted to other devices for broadcasting or distribution.

[0192] For example, a pre-visualization system 700 is configured as a system outside the studio in relation to the above-described shooting system 500. The pre-visualization system 700 includes at least an information processing device 70 that functions as the rendering engine 31, and a monitor 40. The rendering engine 31 is configured by, for example, an information processing device 70 separate from the rendering engine 520 .

[0193] The rendering engine 31 includes a rendering unit 32 and a video processing unit 33 . The rendering unit 32 has the same rendering function as the rendering unit 520a in the rendering engine 520 described above, and can generate the background video vBP based on a 3D model.

[0194] This rendering unit 32 can render the background video vB using at least the 3D model created in the asset creation ST1 process. In other words, the rendering unit 32 obtains the same 3D model as that used in the production ST2 process from, for example, the asset server 530 (see FIGS. 5 and 7), and uses it to perform rendering.

[0195] The rendering unit 32 can then use the 3D model to render the background image vB including the shooting area image vBC (inner frustum). To generate the shooting area video vBC, the rendering unit 32 receives input of the camera position, shooting direction, and angle of view, and performs rendering at a viewpoint position relative to the 3D model according to the input. Therefore, it is assumed that the camera work during actual shooting (the camera path such as the position of the camera 502 and the shooting direction, and transitions in the angle of view) has already been determined. By obtaining information about the camera work during actual shooting, it is possible to obtain the same shooting area video vBC as during shooting. The background image generated by the rendering unit 32 is referred to as "background image vBP" to distinguish it from the shooting area image vBC generated by the rendering engine 520 during shooting.

[0196] The video processing unit 33 has the function of performing various types of video processing on the background video vBP. The video processing unit 33 has the functions of a marker processing unit 85b, a shot video processing unit 85a, and an actual shot video processing unit 33a.

[0197] The video processing unit 33 performs marker processing using the function of the marker processing unit 85b. That is, boundary markers 60 are added to the background video vBP at locations that will become the periphery when the background video vBP is displayed on the LED wall 505.

[0198] The video processing unit 33 also performs actual shooting imaging processing using the function of the actual shooting imaging processing unit 33a. The actual shooting imaging processing is processing for generating a simulation video vSM that imitates the camera 502 actually used for shooting. In this actual shooting imaging processing, the simulation video vSM is generated for the background video vBP using processing parameters that realize at least the brightness or color characteristics of the camera 502 used in the shooting system 500 at the time of shooting. Specifically, the video processing unit 33 matches parameters such as RGB gain, white balance value, gamma value, offset value, etc. to those of the camera 502.

[0199] By setting the brightness or color parameters to the same as those of the camera 502 at the time of shooting, a simulation image vSM with the same color tone as the shot image vC is generated, and the simulation image vSM is displayed on the monitor 40.

[0200] The monitor 40 represents a monitor device that displays the video output from the information processing device 70 serving as the rendering engine 31, and may be, for example, a monitor device located in a different location from the photography studio, or may be a monitor device located in the photography studio. The monitor 40 may be in any form as long as it can display the simulation video vSM at least at the pre-visualization stage.

[0201] In the rendering engine 31, the rendering unit 32 generates a background video vBP, and the video processing unit 33 performs actual shooting video processing to generate a simulation video vSM. By displaying this on the monitor 40, the staff can check the simulation video vSM, which has the same color tone as the background video included in the shot video vC during shooting. This allows the staff to check the color prior to shooting and, if necessary, adjust the color in the camera 502 or camera signal processing unit 515 during the shooting preparation stage.

[0202] Here, the video processing unit 33 executes part of the captured video processing on the simulation video vSM by using the function of the captured video processing unit 85a. For example, an example of processing by the video processing unit 33 is shown in FIG.

[0203] In image processing step S200, the image processing unit 33 acquires information about the position and angle of the LED wall 505 at the time of image capture.

[0204] After starting to generate the simulation video vSM, the video processing unit 33 performs the processes of steps S201 to S208 for each frame of the captured video vC.

[0205] In step S201, the video processing unit 33 acquires the marker generation information Imk for the currently processed frame of the simulation video vSM. In step S202, the video processing unit 33 acquires information on the position, shooting direction, and angle of view of the camera 502 as shooting information during actual shooting of the frame currently being processed. If the camera work is determined in advance, the information on the position, shooting direction, and angle of view of the camera 502 corresponding to the frame of the simulation video vSM can be acquired.

[0206] In step S203, the image processing unit 33 performs the marker position estimation process P12 using the information on the LED wall 505, the shooting information, and the marker generation information Imk (pattern shape, size), and generates a marker position estimation image vMKb.

[0207] Furthermore, in step S204, the video processing unit 33 sets a threshold based on the marker generation information Imk (brightness, color), and performs a marker extraction process P11 from the simulation video vSM to generate a marker-extracted image vMKa.

[0208] In step S205, the image processing unit 33 determines whether or not the current frame includes part or all of the boundary marker 60. If the current frame does not include a boundary marker 60, the process proceeds to step S208.

[0209] On the other hand, if the current frame includes the boundary marker 60, the image processing unit 33 proceeds to step S206 and performs peripheral light falloff and distortion detection processing P13. Then, in step S207, the amount of peripheral light falloff and the amount of distortion are stored as information for the current frame, for example, in DB 42 in Fig. 25, so that they can be referenced when processing captured images during shooting.

[0210] The video processing unit 33 repeats the above process for each frame until it determines in step S208 that the process is finished.

[0211] By this processing, the DB 42 detects the amount of peripheral light falloff and distortion for each frame of the captured video vC before the image is captured. Therefore, step S106 does not need to be executed during the captured video processing of Fig. 18. Furthermore, as a result, it is sufficient to perform either step S103 or S104, because a mask can be generated from either the marker extraction image vMKa or the marker position estimation image vMKb.

[0212] It is also possible to generate the mask vK at the pre-visualization stage, and then read out the mask vK for each frame during the captured image processing at the time of shooting, and perform the synthesis process P19.

[0213] <6. Summary and variations> According to the above embodiment, the following effects can be obtained.

[0214] The information processing device 70 of the embodiment is, for example, a rendering engine 520, a processor in the camera 502, or a device that inputs the captured video vC from the camera 502 during shooting, and performs captured video processing or marker processing. Alternatively, the information processing device 70 of the embodiment is a device that performs video editing in post-production ST3, and performs captured video processing. Furthermore, the information processing device 70 of the embodiment is a rendering engine 31 used in the pre-visualization stage, and performs marker processing or captured video processing.

[0215] Such an information processing device 70 is configured to include a captured image processing unit 85a that performs a determination process on a captured image vC, which is captured by a camera 502 of an LED wall 505 that displays an image with a boundary marker 60 added to a virtual image such as a background image vB, to determine the position of the boundary marker in the target frame of the captured image vC. By detecting the boundary marker 60 in the captured video vC, even if the shooting range of the camera 502 extends beyond the display range of the LED wall 505, the position and amount of the extension can be accurately detected and appropriate processing can be performed. For example, the image of the extension part can be accurately filled in, so that the shooting is not wasted. This also means that the cameraman 512 does not have to worry about making sure that the shooting range fits within the range of the LED wall 505 when taking a picture, and the degree of freedom in framing is increased.

[0216] In the embodiment, the captured video processing unit 85a performs the determination process of extracting boundary markers 60 from target frames of the captured video vC based on the marker generation information Imk. That is, in the marker extraction process P11, the captured video processing unit 85a in the information processing device 70, such as the rendering engine 520, obtains information on the brightness and color of the boundary markers 60 from the marker generation information Imk, and extracts the boundary markers 60 in the captured video vC accordingly. When a boundary marker 60 is generated by selecting a clear brightness or color relative to the color or background content of the background image vB, obtaining the brightness or color information enables accurate extraction of the boundary marker 60 from the captured image vC. In other words, the brightness and color of the boundary marker 60 may be set according to the content of the background image vB, which makes it possible to make the boundary marker 60 stand out within the background image vB.

[0217] In the embodiment, the captured image processing unit 85a performs a marker position estimation process P12 as a determination process to estimate the position of the boundary marker 60 in the target frame of the captured image vC based on the shooting information including the position information, shooting direction, and angle of view of the camera 502 at the time of shooting, and the position information of the LED wall 505. This allows the position of the boundary marker 60 in the captured image vC to be accurately determined without lens distortion or the like.

[0218] In this embodiment, the captured video processing unit 85a performs a first determination process (marker extraction process P11) to extract a boundary marker 60 from a target frame of the captured video vC, and a second determination process (marker position estimation process P12) to estimate the position of the boundary marker 60 in the target frame of the captured video vC. The captured video processing unit 85a then performs a process to calculate the amount of distortion of the captured video vC using the resulting marker extraction image vMKa and marker position estimation image vMKb (see step S106 in FIG. 18 and FIG. 19). By calculating the amount of distortion, it is possible to accurately add distortion to the background video vB, which becomes the outside-of-frame additional video vOT, and to accurately remove distortion from the shot video vC, making it possible to synthesize a natural outside-of-frame additional video vOT, thereby improving the quality of the final output video vCF.

[0219] In addition, in the embodiment, the captured image processing unit 85a performs a process of calculating the amount of peripheral light falloff in the captured image vC using the marker extraction image vMKa and the marker position estimation image vMKb obtained through the first determination process (marker extraction process P11) and the second determination process (marker position estimation process P12) (see step S106 in Figure 18 and Figure 19). By calculating the amount of vignetting, it is possible to accurately add vignetting to the outside-of-frame additional image vOT and remove vignetting from the captured image vC, making it possible to synthesize a natural outside-of-frame additional image vOT, thereby improving the quality of the final output image vCF.

[0220] In the embodiment, the captured video processing unit 85a performs a process of generating a mask vK that distinguishes between a first region 111 that is inside the boundary marker 60 in a target frame of the captured video vC and a second region 112 that is outside the first region 111, and a synthesis process of using the mask vK to synthesize an outside-frame additional image vOT that is an image corresponding to the second region 112 with the image of the first region 111 in the captured video vC. That is, the captured video processing unit 85a performs a synthesis process P19 using the mask vK generated in the mask generation process P14 (see steps S107 and S111 in FIG. 18, FIGS. 21 and 23). This allows a synthesis process to be performed so that the second area 112 including the boundary marker 60 in the captured image vC is replaced with the out-of-frame additional image vOT, and the final output image vCF can be output in which the part that extends beyond the LED wall 505 has an appropriate background.

[0221] As described above, the mask generation process P14 and the synthesis process P19 may be performed in real time during shooting, but one or both of them may also be performed in the post-production stage ST3. For example, if shooting information including the position information, shooting direction, and angle of view of the camera 502 at the time of shooting, position information of the LED wall 505, and marker generation information Imk are stored in association with each frame of the shot video vC, the processing of Figure 18 will be possible during post-production ST3. Furthermore, if the background video vB is stored in association with each frame of the shot video vC, the background video vB can be used as is in the processes of steps S109 and S110. The mask vK may be generated in real time during shooting, and stored in association with each frame of the shot video vC, and the mask vK may be read and used during the composition process in post-production ST3.

[0222] In the embodiment, the captured image processing unit 85a may perform the composition process P19 to compose the image of the first region 111 in the captured image vC with the outside-frame added image vOT to which the amount of distortion or peripheral light falloff that has occurred in the captured image vC has been added. That is, this is the case when Course A is selected in the selection processes P17 and P18 in FIG. 9. As a result, the outside-frame additional image vOT, to which the same distortion and peripheral light falloff have been added, is composited in the second area 112 with the captured image vC in the first area 111. Therefore, the outside-frame additional image vOT is composited seamlessly with the captured image vC. In particular, in this case, it is possible to obtain the final output image vCF in which the lens distortion of the camera 502 and the like are utilized.

[0223] It should be noted that the amount of distortion can be calculated for each frame at the pre-visualization stage (see FIG. 26). This eliminates the need for step S106 in the processing of FIG. 18 during shooting, and reduces the processing load for adding distortion to the background video vB, which becomes the outside-frame additional video vOT. This is therefore suitable for performing the processing of FIG. 18 in real time during shooting.

[0224] In the embodiment, the captured image processing unit 85a may perform the composition process P19 to combine the image of the first region 111 in the captured image vC from which the distortion or peripheral light falloff has been removed with the additional image vOT outside the frame (such as the background image vB). That is, this is the case when Course B is selected in the selection processes P17 and P18 in FIG. 9. As a result, the outside-frame added image vOT, which has no distortion or peripheral light falloff, is composited with the captured image vC, from which the distortion and peripheral light falloff of the first region 111 have been removed. Therefore, the outside-frame added image vOT is composited seamlessly with the captured image vC. In particular, in this case, it is possible to obtain the final output image vCF, from which the lens distortion of the camera 502 has been canceled.

[0225] Note that by calculating the amount of vignetting for each frame at the pre-visualization stage (see FIG. 26), step S106 in the processing of FIG. 18 during shooting becomes unnecessary, and the processing load for removing vignetting for the shot video vC is reduced. Therefore, this is suitable for performing the processing of FIG. 18 in real time during shooting.

[0226] Furthermore, although it has been assumed that a selection between course A and course B is made, it is also possible to select only course A or only course B without performing the selection processes P17 and P18. In other words, in the synthesis process P19, the image of the first area 111 in the captured image vC may always be synthesized with an outside-frame additional image vOT (background image vB, etc.) to which the amount of distortion or peripheral light falloff that has occurred in the captured image vC has been added. Alternatively, the synthesis process P19 may always be performed by synthesizing the image of the first area 111 in the captured image vC from which distortion or peripheral light falloff has been removed with the additional image vOT outside the frame (such as the background image vB).

[0227] In the embodiment, an example has been described in which the captured image processing unit 85a performs the synthesis process P19 by gradually changing the image ratio of the captured image vC and the outside-frame additional image vOT, such as the background image vB, near the boundary between the first area 111 and the second area 112. 22 and 23, the area near the boundary of the first area 111 is set as a blending area 111a, and the α value of the mask vK is changed in stages between 0 and 1. This makes the boundary between the captured image vC and the outside-frame additional image vOT less noticeable, improving the quality of the final output image vCF.

[0228] In the embodiment, an example has been given in which the background video vB rendered for display on the LED wall 505 is used as the outside-frame additional video vOT. For example, the background image vB can be used as the additional image vOT outside the frame. This eliminates the need to create the additional image vOT separately. This also makes it suitable for real-time compositing during shooting. However, an image of the second area 112 may be generated separately from the background image vB and used as the outside-frame additional image vOT.

[0229] In the embodiment, an example has been given in which the information processing device 70 includes the marker processing unit 85b that performs processing to add the boundary marker 60 to the background image vB (virtual image) to be displayed on the LED wall 505. As a result, the boundary marker 60 is displayed on the LED wall 505. This enables the above-described processing by the captured image processing unit 85a. The functions of the captured image processing unit 85a and the marker processing unit 85b may both be provided in the information processing device 70 such as the rendering engine 520, or may be provided in separate information processing devices 70.

[0230] In the embodiment, an example is given in which the marker processing unit 85b generates a boundary marker 60 that indicates the boundary of the display area of ​​the LED wall 505 and has a pattern that can detect the amount of distortion and peripheral light falloff caused by shooting by the camera 502, and adds the boundary marker 60 to the background image vB. For example, the boundary marker 60 has a boundary line 61 and a detection pattern 62. This enables the captured image processing unit 85a to detect the amount of peripheral light falloff and distortion, and to remove or add these. The boundary marker 60 may have at least the boundary line 61 and may be capable of detecting that the photographing range has gone outside the frame of the LED wall 505. In other words, there may be an example in which the detection pattern 62 is not provided.

[0231] In the embodiment, the marker processing unit 85b performs processing to add the boundary marker 60 to the background image vB to be displayed on the LED wall 505 at a position corresponding to the periphery of the LED wall 505. The entire area of ​​the background image vB generated by rendering is not necessarily displayed on the LED wall 505. As shown in FIG. 10A, a background image vB wider than the actual display area 100 may be generated. Therefore, as shown in FIG. 10B, boundary markers 60 are added to the periphery of the actual display area 100, which is the area that is actually displayed on the LED wall 505. As a result, as shown in FIG. 9, boundary markers 60 are displayed on the periphery of the LED wall 505, allowing the captured image processing unit 85a to properly execute processing. For example, if the entire background image vB as shown in FIG. 10A is displayed on the LED wall 505, it is appropriate to add a boundary marker 60 to the periphery of the entire background image vB.

[0232] In the embodiment, the marker processing unit 85b performs processing to record the marker generation information Imk (see step S54 in FIG. 17). For example, the marker generation information Imk, such as the brightness, color, pattern shape, and size of the boundary marker 60, is stored in a recording medium. This makes it possible to extract the marker from the captured video vC even after the marker has been added, for example, at the post-production stage ST3.

[0233] 8 is an example in which the information processing device 70 includes both the captured video processing unit 85a, which has the function of performing captured video processing, and the marker processing unit 85b, which has the function of performing marker processing, as the video processing unit 33, but is not limited to this. For example, an information processing device 70 that includes the marker processing unit 85b but not the captured video processing unit 85a, or an information processing device 70 that includes the captured video processing unit 85a but not the marker processing unit 85b, is also possible.

[0234] In addition, in the embodiment, the background image vB, which is the image displayed on the LED wall 505, is a virtual image obtained by rendering using a 3D model, but even when, for example, live-action image is displayed on the LED wall 505, marker processing that adds boundary markers 60 and photographed image processing for the photographed image vC are effective.

[0235] The program of the embodiment is a program that causes a processor such as a CPU or DSP, or a device including these, to execute the processes shown in FIGS. 17, 18, and 26 described above. That is, the program of the embodiment is a program that causes the information processing device 70 to execute a process of determining the position of the boundary marker 60 in the target frame of the captured image vC, which is captured by a camera of a display that displays an image (e.g., a virtual image such as a background image vB) including the boundary marker 60 to be displayed on the periphery of the display area of ​​a display such as an LED wall 505.

[0236] Such a program can be used to realize an information processing device 70 that operates in production ST2, such as the rendering engine 520 described above, an information processing device 70 used in post-production ST3, or an information processing device 70 used in pre-visualization, using various computer devices.

[0237] Such a program can be pre-recorded on a hard disk drive (HDD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Furthermore, such a program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disc, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Such a program can be installed onto a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0238] Furthermore, such a program is suitable for providing a wide range of information processing devices 70 according to the embodiment. For example, by downloading the program to personal computers, communication devices, mobile terminal devices such as smartphones and tablets, mobile phones, game devices, video devices, PDAs (Personal Digital Assistants), and the like, these devices can function as the information processing devices 70 of the present disclosure.

[0239] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0240] The present technology can also be configured as follows. (1) a captured image processing unit that performs a determination process for determining the position of the boundary marker in a target frame of a captured image obtained by capturing a display that displays an image including a boundary marker to be displayed on the periphery of a display area of ​​the display with a camera; Information processing device. (2) The captured video processing unit performs, as the determination processing, a process of extracting the boundary marker from a target frame of the captured video based on generation information of the boundary marker. The information processing device according to (1) above. (3) The captured image processing unit performs the determination process by estimating the position of the boundary marker in the target frame of the captured image based on image capture information including position information, image capture direction, and angle of view of the camera at the time of image capture and position information of the display. The information processing device according to (1) or (2) above. (4) The captured image processing unit includes: a first determination process for extracting the boundary marker from a target frame of the captured video; a second determination process for estimating the position of the boundary marker in the target frame of the captured video; and calculating a distortion amount of the captured video based on the determination result of the first determination process and the determination result of the second determination process. The information processing device according to any one of (1) to (3) above. (5) The captured image processing unit includes: a first determination process for extracting the boundary marker from a target frame of the captured video; a second determination process for estimating the position of the boundary marker in the target frame of the captured video; and calculating a peripheral light falloff amount of the captured image based on the determination result of the first determination process and the determination result of the second determination process. An information processing device according to any one of (1) to (4) above. (6) The captured image processing unit includes: A process of generating a mask that distinguishes a first region that is inside the boundary marker in the target frame of the captured video from a second region that is outside the first region; performing a synthesis process of synthesizing an additional image, which is an image corresponding to the second area, with an image of the first area in the captured image using the mask; The information processing device according to any one of (1) to (5) above. (7) The captured image processing unit includes: In the synthesis process, The image of the first region in the photographed image is synthesized with the additional image to which the amount of distortion or peripheral light falloff caused in the photographed image has been added. The information processing device according to (6) above. (8) The captured image processing unit includes: In the synthesis process, The image of the first region in the captured image from which the distortion or peripheral light falloff has been removed is synthesized with the additional image. The information processing device according to (6) above. (9) The captured image processing unit includes: In the synthesis process, In a boundary area between the first area and the second area, the image ratio of the captured image and the additional image is gradually changed and combined. The information processing device according to any one of (6) to (8) above. (10) As the additional image, a virtual image rendered for display on the display is used. The information processing device according to any one of (6) to (9) above. (11) a marker processing unit that performs processing to add the boundary marker to the image to be displayed on the display; The information processing device according to any one of (1) to (10) above. (12) The marker processing unit generates the boundary marker having a pattern capable of detecting the amount of distortion or peripheral light falloff caused by photography with a camera, and adds the boundary marker to the image to be displayed on the display. The information processing device according to (11) above. (13) The marker processing unit performs processing to add the boundary marker to a position corresponding to a peripheral portion of the display in the image to be displayed on the display. The information processing device according to (11) or (12) above. (14) The marker processing unit performs a process of recording generation information of the boundary marker. The information processing device according to any one of (11) to (13) above. (15) The image displayed on the display is a virtual image obtained by rendering using a 3D model. An information processing device according to any one of (1) to (14) above. (16) A marker processing unit is provided for adding a boundary marker to be displayed on the periphery of the display area of ​​the display to the image to be displayed on the display that is the subject of the camera. Information processing device. (17) A process for determining the position of the boundary marker in a target frame of a captured image of a display that displays an image including the boundary marker to be displayed on the periphery of the display area of ​​the display, the captured image being captured by a camera. An information processing method performed by an information processing device. (18) A process for determining the position of the boundary marker in a target frame of a captured image of a display that displays an image including the boundary marker to be displayed on the periphery of the display area of ​​the display, the captured image being captured by a camera. A program to be executed by an information processing device. [Explanation of symbols]

[0241] 60 Boundary Marker 70 Information processing device, 71 CPU 85 Video Processing Unit 85a Output video processing section 85b Marker processing section 111 First area 112 Second area 502 Camera 505 LED Wall 520 Rendering Engine vB background video vBC shooting area image (inner frustum) vC shooting footage vCF final output video vMK marker image vMKa marker extraction image vMKb marker position estimation image vOT additional video outside the frame vK Mask Imk marker generation information

Claims

1. The display device includes a captured image processing unit that performs a process of determining the position of the boundary marker in a target frame of a captured image of the display device, the captured image being captured by a camera and including a boundary marker to be displayed on the periphery of the display area of ​​the display device. Information processing device.

2. The captured image processing unit performs, as the determination process, a process of extracting the boundary marker from a target frame of the captured image based on generation information of the boundary marker. The information processing device according to claim 1 .

3. The captured image processing unit performs, as the determination process, a process of estimating a position of the boundary marker in a target frame of the captured image based on shooting information including position information, shooting direction, and angle of view of the camera at the time of shooting, and position information of the display. The information processing device according to claim 1 .

4. The captured image processing unit includes: A first determination process for extracting the boundary marker from a target frame of the captured image; a second determination process for estimating a position of the boundary marker in a target frame of the captured image; calculating a distortion amount of the captured image based on a result of the first determination process and a result of the second determination process; The information processing device according to claim 1 .

5. The captured image processing unit includes: A first determination process for extracting the boundary marker from a target frame of the captured image; a second determination process for estimating a position of the boundary marker in a target frame of the captured image; calculating an amount of peripheral light falloff of the captured image based on a result of the first determination process and a result of the second determination process; The information processing device according to claim 1 .

6. The captured image processing unit includes: A process of generating a mask that distinguishes a first region that is inside the boundary marker in the target frame of the captured image from a second region that is outside the first region; A synthesis process is performed in which an additional image corresponding to the second area is synthesized with an image of the first area in the captured image by using the mask. The information processing device according to claim 1 .

7. The captured image processing unit includes: In the synthesis process, The image of the first area in the photographed image is combined with the additional image to which the amount of distortion or peripheral light falloff caused in the photographed image has been added. The information processing device according to claim 6.

8. The captured image processing unit includes: In the synthesis process, The image of the first area in the captured image from which the distortion or peripheral light falloff amount has been removed is synthesized with the additional image. The information processing device according to claim 6.

9. The captured image processing unit includes: In the synthesis process, The image ratio of the captured image and the additional image is gradually changed in a boundary area between the first area and the second area, and the captured image and the additional image are combined. The information processing device according to claim 6.

10. As the additional image, a virtual image rendered for display on the display is used. The information processing device according to claim 6.

11. A marker processing unit that performs processing to add the boundary marker to the image to be displayed on the display. The information processing device according to claim 1 .

12. The marker processing unit generates the boundary marker having a pattern capable of detecting an amount of distortion or an amount of peripheral light falloff caused by photography with a camera, and adds the boundary marker to the image to be displayed on the display. The information processing device according to claim 11.

13. The marker processing unit performs a process of adding the boundary marker to a position corresponding to a peripheral portion of the display in the image to be displayed on the display. The information processing device according to claim 11.

14. The marker processing unit performs a process of recording generation information of the boundary marker. The information processing device according to claim 11.

15. The image displayed on the display is a virtual image obtained by rendering using a 3D model. The information processing device according to claim 1 .

16. A marker processing unit is provided for performing processing for adding a boundary marker to be displayed on the periphery of a display area of ​​a display to an image to be displayed on the display that is the subject of a camera shot. Information processing device.

17. A process of determining a position of the boundary marker in a target frame of a captured image of a display that displays an image including the boundary marker to be displayed on the periphery of a display area of ​​the display, the captured image being captured by a camera; An information processing method performed by an information processing device.

18. A process of determining a position of the boundary marker in a target frame of a captured image of a display that displays an image including the boundary marker to be displayed on the periphery of a display area of ​​the display, the captured image being captured by a camera; A program to be executed by an information processing device.

Images