Information processing apparatus, information processing method, and imaging system

Abstract

This information processing device implements video processing in which, from among videos obtained by capturing a display video on a display and videos obtained by being independent of an object on the display by a camera, a first captured video including the object and a specific video, and a display that replaces the specific video with a corresponding video generated corresponding to the camera that captures the first captured video, the display displaying the corresponding video generated corresponding to one of the plurality of cameras and the specific video in a time division manner.

Description

Technical Field

[0001] The present technology relates to information processing equipment, information processing methods and imaging systems, and in particular to imaging surveillance videos. Background Art

[0002] As an imaging method for producing video contents such as movies, there is known a technique in which performers perform using a so-called green screen and then composite a background video.

[0003] Furthermore, in recent years, instead of green screen shooting, an imaging system has been developed in which, in a studio provided with a large display, a background video is displayed on the display and performers perform in front of it, so that the performers and the background can be imaged, and the imaging system is called so-called virtual production, in-camera visual effects, or light-emitting diode (LED) wall virtual production.

[0004] Patent Document 1 listed below discloses the technology of a system for imaging a performer performing in front of a background video.

[0005] Reference List

[0006] Patent Literature

[0007] Patent Document 1: U.S. Patent Application Publication No. 2020 / 0145644A Summary of the invention

[0008] Problems to be solved by the present invention

[0009] When a background video is displayed on a large display and the performers and background video are then captured with a camera, there is no need to separately synthesize the background video after capture. The performers and staff can visually understand the scene and perform or judge whether the performance is good or bad, etc., which is more advantageous than green screen shooting.

[0010] However, the appearance of the background should be different according to the position of the camera with respect to the display and the imaging direction. When the background video is simply cast, the background does not change even if the camera position, imaging direction, etc. are different, and the video becomes quite unnatural. Therefore, by changing the background video (at least the video in the display within the viewing angle of the camera) so as to be equivalent to the actual appearance according to the camera position, imaging direction, etc., it is possible to capture a video equivalent to the case of imaging at the actual position.

[0011] However, when using multiple cameras, since the positions of the cameras are physically different, the backgrounds suitable for each camera may overlap on the display, or may image the background of another camera. As a result, the background of the surveillance video of each camera may be in an incorrect state.

[0012] Therefore, the present disclosure proposes a technology for obtaining a surveillance video having an appropriate background for each camera in a case where a video displayed on a display is imaged by a plurality of cameras.

[0013] Solutions to the Problem

[0014] According to the present technology, an information processing device includes a video processing unit, which implements video processing for replacing a first captured video including an object and a specific video with a corresponding video, wherein the corresponding video is generated corresponding to a camera that captures the first captured video, and the first captured video is a video among videos obtained by capturing a display video on a display and an object separated from the display by the camera, and the display displays the corresponding video generated by one camera among a plurality of cameras and the specific video in a time-division manner.

[0015] For example, when a corresponding video such as a background is displayed on a display and imaging is performed by multiple cameras, a specific video is displayed and the timing of capturing the specific video and the object is provided. Subsequently, in this case, a process of replacing the specific video portion with the corresponding video corresponding to each camera is performed for the first captured video. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is an explanatory diagram of an imaging system of an embodiment of the present technology.

[0017] Figure 2 is an explanatory diagram of a background video of a camera position of the imaging system according to the embodiment.

[0018] Figure 3 is an explanatory diagram of a background video of a camera position of the imaging system according to the embodiment.

[0019] Figure 4 It is an explanatory diagram of the video content production steps of the embodiment.

[0020] Figure 5 is a block diagram of the imaging system of the embodiment.

[0021] Figure 6 is a flow chart of background video generation of the imaging system of the embodiment.

[0022] Figure 7 is a block diagram of an imaging system using multiple cameras according to the embodiment.

[0023] Figure 8 is a block diagram of the information processing apparatus of the embodiment.

[0024] Fig. 9 This is an explanatory diagram of the overlap of the inner frustum of each camera.

[0025] Fig.10 is a block diagram of a configuration example including a switch.

[0026] Fig.11 It is a flow chart of the switching control of the switch.

[0027] Fig.12 is an explanatory diagram of an example of a process that affects a switching delay time.

[0028] Fig.13 is a block diagram of a configuration example of the first embodiment.

[0029] Fig.14 is an explanatory diagram of a captured video and a black background captured video according to the embodiment.

[0030] Fig.15 4 is a flowchart of the background video rendering process of the first embodiment.

[0031] Fig.16 is an explanatory diagram of the configuration of the video processing unit of the first embodiment.

[0032] Fig.17 is a flowchart of the processing of the video processing unit of the first embodiment.

[0033] Fig.18 is a block diagram of a configuration example of the second embodiment.

[0034] Fig.19 is an explanatory diagram of the configuration of the video processing unit of the second embodiment.

[0035] Fig. 20 is a flowchart of the processing of the video processing unit of the second embodiment.

[0036] Fig.21 is a block diagram of a configuration example of the third embodiment.

[0037] Fig. 22 is a block diagram of a configuration example of the fourth embodiment.

[0038] Fig.23 is a flowchart of the processing of the video processing unit of the fourth embodiment.

[0039] Fig.24 is a block diagram of a configuration example of the fifth embodiment.

[0040] Fig.25 is an explanatory diagram of the configuration of a video processing unit of the fifth embodiment.

[0041] Fig.26 is a flowchart of the processing of the video processing unit of the fifth embodiment. DETAILED DESCRIPTION

[0042] Hereinafter, each embodiment will be described in the following order.

[0043] <1. Imaging system and content production>

[0044] <2. Configuration of information processing equipment>

[0045] <3. Imaging system using multiple cameras and switchers>

[0046] <4. First embodiment>

[0047] <5. Second Embodiment>

[0048] <6. Third embodiment>

[0049] <7. Fourth embodiment>

[0050] <8. Fifth embodiment>

[0051] <9. Summary and modification examples>

[0052] It should be noted that in the present disclosure, "video" or "image" includes both still images and moving images. In addition, "video" not only refers to the state of displaying the video on the display, but also video data in the state of not displaying the video data on the display can be inclusively referred to as "video".

[0053] For example, in various embodiments, the background video before being displayed on the display, the video captured by the camera, and the background video or captured video switched by the switch are not the actual displayed videos, but video data. Such video data is referred to as "background video", "captured video", etc. for convenience.

[0054] In addition, a plurality of cameras are indicated by adding "a", "b", "c" ... to the reference numerals to write "camera 502a" and "camera 502b", and "camera 502" is used when collectively referred to. In addition, circuit units, functions, signals, etc. corresponding to the respective cameras are distinguished by adding "a", "b", "c" ... to numbers or characters as reference numerals, and only reference numerals excluding "a", "b", and "c" are used when collectively referred to.

[0055] <1. Imaging system and video content production>

[0056] Imaging systems and production of video content to which the techniques of this disclosure may be applied will be described.

[0057] Figure 1 An imaging system 500 is schematically shown. The imaging system 500 is a system that implements imaging as a virtual production, and a part of equipment arranged in an imaging studio is shown in the figure.

[0058] In the imaging studio, a performance area 501 is provided in which a performer 510 performs a performance such as a performance. A large display device is arranged on at least the rear surface, left and right side surfaces, and the upper surface of the performance area 501. Although the device type of the display device is not limited, an LED wall 505 is used as an example of the large display device in the example shown in the figure.

[0059] One LED wall 505 forms a large panel by vertically and horizontally connecting and arranging a plurality of LED panels 506. The size of the LED wall 505 is not particularly limited, but only needs to be a size necessary or sufficient for displaying a background when imaging the performer 510.

[0060] A necessary number of lamps 580 are arranged at necessary positions such as above or at the side of performance area 501 to illuminate performance area 501 .

[0061] Near the performance area 501, for example, a camera 502 for imaging a film or other video content is arranged. A photographer 512 can move the position of the camera 502 and can implement operations such as imaging direction, viewing angle, etc. Of course, it is also conceivable to implement the movement of the camera 502, the viewing angle operation, etc. by remote control. In addition, the camera 502 can automatically or autonomously move or change the viewing angle. For this reason, the camera 502 can be installed on a camera platform or a mobile body.

[0062] Camera 502 uniformly captures performers 510 in performance area 501 and video displayed on LED wall 505. For example, by displaying a scene as background video vB on LED wall 505, it is possible to capture a video similar to the situation where performers 510 actually exist at the scene location and perform.

[0063] The output monitor 503 is arranged near the performance area 501. The video captured by the camera 502 is displayed in real time as a monitoring video vM on the output monitor 503. As a result, the director and the staff who produce the video content can check the captured video.

[0064] As previously described, imaging system 500 for imaging a performer's 510 performance against the backdrop of an LED wall 505 in an imaging studio has various advantages over green screen filming.

[0065] For example, in the case of green screen shooting, it is difficult for the performer to imagine the background and situation of the scene, which may affect the performance. On the other hand, by displaying the background video vB, the performer 510 can easily perform and the quality of the performance is improved. In addition, the director and other staff can easily judge whether the performance of the performer 510 matches the background or situation of the scene.

[0066] Furthermore, post-production after imaging is more efficient compared to the case of green screen shooting. This is because so-called chroma key composition may not be necessary, or color correction or mapping composition may not be necessary. Furthermore, even in the case where chroma key composition is required at the time of imaging, there is no need to add a background screen, which also contributes to improved efficiency.

[0067] In the case of green screen shooting, a green tint is added to the performer's body, costumes, and objects, so it is necessary to correct them. In addition, in the case of green screen shooting, in the case where there is an object in which the surrounding scene is reflected, such as glass, mirror, or snow top, it is necessary to generate and synthesize the reflected image, but this is a very troublesome work.

[0068] On the other hand, through Figure 1 In the case of imaging by the imaging system 500 in , the green hue will not increase, so there is no need to correct it. In addition, by displaying the background video vB, the reflection on the actual object such as glass is naturally obtained and captured, so there is no need to synthesize the reflection video.

[0069] Here we will refer to Figure 2 and 3 To describe the background video vB. Even though the background video vB is displayed on the LED wall 505 and captured together with the performer 510, the background of the captured video becomes unnatural simply by displaying the background video vB. This is because a three-dimensional background having depth is actually used as the background video vB in a flat manner.

[0070] For example, the camera 502 can image the performer 510 in the performance area 501 from various directions, and can also implement a zoom operation. The performer 510 will not stay in one place. Therefore, the actual appearance of the background of the performer 510 should change according to the position, imaging direction, viewing angle, etc. of the camera 502, but such a change cannot be obtained in the background video vB which is a flat video. Accordingly, the background video vB is changed so that the background is similar to the actual appearance including parallax.

[0071] Figure 2 The camera 502 is shown to be imaging a performer 510 from the position on the left side of the drawing. Figure 35 is a diagram showing a state in which the camera 502 is imaging the performer 510 from the right side of the drawing. In each of the drawings, the capture area video vBC is shown in the background video vB.

[0072] It should be noted that the portion of the background video vB excluding the capture area video vBC is called an "outer frustum", and the capture area video vBC is called an "inner frustum".

[0073] The background video vB described here represents the entire video displayed as a background including the capture area video vBC (inner viewing cone).

[0074] The range of the capture area video vBC (inner viewing cone) corresponds to the range of the camera 502 actually imaging in the display surface of the LED wall 505. Thus, the capture area video vBC is a video transformed to express the scene actually viewed when the camera 502 is set as a viewpoint according to the position of the camera 502, the imaging direction, the viewing angle, etc.

[0075] Specifically, as the background, 3D background data as a three-dimensional (3D) model is prepared, and the capture area video vBC is sequentially rendered in real time according to the viewpoint position of the camera 502 with respect to the 3D background data.

[0076] It should be noted that the range of the capture area video vBC is actually slightly wider than the range imaged by the camera 502 at that point in time. This is to prevent the video of the outer viewing cone from being reflected due to drawing delay, and to avoid the influence of diffracted light from the video of the outer viewing cone when the imaging range is slightly changed due to panning, tilting, zooming, etc. of the camera 502.

[0077] The video of the capture area video vBC rendered in real time in this way is synthesized with the video of the outer cone. The video of the outer cone used in the background video vB is pre-rendered based on the 3D background data, and the video is merged into a part of the video of the outer cone as the capture area video vBC rendered in real time to generate the entire background video vB.

[0078] As a result, even when the camera 502 is moved forward and backward or left and right or when a zoom operation is performed, the background of the range imaged together with the performer 510 is captured as a video corresponding to the change in viewpoint position accompanying the actual movement of the camera 502.

[0079] like Figure 2 and 3 As shown in , the monitoring video vM including the performer 510 and the background is displayed on the output monitor 503, and this is the captured video. The background of the monitoring video vM is the capture area video vBC. That is, the background included in the captured video is a real-time rendered video.

[0080] As described above, in the imaging system 500 of the embodiment, the background video vB including the capture area video vBC is changed in real time, so that not only the background video vB is simply displayed in a flat manner, but also a video similar to the actual imaging situation on site can be captured.

[0081] It should be noted that by rendering in real time only the capture area video vBC as the range illuminated by the camera 502 instead of the entire background video vB displayed on the LED wall 505, the processing load of the system is also reduced.

[0082] Here, the production steps for video content will be described as virtual production, where imaging is performed by the imaging system 500. Figure 4 As shown in , the video content production steps are roughly divided into three stages. The stages are asset creation ST1, production ST2 and post-production ST3.

[0083] Asset creation ST1 is a step of making 3D background data for displaying background video vB. As described above, background video vB is generated by performing rendering in real time using 3D background data at the time of imaging. For this purpose, 3D background data as a 3D model is made in advance.

[0084] Examples of methods of producing 3D background data include full computer graphics (CG), point cloud data (Point Cloud) scanning, and photogrammetry.

[0085] Full CG is a method of making a 3D model using computer graphics. Among the three methods, full CG requires the most manpower and time, but is preferably used when it is desired to use non-realistic videos, videos that are difficult to capture in practice, etc. as background videos vB.

[0086] Point cloud data scanning is a method of generating a 3D model based on point cloud data, in which, for example, distance measurement is performed from a specific position using LiDAR, a 360-degree image is captured from the same position by a camera, and color data captured by the camera is placed on the points measured by LiDAR. Compared with full CG, a 3D model can be produced in a short time. In addition, a 3D model with higher definition than photogrammetry is easily produced.

[0087] Photogrammetry is a photographic measurement technique for analyzing parallax information from two-dimensional images to obtain dimensions and shapes, where two-dimensional images are obtained by capturing an object from multiple viewpoints. 3D modeling can be implemented in a short time.

[0088] It should be noted that the point cloud information acquired by LiDAR can be used in 3D data generation by photogrammetry.

[0089] In asset creation ST1, for example, these methods are used to make a 3D model that will serve as 3D background data. Of course, the above methods can be used in combination. For example, a part of a 3D model made by point cloud scanning or photogrammetry is made by CG and synthesized, etc.

[0090] ST2 is made in Figure 1 The steps of implementing imaging in the imaging studio shown in . The essential technologies in this case include real-time rendering, background display, camera tracking, lighting control, etc.

[0091] Real-time rendering is used to Figure 2 and 3 The rendering process of the capture area video vBC is obtained at each time point (each frame of the background video vB) described. This is to render the 3D background data produced in the asset creation ST1 from the viewpoint corresponding to the position of the camera 502, etc. at each time point.

[0092] In this way, real-time rendering is implemented to generate a background video vB including each frame of the capture area video vBC, and the background video vB is displayed on the LED wall 505.

[0093] Camera tracking is implemented to obtain imaging information of the camera 502, and to track the position information, imaging direction, viewing angle, etc. of the camera 502 at each time point. By providing imaging information including such information to the rendering engine in association with each frame, real-time rendering according to the viewpoint position of the camera 502, etc. can be performed.

[0094] The imaging information is information linked or associated with the video as metadata.

[0095] It is assumed that the imaging information includes the position information of the camera 502 at each frame timing, the direction of the camera, the viewing angle, the focal length, the F value (aperture value), the shutter speed, the lens information, etc.

[0096] Lighting control is to control the state of lighting in the imaging system 500, specifically, to control the light quantity, light color, lighting direction, etc. of the lamp 580. For example, lighting control is implemented according to the time setting, location setting, etc. of the scene to be imaged.

[0097] Post-production ST3 refers to various processes performed after imaging, such as video correction, video adjustment, short film editing, video effects, etc.

[0098] As video correction, color gamut conversion, color matching between camera and material, etc. can be implemented.

[0099] As video adjustment, color adjustment, brightness adjustment, contrast adjustment, etc. can be implemented.

[0100] As for short film editing, you can implement cutting of short films, adjustment of sequence, adjustment of time length, etc.

[0101] As a video effect, synthesis of CG video or special effect video can be implemented.

[0102] Next, the configuration of the imaging system 500 used in producing ST2 will be described.

[0103] Figure 5 is shown with reference Figure 1 , 2 3 is a block diagram of an overview of the configuration of the imaging system 500 .

[0104] Figure 5 The imaging system 500 shown in FIG. 5 includes the LED wall 505, the camera 502, the output monitor 503, and the light 580 described above, wherein the LED wall 505 includes a plurality of LED panels 506. Figure 5 As shown in , the imaging system 500 also includes a rendering engine 520, an asset server 530, a synchronization generator 540, an operation monitor 550, a camera tracker 560, an LED processor 570, a lighting controller 581 and a display controller 590.

[0105] Each LED processor 570 is provided to correspond to one or more LED panels 506 , and implements video display driving of the corresponding one or more LED panels 506 .

[0106] The sync generator 540 generates a sync signal for synchronizing the frame timing of the display video of the LED panel 506 and the frame timing of the imaging of the camera 502, and provides the sync signal to the corresponding LED processor 570 and the camera 502. However, this does not prevent the output from the sync generator 540 from being provided to the rendering engine 520.

[0107] The camera tracker 560 generates imaging information of the camera 502 at each frame timing, and provides the imaging information to the rendering engine 520. For example, the camera tracker 560 detects the position information of the camera 502 relative to the LED wall 505 or the predetermined reference position and the imaging direction of the camera 502 as one imaging information, and provides it to the rendering engine 520.

[0108] As a specific detection method of the camera tracker 560, there is a method of randomly arranging reflectors on the ceiling and detecting the position by reflected light of infrared light emitted from the camera tracker 560 to the reflectors, wherein the camera tracker 560 is assembled to the camera 502. In addition, as a detection method, there is also a method of estimating the own position of the camera 502 by information of a gyroscope mounted on a camera platform of the camera 502 or a main body of the camera 502 or image recognition of a video captured by the camera 502.

[0109] In addition, the viewing angle, focal length, F value, shutter speed, lens information, etc. can be provided from the camera 502 to the rendering engine 520 as imaging information.

[0110] The asset server 530 is a server that can store the 3D model (ie, 3D background data) produced in the asset creation ST1 on a recording medium and read the 3D model when necessary. That is, the asset server 530 serves as a database (DB) of 3D background data.

[0111] The rendering engine 520 implements a process of generating a background video vB to be displayed on the LED wall 505. For this reason, the rendering engine 520 reads necessary 3D background data from the asset server 530. Then, the rendering engine 520 generates a video using an outer viewing cone in the background video vB, as a video obtained by rendering the 3D background data in a form viewed from pre-specified spatial coordinates.

[0112] Furthermore, as processing for each frame, the rendering engine 520 specifies the viewpoint position and the like regarding the 3D background data using imaging information provided from the camera tracker 560 or the camera 502 , and renders the capture area video vBC (inner view cone).

[0113] In addition, the rendering engine 520 synthesizes the pre-generated outer viewing cone with the capture area video vBC rendered for each frame, thereby generating a background video vB as video data for one frame. Subsequently, the rendering engine 520 sends the generated video data for one frame to the display controller 590.

[0114] The display controller 590 generates a divided video signal nD obtained by dividing the video data of one frame into video parts to be displayed on each LED panel 506, and transmits the divided video signal nD to each LED panel 506. At this time, the display controller 590 can perform calibration according to individual differences in color rendering, etc. between display units, manufacturing errors, etc.

[0115] It should be noted that the display controller 590 may not be provided, and the rendering engine 520 may implement these processes. That is, the rendering engine 520 may generate the divided video signal nD, implement calibration, and send the divided video signal nD to each LED panel 506.

[0116] The LED processor 570 drives the corresponding LED panel 506 based on each received divided video signal nD, and displays the entire background video vB on the LED wall 505. The background video vB includes the capture area video vBC rendered according to the position of the camera 502 at that time point.

[0117] The camera 502 can capture the performance of the performer 510 including the background video vB displayed on the LED wall 505 in this manner. The video obtained by the imaging of the camera 502 is recorded on a recording medium in the camera 502 or an external recording device (not shown), and is provided to the output monitor 503 in real time and displayed as a monitoring video vM.

[0118] The operation monitor 550 displays an operation image vOP for controlling the rendering engine 520. The engineer 511 can perform necessary settings and operations regarding rendering of the background video vB while viewing the operation image vOP.

[0119] The lighting controller 581 controls the light intensity, light color, irradiation direction, etc. of the lamp 580. For example, the lighting controller 581 can control the lamp 580 asynchronously with the rendering engine 520, or can implement control synchronously with the imaging information and rendering process. Therefore, the lighting controller 581 can implement light emission control according to instructions from the rendering engine 520, a master controller (not shown), etc.

[0120] Figure 6 An example of processing by the rendering engine 520 in the imaging system 500 having such a configuration is shown.

[0121] In step S10 , the rendering engine 520 reads the 3D background data to be used this time from the asset server 530 , and expands the 3D background data in the internal workspace.

[0122] Subsequently, a video is generated which will be used as the outer viewing cone.

[0123] Subsequently, the rendering engine 520 repeats the process from step S30 to step S60 at each frame timing of the background video vB until it is determined in step S20 that the display of the background video vB based on the read 3D background data is finished.

[0124] In step S30, the rendering engine 520 acquires imaging information from the camera tracker 560 and the camera 502. As a result, the position and state of the camera 502 to be reflected in the current frame are checked.

[0125] In step S40, the rendering engine 520 performs rendering based on the imaging information. That is, based on the position, imaging direction, viewing angle, etc. of the camera 502 to be reflected in the current frame, the viewpoint position with respect to the 3D background data is indicated, and rendering is performed. At this time, video processing reflecting focal length, F value, shutter speed, lens information, etc. may also be performed. Through this rendering, video data as the capture area video vBC may be obtained.

[0126] In step S50, the rendering engine 520 implements a process of synthesizing the outer viewing cone (i.e., the entire background video) and the video reflecting the viewpoint position of the camera 502 (i.e., the capture area video vBC). For example, the process is to synthesize the video of the entire background rendered at a specific reference viewpoint with the video generated by reflecting the viewpoint of the camera 502. With this process, a background video vB of one frame displayed on the LED wall 505 is generated, that is, the background video vB including the capture area video vBC.

[0127] The processing in step S60 is implemented by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates divided video signals nD obtained by dividing the background video vB of one frame into videos to be displayed on the respective LED panels 506. Calibration may be implemented. Subsequently, each divided video signal nD is sent to the corresponding LED processor 570.

[0128] Through the foregoing processing, the background video vB including the capture area video vBC captured by the camera 502 is displayed on the LED wall 505 at each frame timing.

[0129] Incidentally, in Figure 5 Only one camera 502 is shown in FIG. 5 , but imaging may be performed by multiple cameras 502 . Figure 7 An example of a configuration in the case of using a plurality of cameras 502a and 502b is shown. The cameras 502a and 502b can independently implement imaging in the performance area 501. In addition, synchronization between the cameras 502a and 502b and the corresponding LED processors 570 is maintained by the synchronization generator 540.

[0130] Output monitors 503a and 503b corresponding to the cameras 502a and 502b are provided, and the output monitors 503a and 503b are configured to display the videos captured by the corresponding cameras 502a and 502b as monitoring videos vMa and vMb, respectively.

[0131] In addition, camera trackers 560a and 560b corresponding to cameras 502a and 502b are provided, and the camera trackers 560a and 560b detect the positions and imaging directions of the corresponding cameras 502a and 502b, respectively. Imaging information from camera 502a and camera tracker 560a and imaging information from camera 502b and camera tracker 560b are sent to rendering engine 520.

[0132] The rendering engine 520 may perform rendering using imaging information from the camera 502 a or the camera 502 b to obtain a background video vB for each frame.

[0133] It should be noted that although Figure 7 An example using cameras 502 a and 502 b is shown, but it is also possible to implement imaging using three or more cameras 502 .

[0134] However, when multiple cameras 502 are used, there is a case where the capture area video vBC corresponding to each camera 502 is interfered. Figure 7 In the example of using two cameras 502a and 502b shown in , the capture area video vBC corresponding to the camera 502a is shown, but in the case of using the camera 502b, the capture area video vBC corresponding to the camera 502b is also necessary. When the capture area video vBC corresponding to each of the cameras 502a and 502b is simply displayed, interference occurs with each other. Therefore, it is necessary to plan the display of the capture area video vBC.

[0135] <2. Configuration of information processing equipment>

[0136] Next, we will refer to Figure 8 A configuration example of the information processing device 70 that can be used in asset creation ST1, production ST2, and post-production ST3 is described.

[0137] The information processing device 70 is a device such as a computer device that can perform information processing, particularly video processing. Specifically, as the information processing device 70, a personal computer, a workstation, a portable terminal device such as a smartphone and a tablet device, a video editing device, etc. are assumed. In addition, the information processing device 70 may be a computer device configured as a server device or an arithmetic device in cloud computing.

[0138] In the case of the present embodiment, specifically, the information processing device 70 can function as a 3D model making device that makes a 3D model in the asset creation ST1.

[0139] Furthermore, the information processing apparatus 70 may also function as the rendering engine 520 and the asset server 530 constituting the imaging system 500 used in producing ST2.

[0140] Furthermore, the information processing device 70 can also function as a video editing device that implements various types of video processing in post-production ST3.

[0141] In addition, the information processing device 70 is also used as a Fig.10 The switch 600 described above includes not only a hardware signal switching circuit but also functions for implementing control and calculation of the information processing device 70 .

[0142] Specifically, in Fig.13 and 22 In the example of , the video processing unit 18 is provided in the switch 600 , but the video processing unit 18 includes the information processing device 70 .

[0143] Furthermore, in the information processing device 70, Fig.18 , 21 The video processing units 24 ( 24 a , 24 b , 24 c ) in 24 are devices that implement rendering similar to the rendering engine 520 , but the video processing units 24 may also include an information processing device 70 .

[0144] Figure 8 The CPU 71 of the information processing device 70 shown in FIG. 1 performs various types of processing according to a program stored in a nonvolatile memory unit 74 such as a ROM 72 or, for example, an electrically erasable programmable read-only memory (EEP-ROM) or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to perform various types of processing.

[0145] The video processing unit 85 is configured as a processor that implements various types of video processing. For example, the video processing unit 85 is a processor that can implement any of the following processes: 3D model generation processing, rendering, DB processing, video editing processing, video analysis / detection processing, video processing such as video extraction and synthesis, or multiple types of processing.

[0146] The video processing unit 85 may be implemented, for example, by a CPU separate from the CPU 71 , a graphics processing unit (GPU), general purpose computing on a graphics processing unit (GPGPU), an artificial intelligence (AI) processor, or the like.

[0147] It should be noted that the video processing unit 85 may be provided as a function in the CPU 71 .

[0148] The CPU 71, the ROM 72, the RAM 73, the nonvolatile memory unit 74, and the video processing unit 85 are connected to one another via a bus 83. To the bus 83, an input / output interface 75 is also connected.

[0149] An input unit 76 including an operator and an operating device is connected to the input / output interface 75. As the input unit 76, various operating elements and operating devices including a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, a remote controller, etc. are assumed, for example.

[0150] A user operation is detected by the input unit 76 , and a signal corresponding to the input operation is interpreted by the CPU 71 .

[0151] A microphone is also assumed as the input unit 76. It is also possible to input a voice spoken by the user as operation information.

[0152] Furthermore, a display unit 77 including a liquid crystal display (LCD), an organic electroluminescence (EL) panel, or the like, and an audio output unit 78 including a speaker, or the like, are integrally or separately connected to the input / output interface 75 .

[0153] The display unit 77 is a display unit that implements various displays, and includes, for example, a display device provided in the housing of the information processing device 70 , a separate display device connected to the information processing device 70 , and the like.

[0154] The display unit 77 displays various images, operation menus, icons, messages, and the like on a display screen based on instructions from the CPU 71 , that is, as a graphical user interface (GUI).

[0155] In some cases, a storage unit 79 including a hard disk drive (HDD), a solid-state memory, or the like, or a communication unit 80 is connected to the input / output interface 75 .

[0156] The storage unit 79 can store various data and programs. A DB can also be configured in the storage unit 79.

[0157] For example, in the case where the information processing device 70 functions as the asset server 530 , a DB storing a 3D background data group may be constructed using the storage unit 79 .

[0158] The communication unit 80 performs communication processing through a transmission path such as the Internet, wired / wireless communication with various devices such as an external DB, an editing device, and an information processing device, bus communication, and the like.

[0159] For example, in the case where the information processing device 70 functions as the rendering engine 520 or the video processing unit 24 to be described later, the communication unit 80 can access the DB as the asset server 530 and receive imaging information from the camera 502 or the camera tracker 560 .

[0160] Furthermore, in the case of the information processing device 70 used in the post-production ST3, the communication unit 80 can also access a DB as the asset server 530 or the like.

[0161] A drive 81 is also connected to the input / output interface 75 as necessary, and a removable recording medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted.

[0162] The drive 81 can read video data, various computer programs, and the like from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio included in the data are output by the display unit 77 and the audio output unit 78. In addition, the computer program and the like read from the removable recording medium 82 are installed in the storage unit 79 as necessary.

[0163] In the information processing apparatus 70, for example, software for the processing of the present embodiment may be installed through network communication by the communication unit 80 or the removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0164] <3. Imaging system using multiple cameras and switchers>

[0165] In the described embodiments, for example, it is assumed that Figure 7 The described situation of using multiple cameras 502 (502a, 502b, ...) to implement imaging.

[0166] The captured videos vC of the multiple systems obtained by capturing the LED wall 505, the performer 510, etc. by the multiple cameras 502 are selectively switched by the switcher 600 described later, and output as the so-called main line video. Of course, it should be noted that each captured video vC can be recorded or transmitted separately from the main line video.

[0167] Here we will describe what happens when two or more cameras 502 are used to implement imaging in a virtual production.

[0168] As described above, the background video vB captured by the camera 502 is not the entire background video vB displayed on the LED wall 505, but the range of the capture area video vBC (hereinafter also referred to as "inner viewing cone vBC").

[0169] Subsequently, the rendering engine 520 renders the video content of the inner visual cone vBC for each frame according to the position and imaging direction of the camera 502, merges it into the entire body of the background video vB, and displays it on the LED wall 505. Figure 2 and 3 As described, the range and content of the inner viewing cone vBC of the background video vB are different for each frame according to the camera position, etc. In addition, the inner viewing cone vBC is displayed corresponding to each camera 502.

[0170] That is, the inner viewing cone vBC is generated corresponding to each camera 502 and is an example of a corresponding video in the present disclosure.

[0171] Here, considering that imaging is performed by two cameras 502a and 502b, the inner cones vBCa and vBCb corresponding to the respective cameras 502a and 502b are displayed on the LED wall 505. Depending on the positions of the cameras 502a and 502b, the inner cones vBCa and vBCb may be as follows: Fig. 9 The overlap is shown in A.

[0172] If the inner visual cones vBCa and vBCb do not overlap, the video vC captured by each camera 502a and 502b is a video including a proper background. Fig. 9 If the background video content is overlapped as shown in , the background video content is not in the correct state.

[0173] Therefore, in the case where imaging is implemented using a plurality of cameras 502 , a background video vB of a plurality of systems including respective inner viewing cones vBC is selectively displayed on the LED wall 505 .

[0174] When switching the main line video, the switching timing of the background video vB and the switching timing of the main line video are controlled so that the state of the video vC captured by a specific camera 502 including the background video vB of another camera 502 will not be output as the main line video.

[0175] exist Fig.10 An example of a configuration in this regard is shown in .

[0176] Fig.10 It is through Figure 7 The configuration shown in FIG. 1 is based on the premise of adding a switch 600 for switching the captured video vC and an output device 610. Figure 5 or Figure 7 The constituent parts described in the present invention are denoted by the same reference numerals, and repeated description is avoided.

[0177] For cameras 502a and 502b, camera signal processing units 515a and 515b are shown that perform signal processing of the captured video signals. Figure 5 and 7 Although omitted, the camera signal processing units 515a and 515b may be formed by a processor or the like in the cameras 502a and 502b, or may be provided as a device of a unit separate from the cameras 502a and 502b.

[0178] The video signal captured by the camera 502a is subjected to development processing, resizing processing, etc. by the camera signal processing unit 515a, and is input as a captured video vCa to the switch 600. The video signal captured by the camera 502b is subjected to development processing, resizing processing, etc. by the camera signal processing unit 515b, and is input as a captured video vCb to the switch 600.

[0179] In addition, imaging information IFa and IFb including camera positions, imaging directions, etc. of the camera trackers 560a and 560b are provided to the rendering engine 520. Although not shown, the viewing angle, focal length, f-number, shutter speed, lens information, etc. of the cameras 502a and 502b are also included in the imaging information IFa and IFb and provided to the rendering engine 520.

[0180] The rendering engine 520 includes one or more information processing devices 70. In this case, the rendering engine 520 is configured to have a plurality of rendering functions as the rendering units 21 and 22, and is configured to be able to simultaneously perform rendering corresponding to at least the camera 502a or 502b.

[0181] The rendering unit 21 performs rendering corresponding to the inner visual cone vBCa of the camera 502a according to the imaging information IFa about the camera 502a, merges the inner visual cone vBCa into the entire background, and outputs the background video vBa matching the camera 502a.

[0182] In addition, the rendering unit 22 performs rendering corresponding to the inner view frustum vBCb of the camera 502b according to the imaging information IFb about the camera 502b, merges the inner view frustum vBCb into the entire background, and outputs the background video vBb matching the camera 502b.

[0183] The switch 600 is provided with a switching unit 11, which receives background videos vBa and vBb as input, selects one of the background videos vBa and vBb, and outputs the selected video. The background video vBa or vBb selected by the switching unit 11 is the background video vB provided to the LED wall 505.

[0184] As previously referred Figure 5 and 7 As described, the background video vB is processed by the display controller 590 and distributed to the plurality of LED processors 570 and each LED panel 506 ( Fig.10 505) is driven by the LED processor 570. As a result, the background video vB is displayed on the LED wall 505.

[0185] Cameras 502a and 502b capture background video vB of LED wall 505 and performer 510.

[0186] As described above, the videos vCa and vCb captured by the cameras 502a and 502b are input to the switcher 600. The switcher 600 is provided with a switching unit 12 for inputting the captured videos vCa and vCb. The switching unit 12 selects one of the captured videos vCa and vCb and outputs the selected video as the main line video vCm.

[0187] The main line video vCm is provided to the output device 610 .

[0188] Here, the output device 610 may be a recording device that records the main line video vCm on a recording medium, or may be a video transmission device that broadcasts and transmits the main line video vCm. In addition, the output device 610 may be a web server that distributes the main line video vCm, etc.

[0189] The switch 600 includes a switching unit 11 for selecting a background video vB and a switching unit 12 for selecting a main line video vCm as described above, and includes a switch controller (hereinafter referred to as a "SW controller") 10 as a control unit for controlling the switching units 11 and 12.

[0190] The SW controller 10 may be composed of Figure 8 The information processing device 70 shown in FIG. 1 is configured. The SW controller 10 only needs to have at least Figure 8 The configuration of the CPU 71, ROM 72, RAM 73, nonvolatile memory unit 74 and input / output interface 75.

[0191] The SW controller 10 performs switching control of the switching units 11 and 12 according to the generation of the switching trigger KP. The SW controller 10 generates a control signal C1 for instructing the switching of the switching unit 11. In addition, the SW controller 10 instructs the switching of the switching unit 12 through a control signal C2.

[0192] The switching trigger KP is, for example, a trigger generated by an operator's switching operation on the main line video vCm (a switching operation on the camera 502 of the main line video vCm).

[0193] For example, the operator can use the switcher panel 611 to implement various operations including the switching operation of the main line video vCm.

[0194] It should be noted that the automatic switching trigger KP can be generated not only by the operator's operation. For example, it can be assumed that the switching trigger KP is generated by automatic switching control according to a predetermined sequence. In addition, it can also be assumed that the switching trigger KP is generated by replacing the operator with AI control.

[0195] Fig.10 A multi-screen viewer 612 is shown. The multi-screen viewer 612 is, for example, a monitoring device that displays monitoring videos vM (vMa, vMb) of the captured video vC of each camera 502 in a divided manner in one screen.

[0196] An operator of the switch 600 or the like can check the content captured by each camera 502 through the multi-viewer 612 .

[0197] Fig.11 The switching control of the switching units 11 and 12 by the SW controller 10 is shown.

[0198] When the switching trigger KP for the main line video vCm as described above is generated, the SW controller 10 proceeds from step S101 to step S102 and implements switching control of the switching unit 11 by the control signal C1. That is, the switching unit 11 is immediately switched according to the generation of the switching trigger KP.

[0199] Subsequently, the SW controller 10 waits for a predetermined time set as the switching delay time Td1 in step S103.

[0200] When the time as the switching delay time Td1 has elapsed, the SW controller 10 implements the switching control of the switching unit 12 by the control signal C2 in step S104. That is, when the SW controller 10 causes the switching unit 11 to select the background video vBa in step S102, the SW controller 10 causes the switching unit 12 to select the capture video vCa in step S104. Similarly, when the SW controller 10 causes the switching unit 11 to select the background video vBb in step S102, the SW controller 10 causes the switching unit 12 to select the capture video vCb in step S104.

[0201] As described previously, the SW controller 10 first causes the switching unit 11 to perform switching in response to the switching trigger KP, and causes the switching unit 12 to perform switching after the switching delay time Td1 has elapsed.

[0202] The switching delay time Td1 will be described below.

[0203] For example, assume that the selected state is a case of switching from camera 502a to camera 502b.

[0204] In this case, the switching delay time Tdl is, for example, the time corresponding to the time lag from when the switching unit 11 selects the background video vBb for the camera 502b to when the captured video vCb obtained by capturing the background video vBb through the camera 502b is input to the switching unit 12.

[0205] Even if the switching unit 11 switches the background video vBa to the background video vBb, the display on the LED wall 505 is not switched to the background video vBb at that moment, and there is a delay until the display is switched.

[0206] That is, as the setting of the operation mode on the LED panel 506 side (hereinafter also referred to as "LED side operation mode"), there are frame rates and various types of signal processing performed by the display controller 590 and the LED processor 570, so that the delay time until the display switching occurs according to these LED side operation modes. For example, in terms of signal processing, for example, the delay time of switching from the switching unit 12 to the switching of the display differs according to the resizing process of the video on the LED wall 505 (LED panel 506) side.

[0207] In addition, the setting of the operation mode related to the imaging of the camera is also referred to as time lag. The operation mode related to the imaging of the camera is an operation mode in the camera 502 or the camera signal processing unit 515. Hereinafter, the operation mode related to the imaging of the camera is also referred to as a "camera-side operation mode". For example, as a setting of the camera 502, a delay occurs due to a frame rate, a shutter speed, a read area from an image sensor, a processing content of the camera signal processing unit 515, and the like.

[0208] For example, Fig.12 An example of a reading range from the image sensor 30 is shown. It is assumed that the solid line indicates the entire pixel area of ​​the image sensor 30. The reading of the photoelectric conversion signal from the image sensor 30 may be implemented in the entire pixel area indicated by the solid line in some cases, or may be implemented in various cases according to the reading mode, such as the range indicated by the dotted line, the range indicated by the dashed line, and the range indicated by the alternating dot-dash line. The delay time is different depending on the difference between these reading ranges. In addition, there is a delay caused by signal processing or resizing processing of the captured video vC.

[0209] With these camera-side operation modes, a time lag occurs from when the background video vBb displayed on the LED wall 505 is captured (exposed) by the camera 502 b until when the captured video vCb is input to the switching unit 12 .

[0210] Therefore, the switching delay time Td1 is set according to the LED side operation mode and the camera side operation mode. As a result, for example, in the case where the switching unit 11 switches the background video vBa to the background video vBb for the camera 502b, the switching unit 12 is switched at the timing when the captured video vCb obtained by capturing the background video vBb by the camera 502b is input to the switching unit 12, and the main line video vCm can be set to the video vCb captured by the camera 502b.

[0211] If the switching timing of the switching unit 12 is advanced, the captured video vCb captured by the camera 502b becomes the main line video vCm when the background video vBa is displayed. This means that a video with a wrong background is included in the main line video vCm. In order to avoid this, an appropriate switching delay time Tdl is set, and Fig.11 The switching timing is controlled as shown in .

[0212] With the above processing, it is possible to switch the camera 502 for capturing the main line video vCm in the shortest time. Specifically, during imaging, the rendering of the background videos vBa and vBb is always performed by the rendering units 21 and 22, so that the background video vB of the switching destination can be output immediately from the switching unit 11 when the switching trigger KP is generated. That is, there is no rendering delay.

[0213] Then, by switching the switching unit 12 at the timing when the background video vB after switching is input to the switching unit 12, the main line video vCm does not become a video with a wrong background.

[0214] exist Fig.10 In the example, the case where imaging is performed by two cameras 502a and 502b is illustrated, but when imaging is performed by three or more cameras 502, the same method is performed. Fig.11 Through the processing in, the video with appropriate background can be set as the main line video vCm.

[0215] However, even in the above manner, the monitoring video vM may still be incorrect.

[0216] For example, assume that camera 502a is selected as the main video vCm. Fig. 9 As shown in FIG. 5B , a background video vB including an inner viewing cone vBCa as a corresponding video of the camera 502 a is displayed on the LED wall 505 .

[0217] In this case, considering the camera 502b in the unselected state, the inner cone vBCb of the corresponding video of the camera 502b is not displayed. Therefore, the monitoring video vM of the camera 502b seen by the photographer, the operator of the switch 600, etc. is a video with a wrong background.

[0218] For example, the inner view frustum vBCa of camera 502a is reflected in the captured video vCb of camera 502b. Alternatively, even if the inner view frustum vBCa is not reflected in the captured video vCb of camera 502b, the monitoring video vM is a video having a background that does not reflect the parallax corresponding to the position of camera 502b.

[0219] By using the viewfinder, monitoring panel, etc. of the camera 502 operated by the photographer, the photographer operates the camera 502 while monitoring the video captured by the camera. Then, as the photographer's operation, when the own camera 502 is not selected as the main line video vCm, the viewing angle is often adjusted by zoom operation, or the position and imaging direction are often changed. At this time, it is inconvenient for the photographer not to be able to visually recognize the correct background in the monitoring video.

[0220] In addition, the operator of the switcher 600 performs the switching operation while viewing the main video vCm and the videos of the other cameras 502 through the multi-screen viewer 612. At this time, it is inconvenient that the videos of the cameras 502 other than the camera 502 selected as the main video vCm do not have the correct background.

[0221] Therefore, in view of these circumstances regarding the monitoring video vM, in this embodiment, even in a case where only the inner visual cone vBC of the camera 502 to be the main line video vCm is displayed on the LED wall 505, the monitoring videos vM of other cameras have the correct background.

[0222] <4. First embodiment>

[0223] Fig.13 An example of a system configuration of the first embodiment is shown. Figure 5 , 7 The same parts as those in 10 are denoted by the same reference numerals, and repeated description is avoided.

[0224] Fig.13 The case where imaging is performed by three cameras 502 a, 502 b, and 502 c is shown. In this case, the videos vCa, vCb, and vCc captured by the respective cameras 502 are input to the switching unit 12 of the switcher 600.

[0225] In response to the imaging of the three cameras 502a, 502b and 502c, in the rendering engine 520, the background videos vBa, vBb and vBc are generated by the corresponding rendering units 21, 22 and 23, respectively. Subsequently, the background videos vBa, vBb and vBc are selected by the switching unit 11.

[0226] Furthermore, in this case, e.g. by Fig.11 In the case where the switching unit 11 is switched by the process in , the SW controller 10 causes the switching unit 12 to perform switching after the switching delay time Td1 has elapsed.

[0227] In the case where the SW controller 10 causes the switching unit 11 to select the background video vBa in step S102, the SW controller 10 causes the switching unit 12 to select the capture video vCa in step S104. Similarly, in the case where the SW controller 10 causes the switching unit 11 to select the background video vBb in step S102, the SW controller 10 causes the switching unit 12 to select the capture video vCb in step S104. Similarly, in the case where the SW controller 10 causes the switching unit 11 to select the background video vBc in step S102, the SW controller 10 causes the switching unit 12 to select the capture video vCc in step S104.

[0228] In this way, in step S104 , the SW controller 10 exercises control so that the switching unit 12 performs switching to the video vC captured by the camera 502 corresponding to the background video vB after the switching of the switching unit 11 .

[0229] It should be noted that although Fig.10 and 13 In the embodiment, the switching units 11 and 12 are configured in one switch 600, and the switch having the switching unit 11 and the switch having the switching unit 12 may be separate devices. In any configuration, the SW controller 10 provided in any switch or in a separate device only needs to be able to implement Fig.11 The switching timing of the switching units 11 and 12 is controlled by the control in.

[0230] exist Fig.13 In the example, except Fig.10 In addition to the configuration in , the video processing unit 18 is provided in the switch 600. It should be noted that in the figure, the video processing unit 18 is included in the switch 600 as an example, but the video processing unit 18 may be configured as a device separate from the switch 600.

[0231] The inner visual cones vBCa, vBCb, and vBCc, which are the corresponding videos of the respective cameras 502a, 502b, and 502c, are output from the rendering units 21, 22, and 23 to the video processing unit 18.

[0232] The video processing unit 18 generates a monitoring video vM (vMa, vMb, vMc) based on the input of the inner visual cones vBCa, vBCb and vBCc, and outputs the monitoring video vM to the multi-screen viewer 612 and the camera signal processing units 515a, 515b and 515c.

[0233] exist Fig.13 In the configuration in , the display of the background video vB and the operation of the camera 502 are as follows.

[0234] The output of the background videos vBa, vBb, and vBc of the rendering units 21, 22, and 23 is set to a frame rate of 60 frames per second (60 fps), for example.

[0235] On the other hand, the input background video vB (60 fps) and the black video also at 60 fps are alternately displayed at 120 fps on the LED wall 505. The black video is a video in which the entire screen is black.

[0236] For example, the LED processor 570 repeatedly displays the background video vB input from the switch 600 on the LED panel 506 during a period of one frame at 120 fps, and then displays the internally generated black video on the LED panel 506 during a period of one frame at 120 fps.

[0237] As a result, if Fig.14 As shown in , a background video vB and a black video BK are alternately displayed on the LED wall 505 in a period T1 of one frame at 120 fps.

[0238] Each camera 502 captures video synchronously on the LED wall 505 side at 120fps.

[0239] As a result, frames including background video vB and objects such as performers 510 and frames including black video BK and objects such as performers 510 are captured alternately. In this case, the camera signal processing unit 515 (515a, 515b, 515c) separates the frames captured at 120fps into odd frames and even frames, and outputs the frames in two systems.

[0240] In a system, as Fig.14 In the captured video vC shown in , each frame is a continuous video in a period T2 of 60fps. This is a video in which the inner visual cone vBCa, the performer 510, etc. are shown in the background video vB.

[0241] In other systems, the video vCbk is captured as a black background, and each frame is a continuous video in the period T2 of 60fps. This is a video in which the black video BK, the performer 510, etc. are shown.

[0242] Therefore, if Fig.13 As shown in , the capture video vCa and the black background capture video vCbka are output from the camera signal processing unit 515a. Similarly, the capture video vCb and the black background capture video vCbkb are output from the camera signal processing unit 515b, and the capture video vCc and the black background capture video vCbkc are output from the camera signal processing unit 515c.

[0243] The captured videos vCa, vCb, and vCc are input to the switching unit 12, and one captured video is selected as the main line video vCm.

[0244] In this case, the captured video vC selected as the main line video vCm is a video with a correct background, but the other captured video vC is a video in which the background (inner visual cone vBC) is wrong and is therefore not used as the monitoring video vM.

[0245] The black background capture videos vCbka, vCbkb, and vCbkc are input to the video processing unit 18. The video processing unit 18 generates monitoring videos vMa, vMb, and vMc by using the black background capture videos vCbka, vCbkb, and vCbkc and the inner cones vBCa, vBCb, and vBCc from the rendering units 21, 22, and 23 described above.

[0246] Rendering units 21, 22 and 23 each implement Fig.15 to output the inner frustum vBCa, vBCb, and vBCc.

[0247] It should be noted that Fig.15 The processing example in the example is obtained by adding step S70 to the reference Figure 6 The processing from step S10 to step S60 is similar to Figure 6 in the processing.

[0248] In step S70 , the rendering units 21 , 22 , and 23 output the inner viewing cone vBC (vBCa, vBCb, vBCc).

[0249] That is, in step S70, the inner view cone vBC generated in step S40 is output without being synthesized with the entire background video vB.

[0250] As described above, for each frame, the background videos vBa, vBb, and vBc are not input to the video processing unit 18, but only the inner viewing cones vBCa, vBCb, and vBCc are input to the video processing unit 18.

[0251] The reason why only the inner viewing cone vBCa, vBCb and vBCc are input to the video processing unit 18 is that the background video within the viewing angle captured by the camera 502 is only the part of the inner viewing cone vBC.

[0252] The video processing unit 18 has the following features: Fig.16 The functional configuration shown in .

[0253] The video processing unit 18 is provided with functions as video processing units 18a, 18b, and 18c corresponding to the three cameras 502a, 502b, and 502c in parallel.

[0254] The video processing units 18 a , 18 b , and 18 c each have a processing function as an object extraction unit 41 and a synthesis unit 42 .

[0255] The black background capture video vCbka captured by the camera 502a and the inner viewing cone vBCa as the corresponding video of the camera 502a are input to the video processing unit 18a.

[0256] The object extraction unit 41 extracts only the video of the object by removing the black video using the brightness key with respect to the input black background capture video vCbka.

[0257] The synthesis unit 42 synthesizes the extracted video of the object and the input inner visual cone vBCa. That is, a process of replacing the black video portion in the black background captured video vCbka with the inner visual cone vBCa is performed.

[0258] Subsequently, the output of the synthesis unit 42 is the monitoring video vMa for the camera 502a.

[0259] In addition, the black background capture video vCbkb captured by the camera 502b and the inner viewing cone vBCb as the corresponding video of the camera 502b are input to the video processing unit 18b.

[0260] The object extraction unit 41 extracts only the video of the object by removing the black video using the brightness key with respect to the input black background capture video vCbkb.

[0261] The synthesis unit 42 synthesizes the extracted video of the object and the input inner view cone vBCb, and replaces the black video portion in the black background capture video vCbkb with the inner view cone vBCb. The output is the monitoring video vMb for the camera 502b.

[0262] In addition, the black background capture video vCbkc captured by the camera 502c and the inner viewing cone vBCc as the corresponding video of the camera 502c are input to the video processing unit 18c.

[0263] The object extraction unit 41 extracts only the video of the object by removing the black video using the brightness key with respect to the input black background capture video vCbkc.

[0264] The synthesis unit 42 synthesizes the extracted video of the object and the input inner view cone vBCc, and replaces the black video portion in the black background capture video vCbkc with the inner view cone vBCc. The output is the monitoring video vMc for the camera 502c.

[0265] Fig.17 is a flow chart showing the processing of the video processing units 18a, 18b, and 18c described above.

[0266] The video processing units 18a, 18b, and 18c repeat the processing from step S201 to step S205 for each frame timing until it is determined in step S200 that the monitoring output is ended.

[0267] In step S201, the video processing unit 18 (18a, 18b, 18c) receives the black background capture video vCbk (vCbka, vCbkb, vCbkc) as input.

[0268] In step S202 , the video processing unit 18 receives the inner viewing cone vBC (vBCa, vBCb, vBCc) as input.

[0269] In step S203 , the video processing unit 18 extracts the video of the object from the black background capture video vCbk.

[0270] In step S204 , the video processing unit 18 synthesizes the video of the object and the inner viewing cone vBC.

[0271] In step S205 , the video processing unit 18 outputs the video in which the background is replaced by the synthesis process as a monitoring video vM (vMa, vMb, vMc).

[0272] like Fig.13 As shown in , surveillance videos vMa, vMb and vMc are provided to the multi-screen viewer 612 and displayed separately.

[0273] In the monitoring videos vMa, vMb, and vMc, the background of objects such as the performer 510 is replaced with the inner viewing cone vBC corresponding to the capture camera 502. Therefore, an operator or the like can monitor a video with a correct background for each camera 502a, 502b, and 502c.

[0274] In addition, monitoring videos vMa, vMb and vMc are respectively provided to camera signal processing units 515a, 515b and 515c, and are respectively provided from camera signal processing units 515a, 515b and 515c to cameras 502a, 502b and 502c, and are displayed on a viewfinder or the like.

[0275] Each cameraman operating cameras 502a, 502b, and 502c can view the video captured by his / her own camera not in the wrong background state in which the video was actually captured, but in the correct background state corresponding to the position and imaging direction of the camera.

[0276] It should be noted that, in the foregoing example, the background video vB and the black video are alternately displayed on the LED wall 505, but the black video is a specific video used to implement video processing for extracting a video of an object in the object extraction unit 41. That is, the black video is a specific video for separating a video of an object within a frame.

[0277] Such a specific video is not limited to a black video. For example, any video that can be used for video separation by a chroma key can be used, and it can be a green video with full green, a blue video with full blue, etc.

[0278] Although the food quality is improved by compositing the green video with the chroma key, when the green video or the blue video is displayed at a high frame rate in a time-division manner, due to the flickering of the green video or the blue video, the visual or psychological burden on the performer 510 may increase. On the other hand, in the case of the black video, it is considered that such a burden on the performer 510 can be reduced.

[0279] It should be noted that, for example, in the case where the video vCbk is captured using a black background and the black is extracted by the brightness key, the black hair of the performer 510 may also be removed, but the video processing unit 18 only generates the monitoring video vM that is not used for the main line video vCm, so it is considered that there is no big problem. In this regard, the specific video is preferably a black video.

[0280] The case of the specific video described above is similar in the second to fifth embodiments.

[0281] <5. Second Embodiment>

[0282] Fig.18 A configuration example of the second embodiment is shown.

[0283] and Fig.13 The difference of the example in is that the video processing units 24 (24a, 24b, 24c) are provided separately from the switch 600, and the video processing units 24a, 24b and 24c each have a rendering function.

[0284] like Fig.18 As shown in , the black background capture videos vCbka, vCbkb and vCbkc output from the camera signal processing units 515a, 515b and 515c are input to the video processing units 24a, 24b and 24c through the switch 600, respectively.

[0285] It should be noted that the black background capture videos vCbka, vCbkb and vCbkc can be directly input to the video processing units 24a, 24b and 24c without passing through the switch 600.

[0286] In addition, Fig.18 , the imaging information IFa, IFb and IFc provided from the camera trackers 560a, 560b and 560c to the rendering engine 520 are also provided to the video processing units 24a, 24b and 24c, respectively.

[0287] Fig.19 The functional configuration of the video processing units 24a, 24b, and 24c is shown.

[0288] The video processing units 24a, 24b, and 24c corresponding to the three cameras 502a, 502b, and 502c each have functions as an object extraction unit 41, a synthesis unit 42, and a rendering unit 43, respectively.

[0289] The object extraction unit 41 and the synthesis unit 42 are reference Fig.16 Describes the functionality.

[0290] Similar to the rendering units 21, 22, and 23 in the rendering engine 520, the rendering unit 43 has a rendering function for generating a video of the inner view cone vBC. However, the rendering unit 43 is different from the rendering units 21, 22, and 23 of the rendering engine 520 in that only the inner view cone vBC needs to be generated, and synthesis with the background video surrounding the inner view cone vBC is not necessary.

[0291] The rendering unit 43 of the video processing unit 24a receives imaging information IFa related to the camera 502a as input, and generates an inner viewing cone vBCa of a corresponding video of the camera 502a which will be a background video based on the viewpoint position of the imaging information IFa.

[0292] Subsequently, in the video processing unit 24a, the synthesis unit 42 synthesizes the video of the object extracted by the object extraction unit 41 and the inner cone vBCa. That is, the black video portion in the black background capture video vCbka is replaced with the inner cone vBCa. Subsequently, the output of the synthesis unit 42 is the monitoring video vMa for the camera 502a.

[0293] The rendering unit 43 of the video processing unit 24b receives imaging information IFb related to the camera 502b as input, and generates an inner viewing cone vBCb of a corresponding video of the camera 502b which will be a background video based on the viewpoint position of the imaging information IFb.

[0294] Subsequently, in the video processing unit 24b, the synthesis unit 42 synthesizes the video of the object extracted by the object extraction unit 41 and the inner cone vBCb, and replaces the black video part in the black background capture video vCbkb with the inner cone vBCb. The output of the synthesis unit 42 is the monitoring video vMb for the camera 502b.

[0295] The rendering unit 43 of the video processing unit 24c receives imaging information IFc related to the camera 502c as input, and generates an inner viewing cone vBCc of a corresponding video of the camera 502c which will be a background video based on the viewpoint position of the imaging information IFc.

[0296] Subsequently, in the video processing unit 24c, the synthesis unit 42 synthesizes the video of the object extracted by the object extraction unit 41 and the inner view cone vBCc, and replaces the black video portion in the black background capture video vCbkc with the inner view cone vBCc. The output of the synthesis unit 42 is the monitoring video vMc for the camera 502c.

[0297] Fig. 20 is a flow chart illustrating the processing of the video processing units 24a, 24b, and 24c described above.

[0298] The video processing units 24a, 24b, and 24c repeat the processing from step S301 to step S306 for each frame until it is determined in step S300 that the monitoring output is ended.

[0299] In step S301, the video processing unit 24 (24a, 24b, 24c) receives imaging information IF (IFa, IFb, IFc) as input.

[0300] In step S302 , the video processing unit 24 generates an inner viewing cone vBC (vBCa, vBCb, vBCc) based on the imaging information IF.

[0301] In step S303 , the video processing unit 24 receives the black background capture video vCbk (vCbka, vCbkb, vCbkc) as input.

[0302] In step S304 , the video processing unit 24 extracts the video of the object from the black background capture video vCbk.

[0303] In step S305 , the video processing unit 24 synthesizes the video of the object and the inner viewing cone vBC generated by the rendering unit 43 .

[0304] In step S306 , the video processing unit 24 outputs the video in which the background is replaced by the synthesis process as the monitoring video vM (vMa, vMb, vMc).

[0305] like Fig.18 As shown in , surveillance videos vMa, vMb, and vMc are provided to the multi-screen viewer 612 and are each displayed.

[0306] In addition, the monitoring videos vMa, vMb, and vMc are provided to the camera signal processing units 515a, 515b, and 515c, respectively, and are provided to the cameras 502a, 502b, and 502c, respectively, from the camera signal processing units 515a, 515b, and 515c, and are displayed on the viewfinder, etc. It should be noted that the monitoring videos vMa, vMb, and vMc can be provided to the camera signal processing units 515a, 515b, and 515c through the switch 600.

[0307] As previously described, the operator and cameraman of each camera 502 can monitor the video for the correct background.

[0308] <6. Third embodiment>

[0309] Fig.21 A configuration example of the third embodiment is shown.

[0310] This is a configuration in which the rendering engine 520 renders only the background video vB of the camera 502 selected as the main line video vCm among the plurality of cameras 502 .

[0311] Imaging information IFa, IFb, and IFc from the camera trackers 560 a , 560 b , and 560 c are selectively provided to the rendering engine 520 through the selector 28 .

[0312] The selector 28 selects one item of the imaging information IFa, IFb, and IFc by the control signal C3 of the SW controller 10 from the switch 600 , and supplies the selected imaging information to the rendering engine 520 .

[0313] The rendering engine 520 generates an inner viewing cone vBC based on the input imaging information IF, generates a background video vB including the inner viewing cone vBC, and outputs the background video vB to the display controller 590 .

[0314] The switch 600 in this example includes a switching unit 12 and a SW controller 10 .

[0315] The SW controller 10 implements control of the selector 28 according to the selection of the main line video vCm by the switching unit 12. That is, during the period in which the switching unit 12 selects the captured video vCa of the camera 502a as the main line video vCm, the SW controller 10 causes the selector 28 to select the imaging information IFa. Similarly, in the period in which the captured video vCb of the camera 502b is selected as the main line video vCm, the SW controller 10 causes the selector 28 to select the imaging information IFb, and in the period in which the captured video vCc of the camera 502c is selected as the main line video vCm, causes the selector 28 to select the imaging information IFc.

[0316] As a result, no matter which camera 502 is selected, the main line video vCm becomes a video with the correct background. In addition, since the rendering engine 520 only needs to synthesize the inner cone vBC corresponding to one camera 502, for example, Fig.18 The configuration in the figure has a smaller processing load than the one in the figure.

[0317] In addition, Fig.21 In the configuration, similar to Fig.18 The imaging information IFa, IFb and IFc from the camera trackers 560a, 560b and 560c are also provided to the video processing units 24a, 24b and 24c. The functional configuration and processing of the video processing units 24a, 24b and 24c are similar to those of FIG. Fig.19 and 20 Similar to .

[0318] As a result, monitoring videos vMa, vMb, and vMc corresponding to the three cameras 502a, 502b, and 502c are generated and provided to the multi-screen viewer 612. In addition, the monitoring videos vMa, vMb, and vMc are provided to the cameras 502a, 502b, and 502c through the camera signal processing units 515a, 515b, and 515c, and are displayed on a viewfinder or the like.

[0319] As previously described, the operator and cameraman of each camera 502 can monitor the video for the correct background.

[0320] <7. Fourth embodiment>

[0321] Fig. 22 A configuration example of the fourth embodiment is shown. Fig. 22 The configuration example in is similar to Fig.13 , but the processing of the rendering units 21, 22 and 23 in the rendering engine 520 is different.

[0322] Imaging information IFa, IFb, and IFc are input to the rendering units 21, 22, and 23, respectively, and parameters PMTa, PMTb, and PMTc are input from the camera signal processing units 515a, 515b, and 515c, respectively.

[0323] The parameters PMT (PMTa, PMTb, PMTc) mentioned herein are parameters related to the video of the camera, and are particularly parameters that affect the brightness and color of the video. Specific examples thereof include values ​​such as exposure and white balance that the photographer uses to implement operations. Alternatively, the parameters may be adjustments to color tone, values ​​of video effects, and the like.

[0324] exist Fig.13 In the case of the first embodiment in FIG. 1 , the monitoring videos vMa, vMb, and vMc are obtained by synthesizing the inner cones vBCa, vBCb, and vBCc by the video processing unit 18, but the inner cones vBCa, vBCb, and vBCc are videos in which the exposure adjustment and the white balance adjustment in the cameras 502a, 502b, and 502c are not reflected. Therefore, even if the photographer performs an adjustment operation of the exposure or white balance for the camera 502, the adjustment is not reflected in the background portion of the monitoring videos vMa, vMb, and vMc. In addition, the brightness and hue of the background portion and the object portion may be different in the monitoring videos vMa, vMb, and vMc. For these reasons, it is inconvenient for the photographer to operate the exposure adjustment and the white balance adjustment.

[0325] Therefore, in the fourth embodiment, in order to reflect the photographer's operation in the monitoring video vM, the parameters PMTa, PMTb and PMTc are provided to the rendering units 21, 22 and 23. Then, the rendering units 21, 22 and 23 implement the following Fig.23 The processing shown in .

[0326] Fig.23 The processing example in is obtained by adding steps S71 and S72 to the reference Figure 6 The processing from step S10 to step S60 is the same as Figure 6 Similar to .

[0327] In step S71, the rendering units 21, 22, and 23 perform signal processing on the inner viewing cone vBC (vBCa, vBCb, vBCc) generated in step S40 using parameters PMTa, PMTb, and PMTc. That is, processing is performed to obtain a video in a state where exposure and white balance are adjusted.

[0328] Subsequently, in step S72 , the rendering units 21 , 22 , and 23 output the inner viewing cone vBC′ subjected to the signal processing in step S71 .

[0329] As described above, in addition to the background videos vBa, vBb and vBc, the inner view cones vBCa', vBCb' and vBCc' subjected to video signal processing reflecting the parameters PMTa, PMTb and PMTc are output from the rendering units 21, 22 and 23, respectively.

[0330] It should be noted that the background videos vBa, vBb, and vBc provided to the LED wall 505 are not subjected to video processing that reflects the parameters PMTa, PMTb, and PMTc. This is because the exposure and white balance adjusted at the time of imaging by the camera 502 are then reflected.

[0331] The video processing unit 18 uses the reference Fig.16 The described functional configuration implementation reference Fig.17 The inner viewing cones vBCa', vBCb' and vBCc' are input, however, subject to video processing reflecting the parameters PMTa, PMTb and PMTc.

[0332] Therefore, the generated monitoring videos vMa, vMb, and vMc are background videos reflecting the exposure and white balance adjusted by the photographer. As a result, the monitoring videos vMa, vMb, and vMc become more appropriate videos.

[0333] It should be noted that Fig. 22 A remote controller 615 is shown, and outputs an operation signal RC to, for example, camera signal processing units 515a, 515b, and 515c. By operating the camera 502 itself or by using the remote controller 615, a photographer, an operator, etc. can implement exposure adjustment, white balance adjustment, color tone adjustment, video effects, etc.

[0334] In this case, it is preferable that the processing of the rendering engine 520 is implemented by the parameter PMT corresponding to the operation.

[0335] <8. Fifth embodiment>

[0336] Fig.24 A configuration example of the fifth embodiment is shown. This configuration example is similar to Fig.21 , and is an example in which parameters PMTa, PMTb, and PMTc are reflected in the video processing units 24a, 24b, and 24c.

[0337] exist Fig.24In the case of, parameters PMTa, PMTb and PMTc from camera signal processing units 515a, 515b and 515c are input to video processing units 24a, 24b and 24c, respectively.

[0338] Fig.25 The functional configuration of the video processing units 24a, 24b, and 24c is shown.

[0339] The video processing units 24a, 24b, and 24c corresponding to the three cameras 502a, 502b, and 502c each have processing functions as an object extraction unit 41, a synthesis unit 42, a rendering unit 43, and a signal processing unit 44, respectively.

[0340] The object extraction unit 41, the synthesis unit 42 and the rendering unit 43 are reference Fig.19 Functionality as described.

[0341] The signal processing unit 44 performs signal processing according to the parameter PMT on the inner viewing cone vBC generated by the rendering unit 43 .

[0342] The rendering unit 43 of the video processing unit 24a receives the imaging information IFa related to the camera 502a as an input, and generates an inner view cone vBCa of the corresponding video of the camera 502a which will be a background video based on the viewpoint position of the imaging information IFa. For the inner view cone vBCa, the signal processing unit 44 implements signal processing according to the parameter PMTa, and the inner view cone vBCa' is provided to the synthesis unit 42. Subsequently, the synthesis unit 42 synthesizes the video of the object extracted by the object extraction unit 41 and the inner view cone vBCa', and outputs as a monitoring video vMa for the camera 502a.

[0343] The rendering unit 43 of the video processing unit 24b receives the imaging information IFb related to the camera 502b as an input, and generates an inner view cone vBCb of the corresponding video of the camera 502b which will be a background video based on the viewpoint position of the imaging information IFb. For the inner view cone vBCb, the signal processing unit 44 implements signal processing according to the parameter PMTb, and the inner view cone vBCb' is provided to the synthesis unit 42. Subsequently, the synthesis unit 42 synthesizes the video of the object extracted by the object extraction unit 41 and the inner view cone vBCb', and outputs as a monitoring video vMb for the camera 502b.

[0344] The rendering unit 43 of the video processing unit 24c receives the imaging information IFc related to the camera 502c as an input, and generates an inner view cone vBCc of the corresponding video of the camera 502c which will be a background video based on the viewpoint position of the imaging information IFc. For the inner view cone vBCc, the signal processing unit 44 implements signal processing according to the parameter PMTc, and the inner view cone vBCc' is provided to the synthesis unit 42. Subsequently, the synthesis unit 42 synthesizes the video of the object extracted by the object extraction unit 41 and the inner view cone vBCc', and outputs as a monitoring video vMc for the camera 502c.

[0345] Fig.26 is a flow chart illustrating the processing of the video processing units 24a, 24b, and 24c described above.

[0346] The video processing units 24a, 24b, and 24c repeat the processing from step S301A to step S306 for each frame timing until it is determined in step S300 that the monitoring output is ended.

[0347] In step S301A, the video processing unit 24 (24a, 24b, 24c) receives the imaging information IF (IFa, IFb, IFc) and the parameter PMT (PMTa, PMTb, PMTc) as input.

[0348] In step S302 , the video processing unit 24 generates an inner viewing cone vBC (vBCa, vBCb, vBCc) based on the imaging information IF.

[0349] In step S310 , the video processing unit 24 performs signal processing on the inner viewing cone vBC according to the parameter PMT.

[0350] Step S305 and subsequent steps are Fig. 20 However, the inner frustum vBCa', vBCb' and vBCc' will be composited.

[0351] By displaying the monitoring video vM obtained by the previous processing, the operator and photographer of each camera 502 can monitor the video having the correct background and reflecting the exposure adjustment and white balance adjustment of the camera 502.

[0352] <9. Summary and modification examples>

[0353] According to the above-described embodiments, the following effects can be obtained.

[0354] In the embodiment, the video processing unit 18 ( 18 a , 18 b , 18 c ) and the video processing unit 24 ( 24 a , 24 b , 24 c ) are described as examples of the information processing device 70 .

[0355] The information processing device 70 receives as input a black background capture video vCbk (first capture video) from among videos obtained by capturing the following items by the camera 502: an inner view cone vBC (corresponding video) corresponding to one camera 502 among the plurality of cameras 502, a display video on the LED wall 505 that displays a black video (specific video) in a time-division manner, and an object. Subsequently, a video process of replacing the black video included in the input black background capture video vCbk with the inner view cone vBC generated corresponding to the camera 502 that captured the video is implemented.

[0356] In the case where a background video vB including an inner view frustum vBC as a corresponding video of one camera is captured by a plurality of cameras 502, a captured video vC of a camera 502 not corresponding to the inner view frustum vBC is not a video having a correct background. Therefore, a black background captured video vCbk is input, and a black video portion is replaced with the inner view frustum vBC corresponding to each camera 502. As a result, videos having a correct background can be obtained for all cameras 502.

[0357] It should be noted that in the first to fifth embodiments described above, not the monitoring video vM generated by the video processing unit 18 or the video processing unit 24, but the main line video vCm may be used as the monitoring video vM for the video as the main line video vCm. That is, the monitoring video vM for the camera 502 may be generated by the video processing unit 18 or the video processing unit 24, wherein the captured video vC is not the main line video vCm.

[0358] In the first to fifth embodiments, the captured video vC (second captured video) including the object and the inner visual cone vBC is a video that can be selected as the main line video vCm, which constitutes the content of the video obtained by capturing the display video on the LED wall 505 and objects such as performers 510 with the camera 502.

[0359] The main line video vCm is a video obtained by the switcher 600 among videos obtained by capturing the inner visual cone vBC displayed on the LED wall 505 with each camera 502. That is, the original video as the virtual production is output as the main line video vCm.

[0360] In the first to fifth embodiments, the video generated by the video processing by the video processing unit 18 or the video processing unit 24 is output as the monitoring video vM for monitoring display.

[0361] For example, by replacing the black video of the background with the inner viewing cone vBC to obtain the monitoring video vM (vMa, vMb, vMc), the monitoring video vM of each camera 502 becomes a video including the correct background and objects for each camera 502.

[0362] In the first to fifth embodiments, the monitoring video vM generated by the video processing of the video processing unit 18 or the video processing unit 24 is output as a video to be used for monitoring display in the camera 502.

[0363] By configuring so that the video obtained by replacing the black video of the background with the inner visual cone vBC is fed back to the camera 502 as the monitoring video vM (vMa, vMb, vMc), the monitoring video vM of all cameras 502 becomes a video including the correct background and objects for the camera 502 regardless of the display on the LED wall 505.

[0364] Therefore, the photographer can appropriately implement the angle of view adjustment, movement of the imaging position and the imaging direction, etc. while checking the correct background state. Specifically, this is extremely appropriate when the photographer implements the angle of view adjustment, angle adjustment, etc. in a state where the captured video vC of the camera 502 operated by the photographer is not the main line video vCm.

[0365] In the first to fifth embodiments, the monitoring video vM generated by the video processing of the video processing unit 18 or the video processing unit 24 is output as a video to be used for monitoring display in the multi-screen viewer 612, which displays multiple videos on one screen in a time-division manner.

[0366] The monitoring video vM (vMa, vMb, vMc) in which the black video of the background is replaced by the inner visual cone vBC is provided to and displayed on the multi-screen viewer 612. As a result, an operator or the like operating the switcher 600 can visually recognize the captured video of each camera 502 with an appropriate background, thereby making it possible to implement a switching operation while appropriately determining the state of each captured video vC.

[0367] In the first to fourth embodiments, an example is described in which the video processing unit 18 performs processing to replace the black video portion with the input inner viewing cone vBC for each frame of the black background capture video vCbk (see Figures 13 to 17 , 22 and 23).

[0368] For example, with reference Figures 13 to 17As in the described example, the video processing unit 18 receives as input the inner view frustum vBC provided from the rendering engine 520, and synthesizes the inner view frustum vBC and the object video extracted from the black background capture video vCbk. As a result, it is possible to generate a video in which the black background is replaced by the inner view frustum vBC.

[0369] In the first to fourth embodiments, the inner view cone vBC for each camera 502 is generated by the rendering engine 520, and a plurality of generated inner view cone vBCs are selectively displayed on the LED wall 505. In this configuration, an example is described in which the video processing unit 18 performs a process of replacing a black video portion with the inner view cone vBC (or vBC′) supplied from the rendering engine 520 for each frame of the black background capture video vCbk (see Figures 13 to 17 , 22 and 23).

[0370] For example, with reference Figures 13 to 17 As in the example described, the rendering units 21, 22 and 23 of the rendering engine 520 generate background videos vBa, vBb and vBc, each including the inner viewing cone vBC corresponding to the camera. These background videos are selected by the switch 600 and displayed on the LED wall 505.

[0371] In the case of this configuration, the video processing unit 18 receives the inner view frustum vBC provided from the rendering engine 520 as an input, and synthesizes the inner view frustum vBC and the object video extracted from the black background capture video vCbk.

[0372] In the case of such a system configuration, since background videos vBa, vBb, and vBc corresponding to the selected camera are always generated after switching to correspond to each camera by the switcher 600, the new background video vB can be immediately displayed on the LED wall 505 when switching is implemented by the switcher 600. This is because the generation of the background video vB of another camera is not started in response to the switching instruction of the switcher 600. Therefore, the delay for generating the background video vB at the time of switching of the switcher 600 can be eliminated, and smooth switching can be implemented.

[0373] Furthermore, even in a period not selected by the switch 600, the inner view cone vBC generated by the rendering units 21, 22, and 23 for each camera is utilized for monitoring without being wasted.

[0374] In the second, third and fifth embodiments, the video processing unit 24 implements a process of generating an inner viewing cone vBC corresponding to the camera 502 that captured the black background capture video vCbk, and a process of replacing the black video portion of each frame of the black background capture video vCbk with the generated inner viewing cone vBC (see Figures 18 to 20, 21 and 24 to 26).

[0375] For example, with reference Figures 18 to 20 As in the described example, in the video processing unit 24, the rendering unit 43 generates the inner view frustum vBC of the corresponding camera 502, and synthesizes the inner view frustum vBC and the video of the object extracted from the black background capture video vCbk. As a result, it is possible to generate a video in which the black background is replaced by the inner view frustum vBC.

[0376] In this case, the video processing unit 24 only needs to be able to render the inner cone vBC for generating the monitoring video vM in the rendering unit 43. That is, since the monitoring video vM is not used as the main line video vCm, the video quality required for the main line video vCm is not required, and only the video quality required for the monitoring video vM is necessary. Therefore, the rendering unit 43 only needs to implement video processing with a relatively light load, and there is no requirement for the processing capabilities of the rendering units 21, 22, and 23 of the rendering engine 520. Therefore, the video processing unit 24 including the rendering unit 43 can be configured using a relatively inexpensive information processing device.

[0377] In the system configuration described in the third and fifth embodiments, only the inner view cone vBC of the selected camera 502 among the plurality of cameras 502 performing imaging is generated by the rendering engine 520, and the background video vB including the generated inner view cone vBC is displayed on the LED wall 505. Subsequently, the video processing unit 24 performs a process of generating the inner view cone vBC corresponding to the camera 502 that captured the black background capture video vCbk, and a process of replacing the black video portion of each frame of the black background capture video vCbk with the generated inner view cone vBC (or vBC') (see Fig.21 and 24 to 26).

[0378] For example, with Fig.21 As in the example in , the rendering engine 520 generates a background video vB for the camera 502 selected by the switching unit 12 of the switch 600.

[0379] With such a configuration, in the rendering engine 520, the processing load for generating a high-definition background video vB to be included in the main line video vCm can be reduced. This is because, for example, there is no need to generate a high-definition background video vB. Fig.18 Background videos vBa, vBb, and vBc corresponding to multiple cameras 502 are generated as in the example in FIG.

[0380] Furthermore, even in the case of such a system configuration, by including the video processing unit 24 and the video processing unit 24 including the rendering unit 43 , a monitoring video vM having a correct background can be generated for each camera 502 .

[0381] In the fourth and fifth embodiments, the video processing unit 18 or the video processing unit 24 implements a process of replacing the black video portion of each frame of the black background capture video vCbk with the inner view cone vBC (vBCa', vBCb', vBCc'), wherein the signal processing according to the parameter PMT related to the video of the camera 502 that captures the black background capture video vCbk is applied to the inner view cone (see Figures 22 to 26 ).

[0382] As a result, when the exposure, white balance, etc. on the camera 502 side are changed through the photographer's operation, automatic adjustment, etc., the changes are also reflected in the monitoring video vM, so that correct monitoring display according to the current parameters can be achieved.

[0383] In the fourth embodiment, an inner view cone vBC for each camera 502 is generated by the rendering engine 520, and a plurality of generated inner view cones vBC are selectively displayed on the LED wall 505. In the example described later, the video processing unit 18 implements a process of replacing a black video portion of each frame of a black background capture video vCbk with an inner view cone vBC (vBCa', vBCb', vBCc') to which the rendering engine 520 applies signal processing according to the parameter PMT (see Fig. 22 and 23 ).

[0384] In the rendering units 21, 22 and 23 of the rendering engine 520, video signal processing according to parameters PMT such as exposure and white balance is performed on the video data of the inner viewing cone vBC, so that correct monitoring display serving as a background according to the adjustment operation of the camera 502 can be achieved.

[0385] In the fifth embodiment, the video processing unit 24 generates an inner viewing cone vBC corresponding to the camera 502 that captures the black background capture video vCbk, performs signal processing according to the parameter PMT, and replaces the black video portion of the black background capture video vCbk with the generated inner viewing cone vBC (see Figures 24 to 26 ).

[0386] The video processing unit 24 performs signal processing according to the parameter PMT on the inner visual cone vBC. By synthesizing the processed inner visual cone vBC (vBCa', vBCb', vBCc') and the object video, it is possible to realize a correct monitoring display serving as a background according to the adjustment operation of the camera 502, etc. It is possible to realize a correct monitoring display serving as a background according to the parameter PMT.

[0387] In the imaging system 500 of the embodiment, the display video on the LED wall 505 is a video in which a black video as the specific video and an inner view cone vBC as the corresponding video (including a background video vB of the inner view cone vBC) are alternately displayed for each frame. Subsequently, each camera 502 captures each frame in synchronization with the display, and outputs a black background capture video vCbk in which each frame including the black video is continuous, and a capture video vC in which each frame including the inner view cone vBC is continuous in parallel. In such an imaging system 500, the video processing unit 18 and the video processing unit 24 only need to receive the black background capture video vCbk as input, and replace each frame with the inner view cone vBC corresponding to the camera 502.

[0388] The specific video in the embodiment is the video used for the video separation process.

[0389] By using a video that can be separated by a chroma key or the like, such as a black video, a green video, or a blue video, as the specific video, video processing for generating the monitoring video vM becomes easy.

[0390] Furthermore, in the embodiment, the specific video used in the video separation process is a black video, and the black background captured video vCbk is a video in which an object is captured having the black video as a background.

[0391] By using a black video as the specific video, compared with a blue video and a green video, the influence of a video change on the LED wall 505 in a high frequency period on the performer 510 can be mitigated.

[0392] The corresponding video in the embodiment is a background video generated according to the position or imaging direction of the camera 502. That is, the corresponding video is an inner visual cone vBC rendered based on imaging information including the position and imaging direction of the camera 502 with respect to the LED wall 505. The monitoring video vM of each camera is a video in which the inner visual cone vBC is optimized.

[0393] The program of the embodiment is used to make a processor such as a CPU, a DSP, or a device including a processor execute the above-described Fig.17 , 20 and the processing procedures shown in 26.

[0394] That is, the program of the embodiment causes the information processing device 70 such as the video processing units 18 and 24 to perform video processing, the information processing device 70 receiving as input a black background capture video vCbk (first capture video) including an object and a specific video among videos obtained by capturing a display video on the LED wall 505 and an object such as a performer 510 by a camera 502, the display video displaying in a time-division manner an inner visual cone vBC generated corresponding to one camera 502 among a plurality of cameras 502 and a specific video such as a black video. The video processing is a video processing of replacing a specific video included in the black background capture video vCbk with the inner visual cone vBC generated corresponding to the camera 502 that captured the black background capture video vCbk.

[0395] Using such a program, the information processing apparatus 70 that performs the video processing described above can be implemented by various computer devices.

[0396] Such a program may be recorded in advance in a HDD as a recording medium built into a device such as a computer device, a ROM in a microcomputer having a CPU, or the like. In addition, such a program may be temporarily or permanently stored (recorded) in a removable recording medium such as a flexible disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium may be provided as a so-called software package.

[0397] Furthermore, such a program may be installed from a removable recording medium into a personal computer or the like, or may be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0398] In addition, such a program is suitable for providing the information processing device 70 of the embodiment in a wide range. For example, by downloading the program to a personal computer, a communication device, a portable terminal device such as a smartphone or a tablet, a mobile phone, a game device, a video device, a personal digital assistant (PDA), etc., these devices can function as the information processing device 70 of the present disclosure.

[0399] It should be noted that the effects described in this specification are merely examples and not limitations, and other effects may be provided.

[0400] It should be noted that the present technology may also have the following configurations.

[0401] (1) An information processing device, comprising a video processing unit, wherein the video processing unit implements video processing for replacing a first captured video including an object and a specific video with a corresponding video, wherein the corresponding video is generated corresponding to a camera that captures the first captured video, wherein the first captured video is a video among videos obtained by capturing, by the camera, a display video on a display and an object separated from the display, and the display displays the corresponding video generated corresponding to one camera among a plurality of cameras and the specific video in a time-division manner.

[0402] (2) An information processing device according to (1), wherein the second captured video is a video that can be selected as a main video, the second captured video including an object and a corresponding video generated by a camera among videos obtained by capturing a display video and the object on a display by a camera.

[0403] (3) The information processing device according to (1) or (2), wherein the video generated by the video processing is output as a video to be used for monitoring display.

[0404] (4) The information processing device according to any one of (1) to (3), wherein the video generated by the video processing is output as a video to be used for monitoring display in a camera.

[0405] (5) The information processing device according to any one of (1) to (4), wherein the video generated by the video processing is output as a video to be used for a monitoring display that displays a plurality of videos in a divided manner on one screen.

[0406] (6) The information processing device according to any one of (1) to (5), wherein a process of replacing a part of the specific video with the input corresponding video for each frame of the first captured video is implemented in the video processing.

[0407] (7) The information processing device according to any one of (1) to (6), wherein

[0408] Generate a corresponding video for each camera through a rendering engine, and selectively display a plurality of generated corresponding videos on a display, and

[0409] A process of replacing a portion of a specific video of each frame of the first captured video with a corresponding video provided from the rendering engine is implemented in the video processing.

[0410] (8) The information processing device according to any one of (1) to (5), wherein the following processing is implemented in the video processing:

[0411] generating a corresponding video corresponding to the camera that captured the first captured video, and

[0412] A process of replacing a portion of the specific video with the generated corresponding video for each frame of the first captured video.

[0413] (9) The information processing device according to any one of (1) to (5), wherein in a system configuration in which the rendering engine generates only corresponding videos for a camera selected from among a plurality of cameras that perform imaging and displays the generated corresponding videos on a display,

[0414] The following processing is performed in the video processing:

[0415] generating a corresponding video corresponding to the camera that captured the first captured video, and

[0416] A process of replacing a portion of the specific video with the generated corresponding video for each frame of the first captured video.

[0417] (10) An information processing device according to any one of (1) to (9), wherein, in the video processing, a process of replacing a portion of the specific video with a corresponding video is implemented for each frame of the first captured video, wherein signal processing based on parameters related to the video of the camera that captured the first captured video is applied to the corresponding video.

[0418] (11) The information processing device according to any one of (1), (2), (3), (4), (5), (6), (7) and (10), wherein

[0419] Generate a corresponding video for each camera through a rendering engine, and selectively display a plurality of generated corresponding videos on a display, and

[0420] A process of replacing a portion of a specific video of each frame of a first captured video with a corresponding video is implemented in the video processing, wherein signal processing according to parameters related to a video of a camera that captured the first captured video is applied to the corresponding video by the rendering engine.

[0421] (12) An information processing device according to any one of (1), (2), (3), (4), (5), (8), (9) and (10), wherein, in the video processing, a corresponding video corresponding to a camera that captures a first captured video is generated, signal processing is performed on the generated corresponding video according to parameters related to the video of the camera that captures the first captured video, and a portion of a specific video of the first captured video is replaced by the corresponding video.

[0422] (13) The information processing device according to (2), wherein

[0423] Displaying video on the display includes video in which the specific video and the corresponding video are alternately displayed for each frame, and

[0424] Each camera receives a first captured video as input and implements the video processing in an imaging system, wherein the first captured video and a second captured video are output in parallel in the imaging system, the frames including the specific video in the first captured video are continuous, and the frames including the corresponding video in the second captured video are continuous.

[0425] (14) The information processing device according to any one of (1) to (13), wherein the specific video includes a video to be used for video separation processing.

[0426] (15) The information processing device according to any one of (1) to (13), wherein

[0427] The specific video includes a black video, and

[0428] The first captured video includes a video in which the object is captured having a black video as a background.

[0429] (16) The information processing device according to any one of (1) to (15), wherein the corresponding video includes a background video generated according to a position or an imaging direction of a camera.

[0430] (17) An information processing method, comprising implementing video processing for replacing a first captured video including an object and a specific video with a corresponding video, wherein the corresponding video is generated corresponding to a camera that captures the first captured video, the first captured video being a video among videos obtained by capturing, by the camera, a display video on a display and an object separated from the display, the display displaying the corresponding video and the specific video generated corresponding to one camera among a plurality of cameras in a time-division manner.

[0431] (18) An imaging system comprising:

[0432] monitor;

[0433] a plurality of cameras that capture video displayed on the display; and

[0434] an information processing device for processing the videos captured by the plurality of cameras,

[0435] wherein the display displays the corresponding video and the specific video generated by one of the plurality of cameras in a time-division manner, and

[0436] The information processing device includes a video processing unit, which implements video processing for replacing a first captured video including an object and a specific video with a corresponding video, wherein the corresponding video is generated corresponding to a camera that captures the first captured video, and the first captured video is a video among videos obtained by capturing a display video on a display and an object separated from the display by a camera.

[0437] Reference numerals list

[0438] 10——Switch controller (SW controller)

[0439] 11 - Switching unit

[0440] 12 - Switching unit

[0441] 18, 18a, 18b, 18c - Video processing unit

[0442] 21, 22, 23 - Rendering Unit

[0443] 24, 24a, 24b, 24c - Video processing unit

[0444] 28 - Selector

[0445] 41——Object Extraction Unit

[0446] 42 - Synthesis Unit

[0447] 43 - Rendering Unit

[0448] 44——Signal processing unit

[0449] 70——Information processing equipment

[0450] 71——CPU

[0451] 85——Video processing unit

[0452] 502——Camera

[0453] 505——LED Wall

[0454] 510 - Performer

[0455] 515a, 515b——Camera signal processing unit

[0456] 520——Rendering Engine

[0457] 600——Switcher

[0458] 612 - Multi-viewer

[0459] vB——Background video

[0460] vBC - Capture Zone Video (Inner View Frustum)

[0461] vC, vCa, vCb, vCc——Capture video

[0462] vCbk - Capture video with black background

[0463] vCm——Mainline Video

[0464] vM, vMa, vMb, vMc——monitoring video

Claims

1. An information processing device, comprising a video processing unit, wherein the video processing unit implements video processing for replacing a first captured video including an object and a specific video with a corresponding video, wherein the corresponding video is generated corresponding to a camera that captures the first captured video, wherein the first captured video is a video among videos obtained by capturing, by the camera, a display video on a display and an object separated from the display, and the display displays the corresponding video generated by one camera among a plurality of cameras and the specific video in a time-division manner.

2. The information processing device according to claim 1, wherein: The second captured video includes a video that may be selected as a main line video, the second captured video including an object and a corresponding video generated corresponding to one camera among videos obtained by capturing the display video and the object on the display through the camera.

3. The information processing device according to claim 1, wherein: The video generated by the video processing is output as a video to be used for monitoring display.

4. The information processing device according to claim 1, wherein: The video generated by the video processing is output as a video to be used for monitoring display in a camera.

5. The information processing device according to claim 1, wherein: The video generated by the video processing is output as a video to be used for a monitoring display that displays a plurality of videos in a divided manner on one screen.

6. The information processing device according to claim 1, wherein: In the video processing, a process of replacing a portion of the specific video with the input corresponding video for each frame of the first captured video is implemented.

7. The information processing device according to claim 1, wherein: Generate a corresponding video for each camera through a rendering engine, and selectively display a plurality of generated corresponding videos on a display, and A process of replacing a portion of a specific video of each frame of the first captured video with a corresponding video provided from the rendering engine is implemented in the video processing.

8. The information processing device according to claim 1, wherein: The following processing is performed in the video processing: generating a corresponding video corresponding to the camera that captured the first captured video, and A process of replacing a portion of the specific video with the generated corresponding video for each frame of the first captured video.

9. The information processing device according to claim 1, wherein: In a system configuration in which the rendering engine generates only corresponding videos for a camera selected from among a plurality of cameras implementing imaging and displays the generated corresponding videos on a display, The following processing is performed in the video processing: generating a corresponding video corresponding to the camera that captured the first captured video, and A process of replacing a portion of the specific video with the generated corresponding video for each frame of the first captured video.

10. The information processing device according to claim 1, wherein: In the video processing, a process of replacing a portion of the specific video with a corresponding video is implemented for each frame of the first captured video, wherein signal processing according to parameters related to a video of a camera that captured the first captured video is applied to the corresponding video.

11. The information processing device according to claim 1, wherein Generate a corresponding video for each camera through a rendering engine, and selectively display a plurality of generated corresponding videos on a display, and A process of replacing a portion of a specific video of each frame of a first captured video with a corresponding video is implemented in the video processing, wherein signal processing according to parameters related to a video of a camera that captured the first captured video is applied to the corresponding video by the rendering engine.

12. The information processing device according to claim 1, wherein: In the video processing, a corresponding video corresponding to a camera that captures a first captured video is generated, signal processing according to parameters related to the video of the camera that captures the first captured video is performed on the generated corresponding video, and a portion of a specific video of the first captured video is replaced with the corresponding video.

13. The information processing device according to claim 2, wherein Displaying video on the display includes video in which the specific video and the corresponding video are alternately displayed for each frame, and Each camera receives a first captured video as input and implements the video processing in an imaging system, wherein the first captured video and a second captured video are output in parallel in the imaging system, the frames including the specific video in the first captured video are continuous, and the frames including the corresponding video in the second captured video are continuous.

14. The information processing device according to claim 1, wherein: The specific video includes a video to be used for a video separation process.

15. The information processing device according to claim 1, wherein The specific video includes a black video, and The first captured video includes a video in which the object is captured having a black video as a background.

16. The information processing device according to claim 1, wherein: The corresponding video includes a background video generated according to the position or imaging direction of the camera.

17. An information processing method, comprising implementing video processing for a first captured video including an object and a specific video to replace the specific video with a corresponding video, wherein the corresponding video is generated corresponding to a camera that captures the first captured video, and the first captured video is a video among videos obtained by capturing, by the camera, a display video on a display and an object separated from the display, and the display displays the corresponding video generated by one camera among a plurality of cameras and the specific video in a time-division manner.

18. An imaging system comprising: monitor; multiple cameras capturing video displayed on a display; as well as an information processing device for processing the videos captured by the plurality of cameras, wherein the display displays the corresponding video and the specific video generated by one of the plurality of cameras in a time-division manner, and The information processing device includes a video processing unit, which implements video processing for replacing a first captured video including an object and a specific video with a corresponding video, wherein the corresponding video is generated corresponding to a camera that captures the first captured video, and the first captured video is a video among videos obtained by capturing a display video on a display and an object separated from the display by a camera.

Images