Switching device, control method, and imaging system

Abstract

A switcher apparatus includes: a first switching unit that receives, as an input, a plurality of video data including video content corresponding to each of a plurality of cameras, selects one of the plurality of videos, and outputs the selected video data as video data to be displayed on a display; a second switching unit that receives, as an input, a plurality of video data obtained by capturing videos displayed on the display by the plurality of cameras, selects one of the plurality of videos, and outputs the selected video data; and a control unit. The control unit performs control to cause the first switching unit to perform switching of the video data in response to triggering of the video switching, and after a predetermined switching delay time, cause the second switching unit to perform switching of the camera-derived video data corresponding to the video data after the switching in the first switching unit.

Description

Technical Field

[0001] The present technology relates to a switcher device for video data, a method for controlling the switcher device, and an imaging system including the switcher device. Background Art

[0002] As an imaging method for producing video contents such as movies, there is known a technique in which performers perform with a so-called green screen and then a background video is composited.

[0003] Furthermore, in recent years, instead of green screen shooting, an imaging system has been developed in which, in a studio equipped with a large monitor, a background video is displayed on the monitor and performers perform in front of it, so that the performers and the background can be imaged. This imaging system is called so-called virtual production, in-camera VFX, or light emitting diode (LED) wall virtual production.

[0004] The following Patent Document 1 discloses a technology of a system for imaging a performer performing in front of a background video.

[0005] Citation List

[0006] Patent Literature

[0007] Patent Document 1: U.S. Patent Application Publication No. 2020 / 0145644A Summary of the invention

[0008] Problems to be solved by the present invention

[0009] When a background video is displayed on a large monitor and then the performers and background video are filmed with a camera, there is no need to synthesize the background video separately after filming. The performers and staff can visually understand the scene and perform or judge the quality of the performance, which is more advantageous than shooting with a green screen.

[0010] However, the appearance of the background should be different according to the position of the camera relative to the display and the imaging direction. When the background video is simply projected, the background does not change even if the position of the camera, the imaging direction, etc. are different, and the video becomes quite unnatural. Therefore, by changing the background video (at least the video in the range of the camera's viewing angle in the display) according to the position of the camera, the imaging direction, etc., so as to be equivalent to the appearance of the actual three-dimensional space, a video equivalent to the case of imaging at the actual position can be shot.

[0011] However, in the case of using multiple cameras, since the positions of the cameras are physically different, the backgrounds suitable for the corresponding cameras may overlap on the display, or when the camera serving as the main line video is switched by a switcher, the background videos of other cameras may be projected.

[0012] Therefore, the present disclosure proposes a technology for obtaining appropriate video in a system that shoots videos displayed on a display using a plurality of cameras and switches main line videos by a switcher.

[0013] Solution to the problem

[0014] A switcher device according to the present technology includes: a first switching unit, which receives as input a plurality of video data including video contents corresponding to each of a plurality of cameras, selects one of the plurality of video data, and outputs the selected video data as video data to be displayed on a display; a second switching unit, which receives as input a plurality of video data obtained by shooting videos displayed on the display by the plurality of cameras, selects one of the plurality of video data, and outputs the selected video data; and a control unit, which controls to execute switching of the video data of the first switching unit in response to a trigger of video switching, and, after a predetermined switching delay time has passed, causes the second switching unit to execute switching to the video data obtained by the camera corresponding to the video data after the switching in the first switching unit.

[0015] In the case of imaging with multiple cameras, for example, a background video or the like having video content corresponding to the position and viewing direction of each camera is prepared and selectively provided to the display via a switcher device for display. Each camera captures a video of the display, and the second switching unit selects and outputs the video. In this case, after the switching of the first switching unit, the second switching unit is switched after a predetermined switching delay time. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is an explanatory diagram of an imaging system according to an embodiment of the present technology.

[0017] Figure 2 is an explanatory diagram of a background video of a camera position of an imaging system according to an embodiment.

[0018] Figure 3 is an explanatory diagram of a background video of a camera position of an imaging system according to an embodiment.

[0019] Figure 4 It is an explanatory diagram of the video content production steps of the embodiment.

[0020] Figure 5 is a block diagram of an imaging system of an embodiment.

[0021] Figure 6 is a flow chart of background video generation of the imaging system of an embodiment.

[0022] Figure 7 is a block diagram of an imaging system using multiple cameras of an embodiment.

[0023] Figure 8 is a block diagram of an information processing apparatus of an embodiment.

[0024] Fig. 9 is an illustration of the overlap of the inner viewing frustum of each camera.

[0025] Fig.10 It is a block diagram of a first configuration example including a switch according to the embodiment.

[0026] Fig.11 This is a flowchart of the switching control of the switch of the embodiment.

[0027] Fig.12 It is an explanatory diagram of an example of processing that affects the switching delay time.

[0028] Fig.13 It is a block diagram of a second configuration example including the switch of the embodiment.

[0029] Fig.14 1 is a flowchart of a first setting process example of the switching delay time applied in the embodiment.

[0030] Fig.15 It is a block diagram of a third configuration example including the switch of the embodiment.

[0031] Fig.16 4 is a flowchart of a second example of a setting process of a switching delay time applied in the embodiment.

[0032] Fig.17 This is a flowchart of a third example of a setting process of a switching delay time applied in the embodiment.

[0033] Fig.18 This is an explanatory diagram of global shutter exposure.

[0034] Fig.19 This is a diagram for explaining exposure using the rolling shutter method.

[0035] Fig. 20 This diagram is an explanatory diagram of video signals output by the rolling shutter method and the global shutter method.

[0036] Fig.21 This is a flowchart of an example of controlling a camera according to the embodiment. DETAILED DESCRIPTION

[0037] Hereinafter, the embodiments will be described in the following order.

[0038] <1. Imaging system and content production>

[0039] <2. Composition of information processing equipment>

[0040] <3. Imaging system using multiple cameras and switchers>

[0041] <4. Delay time setting process>

[0042] <5. Shutter mode>

[0043] <6. Summary and Modifications>

[0044] Note that in the present disclosure, "video" or "image" includes both still images and moving images. In addition, "video" refers not only to the state in which the video is displayed on the display, but also video data in the state in which the video data is not displayed on the display can also be collectively referred to as "video".

[0045] For example, in an embodiment, the background video before being displayed on the display, the video shot by the camera, and the background video or shot video switched by the switcher are not the actual displayed videos, but video data. For convenience, such video data are referred to as "background video", "shot video", etc.

[0046] In addition, multiple cameras, circuit units, functions, signals, etc. are represented by adding "a", "b", "c"... to the figure markings so that they are written as "camera 502a" and "camera 502b", and when referred to collectively, only the figure markings excluding "a", "b" and "c" are used, such as "camera 502".

[0047] <1. Imaging system and video content production>

[0048] An imaging system and production of video content to which the technology of the present disclosure can be applied will be described.

[0049] Figure 1 The schematic diagram illustrates an imaging system 500. The imaging system 500 is a system that performs imaging as a virtual production, and a part of equipment arranged in an imaging studio is illustrated in the figure.

[0050] In the imaging studio, a performance area 501 is provided in which a performer 510 performs a performance, such as a show. Large display devices are arranged at least on the back, left and right sides, and the top of the performance area 501. Although the device type of the display device is not limited, the figure illustrates an example in which an LED wall 505 is used as an example of a large display device.

[0051] One LED wall 505 forms a large panel by vertically and horizontally connecting and arranging a plurality of LED panels 506. The size of the LED wall 505 is not particularly limited, but only needs to be a size required or sufficient as the size of the display background when imaging the performer 510.

[0052] A necessary number of lamps 580 are arranged at necessary positions such as above or at the side of the performance area 501 to illuminate the performance area 501 .

[0053] Near the performance area 501, for example, a camera 502 for imaging a movie or other video content is arranged. A camera operator 512 can move the position of the camera 502 and can operate the imaging direction, viewing angle, etc. Of course, it is also conceivable to move the camera 502, operate the viewing angle, etc. by remote control. In addition, the camera 502 can automatically or autonomously move or change the viewing angle. Therefore, the camera 502 can be installed on a camera pan head or a mobile body.

[0054] Camera 502 jointly captures performers 510 in performance area 501 and video displayed on LED wall 505. For example, by displaying scenery as background video vB on LED wall 505, it is possible to capture a video similar to the situation where performers 510 actually exist at the location of the scenery and perform at the location.

[0055] The output monitor 503 is arranged near the performance area 501. The video shot by the camera 502 is displayed in real time as a monitor video vM on the output monitor 503. As a result, the director and the staff who produce the video content can check the shot video.

[0056] As described above, imaging system 500 for imaging a performance of performer 510 against a backdrop of LED wall 505 in an imaging studio has various advantages over green screen filming.

[0057] For example, in the case of green screen shooting, it is difficult for the performer to imagine the background and the conditions of the scene, which may affect the performance. On the other hand, by displaying the background video vB, the performer 510 can easily perform and improve the quality of the performance. In addition, the director and other staff can easily determine whether the performance of the performer 510 matches the background or the conditions of the scene.

[0058] In addition, post-production after imaging is more efficient compared to the case of green screen shooting. This is because so-called chroma key synthesis may not be necessary, or color correction or image synthesis may not be necessary. In addition, even in the case where chroma key synthesis is required during imaging, there is no need to add a background screen, which also contributes to efficiency.

[0059] In the case of green screen shooting, a green tint is added to the performer's body, clothing, and objects, so it is necessary to correct it. In addition, in the case of green screen shooting, if there is an object such as glass, mirror, or snow top that reflects the surrounding scene, it is necessary to generate and synthesize the reflected image, but this is a troublesome task.

[0060] On the other hand, in using Figure 1 In the case where the imaging system 500 in the image processing system 500 performs imaging, the green hue will not increase, so correction is unnecessary. In addition, by displaying the background video vB, the image on the actual article such as glass is naturally obtained and photographed, so there is no need for a synthetic image video.

[0061] Here, we will refer to Figure 2 and Figure 3 Description of background video vB. Even if the background video vB is displayed on the LED wall 505 and filmed together with the performer 510, the background of the filmed video becomes unnatural by simply displaying the background video vB. This is because a three-dimensional background having depth is actually used in a planar manner as the background video vB.

[0062] For example, the camera 502 can image the performer 510 in the performance area 501 from various directions, and can also perform a zoom operation. The performer 510 will not stay in one place. Then, the actual appearance of the background of the performer 510 should change according to the position, imaging direction, viewing angle, etc. of the camera 502, but such a change cannot be obtained in the background video vB which is a flat video. Therefore, the background video vB is changed so that the background is similar to the actual appearance including parallax.

[0063] Figure 2 The diagram illustrates a state where a camera 502 is imaging a performer 510 from a position on the left side of the diagram. Figure 3 The diagram illustrates a state in which a camera 502 images a performer 510 from a position on the right side of the diagram. In each diagram, a shooting area video vBC is illustrated in a background video vB.

[0064] Note that the portion of the background video vB excluding the shooting area video vBC is called the "outer viewing cone", while the shooting area video vBC is called the "inner viewing cone".

[0065] The background video vB described here indicates the entire video including the shooting area video vBC (inner cone) displayed as the background.

[0066] The range of the shooting area video vBC (inner viewing cone) corresponds to the range actually imaged by the camera 502 in the display surface of the LED wall 505. Then, the shooting area video vBC is a video that is transformed according to the position, imaging direction, viewing angle, etc. of the camera 502 so as to express the scene actually seen when the position of the camera 502 is set as the viewpoint.

[0067] Specifically, three-dimensional (3D) background data, which is a 3D model as a background, is prepared, and the shooting area video vBC is sequentially rendered in real time based on the viewpoint position of the camera 502 with respect to the 3D background data.

[0068] Note that the range of the shooting area video vBC is actually a slightly wider range than the range imaged by the camera 502 at that point in time. This is to prevent the video of the outer viewing cone from being imaged due to rendering delay and to avoid the influence of diffracted light from the video of the outer viewing cone when the imaging range is slightly changed by panning, tilting, zooming, etc. of the camera 502.

[0069] The video of the shooting area video vBC rendered in real time in this way is synthesized with the video of the outer cone. The video of the outer cone used in the background video vB is rendered in advance based on the 3D background data, and the video is merged into a part of the video of the outer cone as the real-time rendered shooting area video vBC to generate the entire background video vB.

[0070] As a result, even when the camera 502 is moved forward and backward or left and right, or a zoom operation is performed, the background of the range imaged together with the performer 510 is captured as a video corresponding to the change in viewpoint position accompanying the actual movement of the camera 502.

[0071] like Figure 2 and Figure 3 As shown in the diagram, the monitor video vM including the performer 510 and the background is displayed on the output monitor 503, which is the filmed video. The background of the monitor video vM is the filming area video vBC. That is, the background included in the filmed video is a real-time rendered video.

[0072] As described above, in the imaging system 500 of this embodiment, the background video vB including the shooting area video vBC is changed in real time, so that not only the background video vC is simply displayed in a two-dimensional manner, but also a video similar to the situation of imaging actually performed on site can be shot.

[0073] Note that by rendering in real time only the shooting area video vBC as the range reflected by the camera 502, rather than the entire background video vB displayed on the LED wall 505, the processing load of the system is also reduced.

[0074] Here, the production steps of a video content as a virtual production in which imaging is performed by the imaging system 500 will be described. Figure 4 As shown in the diagram, the video content production steps are roughly divided into three stages. These stages are asset creation ST1, production ST2, and post-production ST3.

[0075] Asset creation ST1 is a step of producing 3D background data for displaying background video vB. As described above, background video vB is generated by rendering in real time using 3D background data at the time of imaging. For this purpose, 3D background data as a 3D model is produced in advance.

[0076] Examples of methods of producing 3D background data include full computer graphics (CG), point cloud data (point cloud) scanning, and photogrammetry.

[0077] Full CG is a method of creating 3D models using computer graphics. Among the three methods, full CG requires the most man-hours and time, but is preferably used when you want to use unrealistic videos or videos that are difficult to shoot in practice as background videos vB.

[0078] Point cloud data scanning is a method of generating a 3D model based on point cloud data by measuring the distance from a certain position using, for example, LiDAR, taking a 360-degree image with a camera from the same position, and placing the color data taken with the camera at the points measured with LiDAR. Compared to full CG, 3D models can be created in a short time. In addition, it is easy to create 3D models with higher resolution than photogrammetry.

[0079] Photogrammetry is a method of photogrammetry that analyzes parallax information to obtain size and shape from two-dimensional images obtained by photographing an object from multiple viewpoints. 3D models can be created in a short time.

[0080] Note that in 3D data generation by photogrammetry, point cloud information acquired with LiDAR can be used.

[0081] In asset creation ST1, for example, a 3D model as 3D background data is produced using these methods. Of course, the above methods can be used in combination. For example, a part of a 3D model produced by point cloud data scanning or photogrammetry is produced and synthesized by CG, and so on.

[0082] ST2 is made in Figure 1 The steps of imaging in the imaging studio shown in the diagram. The essential technologies in this case include real-time rendering, background display, camera tracking, lighting control, etc.

[0083] Real-time rendering is used to obtain the rendering process of the shooting area video vBC at each time point (each frame of the background video vB), as shown in the reference Figure 2 and Figure 3 This will render the 3D background data produced in asset creation ST1 from a viewpoint corresponding to the position of the camera 502 at each point in time, etc.

[0084] In this way, real-time rendering is performed to generate a background video vB including each frame of the shooting area video vBC, and the background video vB is displayed on the LED wall 505.

[0085] Camera tracking is performed to obtain imaging information of the camera 502, and the position information, imaging direction, viewing angle, etc. of the camera 502 at each time point are tracked. By providing imaging information including such information to the rendering engine in association with each frame, real-time rendering corresponding to the viewpoint position of the camera 502, etc. can be performed.

[0086] Imaging information is information linked or associated with the video as metadata.

[0087] It is assumed that the imaging information includes the position information of the camera 502 at each frame timing, the direction of the camera, the viewing angle, the focal length, the F value (aperture value), the shutter speed, the lens information, and the like.

[0088] Lighting control is to control the state of lighting in the imaging system 500, specifically, to control the light quantity, light color, lighting direction, etc. of the lamp 580. For example, lighting control is performed according to the time setting, location setting, etc. of the scene to be imaged.

[0089] Post-production ST3 indicates various processing after imaging, such as video correction, video adjustment, clip editing, video effects, etc.

[0090] As video correction, color gamut conversion, color matching between cameras and materials, etc. are available.

[0091] As video adjustments, color adjustment, brightness adjustment, contrast adjustment, etc. can be performed.

[0092] As clip editing, you can cut clips, adjust their order, adjust their duration, etc.

[0093] As a video effect, CG video or special effect video can be synthesized.

[0094] Next, the configuration of the imaging system 500 used in producing ST2 will be described.

[0095] Figure 5 Is a graphic description with reference Figure 1 , Figure 2 and Figure 3 A block diagram of an imaging system 500 is described that outlines the configuration of the imaging system 500 .

[0096] Figure 5 The imaging system 500 shown in the diagram includes the above-mentioned LED wall 505, a camera 502, an output monitor 503 and a lamp 580, wherein the LED wall 505 includes a plurality of LED panels 506. Figure 5 As shown in the diagram, the imaging system 500 also includes a rendering engine 520 , an asset server 530 , a synchronization generator 540 , an operation monitor 550 , a camera tracker 560 , an LED processor 570 , a lighting controller 581 , and a display controller 590 .

[0097] Each LED processor 570 is disposed corresponding to one or more LED panels 506 , and performs video display driving of the corresponding one or more LED panels 506 .

[0098] The synchronization generator 540 generates a synchronization signal for synchronizing the frame timing of the display video of the LED panel 506 with the frame timing of the imaging of the camera 502, and provides the synchronization signal to the corresponding LED processor 570 and the camera 502. However, this does not prevent the output of the synchronization generator 540 from being provided to the rendering engine 520.

[0099] The camera tracker 560 generates imaging information of the camera 502 at each frame timing, and provides the imaging information to the rendering engine 520. For example, the camera tracker 560 detects the position information of the camera 502 relative to the LED wall 505 or the predetermined reference position, and the imaging direction of the camera 502 as one of the imaging information, and provides them to the rendering engine 520.

[0100] As a specific detection method of the camera tracker 560, there is a method of randomly arranging reflectors on the ceiling and detecting the position based on the reflected light of infrared light irradiated to the reflectors from the camera tracker 560 mounted on the camera 502. In addition, as a detection method, there is also a method of estimating the own position of the camera 502 by information of a camera pan / tilt mounted on the camera 502 or a main body of the camera 502, or by image recognition of a video shot by the camera 502.

[0101] In addition, the viewing angle, focal length, F value, shutter speed, lens information, etc. can be provided from the camera 502 to the rendering engine 520 as imaging information.

[0102] The asset server 530 is a server capable of storing the 3D model created in the asset creation ST1, that is, 3D background data, on a recording medium and reading the 3D model as needed. That is, the asset server 530 functions as a database (DB) of 3D background data.

[0103] The rendering engine 520 performs a process of generating a background video vB to be displayed on the LED wall 505. Therefore, the rendering engine 520 reads necessary 3D background data from the asset server 530. Then, the rendering engine 520 generates a video of an outer viewing cone used in the background video vB as a video obtained by rendering the 3D background data in a form viewed from pre-specified spatial coordinates.

[0104] Furthermore, as a process of each frame, the rendering engine 520 specifies the viewpoint position and the like relative to the 3D background data using the imaging information provided from the camera tracker 560 or the camera 502 , and renders the shooting area video vBC (inner viewing cone).

[0105] In addition, the rendering engine 520 synthesizes the pre-generated outer viewing cone with the captured area video vBC rendered for each frame to generate a background video vB as video data of one frame. Then, the rendering engine 520 sends the generated video data of one frame to the display controller 590.

[0106] The display controller 590 generates a divided video signal nD obtained by dividing the video data of one frame into video parts to be displayed on the corresponding LED panel 506, and transmits the divided video signal nD to the corresponding LED panel 506. At this time, the display controller 590 can perform calibration according to individual differences in color rendering, etc. between display units, manufacturing errors, etc.

[0107] Note that the display controller 590 may not be provided, and the rendering engine 520 may perform these processes. That is, the rendering engine 520 may generate the segmented video signal nD, perform calibration, and send the segmented video signal nD to the corresponding LED panel 506.

[0108] The LED processor 570 drives the corresponding LED panel 506 based on the corresponding received segmented video signal nD, and the entire background video vB is displayed on the LED wall 505. The background video vB includes a shooting area video vBC rendered according to the position of the camera 502 at that time point.

[0109] The camera 502 can film the performance of the performer 510, including the background video vB displayed on the LED wall 505 in this manner. The video obtained by the imaging of the camera 502 is recorded on a recording medium in the camera 502 or an external recording device (not shown), and is provided to the output monitor 503 in real time and displayed as a monitor video vM.

[0110] The operation monitor 550 displays an operation image vOP for controlling the rendering engine 520. The engineer 511 can perform necessary settings and operations regarding rendering of the background video vB while viewing the operation image vOP.

[0111] The lighting controller 581 controls the light intensity, light color, irradiation direction, etc. of the lamp 580. For example, the lighting controller 581 can control the lamp 580 asynchronously with the rendering engine 520, or can control it synchronously with the imaging information and rendering process. Thus, the lighting controller 581 can perform light emission control according to instructions from the rendering engine 520, a main controller (not shown), etc.

[0112] Figure 6 The diagram illustrates an example of processing by the rendering engine 520 in the imaging system 500 having such a configuration.

[0113] In step S10 , the rendering engine 520 reads the 3D background data to be used this time from the asset server 530 , and expands the 3D background data in the internal workspace.

[0114] Then, a video is generated to be used as the outer viewing frustum.

[0115] Thereafter, the rendering engine 520 repeats the processing from step S30 to step S60 at each frame timing of the background video vB until it is determined in step S20 that the display of the background video vB based on the read 3D background data is finished.

[0116] In step S30, the rendering engine 520 acquires imaging information from the camera tracker 560 and the camera 502. As a result, the position and state of the camera 502 to be reflected in the current frame are checked.

[0117] In step S40, the rendering engine 520 performs rendering based on the imaging information. That is, the viewpoint position relative to the 3D background data is specified based on the position, imaging direction, viewing angle, etc. of the camera 502 to be reflected in the current frame, and rendering is performed. At this time, video processing reflecting the focal length, F value, shutter speed, lens information, etc. can also be performed. Through this rendering, video data as the shooting area video vBC can be obtained.

[0118] In step S50, the rendering engine 520 performs a process of synthesizing the outer view cone as the entire background video and the video reflecting the viewpoint position of the camera 502, that is, the shooting area video vBC. For example, this process is to synthesize the video of the entire background rendered at a specific reference viewpoint with the video generated by reflecting the viewpoint of the camera 502. Through this process, a background video vB of one frame displayed on the LED wall 505 is generated, that is, the background video vB including the shooting area video vBC.

[0119] The processing in step S60 is performed by the rendering engine 520 or the display controller 590. In step S60, the rendering engine 520 or the display controller 590 generates a segmented video signal nD obtained by segmenting a background video vB of one frame into videos to be displayed on each LED panel 506. Calibration may be performed. Then, each segmented video signal nD is sent to the corresponding LED processor 570.

[0120] Through the above-mentioned processing, the background video vB including the shooting area video vBC shot by the camera 502 is displayed on the LED wall 505 at each frame timing.

[0121] By the way, Figure 5 Only one camera 502 is illustrated in FIG. 5 , but imaging may be performed by multiple cameras 502 . Figure 7 The diagram illustrates a configuration example in which a plurality of cameras 502a and 502b are used. The cameras 502a and 502b can independently perform imaging in the performance area 501. In addition, synchronization between the cameras 502a and 502b and the corresponding LED processors 570 is maintained by the synchronization generator 540.

[0122] The output monitors 503a and 503b are provided corresponding to the cameras 502a and 502b, and are configured to display the videos captured by the corresponding cameras 502a, 502b as monitor videos vMa and vMb, respectively.

[0123] In addition, camera trackers 560a and 560b are respectively set corresponding to cameras 502a and 502b, respectively detecting the positions and imaging directions of the corresponding cameras 502a and 502b. Imaging information from camera 502a and camera tracker 560a and imaging information from camera 502b and camera tracker 560b are sent to rendering engine 520.

[0124] The rendering engine 520 may perform rendering using imaging information from the camera 502 a or the camera 502 b to obtain a background video vB for each frame.

[0125] Note that although Figure 7The illustration illustrates an example using two cameras 502a and 502b, but three or more cameras 502 may be used for imaging.

[0126] However, when a plurality of cameras 502 are used, there is a case where the shooting area video vBC corresponding to each camera 502 interferes. Figure 7 As shown in the diagram, in the example of using two cameras 502a and 502b, the shooting area video vBC corresponding to the camera 502a is illustrated, but in the case of using the video of the camera 502b, the shooting area video vBC corresponding to the camera 502b is also necessary. When the shooting area video vBC corresponding to each of the cameras 502a and 502b is simply displayed, they interfere with each other. Therefore, it is necessary to plan the display of the shooting area video vBC.

[0127] <2. Composition of information processing equipment>

[0128] Next, we will refer to Figure 8 A configuration example of the information processing device 70 that can be used for asset creation ST1, production ST2, and post-production ST3 is described.

[0129] The information processing device 70 is a device capable of performing information processing, particularly video processing, such as a computer device. Specifically, as the information processing device 70, a personal computer, a workstation, a portable terminal device such as a smartphone and a tablet, a video editing device, etc. are envisioned. In addition, the information processing device 70 may be a computer device configured as a server device or an arithmetic device in cloud computing.

[0130] In the case of the present embodiment, specifically, the information processing device 70 can function as a 3D model making device that makes a 3D model in the asset creation ST1.

[0131] Furthermore, the information processing apparatus 70 may also function as the rendering engine 520 and the asset server 530 constituting the imaging system 500 used in producing ST2.

[0132] Furthermore, the information processing device 70 can also function as a video editing device that performs various types of video processing in post-production ST3.

[0133] In addition, the information processing device 70 is also used as a reference later. Fig.10 The switch 600 described in the examples above is a switch 600. The switch 600 includes not only a switching circuit based on a signal of hardware but also the functions of the information processing device 70 for performing control and calculation.

[0134] Figure 8The CPU 71 of the information processing device 70 illustrated in the figure executes various types of processing according to a program stored in a nonvolatile storage unit 74, such as a ROM 72 or, for example, an electrically erasable programmable read-only memory (EEP-ROM), or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to execute various types of processing.

[0135] The video processing unit 85 is configured as a processor that performs various types of video processing. For example, the video processing unit 85 is a processor that can perform any one or multiple types of processing among 3D model generation processing, rendering, DB processing, video editing processing, video analysis / detection processing, etc.

[0136] The video processing unit 85 may be implemented by, for example, a CPU separate from the CPU 71 , a graphics processing unit (GPU), a general purpose computing graphics processing unit (GPGPU), an artificial intelligence (AI) processor, or the like.

[0137] Note that the video processing unit 85 may be provided as a function in the CPU 71 .

[0138] The CPU 71, the ROM 72, the RAM 73, the nonvolatile storage unit 74, and the video processing unit 85 are connected to one another via a bus 83. To the bus 83, an input / output interface 75 is also connected.

[0139] An input unit 76 including operating members and operating devices is connected to the input / output interface 75. As the input unit 76, for example, various operating elements and operating devices including a keyboard, a mouse, keys, knobs, a touch panel, a touch pad, a remote controller, etc. are contemplated.

[0140] The input unit 76 detects a user operation, and the CPU 71 interprets a signal corresponding to the input operation.

[0141] A microphone is also conceived as the input unit 76. It is also possible to input a voice uttered by a user as operation information.

[0142] Furthermore, a display unit 77 including a liquid crystal display (LCD), an organic electroluminescence (EL) panel, or the like, and an audio output unit 78 including a speaker or the like are connected to the input / output interface 75 integrally or separately.

[0143] The display unit 77 is a display unit that performs various displays, and includes, for example, a display device provided in a housing of the information processing device 70 , a separate display device connected to the information processing device 70 , and the like.

[0144] The display unit 77 displays various images, operation menus, icons, messages, and the like on a display screen, ie, displays as a graphical user interface (GUI), based on instructions from the CPU 71 .

[0145] In some cases, a storage unit 79 including a hard disk drive (HDD), a solid-state memory, or the like, or a communication unit 80 is connected to the input / output interface 75 .

[0146] The storage unit 79 can store various data and programs. A DB can also be configured in the storage unit 79.

[0147] For example, in the case where the information processing device 70 functions as the asset server 530 , a DB storing a 3D background data group may be constructed using the storage unit 79 .

[0148] The communication unit 80 performs communication processing via a transmission path such as the Internet, wired / wireless communication with various devices such as an external DB, an editing device, and an information processing device, bus communication, and the like.

[0149] For example, in a case where the information processing device 70 functions as the rendering engine 520 , the communication unit 80 may access the DB as the asset server 530 , and receive imaging information from the camera 502 or the camera tracker 560 .

[0150] Furthermore, in the case of the information processing device 70 used in the post-production ST3 , the communication unit 80 can also access a DB or the like as the asset server 530 .

[0151] The drive 81 is also connected to the input / output interface 75 as necessary, and a removable recording medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted.

[0152] The drive 81 can read video data, various computer programs, and the like from the removable recording medium 82. The read data is stored in the storage unit 79, and the video and audio included in the data are output by the display unit 77 and the audio output unit 78. In addition, the computer program and the like read from the removable recording medium 82 are installed in the storage unit 79 as needed.

[0153] In the information processing apparatus 70, for example, software for the processing of the present embodiment may be installed via network communication through the communication unit 80 or the removable recording medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0154] <3. Imaging system using multiple cameras and switchers>

[0155] In this embodiment, for example, it is assumed that a plurality of cameras 502 (502a, 502b, ...) are used for imaging, as shown in FIG. Figure 7 described.

[0156] The captured videos vC of multiple systems obtained by capturing the LED wall 505, performers 510, etc. by multiple cameras 502 are selectively switched by the switcher 600 described later and output as so-called main line videos. Note that, of course, each captured video vC can be recorded or transmitted separately from the main line video.

[0157] Here, what occurs when two or more cameras 502 are used for imaging in virtual production will be described.

[0158] As described above, the background video vB captured by the camera 502 is not the entire background video vB displayed on the LED wall 505, but the range of the captured area video vBC (hereinafter also referred to as “inner visual cone vBC”).

[0159] Then, the video content of the inner visual cone vBC is rendered frame by frame by the rendering engine 520 according to the position and imaging direction of the camera 502, incorporated into the overall background video vB, and displayed on the LED wall 505. Figure 2 and Figure 3 As described above, the range and content of the inner visual cone vBC of the background video vB differ according to the camera position of each frame, etc. In addition, the inner visual cone vBC corresponding to each camera 502 is displayed.

[0160] Here, considering that imaging is performed by two cameras 502a and 502b, the inner cones vBCa and vBCb corresponding to the respective cameras 502a and 502b are displayed on the LED wall 505. Depending on the positions of the cameras 502a and 502b, the inner cones vBCa and vBCb may overlap, such as Fig. 9 As shown in the diagram.

[0161] If the inner visual cones vBCa and vBCb do not overlap, the video vC captured by each camera 502a and 502b is a video including an appropriate background. Fig. 9 If there is overlap as shown in the diagram, the video content of the background of each captured video vC is not in a correct state.

[0162] Therefore, consider a method in which the background video vB for each camera 502 is staggered in the time axis direction and displayed on the LED wall 505, and the shutter phase / angle is adjusted so that the desired video can be shot on the side of the camera 502 to be shot.

[0163] For example, in the case of imaging by four cameras 502a, 502b, 502c, and 502d, the background video vB of camera 502a, the background video vB of camera 502b, the background video vB of camera 502c, and the background video vB of camera 502d are displayed on the LED wall 505 in each 1 / 4 period of one frame, and each camera 502a, 502b, 502c, and 502d is exposed synchronously therewith. For example, camera 502a is exposed in the period of displaying the inner cone vBC of camera 502a.

[0164] In this way, there will be no Fig. 9 The inconvenience caused by the overlap of the inner viewing cone vBC shown in the figure.

[0165] However, since the video is multiplexed in the time direction, the number of cameras that can simultaneously shoot video is limited. It is difficult to increase the number of cameras 502.

[0166] Furthermore, various videos are switched on the actual LED wall 505 during a frame period, which is not an environment in which the performer 510 standing in front of the LED wall 505 can easily perform.

[0167] In addition, since the exposure time is 1 / 2 in the case of 2 cameras and 1 / 4 in the case of 4 cameras, the exposure is reduced, the gain of the captured video vC needs to be increased, and the noise increases. Alternatively, in order to perform sufficient exposure, excessive lighting is required, and the brightness of the LED wall 505 needs to be increased at the same time.

[0168] Thus, in the present embodiment, in the case of performing imaging using a plurality of cameras 502, first, basically, the inner view cone vBC for one specific camera 502 is always displayed on the LED wall 505. Specifically, the background video vB including the inner view cone vBC corresponding to the camera 502 serving as the main line video is displayed. When the main line video is switched, the background video vB is also switched accordingly. That is, the background video vB including the inner view cone vBC for the switched camera 502 is displayed.

[0169] However, in this case, the background videos vB of the plurality of systems including the inner view frustum vBC of the corresponding cameras 502 are continuously generated in parallel and selectively displayed on the LED wall 505. That is, the generation of the inner view frustum vBC also continues for the camera 502 that does not serve as the main line video. As a result, the video delay caused in the case where the rendering corresponding to the new camera 502 is started when the main line video is switched is eliminated.

[0170] In addition, when switching the main line video, the switching timing of the background video vB and the switching timing of the main line video are controlled so that the state in which the video vC shot by a certain camera 502 includes the background video vB of a different camera 502 is not output as the main line video.

[0171] Fig.10 An example of a configuration for this purpose is illustrated in FIG.

[0172] Fig.10 is Figure 7 On the premise of the configuration in , it is obtained by adding a switch 600 for switching the captured video vC and an output device 610. Figure 5 or Figure 7 The constituent parts described in the present invention are denoted by the same reference numerals, and repeated description is avoided.

[0173] For cameras 502a and 502b, camera signal processing units 515a and 515b are illustrated that perform signal processing of captured video signals. Figure 5 and Figure 7 Although omitted in the figure, the camera signal processing units 515a and 515b may be formed by a processor or the like in the cameras 502a and 502b, or may be provided as a device of a unit separate from the cameras 502a and 502b.

[0174] The video signal shot by the camera 502a is subjected to development processing, size adjustment processing, etc. by the camera signal processing unit 515a, and is input as a shot video vCa to the switch 600. The video signal shot by the camera 502b is subjected to development processing, size adjustment processing, etc. by the camera signal processing unit 515b, and is input as a shot video vCb to the switch 600.

[0175] In addition, imaging information including camera positions, imaging directions, and the like of camera trackers 560a and 560b is provided to the rendering engine 520. Although not shown, the viewing angles, focal lengths, f-values, shutter speeds, lens information, and the like of cameras 502a and 502b are also included in the imaging information and provided to the rendering engine 520.

[0176] The rendering engine 520 includes one or more information processing devices 70. In this case, the rendering engine 520 is configured to have a plurality of rendering functions as the rendering units 21 and 22, and is configured to be able to simultaneously perform rendering corresponding to at least the camera 502a or 502b.

[0177] The rendering unit 21 renders the inner visual cone vBCa corresponding to the camera 502a based on the imaging information about the camera 502a, incorporates the inner visual cone vBCa into the entire background, and outputs the background video vBa matching the camera 502a.

[0178] In addition, the rendering unit 22 renders the inner visual cone vBCb corresponding to the camera 502b based on the imaging information about the camera 502b, incorporates the inner visual cone vBCb into the entire background, and outputs the background video vBb matching the camera 502b.

[0179] The switch 600 is provided with a switching unit 11, and the switching unit 11 receives background videos vBa and vBb as input, selects one of the background videos vBa and vBb, and outputs the selected video. The background video vBa or vBb selected by the switching unit 11 is the background video vB provided to the LED wall 505.

[0180] As above reference Figure 5 and Figure 7 As described above, the background video vB is processed by the display controller 590 and distributed to the plurality of LED processors 570, each LED panel 506 ( Fig.10 The background video vB is displayed on the LED wall 505.

[0181] Cameras 502 a and 502 b capture a background video vB of the LED wall 505 and a performer 510 .

[0182] As described above, videos vCa and vCb shot by cameras 502a and 502b are input to the switcher 600. The switcher 600 is provided with a switching unit 12 for inputting the shot videos vCa and vCb. The switcher unit 12 selects one of the shot videos vCa and vCb and outputs the selected video as a main line video vCm.

[0183] The main line video vCm is provided to the output device 610 .

[0184] Here, the output device 610 may be a recording device that records the main line video vCm on a recording medium, or may be a video transmission device that broadcasts and transmits the main line video vCm. In addition, the output device 610 may be a web server that distributes the main line video vCm, etc.

[0185] The switcher 600 includes a switching unit 11 that selects the background video vB and a switching unit 12 that selects the main line video vCm as described above, and includes a switcher controller (hereinafter referred to as a “SW controller”) 10 as a control unit that controls the switching units 11 and 12.

[0186] The SW controller 10 may be composed of Figure 8 The SW controller 10 only needs to have at least Figure 8The CPU 71, ROM 72, RAM 73, nonvolatile storage unit 74 and input / output interface 75 are configured.

[0187] The SW controller 10 performs switching control of the switching units 11 and 12 according to the generation of the switching trigger KP. The SW controller 10 generates a control signal C1 to control the switching of the switching unit 11. In addition, the SW controller 10 controls the switching of the switching unit 12 by a control signal C2.

[0188] Note that the switcher 600 is provided with a switcher panel (not shown) as an interface device that enables a user to perform various operations and performs various displays for the user, and a controller in the switcher panel can perform switching control of the switching units 11 and 12. That is, the SW controller 10 described below may be the processing of the controller in the switcher panel.

[0189] The switching trigger KP is, for example, a trigger generated by an operator's switching operation on the main line video vCm (a switching operation of the camera 502 for the main line video vCm).

[0190] Note that not only the operator's operation but also the automatic switching trigger KP can be generated. For example, it is conceivable to generate the switching trigger KP by automatic switching control according to a predetermined sequence. In addition, it is also conceivable to generate the switching trigger KP by AI control used to replace the operator.

[0191] Fig.11 The diagram illustrates switching control of the switching units 11 and 12 by the SW controller 10 .

[0192] When the switching trigger KP for the main line video vCm as described above is generated, the SW controller 10 proceeds from step S101 to step S102, and performs switching control of the switching unit 11 by the control signal C1. That is, the switching unit 11 is immediately switched according to the generation of the switching trigger KP.

[0193] Subsequently, in step S103, the SW controller 10 waits for a predetermined time set as the switching delay time Td1.

[0194] When the time as the switching delay time Td1 has elapsed, in step S104, the SW controller 10 performs switching control of the switching unit 12 by the control signal C2. That is, when in step S102, the SW controller 10 causes the switching unit 11 to select the background video vBa, in step S104, the SW controller 10 causes the switching unit 12 to select the shooting video vCa. Similarly, when in step S102, the SW controller 10 causes the switching unit 11 to select the background video vBb, in step S104, the SW controller 10 causes the switching unit 12 to select the shooting video vCb.

[0195] As described above, the SW controller 10 first causes the switching unit 11 to perform switching in response to the switching trigger KP, and causes the switching unit 12 to perform switching after the switching delay time Td1 has elapsed.

[0196] The switching delay time Td1 will be described.

[0197] For example, consider a case where the selection state is switched from camera 502a to camera 502b.

[0198] In this case, the switching delay time Tdl is a time corresponding to a time lag from when the switching unit 11 selects the background video vBb for the camera 502b to when the captured video vCb obtained by capturing the background video vB by the camera 502b is input to the switching unit 12, for example.

[0199] Even if the background video vBa is switched to the background video vBb by the switching unit 11, the display on the LED wall 505 will not switch to the background image vBb at this time, but there will be a delay until the display is switched.

[0200] That is, as the setting of the operation mode on the LED panel 506 side (hereinafter also referred to as the "LED side operation mode"), there are settings of the frame rate and settings of various types of signal processing performed by the display controller 590 and the LED processor 570, so that a delay time is generated until the display is switched according to these LED side operation modes. In terms of signal processing, for example, the delay time of switching from the switching unit 12 to the switching of the display differs depending on the resizing processing of the video on the LED wall 505 (LED panel 506) side, etc.

[0201] In addition, the setting of the operation mode related to the imaging of the camera is also related to the time lag. The operation mode related to the imaging of the camera is an operation mode in the camera 502 or the camera signal processing unit 515. Hereinafter, the operation mode related to the imaging of the camera is also referred to as a "camera-side operation mode". For example, a delay occurs due to the frame rate, shutter speed, read area from the image sensor, processing content of the camera signal processing unit 515, etc., which are settings of the camera 502.

[0202] For example, Fig.12The diagram illustrates an example of a reading range from the image sensor 30. It is assumed that the solid line indicates the entire pixel area of ​​the image sensor 30. In some cases, the reading of the photoelectric conversion signal as the imaging signal from the image sensor 30 can be performed in the entire pixel area indicated by the solid line, or can be performed in various cases according to the reading mode, such as the range indicated by the dotted line, the range indicated by the dashed line, and the range indicated by the dashed line. The delay time varies according to the difference between these reading ranges. In addition, there is also a delay due to the signal processing or resizing processing of the captured video vC.

[0203] With these camera-side operation modes, a time lag is generated from when the background video vBb displayed on the LED wall 505 is photographed (exposed) by the camera 502 b until the photographed video vCb is input to the switching unit 12 .

[0204] Thus, the switching delay time Td1 is set according to the LED side operation mode and the camera side operation mode. As a result, for example, in the case where the background video vBa is switched to the background video vBb for the camera 502b by the switching unit 11, for example, at the timing when the shot video vCb obtained by shooting the background video vBb by the camera 502b is input to the switching unit 12, the switching of the switching unit 12 is performed, and the main line video vCm can be set to the video vCb shot by the camera 502b.

[0205] If the switching timing of the switching unit 12 is advanced, the captured video vCb captured by the camera 502b becomes the main line video vCm when the background video vBa is displayed. This means that the main line video vCm contains a video with an incorrect background. In order to avoid this, an appropriate switching delay time Tdl is set, and Fig.11 The control switching timing is shown in the diagram.

[0206] A specific example of the setting process of the switching delay time Td1 will be described later.

[0207] Through the above-mentioned processing, the camera 502 for shooting the main line video vCm can be switched in the shortest time. In particular, during imaging, the rendering of the background videos vBa and vBb is always performed by the rendering units 21 and 22, so that when the switching trigger KP is generated, the background video vB of the switching destination can be immediately output from the switching unit 11. That is, there is no rendering delay.

[0208] Then, by performing switching of the switching unit 12 at the timing of inputting the background video vB after switching to the switching unit 12, the main line video vCm does not become a video with an incorrect background.

[0209] exist Fig.10In the example, the case where imaging is performed by two cameras 502a and 502b is illustrated, but when imaging is performed by three or more cameras 502, similarly Fig.11 Through the processing in, the video with appropriate background can be set as the main line video vCm.

[0210] Fig.13 The diagram illustrates a case where imaging is performed by three cameras 502 a, 502 b, and 502 c. In this case, videos vCa, vCb, and vCc shot by the corresponding cameras 502 are input to the switching unit 12.

[0211] In response to the imaging performed by the three cameras 502a, 502b and 502c, in the rendering engine 520, background videos vBa, vBb and vBc are generated respectively by the corresponding rendering units 21, 22 and 23, and selected by the switching unit 11.

[0212] Also in this case, when Fig.11 When the switching unit 11 is switched in the process of , the SW controller 10 causes the switching unit 12 to perform switching after the switching delay time Td1 has passed.

[0213] When the SW controller 10 causes the switching unit 11 to select the background video vBa in step S102, the SW controller 10 causes the switching unit 12 to select the shooting video vCa in step S104. Similarly, when the SW controller 10 causes the switching unit 11 to select the background video vBb in step S102, the SW controller 10 causes the switching unit 12 to select the shooting video vCb in step S104. Similarly, when the SW controller 10 causes the switching unit 11 to select the background video vBc in step S102, the SW controller 10 causes the switching unit 12 to select the shooting video vCc in step S104.

[0214] In this way, in step S104 , the SW controller 10 performs control so that the switching unit 12 performs switching to the video vC photographed by the camera 502 corresponding to the background video vB after the switching by the switching unit 11 .

[0215] Note that Fig.13, delay buffers 15a, 15b, and 15c are shown in the switch 600. In the case where the delay time attributed to the camera-side operation mode does not differ between the cameras 502a, 502b, and 502c, for example, in the case where the cameras 502a, 502b, and 502c are of the same model, the delay buffers 15a, 15b, and 15c are generally unnecessary. However, in the case where the models of the plurality of cameras 502 are different, or even under the same model, the delay times of the plurality of cameras 502 may be different for some reasons. Therefore, the delay buffers 15a, 15b, and 15c are provided so that the delay times of the captured videos vCa, vCb, and vCc captured by the corresponding cameras 502 are consistent, and the captured videos vCa, vCb, and vCc are input to the switching unit 12.

[0216] The delay buffers 15a, 15b, and 15c provide buffer delay times DTa, DTb, and DTc for the captured videos vCa, vCb, and vCc, respectively.

[0217] This is to adjust the delay time of the captured videos vCa, vCb and vCc so as to match the one with the longest delay time among the captured videos vCenter, vCb and vCc.

[0218] Assume that t1, t2, and t3 are the times from inputting light (exposure) to the image sensor to inputting to the switching unit 12 of the captured videos vCa, vCb, and vCc, respectively. Then, it is assumed that the time t3 is the longest. In this case, the buffer delay times DTa, DTb, and DTc of the delay buffers 15a, 15b, and 15c are, for example,

[0219] DTa=t3-t1

[0220] DTb=t3-t2

[0221] DTc=t3-t3=0

[0222] etc.

[0223] Alternatively, if there is a shortest buffer delay time DBT among the delay buffers 15a, 15b, and 15c, the buffer delay times DTa, DTb, and DTc are

[0224] DTa=t3-t1+DBT

[0225] DTb=t3-t2+DBT

[0226] DTc=t3-t3+DBT=DBT.

[0227] By providing these buffer delay times, the time until the captured videos vCa, vCb, and vCc are input to the switching unit 12 is made consistent. Then, by setting the above-mentioned switching delay time Tdl based on the consistent time, the main line video vCm with the correct background can be output from the switching time point of the switching unit 12.

[0228] Note that although Fig.10 and Fig.13 In the embodiment, the switching units 11 and 12 are configured in one switch 600, but the switch having the switching unit 11 and the switch having the switching unit 12 may be separate devices. In any configuration, the SW controller 10 provided in any switch or separate device only needs to be able to Fig.11 The switching timing of the switching units 11 and 12 can be controlled by the control in .

[0229] <4. Delay time setting process>

[0230] The setting of the switching delay time Td1 will be described.

[0231] First, it is conceivable that the SW controller 10 stores the switching delay time Td1 as a fixed value. As described above, the switching delay time Td1 is determined by the LED side operation mode and the camera side operation mode, but during imaging, the LED side operation mode and the camera side operation mode are usually not changed. In addition, there may be an imaging system 500 in which the same setting is always set as a system setting.

[0232] In this case, it is only necessary to calculate the time lag in the state of the fixed operation mode, set the switching delay time Tdl accordingly, and store the switching delay time Tdl in advance in the internal memory of the SW controller 10 or the like.

[0233] Regardless of whether the switching delay time Tdl is a fixed value, the operator can input the value of the switching delay time Tdl and store the value in the internal memory of the SW controller 10, etc. In the case of a fixed value, the operator sets the switching delay time Tdl as an initial setting, etc., and inputs the switching delay time Tdl in the preparation stage before imaging.

[0234] In the imaging system 500 capable of changing the LED side operation mode or the camera side operation mode, when changing the operation mode setting, the operator sets an appropriate switching delay time Tdl and inputs the switching delay time Tdl in the preparation stage before imaging.

[0235] By letting the operator know the delay values ​​corresponding to various LED-side operation modes and camera-side operation modes, the operator can change the input of an appropriate delay value according to the mode.

[0236] Alternatively, it is also conceivable that the operator inputs information of the LED side operation mode or the camera side operation mode to be set in the preparation stage of imaging. For example, the SW controller 10 can input the current operation mode by storing information of the delay time corresponding to the operation mode as table data, etc., thereby acquiring the delay time, and thus can calculate the switching delay time Tdl.

[0237] Hereinafter, as another method, a processing example in which the SW controller 10 automatically calculates the switching delay time Td1 will be described.

[0238] First, as a premise, it is assumed that a delay management table for managing delay times corresponding to various operation modes is pre-registered in the switch 600. For example, the delay management table is stored in a memory in the SW controller 10 or in a memory provided separately from the SW controller 10 in the switch 600.

[0239] Then, assume that the SW controller 10 can inquire the display controller 590, the LED processor 570, and the LED panel 506 for model information and various setting values. Similarly, assume that the SW controller 10 can inquire each camera 502 and the camera signal processing unit 515 for model information and various setting values.

[0240] Under this configuration, the SW controller 10 performs the following operations in preparation for imaging: Fig.14 in the processing.

[0241] In step S201 , the SW controller 10 acquires information on the model names and operation modes of the LED panels and the like (the display controller 590 , the LED processor 570 , the LED panel 506 ) connected thereto by communicating with these LED panels and the like connected thereto.

[0242] In step S202 , the SW controller 10 obtains the delay time from signal input to light emission of the background video vB by referring to the delay management table using information acquired from the LED panel or the like.

[0243] In step S203 , the SW controller 10 acquires information on the model name and operation mode of the connected cameras and the like (the camera 502 , the camera signal processing unit 515 ) by communicating with these cameras and the like.

[0244] In step S204 , the SW controller 10 obtains the delay time from receiving light to outputting the signal of the captured video vC to the switching unit 12 by referring to the delay management table using information acquired from the camera or the like.

[0245] In step S205, the SW controller 10 adds the respective delay times obtained in steps S202 and S204 to set the switching delay time Td1.

[0246] The switching delay time Td1 is automatically set by the above process and is used as the switching delay time Td1 during subsequent imaging. Fig.11 The switching delay time Td1 in step S103 in FIG.

[0247] exist Fig.14 In the processing, information on the model name and operation mode is obtained from the LED panel, camera, etc., and the delay time is calculated. However, as another method, there are also examples of measuring the actual delay time.

[0248] For example, using Fig.15 The composition shown in the diagram. Fig.15 In the example, the switch 600 is equipped with a video generator 16 and a video detector 17.

[0249] The video generator 16 can input a video signal for displaying a specific measurement video to the switching unit 11. The switching unit 11 selects and outputs the background videos vBa and vBb and the measurement video. Note that the video generator 16 only needs to output the specific measurement video and actually only needs to be composed of a frame memory storing a video signal as the measurement video.

[0250] The video detector 17 receives the outputs of the camera signal processing units 515a and 515b (i.e., the captured videos vCa and vCb as the video signals input to the switching unit 12) as inputs and detects the video contents. Specifically, the video detector 17 only needs to be able to at least detect whether the captured videos vCa and vCb are measurement videos.

[0251] In this way, the switch 600 can output the measurement video and can detect whether the input captured video vC is the measurement video.

[0252] Then, the SW controller 10 performs, for example, Fig.16 in the processing.

[0253] In step S221, the SW controller 10 causes the LED panel 506 to display the measurement video as, for example, full blue. That is, the SW controller 10 causes the video generator 16 to output the full blue measurement video, and causes the switching unit 11 to select the measurement video.

[0254] In step S222, the SW controller 10 waits until the input shot video vC becomes a full blue video. That is, the SW controller 10 waits for the timing at which the video detector 17 detects the shot video vC of full blue.

[0255] When the video detector 17 detects the full blue shooting video vC, the SW controller 10 causes the LED panel 506 to display the measurement video as, for example, full red in step S223. That is, the SW controller 10 causes the video generator 16 to output the full red measurement video. The SW controller 10 causes the switching unit 11 to continue the selection state of the measurement video from the video generator 16.

[0256] In step S224, the SW controller 10 waits for one frame period to acquire information on the color of the screen detected by the video detector 17, and in step S225, checks whether the color is blue or red.

[0257] When the color is blue, the SW controller 10 waits for one frame period again, and checks the information on the color of the screen detected by the video detector 17 .

[0258] When the color is red, the SW controller 10 proceeds to step S226, counts the number of frames until the color switches from blue to red, and sets the switching delay time Td1 accordingly. For example, in the case where it takes a period of 3 frames to switch the color from blue to red, the switching delay time Td1 is the time corresponding to 3 frames.

[0259] The switching delay time Tdl is automatically set by the above process and is used as the switching delay time during subsequent imaging. Fig.11 The switching delay time Td1 in step S103 in FIG.

[0260] In this process, by actually measuring the delay time, there is no Fig.14 In the case of a delay management table, the precise switching delay time Tdl can also be automatically set.

[0261] Note that the above description is based on the premise of a case where the delay times of the plurality of cameras 502 are the same. In this case, the video detector 17 can detect the color of the captured video vC of any camera 502.

[0262] Fig.17 The diagram illustrates that, assuming that the delay times of the plurality of cameras 502 are different from each other, Fig.13 An example of the case where the delay buffers 15a, 15b, and 15c are provided is shown in FIG.

[0263] exist Fig.17 In step S231-1, the SW controller 10 performs a Fig.16 For example, for the camera 502a as a target, the video detector 17 checks the captured video vCa, counts the number of frames, and calculates the switching delay time Td1 for the camera 502a.

[0264] The same process is performed on the other cameras 502b and 502c.

[0265] Step S231-N indicates that the last camera 502 (for example, camera 502c in the case of 3 cameras) is similarly performed. Fig.16 in the processing.

[0266] When N cameras are used for imaging, Fig.16 The N switching delay times Tdl are calculated by the processing in.

[0267] In step S233, the SW controller 10 sets the delay buffer 15. In consideration of the three cameras 502a, 502b, and 502c, the buffer delay times DTa, DTb, and DTc of the respective delay buffers 15a, 15b, and 15c are set.

[0268] As described above, the buffer delay times DTa, DTb, and DTc are set so that the delay times from the other cameras 502 match the camera 502 having the longest delay time.

[0269] In step S234, the SW controller 10 determines the buffer delay times DTa, DTb, and DTc in consideration of the buffer delay times DTa, DTb, and DTc. Fig.11 The switching delay time Tdl used in the processing.

[0270] For example, a value obtained by adding the maximum switching delay time Tdl among the switching delay times Tdl obtained in steps S231-1 to S231-N to the buffer delay time for the camera 502 is set as the switching delay time Tdl to be used.

[0271] For example, in the case where the delay time of the camera 502c is the longest, the switching delay time Tdl to be determined is a value obtained by adding the switching delay time Tdl measured for the camera 502c to the buffer delay time DTc. For example, if the shortest buffer delay time DTc=DBT, the switching delay time Tdl+DBT measured for the camera 502c is determined as the switching delay time Tdl to be used. At this time, if the shortest buffer delay time DBT=0, the switching delay time Tdl measured for the camera 502c is determined as the switching delay time Tdl to be used.

[0272] As described above, in the case of adjusting the delay time difference between cameras using a delay buffer, the switching delay time Td1 is set by reflecting the buffer delay time.

[0273] Alternatively, if the buffer delay time DT=0 for the camera 502 having the maximum delay time, the switching delay time Tdl measured for the camera 502 having the maximum delay time is determined as the switching delay time Tdl to be used.

[0274] <5. Shutter mode>

[0275] Meanwhile, in the case of this embodiment, each camera 502 performs exposure / reading by a global shutter method.

[0276] Fig.18 Schematic diagram illustrating exposure in global shutter mode. Fig.19 Schematic diagram illustrating exposure in rolling shutter mode. The shaded bars indicate the exposure period for each horizontal line.

[0277] As is known, in the case of the global shutter method, the first line (line 1) to the last line (line N) are exposed at the same time. On the other hand, in the case of the rolling shutter method, exposure is performed at a timing gradually shifted from the first line (line 1) to the last line (line N).

[0278] Regarding such a global shutter method and a rolling shutter method, a case where the switcher 600 switches the camera 502 which will become the main line video vCm will be described.

[0279] Fig. 20 The diagram illustrates each period of frames FR0, FR1, FR2, and FR3 displayed on the LED wall 505 as "LED lighting".

[0280] In this case, the state of the output of the exposure and shooting video vC in the camera 502 having the rolling shutter method is schematically illustrated, and the video of one frame is exposed on the two frames before and after. This is not a problem unless switching is performed specifically, but when switching is performed in the switcher 600 in a certain frame, the video content becomes incorrect in the first frame after switching. That is, the first frame after switching is mixed with the video of the immediately preceding frame. This means that the video of the inner cone vBC that does not correspond to the camera 502 used for the main line video after switching is mixed.

[0281] On the other hand, the state of the output of the exposure and shooting video vC in the camera 502 with a global shutter method is also schematically illustrated, and in this case, the exposure of one frame is completed within one frame period. Therefore, even if switching is performed in the switcher 600, different videos before and after the switching are not mixed.

[0282] According to the above, the present embodiment uses a camera with a global shutter method for each camera 502. However, even when a camera with a rolling shutter method is used, if the shutter speed is increased, the mixing of images of the preceding and following frames can be avoided.

[0283] Alternatively, each camera 502 may be a camera that can switch between a rolling shutter mode and a global shutter mode.

[0284] In this case, the control device in the imaging system 500 performs Fig.21 in the processing.

[0285] In step S301, the control device determines whether the imaging system 500 is in a state of performing imaging by the plurality of cameras 502. In the case of performing imaging by one camera 502, no prior control is particularly performed.

[0286] On the other hand, when it is detected that a plurality of cameras 502 are connected to the imaging system 500 to perform imaging, in step S302 , the control device outputs an instruction to each camera 502 to perform exposure by a global shutter method.

[0287] By performing such control, when a plurality of cameras 502 are used, all cameras 502 operate in a global shutter manner.

[0288] Attention, proceed Fig.21 The control device for the processing in only needs to be the information processing device 70 in the imaging system 500, for example, only needs to be the rendering engine 520 or the SW controller 10.

[0289] <6. Summary and Modifications>

[0290] According to the above-described embodiment, the following effects can be obtained.

[0291] The switcher 600 of the present embodiment includes a switching unit 11 (first switching unit), which receives as input a plurality of video data (e.g., background videos vBa and vBb) including video contents corresponding to each of a plurality of cameras 502 as video data to be displayed on the LED wall 505 (LED panel 506), selects one of the plurality of video data, and outputs the selected video data. In addition, the switcher 600 includes a switching unit 12 (second switching unit), which receives as input a plurality of video data (e.g., shooting videos vCa, vCb) obtained by shooting videos displayed on the LED wall 505 by a plurality of cameras 502, selects one of the plurality of video data, and outputs the selected video data. In addition, the switcher 600 includes a SW controller 10, which controls the switching unit 11 to perform switching of the video data in response to a trigger of video switching, and after a predetermined switching delay time Tdl has passed, the switching unit 12 performs switching of the video data of the camera corresponding to the video data after the switching in the switching unit 11.

[0292] For example, when the camera 502a is switched to the camera 502b, the switching unit 11 switches the background video vBa to the background video vBb including the inner visual cone vBC of the camera 502b, and when the shooting video vC2 is input to the switcher 600, the switching unit 12 performs switching to the shooting video vC2. As a result, the shooting video vC2 output from the switching unit 12 to become the main line video vCm becomes a video including the inner visual cone vBC that correctly corresponds to the camera 502b at this time.

[0293] Therefore, when the camera is switched, it is possible to prevent the video of the inner visual cone vBC of the camera before the switch from being output as the main line video vCm, thereby achieving high-quality video output.

[0294] In addition, since the background videos vBa and vBb corresponding to the plurality of cameras 502 are always rendered, and are selected by the switcher 600 and provided to the LED wall 505, when the plurality of cameras perform imaging, the inner frustum vBC does not overlap on the LED wall 505. Thus, the captured video vC is not mixed with the video of the inner frustum vBC of the other cameras 502.

[0295] In addition, as in the embodiment, the background videos vBa and vBb corresponding to the corresponding cameras 502 are always generated and output to the switcher 600. That is, at the camera switching timing, the rendering of the background video vB corresponding to the newly selected camera 502 is not started. As a result, the switching delay time Td1 does not include the rendering delay. Thus, the switching delay time Td1 does not become longer, and after the triggering of the camera switching, the main line video vCm is switched with a shorter delay time.

[0296] Each of the plurality of video data input to the switching unit 11 of the switcher 600 of the embodiment is a background video vB including video content rendered according to the position and imaging direction of the corresponding camera 502 relative to the LED wall 505 .

[0297] For example, background videos vBa and vBb rendered by the rendering units 21 and 22 are input to the switcher 600. That is, the background videos vBa and vBb are images including the inner cone vBC rendered according to the positions and imaging directions of the cameras 502a and 502b, respectively. As a virtual product in which the background is displayed on the LED wall 505 and imaged by the plurality of cameras 502, this enables appropriate imaging.

[0298] The plurality of video data input to the switching unit 12 of the embodiment is a captured video vC obtained by capturing a video on the LED wall 505 and an object (performer 510 or the like) as a foreground by each of the plurality of cameras 502 .

[0299] For example, captured videos vC1 and vC2 obtained by capturing the LED wall 505 and the foreground object with the cameras 502a and 502b are input to the switching unit 12 of the switcher 600, and one of the captured videos vC1 and vC2 is selected by the switching unit 12 as the main line video vCm.

[0300] As a result, as a virtual production in which the background is displayed on the LED wall 505 and imaged by multiple cameras 502, the main line video can be switched appropriately.

[0301] In the embodiment, the example in which the SW controller 10 sets the switching delay time Td1 based on the LED side operation mode and the camera side operation mode has been described.

[0302] The appropriate switching delay time Td1 is a time obtained by adding the time until the background video vB output from the switching unit 11 is displayed on the LED wall 505, and the time until the captured video vC captured by the camera 502 and displayed and subjected to signal processing is input to the switching unit 12. Thus, these times are calculated based on the operation mode regarding the display on the LED wall 505 and the operation mode regarding the imaging and signal processing of the camera 502, and are set as the switching delay time Td1. As a result, after the switching of the switching unit 11, the switching of the switching unit 12 at an accurate timing is achieved.

[0303] In the embodiment, the example in which the SW controller 10 acquires the LED side operation mode and the camera side operation mode through the operation input of the operator has been described.

[0304] For example, an operator or the like inputs information on an operation mode related to the LED wall 505 of the imaging system 500 or an operation method related to the camera 502, and the SW controller 10 of the switcher 600 performs calculations in accordance with the input to set the switching delay time Td1. As a result, the operator or the like inputs the setting states of the display on the LED wall 505 and the imaging by the camera 502, thereby setting an appropriate switching delay time Td1.

[0305] In the embodiment, an example is described in which the SW controller 10 acquires the LED side operation mode and the camera side operation mode through communication (see Fig.14 ).

[0306] For example, the switch 600 communicates with the display controller 590, the LED processor 570, or the LED panel 506 to obtain an operation mode related to the display on the LED wall 505. In addition, the switch 600 communicates with the camera 502 and the camera signal processing unit 515 to obtain operation mode information of the camera and signal processing. As a result, an appropriate switching delay time Td1 can be automatically set through intra-system communication.

[0307] In the embodiment, an example is described in which the SW controller 10 sets the switching delay time Td1 based on the measurement of the time until the video output from the switching unit 11 is input to the switching unit 12 (see Fig.16 ).

[0308] By counting the time until the video actually output from the switching unit 11 is input to the switching unit 12 by the switch 600, the delay until the display on the LED wall 505 and the delay time of the video due to the processing of the camera 502, etc. can be known. Therefore, an appropriate switching delay time Td1 can be set based on this measurement.

[0309] In the embodiment, an example in which the switching delay time Td1 is a fixed value is also described.

[0310] In the imaging system 500 in which the operation mode related to the display on the LED wall 505 and the operation mode related to the imaging by the camera 502 are not particularly changed, the appropriate switching delay time Td1 is always the same. In the switcher 600 in such a system, the SW controller 10 only needs to store the switching delay time Td1 as a fixed value.

[0311] In the embodiment, the example in which the SW controller 10 changes the switching delay time Td1 according to the operation input of the operator has also been described.

[0312] When the switching delay time Td1 is a fixed value, a plurality of fixed values ​​are stored, and the switching delay time Td1 can be switched by operation input. As a result, changes in system settings and the like can be handled.

[0313] In the embodiment, an example is described in which a plurality of captured videos vC captured by a plurality of cameras 502 are input to the switching unit 12 via the delay buffer 15 .

[0314] like Fig.13 As shown in the diagram, the captured videos vC1, vC2, and vC3 are input to the switching unit 12 via the delay buffers 15a, 15b, and 15c, respectively. In this case, by delaying the buffer delay times DTa, DTb, and DTc in the delay buffers 15a, 15b, and 15c, the input timing to the switching unit 12 can be performed simultaneously. That is, even in the case where the appropriate switching delay time Tdl is different in each camera 502 due to different models of cameras, etc., by aligning the delay times using the delay buffers 15a, 15b, and 15c, switching can be performed at appropriate timing.

[0315] In the embodiment, an example is described in which the SW controller 10 sets the switching delay time Td1 by reflecting the setting of the buffer delay time of the delay buffer 15 (see Fig.17 ).

[0316] In the case where the delay times of the captured videos vC1, vC2, and vC3 are aligned by the delay buffers 15a, 15b, and 15c, the buffer delay times DTa, DTb, and DTc may be a portion of the switching delay time Td1. Thus, by setting the switching delay time Td1 according to the settings of the buffer delay times DTa, DTb, and DTc, the switching of the switching unit 12 is performed at an accurate timing.

[0317] Each of the plurality of cameras 502 of the embodiment is a camera that performs exposure using a global shutter method.

[0318] Since the captured video vC is captured by a global shutter method, the video between the previous and next frames will not be mixed when the switching unit 12 is switched. Therefore, at the switching timing, a frame including the inner viewing cone of other cameras will not be generated.

[0319] In the embodiment, all or some of the plurality of cameras 502 are switchable cameras capable of switching between exposure in the global shutter method and exposure in the rolling shutter method, and in the case where there are video inputs from the plurality of cameras 502, an example is described in which a control device such as the SW controller 10 instructs the switchable cameras to perform exposure in the global shutter method (see Fig.21 ).

[0320] When a plurality of cameras 502 are performing imaging, control is performed so that imaging is performed by a global shutter method. As a result, even when a camera capable of switching between exposure in a global shutter method and exposure in a rolling shutter method is used, video mixing between previous and next frames can be prevented when switching by the switching unit 12.

[0321] The program of the embodiment is used to make a processor, such as a CPU, a DSP, etc., or a device including the processor execute the above-mentioned Fig.11 , Fig.14 , Fig.16 and Fig.17 The processing procedure is shown in the diagram.

[0322] That is, the program of the embodiment is a program for causing the information processing device 70 to execute a process of causing the switching unit 11 to execute switching of video data in response to a trigger of video switching, and causing the switching unit 12 to execute switching to the video data of the camera 502 corresponding to the video data after the switching in the switching unit 11 after a predetermined switching delay time Td1 has passed. Specifically, the program is a program executed by the SW controller 10 of the switcher 600.

[0323] By such a program, the information processing apparatus 70 that executes the above-mentioned correction processing can be realized by various computer devices.

[0324] Such a program may be recorded in advance in a HDD as a recording medium built into a device such as a computer device, a ROM in a microcomputer having a CPU, or the like. In addition, such a program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a Blu-ray disk (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such a removable recording medium may be provided as so-called packaged software.

[0325] Furthermore, such a program may be installed from a removable recording medium into a personal computer or the like, or may be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0326] In addition, such a program is suitable for providing the information processing device 70 and the switch 600 of the embodiment in a wide range. For example, by downloading the program to a personal computer, a communication device, a portable terminal device such as a smartphone or a tablet, a mobile phone, a game device, a video device, a personal digital assistant (PDA), etc., these devices can function as the information processing device 70 of the present disclosure.

[0327] Note that the effects described in this specification are merely examples, are not limited, and other effects may be provided.

[0328] Note that the present technology may also have the following configurations.

[0329] (1) A switch device comprising:

[0330] a first switching unit that receives as input a plurality of video data including video contents corresponding to each of a plurality of cameras, selects one of the plurality of video data, and outputs the selected video data as video data to be displayed on a display;

[0331] a second switching unit that receives as input a plurality of video data obtained by shooting videos displayed on the display by the plurality of cameras, selects one of the plurality of video data, and outputs the selected video data; and

[0332] A control unit that controls to execute switching of the video data of the first switching unit in response to a trigger of video switching, and, after a predetermined switching delay time has passed, to cause the second switching unit to execute switching to the video data obtained by the camera corresponding to the video data after the switching in the first switching unit.

[0333] (2) The switcher device according to (1), wherein the plurality of video data input to the first switching unit respectively include background video data containing video content rendered according to a position and imaging direction of a corresponding camera relative to the display.

[0334] (3) A switcher device according to (1) or (2), wherein the multiple video data input to the second switching unit includes video data obtained by shooting the video on the display and the object serving as the foreground of the display by each of the multiple cameras.

[0335] (4) The switcher device according to any one of (1) to (3), wherein the control unit sets the switching delay time based on operation mode information of the display and operation mode information related to imaging by the camera.

[0336] (5) The switcher device according to (4), wherein the control unit acquires operation mode information of the display and operation mode information related to imaging by the camera through operation input.

[0337] (6) The switcher device according to (4), wherein the control unit acquires operation mode information of the display and operation mode information related to imaging by the camera through communication.

[0338] (7) The switcher device according to any one of (1) to (3), wherein the control unit sets the switching delay time based on measurement of a time until a video output from the first switching unit is input to the second switching unit.

[0339] (8) The switch device according to any one of (1) to (3), wherein the switching delay time includes a fixed value.

[0340] (9) The switcher device according to (8), wherein the control unit changes the switching delay time according to an operation input.

[0341] (10) The switcher device according to any one of (1) to (9), wherein each of the plurality of video data captured by the plurality of cameras is input to the second switching unit via a delay buffer.

[0342] (11) The switcher device according to (10), wherein the control unit sets the switching delay time by reflecting a setting of a delay time of a buffer of the delay buffer.

[0343] (12) The switcher device according to any one of (1) to (11), wherein each of the plurality of cameras includes a camera that performs exposure by a global shutter method.

[0344] (13) The switch device according to any one of (1) to (11), wherein

[0345] All or part of the plurality of cameras include switchable cameras capable of switching between global shutter exposure and rolling shutter exposure, and

[0346] In the case where there are video inputs obtained from a plurality of cameras, the control unit instructs the switchable camera to perform exposure in a global shutter manner.

[0347] (14) A method for controlling a switch device, the switch device comprising:

[0348] a first switching unit that receives as input a plurality of video data including video contents corresponding to each of a plurality of cameras, selects one of the plurality of video data, and outputs the selected video data as video data to be displayed on a display;

[0349] a second switching unit that receives as input a plurality of video data in a case where the videos displayed on the display are respectively shot by the plurality of cameras, selects one of the plurality of videos, and outputs the selected video data,

[0350] The method includes executing switching of the video data of the first switching unit in response to a trigger of video switching, and after a predetermined switching delay time has passed, causing the second switching unit to execute switching of the video data obtained by the camera corresponding to the video data after the switching in the first switching unit.

[0351] (15) An imaging system comprising:

[0352] monitor;

[0353] a plurality of cameras, the plurality of cameras capturing video displayed on the display;

[0354] a plurality of rendering units that generate a plurality of video data including video content corresponding to each of the plurality of cameras; and

[0355] Switcher devices,

[0356] in

[0357] The switch device comprises:

[0358] a first switching unit, the first switching unit receiving a plurality of video data from the plurality of rendering units as input, selecting one of the plurality of video data, and outputting the selected video data as video data to be displayed on the display;

[0359] a second switching unit that receives as input a plurality of video data respectively photographed by the plurality of cameras, selects one of the plurality of video data, and outputs the selected video data; and

[0360] A control unit that controls to execute switching of the video data of the first switching unit in response to a trigger of video switching, and, after a predetermined switching delay time has passed, to cause the second switching unit to execute switching to the video data obtained by the camera corresponding to the video data after the switching in the first switching unit.

[0361] Reference numerals list

[0362] 10Switch controller (SW controller)

[0363] 11 Switching unit

[0364] 12 Switching unit

[0365] 15a,15b,15c Delay Buffer

[0366] 16 Video Generator

[0367] 17 Video Detector

[0368] 21,22,23 Rendering Units

[0369] 30 Image Sensor

[0370] 70 Information processing equipment

[0371] 71CPU

[0372] 500 Imaging System

[0373] 502 Camera

[0374] 505 LED Wall

[0375] 506LED Panel

[0376] 515a, 515b Camera signal processing unit

[0377] 520 Rendering Engine

[0378] 560 Camera Tracker

[0379] 570 LED Processor

[0380] 590 Display Controller

[0381] 600 Switcher

[0382] Tdl Switching delay time

[0383] DTa,DTb,DTc buffer delay time

[0384] vB Background Video

[0385] vBC shooting area video (inner cone)

[0386] Videos shot by vC, vCa, vCb, vCc

[0387] vCm main line video

Claims

1. A switch device, comprising: a first switching unit that receives as input a plurality of video data including video contents corresponding to each of a plurality of cameras, selects one of the plurality of video data, and outputs the selected video data as video data to be displayed on a display; a second switching unit that receives as input a plurality of video data obtained by shooting videos displayed on the display by the plurality of cameras, selects one of the plurality of video data, and outputs the selected video data; and A control unit that controls to execute switching of the video data of the first switching unit in response to a trigger of video switching, and, after a predetermined switching delay time has passed, to cause the second switching unit to execute switching to the video data obtained by the camera corresponding to the video data after the switching in the first switching unit.

2. The switcher device according to claim 1, wherein the plurality of video data input to the first switching unit respectively include background video data containing video content rendered according to a position and imaging direction of a corresponding camera relative to a display.

3. The switcher device according to claim 1, wherein the plurality of video data input to the second switching unit includes video data obtained by shooting the video on the display and the object serving as the foreground of the display by each of the plurality of cameras.

4. The switcher device according to claim 1, wherein the control unit sets the switching delay time based on operation mode information of the display and operation mode information related to imaging performed by the camera.

5. The switcher device according to claim 4, wherein the control unit acquires operation mode information of the display and operation mode information related to imaging performed by the camera through operation input.

6. The switcher device according to claim 4, wherein the control unit acquires operation mode information of the display and operation mode information related to imaging performed by the camera through communication. 7 . The switcher device according to claim 1 , wherein the control unit sets the switching delay time based on measurement of a time until a video output from a first switching unit is input to a second switching unit.

8. A switch device according to claim 1, wherein the switching delay time comprises a fixed value.

9. The switcher device according to claim 8, wherein the control unit changes the switching delay time according to an operation input.

10. The switcher device according to claim 1, wherein each of the plurality of video data photographed by the plurality of cameras is input to the second switching unit via a delay buffer.

11. The switcher device according to claim 10, wherein the control unit sets the switching delay time by reflecting the setting of the delay time of the buffer of the delay buffer.

12. The switcher device of claim 1, wherein each of the plurality of cameras comprises a camera that performs exposure via a global shutter.

13. The switch device according to claim 1, wherein All or part of the plurality of cameras include switchable cameras capable of switching between global shutter exposure and rolling shutter exposure, and In the case where there are video inputs obtained from a plurality of cameras, the control unit instructs the switchable camera to perform exposure in a global shutter manner.

14. A method for controlling a switch device, the switch device comprising: a first switching unit that receives as input a plurality of video data including video contents corresponding to each of a plurality of cameras, selects one of the plurality of video data, and outputs the selected video data as video data to be displayed on a display; a second switching unit that receives as input a plurality of video data in a case where the videos displayed on the display are respectively shot by the plurality of cameras, selects one of the plurality of videos, and outputs the selected video data, The method includes executing switching of the video data of the first switching unit in response to a trigger of video switching, and after a predetermined switching delay time has passed, causing the second switching unit to execute switching of the video data obtained by the camera corresponding to the video data after the switching in the first switching unit.

15. An imaging system comprising: monitor; a plurality of cameras, the plurality of cameras capturing video displayed on the display; a plurality of rendering units, the plurality of rendering units generating a plurality of video data including video content corresponding to each of the plurality of cameras; and Switcher devices, in The switch device comprises: a first switching unit, the first switching unit receiving a plurality of video data from the plurality of rendering units as input, selecting one of the plurality of video data, and outputting the selected video data as video data to be displayed on the display; a second switching unit that receives as input a plurality of video data respectively photographed by the plurality of cameras, selects one of the plurality of video data, and outputs the selected video data; and A control unit that controls to execute switching of the video data of the first switching unit in response to a trigger of video switching, and, after a predetermined switching delay time has passed, to cause the second switching unit to execute switching to the video data obtained by the camera corresponding to the video data after the switching in the first switching unit.

Images