Background generation

Abstract

A display controller (8) for a video capture system, the display controller (8) comprising one or more processors (13) configured to: receive indications of a first position of an active camera (3, 4, 5) and a second position of an inactive camera (3, 4, 5); receiving a definition of a three-dimensional scene; forming a first representation and a second representation of the scene from the first location and the second location, respectively; the first representation is transmitted to the display wall (2) and the second representation is transmitted to the production controller (6).

Description

Technical Field

[0001] The present invention relates to generating backgrounds in images captured by multiple cameras. Background Art

[0002] In filming studios or on sets, it's becoming increasingly common to film a subject against a backdrop displayed on an array of display panels. This array is often referred to as a display wall or LED wall. This approach offers many advantages over traditional filming methods. For example, the backdrop can be easily changed to suit different filming requirements, and because the backdrop is illuminated, it can cast light onto the subject, making the subject appear more natural than against an unlit background.

[0003] The display wall displays a two-dimensional image, while the background typically depicts a three-dimensional scene. Therefore, the image displayed on the display wall can be generated to depict the scene as it would appear from a desired position. This position is typically chosen to correspond to the position of the camera being used. The camera can then capture an image of the subject based on a perspective that makes the scene appear realistic from that camera's perspective.

[0004] When multiple cameras are available in a production studio, producers often switch between them to capture images from different positions. Before switching to a new camera, producers may want to see what the image from the new camera will look like. This cannot be accurately determined by simply viewing the image captured by the new camera with a background that fits the currently active camera. This is because the 3D scene, when projected onto a 2D image, will be positioned differently from the new camera. While the background fits the currently active camera, the image captured by the new camera may appear to have a distorted background.

[0005] One approach to addressing this problem is to have the display wall continuously display a background appropriate for each available camera and synchronize each camera with the display wall so that each camera captures an image only when the appropriate background is displayed. However, because display walls typically have relatively high resolutions, this can require extensive data processing to generate all the background images and significant bandwidth to continuously transmit these images to the wall.

[0006] What is needed is an improved method of operating a display wall so that producers and similar operators can realistically visualize what they will see when any one of the multiple cameras is active. Summary of the Invention

[0007] According to one aspect, a display controller for a video capture system is provided, the display controller comprising one or more processors configured to: receive an indication of a first position of an active camera and a second position of an inactive camera; receive a definition of a three-dimensional scene; form first and second representations of the scene from the first and second positions, respectively; and transmit the first representation to a display wall and the second representation to a production controller.

[0008] The system may include such a display controller as well as a display wall and production controller.

[0009] The display controller may be configured to cause the display wall to alternately display (i) the first representation and (ii) a monochrome image of a predetermined color or a complementary image of the first representation. The production controller may be configured to perform a chroma keying process to replace portions of the image received from the inactive camera having the predetermined color with corresponding portions of the second representation to form a modified image.

[0010] The first representation may have a higher resolution than the second representation.

[0011] The system may include an active camera and an inactive camera. The active camera and / or another portion of the system may be configured to capture frames when the display wall displays the first representation and not to capture frames when the display wall displays the monochrome image. The inactive camera may be configured to capture frames when the display wall displays the monochrome image and not to capture frames when the display wall displays the first representation. This selective capture may be accomplished by discarding unnecessary frames.

[0012] The display controller may be configured to cause the display wall to alternately display (i) the first representation, (ii) a monochrome image of a predetermined color, and (iii) a monochrome image complementary to the predetermined color. The production controller may be configured to perform a chroma keying process to replace portions of the image received from the inactive camera having the predetermined color with corresponding portions of the second representation to form a modified image.

[0013] The active camera may be configured to capture frames when all of the first representation, a monochrome image of a predetermined color, and a monochrome image of a color complementary to the predetermined color are displayed. The inactive camera may be configured to capture frames when the display wall displays the monochrome image and not capture frames when the first representation is displayed.

[0014] The display wall may include a local controller responsive to a command message to cause the display wall to display a monochrome image of a predetermined color. The display controller may be configured to cause the display wall to display the monochrome image of the predetermined color by sending such a command. The command message may be in a form that does not include pixel mapping data for the image sub-portion.

[0015] The system may include a control terminal operable with a production controller. The production controller may be configured to output to the control terminal (i) an image captured by the active camera, the image including the first representation as a background, and (ii) the modified image. The production controller may respond to input from the control terminal to cause the active camera to thereafter become an inactive camera, and to cause the inactive camera to thereafter become an active camera.

[0016] The production controller may be configured to transmit the output of the currently active camera as the production output.

[0017] Each camera may include a device configured to determine the position of the camera and transmit the position to the display controller.

[0018] According to a second aspect, a control device for controlling a display wall for use as a studio background is provided, the display wall consisting of a plurality of display panels, the control device comprising an input terminal and one or more processors, the one or more processors being configured to: in response to receiving image data representing an image pixel area at the input terminal, cause the display panels to jointly display the image; and in response to receiving a command message in a predetermined form at the input terminal, cause the display panels to jointly display a monochrome image of a predetermined color.

[0019] Image data may represent the pixels of an image in a compressed form. Command messages may not represent the pixels of an image in a compressed form.

[0020] According to a third aspect, there is provided a method for estimating the position of a subject in a video feed, the method comprising: providing a display wall behind the subject; forming a background image suitable for the position of an active camera; causing the display wall to alternately display the background image and a monochrome image of a predetermined color; capturing a video feed of the subject using the active camera by forming a plurality of frames while displaying the background; capturing an image of the subject while displaying the monochrome image; and estimating the position of the subject by processing the captured image to determine areas therein that are not the predetermined color.

[0021] The step of capturing an image of a subject may be performed by an active camera.

[0022] According to a fourth aspect, there is provided a method of generating a background for a display wall, the method comprising: providing the display wall behind a subject; positioning at least two cameras in front of the subject, displaying the background having a first perspective on the display wall, viewing the background from a first camera at a first position and a second camera at a second position at the first perspective, applying a transformation to generate the background having a second perspective, and adjusting a video feed from the second camera to replace an area of the video feed representing the background having the first perspective with an area representing the background having the second perspective.

[0023] The first perspective is from the first position. The second perspective is from the second position.

[0024] The method may also include subsequently switching to displaying a background having a second perspective on the display wall and replacing the area in the video feed from the first camera.

[0025] The first background and the second background captured by the first camera and the second camera may be appropriate viewing angles of each respective camera determined according to positions of the cameras relative to the background and the subject.

[0026] The perspective transformation can be performed by inputting a first image of the scene and a distance parameter into an artificial intelligence system. Rendering software can be used to apply the transformation to the first image of the scene and generate a second view of the scene.

[0027] The method may further include displaying the complementary image on the display wall having a background with a first viewing angle and / or displaying the complementary image on the display wall having a background with a second viewing angle.

[0028] The complementary image is preferably displayed between the background images. The display duration of the complementary image is preferably the same as the display duration of the corresponding background image. The complementary image and the background image can be displayed alternately for a limited duration so as to cancel each other out as seen by an observer of the display wall. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0030] Figure 1 A filming studio and associated processing equipment are shown.

[0031] Figure 2 The first frame timeline is shown.

[0032] Figure 3 A second frame schedule is shown. DETAILED DESCRIPTION

[0033] Figure 1 A studio space or set is shown, generally designated 1. At the rear of the studio space is a display wall 2. The display wall is positioned to display images into the studio space. A plurality of cameras 3, 4, and 5 are positioned and oriented to image some or all of the studio space and some or all of the display wall.

[0034] The images captured by the cameras are received by a production control system 6. The production control system has a user interface input device 7 (e.g., a keyboard or touch screen). As will be described further below, a user of the production control system can operate the user interface device 7 to select the output of one of the cameras to form a production output stream. The production output stream can be stored and / or output, for example, as a broadcast feed.

[0035] The display controller 8 forms an image output stream to drive the display wall.

[0036] One or more subjects 9 are located in the studio space. The subjects may be humans, animals, or inanimate objects (e.g., props). The camera may be positioned and directed so as to image the one or more subjects against a background displayed on a display wall.

[0037] One or more cameras may be equipped with a device 10 for estimating the position of the camera in the studio space.

[0038] The display wall may be constructed in any convenient manner. For example, it may be formed from an array of multi-pixel display panels, or it may be a screen for displaying images projected onto a screen. The display wall receives image input from a display controller via a data link 11. For convenience, the data link may be a wired link. Alternatively, it may be a wireless data link. In order to reduce the amount of data that needs to be sent over the data link, the display wall may have a local processor 12 that is arranged to decompress data sent from the display controller. Alternatively, the display controller may be local to the display wall. The local display controller 12 may have a processor 16 and a memory 17. The memory stores code that can be executed by the processor to cause the local display controller to perform the functions described herein. As will be discussed further below, the memory may also store definitions of predetermined frames.

[0039] The front display surface of a display wall can be planar or non-planar. In one example, it can be formed from multiple adjacent planes that partially surround the studio space. The display surface can have a vertical extent. It can be parallel to the vertical axis or tilted or curved relative to the vertical axis.

[0040] The display controller 8 includes a processor 13, a program storage device 14, and a scene data storage device 15. The program storage device and the scene data storage device can share the same physical memory. The program storage device stores code executable by the processor in a non-transitory form to perform the functions described herein. The scene data storage device stores scene data defining a three-dimensional scene in a non-transitory form. The display controller can process the scene data to form an image of the scene from a given position. To this end, the display controller can use suitable rendering software (e.g., Unreal Engine™). The scene defined by the scene data can be constant over time or can vary over time. A scene that varies over time can enable the background displayed on the display wall to depict a changing background, such as trees moving in the breeze or waves crashing on a shoreline. In operation, the display controller receives a location in the studio for which a background is to be generated. The manner in which it receives this location will be described below. In practice, this location can be assumed to be the location of the active camera in the studio. The display controller then generates an image of the scene data for the scene in its current state, as it would appear from that location. The display controller then transmits this image (optionally in compressed form) to the display panel so that the display panel displays the image.

[0041] In practice, a display controller may provide a series of video frames to a display panel, each frame being an image representing a scene specified by the scene data when viewed from a specified position. This position may change over time, for example, if the active camera moves or a different camera is activated.

[0042] Once the image (which can be a still image or a frame of a video feed) is displayed on the display panel, the active camera can capture an image of the subject based on the displayed image. The captured image can then be passed to the production control system 6. If the position used by the display controller to generate the image is the position of the active camera, the displayed image will give an appropriate representation of the stored scene with the subject in the foreground.

[0043] The camera positions can be fixed. For example, the cameras can be fixedly mounted on the floor or another fixed part of the set. Alternatively, one or more cameras can be movable during filming. For example, the cameras can be mounted on wheels. The camera positions can be predetermined (e.g., by manual measurement) and input to the display controller. Alternatively, one or more cameras and / or the set can be equipped with a device 10 that can determine the positions of these cameras. For example, this device can be a StarTracker system commercially available from Mo Sys Technology Limited, an ultra-wideband positioning system, or any other suitable system. The positions determined by such a system can be communicated to the display controller from time to time (e.g., in real time). The display controller can then use these positions to form an image of the scene appropriate for the camera's viewpoint. Utilizing the positions of all available cameras, the display controller can form an image of the scene appropriate for all of its viewpoints. The positions can be transmitted to the display controller via any suitable wired or wireless mechanism.

[0044] The trained neural network can be used to estimate which regions in the video stream captured by the auxiliary camera represent foreground and / or background. Regions not estimated as foreground can be considered background. Background regions can be regions in the video stream that are replaced with an image representing the background image from the perspective of the auxiliary camera.

[0045] As described above, the display controller transmits an image to the display wall for display. When the wall is displaying motion or video images, the display controller may transmit consecutive frames, or information defining consecutive frames (e.g., a set of differences from a previous frame or reference frame). When displaying a still image, the display controller may transmit consecutive identical frames, as if it were a video. Alternatively, it may transmit the image, and the display wall or a memory 17 local to the wall may cache the image and enable the wall to continue displaying it until the display controller transmits a replacement image to the display wall.

[0046] The production controller informs the display controller which camera is currently live. This is the camera currently capturing images for the production video stream. The display controller uses this camera's position as the location from which it generates the images it transmits to the display wall for display. When a different camera begins live operation, the display controller is similarly notified of this change, and in response, the display controller begins generating and transmitting images from this camera's position for display.

[0047] The position of each camera may be provided in any suitable reference frame. Conveniently, it may be referenced to an origin defined in the film studio / set. For ease of calculation, the position may optionally coincide with a portion of the display surface of the wall.

[0048] The operation of the system for providing a representative view of the intended video at the production controller when each camera is in real time will now be described.

[0049] The display controller may operate in a mode in which it causes the display wall to alternately display (a) images / frames determined as described above, i.e., images depicting the stored scene from the perspective of the current live operating camera; Figure 2 30), and (b) an image comprising a solid block of a predetermined color (see Figure 2 31 in). The former image will be called the live image. The latter image will be called the chroma keyed image. The chroma keyed image may consist entirely of predetermined colors. Alternatively, it may have a color block in its central portion and a representation of a stored scene in its peripheral portion. The color block may occupy more than 50% of the displayed image, and conveniently more than 80%. Conveniently, the block may be green or blue, as is commonly used for chroma keying. The display controller transmits data to the display wall to cause it to alternately display the live image and the chroma keyed image. This is done in Figure 2 It is explained in Figure 2 A series of images are shown displayed: a live image 30 and a chroma keyed image 31 are alternately displayed.

[0050] The frequency at which the live image and chroma keyed image are displayed can be selected based on the capabilities of the camera being used. In one example, the live image and chroma keyed image are displayed at the camera's capture frequency or an integer multiple thereof. For example, if the camera can capture images at a frequency of 100 Hz, the live image can be displayed 100 times per second, interspersed with the displayed chroma keyed image. In this embodiment, the active camera can be synchronized so that it only captures images when the live image 30 is displayed. The other cameras can be synchronized so that they only capture images when the chroma keyed image 31 is displayed.

[0051] The display controller can operate in a mode in which it causes the display wall to alternately display (a) an image / frame determined as described above, i.e., an image depicting a stored scene from the perspective of the current live camera, and (b) an "inverse image" / frame comprising complementary colors to the previously displayed image. Complementary image is used herein to mean an image having at least one of complementary colors and inverse brightness relative to the displayed image as a ratio of the total displayed brightness. For example, an image of a forest scene having primarily dark greens and browns would have a complementary image having light pinks and blues. The complementary image may be partially the inverse of the image and partially the pure chroma keyed colors. For example, a complementary image of the foreground and a pure color in the sky area at the top of the image.

[0052] In addition, the display controller can display images of the stored scene from each camera's perspective, as will be discussed in more detail below. The display controller can operate in a mode in which it causes the display wall to alternately display a) an image from a first perspective, b) a complementary image from the first perspective, c) an image from a second perspective, and d) a complementary image from a third perspective. The order in which each of (a) through (d) is displayed can be any order, not necessarily the order a, b, c, d.

[0053] Each camera has a field of view, or frustum. Each camera can have an orientation sensor to detect the direction of its field of view, and a lens sensor to sense the state of its lens, and thus the extent of its field of view. From this, the camera's frustum can be estimated. This data can be provided to the display controller, which can then control the operation of the display wall so that the display within the active camera's frustum differs from that outside. For example, a single color chroma keyed color can be displayed for all time periods outside the active camera's frustum.

[0054] The display controller can employ any convenient method for alternating the live image and the chroma-keyed or complementary image. In one example, it can transmit the live image and the chroma-keyed image as pixel or compressed pixel image data. In general, the images can be transmitted at twice the camera capture frequency. To reduce the bandwidth required to provide image data to the display wall, in another approach, a memory local to the display wall can pre-store the chroma-keyed image. Then, when the chroma-keyed image is to be displayed, the display can simply send a command to the display wall to display the chroma-keyed image. For example, multiple chroma-keyed images of different colors can be stored on the display wall, and their display triggered by corresponding commands from the display controller. Thus, it is contemplated that one or more panels of the display wall, or a controller local to the display wall, can pre-store the chroma-keyed image in local memory 17 and, in response to a predetermined command, retrieve and display the image from its storage device.

[0055] The cameras are synchronized with the live display and the chroma-keyed or complementary image. When the live image is displayed, the live or active camera is enabled to capture images. When the chroma-keyed or complementary image is displayed, the other cameras are enabled to capture images. This can be accomplished in a variety of ways. One option is for the production controller to signal each camera whether it is currently active and synchronize the cameras with the display controller using a common clock or by transmitting a synchronization signal between devices. Each camera's processor can then choose whether to capture images synchronously with the live image or the chroma-keyed image, depending on the camera's status (whether it is operating in real time).

[0056] The display controller generates an image of the stored scene from the perspective of each camera; that is, from the position of each camera known to the display controller. This information is passed to the production controller via link 18. The production controller has a processor 19 and memory 20, which stores code executable by the processor in non-transitory form to enable the production controller to perform the functions described herein. The production controller uses the image received from the display controller to perform color keying replacement in each frame received from the non-real-time camera. In each frame received from the camera, the production controller replaces the color area of the chroma keyed image (e.g., green) with the corresponding area of the image from the corresponding camera received from the production controller. The resulting composite image can then be displayed on the production control user interface 7 along with the image received from the active camera (which has not been color keyed). This occurs in successive frames captured by the cameras, so the production control user interface can display video streams, one or more of which have been modified by color keying, and another video stream from the active camera that has not been color keyed. An operator using the production control user interface can select a camera from these images to form an image or video stream for output as output feed 21. If the selected camera is not the currently active camera, a signal is sent to the display controller 8. The display controller 8 then creates an image appropriate for the new active camera's position for display on the wall. It also transmits images directly to the production controller, adapted for the previous active camera's position, so that they can be used for the replacement color keying. The synchronization between the new and previous active cameras is also altered, so that the new active camera captures image 30 while the previous active camera captures image 31.

[0057] The trained neural network can be used to estimate which areas in the generated image of the scene represent the foreground and / or background. Identified background areas in the video stream can be replaced with background images from the perspective of the currently inactive camera. The trained neural network can be integrated with the display controller's processor or provided separately in a suitable processor.

[0058] The active camera is sometimes referred to as a live camera. It is the camera currently contributing to the system's output video stream. Other cameras may be actively capturing video, for example, to provide feeds to a production controller, which can be used to predict what the output stream would look like if one of these cameras were designated active.

[0059] The images provided by the display controller to the production controller for color keying may conveniently be of lower resolution than the images provided to the display wall. This can significantly reduce the processing required by the display controller.

[0060] about Figure 1Another application of the described arrangement is to generate multiple perspectives of a background image for simulating a scene when captured from multiple cameras. A display wall 2 may display a first image of a scene from a first perspective. This image may be computer generated (e.g. rendered as described above). A first camera 3 captures the scene displayed on the display wall 2 and any subjects 9 located between the display wall and the cameras. A second camera 4 is located at a different position to the first camera 3. When viewed from the position of the second camera 4, the image of the scene displayed on the display wall may appear distorted because the perspective of the scene does not fit the angles of the cameras. In a multi-camera studio setup, the fact that a display wall is used instead of a 3D set may be more apparent due to this perspective distortion effect.

[0061] The display controller can operate in a mode in which it causes the display wall to display (a) an image of the scene from a first perspective, i.e., an image depicting the scene from the perspective of a first camera, and (b) an image of the scene from a second perspective, i.e., an image depicting the scene from the perspective of a second camera. The display controller can operate in a manual mode in which the perspective of the background image is changed by an operator. The background image on the display wall can change depending on which camera is active, for example, which camera is recording. The display controller can be configured to display rendered images from perspectives that are not displayed on the display wall. The operator can view displays of other perspectives displayed on the display controller and select a background to be displayed on display wall 2.

[0062] As described above, in order for the system to generate an image stream from the perspective of a camera that is not the active camera, the system can (a) estimate which parts of the image captured by the inactive camera represent the background, (b) generate the background as seen from the perspective of the inactive camera, and (c) replace the areas in the image estimated to be the background with appropriate parts of the generated background. One way to estimate which parts of the image represent the background is to occasionally display an image of a known single color as the background and detect that color in the image. The color can be a chroma keyed color. Another way to estimate which parts of the image represent the background is as follows. The system estimates what the generated background that appears correct from the perspective of the active camera and is displayed on the display wall would look like from the perspective of the inactive camera. For example, this may involve appropriate geometric and lens transformations to account for changes in perspective and lens characteristics. The system can then perform an image matching operation to detect the transformed image in the image captured by the inactive camera and, in this way, estimate the background areas of the image captured by the inactive camera.

[0063] Figure 3Another set of images that can be displayed is shown. In this example, an additional set of images 32 is shown. Image 32 is formed by the display controller to be color-complementary to image 31. This means that the active camera can capture frames throughout a repeating cycle consisting of frames 30, 31, and 32. Because frame 32 is complementary to frame 31, the two will cancel each other out, and the resulting captured image background will correspond only to frame 30. This reduces the precision with which the active camera needs to synchronize with image 30. Frames 31 and 32 conveniently have the same duration.

[0064] The display controller can also be operated to cause the display wall to display a) an image from a first perspective, b) a complementary image from the first perspective, c) an image from a second perspective, and d) a complementary image from the second perspective. In this scenario, the active camera can capture one frame while displaying the image from the first perspective. The second camera can capture frames while displaying the image from the second perspective. The third camera can have a higher frame rate and capture more than one alternating image.

[0065] When the system is in operation, at least two cameras are capturing video or still images of a scene. The background of the scene is provided by a display screen or wall, which can be formed by multiple display screens. Instead of using a display screen, the background can be projected onto a backdrop (e.g., a white wall). The cameras are operated so that they capture images at different times. During video operation, each camera can capture images at a frequency of, for example, 50 Hz, 60 Hz, 100 Hz, or 120 Hz. The cameras are configured to capture images asynchronously, meaning that their captures are interleaved. This means that the display screens (or alternatives) can be operated so that a different background is displayed for each camera. One or more processors can control the camera capture timing, the image displayed as the background at any time, and any post-processing of the captured images. The processors can be configured to execute on demand using suitable software, which can be stored in non-transitory form on a suitable data carrier. The processors can know the position of each camera relative to the screen and the viewpoint direction of each camera (i.e., the centerline of the field of view projected perpendicular to the camera's image plane). The positions and viewpoints can be determined automatically using a known camera position tracker or can be provided manually. The processor has access to a definition of the scene (eg, as a 3D model) and is able to transform the model to form a 2D image that represents the scene as viewed from a specific point and direction.

[0066] Some modes of operation of the system will now be described. The cameras will be referred to as A and B.

[0067] 1. When camera A is capturing images, the screen displays the scene as seen from camera A (i.e., appropriate for the position and viewing angle of camera A). When camera B is capturing images, the screen displays the scene as seen from camera B (i.e., appropriate for the position and viewing angle of camera B). This means that each camera captures an image that matches the intended background.

[0068] 2. As described in 1, except that during the times when camera A is not capturing an image and camera B is not capturing an image, the screen displays an image which is the complement of the image displayed for camera B. This may have the additional advantage that for someone viewing the screen directly, the image from camera B is effectively cancelled and any perception of flicker is reduced.

[0069] 3. As in 1 or 2, except that the image from camera B is a monochrome image (e.g., green or blue) suitable for chroma keying post-processing.

[0070] When a display wall is used as a background, it can be advantageous to know the position of the subject in front of the wall. Certain processing can then be performed to improve the quality of the image around the subject in the output image. Examples include anti-aliasing or displaying the background image / stream only around the edges of the subject, while performing chroma keying replacement elsewhere. To use these techniques, it is useful to estimate the subject's position from the perspective of the active camera. One approach is to use a sensor, such as a lidar sensor, on the active camera. This requires additional equipment. Another approach is to compare the image commanded to be displayed on the wall with the image captured by the active camera. Areas of difference between the two can be considered to correspond to the subject. This can be difficult to do reliably in some practical situations due to the level of detail in the background image. In the above system, the active camera may capture an image when displaying frame 31 or 32. This frame may not form part of the image feed from the camera that generates the output feed at 20. Instead, it may be passed to a suitable processor for detecting the subject's position. Because the background displayed in frames 31 and 32 is monochrome, detecting the subject's position may be easier and more reliable.

[0071] In the above description, units 6, 8 and 12 are described as separate units. In practice, processing tasks may be distributed in any convenient way between one or more physical processing devices, each of which may be local to or remote from studio 1.

[0072] When a camera is required to capture images other than those displayed on the display wall for a certain period of time, this can be achieved by controlling the camera not to sense images or frames during that period of time, or by having it sense images or frames during that time and discard the images or frames sensed during that time.

[0073] The display wall may be provided by any suitable technology, for example a matrix of display screens such as LED screens, or by projection onto the front or rear of a projection screen.

[0074] Applicants disclose herein individually each individual feature described herein and any combination of two or more such features, as long as such feature or combination can be implemented according to the common general knowledge of a person skilled in the art based on the present specification as a whole, regardless of whether such feature or combination of features solves any problem disclosed herein, and without limiting the scope of the claims. Applicants indicate that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be clear to those skilled in the art that various modifications may be made within the scope of the present invention.

[0075] The phrase "configured to" or "arranged to" followed by a term defining a condition or function is used herein to indicate that the object of the phrase is in a state having the condition or being able to perform the function without modifying or further configuring the object.

Claims

1. A display controller for a video capture system, the display controller comprising one or more processors, the one or more processors being configured to: receiving an indication of a first position of an active camera and a second position of an inactive camera; receiving a definition of a three-dimensional scene; forming a first representation and a second representation of the scene from the first position and the second position, respectively; as well as The first representation is transmitted to a display wall, and the second representation is transmitted to a production controller.

2. A video capture system comprising the display controller according to claim 1, the display wall and the production controller.

3. The video capture system of claim 2, wherein: The display controller is configured to cause the display wall to alternately display (i) a first representation and (ii) a monochrome image of a predetermined color or a complementary image of the first representation; as well as The production controller is configured to perform a chroma keying process to replace portions of an image received from the inactive camera having a predetermined color with corresponding portions of the second representation to form a modified image.

4. The video capture system according to claim 3, wherein: The first representation has a higher resolution than the second representation.

5. The video capture system according to claim 3 or 4, comprising the active camera and the inactive camera, wherein: The active camera is configured to capture frames when the display wall displays the first representation and not to capture frames when the monochrome image is displayed, and the inactive camera is configured to capture frames when the monochrome image is displayed on the display wall and not to capture frames when the first representation is displayed.

6. The video capture system of claim 2, wherein: The display controller is configured to cause the display wall to alternately display (i) the first representation, (ii) a monochrome image of a predetermined color, and (iii) a monochrome image of a color complementary to the predetermined color; as well as The production controller is configured to perform a chroma keying process to replace portions of an image received from the inactive camera having a predetermined color with corresponding portions of the second representation to form a modified image.

7. The video capture system of claim 6, comprising the active camera and the inactive camera, wherein: The active camera is configured to capture frames when all of the first representation, a monochrome image of a predetermined color, and a monochrome image of a color complementary to the predetermined color are displayed; and the inactive camera is configured to capture frames when the display wall displays the monochrome image and not capture frames when the first representation is displayed.

8. The video capture system according to any one of claims 3 to 7, wherein: The display wall includes a local controller that is responsive to a command message to cause the display wall to display a monochrome image of a predetermined color, and the display controller is configured to cause the display wall to display a monochrome image of a predetermined color by sending such a command.

9. The video capture system according to any one of claims 3 to 8, comprising a control terminal operable with the production controller, wherein: The production controller is configured to output to the control terminal (i) an image captured by the active camera including the first representation as a background, and (ii) the modified image; as well as The production controller responds to input from the control terminal to make the active camera become the inactive camera from now on, and to make the inactive camera become the active camera from now on.

10. The video capture system according to any one of claims 3 to 9, wherein: The production controller is configured to transmit output from a currently active camera as production output.

11. A video capture system according to claim 5 or 7 or any claim dependent therefrom, wherein: Each camera includes a device configured to determine a position of the camera and send the position to the display controller.

12. The video capture system of claim 2, wherein: The display controller is configured to cause the display wall to display (i) the first representation, and (ii) the second representation; wherein the second representation is transmitted to the display wall for capture by a second camera.

13. A control device for controlling a display wall for use as a studio background, the display wall consisting of a plurality of display panels, the control device comprising an input terminal and one or more processors, the one or more processors being configured to: In response to receiving data representing a pixel region of an image at an input image, causing the display panels to collectively display the image; and The display panels are caused to collectively display a monochrome image of a predetermined color in response to receiving a command message of a predetermined form at an input terminal.

14. The control device according to claim 13, wherein: The image data represents pixels of an image in compressed form.

15. A method for estimating a position of a subject in a video feed, the method comprising: providing a display wall behind the main body; forming a background image appropriate for the location of the active camera; causing the display wall to alternately display a background image and a monochrome image of a predetermined color; capturing a video feed of a subject using the active camera by forming a plurality of frames while displaying a background; When a monochrome image is displayed, an image of a subject is captured; as well as The position of the subject is estimated by processing the captured image to determine areas therein that are not a predetermined color.

16. The method according to claim 16, wherein: The step of capturing an image of a subject is performed by the active camera.

17. A method for generating a background for a display wall, the method comprising: providing a display wall behind the main body; Positioning at least two cameras in front of a subject, displaying a background having a first perspective on the display wall, viewing the background from the first perspective from a first camera at a first position and a second camera at a second position, applying a transformation to generate the background having a second perspective; and adjusting the video feed from the second camera to replace an area of the video feed representing the background having the first perspective with an area representing the background having the second perspective.

18. The method of claim 17, further comprising capturing the background with the first camera.

19. The method according to claim 17 or 18, further comprising displaying a background having a second viewing angle on the display wall, and capturing the background having the second viewing angle from the second camera.

20. The method of claim 17, 18 or 19, further comprising displaying a complementary image on a display wall having a background at the first viewing angle and / or displaying a complementary image on a display wall having a background at a second viewing angle.

Images