Background Generation

Abstract

A display controller for a video capture system, comprising one or more processors configured to receive an indication of a first location of an active camera and a second location of an inactive camera, receive a definition of a three-dimensional scene, construct first and second representations of the scene from the first and second locations, respectively, and transmit the first representation to a display wall and the second representation to a production controller.

Description

[Technical Field]

[0001] The present invention relates to generating a background in images captured by multiple cameras. [Background technology]

[0002] It is becoming increasingly common in film studios and on sets to shoot video of a subject in front of a backdrop that is displayed on an array of display panels. The arrays are commonly known as display walls or LED walls. This approach can offer many advantages over traditional filming approaches. For example, the backdrop can be easily changed to suit different filming requirements, and because the backdrop is illuminated, it can cast light on the subject, making them appear more natural than they would on a non-illuminated backdrop.

[0003] While the display wall displays two-dimensional images, the background typically depicts a three-dimensional scene. Thus, the image displayed on the display wall can be generated to depict the scene as it would appear from a desired location. That location is typically selected to correspond to the location of a camera in use. That camera can then capture an image of a subject against a view of the scene that appears realistic from the camera's perspective.

[0004] When multiple cameras are available in a film studio, the producer can typically switch between cameras to capture images from different locations. Before switching to a new camera, the producer may want to see what the image looks like from the new camera. This cannot be done accurately by simply viewing the image captured by the new camera with a background that suits the camera currently in use. The reason is that the three-dimensional scene will look different in the new camera's position when projected into two dimensions. The image captured by the new camera may appear with a distorted background when the background is aligned with the currently in use camera.

[0005] One way to address this is to have the display wall continuously display backgrounds for each available camera, and then synchronize each camera with the display wall so that each camera captures images only when its respective background is being displayed. However, because display walls typically have relatively high resolution, this approach requires a large amount of data processing to generate all the background images, and a lot of bandwidth to continuously transmit those images to the wall.

[0006] What is needed is an improved approach to operating a display wall so that producers and similar operators can realistically imagine what will be seen when any of the multiple cameras are in use. Summary of the Invention

[0007] According to a first aspect, there is provided a display controller for a video capture system, the display controller including one or more processors configured to receive an indication of a first location of an active camera and a second location of an inactive camera, receive a definition of a three-dimensional scene, construct first and second representations of the scene from the first and second locations, respectively, transmit the first representation to a display wall, and transmit the second representation to a production controller.

[0008] The system may include the display controller described above along with the display wall and production controller.

[0009] The display controller may be configured to cause the display wall to alternately display (i) a first representation and (ii) a monochrome image of a predetermined color or a complementary image of the first representation. The production controller may be configured to perform a chromakey process in which portions of an image received from an inactive camera that are of a predetermined color are replaced with corresponding portions of a second representation to form a modified image.

[0010] The first representation may be of a higher resolution than the second representation.

[0011] Such a system may include an active camera and an inactive camera, where the active camera and / or another portion of the system is configured to capture frames when the first representation is displayed on the display wall but not when the monochrome image is displayed, and the inactive camera is configured to capture frames when the monochrome image is displayed on the display wall but not when the first representation is displayed, such selective capture may be achieved by discarding unwanted frames.

[0012] The display controller may be configured to cause the display wall to alternately display (i) the first representation, (ii) a monochrome image of a predetermined color, and (iii) a monochrome image of a color complementary to the predetermined color. The production controller may be configured to perform a chromakey process in which portions of a predetermined color in an image received from an inactive camera are replaced with corresponding portions of a second representation to form a modified image.

[0013] The camera in use is configured to capture frames when the first representation, the monochrome image of the predetermined color, and the monochrome image of the color complementary to the predetermined color are all displayed, and the camera in the non-use state is configured to capture frames when the monochrome image is displayed on the display wall but not when the first representation is displayed.

[0014] The display wall may include a local controller responsive to a command message for causing the display wall to display a monochrome image of a predetermined color. The display controller is configured to send the command to cause the display wall to display the monochrome image of the predetermined color. The command message may be in a format that does not include pixel mapping data for a subpart of the image.

[0015] The system can include a control terminal operable with a production controller configured to output (i) an image including the first representation captured by the active camera as a background and (ii) a modified image to the control terminal, wherein the production controller, in response to input from the control terminal, causes the active camera to subsequently become an inactive camera and the inactive camera to subsequently become an active camera.

[0016] The production controller is configured to send the output from the currently active camera as the production output.

[0017] Each camera includes a device configured to determine the location of the camera and transmit the location to a display controller.

[0018] According to a second aspect, there is provided a control device for controlling a display wall to operate as a studio backdrop, the display wall comprising a plurality of display panels, the control device including an input section and one or more processors configured to, upon receiving image data representing a pixel region of an image at the input section, collectively cause the display panels to display the image, and, upon receiving a command message in a predetermined format at the input section, collectively cause the display panels to display a monochrome image of a predetermined color.

[0019] Image data represents the pixels of an image in a compressed format. Command messages do not represent the pixels of an image in a compressed format.

[0020] According to a third aspect, there is provided a method of estimating a location of a subject in a video feed, the method comprising: providing a display wall behind the subject, configuring a background image to match the location of an active camera, causing the display wall to alternately display the background image and a monochrome image of a predetermined color, using the active camera to capture a video feed of the subject by composing a plurality of frames when the background is displayed, capturing an image of the subject when the monochrome image is displayed, and processing the captured images to estimate the location of the subject and determine areas in the image that are not the predetermined color.

[0021] The step of capturing an image of the subject of this method may be performed by a camera in use.

[0022] According to a fourth aspect, there is provided a method of generating a background for a display wall, the method including providing a display wall behind a subject, positioning at least two cameras in front of the subject, displaying a background from a first perspective on the display wall, viewing the background from the first perspective from a first camera positioned at a first location and from a second camera positioned at a second location, applying a transformation to generate the background from the second perspective, and adjusting a video feed from the second camera to replace areas in the video feed representing the background from the first perspective with areas representing the background from the second perspective.

[0023] The first view is from a first location. The second view is from a second location.

[0024] The method then further includes switching the display wall to display the second appearance of the background, replacing the area in the video feed from the first camera.

[0025] The first and second backgrounds captured by the first and second cameras are perspectives appropriate to each camera, determined by the camera's position relative to the background and subject.

[0026] The appearance transformation can be performed by inputting a first image of a scene and distance parameters into an artificial intelligence system, and rendering software can be used to apply the transformation to the first image of the scene to generate a second view of the scene.

[0027] The method further includes displaying a complementary image of the background in the first perspective on the display wall and / or displaying a complementary image of the background in the second perspective on the display wall.

[0028] The complementary images are preferably displayed between background images. The complementary images are preferably displayed for the same duration as their respective background images. The complementary and background images are displayed alternately for a limited duration so that they appear to cancel each other out to an observer of the display wall. [Brief explanation of the drawings]

[0029] The invention will now be described by way of example with reference to the drawings. [Figure 1] 1 shows a film studio and associated processing equipment. [Figure 2] The first frame schedule is shown. [Figure 3] The second frame schedule is shown. DETAILED DESCRIPTION OF THE INVENTION

[0030] Figure 1 shows a studio space or set generally designated "1". Behind the studio space is a display wall 2. The display wall 2 is positioned to display images throughout the studio space. A number of cameras 3, 4, 5 are positioned and aimed to capture some or all of the studio space and some or all of the display wall.

[0031] A production control system 6 receives the images captured by the cameras. The production control system 6 has a user interface input device 7, such as a keyboard or touch screen. As described in more detail below, a user of the production control system 6 can operate the user interface device 7 to select the output of any one of the cameras to compose a production output stream. The production output stream can be stored and / or output, for example, as a broadcast feed.

[0032] A display controller 8 configures the image output stream that drives the display wall 2 .

[0033] One or more subjects 9 are located in the studio space. The subjects 9 may be people, animals, or inanimate objects such as props. A camera is aimed and positioned to capture one or more of the subjects 9 in front of a background that is displayed on the display wall 2.

[0034] One or more of the cameras may be equipped with an apparatus 10 for estimating the location of the camera in the studio space.

[0035] Display wall 2 can be constructed in any convenient manner. For example, it may be made up of an array of multi-pixel display panels or it may be a screen that displays projected images. Display wall 2 receives image input from display controller 8 via data link 11. Conveniently, the data link is a wired link. Alternatively, it may be a wireless data link. To reduce the amount of data that needs to be transmitted over the data link, display wall 2 may include a local processor 12 configured to decompress data sent from display controller 8. Alternatively, display controller 8 may be located at display wall 2. Local display controller 12 includes processor 16 and memory 17. Memory 17 stores code executable by the processor that causes local display controller 12 to perform the functions described herein. Memory 17 also stores definitions of predetermined frames, as described further below.

[0036] The front display surface of the display wall 2 may or may not be planar. In one example, it may be composed of multiple adjacent planes that more or less surround the studio space. The display surface also extends vertically. It may be parallel to the vertical axis, or it may be tilted or curved relative to the vertical.

[0037] The display controller 8 includes a processor 13, a program store 14, and a scene data store 15. The program store 14 and the scene data store 15 may share the same physical memory. The program store 14 stores, in a non-transitory format, code executable by the processor to perform the functions described herein. The scene data store 15 stores, in a non-transitory format, scene data defining a three-dimensional scene. The display controller 8 can process the scene data to construct an image of the scene from a given position. To do this, appropriate rendering software, such as Unreal Engine™, may be used. The scene defined by the scene data may be constant over time or may change over time. If changing over time, the scene may depict a changing background displayed on a display wall, such as trees blowing in the wind or waves crashing on a shoreline. In operation, the display controller 8 receives a location within the studio where a background should be generated. The method for receiving this location is described below. In practice, the location may be the location of a camera in use within the studio. The display controller 8 then generates an image of the scene data as it would be seen from that location, for the scene in its current state. The display controller 8 then transmits the image, optionally in compressed form, to the display panel, which displays the image.

[0038] In practice, the display controller 8 provides a series of video frames to the display panel, each frame being an image depicting the scene identified by the scene data as seen from a specified position, which may change over time, for example, if the camera in use moves or if multiple cameras are in use.

[0039] When an image, either a still image or a frame from a video feed, is displayed on the display panel, the active camera can capture an image of the subject with the displayed image as a background, and the captured image is passed to production control system 6. If the position used by display controller 8 to generate the image was the active camera position, the displayed image will provide an appropriate representation of the stored scene, with the subject in the foreground.

[0040] The location of each camera may be fixed. For example, the cameras may be fixedly mounted to the floor or other fixed part of the set. Alternatively, one or more of the cameras may be movable during filming. For example, the cameras may be mounted on wheels. The location of each camera may be measured in advance, for example, by human measurement, and input to the display controller 8. Alternatively, one or more of the cameras and / or the set may be equipped with a device 10 that measures the camera's location. Such a device may be, for example, the StarTracker system available from Mo-Sys Technology Limited, an ultra-wideband positioning system, or other suitable system. The location measured by such a system may be passed to the display controller 8 at any time, for example, in real time. The display controller 8 may then use the location to construct an image of the scene appropriate for that camera's viewpoint. With all camera locations available, the display controller 8 may construct an image of the scene appropriate for all of those viewpoints. The locations are transmitted to the display controller 8 via a suitable wired or wireless mechanism.

[0041] The trained neural network can be used to estimate which regions in the video stream captured by the second camera represent foreground and / or background. Regions that are not estimated to be foreground can be considered background. Background regions are regions in the video stream that are replaced with an image representing the background image from the perspective of the second camera.

[0042] As described above, display controller 8 transmits images to display wall 2 for display. When displaying moving or video images on wall 8, display controller 8 transmits successive frames or information defining the successive frames, such as a set of deltas from a previous frame or a reference frame. When displaying still images, display controller 8 transmits successive identical frames as if they were video. Alternatively, display controller 8 transmits an image and memory 17 associated with display wall 2 or wall 8 caches the image so that wall 2 can continue to display the same image until display controller 8 transmits a replacement image to display wall 2.

[0043] The production controller 6 informs the display controller 8 which camera is currently in use. This is the camera whose capturing images are currently being used to compose the production video stream. The display controller 8 uses that camera's location as the location from which it is generating images to be transmitted and displayed on the display wall 2. If another camera comes into use, this change is likewise notified to the display controller 8, and in response the display controller 8 begins generating and transmitting images for display from that camera's location.

[0044] The location of each camera may be provided in an appropriate reference system (reference coordinate system). It is convenient to refer to an origin defined within the film studio / set. For ease of calculation, the locations optionally coincide with part of the display surface of the wall 8.

[0045] The operation of the system, which provides a representational view of the video that is expected when each camera is in use, in the production controller 6, will now be described.

[0046] The display controller 8 can operate in a mode in which the display wall 2 alternately displays (a) an image / frame determined as described above, i.e., an image depicting a stored scene from the currently active camera's viewpoint (30 in FIG. 2), and (b) an image containing a solid block of a predetermined color (31 in FIG. 2). The former image is called the live image. The latter image is called the chromakey image. The chromakey image can be composed entirely of the predetermined color. Alternatively, the chromakey image can have a color block in its central portion and a representation of the stored scene in its peripheral portion. This color block can occupy more than 50%, preferably more than 80%, of the displayed image. Preferably, the block can be green or blue, a color commonly used in chromakey processing. The display controller 8 sends data to the display wall 2 to alternate between the live image and the chromakey image. This is illustrated in FIG. 2, which shows a sequence of images being displayed. The live image 30 and the chromakey image 31 are displayed alternately.

[0047] The frequency (period) for displaying the live image and the chromakey image can be selected depending on the capabilities of the cameras used. In one example, the live image and the chromakey image are displayed at the capture frequency of the multiple cameras or an integer multiple thereof. For example, if the multiple cameras can capture images at 100 Hz, the live image is displayed 100 times per second, with the chromakey image inserted in between. In this embodiment, the active camera is synchronized to capture images only when the live image 30 is displayed. The other cameras are synchronized to capture images only when the chromakey image 31 is displayed.

[0048] Display controller 8 can operate in a mode that causes display wall 2 to alternately display (a) images / frames determined as described above, i.e., images depicting the scene from the viewpoint of the currently active stored camera, and (b) "inverted images / frames" containing complementary colors to the previously displayed image. Complementary images, as used herein, refer to images that have complementary colors and / or inverted brightness, as a percentage of total display brightness, relative to the image being displayed. For example, an image of a forest scene that is mostly dark green and brown, would have a complementary image with pale pink and blue colors. A complementary image may also be partially inverted and partially solid chromakey colors, e.g., a complementary image in the foreground and a solid color in the sky area above the image.

[0049] Additionally, the display controller 8 can display stored images of the scene from each camera's viewpoint, as described in more detail below. The display controller 8 can operate in a mode that causes the display wall to alternately display (a) an image from a first perspective, (b) a complementary image from the first perspective, (c) an image from a second perspective, and (d) a complementary image from the second perspective. The order in which each of (a)-(d) is displayed is not particularly restricted, and does not necessarily have to be a, b, c, d.

[0050] Each camera has a field of view, or viewing frustum. Each camera has a direction sensor that detects the direction its field of view is pointed, and a lens sensor that detects the state of its lens, i.e., the extent of its field of view. From these, the camera's viewing frustum can be estimated. This data is provided to display controller 8, which can then control the operation of display wall 2 so that what is displayed inside the viewing frustum of an active camera is different from what is displayed outside it. For example, a monochrome chromakey color is displayed outside the viewing frustum of an active camera for all time segments.

[0051] Display controller 8 can take any convenient route to alternating between live images and chromakey or complementary color images. In one example, both live and chromakey images can be transmitted as pixel or compressed pixel image data. Overall, images can be transmitted at twice the camera capture frequency. To reduce the bandwidth required to provide image data to display wall 2, in an alternative approach, a memory associated with display wall 2 can pre-store chromakey images. Then, when a chromakey image is to be displayed, the display need only send a command to display the chromakey image to display wall 2. For example, multiple chromakey images of different colors can be stored on display wall 2, and their display can be triggered by corresponding commands from display controller 8. That is, it is envisioned that one or more panels of display wall 2, or a controller associated with display wall 2, can pre-store chromakey images in local memory 17 and, in response to a predetermined command, retrieve the images from the storage and display them.

[0052] The cameras are synchronized between displaying a live image and displaying a chromakey or complementary image. An active camera is caused to capture an image when the live image is displayed. Other cameras are caused to capture an image when the chromakey or complementary image is displayed. This can be accomplished in several ways. One option is for the production controller 6 to signal each camera whether it is currently active or inactive, and for the cameras to synchronize with the display controller 8 by a common clock or by sending an inter-device synchronization signal. Each camera's processor then selects whether to capture an image in phase with the live image or in phase with the chromakey image, depending on the camera's active or inactive status.

[0053] Display controller 8 generates an image of the stored scene from each camera's perspective, i.e., from each camera's position as known to display controller 8. This information is passed to production controller 6 via link 18. Production controller 6 has processor 19 and memory 20 that stores, in a non-transitory manner, code executable by processor 19 to cause production controller 6 to perform the functions described herein. Production controller 6 uses the images received from display controller 8 to perform a color key substitution for each frame received from an inactive camera. For each frame received from a camera, production controller 6 replaces areas colored with the color of the chromakey image (e.g., green) with the corresponding area of ​​the image received from controller 8 for each camera. The resulting composite image is then displayed on production control user interface 7, along with the non-color-keyed image received from the active camera. As this occurs for successive frames captured by each camera, production control user interface 7 displays a video stream, one or more of which has been modified by color keying and one from the active camera that has not been modified by color keying. From these images, an operator using the production control user interface 7 can select a camera to compose an image or video stream to be output as the output feed 21. If a camera other than the currently active camera is selected, a signal is sent to the display controller 8, which then composes an image appropriate for the new active camera position for display on the wall 2. It also sends an image appropriate for the previously active camera position directly to the production controller 6 for use in color key substitution.The synchronization of the newly activated camera and the previously activated camera is also changed so that the newly activated camera captures image 30 and the previously activated camera captures image 31.

[0054] The trained neural network can be used to estimate which regions in the generated scene image represent foreground and / or background. Identified background regions are replaced in the video stream with background images from the currently unused camera's perspective. The trained neural network can be incorporated into the processor of the display controller 8 or provided separately in a suitable processor.

[0055] The camera in use is also called the on-air camera. This is the camera whose image currently constitutes the system's output video stream. Other cameras may be operational and capturing video, for example, to provide the production controller 6 with a feed that can be used to predict what the output stream would look like if one of these cameras were designated in use.

[0056] It may be preferable that the images provided by the display controller 8 to the production controller 6 for use in color keying are of lower resolution than the images supplied to the display wall 2. This can significantly reduce the processing required in the display controller 8.

[0057] Another use of the setup described with reference to FIG. 1 is to generate multiple views of a background image to simulate a scene when captured from two or more cameras. A display wall 2 can display a first image of the scene from a first view. This image is computer-generated, e.g., rendered as described above. A first camera 3 captures the scene displayed on the display wall 2 and any object 9 positioned between the display wall 2 and the camera 3. A second camera 4 is positioned at a different location from the first camera 3. When viewed from the position of the second camera 4, the image of the scene displayed on the display wall 2 may appear distorted because the view of the scene is not suited to the camera angle. In a multiple-camera studio setup, the fact that a display wall is used rather than a 3D set becomes more apparent due to this view-distorting effect.

[0058] Display controller 8 can operate in a mode that causes display wall 2 to display (a) an image of the scene from a first perspective, i.e., an image depicting the scene from the viewpoint of a first camera, and (b) an image of the scene from a second perspective, i.e., an image depicting the scene from the viewpoint of a second camera. Display controller 8 can operate in a manual mode in which the appearance of the background image is changed by an operator. The background image of display wall 2 can be changed depending on which camera is in use, e.g., which camera is capturing a recording. Display controller 8 is configured to display images rendered from a perspective that is not being displayed on display wall 2. An operator can view the display of the alternative perspective shown on display controller 8 and select this background to be displayed on display wall 2.

[0059] As described above, in order for the system to generate an image stream from the perspective of a camera other than the active camera, the system can (a) estimate which portions of the image captured by the inactive camera represent the background, (b) generate a background as would be seen from the perspective of the inactive camera, and (c) replace the areas in the image estimated as background with the corresponding portions of the generated background. One way to estimate which portions of an image represent the background is to periodically display an image of a known single color as the background and detect that color in the image. This color is the chromakey color. Another way to estimate which portions of an image represent the background is as follows: The system estimates how a background generated and displayed on a display wall to look correct from the perspective of the active camera would appear when viewed from the perspective of the inactive camera. This may include, for example, appropriate geometric and lens transformations to account for changes in viewpoint and lens characteristics. The system can then perform an image matching operation to detect the transformed image in the image captured by the inactive camera, thus estimating the background region of the image captured by the inactive camera.

[0060] 3 shows an alternative set of images that can be displayed. In this example, an additional set of images, 32, is displayed. Image 32 is configured by display controller 8 to be color-complementary to image 31. This means that the camera in use can capture frames throughout a repeating cycle consisting of frame 30, frame 31, and frame 32. Because frame 32 is color-complementary to frame 31, the two cancel each other out, and the resulting captured image background corresponds only to frame 30. This reduces the precision with which the camera in use must be synchronized to image 30. Frames 31 and 32 are preferably of the same duration.

[0061] The display controller 8 may also be operable to cause the display wall 2 to display (a) an image from a first perspective, (b) a complementary image from the first perspective, (c) an image from a second perspective, and (d) a complementary image from the second perspective. In this manner, the active camera may capture frames as the image from the first perspective is displayed. The second camera may capture frames as the image from the second perspective is displayed. The third camera may have a higher frame rate and may capture two or more of the alternating images.

[0062] When the system is operating, at least two cameras capture video or still images of the scene. The background of the scene is provided by a display screen or wall, which may be formed from multiple display screens. Instead of using a display screen, the background may be projected onto a background wall, such as a white wall. The cameras are operated to capture images at different times. During filming, each camera captures images at, for example, 50 Hz, 60 Hz, 100 Hz, or 120 Hz. The cameras are configured to capture images at different phases, meaning that the camera captures are interlaced. This means that the display screen (or alternative) is operable to display different backgrounds for each camera. One or more processors can control both the camera capture timing, the images that are displayed as backgrounds from time to time, and post-processing of the captured images. The processors are configured to operate as desired by appropriate software, which may be stored in a non-transitory format on a suitable data carrier. The location of each camera relative to the screen and the direction of each camera's viewpoint (i.e., the centerline of its field of view projected perpendicularly onto the camera's image plane) are known to the processor. The location and viewpoint can be determined automatically using known camera location trackers or can be provided manually. The processor can access a definition of the scene, for example as a 3D model, and can transform the model to construct a 2D image representing the scene as seen from a particular point and in a particular direction.

[0063] The following describes some modes of operation of the system: The cameras are called A and B.

[0064] 1. When camera A is capturing an image, the screen is made to display an image of the scene as it would be seen by camera A (i.e., appropriate to camera A's position and viewpoint), and when camera B is capturing an image, the screen is made to display an image of the scene as it would be seen by camera B (i.e., appropriate to camera B's position and viewpoint). This means that each camera will capture an image that looks appropriate for the intended background.

[0065] 2. In addition to 1 above, during the time when camera A is not capturing an image and camera B is not capturing an image, the screen is made to display an image that is the complement of the image displayed for camera B. This can have the added benefit that for someone looking directly at the screen, the image for camera B is effectively cancelled out, reducing the perception of flicker.

[0066] 3. In addition to 1 or 2 above, the image for camera B is a single color image, such as green or blue, suitable for chromakey post-processing.

[0067] When a display wall is used as a background, it is advantageous to know the location of the subject in front of the wall. Certain forms of processing can then be performed to improve the quality of the image around the subject in the output image. Examples include anti-aliasing or displaying the background image / stream only at the subject's periphery and performing chromakey substitution elsewhere. To use such techniques, it is useful to estimate the subject's location from the viewpoint of the in-use camera. One way to do this is to use a sensor, such as a LIDAR sensor, in the in-use camera. This requires additional equipment. Another approach is to compare the image commanded to be displayed on the wall with the image captured by the in-use camera. The area of ​​difference between the two is considered to represent the subject. This can be difficult to do reliably in some implementations, depending on the level of detail in the background image. In the system described above, the in-use camera captures an image when frame 31 or frame 32 is displayed. This frame does not form part of the image feed from the camera generating the output feed at 20. Instead, the frame is passed to an appropriate processor to be used to detect the subject's position. Since the background displayed in frames 31 and 32 is monochromatic, it is easier and more reliable to detect the position of the subject.

[0068] In the above description, units 6, 8 and 12 have been described as separate units. In practice, processing tasks may be distributed in various convenient ways between one or more physical processing devices, either at studio 1 or remote from studio 1.

[0069] If it is desired that one camera capture images except for a particular time segment of the image being displayed on the display wall, this can be accomplished by controlling the camera not to sense images or frames during that time, or by the camera sensing images or frames during that time and discarding the images or frames sensed during that time.

[0070] The display wall may be provided by any suitable technology, for example by a matrix of display screens such as LED screens, or by projection onto a front or rear projection screen.

[0071] Applicant has disclosed each individual feature described herein and combinations of two or more such features separately, regardless of whether such feature or combination solves any problem disclosed herein, and to the extent that such feature or combination can be implemented in accordance with the specification as a whole in light of the common general knowledge of a person skilled in the art without creating limitations on the scope of the claims. Applicant indicates that aspects of the invention may consist of any such individual feature or combination of features. In view of the above description, it will be apparent to one skilled in the art that various modifications within the scope of the invention are possible.

[0072] The phrase "configured to" or "arranged to," followed by a definition of a condition or function, is used herein to indicate that the object of the phrase is in a state of having that condition or capable of performing that function without modification or further configuration.

Claims

1. 1. A display controller for a video capture system, comprising: one or more processors; The processor receiving an indication of a first location of an active camera and a second location of an inactive camera; receiving a definition of a three-dimensional scene; constructing a first representation and a second representation of the scene from the first location and the second location, respectively; a display controller configured to send the first representation to a display wall and send the second representation to a production controller.

2. A video capture system comprising the display controller of claim 1, the display wall, and the production controller.

3. the display controller is configured to cause the display wall to alternately display (i) the first representation and (ii) a monochrome image of a predetermined color or a complementary image of the first representation; 3. The video capture system of claim 2, wherein the production controller is configured to perform chromakey processing to replace portions of the image received from the out-of-service camera that are the predetermined color with corresponding portions of the second representation to construct a modified image.

4. The video capture system of claim 3 , wherein the first representation is at a higher resolution than the second representation.

5. a camera in an active state and a camera in an unactive state; the in-use camera is configured to capture frames when the first representation is displayed on the display wall but not when the monochrome image is displayed; 5. The video capture system of claim 3 or 4, wherein the non-operating camera is configured to capture frames when the monochrome image is displayed on the display wall but not when the first representation is displayed.

6. the display controller is configured to cause the display wall to alternately display (i) the first representation, (ii) a monochrome image of a predetermined color, and (iii) a monochrome image of a color complementary to the predetermined color; 3. The video capture system of claim 2, wherein the production controller is configured to perform chromakey processing to replace portions of the predetermined color of the image received from the out-of-service camera with corresponding portions of the second representation to construct a modified image.

7. a camera in an active state and a camera in an unactive state; the camera in use is configured to capture frames when the first representation, the monochrome image of the predetermined color, and the monochrome image of the complementary color to the predetermined color are all displayed; 7. The video capture system of claim 6, wherein the non-active camera is configured to capture frames when the monochrome image is displayed on the display wall but not when the first representation is displayed.

8. the display wall includes a local controller responsive to command messages for causing the display wall to display a monochrome image of a predetermined color; 8. The video capture system of claim 3, wherein the display controller is configured to cause the display wall to display a monochrome image of the predetermined color by sending the command.

9. a control terminal operable with said production controller; the production controller is configured to output to the control terminal (i) an image captured by the in-use camera that includes the first representation as a background, and (ii) the modified image; The video capture system of any one of claims 3 to 8, wherein the production controller, in response to input from the control terminal, sets the camera in use to an unused camera thereafter, and sets the camera in unused state to an used camera thereafter.

10. 10. The video capture system of claim 3, wherein the production controller is configured to send the output from the camera in its current state of use as a production output.

11. 8. A video capture system as described in claim 5 or 7, or any claim that cites claim 5 or 7, wherein each of the cameras includes a device configured to measure the location of the camera and transmit the location to the display controller.

12. the display controller is configured to cause (i) the first representation and (ii) the second representation to be displayed on the display wall; The video capture system of claim 2 , wherein the second representation is transmitted to the display wall for capture by a second camera.

13. a control device for controlling a display wall to act as a studio backdrop, comprising: the display wall is composed of a plurality of display panels; the control device includes an input and one or more processors; The one or more processors: receiving image data representing a pixel region of an image at the input unit, and causing the display panel to display the image in an integrated manner; A control device configured to, when receiving a command message in a predetermined format at the input unit, cause the display panel to collectively display a monochrome image in a predetermined color.

14. 14. The control device of claim 13, wherein the image data represents pixels of the image in a compressed format.

15. 1. A method for estimating a location of an object in a video feed, comprising: providing a display wall behind said subject; Configuring a background image to match the location of the camera in use; displaying the background image and a monochrome image of a predetermined color alternately on the display wall; using the active camera to capture a video feed of the subject by composing a plurality of frames as the background image is displayed; capturing an image of the subject when the monochrome image is displayed; processing the captured image to estimate a location of the object and determine areas in the image that are not of the predetermined color.

16. The method of claim 16 , wherein the step of capturing an image of the object is performed by the in-use camera.

17. 1. A method of generating a background for a display wall, comprising: Providing a display wall behind the subject; placing at least two cameras in front of the subject; displaying a background of a first appearance on the display wall; viewing the scene from a first perspective from a first camera located at a first location and from a second camera located at a second location; applying a transformation to generate a second appearance of the background; adjusting a video feed from the second camera to replace an area in the video feed representing a background from the first perspective with an area representing a background from the second perspective.

18. The method of claim 17 , further comprising capturing the background with the first camera.

19. 19. The method of claim 17 or 18, further comprising: displaying the second perspective background on the display wall; and capturing the second perspective background from the second camera.

20. 20. The method of any one of claims 17 to 19, further comprising displaying on the display wall a complementary image of the background in the first appearance and / or displaying on the display wall a complementary image of the background in the second appearance.

Images