Modulation of video display for human comfort during video image capture of the display

Abstract

Systems and methods to increase human viewing comfort during moving image capture from video displays that use an on / off video control are disclosed. In particular, systems and methods for synchronizing display and camera timing so as to reduce or eliminate artifacts appearing in the captured moving images while minimizing flashing or flickering of the video display perceivable to a human viewer of the video display during image capture times.

Description

MODULATION OF VIDEO DISPLAY FOR HUMAN COMFORT DURING VIDEO IMAGECAPTURE OF THE DISPLAYRELATED APPLICATION DATA

[0001] This application claims the benefit of priority of U.S. Provisional Patent Application Serial No.63 / 534,028, filed August 22, 2023, and titled “Modulation of Video Display for Human Comfort During Video Image Capture of the Display”, which is incorporated by reference herein in its entirety.

[0002] The present application and disclosure is also related to US Patent No 11,310,436, Entitled Elimination of Artifacts in Captured Images of LED Tile Displays,” granted April 19, 2022, and also to International Application Pub. No. WO 2022 / 040300, of the same title, published February 24, 2022, each of which is incorporated by reference herein in its entirety.FIELD OF THE DISCLOSURE

[0003] The present disclosure generally relates to the fields of LED displays and motion picture capture. In particular, the present disclosure is directed to systems and methods for modulating display parameters and image capture timing to provide improved human comfort during image capture while providing desired image capture features and qualities..BACKGROUND

[0004] Motion picture capture is frequently done using cameras employing a “rolling shutter” in which the shutter scans across the scene captured in a single frame. This type of image capture can be simulated electronically in digital cameras. Global shutters also may experience a rolling shutter-like effect depending on the shutter speed setting. In motion picture capture, a common shutter speed is 180 degrees, meaning that 180 degrees out of a 360 degree rotation cycle exposes the camera sensor.

[0005] When capturing moving images with this type of camera device, video displays, for example LED display tiles, can be difficult to capture properly without artifacts because such displays employ an on / off control signal that can cause the pixels of the display to pulse at timings that vary from the timing of the image capture device exposure cycle. For example, when LED display tiles are controlled by pulse width modulation (PWM), the LED pulses are extremely rapidand can often only partially refresh while a scanning type shutter is in a fully open position. The same challenges are presented by other types of displays driven by on / off control signals.

[0006] When pulse timing in the video display does not align with the shutter timing of the image capture device, pulsing or banding artifacts can be created in the image capture device output. The foregoing incorporated US Patent No 11,310,436 and the related, incorporated international application disclose systems and methods for eliminating such artifacts in image capture of video displays. By using techniques employing sub-frame slices as described therein, artifacts in the captured image can be reduced or completely eliminated. While this provides superior captured images, these techniques can cause the video display to appear to flicker significantly when simultaneously viewed by the human eye. This is due to the fact that the human visual system can perceive the differences in content and frequencies and it appears as a flashing / pulsing image and because of the number of milliseconds between sub-frames and the total frame time.

[0007] With the increasing use of video display walls as backgrounds for television and movie production where human actors interact in the in front of the display, or in the case of 3D video volume sound stages surrounded on three or more sides by video display walls, there is a need for new solutions that not only provide improved image capture, but also do so in a manner that does not flash or flicker at rates creating discomfort or health risks for humans interacting with or viewing the displayed images simultaneously with the image capture.SUMMARY

[0008] The present disclosure describes systems and method for modulating display on display surfaces for increased human viewing comfort during image capture of a scene including the video display. Such methods include a method of image capture of a scene including at least one display surface presenting images in image display cycles timed to match a frame rate of the image capture including steps of setting the image display cycle to have a period of not more than about 22 ms, and adjusting the image capture frame rate to match the image display cycle so that one so that each image display cycle substantially aligns with a whole frame of the image capture

[0009] Disclosed systems include systems for image capture of a scene including at least one display surface presenting images to be included in the image capture. Such systems may comprise at least one display surface, at least one image capture device configured to capture within a field of view a scene including the at least one display surface, an image display controller configured toprovide images displayed on the display surface in repeating image display cycles timed to match a frame rate of the image capture device with a period of not more than about 22 ms, and an image capture controller configured to adjust the image capture frame rate to match the image display cycle so that one so that each image display cycle substantially aligns with a whole frame of the image capture.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] For the purpose of illustrating the disclosure, the drawings show aspects of one or more embodiments of the disclosure. However, it should be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein:FIG. 1 schematically depicts examples of light emitting display systems according to embodiments of the present disclosure.FIG. 1 A schematically depicts an example of a projector based display system according to alternative embodiments of the present disclosure.FIG. 2 is a timing diagram illustrating shutter timing relative to timing the control signal of a video display for elimination of artifacts in captured images.FIGS. 3 and 3A are additional timing diagrams illustrating further examples of video display control signal timing and synchronization control methods for elimination of artifacts in captured images.FIG. 4 illustrates a user interface through which video sub-frame slices may be assigned and controlled according to embodiments of the present disclosure.FIG. 5 is another view of a user interface through which video sub-frame slices may be assigned and controlled according to embodiments of the present disclosure.FIG. 6 is a further view of a user interface through which video sub-frame slices may be assigned and controlled according to embodiments of the present disclosure.FIG. 7 illustrates an example of an exposure timeline according to embodiments of the present disclosure..FIGS. 8A, 8B, 8C and 8D show a sequence of image captures illustrating frame blending in embodiments of the present disclosure.FIG. 9 is a block diagram of an image capture device according to an embodiment of the present disclosure.FIG. 10 is a block diagram showing an example of a system, camera or display controller according to the present disclosure.DETAILED DESCRIPTION

[0011] The present disclosure describes systems and methods for moving image capture from video displays that use on / off control signals, such as pulse width modulation (PWM) control. In the disclosed systems and methods, artifacts appearing in the captured images due to unsynchronized interaction between camera shutter / exposure timing and the timing of the display control signal are reduced or eliminated by precisely controlling display on and off times. Systems and methods disclosed herein divide or logically “slice” each frame of video into multiple sub-frame partitions (“time zones”), and then each partition can be individually controlled relative to the image capture device shutter / exposure timing to allow precise timing control of the control signal relative to the shutter timing of the image capture device.

[0012] While LED displays are referenced herein in the exemplary embodiments for illustration purposes, as will be appreciated by persons skilled in the art, the principles of the present disclosure are equally applicable to any type of display employing an on / off control signal. Examples of other display types include, but are not limited to, organic light-emitting diodes (OLED), polymer lightemitting diodes (PLED), active-matrix light-emitting diodes (AMOLED), liquid crystal displays (LCD) or light-emitting electrochemical cells (LEC). The scope of the present disclosure and appended claims is therefore not limited to the illustrative LED display examples.

[0013] FIG. 1 illustrates basic components in an example of a system with which embodiments of the present disclosure may be implemented. As shown therein, system 100 includes an image capture device 110, such as a video camera with a digital image sensor. Image capture device 110 is used to capture a moving scene, which includes within the camera field of view (B) an active video display such as LED display 112. The scene within the camera field of view also may optionally include a presenter 126 or other live action scene occurring in front of the display. LED display 112 may comprise large video display walls formed by assembling panels containing arrays of LED pixels. The basics of constructing and controlling such large LED video display walls are generally understood in the art. International application No. PCT / US2020 / 057385, filed October 26, 2020, by the present Applicant, entitled “Unlimited Pixel Canvas For LED Video Walls” describesembodiments of next-generation large LED video display walls with improved structures and control, and is incorporated by reference in its entirety herein. Image capture device 110 and display 112 are controlled by camera controller 114 and display controller 116, respectively. Each controller comprises one or more processors, memory, software, communications, API and other components as described in more detail hereinbelow. Bi-directional communication links 120 between the controllers and the controlled devices may be wireless or wired. Optionally, communication link 124 may be provided between controllers 114 and 116. In a further alternative, display control 116 and camera control 114 may be combined in a single integrated camera and display control 128. Optionally, in other embodiments, additional image capture devices 110a may be added communicating through additional communication links 120a.

[0014] FIG. 1A illustrate an alternative display system 101 in which the scene captured by image capture device 110 includes a display surface 115 displaying a projected image provided by projector 113. Display and image capture control and required communication links may be configured for system 101 by persons skilled in the art based on the teachings here, including the control and communication arrangements shown in FIG. 1. In a further alternative, systems such as system 100 or system 101 may additionally include a high speed camera communication with one or more of system controllers 114, 116 or 128. The high speed camera can provide feedback on display performance and is configured with a sufficient frame rate that it can detect individual sub-frame time slices as discussed further below. Use of such a camera to provide feedback to system processors provides one option for fully automating the methods and techniques described herein. The present disclosure is not, however, limited to this automation option as persons of ordinary skill will derive other automated options based on the teachings contained herein.

[0015] Table 1 below provides a basic illustration of the shutter timing for an image capture device such as video camera 110. This basic illustration describes shutter timing for both mechanical, “rolling shutter” type capture devices, such as traditional motion picture cameras, and for various types of digital capture devices employing digital image sensors. As shown in Table 1, there are three distinct camera shutter states during the time of a single frame, wherein the “C” state represents a time window when the shutter is fully closed (closed shutter period), the “O” state represents a time window when the shutter is fully open (open shutter period), and the “P” state represents a time window when the shutter is opening or closing such that the amount of lightpermitted to reach the sensor during these time fields will vary with the shutter movement (partial shutter period).The duration and timing of each of these time windows is specific to particular image capture devices and can vary from device to device. The teachings of the present disclosure thus apply to any image capture device employing a shutter mechanism, whether mechanical or digitally simulated, that presents a form of the shutter states shown in Table 1 .

[0016] During P time fields, also referred to herein as “partial shutter,” not all photons emitted by the source reach the image sensor, whether film or a digital sensor. In order to reduce or eliminate artifacts created by the on / off control signal, the LED tile should not output any light during the P time fields. During the O time fields the LED tile should complete all control signal refresh cycles, and during the C time fields no output from the LED tiles is needed. However, if the LED tile is also being observed live by human viewers (in addition to the image capture device), a comfort level light output may be emitted during the C time fields so as to avoid a flicker effect visible to the human eye. It is to be further noted that the time intervals for the P and O time fields may change value based on fixed camera parameters such as characteristics of the digital sensor and selectable camera parameters such as shutter angle settings used to generate the image of the LED display.

[0017] FIG. 2 illustrates timing control of the video display control signal relative to a digital camera sensor charge state across two camera shutter periods for some embodiments disclosed herein. Top row boxes 10 indicate the digital sensor charge states (1-8), beginning at a relative time reference (To), i.e., the beginning of a single camera shutter period. The camera shutter period begins at To and ends at Ts, which is also To for the next shutter period. A single shutter period corresponds to a single video capture frame. Wider black line 20 indicates simulated shutter open vs. closed states, wherein the solid black line portion 22 represents fully closed (C in Table 1), the dashed line portion 24 represents partially open (P in Table 1), and the gap portion 26 represents the fully open state (O in Table 1). Line 30 represents the on / off state of the control signal to the videodisplay and the various arrows (A through C) represent different time control windows across the shutter period.

[0018] During time window B, when the shutter is fully open with the digital sensor receiving full charge (sensor states (3)-(5), corresponding to the O time field in Table 1), control signal 30 is delivered at its full on state 32, which is a pulse control configured to present the desired tile light output as is understood for on / off based control of video tile displays in general. It is the partial shutter periods, e.g., sensor states (1 )-(2) corresponding to time window A2, and sensor states (6)-(7) corresponding to time window Cl, during which the video display control signal 30 must be set to off 34 in order to minimize or eliminate artifacts in the moving image captured from the video display. During times when the shutter is fully closed, e.g., digital sensor state (8) corresponding to time windows C2 and Al, from the perspective of image capture alone, no control signal is necessary. However, if visual appearance to a live, in-person viewer of the display is a concern, then an optional control signal 36 may be delivered to the display, which may correspond to the overall video stream or may present a momentary static color or other still image to smooth or eliminate any visually perceptible flashing of the display.

[0019] In general with respect to FIG. 2, time TO represents the beginning of a shutter period for a single frame. Time T1 represents the beginning of the fully open shutter window during which the LED tiles are to be at full display values, and time T2 represents the end of the fully open shutter window. Time T3 represents the end of the shutter closing state and beginning of the closed shutter duration. Time T4 represents the midpoint of the camera sensor off time within a shutter period. Time T5 represents the end of a shutter period and corresponds to time TO for the next subsequent shutter period. The length of time window B (the O time field in Table 1) when the control signal is set on thus corresponds to T2-T1. As explained further below, the minimum control signal off time for elimination of artifacts corresponds to time windows A2 (T0-T1) and Cl (T2-T3).

[0020] While the video display should be on as much as possible while the digital sensor is fully exposed, i.e.. time window B, which provides a good image on camera without artifacts, a strobing effect in person can be created as mentioned if the display or large sections of the display are not also on during the remainder of the shutter period. However, balancing of artifact elimination with suppression of strobing effects creates added complexity because the display still must be cycled off at least during the partial shutter state of time windows A2 and Cl. Thus, optimally there are veryshort and precise time windows during which the control signal should be off to present good on and off-camera appearance.

[0021] To further illustrate the principles of the present disclosure, FIG. 3 presents an example of synchronization of camera settings with on / off control signal timing based on a camera set to a 180 degree shutter angle at 60 frames per second (fps). With these settings, as is known in the art, the camera frame time is 16.6 milliseconds (ms) and the nominal exposure time is 8.3ms. As explained above, the nominal exposure time includes the camera sensor charge and discharge times. Thus, if the camera sensor has a total charge / discharge time of 5.0 ms, then the time window available for clear video capture is 3.3ms (i.e. time window B), starting after the initial 2.5 ms sensor charge time. However, if the camera manufacturer does not publish the timing specifications for the sensor, the operator may not know how the total exposure time of 8.3 ms is allotted between charge / discharge and fully open, and therefore not know when to set the control signal to on and off as described above.

[0022] To solve this problem, in certain embodiments, the operator may use visual feedback from the captured image to determine the correct timing settings as illustrated in FIG. 3. As illustrated therein, timing synchronization begins at TO, when the camera is turned on or otherwise begins capture of the initial frame. Before TO, the display control signal 30, here represented as a 1000 Hz signal at 50% duty cycle, initially may be on or off at 38 prior to the first camera frame. Portions of the control signal at 34 are off and portions at 32 are on.

[0023] Given the chosen frame rate and camera shutter angle, based on familiarity with the system equipment and its performance, an operator may make an initial estimation of a Sync Delay time and a Sync On time. The Sync Delay time is the time prior to the fully open shutter window B, / .<?., the time window A2 between times TO and T1 in FIG. 2 when the control signal should be off to avoid artifacts in the capture image. The Sync On time is the time following the Sync Delay time corresponding to fully open shutter window B, i.e., time between T1 and T2 in FIG. 2, when the control signal should be on for optimum image capture. For the example camera system described in the preceding paragraph, FIG. 3 illustrates the results of an initial Sync Delay time estimate of 2.0 ms and initial Sync On time estimate of 5.0 ms. After initiating image capture with these initial Sync time settings, the operator observes the captured image to determine the presence of artifacts in the captured image resulting from the display being on during one or both the partial shutter states of A2 and Cl (as identified in FIG. 2).

[0024] FIG. 3 shows that an observer would see artifacts in the captured image at both the leading edge and trailing edge of the fully open shutter window B. Artifacts at the leading edge of the fully open shutter window B would be created by the first control pulse 32 falling at the end of the 2.0 ms Sync Delay time, but before time Tl, because the light emitted in that first pulse would be only partially captured by the image sensor in the partial charge state. Similarly, artifacts at the trailing edge of the fully open shutter window B would be created by the last control pulse 32 falling after the close of fully open shutter window B, after time T2, again in the partial charge (discharge) state of the camera sensor between times T2 and T3. By identifying these artifacts in the captured image, the operator knows that both the Sync Delay time is too short and that the Sync On time is too long for that specific camera set up.

[0025] With the knowledge gained from an initial Sync time estimate, the operator may make a revised estimate, increasing the Sync Delay time to further postpone the control on signal 32 (to eliminate leading edge artifacts) and decreasing the Sync On time to further advance the control off signal 34 at the end of the fully open shutter window B (to eliminate trailing edge artifacts). In order to align the Sync On time more precisely with the open shutter period, the operator may choose to increase the frequency of the of the control signal on timing as shown in FIG. 3 A. In this case, increasing the control signal frequency from 1000 Hz to 2000 Hz (maintaining the 50% duty cycle) allows the same Sync Delay time of 2.5 ms with a Sync On time of 3.3 ms without placing a pulse after time T2 because the trailing edge of the final pulse 32 falls at 3.25 ms of the 3.3 ms fully open shutter window B. As will be noted by persons skilled in the art, the 1000 Hz and 2000 Hz PWM signals used in these examples are convenient for illustrating principles of the present disclosure in a simplified manner. In practice, high resolution video walls may operate at frequencies of 8000 Hz or higher. The principles of the present disclosure apply equally to systems operating at such higher frequencies.

[0026] In using this technique of eliminating artifacts and also employing sub-frame slices as described above, it is possible at lower framerates for the display to flicker significantly. This flicker phenomena can also present itself in other types of displayed image where the displayed image occurs in repeating cycles. This is due to the fact that the human visual system can perceive the differences in content and frequencies and it appears as a flashing / pulsing image and because of the number of milliseconds between sub-frames and the total frame time fall within the human perceivable range. In general, if the repeating cycles of the displayed image have a period of greaterthan about 16.6 ms, most human viewers will begin to notice the flicker of the displayed image. When such flicker or flashing becomes perceivable by a human viewer it can cause discomfort and in some cases may have adverse health effects such as headaches or may even instigate seizures in persons with a predisposition for such health issues.

[0027] Disclosed embodiments that provide for reduction or elimination of human perceivable flashing or pulsing in these control scenarios are illustrated in the following examples with reference to video display control interfaces shown in FIGS. 4-6. For example, FIG. 4 shows display user interface 130A, which include camera frame section 132 corresponding to the length of one image capture frame. In this example, the camera frame is divided into three (3) sub-frame time slices 134 / 1, 134 / 2 and 134 / 3 presented as columns. In this example, rows are provided for video feed (VI) 142, unused still images (1 and 2) 144, chroma key color 1 146, chroma key color 2 148, and all black feed 150. Column 136 and row 150 provides for selectable gain adjustments. Drop down tabs 140 allow for user selection of the chroma key colors. The drawing key included with FIG. 4 identifies field contents for each of FIGS. 4-6.

[0028] In the FIG. 4 example, user interface 130A is configured with three sub-frames enabled 134 / 1, 134 / 2 and 134 / 3, one video feed (VI) 142, one chromakey green 146, and one chroma key magenta 148 to cancel out the green. At 60fps, each sub-frame slice is -5.556 ms within a frame duration of 16.667 ms. These sub-frames cycle through quickly enough that the in-person experience may not be bothersome and it is possible to tune a camera capture period to “see” the video feed or chromakey (whichever the user chooses).

[0029] However, if the FIG. 4 example is instead set with a framerate of 24 fps, then the time of each sub-frame slice 134 / 1-3 is approximately 13.9 ms for a total frame time of 41.7ms. This represents a relatively long time between the sub-frame slices compared to human visual perception capabilities and is thus clearly perceptible by the human visual system and appears as a pulsing / flashing display. The camera image capture is still clean for reasons described above, however the display flashes at a disturbing rate in which it is not a practicable method for human comfort..

[0030] In order to reduce or eliminate the uncomfortable display flashing, it is possible to replicate the sub-frames slices multiple times to effectively increase the refresh rate of the display. Such replication is not required, however, for image capture purposes. FIG. 5 illustrates an exampleof flashing suppression through display user interface 130B, which is essentially the same as display user interface 130A but with additional replicated subframe slices 134 / 4-9.

[0031] As shown in FIG. 5, each sub-frame section 132’, 132” and 132’” corresponds to the 13.9 ms frame slices 134 / 1-3 from the FIG. 4, providing the same 24 fps and total frame time of 41.7ms. However, by further dividing each 13.9 ms sub-frame section into multiple sub-frame slices - in this case three sub-frame slices each for a total of nine sub-frame slices for each frame - then each new sub-frame slice 134 / 1-9 is now only 4.6ms. Note that this is even less time than the 5.556 ms sub-frame slice time for the 60 fps example shown in FIG. 4. The result of multiplying the number of sub-frame slices in this manner is to create a more pleasing in-person experience by eliminating apparent display flashing.

[0032] In a further alternative embodiment, as illustrated in FIG. 6, an additional sub-frame slice with black (134 / 2, 134 / 6 and 134 / 10) is added in each camera sub-frame section (132’, 132” and 132’”). This arrangement can provide a better separation of the image sensor’s capture window and readout duration (as described in greater detail in Applicant’s prior filings cited and incorporated above), but it also causes the sub-frame slices to be even shorter, approximately 3.47ms only in this case with 24 fps frame rate.

[0033] While these smaller and faster sub-frame time slices minimize apparent flashing of the display to increase human viewing comfort, depending on the image capture equipment used, the much smaller exposure time to capture a single sub-frame slice can present difficulties because the camera image sensor may require more time to fully charge than is available in the narrow sub-frame time slice. This challenge can usually be resolved on a case-by-case basis by adjusting individual camera settings to reduce the image sensor capture time. Other potential drawbacks in some situations are that the faster effective frame rate creates a sharper image with little to no motion blur as may be expected in the field of cinematography, thus potentially creating artistic challenges or limitations, and incompatibility of the faster frame rate with some video production or broadcast infrastructure. These drawbacks, however, can be addressed and overcome by further embodiments and post-capture processing as explained below. .

[0034] Embodiments disclosed herein are thus based on image capture methods to capture the video display in the field of view that has sub-frame slices in a repeated sequence, with the image sensor capturing at a faster rate than the nominal frame rate (fps) in a multiple of the repeat (e.g. inthe examples above at a nominal frame rate of 24fps with the video display control signal repeated three (3) times so image sensor capturing 72fps). Continuing with the example of a nominal frame rate of 24 fps, a camera system using the disclosed methodology may operate at least one of the following modes:1. Capturing 72 fps at the image sensor, but only recording or outputting a single repeat such that the resulting video stream produced is at 24fps rather than 72fps. This mode addresses the human comfort issue while producing an output at a lower file size with lower bandwith requirements that is consistent with standard broadcast infrastructure that would not accept a 72 fps video feed.2. Capturing at the image sensor and recording (saving) 72 fps, but outputting 24 fps locally by only outputting every third frame. This mode also addresses the human comfort issue and produces an output suitable for local reference monitoring on standard equipment, but it also preserves all of the captured pixel data in the video stream for later processing.3. Capturing 72 fps at the image sensor and internally frame blending 72 fps to a single 24 fps recording and output so as to reproduce a captured image quality as desired. This mode allows creation of artistically desired motion blur that is missed otherwise.4. Capturing 72 fps at the image sensor and recording 72 fps with the ability in camera or in post-production to have a “software defined shutter angle” by frame blending (in this example) frames 1 / 2 / 3 equally, or giving un unequal weight to one or more frames, such as favoring frame 2 and blending a small amount of 1 and 3, again to provide desired artistic effects.

[0035] Additionally, multiple image capture devices or camera systems could simultaneously capture at 72 fps, with different offsets so that each camera sees a different sub-frame slice. For example, one camera could be timed to “see” VI in FIG. 6 (sub-frame slices 134 / 1, 134 / 5 & 134 / 9), and another camera could be timed to “see” the Chroma 1 Green (sub-frame slices 134 / 3, 134 / 7 & 134 / 11). This could easily extend to additional video content or additional cameras with simply more sub-frame slices within each repeat.

[0036] It is possible to have the same or multiple cameras capturing sub-frames for different purposes. For example, one camera might capture VI subframe repeats while the same or another camera is capturing a sub-frame with a specific pattern for tracking purposes.

[0037] Exposure is shown here for different scenarios. In the example of FIG. 6, the camera must be exposed for no more than 25% of the frame time of each repeat cycle, so the exposure timeline for three successive frames may be as shown in FIG. 7 where each image capture exposure sub-frame 156 is user-positionable within each sub-frame section 132’, 132” and 132’”.

[0038] To further exemplify embodiments employing the “software defined shutter angle” in which not only is the shutter angle defined for the sub-frame capture, but also defined by the user is how many, and with what weighting, each of the (three) repeated sub-frame slices to blend together (in camera or in post-production). FIGS 8A-D provide examples of how embodiments may be implemented to alter the appearance in a final produced video frame. FIG. 8A shows each successive frame at a very fast shutter angle as captured at the image sensor. FIG. 8B shows each successive frame at a larger (slower) shutter angle as captured at the image sensor. FIG. 8C shows frames 1 and 2 blended together and FIG. 8D shows frames 1 / 2 / 3 all blended together according to embodiments of the present disclosure.

[0039] Frame blending as referred to herein is a video editing technique used to create smooth transitions between two video frames. This process is often utilized to produce special effects. However, in embodiments disclosed herein, frame blending is used for the purpose of adding back motion blur to the footage that otherwise is sharpened by the higher frame rates and shorter capture durations described. Frame blending uses two consecutive frames and blendis them together, where the first frame gradually fades out and the second frame fades in, resulting in a seamless transition. In the context of the present disclosure, frame blending can be used for filling the gaps between frames with blended transitions, or to create motion blur when the captured frames are too sharp. There are many types of frame blending. A basic frame blend is less computationally intensive and simply averages the content of two sequential frames together. A more sophisticated but very computationally intensive process is called optical flow - in this case, frames of footage are analyzed for greater control of motion blur as the process estimates the velocity of moving objects by analyzing changes in pixel intensity across consecutive frames. This estimation helps determine how objects move relative to the camera.

[0040] An illustration of the use of frame according to the present disclosure is as follows: If video is captured at 72fps, the ~3.47ms of each sub-frame slice is very fast and can cause the produced video output to have the look and feel of a broadcast TV soap opera instead of a cinematic style. However, ff the 72fps video capture is frame-blended together as disclosed herein to create a single 24 fps video output, then the produced video has an effective 3X 3.47ms = 10.41ms which is a much longer duration, and can be more evenly spaced throughout the 41 ,7ms frame time. Thus, while embodiments described herein can be employed to increase human viewing comfort in the presence of the video display during image capture, the techniques may have a less desirable side effect of making the recorded footage sharper than perhaps intended. Frame-blending as disclosed herein allows for the creation of a frame blended “software-defined shutter angle” that can add back in the motion blur and desired artistic / cinematographic appearance of the produced video.

[0041] In a further alternative, a higher frequency input could be used, capturing several subframes across frame boundaries, blending these captured images together, to produce a lower frequency output. For example a 72 fps input being blended to a 24 fps output by subsequent captures across three frame boundaries. With a combination of these techniques being possible, a higher frequency input with multiple sub-frames, being captured across frame boundaries, to achieve a desired “software defined shutter angle”. Allowing for multiple cameras, producing a desired human comfort level, while still providing enough light output for camera exposure.

[0042] In a further alternative embodiment, image capture of a scene including at least one display surface presenting images in image display cycles timed to match a frame rate of the image capture is accomplished by setting the image display cycle to have a period of not more than about 22 ms and then adjusting the image capture frame rate to match the image display cycle so that one so that each image display cycle substantially aligns with a whole frame of the image capture. The display surface may be a light emitting video display or a surface onto which the display is projected by projectors. An image data flow may be generated from the image capture, for example containing data representing the images captured by the image capture device. Using the generated image data flow may be stored for later uses or may be used to produce a video output. The produced video output may run at the the adjusted frame rate resulting from the adjusting step, or it may run at a modified frame rate different from the adjusted frame rate.

[0043] Typically, the modified frame rate will be at a lower frame rate than the adjusted frame rate. For example, if the adjusted frame rate is set at 72 fps to avoid flashing and promote comfortfor human viewers, the modified frame rate may be set at 24 fps to provide desired appearance or artistic effects in the produced video output. The video output at the lower modified frame rate may be taken directly from an output of the image capture device or it may result from further processing of the generated image data flow. In some embodiments, the video output at the modified frame rate is produced by blending into each frame video content of at least one adjacent frame.

[0044] In a further aspect of the present disclosure, The step of setting the image display cycle involves dividing each image display cycle into plural time slices repeating across all image display cycles and assigning display content to each time slice in the same order for each cycle. It is to be noted that display content in this context may include null content. In some embodiments, the the plural time slices within each cycle are equal in time duration..

[0045] Examples of assigning display content other than null content include one or more of video feed content being assigned to one repeating time slice, second or more video feed content being assigned to a second or more repeating time slices, still image content being assigned to one or more repeating time slices, one or more chroma key color content being assigned to at least one repeating time slice, and / or black color content being assigned to one or more of said repeating time slices. Additionally any of the forgoing display content types may be inverted or negative transpositions.

[0046] An Image display cycle period of 22 ms approximately corresponds to a image capture frame rate of 48 fps. The lower cycle periods correspond to higher capture frame rates, for example the cycle period may be set at 20 ms corresponding approximately a frame rate of 50 fps, at 16.6 ms corresponding approximately to a frame rate of 60 fps, or at 13.9 ms corresponding to a frame rate of approximately 72 fps. In some cases it may be desirable to lower the cycle period to be 5 ms or less.

[0047] Adjustment of the image capture frame rate as described above may involve steps of increasing the image capture frame rate to capture a sequence of display time slices, selecting a specific repeating time slice in each display cycle, and generating the image data flow or producing the video output based on the content of the selected specific repeating time slice. Depending on user preference, the generated image data flow or produced video output will include only the content of the selected specific repeating time slice. Embodiments disclosed herein thus provide systems and methods for simultaneous image capture from, and human viewing of, a video display with a refresh rate set at or above a threshold rate for human viewing comfort (e.g. 60Hz or higher)while simulating image capture output at a frame rate less than the set refresh rate of the display (e.g. 30 fps or slower).

[0048] A feature of disclosed embodiments is modulating image capture at the frame rate needed to ensure human comfort and then processing the captured video stream to provide an output with the visual attributes of a lower frame rate image capture device. Examples of lower frame rates for which disclosed embodiments may be particularly advantageously employed include, but are not limited to 14 fps to 24 fps as may be used in movie product or to achieve film-like characteristics in video production; 25 fps to 30 fps for video production depending on video standard employed and, to a lesser extent, in some implementations up to about 60 fps for broadcast video.

[0049] In a further aspect of the present disclosure, methods described herein may be implemented in systems or devices such as camera system 160 shown in FIG. 9. Camera system 160 includes image sensor 162 communicating with image processor 164, frame rate blending processor 166 and storage / memory 168 that may contain data and information required for the processing functions, including executable processing instructions. The processors cooperate as described above to produce processed video output 170. Camera system 160 may be configured as an integrated single hardware, or may be implemented via a distributed processing network. For example, frame rate blending processing may be performed outside of the physical camera prior to production of the final video output.

[0050] Based on the teachings of the present disclosure, users of the disclosed systems and methods, such as filmmakers, artists or camera operators, may select their preferred camera shutter angles based on desired artistic effect, lighting conditions, lensing, set design, etc. In one control aspect, display / video signal PWM cycles can be reduced and pulses somewhat widened to reduce the number of timing zones, for example increasing the period of the zones. In another control aspect, desired shutter angle versus available slice refresh times can be adjusted. By shifting the timing of the video signal and the camera shutter angle, desired display timing slices are captured along with any live action in the foreground. Embodiments disclosed herein thus provide far more control options than prior art systems by facilitating complimentary control of both the display PWM signal and camera timing / shutter angle. Thus, while the display / video signal control options may be limited by the number of PWM cycles that can be shown per frame period, when combined with shutter timing / angle control, virtually unlimited artistic effects can be created based on user preference.

[0051] As may be appreciated by persons skilled in the art, certain control configurations according to the present disclosure may result in relatively shorter capture periods meaning that less light is received by the camera sensor than otherwise may be the case. A number of options may be employed for adjusting brightness of the captured image in this situation, such as increasing the PWM on time (higher duty cycle) making the display brighter, adjusting the camera’s aperture to let in more of the light, removing any ND (Neutral Density) filters that the camera may have been using, adjusting the shutter angle (within control signal limits), and increasing the camera’s sensor sensitivity (ISO). In specific situations with specific equipment, other options for brightness control may be presented to achieve any desired image capture quality or artistic effects.

[0052] In some embodiments, control functions, such as camera control 114, display control 118 or tile control 156, may be executed as one or more computing devices 200 as illustrated in FIG. 10. In this example, computing device 200 includes one or more processors 202, memory 204, storage device 206, high speed interface 208 connecting to memory 204 and high speed expansion ports 210, and a low speed interface 212 connecting to low speed bus 214 and storage device 206. Each of the components 202, 204, 206, 208, 210, and 212, are interconnected using various busses or other suitable connections as indicated in FIG. 10 by arrows connecting components. The processor 202 can process instructions for execution within the computing device 200, including instructions stored in the memory 204 or on the storage device 206 to display graphical information via GUI 218 with display 220, or on an external user interface device, coupled to high speed interface 208. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 200 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

[0053] Memory 204 stores information within the computing device 200. In one implementation, the memory 204 is a computer-readable medium. In one implementation, the memory 204 is a volatile memory unit or units. In another implementation, the memory 204 is a nonvolatile memory unit or units.

[0054] Storage device 206 is capable of providing mass storage for computing device 200, and may contain information such as timing control, time slice size and / or static color chroma and timing as described hereinabove. In one implementation, storage device 206 is a computer-readable medium. In various different implementations, storage device 206 may be a floppy disk device, ahard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 204, the storage device 206, or memory on processor 202.

[0055] High speed interface 208 manages bandwidth-intensive operations for computing device 200, while low speed controller 212 manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In one implementation, high speed interface 208 is coupled to memory 204, display 220 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 210, which may accept various expansion cards (not shown). In the implementation, low speed controller 212 is coupled to storage device 206 and low speed expansion port 214. The low speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices as part of GUI 218 or as a further external user interface, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0056] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0057] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data toa programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0058] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0059] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of wired or wireless digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[0060] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0061] Further alternative embodiments include systems comprised of one or more video displays and one or more image capture devices such as cameras where the capture device’s total capture duration is more than about 16ms per frame for a given project frame rate, where the video d display effective framerate is at least doubled to lower the effective per-frame milliseconds between different content, and timing of the image capture device capture duration is aligned with the video display in a known portion of the frame. In some embodiments, control processing may just 2x or3x everything including LED wall in sub-frames and also the camera capture rate, or the user need not be aware that tghe system is running 2x or 3x and the user interface for the capture device and display shows the “normalized” framerate (i.e. everything is running at 72 fps for comfort, but the user interface shows that the image capture is occurring at 24 fps.

[0062] In other embodiments, the system user interface allows sub-frames to be repeated in a pattern enough times to reduce human perceptible flicker. This may multiply frame rate such that each sub-frame is less than approximately 5ms in a repeated manner. Alternatively, increasing camera capture frame rate is possible to capture a sequence of sub-frames and selecting a specific sub-frame offset to record / output (e.g., record every 3rd sub-frame), or recording at the displayed framerate (e g., 72fps) for later post processing. When using multiple image capture devices as shown in FIG. 1, for example, the multiple capture devices may use different sub-frame offsets

[0063] Further embodiments comprise capturing multiple repeated sub-frames within an input frame time which are blended together in post-production processing and capturing multiple repeated sub-frames within an input frame time which are blended together in the camera or other image capture device. In some examples, frame blending is used creating a higher frame rate input than 24fps to have a changing camera capture to average (even if the video background is repeated, the foreground can be moving so each sub-frame captured would have a different image). Input source could be 72Hz with fewer sub-frames (3 in the example), and then the camera can capture those and frame-blend or frame-blend in post. Embodiments also include processing within individual video tiles of a video display, for example to allow for the tile repeating frames so that the in person experience does not flicker and the camera footage can be blended or not depending on the sharpness desired. This could work in the opposite way as well in which the display wall control could choose to be “sharp” or “soft” temporally by accepting a sharp 72 fps signal and allowing the wall operator to have the display tile blend a 72fps input into 24fps.

[0064] The foregoing has been a detailed description of illustrative embodiments of the disclosure. It is noted that in the present specification and claims appended hereto, conjunctive language such as is used in the phrases “at least one of X, Y and Z” and “one or more of X, Y, and Z,” unless specifically stated or indicated otherwise, shall be taken to mean that each item in the conjunctive list can be present in any number exclusive of every other item in the list or in any number in combination with any or all other item(s) in the conjunctive list, each of which may also be present in any number. Applying this general rule, the conjunctive phrases in the foregoingexamples in which the conjunctive list consists of X, Y, and Z shall each encompass: one or more of X; one or more of Y; one or more of Z; one or more of X and one or more of Y; one or more of Y and one or more of Z; one or more of X and one or more of Z; and one or more of X, one or more of Y and one or more of Z.

[0065] Various modifications and additions can be made without departing from the spirit and scope of this disclosure. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments, what has been described herein is merely illustrative of the application of the principles of the present disclosure. Additionally, although particular methods herein may be illustrated and / or described as being performed in a specific order, the ordering is highly variable within ordinary skill to achieve aspects of the present disclosure. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this disclosure.

[0066] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various changes, omissions and additions may be made to that which is specifically disclosed herein without departing from the spirit and scope of the present disclosure.

Claims

What is claimed is:

1. A method of image capture of a scene including at least one display surface presenting images in image display cycles timed to match a frame rate of the image capture, the method comprising: setting the image display cycle to have a period of not more than about 22 ms; and adjusting the image capture frame rate to match the image display cycle so that one so that each image display cycle substantially aligns with a whole frame of the image capture.

2. The method of claim 1, wherein the display surface comprises a light emitting video display.

3. The method of claim 1, wherein the display surface presents projected images.

4. The method of claim 1, 2 or 3, further comprising generating a image data flow comprising the image capture.

5. The method of claim 4, further comprising producing a video output from the image data flow, wherein the video output has the adjusted frame rate resulting from the adjusting step.

6. The method of claim 4, wherein the image data flow has the adjusted frame rate resulting from said adjusting step.

7. The method of claim 6, further comprising processing the image data flow to have a modified frame rate different from the adjusted frame rate.

8. The method according to claim 7, further comprising producing a video output at the modified frame rate based on the image data flow.

9. The method according to claim 7 or claim 8, wherein the modified frame rate is lower than the adjusted frame rate.

10. The method according to claim 8 or claim 9, wherein video output is produced at the modified frame rate by blending into each frame video content of at least one adjacent frame.

11. The method of any preceding claim, wherein said step of setting the image display cycle comprises dividing each image display cycle into plural time slices repeating across all said cycles and assigning display content to each time slice in the same order for each cycle.

12. The method of claim 11, wherein the plural time slices within each cycle are equal in time.

13. The method of claim 11 or claim 12, wherein video feed content is assigned to one repeating time slice.

14. The method of claim 13, wherein second video feed content is assigned to a second repeating time slice.

15. The method of any of claims 11-14, wherein still image content is assigned to one or more repeating time slices.

16. The method of any of claims 11-15, wherein chroma key color content is assigned to at least one repeating time slice.

17. The method of any of claims 11-16, wherein black color content is assigned to one or more of said repeating time slices.

18. The method of any of claims 11-17, wherein the assigned content comprises an inverted or negative transposition of said content.

19. The method of any of claims 11-18, wherein null content is assigned to one or more said repeating time slices.

20. The method of any of claims 11-19, wherein the plural equal time slices have a duration of less than approximately 5 ms.

21. The method of any of claims 11-20, wherein said adjusting the image capture frame rate comprises: increasing the image capture frame rate to capture a sequence of display time slices; selecting a specific repeating time slice in each display cycle; generating the image data flow or producing the video output based on the content of the selected specific repeating time slice.

22. The method of claim 21, wherein the generated image data flow or produced video output comprises as image content only the content of the selected specific repeating time slice.

23. The method of any preceding claim wherein the period of the display cycle is set to not more than about 20 ms.

24. The method of claim 23, wherein the period of the display cycle is set to not more than about 16.6 ms.

25. The method of claim 2, wherein the light emitting video display is one of (i) an active-matrix light-emitting diode (AMLED) display with an active-matrix video control signal, or (ii) a light emitting diode (LED) display with a pulse width modulation (PWM) video control signal.

26. An image capture and processing system comprising non-transitory computer storage containing instructions executable on one or more processors to cause the system to perform the steps of any of claims 1-24.

27. A system for image capture of a scene including at least one display surface presenting images to be included in the image capture, the system comprising: at least one display surface; at least one image capture device configured to capture within a field of view a scene including the at least one display surface; an image display controller configured to provide images displayed on the display surface in repeating image display cycles timed to match a frame rate of the image capture device with a period of not more than about 22 ms; and an image capture controller configured to adjust the image capture frame rate to match the image display cycle so that one so that each image display cycle substantially aligns with a whole frame of the image capture.

28. The system of claim 27, wherein the at least one display surface comprises one or more of light emitting video displays or one or more displays surface presents projected images.

29. The system of claim 28, further comprising one or more image projectors.

30. The system of any of claims 27, 28, or 29, further comprising a system processor configured to generate generating an image data flow comprising the image capture from the at least one image capture device.

31. The system of claim 30, wherein the system processor is further configured to produce a video output from the image data flow, wherein the video output has the adjusted frame rate as set by the image capture controller.

32. The system of claim 30, wherein the system processor is further configured to produce the image data flow with the adjusted frame rate.

33. The system of claim 31, wherein the system processor is further configured to generate the image data flow with a modified frame rate different from the adjusted frame rate.

34. The system according to claim 33, wherein the system processor is configured to produce a video output at the modified frame rate based on the image data flow.

35. The system according to claim 33 or claim 34, wherein the modified frame rate is lower than the adjusted frame rate.

36. The system according to claim 34 or claim 35, wherein the system processor is configured to produce the video output at the modified frame rate by blending into each frame video content of at least one adjacent frame.

37. The system of any preceding claim, wherein one or both of the system processor and image controller is configured to divide each image display cycle into plural time slices repeating across all said cycles and to assign display content to each time slice in the same order for each cycle.

38. The system of claim 37, wherein the plural time slices within each cycle are equal in time.

39. The system of claim 37 or claim 38, wherein video feed content is assigned to one repeating time slice.

40. The system of claim 39, wherein second video feed content is assigned to a second repeating time slice.

41. The system of any of claims 37-40, wherein still image content is assigned to one or more repeating time slices.

42. The system of any of claims 37-41, wherein chroma key color content is assigned to at least one repeating time slice.

43. The system of any of claims 37-42, wherein black color content is assigned to one or more of said repeating time slices.

44. The system of any of claims 37-43, wherein the assigned content comprises an inverted or negative transposition of said content.

45. The system of any of claims 37-44, wherein null content is assigned to one or more said repeating time slices.

46. The system of any of claims 37-45, wherein the plural equal time slices have a duration of less than approximately 5 ms.

47. The system of any of claims 37-46, wherein one or both of the system processor and image capture controller is configured to adjust the image capture frame rate by executing steps comprising:increasing the image capture frame rate to capture a sequence of display time slices; selecting a specific repeating time slice in each display cycle; generating the image data flow or producing the video output based on the content of the selected specific repeating time slice.

48. The system of claim 47, wherein the generated image data flow or produced video output comprises as image content only the content of the selected specific repeating time slice.

49. The system of any preceding claim wherein the period of the display cycle is set to not more than about 20 ms.

50. The system of claim 49, wherein the period of the display cycle is set to not more than about 16.6 ms.

51. The system of any preceding claim, wherein the image display controller, image capture controller and the system processor comprise one or more processors and are configured optionally as a single control unit or as multiple communicating control units.

52. The system of claim 2, wherein the light emitting video display is one of (i) an active-matrix light-emitting diode (AMLED) display with an active-matrix video control signal, or (ii) a light emitting diode (LED) display with a pulse width modulation (PWM) video control signal.

53. A method of image capture of a scene including at least one light emitting video display wherein video output of said display is driven by a video control signal and said image capture comprises a series of image capture frames captured sequentially at a frame rate with each image capture frame including a shutter open period, a partial shutter period and a shutter closed period, the image capture method comprising: modulating the video control signal of the video display to be on substantially only during image capture open shutter periods; adjusting the video control signal timing to drive the video display output in repeating cycles with a period of not more than about 16.6 ms; and matching the image capture frame rate to the adjusted video control signal timing so that image capture open shutter periods at least substantially align with a display content displayed during a portion of the output cycle period.

54. The method of claim 53, further comprising producing a video output comprising the sequentially captured image capture frames.

55. The method of claim 54, wherein the video output is presented at the frame rate resulting from said matching step.

Images