Method for wirelessly synchronizing an image displaying device and compact image recording devices

Abstract

The present invention concerns a method for wirelessly synchronizing at least one image displaying device and a plurality of independent compact image recording devices, comprising the steps of: a) displaying a sequence of display image frames at a base displaying frame rate (BDFR) with said image displaying device, wherein each display image frame comprises at least two sub-frames, a first sub-frame showing a first image content and a second sub-frame showing a second image content; b) recording a sequence of recording image frames of scenery including said sequence of display image frames with said image recording device at a recording frame rate (RFR); c) optionally, synchronizing said base displaying frame rate (BDFR) and said recording frame rate (RFR) such that said recording frame rate (RFR) is approximately an integer multiple or an integer fraction of said base displaying frame rate; and d) adjusting a phase shift between said sequence of displaying image frames and said sequence of recording image frames such that each recording image frame captures at least one of said first and second sub-frames of said displaying image frames, wherein said phase shift adjusting step.d) is effected by d1) aligning said sequence of display image data and said sequence of recording image frames with a wirelessly transmitted globally available master clock signal, and / or d2) aligning said sequence of recording image frames with said sequence of display image frames by analyzing image content of said recording image frame, said image content comprises display parameters of said sequence of display image data.

Description

[0001] Max von Braun

[0002] 1

[0003] Method for wirelessly synchronizing an image displaying device and compact image recording devices

[0004] Description

[0005] Background of the Invention

[0006] The present invention concerns a method for wirelessly synchronizing an image displaying device and compact image recording devices, preferably at least one image displaying device and a plurality of independent image recording devices, for instance synchronizing LED projectors or LED panels with cameras having a small form factor, such as compact video cameras or cameras of mobile, portable devices, such as mobile phone cameras or drone cameras.

[0007] In video production, a scenery is usually captured with a camera operating at a recording frame rate RFR to provide a video of a scenery which will be played at a video frame rate VFR. The frame rate is usually indicated as "frames per second" (fps) or as a recording frequency indicated in Hertz (Hz). For instance, in the US, a typical standard video frame rate for broadcasting digital videos is usually 60 fps, corresponding to a frequency of 60 Hz. In this example, one second of video stream comprises 60 images, also denoted as image frames or just frames. Consequently, the available time interval for each frame, denoted frame time interval in the context of the present invention, is the inverse of the corresponding frame rate, i.e. 1 / RFR or 1 / VFR. In the example of a frame rate of 60 fps (60 Hz), the corresponding time interval is 16.7 ms. Typical video frame rates include 12 fps, 24 fps, 25 fps, 30 fps, 50 fps, 60 fps, etc.

[0008] In a typical video production, the video frame rate VFR for playback of the recorded video corresponds to the recording frame rate RFR used for capturing a sequence of camera images of a scenery. In this case, any movement in the scenery is reproduced realistically in the final video stream. However, it is also possible to record a scenery with a recording frame rate which is higher than the video frame rate at which the recorded video is played back which results in a slow-motion effect in the final video. For instance, if a sequence of camera images is captured at a recording frame rate RFR of 100 fps but the resulting video is played back at a video frame rate of 50 fps, the resulting video is slowed down by a slow-motion factor / of 2. Conversely, if the scenery is captured with a recording frame

[0009] M / 65050-PCT Max von Braun

[0010] 2 rate RFR which is lower than the video frame rate VFR, the resulting video will be a timelapse video.

[0011] When recording a video, the exposure of the individual camera frames is determined by the ISO setting of the camera sensor, by the aperture of the camera lens and by the so- called "shutter speed", i.e. the time duration during which electromagnetic radiation is accumulated by the camera sensor for each camera frame, i.e. for each picture produced by the camera. Accordingly, the maximum shutter time (exposure time) for each camera frame (camera image) corresponds to the recording frame time interval 1 / RFR. For certain applications, it is possible to reduce the shutter time, i.e. the exposure time per frame, to values lower than the maximum shutter time 1 / RFR, for instance in order to compensate for a bright environment or to capture specific features of a scenery which are only shown for very short time intervals, as will be described in more detail below. In the context of the present invention, a shutter time (exposure time) which corresponds to the maximum value 1 / RFR is denoted "unshuttered", while camera frames (camera images) produced with a shutter time (exposure time) lower than the maximum value 1 / RFR are denoted "shuttered".

[0012] Rather than using the term "shutter speed", it is also common to use the term "shutter angle" which originates from the world of analogue film cameras and refers to the portion of the mechanical rotary shutter that is open during the exposure of each frame. When applied to modern digital imaging sensors, the term is used rather metaphorically to describe the relationship between exposure time and frame rate, particularly when using digital video cameras. In analogue film cameras, the shutter is a rotating disk with a cutout section, the angle of which determines the exposure. When the disk is rotating, the "shutter angle" refers to the angle of the open segment of the disk. For example, if the shutter angle is 180°, the exposure time for each frame is half of the time interval between frames, i.e. 50 percent of the time duration of each frame time interval, the shutter is open. In digital cameras, where there is no physical rotary shutter, the concept is used to express exposure time relative to frame rate. Instead of an actually rotating disk, the exposure time, i.e. the time period during which the sensor collects light, is referred to using shutter angle terminology. Accordingly, the exposure time can be calculated from the shutter angle and the frame interval as follows:

[0013] M / 65050-PCT Max von Braun

[0014] 3 wherein the frame interval is the time between the start of one frame and the start of the next frame which is the inverse of the frame rate, e.g. for a frame rate of 24 frames per second, the frame interval is 41.67 ms and the shutter angle is a number between 0° (no exposure) and 360° (maximum possible exposure for each frame). In summary, larger shutter angles result in longer exposure times which capture more motion blur and overall light. This is often used for creating a smooth cinematic feel. Smaller shutter angles (closer to 0°) result in shorter exposure times, reducing motion blur and producing sharper images which is often used in action scenes or fast-paced video streams. In digital video production, adjusting the shutter angle affects both motion blur and light sensitivity. A larger shutter angle allows more light to hit the sensor but increases motion blur, while a smaller shutter angle reduces both. In essence, selecting an appropriate shutter angle in digital imaging allows film makers and videographers to fine-tune the visual characteristics of motion and light.

[0015] There are various application in which one or more image recording devices, such as cameras, record a scenery in which, as part of the scenery, image displaying devices, such as LED panels, OLED panels, laser projectors or LED projectors, are included which display a sequence of display image frames, for instance the image frames of a video shown on the panel or projected by the projector. Typical examples include advertisements shown on LED panels at sports events or entertainment events, background image content shown on a large array of LED panels or projected by one or more LED projectors, etc. It is also known that the sequence of display image frames shown on such image displaying devices can include main image content intended for direct viewers watching the image displaying device directly and "hidden" image content which is typically shown only for a fraction of a total frame time interval and is therefore not intended to be viewed by direct viewers but only to be recorded by appropriately synchronized image recording devices. Synchronization is typically carried out by establishing a wire-based connection between the image displaying devices and the image recording devices to a so-called master clock.

[0016] Description of Related Art

[0017] US 2023 / 0113256 A1, titled "Background Display Device, Camera and Method of Displaying a Representation of a Virtual Background," discloses a system and method for displaying virtual backgrounds in a virtual image recording studio. The background display device, typically an LED wall comprising actively illuminating picture elements (e.g.,

[0018] M / 65050-PCT Max von Braun

[0019] 4

[0020] LEDs or OLEDs) arranged in a two-dimensional array, displays a representation of a virtual background (e.g., a 3D scene) that can be recorded by an associated camera. The displaying occurs in accordance with a time control, such as a high light pulse frequency (e.g., 1920 Hz or 3840 Hz), while the camera records the display according to a sequence of exposure times. Synchronization between the display's time control and the camera's exposure times is achieved to ensure uniform illumination and avoid imaging artifacts, such as brightness modulations or tearing effects during background refresh. This synchronization may involve adapting frequencies, phase positions, or durations of exposure times to match integer multiples of the reciprocal of the light pulse frequency, often using network protocols like Precision Time Protocol (PTP). The system is designed for controlled studio environments, where the background display can also illuminate the real subject (e.g., an actor) to simulate virtual light sources, enabling real-time interaction without extensive post-production. This prior art addresses synchronization in virtual production settings, it is limited to scenarios involving a single or small number of expensive professional cameras in a fixed, controlled environment, typically relying on direct or networked control rather than fully wireless methods. Such professional cameras are capable of recording at a very precise preset frame rate without significant jitter. Prior art does not contemplate synchronizing with a plurality of uncontrolled compact image recording devices, such as smartphone cameras, drone cameras or traffic cameras which introduce additional complexities like variable frame rates, independent clock drifts, and heterogeneous hardware. Furthermore, the focus of this prior art document is on pulsed illumination for studio virtual backgrounds to prevent artifacts especially in rolling shutter or global shutter cameras or more generally recording settings with at least 180° shutter angle.

[0021] Technical Problem

[0022] However, when a plurality of compact image recording devices is involved, a wire-based connection to a master clock is no longer feasible. For instance, considering a sports event where a large audience with mobile phones is present at the event and records a scenery including image displaying devices, such as LED panels, and it is intended to show targeted advertisement during the recording, for instance in the native language of the user of the mobile device. Then, the technical problem arises of synchronizing the image recording devices of the individual users with the dedicated image content shown on the image displaying devices.

[0023] M / 65050-PCT Max von Braun

[0024] 5

[0025] Synchronizing an image displaying device with a plurality of compact image recording devices, such as mobile phone cameras, presents unique technical difficulties not encountered in controlled environments with professional cameras or single-device setups. Mobile phones typically employ variable frame rates rather than constant frame rates, where the actual recording frame rate fluctuates around nominal values like 30 fps or 60 fps (often approximating 59.94 fps due to legacy NTSC standards or 50 fps for PAL standard). This variability arises from several factors:

[0026] Power and Processing Optimization: Compact devices dynamically adjust frame rates to conserve battery life and manage processor load. For example, in low-light conditions or during high-motion scenes, the frame rate may drop temporarily to prioritize image quality or reduce heat generation, leading to inconsistencies across devices. Clock Drift and Hardware Heterogeneity: Each mobile phone has an internal recording clock that drifts over time due to manufacturing tolerances, temperature fluctuations, and aging components. Unlike professional cameras with precise, synchronized clocks, compact devices from different manufacturers (e.g., iOS vs. Android) or models exhibit varying drift rates, making uniform synchronization challenging without ongoing adjustments. Environmental and Software Influences: Frame rates can vary based on ambient lighting, motion detection algorithms, or software features like autoexposure and stabilization or flicker reduction caused by tube lights. For instance, Android devices using the Camera2 API may introduce micro-variations in frame timing, while iOS devices might switch to lower rates in dim environments unless manually overridden. This results in desynchronization over time, with cameras drifting out of phase. Even disregarding variable frame rate issues, a displaying device operating at exactly 59.94 fps and a camera operating at exactly 60 fps will drift out of phase by half a frame after approximately 8 seconds.

[0027] Wireless Latency and Scalability: Transmitting synchronization signals wirelessly to hundreds or thousands of devices introduces latency (e.g., 0.25 ms via NTP), compounded by network congestion at events. Prior art approaches, such as those in US 2023 / 0113256 A1, rely on pulsed light synchronization in studio settings with controlled cameras, but do not address the scalability issues of variable frame rate cameras in uncontrolled compact, portable ecosystems, where each device must independently analyze and adapt to display content without a central master clock.

[0028] M / 65050-PCT Max von Braun

[0029] 6

[0030] These challenges necessitate a robust wireless method that accounts for individual device variability, enabling each compact camera to self-synchronize by analyzing display content (e.g., identifier frames) and adjusting its recording parameters dynamically, without requiring hardware modifications or centralized control.

[0031] Therefore, the technical problem underlying the present invention resides in providing a method for wirelessly synchronizing at least one image displaying device and a plurality of independent compact, portable image recording devices. It is also a technical problem of the present invention to provide an image displaying device and / or an image recording device implementing the method of the present invention.

[0032] Summary of the Invention

[0033] This technical problem is solved by the method of claim 1. Preferred embodiments of the method of the present invention are subject to the dependent claims.

[0034] The invention provides a method for wirelessly synchronizing at least one image displaying device and a plurality of independent compact, portable image recording devices. In the method, the image displaying device operates at a base displaying frame rate (BDFR), wherein each display image frame is subdivided into at least two sub-frames. A first sub-frame shows a first image content which can for instance be intended to be viewed by persons present at a the location of the image displaying device (venue spectators) with their naked and / or captured by professional cameras for broadcasting. A second sub-frame shows a second image content which is intended to be captured and viewed via synchronized compact cameras of the venue spectators. Accordingly, the same venue spectators have a different viewing experience when looking at a scenery which includes image displaying devices with their naked eyes or when looking at the same scenery via images captured by their mobile phones. .This allows, for example, venue spectators to perceive one advertisement while mobile phone recordings capture a different, targeted advertisement in synchrony with the display output.

[0035] The compact image recording devices capture a sequence of recording image frames at a recording frame rate (RFR). To ensure reliable capture of complete sub-frames, the method adjusts the phase shift between displaying and recording image frames. This phase alignment is achieved either by synchronizing both devices with a globally available

[0036] M / 65050-PCT Max von Braun

[0037] 7 wireless master clock signal or by analyzing the image content of the recording frames, wherein the image content comprises embedded display parameters of the display sequence.

[0038] By combining time-multiplexed multi-content display frames with robust wireless synchronization, the invention enables simultaneous presentation of different image contents to direct viewers and to independent compact recording devices without requiring a wired connection. This allows, for example, venue spectators to perceive one advertisement while mobile phone recordings capture a different, targeted advertisement in synchrony with the display output.

[0039] Detailed Description of the Invention

[0040] According to one embodiment, the present invention concerns a method for wirelessly synchronizing at least one image displaying device and aa plurality of independent compact image recording devices, comprising the steps of: a) displaying a sequence of display image frames at a base displaying frame rate (BDFR) with said image displaying device, wherein each display image frame comprises at least two sub-frames, a first sub-frame showing a first image content and a second sub-frame showing a second image content; b) recording a sequence of recording image frames of scenery including said sequence of display image frames with said image recording device at a recording frame rate (RFR); c) optionally, synchronizing said base displaying frame rate (BDFR) and said recording frame rate (RFR) such that said recording frame rate (RFR) is approximately an integer multiple or an integer fraction of said base displaying frame rate; and d) adjusting a phase shift between said sequence of displaying image frames and said sequence of recording image frames such that each recording image frame captures at least one of said first and second sub-frames of said displaying image frames, wherein said phase shift adjusting step .d) is effected by d1) aligning said sequence of display image data and said sequence of recording image frames with a wirelessly transmitted globally available master clock signal, and / or

[0041] M / 65050-PCT Max von Braun

[0042] 8 d2) aligning said sequence of recording image frames with said sequence of display image frames by analyzing image content of said recording image frame, said image content comprises display parameters of said sequence of display image data.

[0043] For the purpose of the present invention, the following terms are used with the following meanings:

[0044] Image displaying device: The term "image displaying device" denotes any electronic device capable of presenting a sequence of electronically generated display image frames at a defined base displaying frame rate (BDFR). This includes: direct image generation on panels, such as LED panels, OLED panels, LCD panels or comparable flatpanel display technologies; projection-type devices, such as LED projectors, laser projectors including RGB laser projectors and phosphor-based laser projectors, or other optical projectors that project electronically generated image data onto a passive surface such as a screen, and indirect lighting devices, such as lighting installations or ambient illumination panels, which can modulate light intensity, color, or patterns to encode image content or synchronization information. The image displaying device is capable of showing at least a first and a second image content in a time-multiplexed manner within sub-frames of the display image frames. Furthermore, the image displaying device may embed synchronization information into the displayed content, for example in the form of optical patterns or ambient light modulations, which can be captured by the image recording devices for alignment purposes. In contrast, professional broadcast cameras with dedicated sync lines are not the primary focus of the invention.

[0045] Independent compact image recording devices: In the context of the present invention, the focus is on compact image recording devices with a small form factor and consumer-grade hardware, especially portable cameras such as mobile phone cameras, tablet cameras, laptop-integrated cameras, wearable cameras such as bodycams, or cameras mounted on unmanned vehicles such as drones but also nonportable compact cameras such as CCTV cameras or traffic control cameras. These devices are distinguished from professional broadcast or studio cameras by their portability, autonomous operation, and the absence of a fixed wired synchronization infrastructure. Each device typically operates with its own internal recording clock and frame rate, which may vary between devices. The synchronization method of the present invention enables such independent and compact image recording devices to reliably capture intended sub-frames of the display image frames.

[0046] M / 65050-PCT Max von Braun

[0047] 9

[0048] Plurality: the term "plurality" denotes two or more. In particular, a plurality of independent compact image recording devices comprises at least two separate devices, for example mobile phone cameras of different users, each operating independently and without a wired connection to the image displaying device. Preferably, plurality refers to 10 or more, more preferably 100 or more independent compact image recording devices. At large events, the present invention can refer to tens of thousands compact image recording devices.

[0049] Base displaying frame rate (BDFR): the base displaying frame rate denoted in frames per second (fps) or Hz refers to the overall frame rate at which the image displaying device outputs display image frames. Each display image frame may in turn be composed of a plurality of sub-frames and image slots as defined below. The BDFR therefore corresponds to the rate of complete display image frames, not to the rate of sub-frames. Typically, when considering video image data, the base displaying frame corresponds to the frame rate at which different image content of the same video stream is present. A base displaying frame rate is associated with a base displaying frame time interval 1 / BDFR. Typical base displaying frame rates (and their associated a base displaying frame time intervals) in the context of the present invention are 24 fps (41.67 ms), 29.97 fps (33.37 ms), 50 fps (20 ms), 59.94 fps (16.68 ms), 60 fps (16.67 ms) and 120 fps (8,33 ms).

[0050] First image content: the term "first image content" denotes coherent image data presented within a sub-frame of a display image frame. The first image content may be a video stream, a sequence of still images, or static content such as tracking patterns or chroma key images. The first image content is directed to a first group of recipients, such as direct viewers of the display device and / or broadcast cameras recording the scene for transmission.

[0051] Second image content: the term "second image content" denotes image data different from the first image content, likewise coherent in itself, and presented within another sub-frame of the same display image frame. The second image content may also comprise a video stream, a sequence of still images, or static content such as identifier frames, tracking patterns, or chroma key images. The second image content is directed to a second group of recipients, in particular the independent compact image recording devices which capture this content in a synchronized manner. Typically, the second image content is present in a manner not consciously perceivable by direct viewers.

[0052] Sub-frame: the term "sub-frame" denotes a temporally delimited portion of a base display image frame shown at the base displaying frame rate. Each display image

[0053] M / 65050-PCT Max von Braun

[0054] 10 frame comprises at least two sub-frames, each showing different image content in a time- sliced multiplexed manner. Sub-frames refer essentially refer to a conceptual sub-division of a base display image frame and consist of one or more image slots.

[0055] Image slot: The term image slot or just "slot" denotes a technical subdivision of a base display image frame time interval in which the image displaying device is configured to display image data with different parameters, e.g. different image content and / or different image brightness. The durations of each image slot can vary but typically, each image slot has the same duration. The total duration of all image slots of a base displaying frame time interval add up to the duration of the base displaying frame time interval. For instance, at a BDFR of 60 fps, the base displaying frame time interval of 16.67 ms can consist of 24 image slots, each having a duration of 0.69 ms.

[0056] In the method of the present invention, the image displaying device can be any electronic device which is capable of displaying a sequence of electronically generated display image frames. In the context of the present invention, "displaying" comprises planar displays made out of an array of light emitting elements, such as LED or OLED elements, but also projector-type devices which include an optic for projecting active elements within the image displaying device onto a passive surface, such as a screen. Typical projector-type devices include LED projectors or laser projectors. Typical panel-type devices include LED panels or OLED panels.

[0057] According to step a), the at least one image displaying device displays a sequence of display image frames at a base displaying frame rate BDFR. For instance, if a conventional video is shown on the image displaying device, a base displaying frame rate BDFR = 60 fps can be chosen, i.e. each display image frame is shown for a time period of 16.7 ms. Each display image frame of the sequence of display image frames contains at least two sub-frames, a first sub-frame showing a first image content, for instance a first video intended to be captured by a first sub-group of image recording devices or direct viewers with their naked eyes and second sub-frame showing a second image content, for instance a second video, intended to be captured by a second sub-group of image recording devices. For the first image content, the corresponding sequence of display image frames constitutes a first set of image data and for the second image content, the corresponding sequence of display image frames constitutes a second set of image data. As a matter of course, the sequence of display image frames can comprise more than two sets of different image data. Typically, these at least two sets of different image data are

[0058] M / 65050-PCT Max von Braun

[0059] 11 displayed in a time-sliced multiplexed manner with the image displaying device as for instance described in patent application US 2009 / 102957 A1 (Phelan). In addition, the sequence of display image frames can comprise image frames which are not intended to be captured by an image recording device, for instance inverse images of image frames of the sequence in order to hide the respective image content from direct viewers by creating a perceived homogeneous neutral image, as for instance described in international patent application WO 2018 / 138366 (Appario).

[0060] The image recording device used in step b), can be any compact, optionally portable device as defined above which is capable of capturing a scenery with a certain recording frame rate. Generally, the recording frame rate corresponds to the video frame rate VFR of the final video stream generated from the sequence of recording image frames. However, the recording frame rate can be higher than the video frame rate which results in a slow-motion effect for the final video if all captured recording image frames are used for generating the final video. Likewise, the recording frame rate can be lower than the final video frame rate VFR, resulting in a time-lapse effect. Moreover, the recording frame rate can also be higher than the final video frame rate VFR, for instance if the sequence of display image frames comprises at least two different sets of image data and the camera is configured to capture both sets of image data. From the initial sequence of recording image frames, at least two videos can be created each having a predetermined video frame rate VFR. For instance, for two sets of image data, the recording frame rate is typically twice the video frame rate (RFR = 2 x VFR) to capture both sets and two video streams each having a video frame rate VFR can be created.

[0061] In practice, the base displaying frame rate (BDFR) of the display device and the recording frame rate (RFR) of a compact camera will not always match exactly, and a phase shift may occur between the respective display image frames and recording image frames, i.e. a variable time offset between the leading edge of a display image frame and the leading edge of a recording image frame. Many compact cameras, in particular mobile phone cameras, typically operate at 60 fps. When the display device likewise operates at or close to 60 fps, a precise frequency synchronization is not strictly required, provided that a phase adjustment is performed at regular intervals to compensate for small deviations. In such cases, only an approximate synchronization of the frame rates is sufficient. However, if there is a larger difference between the base displaying frame rate and the recording frame rate, step c) of the method can be employed to adapt one frame rate to the other,

[0062] M / 65050-PCT Max von Braun

[0063] 12 for example to an integer multiple or an integer fraction. The closer the frame rate synchronization, the less frequently a phase adjustment is needed.

[0064] Finally, it has to be ensured that a recording image frame only captures images which include a single display image frame, i.e. a recording image frame should not overlap with subsequent display image frames. As there is initially no link between the displaying of the sequence of display image frames and the recording of the sequence of recording image frames, such an overlap can generally not be avoided. Therefore, according to step d), a phase shift between the sequence of display image frames and the sequence of recording image frames is adjusted such that each recording image frame captures a portion of a single displaying image frame. In the context of the present invention, the term "portion" comprises any temporal fraction of a single displaying image frame up to and including 100 percent of the time period of a single displaying image frame.

[0065] In the context of the present invention, it is preferred that the accuracy of the clocks should be in the microsecond (ps) range, i.e. the time difference At between the display-clocks and the recording-clocks should preferably not be greater than 1 to 10 ps. Under certain conditions, for instance when using long display image frame with black phases at the front and end parts of the display image frames, an accuracy of up to 100 ps may still be acceptable. Lower accuracy, e.g. in the range of 200 ps will require additional compensation measures. Such high accuracies in the microsecond range can be achieved if both the display-clocks and the recording clocks are synchronized with an external wireless master clock signal.

[0066] Therefore, according to a preferred embodiment of the present invention, the synchronizing step c) comprises the sub-step of adapting said base displaying frame rate (BDFR) and said recording frame rate (RFR) to a wireless master clock signal. Thus, the display-clocks and the recordings clocks can be synchronized to the same master clock signal. When a very stable master clock signal is provided, timing accuracies of 1 ps and better can be achieved. This embodiment addresses the above mentioned synchronization challenges by compensating for manufacturing tolerances and environmental variations, ensuring robust and reliable wireless synchronization between image displaying devices and image recording devices.

[0067] M / 65050-PCT Max von Braun

[0068] 13

[0069] According to a first embodiment of step d), denoted step d1), the sequence of display image frames and the sequence of recording image frames are both aligned with a wirelessly transmitted globally available master clock signal. Suitable master clock signals include, for example, satellite-based timing signals (GPS, Galileo, GLONASS, BeiDou), terrestrial radio time signals such as DCF77 in Central Europe, or comparable radio time services worldwide (for example WWVB in the United States, MSP in the United Kingdom, or JJY in Japan), as well as network-based timing signals obtained via the internet, e.g. Network Time Protocol (NTP) or Precision Time Protocol (PTP). By synchronizing both the display clock of the image displaying device and the recording clocks of the compact recording devices to the same global reference, very high timing accuracies in the microsecond range can be achieved, thereby ensuring reliable phase alignment even for a large number of independent devices. In the context of the present invention satellitebased atomic clock signals are preferred because e.g. GPS is far more precise and globally available, making it better for high-accuracy applications like synchronized imaging in compact devices. DCF77 is simpler and cheaper but less accurate and regionally limited, suitable for less demanding tasks. GPS accuracy is in the 1-10 microseconds range in consumer devices due to precise atomic clocks and corrections for signal delays.

[0070] According to a second embodiment of step d), denoted step d2), the phase shift adjustment is achieved by analyzing the image content of the recording image frames themselves. For this purpose, the display image frames may include display parameters or identifiers embedded in the image content, for example in the form of monochromatic fields, bar codes, QR codes, or other patterns. These identifiers are captured by the image recording devices and analyzed to determine the precise timing of the display image frames. On this basis, the phase of the recording sequence is adjusted until each recording image frame reliably captures an intended sub-frame of the display image frames. This alternative has the advantage that no global timing infrastructure is required, since the necessary synchronization information is encoded in the displayed image content itself.

[0071] In one embodiment of the invention, the globally available master clock signal used in aligning step d1) corresponds to a Coordinated Universal Time (UTC) signal. UTC signals provide a standardized international time reference derived from atomic clocks and disseminated via various infrastructures, including satellite-based navigation systems, terrestrial radio time stations (e.g. DCF77, WWVB, MSF, JJY), and internet-based time servers (e.g. NTP or PTP services). By aligning both the display image frames and the

[0072] M / 65050-PCT Max von Braun

[0073] 14 recording image frames to a UTC signal, all participating devices share a common absolute time base. For example, a fixed UTC reference time such as 00:00:00.000000 (midnight) each day can be used as a baseline for phase calculations. Synchronization can then be performed by determining the time offset of the internal display clock and recording clock relative to this UTC reference. Depending on the transmission medium, timing accuracies in the microsecond range can be achieved, in particular when satellitebased signals are used, whereas terrestrial radio time stations typically provide accuracies in the millisecond range, which may still be sufficient for reliable synchronization when regular phase adjustments are applied.

[0074] This alignment with UTC can also be accomplished when base displaying frame rate BDFR and / or recording frame rate RFR have non-integer values in the form ab.cd fps, e.g. 59.94 fps. The sequence of frames align again with full seconds at least after 100 seconds, i.e. after abed frames, in case of even d at least after 50 seconds or abcd / 2 frames. This guarantees that after 100 seconds, an integer number of display frames has been completed and the frame sequence resets to the same phase position. If the denominator simplifies, the base interval may be further reduced. For example, when the decimal part allows simplification by 2, the sequence repeats after 50 seconds, i.e. in the case of 59.94 fps after 2997 frames.

[0075] By aligning both the display device and the recording devices to the same UTC-derived atomic time base, the frame phase can be corrected or re-initialized at these predictable synchronization points (every 50 or 100 seconds). This approach ensures highly accurate and robust synchronization across a large number of independent recording devices while requiring only minimal correction effort in between. Even in the presence of small clock drift, precise frame alignment is automatically restored at the next base interval, as long as all devices are disciplined by the same atomic clock signal.

[0076] In a further embodiment of the invention, corresponding to step d2), the image content of the recording image frames comprises an optical pattern embedded within the sequence of display image frames. Such an optical pattern may be implemented as a one- or two-dimensional code, such as a bar code, a QR code, a checkerboard pattern, or other machine-readable symbols. Even a second video stream could be a marker by using e.g. differential images to extract data. The optical pattern is displayed within a defined subframe of a display image frame and is captured by the image recording devices. By

[0077] M / 65050-PCT Max von Braun

[0078] 15 detecting and decoding the optical pattern in the recorded image, the recording device can determine the exact temporal position of the corresponding sub-frame within the base displaying frame rate. This information allows the device to adjust its phase so that subsequent recording image frames reliably capture complete sub-frames intended for the device. The use of optical patterns has the advantage that synchronization can be achieved without reliance on external timing infrastructures, since the necessary synchronization markers are directly visible in the displayed content. Depending on the size, contrast, and duration of the optical pattern within a sub-frame, reliable detection can be achieved with high temporal accuracy, typically in the sub-millisecond range, which is sufficient to maintain phase alignment in practical scenarios.

[0079] In another embodiment of the invention, corresponding to step d2), the image content of the recording image frames comprises an ambient light pattern. In this embodiment, synchronization information is not necessarily displayed within the visible display surface captured by the recording device, but instead is imparted onto the ambient illumination of the scenery. For example, LED lighting panels or spotlights arranged in the environment may be modulated in intensity, color, or other optical properties to encode synchronization signals. These ambient light variations are captured by the camera sensor even if the lighting panels themselves are not within the field of view of the camera. By analyzing the temporal characteristics of the captured ambient light pattern, the recording device can determine the phase relation between its recording image frames and the display image frames. This approach allows synchronization even in scenarios where the display content does not visibly include identifiers or patterns, and it extends the applicability of the method to large-scale installations where dedicated lighting systems can provide a global optical timing reference. Depending on the modulation scheme, the detection accuracy of the ambient light pattern can reach the sub-millisecond range, ensuring reliable phase alignment across a large number of independent compact recording devices.

[0080] One embodiment of the present invention can be employed if the image displaying devices can be controlled / modified. In this case, the recording frame rate RFR of the image recording device is considered fixed and acting as a master and the synchronizing step c) is essentially implemented by determining the recording frame rate RFR of the image recording device and adapting the image base displaying frame rate BDFR of the image displaying device accordingly to the recording frame rate RFR, for instance by

[0081] M / 65050-PCT Max von Braun

[0082] 16 changing a frequency of a displaying clock of the image displaying device. For instance, the recording frame rate can be read manually on the image recording device and the image displaying device can be configured manually as well. More preferably, however, the image recording device can read-out its recording frame rate and transmit the recording frame rate RFR via a wireless network to the image displaying device and the image displaying device can change its displaying clock accordingly. In this case, the phase shift adjusting step d) comprises phase shifting the sequence of display image frames. Accordingly, for implementation of this variant, it is also necessary to control the image displaying device. Moreover, determining, whether an appropriate phase shift has been established, requires also feedback from the image recording device which can be accomplished in various manners described below in connection to an embodiment, where the method of the present invention is implemented in the image recording device. However, this scheme is less suitable for the intended use case of the invention involving a plurality of independent compact image recording devices, such as mobile phones operated by different users. In such a scenario, there is no single "master" camera to which the display could be adapted, because each device has its own recording frame rate and internal clock. Adapting the display device to one camera would not solve the synchronization problem for the others. This embodiment may only be of practical relevance if all recording devices happen to operate with approximately the same recording frame rate RFR, which can be the case for certain classes of devices (e.g. consumer mobile phones operating at 60 fps). For larger variations between devices, the more robust synchronization approaches described in connection with steps d1) and d2) are preferred.

[0083] Generally, both the recording clock and the displaying clock will experience a certain drift over time so that the synchronizing and phase shift adjusting steps c) and d) will have to be repeated at certain intervals.

[0084] A drawback of the first embodiment of the present invention resides in the fact that adjusting the image displaying device to an image recording device is only feasible when there is only one image recording device or a set of already synchronized image recording devices. In a general setting, for instance if an audience of many mobile phone users employs their cameras to capture a scenery, these individual cameras will be both in recording frequency and in respective phases with respect to the sequence of display image frames. Conversely, a multitude of image displaying devices, for instance an array

[0085] M / 65050-PCT Max von Braun

[0086] 17 of LED panels creating a large LED screen, will generally provide no problem because the individual LED panels are interconnected and synchronized in order to display a meaningful video content.

[0087] Therefore, according to a preferred second embodiment of the present invention, the synchronizing step c) comprises the sub-steps of determining the base displaying frame rate BDFR of the image displaying device and adapting the recording frame rate RFR of the individual image recording devices to the base displaying frame rate BDFR by changing a frequency of the individual recording clocks of the individual image recording devices. This embodiment assumes the usual case that there is only one base displaying frame rate BDFR but multiple image recording devices each having their own recording frame rate and phase shift with respect to the sequence of display image frames.

[0088] In this embodiment, the base displaying frame rate BDFR can be determined by reading out a frequency of a display-clock of the image displaying device and wirelessly transmitting the display-clock frequency to the image recording device. However, wireless networks have a significant latency which limits the accuracy of the synchronizing step c) and the phase shift adjustment step d). For instance, using a variant of the Network Time Protocol (NTP / PTP), which includes an estimate for network latency, an accuracy of around 0.25 ms can be achieved.

[0089] The present invention teaches various solutions to achieve a better accuracy in the synchronizing step c).

[0090] In one embodiment of the invention, the base displaying frame rate BDFR is determined by analyzing the image content of the sequence of recording image frames. Steps c) and d) of the method of the present invention are therefore implemented in the image recording device and any negative influence of network latencies is avoided.

[0091] There are various methods for analyzing the image content of the sequence of recording image frames. For instance, it might be feasible to analyze the overall image for lighting or contrast changes on a pixel level thus identifying individual display image frames which then allows determining the respective base displaying frame rate BDFR. Preferably, however, the sequence of display image frames comprises a pre-determined identifier image frame, for instance a color coded image frame such a monochromatic image frame

[0092] M / 65050-PCT Max von Braun

[0093] 18 or a one- or two-dimensionally patterned image frame, such as a bar code or a QR code. When the camera software is adapted to identify the occurrence of such pre-determined identifier image frames, the corresponding analysis is more time-effective than analyzing generally unknown image content. Such a pre-determined identifier image frame can be much shorter than regular video content shown with the image displaying device because the identifier image frame will generally not constitute a meaningful video content for the final video.

[0094] If a one- or two-dimensional patterned image is used as the pre-determined identifier image, such a patterned image, for instance a bar code or a QR code, can also include additional information, for instance the display frame rate itself can be coded in the patterned image. Thus, in cases where the display clock frequency is stable, it is not necessary to determine the base displaying frame rate from the interval between predetermined identifier image frames but the base displaying frame rate can also be readout directly from the information comprised in the patterned image.

[0095] Generally, however, there will be an inherent variability between the displaying-clock of the image displaying devices and the recording-clocks of image recording devices. For instance, due to manufacturing tolerances, these internal clocks will exhibit slight discrepancies in their oscillation frequencies. Additionally, environmental factors such as temperature differences can cause variations in the clock frequencies over time. These differences result in clock drift, where the internal clock of the image displaying device, referred to as the "display-clock," and the internal clock of the image recording device, referred to as the "recording-clock," gradually lose synchronization.

[0096] For instance, even a minor frequency mismatch can lead to cumulative timing errors over prolonged operation, disrupting the alignment of displayed and recorded frames. This issue is exacerbated in scenarios requiring high precision, such as capturing hidden content displayed for a fraction of a frame time interval. Without a mechanism to dynamically correct for clock drift, stable synchronization cannot be maintained, compromising the quality and accuracy of recorded content.

[0097] The master clock signal can be provided via radio, Bluetooth, internet or via a satellite navigation system To implement this embodiment, both the image displaying and

[0098] M / 65050-PCT Max von Braun

[0099] 19 recording devices are equipped with modules for receiving and processing the atomic clock signals, whether through internet-based NTP- or PTP-servers or satellite navigation systems. The modules ensure that the synchronization process is automatic, requiring minimal user intervention. This guarantees accurate frame alignment between the displaying and recording devices, enabling high-quality content capture and playback without artifacts caused by timing mismatches. In a further embodiment of the invention, the master clock signal may also be derived from the mains frequency of the local power grid. In many regions, the electrical supply grid operates at a highly stable nominal frequency of 50 Hz or 60 Hz, which is continuously monitored and disciplined to longterm atomic clock standards by grid operators. By using the grid frequency as a timing reference, both the display device and the recording devices can align their respective clocks to the zero-crossings of the alternating current. Although the short-term stability of the grid frequency may be lower compared to dedicated atomic time signals, its longterm accuracy ensures that frame alignment remains sufficiently precise for many practical applications. The use of the mains frequency as a master clock signal provides an inexpensive and widely available alternative to satellite navigation systems, radio time signals, or internet-based time synchronization.

[0100] This embodiment not only ensures superior synchronization accuracy but also enhances the versatility and adaptability of the system, making it suitable for a wide range of applications and operating environments.

[0101] In a further embodiment of the present invention, the image displaying device is equipped with a satellite-based, for instance a GPS-based Sync Pulse Generator (SPG) that functions as a master clock, providing a synchronization signal to the compact recording devices. The SPG utilizes highly accurate timing information obtained from GPS satellites to generate synchronization pulses. These pulses define the timing for each display frame and serve as a reference for the recording devices.

[0102] The synchronization signal generated by the SPG can be transmitted to the compact recording devices using various wireless communication methods, such as Wi-Fi, Bluetooth, or RF transmissions on a dedicated frequency band. For example, the SPG can embed timing data within a low-latency broadcast signal, which is periodically transmitted to all recording devices within range. Alternatively, the SPG may use a coded light synchronization method, where imperceptible modulations in the intensity or color of the

[0103] M / 65050-PCT Max von Braun

[0104] 20 displayed content encode the timing data. This coded light is detected by the image sensors of the compact recording devices and used to synchronize their internal recording clocks.

[0105] To ensure robust synchronization in environments with potential interference, the SPG may also implement error-checking mechanisms, such as CRC (Cyclic Redundancy Check), within the transmitted signals. Additionally, the SPG can support real-time adjustments to compensate for drift or latency in the recording devices, providing dynamic resynchronization capabilities. This embodiment ensures that the image displaying device acts as a reliable master clock, enabling compact recording devices to achieve precise frame alignment with minimal dependence on external infrastructure.

[0106] When the method of the present invention is implemented in the image recording device, the phase shift adjusting step d) comprises phase shifting the sequence of recording image frames. Such a phase shift can be accomplished by various techniques, for instance by reset sampling where the camera is restarted until its phase with respect to the sequence of displaying image frames falls within a pre-determined tolerance. Another possibility which allows shifting the sequence of recording image frames is frame injection. Certain camera APIs, such as Android's Camera2 API, allow a high priority capture request to be injected into a normal priority continuous sequence of recording image frames. Typically, the continuous stream is used for the view finder, while the high priority request is used to capture a still image. By injecting a frame with exposure longer or shorter than the individual recording frame time interval 1 / RFR, the phase of a screen can be shifted in one iteration. The above-described variants (reset sampling and frame injection) can be implemented in most commercially available cameras. However, if there is even the possibility to access and modify the camera firmware, a phase shift option can be implemented directly in the camera firmware.

[0107] Typically, the trailing or leading edges of the recording image frames and the displaying image frames are aligned during the phase shift adjusting step d). Preferably, as the frame read-out is accomplished at the trailing edge, it is preferred that the phase shift adjusting step comprises aligning trailing edges of the recording image frames to trailing edges of the displaying image frames. This ensures that the available shutter angle of the image recording device can be maximized to include the whole available displaying image frame, if desired.

[0108] M / 65050-PCT Max von Braun

[0109] 21

[0110] The accuracy of the phase shift adjusting step d) is essentially determined by the precision with which the location of the leading and / or trailing edges of the displaying image frames can be determined by the image recording device. This accuracy is usually determined by the recording frame rate RFR. The higher the recording frame rate, the lower will generally be the resulting image quality of the video because the maximum exposure time for each individual recording image frame is 1 / RFR. However, according to one embodiment of the present invention, it is suggested to increase temporarily the recording frame rate RFR to a high recording frame rate HFR in step b) in order to identify the location of the predetermined identifier image frame with greater precision. The term "temporarily increasing" means in the context of the present invention that the recording frame rate is only increased for such a period of time that the occurrence of an identifier image frame is ensured. For instance, if one identifier image frame is included in each time period 1 / BDFR, it will be sufficient to increase the recording frame rate for one, two or three times the displaying time interval 1 / BDFR in order to provide for a high accuracy phase shift adjustment. This short time interval will generally reduce the image quality of the resulting video for a correspondingly short period of time but this will generally not negatively affect the user experience.

[0111] It will usually also be necessary to adjust a shutter speed of the image recording device in step b) during the phase shift adjusting step d) in order to identify the precise location of the pre-determined identifier image frame with even more precision.

[0112] In certain embodiments, the phase shift adjusting step d) can comprises phase shifting the sequence of display image frames and phase shifting the sequence of recording image frames.

[0113] In embodiments, where a precise external master clock signal is available, for instance a time signal from an atomic clock, both the sequence of display image frames and said sequence of recording image frames can be adjusted to the wireless master clock signal. Accordingly, in a further embodiment of the present invention, said phase shift adjusting step d) comprises adjusting said sequence of display image frames and said sequence of recording image frames to said wireless master clock signal. Synchronization between the image displaying devices and the image recording device can achieved by aligning the start of their respective image sequences to a certain pre-determined times provided by

[0114] M / 65050-PCT Max von Braun

[0115] 22 the external master clock signal, for instance to full-second intervals. In one embodiment, the image displaying device aligns its sequence of display image frames at precisely defined full-second marks. In order to adjust for processing latencies, the display image frames can include a detectable synchronization signal that is detected by the image recording devices' cameras. This synchronization signal acts as a reference point for both devices to achieve phase alignment, ensuring accurate timing for subsequent image recording and display operations.

[0116] Following the initial synchronization signal, the image displaying device divides the time interval into distinct image slots, which can either be predefined or dynamically determined. The dynamic configuration can be facilitated using a QR code embedded within the synchronization signal or transmitted periodically as part of the data stream. The QR code can serve as a versatile data carrier, providing information about the sequence configuration or a reference link to an internet address where additional data can be downloaded. For example, the QR code might contain metadata specifying the frame timing, sequence identifiers, or instructions for dynamic adjustments.

[0117] More generally, to support robust synchronization, two distinct types of signals can be transmitted: higher frequency synchronization pulses (sync pulses) and lower-frequency data pulses carrying, e.g., the QR code. The sync impulse ensures continuous and precise alignment of the image displaying device and recording device clocks, while the data impulse provides additional configuration details. This dual-signal approach ensures that synchronization is maintained even in scenarios with intermittent or variable data transmission rates.

[0118] The QR code embedded in the synchronization system can also be designed for redundancy to handle potential signal degradation or noise in the operating environment. Additionally, the QR code may include error-correcting codes or checksum information to ensure reliable data decoding by the image recording devices. In cases where the QR code serves as a reference to an online resource, the recording device can seamlessly connect to the specified internet address to retrieve the required sequence data, enabling dynamic updates and configuration flexibility.

[0119] This embodiment further enhances the robustness and adaptability of the synchronization method, offering a scalable solution for various use cases, including environments with

[0120] M / 65050-PCT Max von Braun

[0121] 23 multiple image recording devices or complex image displaying configurations. By combining precise phase alignment with dynamic data provisioning, the invention ensures superior synchronization accuracy and operational efficiency.

[0122] While the synchronizing step c) and the phase shift adjusting step d) of the method of the present invention will establish a precise synchronization of the image displaying device and the image recording device for a certain period of time, it will usually be necessary to repeat steps c) and d) at regular intervals while continuously performing displaying step a) and recording step b) in order to compensate for drift of the recording-clock or the displaying-clock. Depending on the required accuracy, steps c) and d) will typically be repeated once every ten seconds or once every minute or even once every five minutes.

[0123] The present invention also concerns an image displaying device and / or an image recording device implementing the method of the present invention.

[0124] The present invention will now be described in more detail with reference to preferred embodiments schematically depicted in the attached drawings. In the drawings:

[0125] Brief Description of the Drawings

[0126] Fig. 1 shows a typical broadcast scheme of a tennis event using a video camera;

[0127] Fig. 2 shows a video image frame of a video stream obtained with the video camera of Fig. 2;

[0128] Fig. 3 shows a time line of a typical image capturing scheme according to the present invention; and

[0129] Fig. 4 shows a more detailed embodiment of a display image frame.

[0130] Description of Preferred Embodiments

[0131] Fig. 1 shows a typical broadcast scheme of a tennis event. Generally, several cameras are distributed around a tennis court 10. For the sake of simplicity, in the present example only one camera 11 having a camera head 12 and a camera lens 13 mounted to the camera head is shown. Camera 11 is placed behind and above a baseline 14 of tennis court 10. Usually, sports events which are broadcast include various kinds of advertisement and marketing elements. In the present case, an LED signboards 15 is arranged behind

[0132] M / 65050-PCT Max von Braun

[0133] 24 baseline 14 of tennis court 10. Camera 11 is operating at a standard video frame rate SVFR, for instance at a standard video frame rate of 12, 24, 25, 30, 50 or 60 fps (frames per second). Especially at events where fast moving objects such as tennis balls are part of a scenery, the camera is usually operated at a shutter time / exposure time which essentially corresponds to the associated standard video frame rate time interval • T = 1 / SVFR, i.e. in the example of SVFR = 50 fps at a standard video frame rate time interval of 20 ms. Any fast moving object such as a tennis ball consequently appears as an extended element blurred along its trajectory over the exposure time of 20 ms. This allows broadcast viewers to more easily perceive the movement of the ball because "jumps" of the ball between consecutive video image frames are avoided. The final result is a blurred line following the path of the tennis ball.

[0134] In the example of Fig. 1, a data transmission line 16 connects camera 11 with a camera control unit (CCU) 17 installed in an OB van 18. A raw video output line 19 connects the camera control unit 17 with a vision mixer 20 which has multiple inputs 21a - 21e for video data so that graphic, text or video inserts or overlays can be added to the raw video captured by camera 11. The final video stream generated by the vision mixer (and optional other video processing equipment not shown in Fig.1) is then provided at a video output 22 of the vision mixer as an output video stream which can, for instance, be broadcast at the standard video frame rate via an output transmission line 23 and a satellite dish 24 installed on top of the OB van 18 to the broadcast viewers. As a matter of course, wirebased transmission using, e.g. a fiber line, or transmission via a broad band cellular network, e.g. a 5G network, are also possible.

[0135] In the present embodiment, an OB van is only a typical example for an existing broadcasting technology. However, remote broadcast technology can be employed as well where rather than transmitting final video feeds from an OB van to a rebroadcast center, raw camera video feeds are transmitted to a remote location where they are processed / assembled into the final video signal.

[0136] Fig. 2 shows a typical video image frame obtained by projecting an image caught be the camera lens 13 of camera 11 onto the image sensor of camera 11. The video image frame has an outer image border 30 and an inner image region 31 corresponding to the camera image captured during a standard video frame rate time interval • T. In the present example, camera 11 of Fig. 1 captures a scenery which comprises the tennis court 10. In

[0137] M / 65050-PCT Max von Braun

[0138] 25 the present example, one tennis player 32 is in the process of serving while the second player 33 is expecting the serve. LED signboard 15 shows advertisements intended to be seen by the direct viewers (spectators) 34 and is therefore part of the scenery shown in the image frame of Fig. 2. Conventionally, the same advertisement is shown to all broadcast viewers. As explained in connection with Fig. 4, the shutter angle of camera 11 is usually large, e.g. between 270 and 360° (the latter orresponding to the standard frame rate time interval of e.g. 20 ms (50 fps) that the fast moving tennis ball 35 is no longer a sharp object but motion-blurred along a part of its trajectory. This effect is intended in conventional recording of tennis matches to increase visibility of the tennis ball for the broadcast viewers because it avoids "jumps" of the tennis ball 35 from one video image frame to the next.

[0139] When broadcasting international sports events, it became common to employ multicontent advertisement. Multi-content advertisement in the sense of the present invention means that on one and the same advertisement signboard which is part of a scenery captured by the camera, different types of image content are shown for different sub-sets of viewers, for instance the direct viewers present at the event and one or more different sets of broadcast viewers. Therefore, the dedicated image content (e.g. different advertisements) can be presented simultaneously throughout the event to the direct viewers and different sets of broadcast viewers, for instance to broadcast viewers in different countries where each can see an advertisement in their native language. To effect multi-content advertisement, different technologies have been developed in the past. For instance, technologies denoted "virtual advertisement" use image processing technology to identify the location of the physical advertisement signboards in a scenery of the video stream and selectively replace the area of the detected signboards by alternative image content for different sub-sets of broadcast viewers resulting in different final videos being produced of the event where each shows the same scenery but the advertisement boards show different advertisements. This technique still faces problems with properly identifying background and foreground (for instance, players moving before the signboards), etc. In another technology denoted "dynamic content multiplication" described in international patent application WO 2018 / 138366 A1, every type of advertisement of the multi-content advertisement is actually shown on the physical display but only one type long enough so that it can be perceived by the direct viewers present at the event (the so-called venue feed). Additional alternative advertisements are shown in a time-sliced multiplexed manner on the physical display for only brief periods

[0140] M / 65050-PCT Max von Braun

[0141] 26 of time too short to be noticed by the direct viewers. Additionally, in order to avoid the direct viewers experiencing flickering of the physical LED signboards during the event, international patent application WO 2018 / 138366 A1 suggests to present the additional image content (the so-called "parallel feeds") not only as a sequence of the respective images but as a sequence of images and corresponding inverse images so that the resulting inserted image is a grey image which reduces the disturbance of the direct viewers. A camera is employed which captures images synchronized to the particular image content feeds shown on the LED signboards but do not capture the inverse images shown on the LED signboards. Thus alternative image content in the sense of the present invention can be actual advertisements shown on the physical displays which are captured by synchronized cameras or image content which is only intended to identify the location of the signboards in the captured camera images for replacing the image content of the signboards by alternative image content in video post-production. The alternative image contents shown on the physical displays can in this case be for instance monochromatic chroma key images or pattern images. Such pattern images can also include tracking patterns which allow to establish the exact viewing angle / orientation between camera and therefore facilitate exact discrimination between background and foreground.

[0142] Such additional image content is usually presented for such short time intervals that it is not consciously seen by the direct views. Moreover, it is also not seen by broadcast viewers because it is only serves as an information source for video post-processing and will ultimately be replaced by different image content in the final video stream sent to broadcast viewers.

[0143] Fig. 3 shows an implementation of the method of the present invention according to an embodiment where the base displaying frame rate BDFR is pre-set and the method of the present invention is implemented in the recording device so that the recording frame rate RFR and the phase shift of the sequence of the recording image frames is adapted to the sequence of the display frames.

[0144] We assume that the LED signboards at the tennis event of Fig. 2 show a sequence of display image frame in such a manner that additional video content not intended for the direct viewers present at the event is shown. Fig. 3a) shows a sequence of display image frames where we assume that a camera capturing a video of the event is operating at a standard video frame rate VFR of 60 fps, corresponding to a standard video frame time

[0145] M / 65050-PCT Max von Braun

[0146] 27 interval of 1 / 60 s-1= 16.7 ms. The large vertical bars in Fig. 3a) delimit these standard frame rate time intervals of 16.7 ms. The LED signboard operates at a displaying frame rate of 240 fps, i.e. each time interval of 16.7 ms, the LED signboards show in the present example four different sets of image data denoted D1.1, D1.2, D1.3, D1.4, D2.1, D2.2, ..., each having a duration of 4.2 ms. Accordingly, display image frames D1.1, D2.1, etc. correspond to the same set of image data. In the present example, it is assumed that image data D1.1 and D1.2 are actually the same image data intended to be viewed by the direct viewers thus ensuring that this image content is presented for 50 percent of the frame time interval. Image content D1.3, D2.3, etc. is intended to be captured by a camera only, for example containing dedicated video content for foreign visitors present at the sports event in their native language. Image content D1.4, D2.4, etc. can be an inverse image of image content D1.3, D2.3, etc., respectively, in order to result in the perception of a neutral grey image when the combination of D1.3 and D1.4, etc. is viewed by the direct viewers present at the event.

[0147] Fig. 3b) shows the recording sequence of a camera operating in normal camera mode, i.e. capturing a recording image frame R1.1, R2.1 every 16.7 ms. The black rectangle in each recording image frame indicates the shutter angle ending at the trailing edge of each frame, i.e. visualizing the actual exposure time which usually depends on the lighting conditions at the event. In the present case, we assume a shutter angle of 180°, i.e. half of each frame time interval of 16.7 ms is used for the exposure of the frame. As can be taken from Fig. 3b), the first recording frame R1.1 will capture half of the display image frame 1.3, part of its inverse display image frame 1.4 and part of the subsequent image frame 2.1 resulting in a superposition of the respective image content in the captured video image frame.

[0148] In the present example, it is also assumed that the image content D1.3 and the image content D1.4 are not shown for their whole period of 4.2 ms but one half of a short identifier frame IF is shown at the end of frame D1.3 and the start of frame D1.4, respectively. Accordingly, in the image frame captured in Fig. 3b), the identifier frame, e.g. a QR code, is also overlayed over the resulting recoded image R1.1.

[0149] Through image processing, the mobile phone of the user can therefore retrieve from analyzing the QR code information of the captured image R1.1 that the LED panel operates at a display frame rate BDFR of 240 fps.

[0150] M / 65050-PCT Max von Braun

[0151] 28

[0152] In Fig. 3d), the corresponding recording frame rate of the mobile phone camera is adjusted accordingly and operated at full or almost full shutter angle in order to determine in which recoding frame the identifier frame IF is recoded (in this example in frame R2.3). In Fig. 3d) the standard video frame time interval corresponding to recording image frames 2.1 has been shown below the first standard video frame time interval of Fig. 3a1) just to denote the relative phase shift between displaying image frames and recording image frames. In practice, the adaptation of the recording frame takes place in relation to the subsequent displaying image frame D2.1-D2.4.

[0153] As can be seen, there is still a phase shift 5 between the display image frames and the recording image frames and in step c2), the recording frame rate is adjusted to a high recording frame rate HRFR at least during the period corresponding to third frame in the sequence corresponding to third displaying image frame Dx.3 (here shown beneath D1.1- D1.4) but in practice in the third frame of the subsequent standard video frame time interval D3.1-D3.4, i.e. in order to identify the location of the identifier image frame IF more precisely and adjusting the phase of the recording image frames accordingly.

[0154] Thus, in the present example, frame rate synchronization and phase shift adjustment have been accomplished within three standard video frame time intervals, i.e. within 50 ms. Subsequently synchronized video recording commences as shown in Fig. 3d).

[0155] Fig. 3d) shows the resulting adjusted sequence of recording image frames. The identifier can also comprise the information that the third set of image data contains foreign language information so that the camera software discards recording frames R.4.1, R4.2 and R4.4 and generates a video at a standard video frame rate VFR of 60 fps using only image data recorded in recording frames R4.3, R5.3, etc.

[0156] Figure 4 shows a more detailed embodiment of a typical display image frame. In this example, the display image frame is subdivided into a plurality of time slots. The base displaying frame rate (BDFR) is 60 fps, so that each display image frame has a total duration of 16.67 ms. This frame interval is further divided into 24 image slots of equal length, each slot having a duration of approximately 0.69 ms. Different sub-frames can be assigned to groups of these slots.

[0157] M / 65050-PCT Max von Braun

[0158] 29

[0159] In the embodiment shown, a first sub-frame consisting of slots 1 to 6 corresponds to a first video content denoted "V-A," which represents the first image content, i.e. coherent video data intended for direct viewers and / or broadcast cameras. The frame of video V- A is repeated in slots 2, 3, 4 and 5 at double nominal intensity. Slot 1 and slot 6 of this sub-frame show black images, which can be generated either by transmitting a black image to the display or by turning the active light-emitting elements of the display off for the respective slot interval.

[0160] A second sub-frame consisting of slots 7 to 12 corresponds to a static image denoted "S- I," representing a second image content. The native S-l content is displayed in slots 8, 9 and 10, but is neutralized for the human eye by showing in slot 11 an inverse image of S-l at triple intensity. Slot 7 and slot 12 of this sub-frame show black images. The static S-l content may comprise tracking patterns or chroma key images to facilitate subsequent video post-processing. Alternatively or additionally, the static images can include identifier patterns, QR codes, or bar codes which serve as synchronization markers for aligning the phases of the display device and the image recording devices, as described below.

[0161] A third sub-frame consisting of slots 13 to 18 again corresponds to the first video content V-A. Slot 13 and slot 18 show black images, while the intermediate slots 14 to 17 repeat the video frames of V-A at double gain, i.e. at double nominal intensity.

[0162] Finally, a fourth sub-frame consisting of slots 19 to 24 corresponds to a second video content denoted "V-B," which represents a third image content, i.e. coherent video data intended primarily to be captured by independent compact image recording devices such as mobile phone cameras, tablets, or drones. The V-B content may include, for example, foreign-language advertising or additional video feeds tailored to specific target audiences. As in the S-l case, the V-B content shown in slots 20 to 22 is perceptually canceled by an inverse image of V-B shown in slot 23 at triple intensity, so that direct viewers perceive only a homogeneous grey background. Slot 19 and slot 24 of this subframe show black images.

[0163] When a compact image recording device is activated while the sequence of display image frames is shown according to the scheme of Figure 4, the device may initially record the scenery with a large shutter angle and a high recording frame rate to detect image frames with characteristic grey content. In the present example, such grey content may occur in

[0164] M / 65050-PCT Max von Braun

[0165] 30 frames corresponding to slots 8-11 and slots 20-23. When the shutter angle is narrowed and / or the recording frame rate is further increased, the camera can identify synchronization data embedded specifically in the inverse S-l image of slot 11. Using this data, the recording device can align its phase precisely with slot 11 of the display image frame. The synchronization data may include the base displaying frame rate (BDFR), enabling the recording device to adapt its recording frame rate (RFR) if necessary. More importantly, the synchronization data specifies the phase relation and permissible maximum shutter angle to reliably capture the third image content, i.e. the video stream V-B, without overlap with other image contents. For example, the synchronization data may prescribe certain shutter angles (e.g. 15°, 30°, ..., up to 360°) relative to the display image frame.

[0166] The slot structure of the embodiment of Figure 4 thus enables flexible presentation of multiple coherent image contents within each display image frame. The first image content (V-A) dominates visually for direct viewers, while the second image content (S-l) and the third image content (V-B) are perceptually suppressed through inverse-slot neutralization. Nevertheless, independent recording devices can be synchronized and aligned to selectively capture the second or third image content by means of shutter control adapted to the slot structure.

[0167] M / 65050-PCT

Claims

Max von Braun31Claims1. A method for wirelessly synchronizing at least one image displaying device and a plurality of independent compact image recording devices, comprising the steps of: a) displaying a sequence of display image frames at a base displaying frame rate (BDFR) with said image displaying device, wherein each display image frame comprises at least two sub-frames, a first subframe showing a first image content and a second sub-frame showing a second image content; b) recording a sequence of recording image frames of scenery including said sequence of display image frames with said image recording device at a recording frame rate (RFR); c) optionally, synchronizing said base displaying frame rate (BDFR) and said recording frame rate (RFR) such that said recording frame rate (RFR) is approximately an integer multiple or an integer fraction of said base displaying frame rate; and d) adjusting a phase shift between said sequence of displaying image frames and said sequence of recording image frames such that each recording image frame captures at least one of said first and second sub-frames of said displaying image frames, wherein said phase shift adjusting step d) is effected by d1) aligning said sequence of display image data and said sequence of recording image frames with a wirelessly transmitted globally available master clock signal, and / or d2) aligning said sequence of recording image frames with said sequence of display image frames by analyzing image content of said recording image frame, said image content comprises display parameters of said sequence of display image data.

2. The method according to claim 1, wherein, in said aligning step d1), said globally available master clock signal is a Coordinated Universal Time (UTC) signal.M / 65050-PCTMax von Braun323. The method according to any one of claims 1 or 2, wherein in said aligning step d2) said image content of said recording frame comprises an optical pattern within said sequence of display image frames.

4. The method according to any one of claims 1 to 3, wherein in said aligning step d2) said image content of said recording image frames comprises an ambient light pattern.

5. The method according to any one of claims 1 to 4, wherein said synchronizing step c) comprises the sub-steps of determining said base displaying frame rate (BDFR) and adapting said recording frame rate (RFR) to said base displaying frame rate (BDFR) by changing a frequency of a recording-clock of said image recording device, wherein, preferably, said base displaying frame rate (BDFR) is determined by reading-out a frequency of a display-clock of said image displaying device and wirelessly transmitting said display-clock frequency to said image recording device.

6. The method according to claim 5, wherein said base displaying frame rate (BDFR) is determined by analyzing the image content of said sequence of recording image frames.

7. The method of any one of claims 1 to 6, wherein said sequence of display image frames comprise a pre-determined identifier image frame, for instance a color-code image.

8. The method according to any one of claims 1 to 7, wherein said pre-determined identifier image frame comprise a one- or two-dimensional patterned image, such as a bar-code or a QR-code.

9. The method according to any one of claims 1 to 8, wherein the master clock signal is provided via radio, Bluetooth, internet, via a satellite navigation system, or derived from a mains power grid frequency.

10. The method according to any one of claims 1 to 9, wherein said phase shift adjusting step d) comprises phase shifting said sequence of display image frames.M / 65050-PCTMax von Braun3311. The method according to any one of claims 1 to 10, wherein said phase shift adjusting step d) comprises phase shifting said sequence of recording image frames.

12. The method according to any one of claims 10 or 11, wherein said phase shift adjusting step d) comprises aligning trailing edges of said recording image frames to trailing edges of said displaying image frames.

13. The method according to any one of claims 10 or 11, wherein said phase shift adjusting step d) comprises temporarily increasing said recording frame rate (RFR) to a high recording frame rate (HRFR) in said recording step b).

14. The method according to any one of claims 1 to 11, wherein recording step b) comprises adjusting a shutter speed of said image recording device.

15. The method according to any one of claims 7 to 9, wherein said phase shift adjusting step d) comprises adjusting said sequence of display image frames and said sequence of recording image frames to said wireless master clock signal.

16. The method according to any one of claims 1 to 15, wherein said image displaying device is selected from a group comprising LED panels, laser projectors and LED projectors and said image recording device is selected from a group comprising compact digital cameras such as compact video cameras or mobile phone cameras.M / 65050-PCT

Images