Method and system for projecting multiple temporally or spatially multiplexed image streams

Abstract

The invention relates to a method and a system for projecting multiple temporally and / or spatially multiplexed image streams using one or more laser projectors. A standard projecting frame rate at high spatial resolution and a higher projecting frame rate at lower spatial resolution are selected. Each high-rate interval is subdivided into an integer number of sub-frame slots, whose sub-frames are time-shifted to form an output stream with an effectively increased frame rate. Multiple image streams are assigned to the sub-frames in a temporally and / or spatially multiplexed manner and projected onto an image area, optionally comprising a movable diffuser screen for volumetric or hologram-like 3D display. The invention enables higher sub-frame projection rates and volumetric image formation while maintaining high brightness and spatial resolution, thereby overcoming bandwidth limitations of conventional projector systems.

Description

[0001] Max von Braun

[0002] 1

[0003] Method and system for projecting multiple temporally or spatially multiplexed image streams

[0004] BACKGROUND OF THE INVENTION

[0005] Field of the Invention

[0006] The present invention concerns a method and a system for projecting multiple temporally and / or spatially multiplexed image streams onto a surface or a volume, including volumetric 3D displays, using one or more illumination devices, in particular a laser projector, with temporal and / or spatial multiplexing to overcome bandwidth limitations in high-frame-rate or high-resolution applications, including volumetric 3D displays.

[0007] Description of Related Art

[0008] In professional video production, it is important to adapt the sources of light to the physiological human perception. As human perception of color depends on the use of three different cones in the retina, sensitive to long, medium and short wavelengths of light, which can be represented by red, green and blue light, a suitable source of light for video projection systems should also include sources of red, green and blue light (RGB light). Modern video projection systems use light emitting diodes (LEDs), LCD beamers or direct laser illumination, e.g. via laser diodes. While LED projectors are typically used in low brightness applications, direct laser projectors offer benefits in brightness and image quality and are, therefore, used for profession applications. Even combined Systems with RGB laser and phosphor wheel have recently reached the market.

[0009] Laser projection systems have gained significant popularity due to their high brightness, enhanced color reproduction and longer operational life compared to traditional land-based projectors. In particular, two types of laser projecting systems are commonly used, namely laser projectors with phosphor wheels and RGB laser projectors.

[0010] Laser projectors using phosphor wheels typically utilize a blue laser that excites a segmented rotating phosphor wheel to produce a yellow beam which includes a broader range of colors, typically generating green and red light from the phosphor material while the blue component is often directly produced by the laser. This type of system offers a cost-effective solution with high brightness and good color reproduction, especially if a

[0011] M / 65056-PCT (Subframe Projection) Max von Braun

[0012] 2 color wheel is placed in cascade with the phosphor wheel in order to achieve acceptable colorimetry for use in video production systems. The color wheel spins mechanically which imposes a limitation on how quickly different primary colors can be switched, which also leads to a reduction in brightness due to spoke time, i.e. the transition time between colors in the color wheel, and visual artefacts due to limited color cycling rate. Typically, a relatively low cost blue laser is used to illuminate spatial light modulators (SLMs) in order to form the projected images. In certain devices, the three color components are separated spatially and directed to the individual SLM devices and then recombined and projected via a lens system. In other devices, the three (or more) color components are sequentially projected onto a single SLM device. Suitable SLMs include digital micromirror devices (DMDs), liquid crystal devices (LCD) and grating light valves (GLV). The blue laser can also be used to illuminate a phosphor material applied to a rotating phosphor wheel which will then emit a broader light spectrum having longer wavelengths than the blue light of the laser. The emitted light is filtered, for instance using dichroic mirrors to generate red and green light beams. Typically, a spinning segmented phosphor wheel and a spinning segmented color wheel is used in cascade to filter and manage light output. Such phosphor laser projectors face several problems, such as quenching, due to overheating of the phosphor material, red deficiency as phosphors often emit more orange than true red, the above-mentioned spoke time, i.e. the transition time between color segments of the spinning wheels, and color cycle limitation, i.e. a limited number of color cycles per video frame can lead to perceptible image changes reducing image quality.

[0013] In order to overcome these problems, RGB laser projectors are employed which use separate red, green and blue lasers allowing for a direct control of the color channels. Consequently, RGB laser projectors provide superior color accuracy and a wider color gamut, but are generally more expensive and complex than phosphor-based laser projecting systems. However, RGB laser projectors also exhibit certain drawbacks in practical use. When scenes projected by pure RGB laser projectors are recorded using consumer mobile phone cameras, the resulting video footage often shows a noticeable magenta hue, leading to unnatural color reproduction. This effect is particularly undesirable in professional environments where both direct viewers and mobile devices are used for capturing content, such as in entertainment installations or live productions.

[0014] M / 65056-PCT (Subframe Projection) Max von Braun

[0015] 3

[0016] In modern high-speed laser projectors, the image generation and color control are managed through a so-called bit-plane formatter. The bit-plane formatter decomposes each image frame into a sequence of binary sub-frames (bit planes) corresponding to the individual bits of the pixel intensity values. These bit planes are displayed sequentially on the spatial light modulator (SLM), typically a digital micromirror device (DMD), with illumination durations weighted according to the binary significance of each bit. This process enables precise temporal modulation of the light output, resulting in high dynamic range and accurate color reproduction.

[0017] However, even the latest flag ship laser projector generation which allows a 4K or UHD spatial resolution (3840 x 2160 pixel) at 60 Hz (actually in most cases 59.94 Hz) suffers from the drawback that at higher frame rates, the overall resolution is drastically reduced. For instance, at a frame rate of 479.52 Hz (8 x 59.94 Hz), only a HD resolution of 1920 x 1080 pixel is achievable. At this frame rate, 8 sub-frame slots could be presented during a standard 59.94 Hz frame rate time interval ST = 1 / 59.94 = 16.7 ms.

[0018] Other image displaying technologies using LED screens can achieve much higher image display frame rates. For instance, international patent application WO 2022 / 112579 A2 describes the use of LED screens made up of multiple LED panels in the context of a studio environment, an entertainment event or a sports event for displaying multiple time- multiplexed image streams on the LED panels. The multiple time-multiplexed image streams can, for instance, be different video streams where video frames of each video stream are shown alternately on the LED screens. When cameras capture a scenery including these LED panels and the cameras are synchronized to the presentation of the time-multiplexed image streams, videos can be created from these cameras, which show the scenery with only one selected image stream shown on the LED panels. As described in WO 2022 / 112579 A2, such LED systems can be used to show one main image stream for the direct viewers and certain additional video streams which are only visible to broadcast viewers receiving the final video obtained by the synchronized cameras.

[0019] Generally, in order to achieve usable multiple time-multiplexed image streams, especially using techniques for hiding additional video streams by showing sequences of images and inverse images, as also described in WO 2022 / 112579 A2, one should at least be able to present 12 sub-frame slots at approximately 60 Hz for the time-multiplexed image content

[0020] M / 65056-PCT (Subframe Projection) Max von Braun

[0021] 4 during each standard frame rate time interval. However, even high-end laser projectors are not capable of operating at, for instance, 12 x 59.94 Hz = 719.28 Hz, which corresponds to sub-frames having only a length of 1.4 ms due to limitations of the bit-plane formatter.

[0022] Volumetric 3D displays using rotating screens have been proposed to create photorealistic three-dimensional images through persistence of vision, but they are constrained by high bandwidth requirements for data processing and projection to achieve sufficient resolution and refresh rates.

[0023] For example, US 12,039,686 B2 discloses a method for dynamically displaying a three- dimensional image object in a volumetric display apparatus. The system involves rotating a display plate around an axis to form a volumetric display volume. A first 3D image is displayed based on coordinates in an initial system, while a second 3D image is generated by reorienting the object in response to sensed position or movement of an external real- world object within a sensing zone, using sensors and motors. The display adapts in realtime, correlating reorientation with external movements. The technology described requires high computational power and data bandwidth needs for real-time reorientation and high-voxel-density rendering to maintain photorealism, as rapid rotation and precise synchronization demand fast data transfer rates that exceed conventional projector capabilities, limiting resolution and frame rates for lifelike projections.

[0024] Similarly, US 2005 / 0062684 Al describes a method and apparatus for an interactive volumetric three-dimensional display using a rotating helical screen to generate large-scale 3D images with over one million voxels. A pulsed laser beam of a laser pointer is projected onto the rotating helical screen, synchronized to create voxels in 3D space. The system includes a computer, a spatial light modulator, optics, and the helical screen rotated at high speed. Again, photorealistic quality requires ultra-high frame and data rates for processing millions of voxels in real-time, as the SLM must handle rapid updates without resolution loss, a limitation in standard projectors that reduces image fidelity in large-scale or interactive scenarios.

[0025] These examples illustrate the persistent problem in prior art volumetric displays: the high bandwidth demands for data transmission, processing, and projection to support photorealistic 3D with sufficient voxel density, refresh rates, and interactivity, often resulting in trade-offs between resolution, speed, and realism. In addition, such systems

[0026] M / 65056-PCT (Subframe Projection) Max von Braun

[0027] 5 typically suffer from insufficient brightness, making them usable only in completely dark or heavily dimmed environments. This limitation severely restricts their applicability in open or partially illuminated settings, further emphasizing the need for projection systems capable of providing higher brightness levels while maintaining high spatial and temporal resolution.

[0028] BRIEF SUMMARY OF THE INVENTION

[0029] It is, therefore, an object of the present invention to provide a method and a system for projecting multiple temporally and / or spatially multiplexed image streams with one or more laser projectors which achieve a higher sub-frame rate while maintaining a high overall spatial resolution.

[0030] This object is achieved by the method and system of the appended claims.

[0031] Accordingly, the present invention concerns a method for projecting multiple temporally and / or spatially multiplexed image streams, comprising the steps of: a) selecting a standard projecting frame rate at a high spatial image resolution defining a standard projecting frame rate time interval; b) selecting a high projecting frame rate at a low spatial image resolution defining a high projection frame rate time interval, said high projecting frame rate being at least twice the standard projecting frame rate; c) subdividing each high projection frame rate time interval into an integer number of sub-frame slots, each sub-frame slot comprising sub-frames having a duration which is a fraction of said high projection frame rate time interval, said fraction being one over said number of sub-frame slots, wherein said sub-frames of each sub-frame slot are generated at said high projecting frame rate; d) time-shifting said sub-frames of each high projection frame rate time interval by an integer multiple of said sub-frame-duration in order to create an output stream of sub-frames having a output frame rate which corresponds to said high projecting frame rate multiplied by said integer number of sub-frame slots; e) assigning at least two image streams in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream; and

[0032] M / 65056-PCT (Subframe Projection) Max von Braun

[0033] 6 f) projecting said output stream onto an image area.

[0034] The frame rate is usually indicated as “frames per second” (fps) or as a recording frequency indicated in Hertz (Hz). For instance, in the USA, a typical standard frame rate for digital videos is usually 60 fps corresponding to a frequency of 60 Hz. In this example, one second of a video stream comprises 60 frames (images). Consequently, the available time interval for each frame, denoted frame time interval in the context of the present invention, is the inverse of the corresponding frame rate, i.e. 1 / CFR or 1 / VFR. In the example of a frame rate of 60 fps (60 Hz), the corresponding frame time interval is 16.7 ms. Typical standard frame rates include 12 fps, 24 fps, 25 fps, 30 fps, 50 fps, 60 fps etc. For applications involving stereoscopic 3D display, certain projectors are configured to operate at higher frame rates in order to generate separate image streams for each eye when used with 3D glasses. In such cases, typical frame rates range from 60 to 120 Hz per eye, resulting in an overall projector operation of 120 to 240 Hz to output both images within one standard frame interval.

[0035] The term “high spatial resolution” at the standard projection frame rate corresponds typically to UHD resolution or 4K resolution (3840 x 2160 pixels). In the near future, projectors are expected to reach 8K resolution which is also a suitable “high spatial resolution”.

[0036] The “high projecting frame rate” is typically at least twice, preferably at least 8 times the standard projection frame rate. For certain application directed to projection of temporally multiplexed image content, the “high projecting frame rate” is at least 8 times, preferably 12, 16, 24 or 32 times the standard projection frame rate. For other application, especially in the context of volumetric image generation, the “high projecting frame rate” has to be much higher, for instance 500 times, 1 000 times or even 2 000 times the standard projection frame rate.

[0037] The term “low spatial image resolution” means a spatial resolution that is lower than the “high spatial resolution”, for instance 2K (2048 x 1080 pixels). Full HD resolution (1920 x 1080 pixel) or lower.

[0038] The image streams can include any type of image data including videos, still images, any type of monochromatic images including black images. In the context of volumetric image

[0039] M / 65056-PCT (Subframe Projection) Max von Braun

[0040] 7 display, the image streams can include layers or partial layers of the volumetric image to be depicted. In this context, the term “layer” is to be construed broadly and may represent any two-dimensional configuration corresponding to the position of the diffuser screen at a given time.

[0041] The projected image area is usually a two-dimensional surface which can be planar or have any type of shape. However, the projected image area can also include non-surface areas such as fog, steam etc., especially for volumetric 3D image display. When two- dimensional surfaces are used as an image area for volumetric image generation, the two- dimensional surface is usually a semitransparent diffuser screen. The two-dimensional surface will have to be swept across the projection volume while slices corresponding to the respective positions and configurations of the diffuser screen are projected onto the screen. Sweeping the diffuser screen across the projection volume can, for instance, be accomplished by reciprocal movement of the diffuser screen or by a rotational movement of the diffuser screen. The diffuser screen can, for instance, be planar rotating screen or helical rotating screen.

[0042] Step e), i.e. assigning the at least two image streams in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream is typically accomplished by operating the SLM of a projector, for instance a DMD, accordingly.

[0043] In one embodiment, the method of the present invention is implemented in a laser projector system comprising at least one laser projector. Typical laser projectors include RGB Laser projectors or Laser Phosphor projectors. By now also such combinations are reaching the market.

[0044] In one embodiment of step c), subdividing said high projection frame rate time interval is carried out in a control unit of said laser projector system.

[0045] The control unit can include a processing unit integrated in the projector directly or a sparate processing unit. In practical implementations, the dedicated multiplexer control unit corresponds to the so-called formatter. The output of the video processing stage is provided as input to the formatter block, which converts the image data into DMD control data. The formatter block comprises a sequencer that decomposes each image frame into corresponding DMD mirror positions for display. Output data from the formatter block are

[0046] M / 65056-PCT (Subframe Projection) Max von Braun

[0047] 8 transmitted to the DMD via its high-speed and low-speed data ports, while additional synchronization signals are generated to control the illumination timing in coordination with DMD events throughout each video frame. Such formatter logic can also be adapted or extended to implement the temporal or spatial subdivision and sub-frame multiplexing described in the present invention. A suitable play out system can be implemented in hardware and software.

[0048] In one embodiment, step e) comprises reducing image data of said at least two image streams assigned to said sub-frames by image analysis. In particular, in data-intensive volumetric projection modes, a significant reduction of data bandwidth can be achieved by limiting the displayed content to the contours or outlines of the projected objects rather than rendering full volumetric frames. Since, in such volumetric operation, typically only a small fraction of the available pixel array is actively used at any given time — often less than 20% of the total pixel capacity — the effective data rate requirement is considerably reduced. Consequently, for example, a 4K projection operating at 120 Hz may effectively correspond to a data load equivalent to approximately 600 Hz for contour-only projection, thereby allowing substantially higher temporal resolution within the same system bandwidth.

[0049] Single projector performance can be optimized to achieve lower lifted black effect (greyed out experience by the naked eye of direct viewers). It is also possible to employ a “per pixel / pixel area” detection of activated feature in real-time combining camera captures and media playout systems. For instance, projected image areas which are not captured by a camera can be played-out using less data. In other embodiments all “pixels / pixel areas” that only require emitting one image can stay in single picture mode. In addition or alternatively, all “pixels / pixel areas” with RGB values of 0 / 0 / 0 (K) content stay turned off Adjusting black / brightness performance can be effected by content analysis, for instance during preproduction and live operation. In addition, overall brightness settings can be adjusted in real-time by analyzing the content before emitting the light from the projector. Darker scenes then can run at lower brightness (30% brightness is a typical minimum setting in professional projectors).

[0050] In another embodiment, the projection system may be operated in a single-color or monochromatic mode in order to achieve extremely high frame rates, for example for

[0051] M / 65056-PCT (Subframe Projection) Max von Braun

[0052] 9 holographic or hologram-like image generation. By limiting the projection to one wavelength, e.g. blue or green laser light, the system can significantly reduce data volume and processing complexity, allowing higher temporal resolution and increased sub-frame rates. Such single-color operation is particularly advantageous for holographic display modes, where spatial phase or contour information is of greater relevance than full color reproduction.

[0053] In one embodiment, said laser projector systems include a hybrid laser projector comprising an RGB laser source and a Laser-Phosphor laser source.

[0054] In one embodiment, said laser projector system includes a Laser-Phosphor projector comprising at least two phosphor wheels. Other embodiments include two color wheels.

[0055] In one embodiment of step c), subdividing each high projection frame rate time interval into at an integer number of sub-frame slots is accomplished by using, for each projected image area, a separate laser projector for each sub-frame slot for each image area.

[0056] When separate laser projectors with adjacent or overlapping image projection areas are used, it is possible to operate certain projectors or each projector with reduced or partial DMD data load.

[0057] In one embodiment, said laser projectors for each sub-frame slot comprise at least one RGB laser projector and at least one Laser-Phosphor projector.

[0058] In one embodiment of step e), said at least two image streams are assigned in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream in a manner to optimize color gamut of said output stream.

[0059] In one embodiment of step e), said at least two image streams are assigned in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream in a manner to increase spatial resolution of said projected image area.

[0060] In one embodiment of step e), said at least two image streams are assigned in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream in a manner to increase brightness headroom said projected image area.

[0061] M / 65056-PCT (Subframe Projection) Max von Braun

[0062] 10

[0063] In one embodiment of step e) said at least two image streams include alignment patterns for projector alignment or tracking patterns for cameras which are assigned to sub-frames of said output stream, enabling for instance alignment of multiple image layers of multiple projectors.

[0064] In one embodiment of step e) said at least two image streams include chroma key images.

[0065] In one embodiment of step e) said at least two image streams include images and corresponding inverse images.

[0066] In one embodiment, said laser projector system includes a first set of laser projectors for a first projected image area and a second set of lasers projectors for a second projected image area partially overlapping with said first projected image area, wherein each of said sub-frame slots comprises sub-frames with calibration data for aligning and adapting said first and second projected image areas.

[0067] In another embodiment of step e), assigning at least two image streams in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream comprises subdividing each image frame area within a single frame into two or more spatially distinct area portions. Each of these area portions is assigned to a respective image stream, wherein each image stream represents a layer or partial layer of a volumetric image area. In this embodiment, the sub-frames within the output stream contain different layer information projected simultaneously onto different area portions of the same image frame area. This spatial subdivision enables parallel projection of multiple depth layers within a single frame, allowing reconstruction of volumetric or quasi- volumetric image content when viewed through a suitable optical arrangement or a moving or diffusive projection medium. For instance, the image frame area may be divided into concentric zones, grid-like segments, or irregular spatial portions, each corresponding to a distinct depth level or cross-section of the volumetric image. The respective image streams assigned to these spatial portions can represent slices or voxels of the three-dimensional image, and are combined optically or perceptually by the viewer or by interaction with a rotating or reciprocating diffuser screen to form a continuous volumetric image. Such spatial multiplexing of layer information allows the available projection bandwidth to be utilized efficiently, as multiple layers of a volumetric scene can be represented in a single subframe. The approach can be implemented by a suitable control logic in the projector or in a

[0068] M / 65056-PCT (Subframe Projection) Max von Braun

[0069] 11 dedicated multiplexer unit, such as a bit-plane formatter, that partitions the image frame data into spatially distinct layer regions prior to projection.

[0070] In a further embodiment, the image area comprises at least one movable diffuser screen. The term “movable” in this context denotes any type of mechanical movement that allows the diffuser screen to sweep across a projection volume or change its spatial position relative to the projector output. The movement can, for example, be a reciprocating motion along a linear or curved path, or a rotational motion about an axis, such as in a planar rotating or helical rotating diffuser screen. By synchronizing the motion of the diffuser screen with the temporally and / or spatially multiplexed projection of the image streams, the successive or spatially offset layers of the volumetric image can be reconstructed within a three-dimensional image volume.

[0071] In another embodiment, all components of the projection setup, including the play out controller, the projector system, and any movable or rotating holographic or diffuser screen, are synchronized with ultra-low latency to ensure precise temporal alignment of all sub-frame and illumination events. Such high-speed synchronization between content generation, light modulation, and physical screen movement constitutes an essential aspect of the present invention. The implementation of this ultra-synchronization enables stable holographic and volumetric image formation at sub-millisecond accuracy, which is not achievable with conventional projection control systems.

[0072] In specific embodiments of the invention, the movable diffuser screen of step f) may take different structural forms adapted to the desired volumetric image generation.In one embodiment, the diffuser screen is a planar rotating diffuser screen. In this configuration, a substantially flat, semitransparent diffuser plate is rotated about an axis — typically a central or offset vertical or horizontal axis — such that successive projections onto the rotating surface form spatially displaced slices of the volumetric image. The persistence of vision of the viewer or the integration of camera capture across multiple rotations results in a perceived three-dimensional representation of the projected content. In this embodiment, the system is capable of displaying both hologram-like volumetric contours and closed- surface three-dimensional models within the projection volume. Owing to the fact that in certain orientations, the diffuser screen does not obstruct the background, the background behind the projection remains at least partially visible to direct viewers and cameras. This

[0073] M / 65056-PCT (Subframe Projection) Max von Braun

[0074] 12 feature allows realistic integration of volumetric or holographic content into real-world environments, enabling applications where both virtual 3D objects with closed surfaces and the real background are simultaneously perceptible — a visual effect that cannot be achieved with conventional opaque or emissive display technologies.

[0075] In another embodiment, the diffuser screen consists of two crossed rotating diffuser plates, each arranged at an angle — preferably perpendicular — to one another. The use of two crossed diffuser plates increases the effective voxel density of the reconstructed volumetric image and reduces directional artifacts, as light from the projectors is diffused along two intersecting planes. This configuration allows generation of volumetric images with enhanced depth uniformity and improved viewing angles. Owing to the fact that in this embodiment parts of the crossed diffuser screens always obstruct the background, the background behind the projection remains generally less visible or even not visible to direct viewers and cameras.

[0076] In a further embodiment, the diffuser screen is implemented as a rotating helical diffuser screen. The helical configuration enables continuous sweeping of the projection surface through the image volume during rotation, thereby allowing dense spatial coverage and smooth transitions between successive volumetric layers. The helical diffuser may be realized as a translucent film, a mesh, or a rigid light-diffusing structure formed into a helix, rotated at a speed synchronized with the sub-frame projection rate to reconstruct a three-dimensional image volume.

[0077] In embodiments where the rotation axis of the planar or helical diffuser screen is arranged vertically, the projection in step f) is preferably accomplished from below the rotating diffuser screen. In this configuration, the projection light is directed upward into the diffuser volume using a suitable optical path arrangement, which may include projection lenses, beam shaping optics, and one or more co-rotating or tilting mirrors for dynamic beam deflection. Such an optical configuration allows the projector to maintain a compact footprint beneath the rotating diffuser structure while ensuring that the projected image layers intersect the diffuser surface at the correct spatial positions throughout its rotation. The use of rotating or synchronized mirror systems further enables precise tracking of the diffuser surface during its movement, thereby maintaining consistent focus and layer alignment within the volumetric image volume.

[0078] M / 65056-PCT (Subframe Projection) Max von Braun

[0079] 13

[0080] The present invention also concerns a system for projecting multiple temporally and / or spatially multiplexed image streams comprising a laser projector system and a control unit configured to carry out the above described method, preferably at least one SLM-based laser projector, a controller configured to subdivide high-rate intervals into multiple slots, apply a time-shift per slot and map at least image streams to sub-frames, and output to the projection optics. For volumetric projections, the system may further comprise at least one movable diffuser screen.

[0081] BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The invention will now be described in more detail in connection with the attached drawings.

[0083] In the drawings:

[0084] Fig. 1 is a schematic representation of a scenery including a screen and projectors and direct viewers capturing the scenery with cameras;

[0085] Fig. 2 is a schematic representation of an RGB laser projector;

[0086] Fig. 3 is a schematic representation of a laser-phosphor projector;

[0087] Fig. 4 is an embodiment of the present invention where the performance of an individual projector is optimized via an adapted playout software;

[0088] Fig. 5 is an embodiment of the present invention where projector performance is improved by using dual phosphor wheels;

[0089] Fig. 6 is an embodiment of the present invention using multiple projectors to achieve additional time slots and / or a higher spatial resolution and / or an increased color gamut;

[0090] Fig. 7 is an embodiment of the present invention using multiple projectors to increase maximum brightness;

[0091] Fig. 8 is an embodiment of the present invention using multiple projectors to optimize image overlay;

[0092] M / 65056-PCT (Subframe Projection) Max von Braun

[0093] 14

[0094] Fig. 9 is a schematic representation of an embodiment of the present invention used for volumetric hologram-like 3D image projection;

[0095] Fig. 10 is a first variant of the projection scheme of Fig. 9 using a single planar rotating diffuser screen;

[0096] Fig. 11 is a second variant of the projection scheme of Fig. 9 using a two crossed planar diffuser screens; and

[0097] Fig. 12 is a third variant of the projection scheme of Fig. 9 using a helical rotating diffuser screen. DET AILED DESCRIPTION OF THE INVENTION

[0098] Fig. 1 shows a scenery comprising a screen 10, which is illuminated by multiple laser projectors two of which are schematically indicated by reference signs 11 and 12, respectively. Typically, each laser projector illuminates a certain portion of the stream. Typically, there is an overlap of portions illuminated by different projectors. A controller 13 controls the playout of the image screens to the individual projectors 11, 12 ensuring that a seamless overlap is provided and that the intended overall image is produced on the screen. Depending on application and circumstances, various direct viewers 14, 15 are present at or near the scenery viewing the screen directly with their naked eyes and / or taking pictures or videos of the scenery including the illuminated screen 10. As an example, direct viewer 14 uses the camera of a handheld device 16, such as a mobile phone, while direct viewer 15 uses a professional broadcasting camera 17 to generate a professional video of the scenery including screen 10.

[0099] Fig. 2 and 3 show two typical types of laser projectors used in the art. Fig 2 shows a schematic overview of a RGB laser projector 20 comprising individual laser light sources 21, 22, 23 for red, green and blue laser light, respectively. Each laser light source is provided with its light combiner 21a, 22a and 23a coupling the laser light into individual optical fibers 21b, 22b and 23b, respectively. Via a despeckler 24, the light of the individual optical fibers is transmitted into a light pipe 25 and the resulting light beam 26 is extended via a lens 27 onto a prism 28, which projects the light onto a digital mirror device (DMD) 29, which includes an array 29a of adjustable micromirrors which can reflect the light from prism 28 into a projection lens 30 or away from the projection lens and a bitplane formatter 29b which transforms the respective image data into control signals for

[0100] M / 65056-PCT (Subframe Projection) Max von Braun

[0101] 15 micromirror array 29a. The image is generated by suitably adjusting the individual micromirrors in order to generate the image. Fig. 3 shows a laser phosphor projector 40, which uses one or more blue laser light sources 41 which generate a blue laser light beam 42, which is transmitted onto a phosphor wheel 43, which generates a broad light spectrum containing the original blue laser light and longer wavelengths such as green light and red light. A suitable optic 44, which includes filters to generate red, green and blue light components, transmits the light onto a digital mirror device 45, which operates similarly as the digital mirror device 29 of Fig. 2, i.e. it also includes an array of micromirrors 45a, which can reflect the light from optics 44 onto a projection lens 46 and a bit-plane formatter 45b.

[0102] In the embodiment shown in Fig. 4, the problem of achieving high sub-frame rates is solved by optimizing the performance of a single projector in its current commercially available configuration via an optimization of the control hardware and / or software. Conventionally, the projector 50 of Fig. 4 will output a single frame for each standard frame rate time interval, for instance when operating at 60 Hz (which is actually 59.94 Hz), each frame has a duration of 16.7 ms (corresponding to step a) of the method of the present invention). According to step b) of the present invention, a high projecting frame rate at a low spatial image resolution defining a high projection frame rate time interval is selected. As shown in the present example. With current technology, it is feasible to operate the projector at eight times the standard frame rate of 59.94 Hz, i.e. at 479.52 Hz, while maintaining at least HD resolution, i.e. 1920 x 1080 pixel. Rather than outputting a single frame 51 at 59.94 Hz, the present invention suggests modifying the playout software in order to create additional slicing via time slicing of individual sections, i.e an incremental playout system. In the example shown in Fig. 4, a eight times time slicing is produced to produce high projection frame rate time interval 52.1-52.8, each having a duration of 2.1 ms, i.e. operating at 479.52 Hz. This scheme is applied to each sub-frame slot, i.e. a first sub-frame slot denoted by reference sign 53 leading to sub-frame 53.1-53.8, a second subframe slot, denoted by reference sign 54 (sub-frames 54.1-54-8), and a third sub-frame slot, denoted by reference sign 55 (sub-frames 55.1-55.8). Each sub-frame is presented for only a third of the high projection frame rate time interval of 2.1 ms, i.e. for 0.7 ms, and the second and third sub-frame slots are shifted by 0.7 ms and 1.4 ms with respect to the first sub-frame slot, respectively. In summary, 3 x 8 = 24 image slots are created with the

[0103] M / 65056-PCT (Subframe Projection) Max von Braun

[0104] 16 suitably adapted playout software as shown by the combined output stream 56. The formatter transmits the required data is sent to the digital mirror device of the projector, individual image content can be applied to each of the 3 x 8 image slots of playout line 56.

[0105] Fig. 5 shows an embodiment of the present invention where a single projector is modified on a hardware basis to increase projector performance. In this embodiment, a laser phosphor projector is used but, contrary to the conventional projector shown in Fig. 3, the projector of Fig. 5 is equipped with two phosphor wheels 43a, 43b. Otherwise, the same reference signs as in the embodiment of Fig. 3 are used to denote similar elements of the phosphor laser projector. Using two phosphor wheels, two different colors can be created at the same time with the option to overlay positive and / or negative colors using appropriately segmented phosphor wheels. For simplicity, the two phosphor wheels 43a, 43b are shown in cascade in Fig. 5, i.e. in a serial arrangement. They can, however, be used in parallel as well, for instance by adapting the beam splitters 48 accordingly when using multiple blue laser light sources. When only one laser light source is employed, a beam splitter can be employed to generate two beams out of one initial beam.

[0106] In another embodiment, the conventional RGB laser projector of Fig. 2 and the conventional phosphor laser projector of Fig. 3 can be combined to employ a hybrid RGB / phosphor laser projector variably combining the benefits of both approaches. This approach has the advantage of improving the overall color gamut of the projection, especially with respect to applications where the scenery is typically captured by mobile phone cameras which suffer from the drawback that, when using phosphor laser projectors only, the captured images typically experience a magenta hue.

[0107] Fig. 6 shows an embodiment of the present invention where multiple projectors, in the present example three projectors 61, 62, 63, are used to overlay a first image 61a, a second image 62a and a third image 63a, simultaneously. Again, each projector is operated at eight times the standard frame rate, for instance eight times 59.94 Hz, i.e. at 479.52 Hz. While stepping up the output rate of the projector, the available spatial resolution decreases from 4K / UHD down to HD resolution. Using multiple projectors, the incremental slots, which again are presented only for one third of each sub-frame time interval, can be combined to create 3 x 8 image slots per 16.68 ms standard frame rate time interval. Contrary to the embodiment of Fig. 4, no modification of the playout software is required in this example.

[0108] M / 65056-PCT (Subframe Projection) Max von Braun

[0109] 17

[0110] In one embodiment, each of the 3 x 8 image slots 61b, 62b, 63b can be provided with different image content from different image streams in order to generate the time- multiplexed image stream 64. Accordingly, a high variability of image content is achieved at HD spatial resolution.

[0111] In another embodiment, the scheme of Fig. 6 can be employed to increase spatial image resolution by subdividing an initial UHD / 4K image onto different sub-frame slots, respectively. Overlaying three images from three different projectors simultaneously will then result in a higher resolution than otherwise obtainable using an 8 x sub-frame increase.

[0112] Finally, the multiple projector scheme of Fig. 6 can also be employed to increase the overall color gamut, for instance by using different types of laser projectors. For instance, in the example of Fig. 6, two RGB projectors 61, 62 and one phosphor projector 63 can be employed or two phosphor projectors 61, 62 and one RGB projector 63. The corresponding 3 x 8 image slot 61b, 62b, 63b stream is a mixture of sub-frame slots created by RGB projectors and phosphor projectors, respectively.

[0113] The embodiment of Fig. 7 shows a multiple projector approach to increase brightness headroom. In the example of Fig. 7, three projectors 71, 72, 73 are used to overlay three images simultaneously to create an overlayed image 74. Each projector can operate at a maximum of 100 percent design brightness for image frames 71.1.-71.8, 72.1-72.8 and 73.1-73.8, respectively, but the overlayed image has up to three times the brightness of each individual projector for overlayed image frames 74.1-74.8, depending on image parameters usually between 200 and 300%. Such an approach can be used to ensure that each sub-frame slot is bright enough to be captured by an appropriately synchronized camera.

[0114] In the embodiment of Fig. 8, multiple projectors 81, 82 are used to overlay multiple images 81a, 82a, 83 simultaneously. An overlay optimization for multiple output signals is provided, for instance by calculating overlapping areas for more than one image simultaneously. The laser projector system includes a first set of laser projectors for a first projected image area and a second set of lasers projectors for a second projected image area partially overlapping with said first projected image area, wherein each of said sub

[0115] M / 65056-PCT (Subframe Projection) Max von Braun

[0116] 18 frame slots comprises sub-frames with calibration data for aligning and adapting said first and second projected image areas.

[0117] Figs. 9 to 12 illustrate exemplary embodiments of the invention applied to volumetric or hologram-like three-dimensional image projection systems, in which the projection method described in steps e) and f) is implemented using one or more movable diffuser screens. In these embodiments, the volumetric image area is generated by sequentially or simultaneously projecting a multiplicity of sub-frames corresponding to different image layers or partial layers into a three-dimensional space. The volumetric image is thereby formed through the spatial and / or temporal multiplexing of the projected sub-frames in combination with the motion of one or more diffuser screens that sweep through the projection volume. The rotation or reciprocation of the diffuser screen enables reconstruction of a continuous volumetric image from layer data generated in accordance with the subdivision of the image frame area as described above in connection with claim 16.

[0118] Fig. 9 depicts a schematic representation of an embodiment of the present invention used for volumetric, hologram-like 3D image projection. The projection system 100 comprises a volumetric image area or holographic projection volume 101, having, for example, a diameter of approximately 1 m and a height of about 2 m, thereby allowing realistic lifesize representations of persons 102, such as members of a sports team or a musical band. The projection system 100 further comprises a projector 103, an ultrashort-throw projection lens 104, and an optical package 105 for directing and shaping the projection beams. A play out controller 106 stores and / or computes the image data to be projected into the volumetric image area 101. A master controller 107 communicates with the play out controller 106, the projector 103, and the optical package 105 to synchronize projection timing, sub-frame assignment, and diffuser screen motion.

[0119] Fig. 10 depicts a first variant of the projection scheme of Fig. 9, employing a single planar rotating diffuser screen. In this embodiment, the optical package 105 includes the stationary ultrashort-throw projection lens 104 of the projector 102 (not shown in detail in this figure). The volumetric image area 101 is enclosed by a stationary transparent cylindrical tube 108. Inside the tube 108, a planar diffuser plate 109 is arranged vertically and rotates about a vertical axis A.

[0120] M / 65056-PCT (Subframe Projection) Max von Braun

[0121] 19

[0122] In this embodiment, the image frame area of the projection system is spatially subdivided into two distinct area portions corresponding to two partial layers of the volumetric image area 101. The optical package 105 further comprises a first set of deflection mirrors 110, 111 co-rotating with the diffuser plate 108, and a second set of deflection mirrors 112, 113, likewise co-rotating with the first set and the diffuser plate 109, thereby enabling simultaneous projection onto the front side and the backside of the rotating diffuser plate 108. This allows for volumetric objects with closed surfaces and therefore realistic copies of e.g. human beings while also keeping the background available to make the overall setting seem so realistic a difference between the volumetric object and the real world object look identical.

[0123] To estimate the required sub-frame frequency, it is assumed that the circumferential resolution across a volumetric image area of 1 m diameter should be 1.6 mm or better. Accordingly, approximately 2 000 discrete circumferential layers are required to provide a layer spacing of 1.6 mm at the outer circumference. The resolution increases with decreasing distance from the rotational axis. With a standard frame rate of 30 fps (30 Hz) corresponding to a frame time interval of 33.33 ms, and since front and back projections are performed simultaneously, 1 000 different layers / images must be displayed within one rotation. This results in a required sub-frame projection rate of approximately 30 kHz. In this configuration, the lateral resolution of each layer is approximately half the maximum lateral resolution of the projector system due to the two-portion subdivision of the frame area.

[0124] Fig. 11 depicts a second variant of the projection scheme of Fig. 9, using two crossed planar rotating diffuser screens. In this embodiment, the optical package 105 comprises a first set of four co-rotating mirrors 122-125 and a second set of four co-rotating mirrors 126-129, enabling simultaneous projection onto four spatially distinct area portions corresponding to four partial layers or cross-sections of the volumetric image area 101. Between the ultrashort-throw lens 103 and the first mirror set 122-125, a Fresnel lens 130 is provided to collimate (parallelize) the divergent light beams emitted by the ultrashort- throough lens.

[0125] Since four projections are generated simultaneously, only 500 different layers or angular positions must be addressed within one rotation, resulting in a sub-frame projection

[0126] M / 65056-PCT (Subframe Projection) Max von Braun

[0127] 20 frequency of approximately 15 kHz. While this reduces the required sub-frame rate, it also decreases the spatial image resolution, since four images are displayed concurrently. For example, in the case of a 4K projector system, each area portion provides an effective resolution equivalent to Full HD (1920 x 1080 pixels). However, in volumetric or holographic operation modes, only a fraction of the total pixel array is typically required, as the projected content primarily represents contours or outlines of objects rather than full-surface imagery. This contour-based rendering substantially reduces the data load and allows a greater effective frame rate or increased number of volumetric layers to be displayed within the same projection bandwidth.

[0128] Fig. 12 depicts a third variant of the projection scheme of Fig. 9, using a rotating helical diffuser screen 131. Similar to the configuration of Fig. 11, a Fresnel lens 130 is used to collimate the divergent beams from the projection lens 103 before they enter the helical diffuser structure. In this embodiment, the entire image content is projected continuously onto the rotating helical diffuser screen 131, which sweeps through the volumetric image area 101 as it rotates about the vertical axis A. The helical surface thereby traverses all depth positions of the volumetric volume during a single revolution, enabling the reconstruction of a continuous 3D image volume from temporally multiplexed layer information.

[0129] Because no subdivision of the image frame area is applied in this case, the full set of 2 000 layers must be projected within each rotation, resulting in a required sub-frame projection rate of approximately 60 kHz. The use of the Fresnel lens 130 ensures that the projection remains sharply focused on the moving helical diffuser surface throughout rotation.

[0130] M / 65056-PCT (Subframe Projection) Max von Braun

[0131] List of reference signs

[0132] 10 screen (projected image area)

[0133] 11, 12 laser projectors illuminating the screen

[0134] 13 controller (control unit for projector playout)

[0135] 14, 15 direct viewers

[0136] 16 handheld device (e.g. mobile phone camera)

[0137] 17 professional broadcasting camera

[0138] 20 rgb laser projector

[0139] 21, 22, 23 laser light sources for red, green, and blue light, respectively

[0140] 21a, 22a, 23a light combiners

[0141] 21b, 22b, 23b optical fibers

[0142] 24 despeckler

[0143] 25 light pipe

[0144] 26 resulting light beam

[0145] 27 lens)

[0146] 28 prism

[0147] 29 digital micromirror device (DMD)

[0148] 29a array of micromirrors within DMD 29

[0149] 29b bit-plane formatter (DMD control logic)

[0150] 30 projection lens

[0151] 40 laser-phosphor projector

[0152] 41 blue laser light source

[0153] 42 blue laser beam

[0154] 43 phosphor wheel

[0155] 43a, 43b dual phosphor wheels in modified projector

[0156] 44 optic / filter system

[0157] 45 DMD of phosphor projector

[0158] 45a array of micromirrors within DMD 45

[0159] 45b bit-plane formatter of phosphor projector

[0160] 46 projection lens of phosphor projector

[0161] 48 beam splitter

[0162] M / 65056-PCT (Subframe Projection) Max von Braun

[0163] 22

[0164] 50 projector in software-optimized embodiment

[0165] 51 single frame at standard frame rate

[0166] 52.1 - 52.8 sub-frame time intervals at high frame rate

[0167] 53, 54, 55 sub-frame slots

[0168] 53.1 - 55.8 sub-frames within each sub-frame slot 56 combined output stream of sub-frames

[0169] 61, 62, 63 multiple projectors used in combined / overlay embodiments 61a, 62a, 63a images projected by projectors 61-63 64 time-multiplexed image stream

[0170] 71, 72, 73 multiple projectors used to increase brightness headroom 74 overlay ed image (resulting combined projection)

[0171] 81, 82, 83 projectors in overlay optimization embodiment 81a, 82a, 83a overlay ed images from projectors 81-83 100 volumetric projection system

[0172] 101 volumetric image area / holographic projection volume

[0173] 102 projected person or object

[0174] 103 projector for volumetric projection

[0175] 104 ultrashort-throw projection lens

[0176] 105 optical package for directing and shaping beams

[0177] 106 playout controller

[0178] 107 master controller

[0179] 108 stationary transparent cylindrical tube

[0180] 109 planar rotating diffuser plate

[0181] 110, 111 first set of co-rotating mirrors for single planar diffuser 112, 113 second set of co-rotating mirrors for single planar diffuser 122 - 125 first set of co-rotating mirrors for crossed diffusers 126 - 129 second set of co-rotating mirrors for crossed diffusesr 130 Fresnel lenses (collimating optics) 131 rotating helical diffuser screen A vertical rotation axis of diffuser screen

[0182] M / 65056-PCT (Subframe Projection)

Claims

Max von Braun23Claims1. A method for projecting multiple temporally and / or spatially multiplexed image streams, comprising the steps of: a) selecting a standard projecting frame rate at a high spatial image resolution defining a standard projecting frame rate time interval, b) selecting a high projecting frame rate at a low spatial image resolution defining a high projection frame rate time interval, said high projecting frame rate being at least twice the standard projecting frame rate; c) subdividing each high projection frame rate time interval into an integer number of sub-frame slots, each sub-frame slot comprising sub-frames having a duration which is a fraction of said high projection frame rate time interval, said fraction being one over said number of sub-frame slots, wherein said sub-frames of each sub-frame slot are generated at said high projecting frame rate; d) time-shifting said sub-frames of each high projection frame rate time interval by an integer multiple of said sub-frame-duration in order to create an output stream of sub-frames having an output frame rate which corresponds to said high projecting frame rate multiplied by said integer number of sub-frame slots; e) assigning at least two image streams in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream; and f) projecting said output stream onto an image area.

2. The method according to claim 1, wherein said method is implemented in a laser projector system comprising at least one laser projector.

3. The method according to claim 2, wherein in step c) subdividing said high projection frame rate time interval is carried out in a control unit of said laser projector system.M / 65056-PCT (Subframe Projection)Max von Braun244. The method according to any one of claims 1 to 3, wherein step e) comprises reducing image data of said at least two image streams assigned to said sub-frames by image analysis.

5. The method according to any one of claims 2 to 4, wherein said laser projector system includes a hybrid laser projector comprising an RGB laser source and a Laser-Phosphor laser source.

6. The method according to any one of claims 2 to 5, wherein said laser projector system includes a Laser-Phosphor projector comprising at least two phosphor wheels.

7. The method according to any one of claims 2 to 6, wherein in step c) subdividing each high projection frame rate time interval into at an integer number of sub-frame slots is accomplished by using, for each projected image area, a separate laser projector for each sub-frame slot for each image area.

8. The method according to claims 7, wherein said laser projectors for each sub-frame slot comprise at least one RGB laser projector and at least one Laser-Phosphor laser projector.

9. The method according to any one of claims 7 or 8, wherein in step e), said at least two image streams are assigned in a time-multiplexed manner to said sub-frames of said output stream in a manner to optimize color gamut of said output stream.

10. The method according to any one of claims 7 to 9, wherein in step e), said at least two image streams are assigned in a time-multiplexed manner to said sub-frames of said output stream in a manner to increase spatial resolution of said projected image area.

11. The method according to any one of claims 7 to 10, wherein in step e), said at least two image streams are assigned in a time-multiplexed manner to said sub-frames of said output stream in a manner to increase brightness headroom of said projected image area.M / 65056-PCT (Subframe Projection)Max von Braun2512. The method according to any one of claims 2 to 11, wherein in step e) said at least two image streams include alignment patterns for projector alignment or tracking patterns for cameras which are assigned to sub-frames of said output stream.

13. The method according to any one of claims 2 to 12, wherein in step e) said at least two image streams include chromakey images.

14. The method according to any one of claims 1 to 13, wherein said in step e) said at least two image streams include images and corresponding inverse images.

15. The method according to any one of claims 2 to 14, wherein said laser projector system includes a first set of laser projectors for a first projected image area and a second set of lasers projectors for a second projected image area partially overlapping with said first projected image area, wherein each of said sub-frame slots comprises sub-frames with calibration data for aligning and adapting said first and second projected image areas.

16. The method according to any one of claims 1 to 15, wherein step e) of assigning at least two image streams in a temporally and / or spatially multiplexed manner to said sub-frames of said output stream comprises subdividing each image frame area into two or more spatially distinct area portions, each area portion being assigned to one of said image streams and corresponding to a layer or partial layer of a volumetric image area.

17. The method of claim 16, wherein in step f) said image area comprises at least one movable diffuser screen.

18. The method of claim 17, wherein said diffuser screen is a planar rotating diffuser screen.

19. The method of claim 17, wherein said diffuser screen consists of two crossed rotating diffuser plates.

20. The method of claim 17, wherein said diffuser screen is a rotating helical diffuser screen.M / 65056-PCT (Subframe Projection)Max von Braun2621. System for projecting multiple temporally and / or spatially multiplexed image streams comprising a laser projector system and a control unit configured to carry out the method of any one of claims 1 to 20.M / 65056-PCT (Subframe Projection)

Images