Holographic display of a live scene including three-dimensional objects

Abstract

Methods, apparatuses, devices, subsystems, and systems are provided for holographically displaying a live scene containing one or more three-dimensional (3D) objects. In one embodiment, the method includes capturing an optical hologram of the live scene, digitizing / processing the optical hologram, and holographically reconstructing the live scene based on the digitized / processed hologram. In another embodiment, the method includes capturing an image / video of the live scene, calculating a corresponding hologram, and holographically reconstructing the live scene based on the calculated hologram.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Patent Application No. 18 / 410,185, filed on January 11, 2024; U.S. Provisional Patent Application No. 63 / 613,622, filed on December 21, 2023; U.S. Patent Application No. 18 / 468,571, filed on September 15, 2023; and U.S. Provisional Patent Application No. 63 / 501,928, filed on May 12, 2023, the entire contents of which are incorporated herein by reference.

[0002] This disclosure relates to displaying three - dimensional (3D) objects, and more particularly, to holographically displaying a live scene that includes 3D objects.

Background Art

[0003] Advances in conventional two - dimensional (2D) projection and 3D rendering have led to new approaches for 3D displays that incorporate head and eye tracking into conventional display devices for virtual reality (VR), augmented reality (AR), and mixed reality (MR). These techniques attempt to recreate the experience of holographic images by combining tracking and measurement - based calculations to simulate stereo or in - eye light fields that can be represented by actual holograms. Holograms that provide a 3D view of an object offer levels of detail not achievable with ordinary two - dimensional (2D) images. Due to the ability to provide a realistic and immersive experience of 3D objects, holograms hold great potential for use in various fields including medical imaging, manufacturing, and virtual reality.

Summary of the Invention

[0004] This disclosure describes apparatus, devices, subsystems, and systems relating to holographic display of a live scene including one or more three-dimensional (3D) objects by, for example, i) capturing an optical hologram of a live scene, digitizing / processing the optical hologram, and holographically reconstructing the live scene based on the digitized / processed hologram, and / or ii) capturing an image / video of the live scene, calculating a corresponding hologram, and holographically reconstructing the live scene based on the calculated hologram.

[0005] One aspect of the present disclosure features a system comprising a holographic capture system and a holographic display system. The holographic capture system includes an optical system configured to generate an optical hologram of a live scene including one or more three-dimensional (3D) objects, and an optical sensor configured to capture sequential optical holograms of the live scene and output sequential hologram data associated with the sequential optical holograms, wherein each optical hologram is associated with individual hologram data. The holographic display system is configured to optically reconstruct the live scene in 3D space based on at least a portion of the sequential hologram data.

[0006] In some implementations, the system further includes a computing device coupled between the holographic capture system and the holographic display system. The computing device is configured to receive at least a portion of the sequential holographic data from the optical sensor and to generate a digital hologram associated with the live scene based on at least a portion of the sequential holographic data. The holographic display system is configured to receive the digital hologram associated with the live scene from the computing device and to reconstruct the live scene in 3D space based on the digital hologram.

[0007] In some implementations, a holographic capture system is configured to capture a sequential optical hologram and generate sequential hologram data without storing the sequential optical hologram and the sequential hologram data. A computing device is configured to process at least a portion of the sequential hologram data to generate a digital hologram without storing at least a portion of the sequential hologram data and the digital hologram.

[0008] In some implementations, the holographic capture system, computing device, and holographic display system are configured together to capture an optical hologram of a live scene and optically reconstruct the live scene in real time.

[0009] In some implementations, digital holograms include amplitude-like holograms, and holographic display systems include displays for phase modulation.

[0010] In some implementations, the optical sensor includes a digital sensor, and the sequential hologram data includes a stream of digital data. The digital data can include an array of data bits.

[0011] In some implementations, the system further includes a frame grabber configured to select individual hologram data from one or more sequential optical holograms coupled to an optical sensor and transmitted to a computing device.

[0012] In some implementations, the frame grabber includes a frame buffer-based grabber configured to deposit individual hologram data into the frame grabber's frame buffer, and then to the computing device.

[0013] In some implementations, the frame grabber includes a first-in, first-out (FIFO) based grabber configured to directly deposit individual hologram data into the computing device.

[0014] In some implementations, the frame grabber is externally coupled to the optical sensor and the computing device, respectively.

[0015] In some implementations, the frame grabber is included in an optical sensor or computing device.

[0016] In some implementations, an optical sensor includes multiple sensing pixels in its active area, and a holographic display system includes a display having multiple display elements. A computing device is configured to process at least a portion of sequential hologram data to generate a digital hologram associated with a live scene, based on at least one of the pitch of the sensing pixels, the pitch of the display elements, the size of the active area of ​​the optical sensor, or the size of the display.

[0017] In some implementations, the pitch of the sensing pixels is related to the resolution of the captured optical hologram and the size of the scene that can be captured.

[0018] In some implementations, the pitch of the display elements is related to the acceptable viewing angle of the reconstructed scene and the size of the display.

[0019] In some implementations, computing devices are configured to minimize mismatches between a captured optical hologram of a live scene and a reconstruction of the live scene. The mismatch is associated with at least one of the following: the difference between the pitch of the sensing pixels and the pitch of the display elements, or the difference between the size of the active area of ​​the optical sensor and the size of the display.

[0020] In some implementations, the computing device is configured to perform at least one of the following: scaling a first digital hologram associated with a captured optical hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements; or adjusting the scaled first digital hologram based on the size of the display and the size of the hologram data in order to generate a second digital hologram that is modulated on the display.

[0021] In some implementations, the computing device is configured to perform scaling using at least one of one or more interpolation algorithms, including linear interpolation, nearest neighbor interpolation, cubic spline interpolation, shape-preserving interpolation, biharmonic interpolation, and thin-plate spline interpolation.

[0022] In some implementations, the computing device is configured to resample a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on a display using Fourier transforms and inverse Fourier transforms, where the first digital hologram is associated with the pitch of the sensing pixels and the second digital hologram is associated with the pitch of the display elements.

[0023] In some implementations, the computing device is configured to perform a Fourier transform on a first digital hologram to generate a transformed first digital hologram, to perform zero padding on the transformed first digital hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements if the pitch of the sensing pixels is greater than the pitch of the display elements, and then to perform an inverse Fourier transform on the transformed first digital hologram with zero padding to obtain a second digital hologram.

[0024] In some implementations, the computing device is configured to perform a Fourier transform on a first digital hologram to generate a transformed first digital hologram, to crop the transformed first digital hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements, depending on whether it has determined that the pitch of the sensing pixels is smaller than the pitch of the display elements, and then to perform an inverse Fourier transform on the cropped transformed first digital hologram to obtain a second digital hologram.

[0025] In some implementations, the computing device is configured to resample a first digital hologram into a second digital hologram by resampling the center points of multiple sensing pixels of an optical sensor so that they match the geometric centers of multiple display elements of the display.

[0026] In some implementations, multiple sensing pixels of an optical sensor are regularly arranged within the active area of ​​the optical sensor, multiple display elements are regularly arranged within the display, the center points of the multiple sensing pixels are regularly spaced apart, and the geometric centers of the multiple display elements of the display are regularly spaced apart.

[0027] In some implementations, a plurality of sensing pixels of an optical sensor are regularly arranged in an active area of the optical sensor, a plurality of display elements are irregularly arranged on a display, central points of the plurality of sensing pixels are regularly spaced apart, and geometric centers of the plurality of display elements of the display are irregularly spaced apart.

[0028] In some implementations, a computing device is configured to resample regularly spaced central points of a plurality of sensing pixels to match irregularly spaced geometric centers of the plurality of display elements by using one or more weighting algorithms to determine positions of geometric centers of each of the plurality of display elements based on weighted sums of adjacent central points around the geometric center. The plurality of display elements can form a Voronoi pattern.

[0029] In some implementations, a plurality of sensing pixels of an optical sensor are irregularly arranged in an active area of the optical sensor, a plurality of display elements are irregularly arranged on a display, central points of the plurality of sensing pixels are irregularly spaced apart, and geometric centers of the plurality of display elements of the display are irregularly spaced apart. In some implementations, an irregular pattern formed by the plurality of sensing pixels matches an irregular pattern formed by the plurality of display elements.

[0030] In some implementations, an optical system includes an interferometer, and an optical hologram includes an interference pattern of an object beam interacting with a live scene and a reference beam interfering with the object beam by the interferometer.

[0031] In some implementations, an optical system includes a coherent light source configured to emit a coherent light beam and a beam splitter that separates the coherent light beam from the coherent light source into an object beam and a reference beam.

[0032] In some implementations, the optical system further includes a beam combiner, where the live scene lies on the optical path of the object beam upstream of the beam combiner, and the beam combiner is configured to superimpose the reference beam and the object beam to form an interference pattern.

[0033] In some implementations, the optical sensor is positioned downstream of the beam combiner and configured to directly capture interference patterns on the active area of ​​the optical sensor.

[0034] In some implementations, there is no optical lens between the beam combiner and the optical sensor.

[0035] In some implementations, there are no optical lenses in the optical path of the reference beam between the beam splitter and the beam combiner.

[0036] In some implementations, the live scene is transparent, and the object beam passes through the live scene before entering the beam combiner.

[0037] In some implementations, the live scene is reflective, and the optical system includes one or more reflective mirrors configured to guide the object beam toward the live scene at an angle such that the object beam is reflected or scattered from the live scene before entering the beam combiner.

[0038] In some implementations, the optical system further includes one or more optical lenses configured to enlarge or reduce the object beam, after it has interacted with the live scene, to fit the active area of ​​the optical sensor, on the optical path of the object beam between the live scene and the beam combiner.

[0039] In some implementations, the optical system further includes absorbers positioned on the sides of the beam combiner and configured to absorb other portions of the reference beam that propagate away from the interference pattern.

[0040] In some implementations, the optical system further includes one or more optical lenses positioned between the beam splitter and the live scene, configured to enlarge or reduce the object beam to fit the active area of ​​the optical sensor.

[0041] In some implementations, the optical system includes a collimator positioned upstream of the beam splitter and configured to collimate a coherent light beam from a coherent light source.

[0042] In some implementations, the optical system includes a phase adjuster positioned on one of the optical paths of the object beam and the reference beam, configured to dynamically adjust the phase shift of one of the beams before an interference pattern is formed, and the phase adjuster includes a phase shifter or dynamic retarder.

[0043] In some implementations, the phase adjuster is configured to sequentially adjust the phase shift to a set of predetermined values ​​within a given time period so that the optical sensor captures a corresponding sequential optical hologram of the live scene over that time period.

[0044] In some implementations, the system further includes a computing device coupled between the holographic capture system and the holographic display system. The computing device is configured to generate a noise-reduced digital hologram of the live scene based on the corresponding sequential optical hologram.

[0045] In some implementations, the phase adjuster includes a liquid crystal cell configured to adjust the phase shift by corresponding voltages to a set of predetermined values.

[0046] In some implementations, the liquid crystal cell is a single cell with a size greater than or equal to the size of either the object beam or the reference beam.

[0047] In some implementations, the correspondence between a predetermined value of the phase shift and the corresponding voltage is pre-calibrated and predetermined.

[0048] In some implementations, the computing device is configured to process a corresponding sequential optical hologram to obtain a corresponding raw digital hologram, and the computing device is configured to perform one or more mathematical operations on the corresponding raw digital hologram to generate a digital hologram of the live scene.

[0049] In some implementations, the set of predetermined values ​​for the phase shift includes 0, pi / 2, pi, and 3pi / 2, and the digital hologram is calculated based on the following formula. final_hologram=(hologram_0-hologram_pi) / (hologram_pi / 2)-hologram_3pi / 2, In the formula, final_hologram represents a digital hologram, hologram_0 represents a first corresponding raw digital hologram based on a first corresponding optical hologram with a phase shift of 0, hologram_pi / 2 represents a second corresponding raw digital hologram based on a second corresponding optical hologram with a phase shift of pi / 2, hologram_pi represents a third corresponding raw digital hologram based on a third corresponding optical hologram with a phase shift of pi, and hologram_3pi / 2 represents a fourth corresponding raw digital hologram based on a fourth corresponding optical hologram with a phase shift of 3pi / 2.

[0050] In some implementations, a coherent light source includes multiple optical elements, each of which emits a distinct color.

[0051] In some implementations, multiple coherent optical elements are configured to emit light of their respective colors sequentially and alternately, and the holographic capture system further includes corresponding color filters positioned upstream of the optical sensor for each of the multiple coherent optical elements, the corresponding color filters being configured to transmit light of the corresponding color from the coherent optical elements and block light of other colors reaching the optical sensor.

[0052] In some implementations, multiple coherent optical elements are configured to emit light of their respective colors simultaneously. The holographic capture system further includes a color filter array containing groups of different color filters on multiple sensing pixels of an optical sensor, where each color filter is associated with a specific color, and each group of different color filters is positioned on a corresponding group of adjacent sensing pixels of the multiple sensing pixels.

[0053] In some implementations, the optical sensor is configured to determine holographic data for each color based on captured optical holograms captured by corresponding groups of adjacent sensing pixels of multiple sensing pixels.

[0054] In some implementations, the system further includes a computing device coupled between the holographic capture system and the holographic display system, the computing device configured to generate digital holograms for each color based on captured optical holograms captured by corresponding groups of adjacent sensing pixels of multiple sensing pixels.

[0055] In some implementations, a holographic display system includes a display containing multiple display elements and a driver device coupled to the display. The driver device is configured to generate control signals for the multiple display elements of the display based on a digital hologram associated with a live scene, and to transmit the control signals to the display in order to modulate the multiple display elements of the display based on the control signals.

[0056] In some implementations, the system further includes a computing device coupled between the holographic capture system and the holographic display system. The computing device is configured to receive at least a portion of the sequential holographic data from the optical sensor and to generate a digital hologram associated with the live scene based on at least a portion of the sequential holographic data. The holographic display system is configured to receive the digital hologram associated with the live scene from the computing device and to reconstruct the live scene in 3D space based on the digital hologram.

[0057] In some implementations, a digital hologram includes a group of digital holograms for multiple colors, and a holographic display system further includes an illuminator containing multiple coherent light elements for multiple colors.

[0058] In some implementations, the driving device is configured to control the illuminator to sequentially modulate the display with a first digital hologram for a first color during a first time period, then modulate the display with a second digital hologram for a second color during a subsequent second time period, and to sequentially light up a first coherent optical element to emit light of the first color during the first time period, and a second coherent optical element to emit light of the second color during a subsequent second time period.

[0059] In some implementations, the driving device includes at least one of the following: a display driver coupled to a display, a lighting driver coupled to a lighting fixture, or memory coupled to at least one of the display driver or the lighting driver.

[0060] In some implementations, the reconstructed live scene in 3D space has one or more holographic properties, including occlusion, parallax, and accommodation.

[0061] Another aspect of this disclosure is characterized by the method performed by the system described above.

[0062] Another aspect of the present disclosure features a method comprising: optically generating an optical hologram of a live scene including one or more three-dimensional (3D) objects; capturing a successive optical hologram of the live scene; generating sequential hologram data associated with the successive optical hologram of the live scene, wherein each optical hologram is associated with individual hologram data; and reconstructing the live scene in 3D space based on at least a portion of the sequential hologram data.

[0063] In some implementations, the method further includes processing at least a portion of sequential hologram data to generate a digital hologram associated with the live scene. Reconstructing the live scene in 3D space based on at least a portion of the hologram data includes reconstructing the live scene in 3D space based on the digital hologram.

[0064] In some implementations, each digital hologram includes an amplitude-like hologram. Reconstructing a live scene in 3D space based on a digital hologram involves directly modulating the display for phase modulation with the digital hologram.

[0065] In some implementations, sequential hologram data includes a stream of digital data, and the digital data includes an array of data bits.

[0066] In some implementations, the method further includes a frame grabber selecting individual hologram data of one or more optical holograms from a sequence of optical holograms. Processing at least a portion of the sequence hologram data to generate a digital hologram associated with the live scene includes generating a digital hologram associated with the live scene based on the selected individual hologram data of one or more optical holograms.

[0067] In some implementations, the frame grabber includes either a frame buffer-based grabber configured to deposit individual hologram data into the frame buffer of the frame grabber before transmitting it to generate a digital hologram, or a first-in, first-out (FIFO)-based grabber configured to transmit individual hologram data directly to generate a digital hologram.

[0068] In some implementations, processing at least a portion of sequential hologram data to generate a digital hologram associated with a live scene includes processing at least a portion of sequential hologram data based on at least one of the pitch of the sensing pixels of an optical sensor, the pitch of the display elements of a display, the size of the active area of ​​the optical sensor, or the size of the display, in order to generate a digital hologram associated with a live scene.

[0069] In some implementations, the pitch of the sensing pixels is related to the resolution of the captured optical hologram and the size of the scene that can be captured, while the pitch of the display elements is related to the acceptable viewing angle of the reconstructed scene and the size of the display.

[0070] In some implementations, processing at least a portion of the sequential hologram data to generate a digital hologram associated with the live scene involves suppressing a mismatch between the captured optical hologram of the live scene and the reconstruction of the live scene. The mismatch is associated with at least one of the following: the difference between the pitch of the sensing pixels and the pitch of the display elements, or the difference between the size of the active area of ​​the optical sensor and the size of the display.

[0071] In some implementations, processing at least a portion of sequential hologram data to generate a digital hologram associated with a live scene includes at least one of scaling a first digital hologram associated with a captured optical hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements, or cropping the scaled first digital hologram based on the size of the display and the size of the hologram data to generate a second digital hologram that is modulated on the display.

[0072] In some implementations, scaling the size of a first digital hologram associated with a captured optical hologram involves using at least one of one interpolation algorithms, including linear interpolation, nearest neighbor interpolation, cubic spline interpolation, shape-preserving interpolation, biharmonic interpolation, and thin-plate spline interpolation.

[0073] In some implementations, processing at least a portion of the sequential hologram data to generate a digital hologram associated with a live scene includes resampling a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on a display using a Fourier transform (e.g., FFT) and an inverse Fourier transform (e.g., inverse FFT), wherein the first digital hologram is associated with the pitch of the sensing pixels and the second digital hologram is associated with the pitch of the display elements.

[0074] In some implementations, resampling a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on a display involves performing a Fourier transform on the first digital hologram to generate the transformed first digital hologram, cropping the transformed first digital hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements, depending on whether the pitch of the sensing pixels is smaller than the pitch of the display elements, and then performing an inverse Fourier transform on the cropped transformed first digital hologram to obtain the second digital hologram.

[0075] In some implementations, resampling a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on a display involves performing a Fourier transform on the first digital hologram to generate the transformed first digital hologram, adding one or more zero-pads to the transformed first digital hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements if the pitch of the sensing pixels is greater than the pitch of the display elements, and then performing an inverse Fourier transform on the transformed first digital hologram with the added one or more zero-pads to obtain the second digital hologram.

[0076] In some implementations, resampling a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on a display involves resampling the first digital hologram into a second digital hologram by resampling the center points of multiple sensing pixels of an optical sensor to match the geometric centers of multiple display elements of the display.

[0077] In some implementations, multiple sensing pixels of an optical sensor are regularly arranged within the active area of ​​the optical sensor, multiple display elements are regularly arranged within the display, the center points of the multiple sensing pixels are regularly spaced apart, and the geometric centers of the multiple display elements of the display are regularly spaced apart.

[0078] In some implementations, multiple sensing pixels of an optical sensor are regularly arranged within the active area of ​​the optical sensor, multiple display elements are irregularly arranged on the display, the center points of the multiple sensing pixels are regularly spaced apart, and the geometric centers of the multiple display elements of the display are irregularly spaced apart.

[0079] In some implementations, resampling a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on a display involves resampling regularly spaced center points of multiple sensing pixels to match irregularly spaced geometric centers of multiple display elements by determining the position of each geometric center of multiple display elements based on a weighted sum of adjacent center points around the geometric center using one or more weighting algorithms.

[0080] In some implementations, multiple sensing pixels of an optical sensor are irregularly arranged in the active area of ​​the optical sensor, multiple display elements are irregularly arranged in the display, the center points of the multiple sensing pixels are irregularly spaced apart, the geometric centers of the multiple display elements of the display are irregularly spaced apart, and the irregular pattern formed by the multiple sensing pixels matches the irregular pattern formed by the multiple display elements.

[0081] In some implementations, optically generating an optical hologram of a live scene involves forming an interference pattern by interfering an object beam interacting with the live scene with a reference beam, wherein the object beam and the reference beam are coherent light beams, and the optical hologram includes the interference pattern.

[0082] In some implementations, capturing a sequential optical hologram of a live scene involves directly capturing interference patterns on the active area of ​​an optical sensor.

[0083] In some implementations, the method further includes scaling the object beam, after it has interacted with the live scene, to fit the active area of ​​the optical sensor.

[0084] In some implementations, the method further includes dynamically adjusting the phase shift of either the object beam or the reference beam before the interference pattern is formed.

[0085] In some implementations, dynamically adjusting the phase shift of either the object beam or the reference beam involves sequentially adjusting the phase shift to a series of predetermined values ​​over a time period. Capturing a successive optical hologram of a live scene involves capturing the corresponding successive optical hologram of the live scene over that time period.

[0086] In some implementations, the method further includes generating a noise-suppressed digital hologram of the live scene based on the corresponding sequential optical hologram.

[0087] In some implementations, generating a noise-suppressed digital hologram of a live scene based on a corresponding sequential optical hologram involves processing the corresponding sequential optical hologram to obtain the corresponding raw digital hologram, and performing one or more mathematical operations on the corresponding raw digital hologram to generate the digital hologram of the live scene.

[0088] In some implementations, the set of predetermined values ​​for the phase shift includes 0, pi / 2, pi, and 3pi / 2, and the digital hologram is calculated based on the following formula. final_hologram=(hologram_0-hologram_pi) / (hologram_pi / 2-hologram_3pi / 2), In the formula, final_hologram represents a digital hologram, hologram_0 represents a first corresponding raw digital hologram based on a first corresponding optical hologram with a phase shift of 0, hologram_pi / 2 represents a second corresponding raw digital hologram based on a second corresponding optical hologram with a phase shift of pi / 2, hologram_pi represents a third corresponding raw digital hologram based on a third corresponding optical hologram with a phase shift of pi, and hologram_3pi / 2 represents a fourth corresponding raw digital hologram based on a fourth corresponding optical hologram with a phase shift of 3pi / 2.

[0089] In some implementations, optically generating an optical hologram of a live scene involves sequentially and alternately emitting light of multiple colors to sequentially and alternately generate optical holograms for multiple colors. Capturing a sequential optical hologram of a live scene involves sequentially transmitting only light of individual colors and blocking light of other colors.

[0090] In some implementations, optically generating an optical hologram of a live scene involves simultaneously emitting light with multiple colors. Capturing a sequential optical hologram of a live scene involves capturing the optical hologram using a color filter array placed on an optical sensor, wherein the color filter array includes groups of different color filters on the multiple sensing pixels of the optical sensor, each of which is associated with multiple colors, and each group of different color filters is placed on a corresponding group of adjacent sensing pixels.

[0091] In some implementations, generating sequential hologram data associated with a sequential optical hologram of a live scene involves determining hologram data for each of multiple colors based on the optical hologram.

[0092] In some implementations, the method further includes generating digital holograms for multiple colors based on optical holograms, and reconstructing a live scene in 3D space based on at least a portion of the hologram data, or reconstructing a live scene in 3D space based on digital holograms.

[0093] In some implementations, reconstructing a live scene in 3D space based on at least a portion of the hologram data involves generating control signals for multiple display elements of a display based on the digital hologram associated with the live scene, and modulating the multiple display elements of the display based on the control signals.

[0094] In some implementations, the method further includes generating a digital hologram associated with a live scene based on at least a portion of the sequential hologram data. The digital hologram comprises a group of digital holograms for multiple colors.

[0095] In some implementations, reconstructing a live scene in 3D space based on at least a portion of holographic data includes sequentially modulating the display with a first digital hologram for a first color during a first time period, then modulating the display with a second digital hologram for a second color during a subsequent second time period, and sequentially illuminating a first coherent optical element to emit light in the first color during the first time period, and a second coherent optical element to emit light in the second color during a subsequent second time period.

[0096] Another aspect of the present disclosure features a system comprising: a hologram generation system configured to generate one or more digital holograms corresponding to a live scene including one or more three-dimensional (3D) objects; and a holographic display system configured to reconstruct the live scene in 3D space based on one or more digital holograms. The hologram generation system comprises: one or more scene acquisition devices configured to capture visual data of a live scene from one or more views, which includes at least one of one or more images or one or more videos; and a computing system further configured to acquire primitive data associated with the live scene based on the captured visual data of the live scene, and to generate a digital hologram corresponding to the live scene based on the primitive data associated with the live scene and display element information of the display of the holographic display system.

[0097] In some implementations, the computing system includes a computing device coupled with one or more scene acquisition devices, configured to generate a 3D representation of a live scene based on captured visual data of the live scene, and to acquire primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, wherein the primitive data associated with the live scene includes primitive data of the 3D representation of the live scene.

[0098] In some implementations, the computing device is configured to use a 3D rendering algorithm to generate a 3D representation of the live scene based on captured visual data of the live scene.

[0099] In some implementations, one or more scene acquisition devices are configured to transmit sequential visual data of a live scene over a period of time to a computing device, which includes first visual data and second visual data following the first visual data. The computing device is configured to generate a first 3D representation of the live scene based on the first visual data of the live scene using a 3D rendering algorithm, and to generate a second 3D representation of the live scene by updating the first 3D representation of the live scene based on the difference between the first and second visual data using a 3D rendering algorithm.

[0100] In some implementations, the computing device is configured to load a 3D representation of a live scene into a 3D simulation application and retrieve primitive data of the 3D representation of the live scene based on the output associated with the 3D representation of the live scene in the 3D simulation application.

[0101] In some implementations, the primitive data for the 3D representation of a live scene is data for multiple primitives corresponding to the 3D representation of the live scene, where the data includes primitive data for each of the multiple primitives, where each primitive includes at least one vertex, and the primitive data of the primitive includes data for at least one vertex.

[0102] In some implementations, the primitive data of a primitive includes at least one of the primitive identifier of the primitive, at least one vertex identifier of at least one vertex, coordinate information of the primitive in a 3D coordinate system, color information of the primitive, texture coordinate information of the primitive, shading information for the primitive, viewpoint-dependent shading information associated with the primitive, or occlusion information of the primitive.

[0103] In some implementations, the computing system includes a processing device coupled to the computing system, which is configured to determine, for each of a plurality of primitives, the contribution of the electromagnetic (EM) field to each of a plurality of display elements of the display based on the primitive data of the primitive, and to generate, for each of the plurality of display elements of the display, the sum of the EM field contributions of the plurality of primitives to the display element, and the digital hologram includes the sum of the EM field contributions to the plurality of display elements of the display.

[0104] In some implementations, the computing device is configured to use an application programming interface (API) to generate primitive data for a 3D representation of a live scene based on the output of a 3D simulation application. The computing device is configured to execute the API to associate a unique vertex identifier of each of the multiple vertices of a multiple primitive with the unique vertex data of the vertex, and to store the association between the unique vertex identifier and the unique vertex data of the vertex in the computing device's memory. For each of the multiple primitives, the API is configured to associate a unique primitive identifier of the primitive with the vertex identifier of one or more of the vertices of the primitive in memory, and to store the association between the unique primitive identifier and the vertex identifier of one or more of the primitives in memory.

[0105] In some implementations, the computing device is configured to execute an API to determine the primitive identifiers of multiple primitives associated with a command instruction, determine the vertex identifiers associated with the primitive identifiers, and send a command containing the command instruction, the vertex identifiers associated with the primitive identifiers, and the primitive identifiers of the multiple primitives, without the primitive data of the multiple primitives. The command instructs the device to draw the multiple primitives according to the command instruction, based on at least one of the primitive identifiers of the multiple primitives or the vertex identifiers associated with the primitive identifiers.

[0106] In some implementations, the processing device includes a command processor, multiple computing units, and an accumulator. The command processor is configured to receive commands from the computing device and, based on the commands, process the commands to obtain primitive data for multiple primitives from the computing device. The multiple computing units may be configured to calculate the contribution of each of the multiple primitives' electromagnetic (EM) fields to each of the multiple display elements, based on the primitive data of the multiple primitives. The accumulator may be configured to accumulate the EM field contributions of the multiple primitives to each of the multiple display elements.

[0107] In some implementations, the command processor, multiple computing units, and accumulator are connected in series, while the multiple computing units are connected in parallel between the command processor and the accumulator.

[0108] In some implementations, digital holograms are complex value holograms, phase holograms, or amplitude holograms.

[0109] In some implementations, a holographic display system includes a driver device coupled to the display, which is configured to generate modulation control signals for multiple display elements of the display based on a digital hologram corresponding to a live scene.

[0110] In some implementations, the digital hologram is a complex-value hologram, and the driving device is configured to convert the complex-value hologram into a phase-only hologram, and then generate the respective modulation control signals for multiple display elements based on the phase-only hologram.

[0111] In some implementations, the holographic display system further includes an illuminator. The driving device is configured to transmit illumination control signals to the illuminator to act on the illuminator to illuminate the display such that the light, through the modulated display elements of the display, forms a volume light field corresponding to the live scene. In conjunction with transmitting illumination control signals to the illuminator, the driving device is configured to output individual modulation control signals to each of the multiple display elements.

[0112] In some implementations, the driving device is configured to sequentially output a first modulation control signal for modulating the display with a first digital hologram associated with a first color during a first time period, a second modulation control signal for modulating the display with a second digital hologram associated with a second color during a subsequent second time period, a first illumination control signal for activating an illuminator to turn on a first coherent optical element to emit light in the first color during the first time period, and a second illumination control signal for activating an illuminator to turn on a second coherent optical element to emit light in the second color during the second time period.

[0113] In some implementations, the hologram generation system is configured to generate sequential digital holograms corresponding to the live scene, and the holographic display system is configured to sequentially reconstruct the live scene in 3D space based on the sequential digital holograms.

[0114] Another aspect of the present disclosure is a method comprising capturing visual data of a live scene from one or more views, wherein the live scene comprises one or more three-dimensional (3D) objects, and the visual data comprises at least one of one or more images or one or more videos; obtaining primitive data associated with the live scene based on the captured visual data of the live scene; generating one or more digital holograms corresponding to the live scene based on the primitive data associated with the live scene and display element information of a display; and reconstructing the live scene in 3D space by modulating the display with one or more digital holograms.

[0115] In some implementations, obtaining primitive data associated with a live scene based on captured visual data of the live scene includes generating a 3D representation of the live scene based on the captured visual data of the live scene, and obtaining primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, wherein the primitive data associated with the live scene includes primitive data of the 3D representation of the live scene.

[0116] In some implementations, generating a 3D representation of a live scene based on captured visual data of the live scene involves processing the captured visual data of the live scene using a 3D rendering algorithm to generate a 3D representation of the live scene.

[0117] In some implementations, the method includes generating sequential visual data of a live scene over a certain period of time, comprising first visual data and second visual data following the first visual data; generating a first 3D representation of the live scene based on the first visual data of the live scene using a 3D rendering algorithm; and generating a second 3D representation of the live scene by updating the first 3D representation of the live scene based on the difference between the first and second visual data using a 3D rendering algorithm.

[0118] In some implementations, obtaining primitive data of a 3D representation of a live scene based on the 3D representation of the live scene involves loading the 3D representation of the live scene into a 3D simulation application and obtaining primitive data of the 3D representation of the live scene based on the output associated with the 3D representation of the live scene in the 3D simulation application.

[0119] In some implementations, the primitive data for the 3D representation of the live scene is data for multiple primitives corresponding to the 3D representation of the live scene, wherein the data includes primitive data for each of the multiple primitives, each primitive includes at least one vertex, and the primitive data of the primitive includes data for at least one vertex.

[0120] In some implementations, the primitive data of a primitive includes at least one of the primitive identifier of the primitive, at least one vertex identifier of at least one vertex, coordinate information of the primitive in a 3D coordinate system, color information of the primitive, texture coordinate information of the primitive, shading information for the primitive, viewpoint-dependent shading information associated with the primitive, or occlusion information of the primitive.

[0121] In some implementations, generating one or more digital holograms corresponding to a live scene involves determining the electromagnetic (EM) field contribution to each of the display elements of the display based on the primitive data of each of the multiple primitives, and generating the sum of the EM field contributions of the multiple primitives to each of the display elements of the display. The digital hologram includes the sum of the EM field contributions to the display elements of the display.

[0122] In some implementations, the method further includes associating a distinct vertex identifier of each of the multiple vertices of a multiple primitive with the distinct vertex data of the vertex, storing the association between the distinct vertex identifier and the distinct vertex data of the vertex in memory, associating a distinct primitive identifier of the primitive with the vertex identifier of one or more of the vertices of the primitive in memory, and storing the association between the distinct primitive identifier and the vertex identifier of one or more of the primitives in memory.

[0123] In some implementations, the method further includes determining the primitive identifiers of multiple primitives associated with a command instruction, determining the vertex identifiers associated with the primitive identifiers, and generating a command that includes the command instruction, the vertex identifiers associated with the primitive identifiers, and the primitive identifiers of the multiple primitives. The command instructs the system to draw the multiple primitives according to the command instruction, based on at least one of the primitive identifiers of the multiple primitives or the vertex identifiers associated with the primitive identifiers.

[0124] In some implementations, the method includes processing a command to obtain primitive data for multiple primitives based on the command, calculating the contribution of each of the multiple primitives' electromagnetic (EM) fields to each of the multiple display elements based on the primitive data for the multiple primitives, and accumulating the contributions of the multiple primitives' EM fields to each of the multiple display elements.

[0125] In some implementations, reconstructing a live scene in 3D space by modulating a display with one or more digital holograms involves generating modulation control signals for multiple display elements of the display based on the digital holograms corresponding to the live scene.

[0126] In some implementations, the digital hologram is a complex-value hologram, and the method involves converting the complex-value hologram into a phase-only hologram, and generating respective modulation control signals for multiple display elements based on the phase-only hologram.

[0127] In some implementations, reconstructing a live scene in 3D space by modulating a display with one or more digital holograms involves sending illumination control signals to an illuminator to cause the illuminator to illuminate the display so that the light forms a volume light field corresponding to the live scene through the modulated display elements of the display, and, in coordination with sending illumination control signals to the illuminator, outputting individual modulation control signals to each of the multiple display elements.

[0128] In some implementations, reconstructing a live scene in 3D space by modulating a display with one or more digital holograms includes sequentially outputting a first modulation control signal for modulating the display with information associated with a first color during a first time period, and a second modulation control signal for modulating the display with information associated with a second color during a subsequent second time period, and sequentially outputting a first illumination control signal for activating an illuminator to turn on a first coherent photon element to emit light of the first color during the first time period, and a second illumination control signal for activating an illuminator to turn on a second coherent photon element to emit light of the second color during the second time period.

[0129] In some implementations, the method includes generating a sequential digital hologram corresponding to a live scene based on sequentially captured visual data of the live scene, and sequentially reconstructing the live scene in 3D space based on the sequential digital hologram.

[0130] Another aspect of the present disclosure features a system for generating a digital hologram corresponding to a live scene. The system includes one or more scene acquisition devices and a computing system. The computing system is configured to capture visual data of the live scene from one or more views, wherein the live scene includes one or more three-dimensional (3D) objects and the visual data includes at least one of one or more images or one or more videos. Based on the captured visual data of the live scene, the computing system may be configured to acquire primitive data associated with the live scene and generate a digital hologram corresponding to the live scene based on the primitive data associated with the live scene and display element information of a display.

[0131] In some implementations, the computing system is implemented by the computing system described herein.

[0132] Another aspect of this disclosure features a method performed by the system described herein.

[0133] This disclosure provides techniques that can overcome the limitations present in known techniques. For example, the techniques disclosed herein can capture and display live scenes containing 3D objects in real time, with or without minimum latency in the real world, and / or with high processing speed or minimal data processing. As another example, the techniques disclosed herein provide actual 3D reconstruction using holography, which can be implemented without the use of cumbersome wearable devices such as "3D glasses." As yet another example, the techniques disclosed herein can be implemented at will without being limited by the accuracy of the tracking mechanism, the quality of the display device, relatively long processing times and / or relatively high computational requirements, and / or the inability to display objects to multiple viewers simultaneously. As a further example, the techniques can be implemented without dedicated tools and software for developing content that extends beyond the tools and software used in conventional 3D content creation. Various embodiments can demonstrate one or more of the aforementioned advantages. For example, certain implementations of this disclosure can generate real-time, full-color, authentic 3D images that appear as if they were real 3D objects and can be viewed simultaneously by multiple viewers from different locations without any wearable devices.

[0134] In this disclosure, the term “live scene” means a real-world scene, a real-life scene, or a physical scene, as opposed to a computer-generated scene or a digital scene. A live scene may include one or more three-dimensional (3D) objects in the real world. The term “real-time” means an event that occurs instantaneously (e.g., capturing a live scene and holographic reconstruction of a live scene) or a series of events where the delay between the events is a predetermined threshold (e.g., 10 milliseconds (ms), 50 ms, 66 ms, 100 ms, or any other appropriate value).

[0135] The term "primitive" refers to a basic element for input or output within a computing system. An element can be a geometric element or a graphic element. For example, in vector computer graphics, CAD systems, and geographic information systems, a geometric primitive (or prim) is the simplest (e.g., "atomic" or less) geometric shape that the system can process (e.g., draw, store). The term "vertex" refers to a node of a primitive that can connect to one or more other nodes to form the primitive.

[0136] The term "hologram" refers to a pattern displayed by (or uploaded to) a display that includes amplitude information, phase information, or any combination thereof, about an object. The term "optical hologram" refers to the physical representation of a hologram, e.g., the interference pattern between a reference beam and an object beam of coherent light (e.g., laser light). The term "digital hologram" refers to the digital representation of a hologram. Digital holograms can be produced by a) processing the hologram data of an optical hologram, b) processing an image / video of the object itself, c) simulating the interference process on a computer using mathematical models and algorithms (this may also be called a computational hologram), or d) calculating the electromagnetic contribution from the primitives of the object to the display elements (e.g., produced using Maxwell holography). The term "amplitude-like hologram" can refer to a hologram that functions like an amplitude hologram, but the hologram includes both amplitude and phase information of the object, enabling the reconstruction of the object in 3D space. The term "holographic reconstruction" refers to the volume light field (e.g., holographic light field) from a display modulated with a hologram when illuminated.

[0137] As used herein, the term “irregular” means “non-periodic” and / or “non-uniform.” For example, the term “irregular shape” may indicate that the shape has sides and / or angles of different lengths and / or sizes. The term “irregular pattern” may indicate that i) the components (e.g., phasels) within an area of ​​the pattern are arranged non-periodically, and the components may be the same or different from one another, or ii) the components have different irregular shapes.

[0138] Shading is the process of adding values ​​to create illusions of shape, space, and light in a drawing. Shading can display a drawing in three dimensions and create a compelling image. Shading may differ from techniques that add shadows, such as shadow mapping or shadow volume, which deal with the global behavior of light. The term "shading information" refers to information that describes the depth perception of a 3D model (e.g., in the field of 3D computer graphics) or an illustration (e.g., in visual arts) by changing the level of darkness. Shading information can approximate the local behavior of light on the surface of an object. Shading information can be obtained by any conventional computer-generated imagery (CGI) surface shading method that involves modulating the color or brightness of the primitive's surface. Primitive data disclosed herein may include shading information associated with the primitive.

[0139] The term “view-dependent shading information” can be considered a broader generalization of the term “geometric specular reflection.” Specular reflection is a subset of view-dependent shading. Specular reflection is like a blurred, color-shifted image of a light source, described by the bi-directional reflectance distribution function ("BRDF") of a particular material, such as plastic or glossy wood. View-dependent shading can encompass specular BRDFs as well as perfect specular and image-based lighting. For example, to render a spherical Christmas ornament, the environment surrounding the ornament, including its image, as well as the position and size of each light source, can be reflected, and the viewer's position can be part of the reflection calculation. The reflection appears to move as the viewpoint changes, revealing different parts of the environment. Similarly, the position of a bright specular area on a plastic surface can be the sum of the view-dependent projections of Christmas lights reflected from the spherical ornament multiplied by the BRDF. Primitive data disclosed herein may include view-dependent shading information associated with the primitive.

[0140] Details of one or more implementations of the subject matter described herein are shown in the accompanying drawings and related descriptions. Other features, embodiments, and advantages of the subject matter will become apparent from the following detailed description, drawings, and claims.

[0141] It should be understood that various implementation forms can be combined in different ways. For example, features from one particular method, device, or system can be combined with features from another method, device, or system. [Brief explanation of the drawing]

[0142] [Figure 1A] A schematic diagram of an exemplary system for capturing and displaying live scenes using computational methods is shown.

[0143] [Figure 1B] This demonstrates an exemplary holographic reconstruction for live scenes.

[0144] [Figure 2] A schematic diagram of an exemplary system for capturing and displaying live scenes using optical techniques is shown.

[0145] [Figure 3A] An illustrative schematic diagram of a holographic capture system is shown.

[0146] [Figure 3B] This diagram shows an exemplary holographic capture system for capturing transparent objects.

[0147] [Figure 3C] An illustrative schematic diagram of a holographic capture system for capturing opaque objects is shown.

[0148] [Figure 3D] A schematic diagram of an exemplary holographic capture system for noise suppression is shown.

[0149] [Figure 4A] This shows an example of processing hologram-based images.

[0150] [Figure 4B] Here's another example of processing hologram-based images.

[0151] [Figure 4C] Examples of raw holograms and noise-reduced processed holograms are shown.

[0152] [Figure 5A] This document illustrates an exemplary system for a 3D display, including a reflective display with optical diffraction illumination.

[0153] [Figure 5B] Another exemplary system for a 3D display, including a reflective display with optical diffraction illumination, is shown.

[0154] [Figure 5C] Another exemplary system for a 3D display, including a transmissive display with optical diffraction illumination, is shown.

[0155] [Figure 5D] Another exemplary system for a 3D display, including a transmissive display with optical diffraction illumination, is shown.

[0156] [Figure 6A] An example of an irregular display is shown.

[0157] [Figure 6B] This shows an example of designing an irregular display.

[0158] [Figure 7A] This shows an exemplary queue for the holographic display of an object, including occlusion. [Figure 7B] This shows an exemplary queue for holographic display of an object, including parallax. [Figure 7C] An exemplary queue for the holographic representation of an object, including accommodation (Figure 7C), is shown.

[0159] [Figure 8A] This is an exemplary flowchart of the process for displaying a live scene holographically.

[0160] [Figure 8B] This is a flowchart of another exemplary process for displaying a live scene holographically.

[0161] Similar reference numbers and names in various drawings refer to the same components. It should also be understood that the various exemplary implementation configurations shown in the drawings are merely illustrative and not necessarily drawn to scale. [Modes for carrying out the invention]

[0162] Herein, we refer in detail to embodiments illustrated in the attached drawings. Several specific details are given in the following detailed description to provide a complete understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be carried out without these specific details. In other examples, well-known methods, procedures, components, circuits, and networks are not described in detail so as not to unnecessarily obscure the aspects of the embodiments.

[0163] Several features are described below, each of which can be used independently of others or in any combination with other features. However, no single feature may be able to address any of the problems described above, or only one of them. Some of the problems described above may not be fully addressed by any of the features described herein. Headings are provided, but information regarding a particular heading that is not found in the section containing that heading may be found elsewhere in this description.

[0164] The implementation of this disclosure is described herein in accordance with the following general summary. 1. Overview 2. An example system using computational methods 3. Exemplary Systems Using Optical Techniques 4. Exemplary Holographic Capture System 5. Exemplary Holographic Display System 6. Exemplary Irregular Display 7. Illustrative Holographic Cubes 8. Exemplary Process 9. Exemplary Applications

[0165] 1. Overview Implementations of the present disclosure feature techniques for enabling the holographic display of a live scene containing one or more three-dimensional (3D) objects. The live scene can be an event taking place in the real world, real life, or physical space, such as a soccer match in a stadium, and can be different from a complex computer-generated, simulated, or digitally animated scene (e.g., by a 3D simulation application). These techniques enable the physical capture of the live scene, the creation of a hologram of the live scene (e.g., a monochromatic or multicolored hologram), and the holographic display of the live scene (in one or more colors), which can be done in real time, for example, by transmitting the captured data of the live scene to a holographic display system for reconstructing the live scene with no or minimal latency, and / or without storing the captured data and / or generated hologram of the live scene, and / or with improved processing speed.

[0166] In some implementations, as shown in Figure 1, for example, the live scene can be captured by one or more visual acquisition devices (e.g., cameras or image / video recorders) that can be positioned around the live scene and / or from one or more views. One or more visual acquisition devices can generate visual data of the live scene, which may include one or more images (or videos) of the live scene from one or more views. One or more visual acquisition devices can transmit the visual data to a computing device (e.g., in real time). The computing device can generate a corresponding digital hologram of the live scene based on the visual data. For example, the computing device can use a 3D rendering algorithm (e.g., NeRF) to generate a 3D representation of the live scene based on the visual data (e.g., from two or more views). The computing device can further use a 3D simulation application (e.g., Unity) to acquire primitive data of multiple primitives associated with the 3D representation of the live scene. Next, a processing device (e.g., a Fazel processing device) can generate a digital hologram corresponding to the live scene, for example, using the MAXWELL HOLOGRAPHY® (registered trademark) (or MAXWELL HOLOGRAPHY® trademark) technique, which can speed up the processing of real-time holography compared to conventional digital holography and / or computational holography, based on primitive data associated with the live scene and display element information of the display modulated with the digital hologram. The digital hologram can then be transmitted to a holographic display system, and the display can be modulated with the digital hologram so that light from an illuminator is diffracted (for example, as shown in Figure 1B) to reconstruct the live scene in 3D space.In some implementations, different colors of digital holograms (e.g., red, blue, and green) can be generated by computing and / or processing devices, allowing the holographic display system to reconstruct live scenes in different colors, for example, by sequentially modulating the display with alternating holograms of different colors.

[0167] The calculation (or computation) in MAXWELL HOLOGRAPHY® can be expressed as Maxwell holographic calculation (or Maxwell holographic computation). In conventional digital holography, the digital intensity image of the hologram is first processed, such as by complex wave retrieval, and then the hologram is numerically propagated by the Fresnel approximation, which describes the propagation of waves to the image plane in free space. This results in a complex image containing both amplitude and phase information. In contrast, Maxwell holographic computation approaches the hologram as a general Dirichlet or Cauchy boundary condition problem of an electric field, using tools including field theory, topology, analytical continuation, and / or symmetry groups, which enable solving for holograms in real time without the limitations of legacy holographic systems such as conventional digital holography, and achieve faster processing speeds.

[0168] In some implementations, as shown in Figure 2, for example, the live scene is captured as an optical hologram by a holographic capture system (e.g., as shown in Figures 3A-3C). The optical hologram of the live scene can be an interference pattern of an object beam interacting with the live scene and a reference beam interfering with the object beam using an interferometer such as a Mach-Zehnder or Michelson interferometer. The object beam and reference beam can be from coherent light sources (e.g., lasers). The optical hologram can be captured by a digital sensor (e.g., CCD or CMOS) that outputs the hologram data (e.g., an array of bits) of the optical hologram. The digital sensor can have a high frame rate, such as 90 Hz, and a frame grabber can be used to select at least a portion of the hologram data to send to a computing device. The computing device can process at least a portion of the hologram data to produce a digital hologram that can be displayed by the display of the holographic display system. For example, the digital sensor and the display may have different pitches, and the computing device can convert the hologram data to the correct resolution for the display. Holographic display systems can reconstruct live scenes in 3D space based on digital holograms. Digital holograms can function like amplitude holograms, simplifying the processing of hologram data and increasing processing speed. Displays (e.g., phase-modulated liquid crystal on silicon: LCOS) can be directly modulated with digital holograms without additional devices for conversion. In some implementations, a coherent light source may emit light of different colors (e.g., red, green, blue) alternately or continuously.One or more notch filters or color filter arrays can be placed in front of the digital sensor to generate optical holograms of different colors, and these different-color optical holograms can be processed to generate corresponding digital holograms of different colors. The display can be sequentially modulated by different-color digital holograms to reconstruct a live scene in different colors.

[0169] 2. An example system using computational methods Figure 1A shows a schematic diagram of an exemplary system 100 for capturing and displaying a live scene using computational methods. System 100 may be configured to capture the live scene in the form of images / videos from different views / angles, generate a 3D digital representation of the live scene based on the images / videos, compute a corresponding digital hologram, and then holographically reconstruct the live scene based on the digital hologram. Since the digital hologram can be computed at high speed using Maxwell Holography computation, system 100 enables real-time holographic display of the live scene by dynamically capturing the live scene and displaying the digital hologram. Several techniques can be implemented in system 100 to achieve high computation speed, high display refresh rate, high image quality, and high performance in 3D reconstruction / display, as will be described in more detail below.

[0170] In some implementations, system 100 includes a hologram generation system 101 and a holographic display system 130. The hologram generation system 101 is configured to generate one or more digital holograms corresponding to a live scene that may contain one or more three-dimensional (3D) objects in physical space. The holographic display system 130 is configured to reconstruct the live scene in 3D space based on one or more digital holograms.

[0171] In some implementations, as shown in Figure 1A, for example, the hologram generation system 101 includes a scene acquisition system 102 and a computing system 103. The scene acquisition system 102 may include one or more scene acquisition devices 104 (e.g., cameras, image recorders, or video recorders) configured to acquire scene data of a live scene. The scene data may include visual data 105 (e.g., images and / or video) and / or audio data (e.g., audio or sound). One or more scene acquisition devices 104 may be positioned around the live scene and configured to capture corresponding scene data from different angles or fields of view. In some implementations, the scene acquisition system 102 includes a buffer 106 configured to buffer the scene data before transmitting it from one or more scene acquisition devices 104 to the computing system 103.

[0172] The scene acquisition system 102 can transmit visual data 105 (e.g., images and / or video of the live scene) to the computing system 103. The computing system 103 may be configured to acquire primitive data associated with the live scene based on the captured visual data of the live scene, and to generate a digital hologram corresponding to the live scene based on the primitive data associated with the live scene and the display element information of the display 150 (e.g., phase-only LCOS) of the holographic display system 130, for example, using Maxwell Holography calculations. Data transmission between the scene acquisition system 102 and the computing system 103 can be via a wired connection, a wireless connection, or any high-speed connection. Data transmission between the computing system 103 and the holographic display system 130 can be via a wired connection, a wireless connection, or any high-speed connection.

[0173] In some implementations, the computing system 103 includes a computing device 110 and a processing device 120. The computing device 110 is configured to prepare data for a list of primitives corresponding to a live scene and to transmit this data to the processing device 120 via an interface 115, for example, a PCIe slot or any other high-speed connection. The processing device 120 may be configured to calculate the electromagnetic (EM) field contribution from each of the primitives in the list to each of the display elements of the display 150, for example, using Maxwell Holography calculations, and to output a hologram 127 to the holographic display system 130. In this specification, the hologram 127 refers to modulated data for the display 150, which includes complex information, amplitude information, or phase information, or any combination thereof, relating to at least one object. The driving device 131 of the holographic display system 130 may be configured to generate control signals, based on a hologram, for modulating the display elements of the display 150, which diffract light 145 from the illuminator 140 of the holographic display system 130 to form a holographic light field 160 corresponding to a live scene in 3D space. The holographic light field 160 can be the volume light field from the display 150 when illuminated, and can also refer to a holographic reconstruction. The holographic reconstruction includes a reconstructed scene corresponding to the live scene.

[0174] The processing device 120 can be implemented, for example, as an ASIC, FPGA, integrated circuit, one or more computing units, or any combination thereof. In some implementations, the processing device 120 is packaged as a circuit board integrated with the computing device 110 via a PCIe slot within the computing device 110. In some implementations, the processing device 120 is integrated with the driver device 131 to function as a controller that can be externally coupled to the computing device 110 and the display 150 and / or illuminator 140, for example. In some implementations, the processing device 120 and the driver device 131 are integrated with the display 150 (and optionally the illuminator 140) and mounted together, for example, to form an integrated device that may be called a holographic display device or holographic display system 130.

[0175] The computing device 110 may be a computing device associated with a user (e.g., an operator, developer, programmer, customer, or any appropriate entity). The computing device 110 may be any appropriate type of device, such as a desktop computer, personal computer, notebook, tablet computing device, personal digital assistant (PDA), network equipment, smart mobile phone, smartwatch, enhanced general packet radio service (EGPRS) mobile phone, media player, navigation device, email device, game console, or any two or more of these computing devices or any other computing devices in any appropriate combination.

[0176] The computing device 110 includes an operating system that runs several applications 111, 112 as a graphics engine. In some implementations, the computing device 110 is configured to generate a 3D representation of a live scene based on captured visual data of the live scene, using, for example, a 3D rendering algorithm or application 111, and to obtain primitive data of the 3D representation of the live scene based on the 3D representation of the live scene. The 3D representation of the live scene can be stored in the memory 118 of the computing device 110.

[0177] In some examples, 3D rendering algorithms or applications 111 include neural radiance field (NeRF) algorithms. NeRF is a technique that uses advanced machine learning to generate a 3D representation (e.g., a continuous volumetric scene) of an object or scene from a 2D image (e.g., from a preliminary set of input views). The NeRF technique involves encoding the entire object or scene into an artificial neural network that predicts light intensity or radiance at any point in the 2D image to generate 3D views from different angles. NeRF can automatically generate highly realistic 3D objects. When used in conjunction with other techniques, NeRF can be used to compress a 3D representation of a live scene on a large scale, for example, from gigabytes to tens of megabytes. In some examples, the NeRF algorithm represents a scene using a fully connected (non-convolutional) deep network where the input is a single continuous 5D coordinate (spatial position (x, y, z) and line of sight direction (θ, φ)) and the output is the volumetric density and field-of-view dependent radiance at that spatial position. The NeRF algorithm synthesizes views by querying 5D coordinates along camera rays and can project output color and density onto an image using volumetric rendering techniques. Since volumetric rendering is naturally differentiable, the input for optimizing the 3D representation can be a set of images with known camera poses. NeRF has several advantages over photogrammetry. It can create more realistic 3D models and can do so with fewer images. NeRF is also more efficient because it can generate new views of a 3D scene in real time.

[0178] In some implementations, the scene acquisition system 102 is configured to transmit sequential visual data of a live scene over a certain period of time to a computing device 110. The sequential visual data may include first visual data and second visual data following the first visual data. The computing device 110 may be configured to generate a first 3D representation of the live scene based on the first visual data of the live scene using a 3D rendering algorithm, and to generate a second 3D representation of the live scene by updating the first 3D representation of the live scene based on the difference between the first and second visual data using a 3D rendering algorithm, which can improve the rendering speed of the 3D representation of the live scene.

[0179] In some implementations, the computing device 110 obtains primitive data of primitives of a 3D representation of a live scene from a 3D rendering algorithm, which can be used to generate a digital hologram. The primitive data may include density, color, and / or texture.

[0180] In some implementations, the computing device 110 loads a 3D rendering algorithm or a 3D representation of a live scene from application 111 into a 3D simulation application 112, and retrieves primitive data of the 3D representation of the live scene based on the output associated with the 3D representation of the live scene in the 3D simulation application. The 3D software application 112 can be, for example, 3ds Max®, SOLIDWORKS®, Maya®, or Unity. In some implementations, one application 112 or multiple applications 112 (operating in parallel) are configured to render a 3D representation of a live scene to obtain application scene data or graphics abstractions. In some cases, application scene data is obtained by processing graphics abstractions. The application scene data can be stored in the memory 118 of the computing device 110. In some implementations, the application scene data is provided directly to a processing device 120 for further processing. In some implementations, the application scene data can also be provided to an application programming interface (API) 114 for further processing.

[0181] 2.1 Application Programming Interface (API) An API is a type of software interface. An API can specify an interface between a software application and another software application or computer program, operating system, computer hardware, or external device. In system 100, API 114 is configured to be a holographic API that enables a developer or user to interact with a holographic device, such as a processing device 120, using a computing device 110. The holographic API can convert computer graphic primitives into discrete holographic scene primitives, enabling the generation of rich holographic content using general-purpose and specially designed holographic computing hardware.

[0182] In some implementations, API 114 specifies an interface between application 112 and the computer hardware of computing device 110 (e.g., memory 118), for example via driver 116. Driver 116 may include machine-readable or executable programming instructions or software. Driver 116 is configured to communicate between API 114 and memory 118, for example, to store data (such as tables and commands) from API 114 in memory 118, or to retrieve data from memory 118 to API 114.

[0183] API 114 can retrieve application scene data from application 112. In some examples, application scene data includes data for multiple primitives corresponding to one or more objects in the scene. In some examples, API 114 processes the application scene data to retrieve data for multiple primitives. Multiple primitives can be indexed in a specific order. A primitive can include at least one of a point primitive, a line primitive, or a polygon primitive (e.g., a triangle primitive). The primitive data can include primitive data for each primitive of a group of primitives. A primitive has at least one vertex, and the primitive data of a primitive can include vertex data for at least one vertex. For example, a triangle primitive has three vertices connected to each other.

[0184] In some examples, primitive data for a primitive includes at least one of the primitive's coordinate information in a 3D coordinate system, the primitive's color information (e.g., texture color, gradient color, or both), the primitive's texture coordinate information, viewpoint-dependent shading information associated with the primitive (e.g., geometric specular reflection information), shading information associated with the primitive, or occlusion information associated with the primitive. Primitive data may also include the primitive identifier of a primitive among multiple primitives, and / or at least one vertex identifier for at least one vertex.

[0185] In some examples, the vertex data of a vertex includes at least one of the following: the vertex's coordinate information in a 3D coordinate system, the vertex's associated color information (e.g., texture color, gradient color, or both), the vertex's associated texture coordinate information, the vertex's associated viewpoint-dependent shading information (e.g., geometric specular reflection information), the vertex's associated shading information, or the vertex's associated occlusion information. The vertex data may also include the vertex's vertex identifier.

[0186] In some implementations, API 114 can adjust the vertex data of multiple primitives associated with an object or holographic scene in response to, for example, user input, trigger signals or commands, or the reception of a predetermined command. Based on the result of the adjustment, API 114 can update the vertex data of the vertices in memory 118 for further processing. For example, the gap between adjacent primitives can be adjusted to avoid kissing or overlapping, or to create an overlapping effect by adjusting the vertex coordinate information.

[0187] API 114 may be configured to process primitive data of multiple primitives and / or vertex data of multiple vertices (or vertexes) of multiple primitives to obtain data that can be processed by processing device 120, which includes, but is not limited to, generating a table showing vertex information for each primitive, organizing vertices for parallel processing, and / or generating commands for processing device 120 to draw the primitives.

[0188] In some implementations, API 114 is configured to associate a unique vertex identifier for each of the multiple vertices of a multiple primitive with its unique vertex data, and to store the association between the unique vertex identifier and the unique vertex data in memory 118, for example, along with the unique vertex data. API 114 can determine the unique vertex identifier for each of the multiple vertices based on the order of the multiple vertices in the vertex stream corresponding to the multiple primitives.

[0189] API 114 can store associations about vertices in a table in memory 118. The table may show information associated with a vertex, including but not limited to vertex identifiers (numbers), 3D coordinates (x, y, z), color information, texture mapping information, occlusion information, shading information, and / or viewpoint-dependent shading information.

[0190] In some implementations, API 114 is configured to associate each of multiple primitives with a unique primitive identifier for that primitive, with each of the vertex identifiers (and optionally the unique primitive data for that primitive) of one or more vertices of the primitive in memory 118, and to store in memory 118 the association between the unique primitive identifier and each of the vertex identifiers (and optionally the unique primitive data for that primitive). API 114 can determine the unique primitive identifier for multiple primitives based on the order of the multiple primitives in the primitive stream corresponding to the scene.

[0191] API 114 can store associations about vertices in a table in memory 118. The table may include, but is not limited to, primitive identifiers (numbers), vertex identifiers of the primitive's vertices, color information (PC), texture mapping information (PT), occlusion information (PO), shading information (PS), and / or viewpoint-dependent shading information (PVDS).

[0192] In some implementations, API 114 generates commands to be sent to processing device 120. These commands can be generated, for example, based on instructions from an application 112 or processor on computing device 110. The instructions can dictate the reconstruction of a holographic scene containing one or more objects. For example, a command may include instructions for drawing a set of primitives associated with one or more objects.

[0193] In some cases, API 114 can determine primitive identifiers for a set of primitives associated with a command instruction and send a command containing the command instruction and primitive identifiers to processing device 120 for further processing. In some cases, API 114 can determine vertex identifiers associated with primitive identifiers and send a command containing a command instruction containing vertex identifiers (and optionally primitive identifiers) to processing device 120 for further processing. The command can instruct processing device 120 to draw a set of primitives based on primitive identifiers, vertex identifiers, or a combination thereof. In some examples, API 114 generates an index table showing the association between each primitive identifier and its associated vertex identifier. API 114 can also generate a vertex table listing the vertex identifiers associated with a command, along with the vertex data associated with the vertex identifiers, optionally. The index table and vertex table for the command can be stored in memory 118. In some cases, the index table and vertex table are stored in a buffer 119 (or cache) in memory 118. Buffer 119 can be a ring buffer.

[0194] In some examples, API 114 sends a command list to processing device 120. The command list can contain several DRAW commands. Each DRAW command specifies a vertex (e.g., vertex_ptr) and index (e.g., index_ptr) associated with a sequence of command instructions (e.g., DRAW type and count). vertex_ptr can be a vertex identifier associated with the Draw command, and index_ptr can be an association between each primitive and its vertex identifier. In this way, the size of the DRAW commands can be made very small, and the DRAW commands can be sent to processing device 120 for fast processing. The DRAW commands can be cached in buffer 119 in memory 118 and then sent to processing device 120 via interface 115. Sending DRAW commands (or command lists) can be far more efficient and faster than sending primitive data or vertex data for multiple primitives corresponding to a holographic scene (or one or more objects) from API 114 to processing device 120. Furthermore, vertex_ptr and index_ptr contain information on multiple primitives within the same DRAW command, which allows the processing device 120 to perform parallel processing and improve the computation speed.

[0195] 2.2 Processing Devices The processing device 120 communicates with the computing device 110 and is configured to generate a hologram corresponding to the reconstructed holographic scene based on data transmitted from the computing device 110. The holographic scene includes one or more objects (e.g., 2D or 3D) in a 3D coordinate system. The data may include information about primitives corresponding to one or more objects. The hologram corresponds to the electromagnetic (EM) contribution from the primitives to the display elements (or feszels) of the display 150. The processing device 120 may be called a feszel processing unit (PPU). The processing device 120 is configured to compute the EM contributions at high speed, for example by parallel processing, simplified formulas, and / or any other techniques described in more detail below.

[0196] The processing device 120 may include at least one of an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable gate array (PGA), a central processing unit (CPU), a graphics processing unit (GPU), a PPU, or a standard computing unit. In some implementations, the processing device 120 communicates with the computing device 110 via Peripheral Interconnect Express (PCIe). Interface 115 may be a PCIe slot on the computing device 110. The processing device 120 may be an integrated chip insertable into the PCIe slot of the computing device 110. In some implementations, the processing device 120 is configured to be integrated with a driver device 131, optionally a display 150, and / or an illuminator 140 in an external package of the computing device 110. The processing device 120 may communicate with the computing device 110 via a wired or wireless connection, such as a USB-C connection or any other high-speed serial connection. In some implementations, API 114 (and optionally driver 116) can be implemented on the processing device 120.

[0197] In some implementations, the processing device 120 includes a command processor 122 that can be coupled in series, a local memory 123, a plurality of computing units 124, and at least one accumulator 126. The plurality of computing units 124 can be coupled in parallel, for example, for parallel processing. In some implementations, groups of computing units 124 are coupled in series, and multiple groups are coupled in parallel with each other. In some implementations, the processing device 120 includes a plurality of accumulators 126 that can be coupled in parallel, and each accumulator 126 is coupled to a separate group of computing units 124.

[0198] The command processor 122 is configured to communicate with the computing device 110 (for example, a buffer 119 in memory 118) and to receive commands from the computing device 110. A command may include information about multiple primitives corresponding to at least one object. This information may include primitive identifiers for the multiple primitives, vertex identifiers associated with the primitive identifiers, and an index table showing the associations between primitive identifiers and vertex identifiers. A command may include instructions for drawing the multiple primitives based on the information. As mentioned above, a command does not have to include primitive data for the multiple primitives and vertex data for the vertices of the multiple primitives.

[0199] The command processor 122 is configured to process commands to identify primitive identifiers or vertex identifiers, and to retrieve primitive data for multiple primitives from the computing device 110 based on the primitive identifiers or vertex identifiers. For example, the command processor 122 retrieves primitive data from memory 118 based on the primitive identifier in the command. In some implementations, the command processor 122 retrieves vertex data for a vertex based on the vertex identifier in the command.

[0200] In some implementations, an object is represented by multiple primitives. Drawing an object can be performed by a series of commands, each command associated with a separate small group of primitives. In this way, the size of each command can be reduced. The transmission speed when sending commands, extracted primitive data, and / or extracted vertex data from the computing device 110 to the command processor 122 can be increased. The processing speed of the processing device 120 can also be increased.

[0201] In some implementations, as described above, the API 114 in the host device 110 can decode an object to retrieve vertex information of vertices associated with the object and / or primitive information of primitives associated with the object, which can be stored in memory 118 (e.g., in buffer 119). Before sending a command to the processing device 120, the API 114 can first send the vertex information of vertices and / or primitive information of primitives (stored in buffer 119, for example) to the processing device 120. The vertex information of vertices and / or primitive information of primitives can be stored in local memory 123. After API 114 sends a command to command processor 122 in processing device 120, command processor 122 retrieves the corresponding index information (e.g., index table 422 in Figure 4B) from memory 118 in host device 110 and the corresponding vertex information (e.g., in table 400 in Figure 4A or vertex table 424 in Figure 4B) from local memory 123 in processing device 120, and can execute a command based on the retrieved corresponding index information and vertex information. In this way, API 114 only needs to send index information and not vertex information from host device 110 to processing device 120, thus reducing the data on interface 115 and improving transmission speed.

[0202] Unlike conventional three-dimensional graphics systems that acquire a three-dimensional scene and render it on a two-dimensional display device, system 100 is configured to generate a three-dimensional output, such as a holographic reconstruction of a form of light field, e.g., a three-dimensional volume of light. In a hologram, each display element can contribute to any part of the holographic reconstruction of the scene. Thus, each display element can potentially be modulated for any part of the scene, e.g., for each primitive in the list of primitives generated by application 112, for a complete holographic replica of the scene. In some implementations, the modulation of a particular element can be omitted or simplified, for example, based on an acceptable level of accuracy in the replicated scene or in certain areas of the scene, or on occlusion.

[0203] In some implementations, the processing device 120 is configured to calculate the EM field contribution from each primitive to each display element, e.g., phase, amplitude, or both, and for each display element, generate the sum of the EM field contributions to the display element from the list of primitives. This can be done by iterating through all primitives and obtaining their contributions to a given display element, or by iterating through each display element for each primitive, or by a hybrid blend of these two techniques.

[0204] The processing device 120 can calculate the contribution of the EM field from each primitive to each display element based on a predetermined equation for the primitive. Different primitives may have corresponding equations. In some cases, the predetermined expression is an analytical equation. In some cases, the predetermined equation is determined by solving Maxwell's equations with respect to boundary conditions defined in the display 150. The boundary conditions may include Dirichlet boundary conditions or Cauchy boundary conditions. The display elements can then be modulated based on the sum of the EM field contributions by, for example, modulating at least one of the refractive index, amplitude exponent, birefringence, or retardance of the display elements.

[0205] If the values ​​of the EM field at each point on the surface that defines the field, for example, the solutions to Maxwell's equations, are known, then the exact and unique configuration of the EM field within the volume bounded by the boundary surface can be determined. The list of primitives (or the corresponding holographic reconstruction of the hologram) and the display 150 define a 3D space, and the surface of the display 150 forms a portion of the boundary surface of the 3D space. By setting EM field states (e.g., phase or amplitude, or phase and amplitude states) on the surface of the display 150, for example, by irradiating the display surface with light, the boundary conditions of the EM field can be determined. Due to the time symmetry of Maxwell's equations, the display elements are modulated based on the contribution of the EM field from the primitives corresponding to the hologram, so that the volume light field corresponding to the hologram can be obtained as a holographic reconstruction.

[0206] For example, a line primitive of a specific color of illumination can be set in front of display 150. The analytical formula for the linear aperture can be written as a function in space. Then, the contribution of the EM field from the line primitive on the interface containing display 150 can be determined. Due to the time symmetry of Maxwell's equations, if the EM field value corresponding to the calculated EM field contribution is set on display 150, the same linear aperture used in the calculation can appear in a specific color at the corresponding location, for example, at the coordinate position of the linear primitive in a 3D coordinate system.

[0207] In some examples, we assume there is a line of light between two points A and B in 3D space. The light is uniform and has an intensity I for every l of the line distance. At each local minimum dl along the line from A to B, an amount of light proportional to I*dl is emitted. The local minimum dl acts as a delta (point) source, and from the local minimum dl we can determine the contribution of the EM field to any point on the surrounding interface of the scene corresponding to a list of primitives. Thus, for any display element of display 150, we can determine an analytical equation that represents the contribution of the EM field from the local minimum segment of the line in the display element. A special addition / integral that proceeds along the line and accumulates the contribution of the EM field of the entire line to the EM field in the display element of the display can be determined as an equation. The values ​​corresponding to the equation can be set in the display element, for example, by modulating the display element and illuminating the display element. Then, with time reversal and correction constants, we can create a line at the same position defined by points A and B in 3D space.

[0208] Referring to Figure 1A, after the command processor 122 acquires primitive data or vertex data for multiple primitives, the command processor 122 transmits the primitive data or vertex data to multiple computing units 124 for further processing. Each computing unit 124 is configured to determine the contribution of the primitive's EM field to each of the multiple display elements of the display 150 in a 3D coordinate system, based on the primitive data or vertex data of the multiple primitives. Multiple computing units 124 can be operated in parallel. The accumulator 126 is configured to accumulate the EM field contributions of the multiple primitives to each of the multiple display elements from the multiple computing units 124 and generate a separate sum of the EM field contributions of the multiple primitives to each of the multiple display elements. The accumulator 126 can generate a hologram containing the separate sum of the EM field contributions of the multiple primitives for each of the multiple display elements.

[0209] The 3D coordinate system can be, for example, a Cartesian coordinate system (XYZ), a polar coordinate system, a cylindrical coordinate system, or a spherical coordinate system. Multiple display elements within the display 150 may also have corresponding coordinate information in the 3D coordinate system. Primitives at coordinate positions can represent, for example, 3D objects adjacent to multiple display elements, in front of a display element, behind a display element, or across display elements.

[0210] In some implementations, the computing unit 124 is configured to determine at least one distance between the display elements and primitives of the display 150 based on the coordinate information of the display elements and the coordinate information of the primitives, and to determine the contribution of the primitives' EM field to the display elements based on a predetermined formula for the primitives and at least one distance. The predetermined formula can be determined by analytically calculating the propagation of the EM field from the primitives to the display elements, by a solution to Maxwell's equations under boundary conditions defined by the display, or by at least one of at least one functions from a group of functions comprising sine, cosine, and exponential functions, and determining the contribution of the EM field includes identifying the value of at least one function in a table stored in memory.

[0211] In some implementations, adjacent first and second primitives have at least one shared vertex. The computing unit 124 can determine the contribution of the first primitive's first EM field to the display elements of the display 150 based on the primitive data of the first primitive, and can determine the contribution of the second primitive's second EM field to the display elements of the display based on the contribution of the first EM field and the primitive data of the second primitive, for example, by the distance between the coordinates of the first and second primitives.

[0212] In some implementations, the computing unit 124 (or Fazel processing unit) is configured to determine the first EM contribution of a primitive to a first display element of the display 150, and, based on the first EM contribution, to determine the second EM contribution of a primitive to a second display element of the display adjacent to the first display element.

[0213] The computing unit 124 can determine in parallel the contribution of multiple primitives' EM fields to the display elements of the display 150. In some implementations, the computing unit 124 is configured to determine the contribution of the first primitive to the first display element in parallel with determining the contribution of the second primitive to the first display element. In some implementations, the computing unit 124 is configured to determine the contribution of the first primitive to the first display element in parallel with determining the contribution of the second primitive to the second display element. In some implementations, the computing unit 124 is configured to determine the contribution of the first primitive to the first display element in parallel with determining the contribution of the first primitive to the second display element.

[0214] In some implementations, the computing unit 124 is configured to determine the contribution of each first EM field from a first primitive of the plurality of primitives to each display element of the plurality of display elements, and in parallel to this, to determine the contribution of each second EM field from a second primitive of the plurality of primitives to each display element of the plurality of display elements. The accumulator 126 may be configured to accumulate the EM field contributions to each display element of the plurality of display elements by adding the contributions of each first EM field and each second EM field corresponding to the display elements.

[0215] In some implementations, the processing device 120 is configured to obtain the sum of the EM field contributions to the multiple display elements of the display 150 by a pipeline process that determines the contribution of each of the multiple primitives' EM fields to each of the multiple display elements and generates the sum of the EM field contributions from the multiple primitives to each of the multiple display elements.

[0216] The processing device 120 can perform pipeline calculations using the computing unit 124. As described above, the processing device 120 obtains the sum of the EM field contributions from multiple primitives to each display element through a series of steps. For example, calculating a given formula can be divided into multiple steps. Each row can represent a set of steps for a corresponding display element. Steps for multiple display elements can be executed in parallel. After the first step for the first display element is completed, the second step for the first display element is executed, while the first step for the second display element is executed (for example, based on the result of the first step for the first display element). Then, after the second step for the first display element is completed and the first step for the second display element is also completed, the third step for the first display element is executed (for example, based on the result of the second step for the first display element and the result of the first step for the second display element), and the second step for the second display element may also be executed. On the other hand, a first step for a third display element may be performed (for example, based on the result of the first step for a second display element and / or the result of the first step for a first display element). In this way, the processing device 120 enables the parallel execution of steps for multiple display elements according to pipeline calculations, except for a latency period at the start of the calculation. In some implementations, to improve calculation speed and / or accuracy, the processing device 120 calculates one or more mathematical functions using fixed-point representation, updated floating-point representation, or a combination thereof.

[0217] In some implementations, the processing device 120 calculates the contribution of each primitive of a plurality of primitives to each of a plurality of display elements of a plurality of display elements. The calculation of each EM field contribution can be performed without extending the object's geometry to multiple display elements, applying visibility tests before packing wavefronts, or making decisions or communicating between parallel calculations of different primitives of a plurality of primitives. The calculation of each EM field contribution may be configured to cause at least one of the following: optimizing parallel calculations of multiple primitives for speed, cost, size, or energy; reducing latency from the start of drawing to the ready to display the results; increasing precision by using fixed-point representations; skipping unpacking and repacking of floating-point representations between mathematical operations; or optimizing mathematical functions to optimize computation speed.

[0218] After acquiring primitive data or vertex data associated with multiple primitives, the processing device 120 may be configured to adjust the primitive data or vertex data of at least one of the multiple primitives according to the corresponding settings, and to calculate the contribution of the EM field associated with at least one of the multiple primitives based on the adjusted primitive data or vertex data.

[0219] In some examples, the processing device 120 adjusts the primitive data or vertex data of at least one of adjacent primitives to create a gap between adjacent primitives so that no shared vertices exist between them. The gap may be identical to or larger than a predetermined diffraction limit of the display. Based on the adjusted vertex data associated with at least one of the adjacent primitives, the processing device 120 can determine the EM field contribution of at least one of the adjacent primitives.

[0220] After obtaining the sum of the EM field contributions to multiple elements of the display 150, the processing device 120 (e.g., accumulator 126) can generate a hologram based on the sum of the EM field contributions. In some examples, the hologram is a complex value hologram. The processing device 120 can further convert the complex value hologram into an amplitude-only hologram or a phase-only hologram. The processing device 120 can also transmit the complex value hologram to a driving device 131, which can convert the complex value hologram into a phase-only hologram or an amplitude-only hologram.

[0221] In some examples, the hologram is either a phase hologram or an amplitude hologram. The processing device 120 can also transmit the phase hologram or amplitude hologram to the driving device 131. The driving device 131 can then generate corresponding control signals for modulating multiple display elements based on the hologram.

[0222] To suppress the zeroth order of display light, the zeroth order of display light can be deflected from the reconstruction cone of the holographic scene formed by the first-order diffracted light from the display. To achieve this, in some implementations, the processing device 120 can modify the holograms for multiple display elements. In some implementations, instead of the processing device 120, the driving device 131 can modify the holograms for multiple display elements after receiving the holograms from the processing device 120.

[0223] The hologram can be modified by the processing device 120 or the driving device 131, for example, by adjusting the individual phase for each of the multiple display elements. The individual phases can be adjusted, for example, by adding a corresponding phase to the individual phase for each of the multiple display elements.

[0224] In some examples, the corresponding phases for each of the multiple display elements can be expressed as follows: φ = 2π(xcosθ + ysinθ) / λ In the formula, φ represents the corresponding phase for the display element, λ represents the wavelength of light incident on the display element at a certain angle of incidence, θ represents the angle corresponding to the reversal angle of an optical reversal device (e.g., the zero-order reversal lattice structure 508-3 in Figure 5A) configured to reversal the direction of light from the display 150, and x and y represent the coordinates of the display element in a global 3D coordinate system in which the contribution of each EM field of multiple display elements to the display element is determined. Adding these corresponding phases to a hologram can have the same or similar effect as pre-configuring a hologram using a software application (e.g., Unity) in which a construction cone containing one or more corresponding virtual objects is rotated by the corresponding angle.

[0225] In some examples, the corresponding phases for each of the multiple display elements are expressed as follows:

number

[0226] In some implementations, instead of adjusting the hologram, the processing device 120 can adjust primitive data or vertex data associated with multiple primitives. The adjusted primitive data of the multiple primitives corresponds to virtual objects moved relative to the display in a global 3D coordinate system. The processing device 120 can then determine the contribution of the primitive's EM field to each of the multiple display elements of the display based on the adjusted primitive data of the primitives in the 3D coordinate system.

[0227] In some examples, coordinated primitive data of multiple primitives corresponds to a virtual object rotated at a certain angle relative to the display in a global 3D coordinate system, where the angle corresponds to the reorientation angle of an optical reorientation device (e.g., the zero-order reorientation grid structure 508-3 in Figure 5A) configured to reorient light from the display, thereby forming a holographic scene with light modulated by multiple display elements, while the zero-order display light from the display is reoriented away from the holographic scene.

[0228] In some examples, the coordinated primitive data of multiple primitives corresponds to a virtual object moved a certain distance relative to the display in a global 3D coordinate system, along a direction perpendicular to the display. The distance corresponds to the focal length of an optical divergent component configured to diverge light from the display, thereby forming a holographic scene without divergence of light modulated by multiple display elements, while the zeroth-order display light from the display is diverged and suppressed within the holographic scene.

[0229] In some implementations, the primitive data of a primitive includes the primitive's texture coordinate information. In some cases, the primitive data obtained from API 114 includes values ​​associated with the discrete cosine transform (DCT) amplitude for an image pixel mapped to a specified surface of one or more primitives, where the DCT amplitude for the image pixel is associated with the DCT weight of the image pixel. In some cases, after obtaining the primitive data, the processing device 120 can adjust the primitive data to include values ​​associated with the DCT amplitude for the image pixel. For each primitive of the multiple primitives, the processing device 120 can use the values ​​associated with the DCT amplitude for the image pixel to calculate the EM field contribution from each of the one or more primitives to each of the multiple display elements.

[0230] In some implementations, the primitive data of a primitive includes occlusion information for that primitive. In some examples, the processing device 120 can determine, based on the occlusion information of a given primitive, one or more specific display elements that do not contribute to the reconstruction of a given primitive. For each of the one or more specific display elements, the processing device 120 can generate a separate sum of the EM field contributions of multiple primitives to that specific display element by excluding the EM field contribution of the given primitive to that specific display element. In some examples, the processing device 120 may be configured to generate a sum of the EM field contributions from multiple primitives to the display element by determining, for each display element of the multiple display elements, based on the occlusion information of a given primitive, a specific portion of the given primitives that do not contribute to the EM field of the display element, and by excluding the EM field contribution to the display element from that specific portion of the given primitives.

[0231] In some implementations, the primitive data of a primitive includes viewpoint-dependent shading information about the primitive. The processing device 120 may be configured to determine the contribution of each primitive of a plurality of primitives to each of a plurality of display elements by considering the viewpoint-dependent shading information about the primitive.

[0232] In some implementations, the display 150 is a regular display in which multiple display elements have the same shape and are arranged at equal intervals in the display area. In some implementations, the display 150 is an irregular display in which multiple display elements form an irregular pattern, and for example have different shapes or sizes and / or are arranged irregularly, as shown in Figure 6A below.

[0233] For either a regular or irregular display, the processing device 120 may be configured to acquire information about the display 150, including coordinate information of multiple points corresponding to multiple display elements. For each of the multiple primitives, the processing device 120 calculates the EM propagation from the primitive to the display in a three-dimensional (3D) coordinate system based on the primitive's coordinate information and the coordinate information of the points corresponding to the display elements, thereby determining the contribution of the electromagnetic (EM) field to each of the multiple display elements of the display based on the primitive data of the display.

[0234] In some implementations, in the case of an irregular display, each of the multiple display elements has a distinct shape of multiple shapes within the display area, each of the multiple shapes uniquely encloses a distinct point of multiple points, and adjacent shapes of multiple shapes are distinct from each other. The coordinate information of the multiple points includes the coordinate information of multiple spaced points within the display area, and offset data that includes a distinct offset between each point of the multiple points and the corresponding spaced point of the multiple spaced points.

[0235] 2.3 Drive Devices The driving device 131 is coupled to the processing device 120 and configured to receive a hologram from the processing device 120 (e.g., from the accumulator 126). The hologram is for modulating the display 150 and may include, for each of the multiple display elements, a separate sum of the electromagnetic (EM) field contributions to the display element of multiple primitives corresponding to at least one object. The driving device 131 may be configured to generate a separate modulation control signal (e.g., a voltage signal) for each of the multiple display elements based on the separate sum of the EM field contributions of multiple primitives to the display element, and to output the separate modulation control signal to each of the multiple display elements in order to modulate the display element.

[0236] In some implementations, for example as shown in Figure 1A, the drive device 131 includes a drive memory 132 coupled to the processing device 120, an illuminator driver 134 coupled to the drive memory 132 and the illuminator 140, and a display driver 136 coupled to the drive memory 132 and the display 150. The drive memory 132 may include a first memory buffer coupled to the display driver 136 and a second memory buffer coupled to the illuminator driver 134. The second memory buffer may be configured to store information for controlling the amplitude and / or brightness of the light-emitting elements in the illuminator 140. The second memory buffer may have a smaller size than the first memory buffer.

[0237] The drive memory 132 is configured to receive and store holograms from the processing device 120. In some cases, the hologram is a phase-only hologram. In other cases, the hologram is a complex-value hologram. The drive device 131 can convert a complex-value hologram to a phase-only hologram and generate separate modulation control signals for multiple display elements based on the phase-only hologram. The drive device 131 can also store the phase-only hologram, along with the complex-value hologram, in the drive memory 132. A phase-only hologram for the display 150 may include individual phases for each of the multiple display elements of the display 150. The individual phases correspond to the sum of the electromagnetic (EM) field contributions to the display element from multiple primitives corresponding to at least one object.

[0238] In some implementations, the display driver 136 includes a phase converter 137 and a scan-out unit 138. The phase converter 137 is configured to convert the individual phases of each of a plurality of display elements into corresponding modulation control signals (e.g., voltage signals), and the scan-out unit 138 is configured to transmit the corresponding modulation control signals to the display elements. The phase converter 137 can sequentially and continuously convert the phases of each of the plurality of display elements, and the scan-out unit 138 can sequentially and continuously transmit the corresponding modulation control signals to the plurality of display elements as each of the corresponding modulation control signals is generated by the phase converter 137.

[0239] In some implementations, the phase converter 137 is included in the processing device 120 and coupled to the accumulator 126. The phase converter 137 can convert the complex hologram from the accumulator 126 into a phase hologram. The phase hologram can be transmitted from the processing device 120 to the drive device 130 and stored in the drive memory 132 of the drive device 130.

[0240] In some implementations, both the phase converter 137 and the drive memory 132 are located within the processing device 120, with the drive memory 132 included in the local memory 123, and the phase converter 137 coupled to the accumulator 126 and then to the local memory 123. In this way, the processing device 120 can output the phase hologram directly to the drive device 130. The drive device 130 may include an illuminator driver 134 and a display driver 136 which includes only a scan-out unit 139, which can greatly simplify the drive device 130 and allow it to be integrated with the display 150.

[0241] The display driver 136 and the display 150 can be connected using an interface, such as a low-voltage differential signal (LVDS) interface. The display driver 136 may be configured to drive digital signals to the display 150 using the interface. For example, the digital signal may be a 9-bit digital value that can be converted into 512 different voltage signals to modulate the display elements of the display 150.

[0242] In addition to the display elements, the display 150 may include digital and analog circuits. The digital circuit may be configured to receive digital data for modulating the elements and optionally perform data processing on the digital phase signals (for example, to reduce noise in the phase of the display elements). The analog circuit may include digital-to-analog converters (DACs), drivers, and row scanners. The DACs are configured to convert digital data about the display elements into analog voltage signals. The drivers are coupled to the digital circuits and are configured to drive the analog voltage signals to the corresponding display elements. The row scanners may be configured to sequentially select rows of display elements to modulate the rows of display elements using the corresponding voltage signals driven by the drivers.

[0243] The illuminator driver 134 may be configured to transmit illumination control signals (for example, to control amplitude and / or brightness) to the illuminator 140 to cause the illuminator 140 to illuminate the display such that the light, through the modulated display elements of the display 150, forms a volume light field, such as a holographic light field 160, corresponding to at least one object. The resulting volume light field corresponds to a solution to Maxwell's equations under boundary conditions defined by the modulated elements of the display 150. The display driver 136 and the illuminator driver 134 may be configured to communicate with each other such that the output of individual modulation control signals by the display driver 136 to each of the multiple display elements is coordinated with the transmission of illumination control signals by the illuminator driver 134 to the illuminator 140.

[0244] In some implementations, the illuminator 140 includes two or more light-emitting elements, each configured to emit light of a different color (e.g., red, blue, or green), and the illuminator driver 134 can sequentially transmit individual lighting control signals to sequentially activate each of the two or more light-emitting elements of the illuminator.

[0245] In some implementations, the display driver 136 sequentially outputs i) a first modulation control signal for modulating the display 150 with information associated with a first color during a first time period, and ii) a second modulation control signal for modulating the display 150 with information associated with a second color during a subsequent second time period. The illuminator driver 134 can sequentially output a first illumination control signal for operating the illuminator 140 to turn on a first light-emitting element so that it emits light of a first color during a first time period, and a second illumination control signal for operating the illuminator 140 to turn on a second light-emitting element so that it emits light of a second color during a second time period.

[0246] 2.4 Display The display 150 includes several display elements. In some implementations, the display 150 includes a spatial light modulator (SLM). The SLM may be a phase SLM, an amplitude SLM, or a phase and amplitude SLM. In some examples, the display 150 is a digital micro-mirror device (DMD), a liquid crystal on silicon (LCOS) device, an optically addressed spatial light modulator (OASLM), or any other type of light modulator capable of performing holography.

[0247] In some implementations, the display 150 is a regular display in which multiple display elements have the same shape and are arranged at equal intervals in the display area (for example, as shown in Figure 6A below). In some implementations, the display 150 is an irregular display in which multiple display elements form an irregular pattern and have, for example, different shapes or sizes and / or are arranged irregularly, as shown in Figure 6A or Figure 6B below.

[0248] The display elements of the display 150 can be modulated by modulated control signals from the display driver 136. Each modulated control signal is used to modulate individual display elements based on the sum of the EM field contributions from multiple primitives corresponding to a scene (e.g., including one or more objects) for each individual display element.

[0249] In some implementations, the number of light-emitting elements in the illuminator 140 can be less than the number of display elements in the display 150, as long as the light from the light-emitting elements can illuminate almost the entire surface of the display 150. For example, an illuminator with 64 × 64 OLEDs (organic light-emitting diodes) can be used with a display having 1024 × 1024 elements. The driving device 131 (e.g., illuminator driver 134) may be configured to operate multiple lighting elements of the illuminator 140 simultaneously.

[0250] The illuminator 140 may include one or more coherent light sources (e.g., lasers), one or more semi-coherent light sources (e.g., LEDs (light-emitting diodes) or superluminescent diodes (SLEDs)), one or more non-coherent light sources, or a combination of such light sources. In some implementations, the illuminator 140 is a monochromatic light source configured to emit substantially monochromatic light, such as red light, green light, yellow light, or blue light. In some implementations, the illuminator 140 includes two or more light-emitting elements, such as lasers or light-emitting diodes (LEDs), each configured to emit light of a different color. For example, the illuminator 140 may include red, green, and blue lighting elements.

[0251] To display 3D objects of all colors, three or more separate holograms for colors including at least red, green, and blue can be calculated by, for example, a processing device 120. That is, contributions of at least three EM fields from the corresponding primitives to the display elements can be obtained. The display elements can be sequentially modulated based on the contributions of at least three EM fields, and the illuminator 140 can be controlled to sequentially turn on at least red, green, and blue lighting elements. For example, the drive device 131 may first transmit a first timing signal to turn on the blue lighting element and then transmit a first modulation control signal corresponding to the blue hologram to the display elements of the display 150. After the blue hologram on the display 150 has been illuminated with blue light for a first time period, the drive device 131 may transmit a second timing signal to turn on the green lighting element and then transmit a second control signal corresponding to the green hologram to the display elements of the display 150. After the green hologram on display 150 is illuminated with green light for a second time period, the driver device 131 can transmit a third timing signal to turn on the red lighting element and transmit a third control signal corresponding to the red hologram to the display element of display 150. After the red hologram on display 150 is illuminated with red light for a third time period, the driver device 131 can repeat the above steps. Depending on the temporal integration effect of vision in the viewer's eye, the three colors can be combined in the eye to give the appearance of all colors. In some cases, the illuminator 140 is turned off during a state change (or holographic reconstruction) of the displayed image and turned on when a valid image (or holographic reconstruction) is presented for a certain period of time. This also allows the image (or holographic reconstruction) to be displayed stably, depending on the temporal interference of vision.

[0252] In some implementations, the display 150 has a resolution small enough to diffract visible light, for example, about 0.5 μm or less. The illuminator 140 may include a single white light source, and the emitted white light can be diffracted into different colors by the display 150 for holographic reconstruction.

[0253] As will be explained in more detail below with respect to Figures 5A to 5D, the system 100 can have different configurations. The display 150 can be reflective or transmissive. The display 150 can have various sizes ranging from small (e.g., sides of 1 to 10 cm) to large (e.g., sides of 100 to 1000 cm). The illumination from the illuminator 140 can come from the front of the display 150 (e.g., in the case of a reflective or semi-transmissive display) or from the rear of the display 150 (e.g., in the case of a transmissive display). The illuminator 140 can provide uniform illumination across the display 150. In some implementations, an optical waveguide can be used to uniformly illuminate the surface of the display 150.

[0254] In some implementations, the illuminator driver 134 and the illuminator 140 are integrated as a lighting device, and the display driver 136 and the display 150 are integrated as a display device. The drive memory 132 can be separated from the illuminator device and the display device, or the drive memory 132 can be integrated with either the illuminator device or the display device. The illuminator device and the display device can communicate with each other.

[0255] In some implementations, the processing device 120 and the drive device 131 are integrated as a controller for the display 150 and / or the illuminator 140. In some implementations, the processing device 120, the drive device 131, and the display 150 are integrated as a single unit. This single unit can also be integrated with the illuminator 140. The single units can be arranged (or tiled) in an array to form a larger device.

[0256] In some implementations, the drive device 131, the illuminator 140, and the display 150 are integrated as a single unit, for example, a holographic display device. The processing device 120 can be integrated with the computing device 110, for example, via a PCIe slot. As described above, the computing device 110 can use a 3D simulation application (e.g., application 112) to generate application scene data, which includes information on multiple primitives corresponding to at least one object, and can use API 114 to generate primitive data for multiple primitives corresponding to at least one object based on the application scene data. API 114 may be configured to adjust the initial primitive data of multiple primitives generated from the application scene data to generate primitive data for multiple primitives, vertex data for vertices associated with multiple primitives, and / or tables and commands, for example, as described in detail in Section 2.1.

[0257] In some implementations, as shown in Figures 5A to 5D, an optical diffraction device, such as a field grating device, waveguide device, or light guide device, is coupled between the illuminator 140 and the display 150, diffracting light from the illuminator 140 to the display 150, which then diffracts that light to the viewer's eyes. The optical diffraction device can be positioned adjacent to the display 150, for example, as shown in Figures 5A to 5D.

[0258] In some examples, light from the illuminator 140 can be incident on the optical diffraction device from the side at a large angle of incidence, such that the illuminator 140 does not obstruct the viewer's field of view of the display 150. In some examples, diffracted light from the optical diffraction device can be diffracted at an angle of incidence nearly perpendicular to the display, thereby allowing the light to illuminate the display 150 relatively uniformly and be diffracted to the viewer's eyes with reduced (e.g., minimal) loss.

[0259] The optical diffraction device may include a field grating structure (e.g., field grating structure 508-1 in Figure 5A) formed on a first side of the substrate (e.g., substrate 508-2 in Figure 5A) and configured to suppress color crosstalk between different colors of light incident on the display 150, and / or a zero-order direction reversal structure (e.g., zero-order direction reversal grating structure 508-3 in Figure 5A) formed on a second side opposite the substrate and configured to suppress zero-order light of the display. In some implementations, exemplary optical diffraction devices are configured and performed as described in International Application PCT / US2021 / 50271, filed September 14, 2021, entitled “DISPLAYING THREE-DIMENSIONAL OBJECTS”, and International Application PCT / US2021 / 50275, filed September 14, 2021, entitled “RECONSTRUCTING OBJECTS WITH DISPLAY ZERO ORDER LIGHT SUPPRESSION”.

[0260] In some implementations, the field grating structure includes at least one optical diffraction component and at least one of one or more color-selective polarizers, or at least one of one or more reflective layers or one or more transmissive layers. In some implementations, for example, as described in more detail in Section 4.5, the field grating structure includes at least one diffraction grating having a low diffraction efficiency for light from the illuminator 140. The diffraction efficiency can be lower than a predetermined threshold, for example, 20%, 15%, or 10%.

[0261] In some implementations, a coupling device is placed between the illuminator 140 and the optical diffraction device (e.g., a field grating structure). The illuminator 140 can illuminate with multiple different colors of light. The coupling device may be configured to receive multiple different colors of light from the illuminator 140 and output multiple different colors of light to the optical diffraction device.

[0262] In some implementations, the coupled device includes a prism element located between the illuminator 140 and the optical diffraction device and configured to receive a plurality of different colors of light from the input surface of the prism element; one or more extension gratings adjacent to the output surface of the prism element, each configured to extend the beam profiles of different corresponding colors of light by a coefficient in at least one dimension; and one or more reflectors downstream of the extension gratings, each configured to reflect individual colors of light to the optical diffraction device, the tilt angle of each of the reflectors being independently adjustable to cause uniformity of diffraction from the optical diffraction device to the display 150.

[0263] In some implementations, the optical diffraction device (e.g., a field grating structure) is positioned facing the display surface of the display 150 along a direction perpendicular to the display surface. In some implementations, the coupling device is configured to couple multiple different colors of light to the optical diffraction device (e.g., a field grating structure) from the bottom or top surface of the coupling device. In some implementations, the coupling device is configured to couple multiple different colors of light to the optical diffraction structure (e.g., a field grating structure) from the side surface of the coupling device.

[0264] In some implementations, multiple optical components can be configured between the illuminator 140 and the display 150 for dispersion compensation. For example, with respect to light incident on the display, having a spectral bandwidth with a peak wavelength, at least one first optical component (e.g., a diffraction grating) may be configured to induce positive optical dispersion, and at least one second optical component (e.g., another diffraction grating) may be configured to induce negative optical dispersion. The positive and negative optical dispersions can compensate for each other so that the holographic scene reconstructed from the light has little to no optical dispersion.

[0265] 2.5 Exemplary Holographic Reconstruction for Live Scenes Figure 1B shows an exemplary holographic reconstruction for a live scene. System 170 may include a computing device, for example, the computing device 110 of Figure 1A, which optionally has the processing device 120 of Figure 1A, and a holographic display device 172, for example, the holographic display 150 of Figure 1A, which optionally has the processing device 120 of Figure 1A and / or the driving device 131 of Figure 1A. A user may use input devices such as a keyboard 174 and / or a mouse 176 to operate System 170. The computing device or holographic display device 172 may include a holographic renderer that renders a 3D representation of a live scene (e.g., a soccer match in a stadium) to generate graphic data corresponding to the live scene, which may be implemented by the processing device 120 of Figure 1A. The graphic data may include separate primitive data for a list of primitives corresponding to the live scene.

[0266] The holographic display device 172 may include a processing device (e.g., processing device 120 in Figure 1A), a driving device (e.g., driving device 131 in Figure 1A), and a display 173 (e.g., display 150 in Figure 1A). The processing device can calculate the individual sums of the EM field contributions from primitives to each display element of the display 173 and generate control signals to modulate each display element based on the individual sums of the EM field contributions. The holographic display device 172 may further include an illuminator (e.g., illuminator 140 in Figure 1A). The driving device can generate timing control signals to activate the illuminator. When light from the illuminator illuminates the surface of the display 173, the modulated display elements can propagate the light in 3D space to form a volume light field corresponding to a live scene, for example, a holographic reconstruction 178 for a soccer match in a stadium including two players and a soccer ball. Therefore, the 3D holographic reconstructions 178 of the live scene are displayed as separate holographic reconstructions floating in 3D space in front of, behind, or across the display 173.

[0267] 3. Exemplary Systems Using Optical Techniques Figure 2 shows a schematic diagram of an exemplary system 200 for capturing and displaying a live scene using optical techniques. Unlike system 100 in Figure 1A, where visual data of the live scene is captured and processed to generate a 3D digital representation of the live scene and a corresponding computed hologram for holographic reconstruction, system 200 is configured to capture an actual optical hologram of the live scene (e.g., interference patterns associated with the live scene) using a digital sensor (e.g., CCD or CMOS) that outputs the digital hologram data to a holographic display system for holographic reconstruction in real time, for example, at an acceptable quality, without post-processing. Optionally, the digital hologram data can be processed by a computing device that becomes a holographic display system for holographic reconstruction to generate a corresponding digital hologram, which can improve the quality of the reconstructed live scene. If the processing speed is fast enough, the capture and display of the live scene can still be achieved in real time.

[0268] In some implementations, for example, as shown in Figure 2, the system 200 includes a holographic capture system 210 and a holographic display system 230. The holographic capture system 210 may include an optical system 212 configured to generate an optical hologram of a live scene, and an optical sensor 214 configured to capture a sequential optical hologram of the live scene and output sequential hologram data, where each optical hologram is associated with individual hologram data, sequential optical hologram data of the live scene associated with the sequential optical hologram. The live scene may include one or more three-dimensional (3D) objects in the real world or physical space. The holographic display system 230 may be configured to optically reconstruct the live scene in 3D space based on at least a portion of the sequential hologram data.

[0269] In some implementations, system 200 further includes a computing device 220 coupled between the holographic capture system 210 and the holographic display system 230. The computing device 220 may be configured to receive at least a portion of sequential hologram data from the optical sensor 214 and generate a digital hologram 226 associated with the live scene based on at least a portion of the sequential hologram data. The holographic display system 230 may be configured to receive the digital hologram 226 associated with the live scene from the computing device and reconstruct the live scene in 3D space based on the digital hologram 226.

[0270] In some implementations, system 200 includes one or more holographic capture systems 210, which are positioned around a live scene (e.g., a soccer match in a stadium) and configured to capture optical holograms of the live scene from one or more different views. One or more holographic capture systems 210 can be coupled to one or more computing devices 220 that can operate in parallel. Data transmission between the holographic capture systems 210 and the computing devices 220 can be via wired, wireless, or any high-speed connection. Data transmission between the computing devices 220 and the holographic display system 230 can be via wired, wireless, or any high-speed connection.

[0271] In some implementations, to achieve real-time display of a live scene, the holographic capture system 210 is configured to capture sequential optical holograms and generate sequential hologram data without storing the sequential optical holograms and the sequential hologram data. The computing device 220 can also be configured to process at least a portion of the sequential hologram data to generate digital holograms without storing at least a portion of the sequential hologram data and the digital holograms. The holographic capture system 210, the computing device 220, and the holographic display system 230 are all configured to capture an optical hologram of a live scene and optically reconstruct the live scene in real time.

[0272] The refresh rate for capturing and displaying a live scene may be limited by the refresh rate (or frame rate) of the optical sensor of the holographic capture system 210 (e.g., 90 frames per second) and the refresh rate (or frame rate) of the display of the holographic display system 230 (e.g., 30 frames per second). In the case of real-time holography, the limit is 15 frames per second, which is greater than the persistence of the visual characteristics of the human eye and brain, indicating that reconstructing a live scene at a rate exceeding 15 frames per second can make the live scene appear continuous to the human eye and brain.

[0273] 3.1 Holographic Capture System Referring to Figure 2, the holographic capture system 210 includes an optical system 212 and an optical sensor 214. The optical system 212 may include a holographic setup configured to produce an optical hologram. For example, the holographic setup may include an interferometer (e.g., a Mach-Zehnder interferometer or a Michelson interferometer), a Gabor holographic setup (e.g., inline), or a Leith-Upatnieks holographic setup (e.g., off-axis). For illustrative purposes, a Mach-Zehnder interferometer is described as an exemplary holographic setup of the optical system 212, as shown in more detail in Figures 3A to 3D, for example. The optical hologram includes interference patterns associated with the live scene, e.g., interference between an object beam interacting with the live scene and a reference beam interacting with the object beam.

[0274] The optical sensor 214 is configured to capture an optical hologram and output hologram data of the optical hologram. The optical sensor 214 can be a digital sensor, such as a digital camera like a CCD or CMOS. The optical sensor 214 may include several sensing pixels within its active area. An optical hologram, such as an interference pattern, can be formed on some of the sensing pixels. The hologram data may include data bits, such as an array of bits "0" and "1". In some examples, the array of data bits includes a one-dimensional data array (e.g., 1 × 12,000,000). In some examples, the array of data bits includes a two-dimensional data array (e.g., 3,000 × 4,000). In some implementations, the optical sensor 214 includes a buffer for buffering the hologram data. The optical sensor 214 can output sequential hologram data of a live scene sequential optical hologram, such as a stream of data bit arrays.

[0275] In some implementations, the frame rate of the optical sensor 214 is adapted to the processing speed of the computing device 220. The optical sensor 214 can directly send hologram data of an optical hologram to the computing device 200 for processing.

[0276] In some implementations, the optical sensor 214 has a high frame rate, such as 90 Hz for example, that is, the optical sensor 214 can capture 90 images (for example, optical holograms) per second. The frame rate of the optical sensor 214 may be higher than the processing speed of the computing device 220. In some implementations, the system 200 includes a frame grabber 216 coupled between the optical sensor 214 and the computing device 220 and configured to select a portion of the hologram data of the optical hologram to send to the computing device 220. For example, if the frame rate of the optical sensor 214 is 90 Hz and the computing device 220 can process 30 images per second, the frame grabber 216 can select exactly 1 / 3 of the images captured by the optical sensor 214 to send to the computing device 220. As an example, the frame grabber 216 can select the first image, the fourth image, the seventh image,..., the 88th image from among 90 images to send to the computing device 220.

[0277] The frame grabber 216 can be coupled to the optical sensor 214 by a wired connection, such as one or more transmission cables, or by a wireless connection. The frame grabber 216 can be coupled to the computing device 220 by an interface 215 (e.g., interface 115 in Figure 1A), which can be a PCIe slot, or by any other high-speed connection. In some implementations, the frame grabber 216 is included in the holographic capture system 210 (e.g., integrated with the optical sensor 214), as shown in Figure 2, for example. In some implementations, the frame grabber 216 is integrated into the computing device 220, for example, by a PCIe slot.

[0278] In some implementations, the frame grabber 216 includes a first-in, first-out (FIFO) based grabber configured to directly deposit individual hologram data output from the optical sensor 214 to the computing device 220. In some implementations, the frame grabber 216 includes a frame buffer based grabber configured to deposit hologram data output from the optical sensor 214 into the frame buffer of the frame grabber, and then to the computing device 220.

[0279] In some implementations, the frame grabber 216 includes at least one of a grabber, a frame buffer, or a controller (e.g., a Direct Memory Access (DMA) controller). The grabber is responsible for all timing and control tasks required by the optical sensor 214 and may include a set of hardware resources to coordinate the digital hologram data provided by the optical sensor 214. The frame buffer may include an internal storage area large enough to hold the full frame image issued by the optical sensor 214. The controller is configured to transfer the stored image from the frame buffer to the memory 222 of the computing device 220, for example, using a Direct Memory Access (DMA) method. This transfer does not require CPU intervention (e.g., intervention of the processor 224 of the computing device 220).

[0280] In some implementations, the optical system 212 generates optical holograms of different colors using a coherent light source (e.g., lasers of different colors) which includes coherent optical elements of different colors (e.g., red, green, blue).

[0281] In some implementations, a coherent optical element sequentially and alternately emits coherent light beams of each color, e.g., red, green, and blue (RGB). For each color, the holographic capture system 210 includes a corresponding color filter positioned upstream of the optical sensor 214 for each coherent optical element when the coherent optical element emits the corresponding coherent light beam of that color. The corresponding color filter can be a notch filter configured to transmit light of that color and block light of other colors reaching the optical sensor 214. The optical sensor 214 can capture sequential groups of optical holograms of different colors, each group containing a corresponding optical hologram of a different color (e.g., a red optical hologram, a green optical hologram, and a blue optical hologram). In some implementations, a frame grabber 216 alternately selects one hologram data from three adjacent groups of optical holograms of different colors, e.g., red, green, and blue. In this way, the processing speed of the computing device 220 can be matched with the hologram data transmitted from the frame grabber 216.

[0282] In some implementations, coherent optical elements of different colors simultaneously emit corresponding coherent light beams of their respective colors. The holographic capture system 210 may further include a color filter array located on the active area of ​​the optical sensor 214. The color filter array may include groups of different color filters on multiple sensing pixels of the optical sensor. Each color filter is associated with a different color, and each group of different color filters is located on a corresponding group of adjacent sensing pixels of the multiple sensing pixels. Holographic data for each color can be determined based on captured optical holograms captured by the corresponding groups of adjacent sensing pixels of the multiple sensing pixels.

[0283] In some implementations, the color filter array includes a Bayer filter mosaic, which is a color filter array for arranging RGB color filters on a square grid of photosensors. The Bayer filter mosaic has a specific arrangement (or filter pattern) of color filters for creating a color image. The filter pattern can be semi-green, quarter red, and quarter blue, for example, BGGR, RGBG, GRBG, or RGGB. The color image is color-coded with Bayer filter colors, and the image of individual colors can be reconstructed after interpolating missing color information. Demosaicing can be performed in various ways. In some examples, the color values ​​of neighboring pixels of the same color are interpolated. For example, when a sensing pixel is exposed to an optical hologram, each sensing pixel can be read. A sensing pixel with a green filter provides an accurate measurement of the green component. The red and blue components for this sensing pixel are obtained from neighboring pixels. In the case of a green pixel, the red value can be obtained by interpolating two red neighboring pixels, and the blue value can be obtained by interpolating two blue pixels.

[0284] In some implementations, the optical sensor 214 is configured to perform demosaicing to generate separate hologram data with different colors based on the same optical hologram captured by the optical sensor 214 with a Bayer filter mosaic. In some implementations, the computing device 220 is configured to perform demosaicing to generate separate hologram data with different colors based on the same optical hologram captured by the optical sensor 214 with a Bayer filter mosaic, and to generate digital holograms for even different colors.

[0285] 3.2 Computing Devices The computing device 220 may be a computing device associated with a user (e.g., an operator, developer, programmer, customer, or any appropriate entity). The computing device 220 may be any appropriate type of device, such as a desktop computer, personal computer, notebook, tablet computing device, personal digital assistant (PDA), network equipment, smart mobile phone, smartwatch, Enhanced General-Purpose Packet Radio Service (EGPRS) mobile phone, media player, navigation device, email device, game console, or any two or more appropriate combinations of these computing devices or other computing devices.

[0286] In some implementations, the computing device 220 includes at least one memory 222 and at least one processor 224. As described above, the memory 222 can receive and store hologram data of an optical hologram of a live scene from the holographic capture system 210 (e.g., from an optical sensor 214 or a frame grabber 216). The optical sensor 214 can be a digital sensor (e.g., a CCD or CMOS) that picks up only the intensity of the optical hologram (e.g., an interference pattern), so the hologram data of the optical hologram can be an array of data bits. The processor 224 may be configured to process the hologram data to generate a corresponding digital hologram 226 that is sent to the holographic display system 230 for holographic reconstruction of the live scene in 3D space. As will be described in more detail below, the processor 224 can perform at least one of the following mathematical operations: resolution correction, subtraction, zero padding, division, fast Fourier transform (FFT), inverse FFT, or any other operations as shown in Figures 4A-4C, for example.

[0287] In some implementations, the optical sensor 214 includes multiple sensing pixels in the active area of ​​the optical sensor, and the holographic display system 230 includes a display 250 having multiple display elements. The processor 224 is configured to process at least a portion of sequential hologram data based on at least one of the pitch of the sensing pixels, the pitch of the display elements, the size of the active area of ​​the optical sensor, or the size of the display, in order to generate a digital hologram associated with the live scene.

[0288] The pitch of the sensing pixels is related to the resolution of the captured optical hologram and the size of the scene that can be captured. On the one hand, the optical sensor 214 needs to have sensing pixels small enough to record high-resolution images. On the other hand, the optical sensor 214 also needs to be large enough to record sufficiently large scenes. In some examples, the pitch of the sensing pixels of the optical sensor 214 is approximately 3.45 μm.

[0289] The pitch of the display elements is related to the acceptable viewing angle of the reconstructed scene and the size of the display. On the other hand, the pixel pitch of the display must be small enough for the display to reconstruct 3D objects or scenes within an acceptable viewing angle. In addition to a small pixel pitch, the size of the display must be large enough to display a scene that makes sense when viewed directly. In some examples, the pitch of the display elements of Display 250 is approximately 3.74 μm.

[0290] As described above, an optical hologram (e.g., an interferogram) can be captured by the optical sensor 214 and displayed on the display 250 in real time without post-processing by the computing device 220, and the holographic reconstruction can have an acceptable quality. Higher quality of holographic reconstruction can be achieved at the level of post-processing by the computing device 220. The reason for post-processing is that the pixel sizes of the sensing pixels of the optical sensor 214 and the display elements of the display 250 are not equal. In some examples, the pitch of the sensing pixels of the optical sensor 214 is 3.45 μm, and the pitch of the display elements of the display 250 is 3.74 μm, which results in a mismatch between the recording process and the display process, for example, between the captured optical hologram of a live scene and the reconstruction of the live scene. Another mismatch occurs between the size of the display 250 and the active area of ​​the optical sensor 214. The size of the display 250 is the same as the pixel size multiplied by the number of pixels. If the active area of ​​the optical sensor 214 is larger than the size of the display 250, the live scene captured by the optical sensor 214 cannot be fully displayed on the smaller display 250. Conversely, if the active area of ​​the optical sensor 214 is smaller than the size of the display 250, the live scene captured by the optical sensor 214 will only be displayed on a portion of the display 250.

[0291] In some implementations, the mismatch between the captured optical hologram of the live scene and the reconstruction of the live scene is associated with at least one of the differences between the pitch of the sensing pixels of the optical sensor 214 and the pitch of the display elements of the display 250, or the difference between the size of the active area of ​​the optical sensor 214 and the size of the display 250. The computing device 220 (e.g., the processor 224) may be configured to suppress the mismatch by post-processing the hologram data of the optical hologram to generate a corresponding digital hologram 226.

[0292] In some implementations, computing device 220 is configured to perform at least one of scaling a first digital hologram associated with a captured optical hologram based on a ratio between a pitch of the sensing pixels and a pitch of the display elements, or adjusting the scaled first digital hologram based on the size of display 250 and the size of the hologram data to generate a second digital hologram that is modulated on display 250.

[0293] For example, pixelpitch_Display = L, pixelpitch_Sensor = C, If the ratio = L / C, post_hologram = recorded_hologram x ratio, where pixelpitch_Display and pixelpitch_Sensor represent the pitch of the display elements of display 250 and the pitch of the sensing pixels of optical sensor 214 respectively, and recorded_hologram and post_hologram represent the digital hologram before processing and the digital hologram after processing respectively. Thereafter, post_hologram can be further adjusted to be displayed on display 250.

[0294] Computing device 220 can perform the scaling using at least one of one or more interpolation algorithms including linear interpolation, nearest neighbor interpolation, cubic spline interpolation, shape-preserving interpolation, Biharmonic interpolation, and thin plate spline interpolation. For example, in linear interpolation, when a new pixel is added between two adjacent pixels, the value of the new pixel can be the average value of the two adjacent pixels.

[0295] The computing device 200 may be configured to resample a digital hologram so that the resampled digital hologram can be modulated on the display 250 at the same sampling frequency. One way to achieve this is to use the Fourier transform to find the range of spatial frequencies that constitute the digital hologram associated with the optical hologram captured by the optical sensor 214 at a specific pixel pitch, then process the frequencies (e.g., cropping or summing) and resample them on a grid equal to the pixel pitch of the display 250. In some implementations, the computing device 220 is configured to resample a first digital hologram associated with the captured optical hologram to a second digital hologram modulated on the display 250 using the Fourier transform (e.g., FFT) and inverse Fourier transform (e.g., inverse FFT). The first digital hologram is associated with the pitch of the sensed pixels, and the second digital hologram is associated with the pitch of the display elements.

[0296] Figure 4A shows an example 400 of data processing a hologram-based image. The computing device 220 receives hologram data (e.g., an array of data bits) of an optical hologram captured by the optical sensor 214 and can convert the hologram data (e.g., 1 × 12,000,000) into an image 402 with X × Y pixels (e.g., 3,000 × 4,000). The computing device 220 can perform a Fourier transform (e.g., FFT) on the image 402 in the time domain to obtain an image 404 in the frequency domain, which may still have X × Y pixels.

[0297] Zero-padding the FFT can increase the resolution of the frequency-domain result, which can be useful in narrowband applications with limited data. In some implementations, if the pitch of the sensing pixels is greater than the pitch of the display elements, the computing device 220 processes image 404 in the frequency domain to perform zero-padding on image 404 based on the ratio between the pitch of the sensing pixels and the pitch of the display elements. As shown in Figure 4A, for example, zeros 405 can be added around image 404 to obtain image 406 having M × N pixels that can match the pitch of the display elements, using the formula: post_hologram = recorded_hologram x ratio. The computing device 220 can further perform an inverse Fourier transform (e.g., inverse FFT) on the transformed image 406 with zero-padding in the frequency domain to obtain image 408 having M × N pixels in the time domain. Image 408 can be a new hologram with the correct sampling frequency. Zero-padding adds background noise but allows resampling to the correct resolution. Image 408 in the time domain can be a digital hologram (e.g., hologram 226). Since the optical sensor 214 (e.g., digital CCD or CMOS) picks up only the intensity of the optical hologram (e.g., interference pattern), the digital hologram functions like an amplitude hologram (or amplitude-like hologram), and although the digital hologram still contains both amplitude and phase information of the live scene, this allows the live scene to be reconstructed in 3D space. The display 250 (e.g., LCoS) can convert the digital hologram (e.g., amplitude-like hologram) into a phase hologram, since the orientation of the liquid crystal determines the phase of the display elements, and thus allows 3D objects in the live scene to be reconstructed for different refractive indices of different display elements.

[0298] Figure 4B shows another example 430 for data processing of a hologram-based image. Similar to example 400 in Figure 4A, the computing device 220 can receive hologram data (e.g., an array of data bits) of an optical hologram captured by the optical sensor 214 and convert the hologram data (e.g., 1 × 12,000,000) into an image 432 with X × Y pixels (e.g., 3,000 × 4,000). The computing device 220 can then perform a Fourier transform (e.g., FFT) on the image 432 in the time domain to obtain an image 434 in the frequency domain, which may still have X × Y pixels.

[0299] If the pixel pitch of the sensing pixels of the optical sensor 214 is smaller than the pixel pitch of the display elements of the display 250, the computing device 220 may be configured to crop the image 432 based on the ratio between the pitch of the sensing pixels and the pitch of the display elements, for example, using the formula: post_hologram = recorded_hologram x ratio, in order to obtain an M × N pixel image 436. Then, for example, as shown in Figure 4B, the computing device 220 may further perform an inverse Fourier transform (e.g., inverse FFT) on the transformed image 436 in the frequency domain to obtain an image 438 with M × N pixels in the time domain. Image 438 can be a new hologram with the correct sampling frequency. In this way, the information that the display 250 cannot display due to the large pixel pitch of the display elements can be removed.

[0300] The computing device 220 may be configured to resample a first digital hologram into a second digital hologram by resampling the center points of each of the multiple sensing pixels of the optical sensor 214 so as to match the geometric centers of the multiple display elements of the display 250. In some implementations, the multiple sensing pixels of the optical sensor 214 are regularly arranged in the active area of ​​the optical sensor 214, the multiple display elements are regularly arranged in the display 250, the center points of the multiple sensing pixels are regularly spaced apart, and the geometric centers of the multiple display elements of the display 250 are regularly spaced apart.

[0301] In some implementations, multiple sensing pixels of the optical sensor 214 are regularly arranged in the active area of ​​the optical sensor 214, and multiple display elements are irregularly arranged in the display 250. For example, as shown in Figures 6A to 6B, the center points of the multiple sensing pixels are regularly spaced apart, and the geometric centers of the multiple display elements of the display are irregularly spaced apart. The multiple display elements can form an irregular pattern (e.g., a Voronoi pattern). Among other effects, the irregular pattern of the display 250 can significantly reduce or eliminate diffraction aberration and thus improve image quality. The computing device 220 may be configured to resample the regularly spaced center points of the multiple sensing pixels to match the irregularly spaced geometric centers of the multiple display elements by determining the position of each geometric center of the multiple display elements based on a weighted sum of adjacent center points around the geometric center using one or more weighting algorithms (e.g., Gaussian weighting or optimized windowed-sync weighting).

[0302] In some implementations, multiple sensing pixels of an optical sensor are irregularly arranged in the active area of ​​the optical sensor 214, and multiple display elements are irregularly arranged in the display 250. The center points of the multiple sensing pixels are irregularly spaced apart, and the geometric centers of the multiple display elements of the display 250 are irregularly spaced apart. The irregular pattern formed by the multiple sensing pixels can be matched with the irregular pattern formed by the multiple display elements.

[0303] Unwanted noise may be present in the captured optical hologram. This unwanted noise may include zero-order noise arising from the undiffracted portion of the reference beam or object beam. Unwanted noise may also include noise coming from the cover plate in front of the optical sensor 214, which can add interference patterns (like a diffraction grating or Gabor zone plate) with different frequencies, sometimes low and sometimes high, depending on the flatness of the cover plate. Since these are constant parts of the optical hologram and do not change due to additional phase shifts in the optical path within the optical system 212, post-processing of the captured optical hologram can remove these types of noise.

[0304] In some implementations, as illustrated in more detail in Figure 3D, for example, a known phase shift can be introduced into one of the coherent light beams (reference beam or object beam) in the optical system 212 by adding a phase adjuster, such as a voltage-controllable phase adjuster or a dynamic retarder, to remove unwanted noise from the optical hologram captured by the optical sensor 214. The computing device 220 can then process the hologram data of the captured optical hologram using one or more mathematical operations, such as subtraction and / or division, to obtain a new digital hologram with a lower level of noise.

[0305] As described above, in conventional digital holography, the digital intensity image of the hologram is first processed, such as by complex wave retrieval, and then the hologram is numerically propagated by the Fresnel approximation, which describes the propagation of waves to the image plane in free space. This results in a complex image containing both amplitude and phase information. In contrast, the computing device 220 can generate a digital hologram (e.g., an amplitude-like hologram) with the simpler processing described above in order to improve processing speed and achieve holographic reconstruction or display in real time.

[0306] 3.3 Holographic Display Systems The holographic display system 230 may be similar to the holographic display system 130 in Figure 1A. As shown in Figure 2, the holographic display system 230 includes a driving device 231, an illuminator 240 (e.g., illuminator 140 in Figure 1A), and a display 250 (e.g., display 150 in Figure 1A or display device 172 in Figure 1B). The driving device 231 may include a driving memory 232 (e.g., driving memory 132 in Figure 1A) coupled to the computing device 220, an illuminator driver 234 (e.g., illuminator driver 134 in Figure 1A) coupled to the driving memory 232 and the illuminator 240, and a display driver 236 (e.g., display driver 136 in Figure 1A) coupled to the driving memory 232 and the display 250.

[0307] The drive memory 232 may be configured to receive and store the hologram 226 from the computing device 220. In some cases, the hologram is an amplitude-like digital hologram corresponding to an interference pattern associated with the live scene.

[0308] In some implementations, the display driver 236 is configured to convert the hologram 226 into corresponding modulation control signals (e.g., voltage signals) for multiple display elements of the display 250. The display 250 (e.g., LCoS) can convert the hologram 226 (e.g., amplitude-like hologram) into a phase hologram, since the orientation of the liquid crystals determines the phase of the display elements, thus allowing 3D objects in the live scene to be reconstructed for the different refractive indices of different display elements. The illuminator driver 234 may be configured to send illumination control signals (e.g., to control amplitude and / or brightness) to the illuminator 240 to act on the illuminator 240 to illuminate the display 250 so that the light forms a holographic light field 260 (e.g., holographic light field 160 in Figure 1A or 3D holographic reconstruction 178 in Figure 1B) corresponding to the live scene by the modulated display elements of the display 250. The display driver 236 and the illuminator driver 234 may be configured to communicate with each other so that the display driver 236 outputs individual modulation control signals to each of the multiple display elements in coordination with the illuminator driver 234 transmitting illumination control signals to the illuminator 240.

[0309] In some implementations, the illuminator 240 includes two or more light-emitting elements, each configured to emit light of a different color (e.g., red, blue, or green), and the illuminator driver 234 can sequentially transmit individual illumination control signals to sequentially activate each of the two or more light-emitting elements of the illuminator. In some implementations, the display driver 236 sequentially outputs i) a first modulation control signal to modulate the display 250 with information associated with a first color during a first time period, and ii) a second modulation control signal to modulate the display 250 with information associated with a second color during a subsequent second time period. The illuminator driver 234 can sequentially output a first illumination control signal to activate the illuminator 240 to turn on a first light-emitting element to emit light of a first color during a first time period, and a second illumination control signal to activate the illuminator 240 to turn on a second light-emitting element to emit light of a second color during a second time period.

[0310] In some implementations, the display 250 includes a spatial light modulator (SLM). The SLM may be a phase SLM, an amplitude SLM, or a phase and amplitude SLM. In some examples, the display 250 is a digital micromirror device (DMD), a liquid crystal on silicon (LCOS or LCoS) device, optically addressable spatial light modulators (OASLMs), or any other type of optical modulator capable of performing holography.

[0311] In some implementations, the display 250 is a regular display in which multiple display elements have the same shape and are arranged at equal intervals in the display area. In some implementations, the display 250 is an irregular display in which multiple display elements form an irregular pattern, for example, as shown in Figures 6A and 6B below, and have different shapes or sizes and / or are arranged irregularly.

[0312] As will be explained in more detail below with respect to Figures 5A to 5D, the system 200 can have different configurations. The display 250 can be reflective or transmissive. The display 250 can have various sizes ranging from small (e.g., sides of 1 to 10 cm) to large (e.g., sides of 100 to 1000 cm). The illumination from the illuminator 240 can come from the front of the display 250 (e.g., in the case of a reflective or semi-transmissive display) or from the rear of the display 250 (e.g., in the case of a transmissive display). The illuminator 240 can provide uniform illumination across the display 250. In some implementations, an optical waveguide can be used to uniformly illuminate the surface of the display 250.

[0313] 4. Exemplary Holographic Capture System Figure 3A shows a schematic diagram of an exemplary holographic capture system 300, which can be implemented as the holographic capture system 210 in Figure 2. The holographic capture system 300 is configured to generate an optical hologram of a live scene using a holographic setup 301 and to capture the optical hologram using an optical sensor 320. The holographic setup 301 can be implemented as the optical system 212 in Figure 2.

[0314] For illustrative purposes, as an example of a holographic setup 301, a Mach-Zehnder (MZ) interferometer is implemented in the holographic capture system 300. The MZ interferometer may be configured to determine the relative phase shift variation between two collimated beams derived by splitting light from a single coherent light source. The phase shift variation can arise from changes in the scene or the length of one of the optical paths of the collimated beams.

[0315] In some implementations, as shown in Figure 3A, the holographic capture system 300 includes a coherent light source 302 (e.g., a laser source) configured to emit a coherent light beam. The coherent light beam can be collimated using a collimator 304 (e.g., an optical lens) and then split by a beam splitter 306 into two light beams, one as the object beam 303 and the other as the reference beam 305. The holographic setup 301 may include a number of reflective mirrors 308a, 308b to guide the object beam 303 and the reference beam 305 to an optical combiner 310 (e.g., a beam splitter). The beam combiner 310 is configured to superimpose the reference beam 305 and the object beam 303 to form an interference pattern 330. An optical sensor 320 may be placed downstream of the beam combiner 310 to capture the interference pattern 330 on the active area of ​​the optical sensor 320.

[0316] The optical sensor 320 may be similar to or identical to the optical sensor 214 in Figure 2. The optical sensor 320 may be a digital optical camera (e.g., CCD or CMOS) configured to record the interference pattern 330 as an array of data bits. Since there is no need to image the scene, there is no lens on the optical sensor 320 or between the beam combiner 310 and the optical sensor 320, and the interference pattern 330 is captured directly on the active area of ​​the optical sensor 320. Also, there is no optical lens in the optical path of the reference beam 305 between the beam splitter 306 and the beam combiner 310.

[0317] In some implementations, the holographic setup 301 includes one or more optical lenses positioned between the beam splitter 306 and the scene, configured to enlarge or reduce the object beam 303 to fit the active area of ​​the optical sensor 320. In some implementations, the holographic setup 301 includes one or more optical lenses on the optical path of the object beam 303 between the scene and the beam combiner 310, configured to enlarge or reduce the object beam 303 after it has interacted with the scene to fit the active area of ​​the optical sensor 320.

[0318] In some implementations, the holographic setup 301 further includes an optical absorber 312 positioned on the side of the beam combiner 310 and configured to absorb a portion of the reference beam 305 that propagates away from the interference pattern 330.

[0319] As will be described later, the holographic setup 301 may be configured to operate in a transmission mode for capturing optical holograms of transparent objects, or in a reflection mode for capturing optical holograms of reflective objects.

[0320] Figure 3B shows a schematic diagram of an exemplary holographic capture system 300a for capturing a transparent object 340 in the optical path of an object beam 303. The holographic capture system 300a may be the same as the holographic capture system 300 in Figure 3A, and the holographic setup 301a may be the same as the holographic setup 301 in Figure 3A. The object beam 303 is incident on the beam combiner 310 through the transparent object 340. The object beam 303 interacting with the transparent object 340 interacts with the reference beam 305 to form an interference pattern 330a that is captured by the optical sensor 320.

[0321] Figure 3C shows a schematic diagram of an exemplary holographic capture system 300b for capturing an opaque or reflective object 350. Compared to the holographic capture system 300a in Figure 3B, the holographic capture system 300b includes at least one additional reflective mirror 308c in the holographic setup 301b. The reflective mirror 308c can be positioned in the optical path of the object beam 303 and can be configured together with the reflective mirror 308a to guide the object beam 303 toward the opaque object 350 at an angle such that the object beam 303 is reflected or scattered from the opaque object 350 and incident into the beam combiner 310. The object beam 303 interacting with the transparent object 340 interacts with the reference beam 305 to form an interference pattern 330b that is captured by the optical sensor 320.

[0322] Figure 3D shows a schematic diagram of an exemplary holographic capture system 300c for noise suppression. As described above, a phase adjuster (e.g., a phase shifter or dynamic retarder) can be added to the holographic capture system 300c to remove noise from the optical hologram captured by the optical sensor 320. The phase adjuster can introduce a phase shift to either the reference beam 305 or the object beam 303. For illustrative purposes, a phase adjuster 360 has been added to the optical path of the reference beam 305 in the holographic capture system 300b in Figure 3C to form the holographic capture system 300c with the holographic setup 301c.

[0323] The phase adjuster 360 may be configured to dynamically adjust the phase shift of the reference beam 305, and thus adjust the phase difference between the object beam 303 interacting with the scene and the reference beam 305 before the optical hologram (or interference pattern) 300c is formed on the optical sensor 320.

[0324] In some implementations, the phase tuner 360 is configured to sequentially adjust the phase shift of the reference beam 305 to a series of predetermined values ​​within a given time period, so that the optical sensor 320 captures corresponding sequential optical holograms of a scene over a given time period. In some examples, the phase tuner 360 includes a liquid crystal (LC) cell configured to adjust the phase shift by corresponding voltages to a series of predetermined values. The LC cell may be a single cell having a size greater than or equal to the size of the reference beam 305 so that all optical holograms can be adjusted by phase shift. The LC cell can be pre-calibrated to determine the correspondence between predetermined values ​​of the phase shift and corresponding voltages.

[0325] In some implementations, the computing device (e.g., computing device 220 in Figure 2) is configured to receive hologram data of an optical hologram captured by the optical sensor 320. The computing device may be configured to generate a noise-reduced digital hologram of the scene based on the corresponding sequential optical hologram. For example, the computing device may be configured to process the corresponding sequential optical hologram to obtain the corresponding raw digital hologram and to perform one or more mathematical operations on the corresponding raw digital hologram to generate the digital hologram of the scene.

[0326] In some implementations, the set of predetermined values ​​for the phase shift includes 0, pi / 2, pi, and 3pi / 2, and the digital hologram is calculated based on the following formula. final_hologram=(hologram_0-hologram_pi) / (hologram_pi / 2-hologram_3pi / 2), In the formula, final_hologram represents the digital hologram, hologram_0 represents the first corresponding raw digital hologram based on the first corresponding optical hologram with a phase shift of 0, hologram_pi / 2 represents the second corresponding raw digital hologram based on the second corresponding optical hologram with a phase shift of pi / 2, hologram_pi represents the third corresponding raw digital hologram based on the third corresponding optical hologram with a phase shift of pi, and hologram_3pi / 2 represents the fourth corresponding raw digital hologram based on the fourth corresponding optical hologram with a phase shift of 3pi / 2. For example, Figure 4C(a) shows the raw digital hologram 450, e.g., hologram_0 with a phase shift of 0, and Figure 4C(b) shows the processed digital hologram 460, e.g., final_hologram, calculated based on the above formula. Compared to the raw digital hologram 450, it can be seen that noise has been suppressed or removed in the processed digital hologram 460.

[0327] 5. Exemplary Holographic Display System 5.1 Exemplary System Setup A system for a 3D display may have a reflective or transmissive display with front illumination, back illumination, waveguide illumination, or optical diffraction illumination. For illustrative purposes, Figures 5A to 5D show exemplary implementations of systems for a 3D display with optical diffraction illumination. Any of these systems may correspond, for example, to system 100 in Figure 1A, system 170 in Figure 1B, or system 200 in Figure 2. Figures 5A and 5B show exemplary systems having a reflective display with optical diffraction illumination using a transmissive grid structure (Figure 5A) and a reflective grid structure (Figure 5B). Figures 5C and 5D show exemplary systems having a transmissive display with optical diffraction illumination using a reflective grid structure (Figure 5C) and a transmissive grid structure (Figure 5D).

[0328] Figure 5A shows an exemplary system 500 for a 3D display, including a reflective display 504 with optical diffraction illumination, for example, using an optical diffraction device 508. The optical diffraction device 508 can be considered a light guide (or waveguide) device for guiding light. The optical diffraction device 508 can be a transmission field grating-based structure that can include one or more transmission holographic gratings. The reflective display 504 can be the display 150 in Figure 1A, the holographic display device 172 in Figure 1B, and the display 250 in Figure 2. In some examples, the reflective display 504 is a reflective LCOS device.

[0329] The controller 502 may be configured to receive graphic data corresponding to one or more objects from the computer 501 (e.g., computing device 110 in Figure 1A or computing device 220 in Figure 2), perform calculations on the graphic data or process hologram data, and / or generate control signals for modulation, and transmit them to the display 504 via the memory buffer 503 (e.g., memory 132 in Figure 1A or memory 232 in Figure 2). The controller 502 may also be coupled to an illuminator 506 (e.g., illuminator 140 in Figure 1A or illuminator 240 in Figure 2) and configured to provide timing signals to activate the illuminator 506 to supply light. In some implementations, the controller 502 includes a processing device (e.g., processing device 120 in Figure 1A or processing device 224 in Figure 2) and a drive device (e.g., drive device 131 in Figure 1A or drive device 231 in Figure 2). In some implementations, the controller 502 includes the drive device, and the processing device is integrated with the computer 501.

[0330] Light is diffracted by the optical diffraction device 508 so as to enter the display 504, and then diffracted by the display 504 to form a holographic light field 509 corresponding to one or more objects. The display 504 may include a back mirror on its back surface to reflect light toward the viewer. The optical diffraction device 508 may be optically transparent. An illuminator 506 may be positioned below the display 504, thereby allowing the illuminator 506 to be mounted or housed together with the other components of the system 500 and positioned below the viewer's line of sight.

[0331] Bragg selectivity allows off-axis illumination light to be diffracted from the optical diffraction device 508 toward the display 504, while the reflected light diffracted from the display 504 can be close to the axis and therefore off-Bragg with respect to the grating in the optical diffraction device 508, and thus can pass through the optical diffraction device 508 almost completely without being diffracted again by the grating in the optical diffraction device 508 and reach the viewer. In some implementations, light from the illuminator 506 can be incident on the optical diffraction device 508 from the side of the display 504 at a large angle of incidence so that the illuminator 506 does not obstruct the viewer's field of view and does not enter the holographic light field 509. The angle of incidence can be a positive or negative angle with respect to the normal of the display 504. For illustrative purposes, the angle of incidence is presented as a positive angle. For example, the angle of incidence can be in the range of 70 to 90 degrees, e.g., 80 to 90 degrees. In a particular example, the angle of incidence is 84 degrees. The diffracted light from the optical diffraction device 508 can be diffracted upon near-perpendicular incidence to the display 504, thereby allowing the light to uniformly illuminate the display 504 and to diffract back to the viewer's eye through the optical diffraction device 508 near-perpendicular, minimizing power loss due to undesirable reflection, diffraction, and / or scattering inside or on the surface of the optical diffraction device 508. In some examples, the diffraction angle from the optical diffraction device 508 to the reflective display 504 can be in the range of -10° (or 10 degrees) to 10° (or 10 degrees), for example, -7° to 7°, or 5° to 7°. In a particular example, the diffraction angle is 6°. In another example, the diffraction angle is 0°.

[0332] In some implementations, as shown in Figure 5A, the optical diffraction device 508 is positioned in front of the reflective display 504, for example, along the Z-direction toward the viewer. The optical diffraction device 508 may include a field grid structure 508-1 disposed on a substrate 508-2. The back surface of the field grid structure 508-1 faces the front surface of the reflective display 504, and the front surface of the field grid structure 508-1 is attached to the substrate 508-2. Light from the illuminator 506 can be incident on the front surface of the field grid structure 508-1 via the substrate 508-2, for example, from the side surface of the substrate 508-2. For example, the substrate 508-2 may have wedge-shaped sides so that light at large incident angles can have less reflection loss.

[0333] If the diffraction efficiency of a diffraction structure, such as a holographic grating, is less than 100%, light incident at an incident angle can be diffracted by the diffraction structure to the 0th and 1st order. The 1st order light (or 1st order light) is diffracted by the diffraction structure toward the display at a certain diffraction angle and is diffracted again within the display to reconstruct the holographic light field 509. The 1st order may also be called the 1st order diffraction. Light in the 0th order (or 0th order light, or non-diffracted light, or non-diffracted order) is not diffracted (or deflected) by the diffraction structure and is transmitted by the diffraction structure at an angle corresponding to the incident angle. The 0th order light can cause undesirable effects, such as ghost images, if, for example, the 0th order light is incident directly on a reflective display 508-1 or incident following reflection from a surface within the optical diffraction device 508.

[0334] To eliminate undesirable effects, the field grid structure 508-1 can be separated from the display 504. In some implementations, the back surface of the field grid structure 508-1 is separated from the front surface of the display 504 by a certain gap. The gap can have any suitable distance, e.g., 1 mm. The gap can be filled with air or any low refractive index material to satisfy the internal total internal reflection (TIR) ​​at the interface. For example, air has a refractive index (e.g., n ≈ 1.0) that is much smaller than the refractive index of the back surface layer of the field grid structure 508-1 (e.g., n ≈ 1.5), and therefore any residual light at an incident angle (e.g., >70°) can be internally totally reflected by the back surface of the field grid structure 508-1 when the incident angle is greater than the critical angle (e.g., ≈ 41.8° for n ≈ 1.5). That is, residual light at an incident angle cannot reach the reflective display 504 and produce undesirable effects. In some examples, at least one of the front surface of the reflective display 504 or the back surface of the field grid structure 508-1 is treated with an anti-reflective coating, which can substantially reduce a portion of the holographic light field that is reflected from the reflective display 504 and returns from the back surface of the field grid structure 508-1 towards the reflective display 504, which could otherwise cause further ghosting. In some examples, the back surface of the field grid structure 508-1 can be protected by an additional layer, such as a glass layer.

[0335] In some implementations, instead of being separated by a gap, the back surface of the field lattice structure 508-1 can be attached to the front surface of the reflective display 504 using an intermediate layer. The intermediate layer can be an optically clear adhesive (OCA) layer with a refractive index substantially lower than that of the back layer of the field lattice structure 508-1, so that internal total internal reflection (TIR) ​​can occur and residual zero-order light can be totally reflected at the interface between the intermediate layer and the back layer of the field lattice structure 508-1 and returned to the optical diffraction structure 508.

[0336] In some implementations, the field grid structure 508-1 and the display 504 can be separated by a gap so that no residual light can reach the display 504. The gap can be filled with any suitable transparent material, refractive index matching fluid, or OCA. In some implementations, the field grid structure 508-1 can be formed in the cover layer (e.g., cover glass) of the display 504.

[0337] In some cases, the active area of ​​the field grid structure 508-1 may be greater than the area of ​​the entire surface of the reflective display 504 in order to illuminate the entire surface of the reflective display 504 with light diffracted from the active area of ​​the field grid structure 508-1. In some implementations, the field grid structure 508-1 and the reflective display 504 have a rectangular shape with height along the X direction and width along the Y direction. The active area of ​​the field grid structure 508-1 may have a height greater than or equal to the height of the reflective display 504 and a width greater than or equal to the width of the reflective display 504. If there is a substantial gap between the field grid structure 508-1 and the reflective display 504, the field grid structure 508-1 and the substrate 508-2 may be further extended so that the expanding cone (or frustration) of light from the reflective display 504, for example from a holographic light field 509, is visible through the front of the optical diffraction device 508 over the entire vertical and horizontal field of view (around the +Z axis) of the holographic light field 509. The substrate 508-2 can be made slightly wider and taller than the field grid structure 508-1.

[0338] When light is incident on the field grid structure 508-1 in one dimension, for example in the Z direction, at a substantially off-axis angle, the light can be narrowed by the cosine of the angle of incidence in that dimension. The light from the illuminator 506 may have a narrow rectangular shape incident on the field grid structure 508-1, which can then expand the light into a larger rectangular shape incident on the reflective display 504. For example, one or more optical components such as mirrors, prisms, optical slabs, and / or optical fillers are placed between and within the illuminator 506, the optical diffraction structure 508, and the reflective display 504 so that the light can be further expanded and its bandwidth filtered. In some examples, the expanded light may have a beam area somewhat smaller than the active area of ​​the reflective display 504, thereby making the edges and surrounding areas of the illuminated area of ​​the reflective display 504 less noticeable in reflection or scattering to the viewer. In some examples, the extended light may have a beam area somewhat larger than the active area of ​​the reflective display 504, thereby fully illuminating the edges of the illuminated area of ​​the reflective display 504 even if the edges of the extended light are not uniform (for example, due to diffraction from masking edges).

[0339] In some implementations, the illuminator 506 may include one or more color light-emitting elements, such as red, blue, or green lasers (or LEDs), configured to emit light of the corresponding color. The optical diffraction device 508 may be configured to diffract several different colors of light at diffraction angles that are substantially identical to each other. Each of the diffraction angles may be substantially identical to a range of 0° to ±10°, for example, 0°, + or -1°, + or -2°, + or -3°, + or -4°, + or -5°, + or -6°, + or -7°, + or -8°, + or -9°, or + or -10°.

[0340] In some implementations, the controller 502 is configured to sequentially modulate the display 504 with information associated with multiple colors of light during a series of time periods. For example, the information may include a series of color holograms or color images. The controller 502 can control the illuminator 506 to sequentially emit each of the multiple colors of light to the optical diffraction device 508 during individual time periods of a series of time periods, thereby diffracting each of the multiple colors of light to the reflective display 504 by the optical diffraction device 508 and diffracting by the modulated display elements of the reflective display 504 to form a three-dimensional holographic light field 509 of individual colors corresponding to objects during individual time periods. Depending on the temporal integration effect of vision in the viewer's eye, the multiple colors can be combined in the eye to give the appearance of all colors. In some cases, the illuminator 506 may be turned off between different light-emitting elements during black insertion subframes between color subframes, or during blanking or retrace periods of the video source, or during state changes (or holographic reconstruction) of the displayed image such as LC rising, falling, or DC balance inversion transitions, or during system warm-up, or when the intended holographic light field is completely black, or during calibration procedures, and turned on once a valid image (or holographic reconstruction) has been presented for a certain period of time. This also allows the image (or holographic reconstruction) to be stabilized and presented flicker-free, depending on the persistence of vision.

[0341] If a portion of the holographic light field 509 appears in front of the display 504, as shown by light field 509-1 in Figure 5A, then that portion of the holographic light field 509 is a real portion of the reconstructed image or holographic reconstruction (also called a real image or real holographic reconstruction). When a viewer sees a light point in front of the display 504, there is actually light reflected from the display 504 to that point. If a portion of the light field 509 appears to the viewer to be behind (or inside) the display 504, as shown by light field 509-2 in Figure 5A, then that portion of the holographic light field 509 is a virtual portion of the reconstructed image or holographic reconstruction (also called a virtual image or virtual holographic reconstruction). When a viewer sees a light point that appears to be behind or inside the display 504, there is actually no light diffracted from the display 504 to that virtual point; rather, it appears that some of the light diffracted from the display 504 originates from that virtual point.

[0342] Computer 501 and / or controller 502 may be configured to coordinate (e.g., by equations) the calculation of information modulated within the display 504 (e.g., a two-dimensional hologram, image, or pattern) to move the reconstructed holographic light field 509 back and forth along the normal direction (e.g., the Z direction) of the display 504. The calculation may be based on a holographic rendering process. In some cases, the holographic light field 509 may be entirely in front of the display 504. In some cases, the holographic light field 509 may appear to be entirely behind the display 504. In some cases, as shown in Figure 5A, the holographic light field may have a portion in front of the display 504, e.g., a real portion 509-1, and another portion that appears to be behind the display, e.g., a virtual portion 509-2. That is, the light field 509 may appear to straddle the surface of the display 504, and this may be called image planning.

[0343] The optical diffraction device 508 can be implemented in different configurations. In some implementations, the optical diffraction device 508 includes a holographic grating for a specific color, such as a Bragg grating, and the holographic light field 509 can correspond to a specific color. In some implementations, the optical diffraction device 508 includes multiple holographic gratings for different colors in a single recording layer.

[0344] In some implementations, the field grating structure 508-1 of the optical diffraction device 508 includes multiple holographic gratings for different colors on different recording layers. A grating for a particular color may diffract not only light of that particular color but also light of other colors, which can cause crosstalk between different colors. In some examples, the field grating structure 508-1 of the optical diffraction device 508 may include multiple holographic gratings with one or more color-selective polarizers to suppress (e.g., eliminate or minimize) color crosstalk. In some examples, the field grating structure 508-1 of the optical diffraction device 508 may include multiple holographic gratings with one or more reflective layers for light of different colors incident at each incident angle to suppress color crosstalk and zero-order diffraction. In some examples, the field grating structure 508-1 of the optical diffraction device 508 may include multiple holographic gratings with one or more color-selective polarizers and one or more reflective layers to suppress color crosstalk and zero-order diffraction. Each of the color-selective polarizers can be configured for monochromatic or multichromatic use. Each of the reflective layers can be configured for monochromatic or multichromatic applications. In some implementations, exemplary field grid structures are configured and implemented as described in International Application PCT / US2021 / 50271, filed September 14, 2021, entitled “DISPLAYING THREE-DIMENSIONAL OBJECTS,” which is jointly owned and fully incorporated herein by reference.

[0345] To improve the effect of the reconstructed holographic scene and therefore the performance of the display system, it is desirable to suppress (or further eliminate) the display zero-order light in the reconstructed holographic scene. The display zero-order light may include any unwanted light from the display, e.g., light reflected / diffracted in the gaps between display elements, reflected light from the display elements, or reflected light from the display cover on the display. In some implementations, the optical diffraction device 508 is configured to suppress the display zero-order light by at least one of zero-order light deflection, zero-order light blockage, or zero-order light redirection. In some implementations, the exemplary optical diffraction device 508 is configured and performed as described in International Application PCT / US2021 / 50275, filed on 14 September 2021, entitled “RECONSTRUCTING OBJECTS WITH DISPLAY ZERO ORDER LIGHT SUPPRESSION,” which is jointly owned and fully incorporated herein by reference.

[0346] In some examples, for zero-order light deflection, the field grating structure 508-1 of the optical diffraction device 508 may be configured to couple the input light to illuminate the display 504 at an incident angle greater than half the viewing angle of the reconstructed cone forming the holographic scene. The display zero-order light propagates away from the display 504 at the same reflection angle as the incident angle. The hologram corresponding to the holographic scene can be pre-configured so that the diffracted first-order light propagates away from the display to form the reconstructed cone as it would be at an incident angle of 0°. Thus, the display zero-order light is deflected away from the reconstructed cone and therefore away from the holographic scene.

[0347] In some examples, for zero-order light blocking, the zero-order light of the display is first deflected away from the primary light diffracted according to the zero-order light deflection, and then blocked (or absorbed) by an optical blocking component (e.g., an anisotropic optical element such as a metamaterial layer or a louver film). The optical blocking component is configured to transmit light beams with angles smaller than a given angle and block light beams with angles larger than a given angle. The given angle can be smaller than the incident angle of the input light and larger than half the field of view angle of the reconstructed cone. The optical blocking component can be formed on the side of the optical diffraction device 508 opposite to the field grating structure 508-1.

[0348] In some examples, for the reorientation of zero-order light, the zero-order light of the display can be first deflected away from the primary light diffracted according to the zero-order light deflection, and then reoriented further away from the diffracted primary light by a reorientation grating structure 508-3 within the optical diffraction device 508. If the input light contains different colors of light simultaneously or sequentially, the optical diffraction component can include one or more corresponding diffraction gratings configured to diffract the different colors of light in different directions in a plane or space to reduce color crosstalk between the different colors of light. The reorientation grating structure 508-3 can be formed on the substrate 508-2, on the side opposite to the field grating structure 508-1.

[0349] Figure 5B shows another exemplary system 510 for a 3D display, including a reflective display 514 with optical diffraction illumination, for example, using an optical diffraction device 518. The reflective display 514 may be similar to or the same as the reflective display 504 in Figure 5A. Unlike the optical diffraction device 508 in system 500 of Figure 5A, the optical diffraction device 518 in system 510 may have a reflective field grating-based structure that includes a reflective field grating structure 518-1 and a substrate 518-2. The substrate 518-2 may be a glass substrate. The reflective field grating structure 518-1 may include one or more reflective holographic gratings for one or more different colors. The reflective field grating structure 518-1 is positioned in front of the substrate 518-2, for example, along the Z direction. An illuminator 506 is positioned behind the reflective field grating structure 518-1 and is configured to illuminate the reflective field grating structure 518-1 with light at a large angle of incidence. Light is diffracted (along the -Z direction) towards the reflective display 514, which further diffracts the light back through the optical diffraction device 518 to form a holographic light field 519. The holographic light field 519 may be similar to or the same as the holographic light field 509 in Figure 5A and may include a real holographic reconstruction 519-1 (e.g., 509-1 in Figure 5A) and a virtual holographic reconstruction 519-2 (e.g., 509-2 in Figure 5A). In some implementations, the optical diffraction device 518 also includes a reorientation lattice structure for display zero-order suppression (e.g., reorientation lattice structure 508-3 in Figure 5A). For example, the direction-changing lattice structure can be formed on the side of the field lattice structure 518-1 away from the reflective display 514 by attaching the direction-changing lattice structure to the field lattice structure 518-1 using an adhesive material with a low refractive index, such that i) light diffracted by the field lattice structure 518-1 is reflected by the interface between the field lattice structure 518-1 and the adhesive material and returns to the reflective display 514, and ii) light diffracted by the reflective display 514 is transmitted through the adhesive material to the direction-changing lattice structure.

[0350] Figure 5C shows another exemplary system 520 for a 3D display, including a transmissive display 524 using, for example, an optical diffraction device 528, which is equipped with optical diffraction illumination. Similar to the optical diffraction structure 518 in Figure 5B, the optical diffraction structure 528 can be a reflective field grating-based structure that includes a reflective field grating structure 528-1 and a substrate 528-2. The substrate 528-2 can be a glass substrate. The reflective field grating structure 528-1 can include one or more reflective holographic gratings for one or more different colors. Unlike the optical diffraction structure 518 in Figure 5B, the reflective field grating structure 528-1 in the optical diffraction structure 528 is located on the back surface of the substrate 528-2. The illuminator 506 is positioned in front of the reflective field grating structure 528-1 and is configured to illuminate the reflective field grating structure 528-1 with light at a large angle of incidence. Light is diffracted (along the -Z direction) towards the transmissive display 524, which further diffracts the light to form a holographic light field 529. The holographic light field 529 may be similar to or identical to the holographic light field 509 in Figure 5A, and may include real holographic reconstruction 529-1 (e.g., 509-1 in Figure 5A) and virtual holographic reconstruction 529-2 (e.g., 509-2 in Figure 5A). In some implementations, the optical diffraction device 528 also includes a reorientation grating structure for display zero-order suppression (e.g., reorientation grating structure 508-3 in Figure 5A).

[0351] Figure 5D shows another exemplary system 530 for a 3D display, including a transmissive display 534 using, for example, an optical diffraction device 538, which is equipped with optical diffraction illumination. The transmissive display 534 may be the same as the transmissive display 524 in Figure 5C. Similar to the optical diffraction structure 508 in Figure 5A, the optical diffraction structure 538 may have a transmissive field grating-based structure that includes a transmissive field grating structure 538-1 and a substrate 538-2. The substrate 538-2 may be a glass substrate. The transmissive field grating structure 538-1 may include one or more transmissive holographic gratings for one or more different colors. Unlike the optical diffraction structure 508 in Figure 5A, the transmissive field grating structure 538-1 in the optical diffraction structure 538 is located in front of the substrate 538-2. The illuminator 506 is located behind the transmissive field grating structure 538-1 and is configured to illuminate the transmissive field grating structure 538-1 with light at a large angle of incidence. Light is diffracted (along the +Z direction) toward the transmissive display 534, which further diffracts the light to form a holographic light field 539. The holographic light field 539 may be similar to or the same as the holographic light field 509 in Figure 5A, and may include real holographic reconstruction 539-1 (e.g., holographic reconstruction 509-1 in Figure 5A) and virtual holographic reconstruction 539-2 (e.g., holographic reconstruction 509-2 in Figure 5A). In some implementations, the optical diffraction device 538 also includes a reorientation grating structure for display zero-order suppression (e.g., reorientation grating structure 508-3 in Figure 5A).

[0352] 7. Exemplary Irregular Display In a display (e.g., an LCOS device), a circuit chip, such as a complementary metal-oxide-semiconductor (CMOS) chip or equivalent, controls a voltage on a reflective metal electrode embedded beneath the chip surface, each controlling one facel (or display element). A common electrode for all facels is supplied by a transparent conductive layer made of indium tin oxide on a coverslip. In some examples, the chip may have a 1024 × 768 plate, each with an independently addressable voltage. The facels may be of the same size and shape (e.g., square). When the facel gap becomes comparable to the wavelength of incident light, diffraction effects appear in the periodic structure of the display, potentially causing significant optical loss.

[0353] In some implementations, as shown in Figure 6A, the display 600 includes a plurality of non-uniform (or irregular) fascies 602. Among other effects, the non-uniform shape of the fascies 602 can significantly reduce or eliminate diffraction aberration and thus improve image quality. The plurality of fascies 602 can form an irregular pattern. In some implementations, the irregular pattern includes a Voronoi pattern. In some implementations, the irregular pattern includes a HOLOCHROME® (or HOLOCHROME® trademark) pattern. At least two fascies 602 in the display 600 have different shapes. For example, adjacent fascies 602 may have different shapes. In some examples, at least one fascie 602 has an irregular polygonal shape.

[0354] In some cases, the gap between adjacent fazels of the display 600 is smaller than the wavelength of the incident light, which can reduce light loss between adjacent fazels. In some examples, the wavelength of the incident light is about 450 nm (for example, blue light), about 530 nm (for example, green light), or about 630 nm (for example, red light), but the gap may be about 200 nm. In some cases, the size distribution of the multiple fazels 602 of the display 600 may be the same as the spatial frequency response of the display 600, for example, around 3 μm. The size of a fazel can refer to the maximum width of the fazel.

[0355] In some implementations, each fezel 602 surrounds a corresponding spaced point 604. As shown in Figure 6A, the corresponding spaced point 604 can be at a random position within the shape of the fezel 602, for example, at the center or edge of the fezel 602. The fezel 602 can be designed and / or manufactured based on the spaced points 604. In some implementations, the base points 604 form an irregular pattern. The irregular pattern of the spaced points 604 may be the same as or different from the irregular pattern of the fezel 602.

[0356] In some implementations, the shape and / or area of ​​the fascia 602 may form an irregular pattern, while the spaced points 604 may form a regular shape, such as a square. The distance between the centers of adjacent base points 604 may be the same among the spaced points 604. For example, the spaced points 604 may be the center points of the fascia of a regular display. In this way, the display 600 can be manufactured using the same backplane as that used for a regular display. In some implementations, the spaced points 604 are regularly spaced in a first region of the display 600 with a first spacing period and in a second region with a second spacing period. The second spacing period may differ from the first spacing period. In some embodiments, the spaced points 604 correspond to conductive vias electrically connected to a fascia drive circuit.

[0357] As described herein, an irregular display (e.g., display 600 in Figure 6A) can be used as a display (display 150 in Figure 1A, display 172 in Figure 1B, display 250 in Figure 2, reflective display 504 in Figure 5A, reflective display 514 in Figure 5B, transmissive display 524 in Figure 5C, transmissive display 534 in Figure 5D). An irregular display can be used in a system for a 3D display (e.g., system 100 in Figure 1A, system 170 in Figure 1B, system 200 in Figure 2, system 500 in Figure 5A, system 510 in Figure 5B, system 520 in Figure 5C, system 530 in Figure 5D). An irregular display may be configured to reduce or eliminate diffraction effects, diffraction aberrations, aliasing, or other effects, thereby improving the quality of the reconstructed image, object, or scene.

[0358] In some implementations, the irregular displays or irregular faces described herein can also be used for capturing (or sensing) images or videos. For example, a device having an array of irregular pixels (e.g., the irregular face 602 in Figure 6A) can be implemented as a camera (e.g., the scene acquisition device 104 in Figure 1A), or an optical sensor (e.g., the optical sensor 214 in Figure 2) can be configured to capture interference patterns, images, or videos. The array of irregular pixels can form a photodiode array. The device may include one or more other components, such as a driver circuit, an optical lens, or a color filter.

[0359] Irregular pixels can form irregular patterns, such as Voronoi patterns. Devices with irregular pixels can mitigate or eliminate several potential problems. For example, aliasing is a phenomenon where digital cameras fail to properly translate complex patterns, which can occur when digital information is broken down into pixels and bits, potentially resulting in many odd visual artifacts in the image (or photograph) or video. To eliminate such aliasing problems (e.g., moiré or glitch problems / effects) in captured images or videos, a camera can be configured with irregular pixels. Additionally or alternatively, the irregular patterns of irregular pixels in a camera can eliminate undesirable regular diffraction that may appear due to regular pixels.

[0360] Figure 6B shows an example 610 of designing an irregular display 630 based on a regular display 620. The regular display 620 includes an array of fazels 622, each having a center point 624 (indicated by the "○" symbol in Figure 6B). The center point 624 can be the center point of the corresponding conductive vias coupled to the display element to be formed. The regular display 620 can have a square shape, and each fazel 622 can also have a square shape. The center point 624 can also form a square shape.

[0361] For comparison, the irregular display 630 may be similar to or identical to the irregular display 600 in Figure 6A. The irregular display 630 includes a plurality of fazels 632 (e.g., fazel 602 in Figure 6A). Each fazel 632 has individual points 634 within it (indicated by solid dots "·" in Figure 6B). The shape of the fazel 632 can be generated based on the individual points 634 according to an irregular pattern (e.g., a Voronoi pattern). The individual points 634 can be called seed points (or seed centers) of the fazel 632. In some examples, the fazel 632 has a polygonal shape, and the individual points 634 are the polygonal geometric centers of the fazel 632. In some examples, the individual points 634 are different from the polygonal geometric centers of the fazel 632.

[0362] Multiple faces 622 of a regular display 620 are the same as multiple faces 632 of an irregular display 630. Each face 632 can surround the corresponding center point 624 of the regular display 620, having an offset 636 from an individual point 634 of the face 632. The offsets 636 from the face 632 may be different from each other, for example, as shown in Figure 6B. Each point 634 of the face 632 may be irregularly arranged within the area of ​​the irregular display 630. The irregular display 630 may have a larger area than the regular display 620. The contour of the irregular display 630 may be a regular shape, for example, a rectangle or a square.

[0363] As described above, in a system for capturing and displaying a live scene holographically in real time (e.g., system 200 in Figure 2), the optical sensor for capturing the optical hologram (e.g., optical sensor 214 in Figure 2) and / or the display modulated with the hologram for holographic reconstruction (e.g., display 250 in Figure 2) may be irregular.

[0364] In some implementations, (as shown in Figures 6A-6B) multiple sensing pixels of an optical sensor are regularly arranged in the active area of ​​the optical sensor, and multiple display elements are irregularly arranged in the display. For example, as shown in Figures 6A-6B, the center points of multiple sensing pixels are regularly spaced apart, and the geometric centers of multiple display elements of the display are irregularly spaced apart. Multiple display elements can form an irregular pattern (e.g., a Voronoi pattern). Among other effects, the irregular pattern of the display can significantly reduce or eliminate diffraction aberration and thus improve image quality. The computing device may be configured to resample the regularly spaced center points of multiple sensing pixels to match the irregularly spaced geometric centers of multiple display elements by determining the position of each geometric center of multiple display elements based on a weighted sum of adjacent center points around the geometric center using one or more weighting algorithms (e.g., Gaussian weighting or optimized windowed-sync weighting).

[0365] Computing devices can sample regularly spaced center points of a regular grid to compute the irregularly spaced geometric centers of an irregular grid. For example, the phase at an irregular location can be computed using a weighted average of four adjacent regular points. Alternatively, a weighted sum of regular points around an irregular target point can be used, employing a Gaussian or optimized window-synchronous weighting algorithm.

[0366] In some implementations, the computing device is configured to perform resampling based on a predetermined relationship between a regular center point of a regular display (e.g., the center point 624 of the regular display 620 in Figure 6B) and a corresponding seed point of a random display (e.g., point 634 of the random display 630 in Figure 6C) or the geometric center of the random display.

[0367] In some implementations, multiple sensing pixels of an optical sensor are irregularly arranged within the active area of ​​the optical sensor, and multiple display elements are irregularly arranged within the display. The centers of the multiple sensing pixels are irregularly spaced apart, and the geometric centers of the multiple display elements of the display are irregularly spaced apart. The irregular pattern formed by the multiple sensing pixels can be matched with the irregular pattern formed by the multiple display elements.

[0368] 7. Illustrative Holographic Cubes As described above, real-time holography may require the reconstruction of the live scene at a rate exceeding 15 frames per second, thereby making the live scene continuous to the human eye and brain. In addition to time, real-time holography may also need to include three main holographic cues: occlusion, parallax, and accommodation. The systems disclosed herein (e.g., system 100 in Figure 1A or system 200 in Figure 2) are capable of performing real-time holography.

[0369] Figures 7A–7C show exemplary cues for the holographic representation of an object, including occlusion (Figure 7A), parallax (Figure 7B), and accommodation (Figure 7C).

[0370] As shown in Figure 7A, Figure 700, occlusion is the process by which an occluded object 702 is hidden from view or made inconspicuous by an occluder 704. In holography, if the viewer changes their field of view, the occluded object 702 may be visible from several fields of view. In contrast, a 2D screen does not have this feature, and how the viewer sees the image is irrelevant; the scene remains the same, and the blocked object remains blocked even when the field of view is changed.

[0371] Parallax is the property where each eye sees objects in a scene from a different perspective. For example, in holography, as shown in Figure 7B, the left eye sees a separate scene 710 where the cylindrical object 712 and the rectangular object 714 are separated from each other, while the right eye sees a different scene 720 where the cylindrical object 712 and the rectangular object 714 overlap. In contrast, with a 2D display, the image is the same regardless of the field of view from the left or right eye.

[0372] Accommodation is the ability of the eye to perceive the depth of objects in a scene. For example, as shown in Figure 730 of Figure 7C, the cylindrical object 732 is further from the viewer's eye than the rectangular object 734. In holography, the depth of the cylindrical object 732 is greater than that of the rectangular object 734. In contrast, with a 2D display, the viewer's eye remains focused on the surface of the 2D screen, and no depth information is displayed on the screen surface.

[0373] 8. Exemplary Process Figure 8A is a flowchart of an exemplary process 800 for holographic display of a live scene. Process 800 can be performed by a system such as system 200 in Figure 2. A live scene can include one or more three-dimensional (3D) objects. A live scene can be a scene or event occurring in the real world, real life, or physical space. The live scene can be captured and displayed by the system in real time.

[0374] In some implementations, the system includes a holographic capture system (e.g., holographic capture system 210 in Figure 2) and a holographic display system (e.g., holographic display system 230 in Figure 2). The holographic capture system may include an optical system (e.g., optical system 212 in Figure 2 or holographic setup 300 in Figure 3A, 300a in Figure 3B, 300b in Figure 3C, or 300c in Figure 3D) and optical sensors (e.g., optical sensor 214 in Figure 2, or optical sensor 320 in Figures 3A to 3D). In some implementations, the system further includes a computing device coupled between the holographic capture system and the holographic display system (e.g., computing device 220 in Figure 2).

[0375] In 802, the optical system optically generates an optical hologram of the live scene. In 804, the optical sensor captures a successive optical hologram of the live scene and generates successive hologram data associated with the successive optical hologram of the live scene. Each optical hologram can be associated with individual hologram data. In 806, the holographic display system optically reconstructs the live scene in 3D space based on at least a portion of the successive hologram data.

[0376] In some implementations, a computing device processes at least a portion of sequential hologram data to generate a digital hologram associated with a live scene (e.g., hologram 226 in Figure 2), and a holographic display system reconstructs the live scene in 3D space based on the digital hologram. Each digital hologram may include an amplitude-like hologram, and the amplitude-like program can directly modulate the display of the holographic display system (e.g., display 250 in Figure 2) for the holographic reconstruction of the live scene. The display may also be phase-modulated.

[0377] In some implementations, the optical sensor is a digital sensor (e.g., CCD or CMOS), and the sequential hologram data includes a stream of digital data. The digital data may include an array of data bits (e.g., 0s or 1s).

[0378] In some implementations, the system includes a frame grabber (e.g., frame grabber 216 in Figure 2). Process 800 further includes the frame grabber selecting individual hologram data of one or more optical holograms from a series of optical holograms. Based on the selected individual hologram data of one or more optical holograms, the computing device can generate a digital hologram associated with the live scene.

[0379] In some implementations, the frame grabber includes a frame buffer-based grabber configured to deposit individual hologram data into the frame grabber's frame buffer before transmitting it to generate a digital hologram. In some implementations, the frame grabber includes a first-in, first-out (FIFO) based grabber configured to transmit individual hologram data directly to generate a digital hologram.

[0380] In some implementations, a computing device processes at least a portion of sequential hologram data based on at least one of the following: the pitch of the sensing pixels of an optical sensor, the pitch of the display elements of a display, the size of the active area of ​​the optical sensor, or the size of the display. The pitch of the sensing pixels can be associated with the resolution of the captured optical hologram and the captureable size of the scene, and the pitch of the display elements can be associated with the acceptable viewing angle of the reconstructed scene and the size of the display.

[0381] In some implementations, computing devices suppress mismatches between the captured optical hologram of a live scene and the reconstruction of the live scene. These mismatches can be associated with at least one of the following: the difference between the pitch of the sensing pixels and the pitch of the display elements, or the difference between the size of the active area of ​​the optical sensor and the size of the display.

[0382] In some implementations, the computing device processes at least a portion of sequential hologram data to generate a digital hologram associated with a live scene by scaling a first digital hologram associated with a captured optical hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements, and / or adjusting the scaled first digital hologram based on the size of the display and the size of the hologram data to generate a second digital hologram that is modulated on the display.

[0383] In some implementations, the computing device scales the size of a first digital hologram associated with a captured optical hologram using at least one of one or more interpolation algorithms, including linear interpolation, nearest neighbor interpolation, cubic spline interpolation, shape-preserving interpolation, biharmonic interpolation, and thin-plate spline interpolation.

[0384] In some implementations, the computing device processes at least a portion of the sequential hologram data to generate a digital hologram associated with a live scene by resampling a first digital hologram associated with a captured optical hologram, which is associated with the pitch of a sensing pixel, into a second digital hologram modulated on a display, which is associated with the pitch of a display element, using Fourier transforms and inverse Fourier transforms.

[0385] In some implementations, for example as shown in Figure 4B, the computing device first resamples the first digital hologram associated with the captured optical hologram into a second digital hologram that is modulated on the display by performing a Fourier transform on the first digital hologram to generate a transformed first digital hologram. If the pitch of the sensing pixels is smaller than the pitch of the display elements, the computing device crops the transformed first digital hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements, and then performs an inverse Fourier transform on the cropped transformed first digital hologram to obtain the second digital hologram.

[0386] In some implementations, for example as shown in Figure 4A, the computing device resamples the first digital hologram associated with the captured optical hologram into a second digital hologram modulated on the display by first performing a Fourier transform on the first digital hologram to generate a converted first digital hologram; if the pitch of the sensing pixels is greater than the pitch of the display elements, it performs zero padding on the converted first digital hologram based on the ratio between the pitch of the sensing pixels and the pitch of the display elements; and then performs an inverse Fourier transform on the converted first digital hologram with the added zero padding to obtain the second digital hologram.

[0387] In some implementations, resampling a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on a display involves resampling the first digital hologram into a second digital hologram by resampling the center points of multiple sensing pixels of an optical sensor to match the geometric centers of multiple display elements of the display.

[0388] In some implementations, multiple sensing pixels of an optical sensor are regularly arranged within the active area of ​​the optical sensor, and multiple display elements are regularly arranged within the display. The center points of the multiple sensing pixels are regularly spaced apart, and the geometric centers of the multiple display elements of the display are regularly spaced apart.

[0389] In some implementations, (for example, as shown in Figure 6A or Figure 6B) multiple sensing pixels of an optical sensor are regularly arranged in the active area of ​​the optical sensor, and multiple display elements are irregularly arranged on the display. The center points of the multiple sensing pixels are regularly spaced apart, and the geometric centers of the multiple display elements of the display are irregularly spaced apart. In some implementations, resampling a first digital hologram associated with a captured optical hologram into a second digital hologram modulated on the display involves resampling the regularly spaced center points of the multiple sensing pixels to match the irregularly spaced geometric centers of the multiple display elements by determining the position of each geometric center of the multiple display elements based on a weighted sum of adjacent center points around the geometric center using one or more weighting algorithms.

[0390] In some implementations, multiple sensing pixels of an optical sensor are irregularly arranged within the active area of ​​the optical sensor, and multiple display elements are irregularly arranged within the display. The centers of the multiple sensing pixels are irregularly spaced apart, and the geometric centers of the multiple display elements of the display are irregularly spaced apart. The irregular pattern formed by the multiple sensing pixels can be matched with the irregular pattern formed by the multiple display elements.

[0391] In some implementations, the optical system forms an interference pattern by interfering an object beam interacting with the live scene (e.g., object beam 303 in Figures 3A-3D) with a reference beam (e.g., reference beam 305 in Figures 3A-3D). The object beam and reference beam are coherent light beams, and the optical hologram contains the interference pattern. The optical sensor can directly capture the interference pattern on the active area of ​​the optical sensor. The optical system can enlarge or reduce the object beam after it has interacted with the live scene to fit the active area of ​​the optical sensor.

[0392] In some implementations, process 800 further includes dynamically adjusting the phase shift of one of the object beam and the reference beam in a phase adjuster, for example, phase adjuster 360 in Figure 3D, before the interference pattern is formed. The phase adjuster may include a phase shifter or a dynamic retarder. The phase adjuster may include a liquid crystal cell. The phase adjuster may be configured to dynamically adjust the phase shift of one of the object beam and the reference beam by sequentially adjusting the phase shift to a series of predetermined values ​​over a time period. An optical sensor can capture a corresponding sequential optical hologram of the live scene over that time period.

[0393] Process 800 further includes generating a noise-suppressed digital hologram of the live scene based on the corresponding successive optical hologram, for example, by a computing device. The computing device may process the corresponding successive optical hologram to obtain the corresponding raw digital hologram and perform one or more mathematical operations on the corresponding raw digital hologram to generate the digital hologram of the live scene. In some examples, a set of predetermined values ​​for the phase shift include 0, pi / 2, pi, 3pi / 2, and the digital hologram is calculated based on the following formula: final_hologram=(hologram_0-hologram_pi) / (hologram_pi / 2-hologram_3pi / 2), In the formula, final_hologram represents a digital hologram, hologram_0 represents a first corresponding raw digital hologram based on a first corresponding optical hologram with a phase shift of 0, hologram_pi / 2 represents a second corresponding raw digital hologram based on a second corresponding optical hologram with a phase shift of pi / 2, hologram_pi represents a third corresponding raw digital hologram based on a third corresponding optical hologram with a phase shift of pi, and hologram_3pi / 2 represents a fourth corresponding raw digital hologram based on a fourth corresponding optical hologram with a phase shift of 3pi / 2.

[0394] In some implementations, optically generating an optical hologram of a live scene involves sequentially and alternately emitting light of multiple colors to sequentially and alternately generate optical holograms for multiple colors. Capturing a sequential optical hologram of a live scene may involve sequentially transmitting only light of individual colors and blocking light of other colors.

[0395] In some implementations, optically generating an optical hologram of a live scene involves simultaneously emitting light with multiple colors. Capturing a successive optical hologram of a live scene may involve capturing the optical hologram using a color filter array (e.g., Bayer filters) placed on the optical sensor, with corresponding groups of adjacent sensing pixels of multiple sensing pixels of the optical sensor. The color filter array may include groups of different color filters on multiple sensing pixels of the optical sensor, where different color filters are associated with multiple colors, and each group of different color filters is placed on a corresponding group of adjacent sensing pixels of multiple sensing pixels. Generating successive hologram data associated with a successive optical hologram of a live scene may involve determining hologram data for each of multiple colors based on the optical hologram.

[0396] In some implementations, process 800 further includes generating digital holograms for multiple colors based on optical holograms, and reconstructing a live scene in 3D space based on at least a portion of the hologram data, or reconstructing a live scene in 3D space based on digital holograms.

[0397] In some implementations, reconstructing a live scene in 3D space based on at least a portion of the hologram data involves generating control signals for multiple display elements of a display based on the digital hologram associated with the live scene, and modulating the multiple display elements of the display based on the control signals.

[0398] In some implementations, the process further includes generating a digital hologram associated with the live scene based on at least a portion of the sequential hologram data. The digital hologram may include a group of digital holograms for multiple colors.

[0399] In some implementations, reconstructing a live scene in 3D space based on at least a portion of holographic data includes sequentially modulating the display with a first digital hologram for a first color during a first time period, then modulating the display with a second digital hologram for a second color during a subsequent second time period, and sequentially lighting up a first light-emitting element to emit light in the first color during the first time period, and a second light-emitting element to emit light in the second color during a subsequent second time period.

[0400] Figure 8B is a flowchart of another exemplary process 850 for holographic display of a live scene. Process 850 can be performed by a system such as system 100 in Figure 1A. A live scene can include one or more three-dimensional (3D) objects. A live scene can be a scene or event occurring in the real world or real life, or in physical space. A live scene can be captured and displayed by the system in real time.

[0401] In some implementations, the system includes a hologram generation system (e.g., hologram generation system 101 in Figure 1A) configured to generate one or more digital holograms corresponding to a live scene, and a holographic display system (e.g., holographic display system 130 in Figure 1A) configured to reconstruct the live scene in 3D space based on one or more digital holograms. The hologram generation system may include one or more scene acquisition devices (e.g., 104 in Figure 1A) and a computing system (e.g., 103 in Figure 1A). The computing system may include a computing device coupled to one or more scene acquisition devices (e.g., computing device 110 in Figure 1A) and a processing device coupled to the computing device (e.g., processing device 120 in Figure 1A).

[0402] In step 852, visual data of the live scene is captured from one or more views by, for example, one or more scene acquisition devices. In step 854, primitive data associated with the live scene is acquired (for example, by a computing device) based on the captured visual data of the live scene. In step 856, a digital hologram (for example, hologram 127 in Figure 1A) corresponding to the live scene is generated (for example, by a processing device) based on the primitive data associated with the live scene and the display element information of the display of the holographic display system (for example, display 150 in Figure 1A). In step 858, the live scene is reconstructed in three-dimensional space by, for example, the holographic display system modulating the display with one or more digital holograms.

[0403] In some implementations, obtaining primitive data associated with a live scene based on captured visual data of the live scene includes generating a 3D representation of the live scene based on the captured visual data of the live scene, and obtaining primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, for example, by a computing device. The primitive data associated with the live scene includes primitive data of the 3D representation of the live scene. The 3D representation of the live scene can be generated by processing the captured visual data of the live scene using a 3D rendering algorithm (e.g., NeRF).

[0404] In some implementations, process 850 includes generating sequential visual data of a live scene over a certain period of time, comprising first visual data and second visual data following the first visual data (e.g., by one or more scene acquisition devices); generating a first 3D representation of the live scene based on the first visual data of the live scene using a 3D rendering algorithm; and generating a second 3D representation of the live scene by updating the first 3D representation of the live scene based on the difference between the first and second visual data using a 3D rendering algorithm.

[0405] In some implementations, obtaining primitive data of a 3D representation of a live scene based on the 3D representation of the live scene involves loading the 3D representation of the live scene into a 3D simulation application (e.g., Unity) and obtaining primitive data of the 3D representation of the live scene based on the output associated with the 3D representation of the live scene in the 3D simulation application.

[0406] In some implementations, the primitive data for the 3D representation of a live scene includes data for multiple primitives corresponding to the 3D representation of the live scene, wherein the data comprises primitive data for each of the multiple primitives, each primitive has at least one vertex, and the primitive data of the primitive comprises data for at least one vertex. The primitive data of the primitive may include at least one of the primitive identifier of the primitive, at least one vertex identifier of at least one vertex, coordinate information of the primitive in a 3D coordinate system, color information of the primitive, texture coordinate information of the primitive, shading information for the primitive, viewpoint-dependent shading information associated with the primitive, or occlusion information of the primitive.

[0407] In some implementations, generating one or more digital holograms corresponding to a live scene involves determining, for each of several primitives, the contribution of the electromagnetic (EM) field to each of several display elements of the display, based on the primitive data of the primitive; and generating, for each of the several display elements of the display, the sum of the EM field contributions of the several primitives to the display element. The digital hologram may include the sum of the EM field contributions to the several display elements of the display.

[0408] In some implementations, process 850 further includes associating a specific vertex identifier of a vertex with specific vertex data of a vertex for each of the multiple vertices of a multiple primitive, and storing the association between the specific vertex identifier and the specific vertex data of a vertex in memory; and associating a specific primitive identifier of a primitive with each of the vertices of one or more vertices of the primitive in memory, and storing the association between the specific primitive identifier and each of the vertex identifiers of the primitive in memory.

[0409] In some implementations, process 850 further includes determining primitive identifiers of multiple primitives associated with a command instruction, determining vertex identifiers associated with the primitive identifiers, and generating a command that includes the command instruction, the vertex identifiers associated with the primitive identifiers, and the primitive identifiers of the multiple primitives. The command instructs to draw the multiple primitives according to the command instruction, based on at least one of the primitive identifiers of the multiple primitives or the vertex identifiers associated with the primitive identifiers.

[0410] In some implementations, process 850 includes processing commands to obtain primitive data for multiple primitives based on the commands, calculating the contribution of each of the multiple primitives' electromagnetic (EM) fields to each of the multiple display elements based on the primitive data for the multiple primitives, and accumulating the contributions of the multiple primitives' EM fields to each of the multiple display elements.

[0411] In some implementations, reconstructing a live scene in 3D space by modulating the display with one or more digital holograms involves generating modulation control signals for multiple display elements of the display based on the digital holograms corresponding to the live scene. The digital holograms can be complex value holograms, and process 850 may include converting the complex value holograms to phase-only holograms and generating respective modulation control signals for multiple display elements based on the phase-only holograms.

[0412] In some implementations, reconstructing a live scene in 3D space by modulating a display with one or more digital holograms involves sending illumination control signals to an illuminator (e.g., illuminator 140 in Figure 1A) to cause the illuminator to illuminate the display so that the light forms a volume light field corresponding to the live scene through the modulated display elements of the display, and, in coordination with sending illumination control signals to the illuminator, outputting individual modulation control signals to each of the multiple display elements.

[0413] In some implementations, reconstructing a live scene in 3D space by modulating a display with one or more digital holograms includes sequentially outputting a first modulation control signal for modulating the display with information associated with a first color during a first time period, and a second modulation control signal for modulating the display with information associated with a second color during a subsequent second time period, and sequentially outputting a first illumination control signal for activating an illuminator to turn on a first light-emitting element to emit light of the first color during the first time period, and a second illumination control signal for activating an illuminator to turn on a second light-emitting element to emit light of the second color during the second time period.

[0414] In some implementations, process 850 includes generating a sequential digital hologram corresponding to the live scene based on captured sequential visual data of the live scene, and sequentially reconstructing the live scene in 3D space based on the sequential digital hologram.

[0415] 9. Exemplary Applications The systems, methods, and techniques described herein can be applied to any suitable use.

[0416] For example, a system (e.g., system 100 in Figure 1A or system 200 in Figure 2) can be used to holographically display a live scene, and live matches such as soccer matches can be broadcast holographically, as described above.

[0417] As another example, a system (for example, system 100 in Figure 1A or system 200 in Figure 2) can be used to holographically display moving objects. One or more objects can be placed on a rotating stage. As the rotating stage rotates, optical holograms or scene data of the rotating objects can be captured and then simultaneously reconstructed by a holographic display.

[0418] As another example, a system (for example, system 100 in Figure 1A or system 200 in Figure 2) can create holographic stop-motion animation. Stop-motion is an animation filmmaking technique in which objects are physically manipulated little by little between individually filmed frames so that when a series of frames are played back, the objects appear to exhibit independent movement or change.

[0419] As another example, a system (for example, system 100 in Figure 1A or system 200 in Figure 2) can be used to capture a scene with optical effects. By adding optical effects to the scene, such as introducing prisms or mirrors, artistic holographic scenes can be created.

[0420] Implementations of the subject matter and functional operations described herein may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Implementations of the subject matter described herein may be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a tangible non-temporary computer storage medium for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, program instructions may be encoded in artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable memory board, a random-access or serial-access memory device, or one or more combinations thereof.

[0421] The terms “data processing device,” “computer,” or “electronic computer device” (or equivalent as understood by those skilled in the art) refer to data processing hardware and encompass all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may be, or further include, application-specific logic circuits, such as a central processing unit (CPU), an FPGA (field-programmable gate array), or an ASIC (application-specific integrated circuit). In some implementations, data processing devices and application-specific logic circuits may be hardware-based or software-based. A device may optionally include code that creates an environment for the execution of computer programs, such as processor firmware, a protocol stack, a database management system, an operating system, or code that constitutes one or more of these. This specification assumes the use of data processing devices with or without a conventional operating system.

[0422] Computer programs, also called or written as programs, software, software applications, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed as standalone programs or in any form, including modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may or may not correspond to files in a file system. A program may be stored in a portion of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordinated files, for example, a file that stores one or more modules, subprograms, or portions of code. Computer programs can be deployed to run on one computer, or on multiple computers located in one site, or distributed across multiple sites and interconnected by a communication network. While the parts of a program shown in various diagrams are shown as individual modules implementing various features and functionalities through various objects, methods, or other processes, a program may instead include several submodules, third-party services, components, libraries, etc., as needed. Conversely, the characteristics and functionalities of various components can be combined into a single component as needed.

[0423] The processes and logic flows described herein can be executed by one or more programmable computers that run one or more computer programs to perform a function by manipulating input data to produce an output. The processes and logic flows can also be executed by application-specific logic circuits such as CPUs, GPUs, FPGAs, or ASICs, and the devices can be implemented as these application-specific logic circuits.

[0424] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, both, or any other type of CPU. Generally, the CPU receives instructions and data from read-only memory (ROM) or random access memory (RAM), or both. The main components of a computer are the CPU for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes, or is operably coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or receives data from them, transfers data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be integrated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, global positioning system (GPS) receivers, or portable storage devices, such as, to name just a few, universal serial bus (USB) flash drives.

[0425] Computer-readable media suitable for storing computer program instructions and data (temporary or non-temporary, as necessary) include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM disks, DVD-R disks, DVD-RAM disks, and DVD-ROM disks. Memory can store a variety of objects or data, including caches, lookup tables, classes, frameworks, applications, backup data, jobs, web pages, web page templates, database tables, repositories for storing business and dynamic information, and any other appropriate information, including any parameters, variables, algorithms, instructions, rules, constraints, or references thereto. Furthermore, memory may include any other appropriate data, such as logs, policies, security or access data, and report files. The processor and memory may be complemented by or incorporated into application-specific logic circuits.

[0426] To provide user interaction, the implementations of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a cathode ray tube (CRT), liquid crystal display (LCD), light-emitting diode (LED), holographic or optical field display, or plasma monitor, and a keyboard and pointing device, such as a mouse, trackball, or trackpad, on which the user can provide input to the computer. Input may also be provided to the computer using a touchscreen, such as a pressure-sensitive tablet computer surface, a multi-touch screen using capacitive or electrical sensing, or other types of touchscreens. User interaction can also be provided using other types of devices. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback. Input from the user can be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0427] The term “Graphical User Interface” or “GUI” can be used singly or plural to describe one or more graphical user interfaces and each display of a particular graphical user interface. Therefore, GUI can represent any graphical user interface, including but not limited to web browsers, touchscreens, or command-line interfaces (CLIs), that process information and efficiently present the results to the user. Generally, GUI can include several user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, pull-down lists, and buttons operable by business suite users. These and other UI elements may relate to or represent the functionality of the web browser.

[0428] Implementations of the subject matter described herein may be implemented in a computing system including, for example, a backend component such as a data server, or a middleware component such as an application server, or a frontend component such as a client computer having a graphical user interface or web browser on which a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system may be interconnected by any form or medium of wireline or wireless digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs), radio access networks (RANs), metropolitan area networks (MANs), wide area networks (WANs), worldwide interoperability for microwave access (WiMAX), wireless local area networks (WLANs) using, for example, 602.11 a / b / g / n and 602.20, all or part of the Internet, and any other communication systems in one or more locations. Networks can communicate voice, video, data, or other appropriate information between network addresses using, for example, Internet Protocol (IP) packets, frame relay frames, and asynchronous transfer mode (ATM) cells.

[0429] A computing system can include clients and servers. Clients and servers are generally remote from each other and typically interact via a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other.

[0430] In some implementations, any or all components of a computing system, both hardware and software, can interface with each other or with each other using an application programming interface (API) or a service layer. An API can include specifications for routines, data structures, and object classes. An API may be language-independent or language-dependent and may refer to a complete interface, a single function, or even a set of APIs. The service layer provides software services to the computing system. The functionality of various components of the computing system may be accessible to all service consumers through this service layer. Software services provide defined business functionality that is reusable through defined interfaces. For example, an interface may be software written in any suitable language that provides data in any suitable format. The API and service layer may be integrated or standalone components in relation to other components of the computing system. Furthermore, any or all parts of the service layer may be implemented as a child module or submodule of another software module, enterprise application, or hardware module without departing from the scope of this specification.

[0431] This specification includes many specific details of implementation, which should not be interpreted as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of separate implementations may be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may be implemented separately or in any appropriate partial combination in multiple implementations. Furthermore, features may be described above as acting in a particular combination, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from the combination, and the claimed combination may cover a partial combination or a variation of a partial combination.

[0432] A specific implementation of this subject matter has been described. Other implementations, alternatives, and permutations of the described implementation are within the scope of the following claims, as will be apparent to those skilled in the art. Although the operations are shown in a specific order in the drawings or claims, this should not be understood as requiring that such operations be performed in the specific order or sequence shown, or that all illustrated operations be performed (some operations may be considered optional). In certain circumstances, multitasking or parallel processing may be performed if it is deemed advantageous and appropriate.

[0433] For the sake of brevity, conventional techniques for constructing and using holographic gratings, LCOS devices, and other optical structures and systems may not be described in detail herein. Furthermore, the connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships, signal or optical paths, and / or physical connections between various elements. It should be noted that many alternative or additional functional relationships, signal or optical paths, or physical connections may exist in exemplary holographic gratings, LCOS, or other optical structures or systems, and / or their components.

[0434] Detailed descriptions of the various exemplary embodiments described herein refer to the accompanying drawings and figures illustrating various exemplary embodiments. While these various exemplary embodiments are described in sufficient detail to enable those skilled in the art to implement the disclosure, it should be understood that other exemplary embodiments may be realized, and logical, optical, and mechanical modifications may be made without departing from the spirit and scope of the disclosure. Therefore, the detailed descriptions herein are presented for illustrative purposes only and not limitation. For example, the steps listed in any description of a method or process may be performed in any suitable order and are not limited to the order presented unless expressly stated. Furthermore, any function or step may be outsourced to or performed by one or more third parties. Modifications, additions, or omissions may be made to the systems, apparatus, and methods described herein without departing from the scope of the disclosure. For example, the components of systems and apparatus may be integrated or separate. Furthermore, the operation of the systems and apparatus disclosed herein may be performed by more, fewer, or other components, and the described methods may include more, fewer, or other steps.

[0435] As used herein, “each” refers to each member of a set or each member of a subset of a set. Furthermore, any reference to the singular form may include multiple exemplary embodiments, and any reference to two or more components may include a singular exemplary embodiment. While certain advantages are listed herein, various exemplary embodiments may include some of the listed advantages, none of them, or all of them.

[0436] Benefits, other advantages, and solutions to problems are described herein in relation to specific exemplary embodiments. However, benefits, advantages, solutions to problems, and any elements that may cause any benefit, advantage, or solution to occur or be enhanced should not be construed as important, necessary, or essential features or elements of this disclosure. Accordingly, the scope of this disclosure is not limited by anything other than the appended claims, and references to singular elements mean "one or more" rather than "one and only" unless expressly stated. Furthermore, where phrases similar to "at least one of A, B, and C" or "at least one of A, B, or C" are used in the claims or specification, these phrases are intended to mean that A may be present alone in an exemplary embodiment, B may be present alone in an exemplary embodiment, C may be present alone in an exemplary embodiment, or any combination of elements A, B, and C, e.g., A and B, A and C, B and C, or A, B, and C may be present in a single exemplary embodiment.

[0437] Therefore, the description of exemplary implementations provided previously does not define or limit this Spec. Other changes, substitutions, and modifications are also possible without departing from the spirit and scope of this Spec.

Claims

1. It is a system, It is a holographic capture system, An optical system configured to generate an optical hologram of a live scene comprising one or more three-dimensional (3D) objects, A holographic capture system comprising: an optical sensor configured to capture a sequential optical hologram of the live scene and output sequential hologram data associated with the sequential optical hologram, wherein each optical hologram is associated with individual hologram data; A holographic display system configured to optically reconstruct the live scene in 3D space based on at least a portion of the sequential hologram data, system.

2. The system further comprises a computing device coupled between the holographic capture system and the holographic display system, The computing device is configured to receive at least a portion of the sequential hologram data from the optical sensor and to generate a digital hologram associated with the live scene based on at least a portion of the sequential hologram data. The system according to claim 1, wherein the holographic display system is configured to receive the digital hologram associated with the live scene from the computing device and to reconstruct the live scene in the 3D space based on the digital hologram.

3. The holographic capture system is configured to capture the sequential optical hologram and generate the sequential hologram data without storing the sequential optical hologram and the sequential hologram data. The system according to claim 1 or 2, wherein the computing device is configured to process at least a portion of the sequential hologram data in order to generate the digital hologram, without storing the sequential hologram data and at least a portion of the digital hologram.

4. The system according to claim 2 or 3, wherein the holographic capture system, the computing device, and the holographic display system are together configured to capture an optical hologram of the live scene and optically reconstruct the live scene in real time.

5. The system according to any one of claims 2 to 4, wherein the digital hologram comprises an amplitude-like hologram, and the holographic display system comprises a display for phase modulation.

6. The system according to any one of claims 2 to 5, wherein the optical sensor comprises a digital sensor and the sequential hologram data comprises a stream of digital data.

7. The system according to claim 6, wherein the digital data includes an array of data bits.

8. The system according to any one of claims 2 to 7, further comprising a frame grabber coupled to the optical sensor and configured to select individual hologram data of one or more optical holograms from the sequential optical holograms transmitted to the computing device.

9. The system according to claim 8, wherein the frame grabber comprises a frame buffer-based grabber configured to deposit the individual hologram data into the frame buffer of the frame grabber, and subsequently to the computing device.

10. The system according to claim 8, wherein the frame grabber comprises a first-in, first-out (FIFO) based grabber configured to directly deposit the individual hologram data into the computing device.

11. The system according to any one of claims 8 to 10, wherein the frame grabber is externally coupled to the optical sensor and the computing device, respectively.

12. The system according to any one of claims 8 to 11, wherein the frame grabber is included in the optical sensor or the computing device.

13. The optical sensor comprises a plurality of sensing pixels within the active area of ​​the optical sensor, and the holographic display system comprises a display having a plurality of display elements. The system according to any one of claims 2 to 12, wherein the computing device is configured to process at least a portion of the sequential hologram data to generate the digital hologram associated with the live scene based on at least one of the pitch of the sensing pixels, the pitch of the display elements, the size of the active area of ​​the optical sensor, or the size of the display.

14. The system according to claim 13, wherein the pitch of the sensing pixels is associated with the resolution of the captured optical hologram and the size of the scene that can be captured.

15. The system according to claim 13 or 14, wherein the pitch of the display elements is associated with the acceptable viewing angle of the scene to be reconstructed and the size of the display.

16. The computing device is configured to suppress mismatches between the captured optical hologram of the live scene and the reconstruction of the live scene. The mismatch is the difference between the pitch of the sensing pixel and the pitch of the display element, or The difference between the size of the active area of ​​the optical sensor and the size of the display is associated with at least one of the following: The system according to any one of claims 13 to 15.

17. The computing device scales a first digital hologram associated with the captured optical hologram based on the ratio between the pitch of the sensing pixel and the pitch of the display element, or The system is configured to perform at least one of the steps of adjusting the scaled first digital hologram based on the size of the display and the size of the hologram data in order to generate a second digital hologram that is modulated on the display. The system according to any one of claims 13 to 16.

18. The system according to claim 17, wherein the computing device is configured to perform the scaling using at least one of one or more interpolation algorithms, including linear interpolation, nearest neighbor interpolation, cubic spline interpolation, shape-preserving interpolation, biharmonic interpolation, and thin-plate spline interpolation.

19. The system according to any one of claims 16 to 18, wherein the computing device is configured to resample a first digital hologram associated with a captured optical hologram, the first digital hologram associated with the pitch of the sensing pixel, to a second digital hologram modulated on the display, the second digital hologram associated with the pitch of the display element, using Fourier transforms and inverse Fourier transforms.

20. The computing device, To generate the converted first digital hologram, the Fourier transform is performed on the first digital hologram. If the pitch of the sensing pixel is larger than the pitch of the display element, The system according to claim 19, configured to perform zero padding on the converted first digital hologram based on the ratio between the pitch of the sensing pixel and the pitch of the display element, and then perform the inverse Fourier transform on the converted first digital hologram with zero padding to obtain the second digital hologram.

21. The computing device, To generate the converted first digital hologram, the Fourier transform is performed on the first digital hologram. In response to determining that the pitch of the sensing pixel is smaller than the pitch of the display element, Based on the ratio between the pitch of the sensing pixel and the pitch of the display element, the converted first digital hologram is cropped, and then, The system according to claim 19, configured to perform the inverse Fourier transform on the cropped transformed first digital hologram in order to obtain the second digital hologram.

22. The system according to any one of claims 19 to 21, wherein the computing device is configured to resample the first digital hologram into a second digital hologram by resampling the center points of the plurality of sensing pixels of the optical sensor so as to match the geometric centers of the plurality of display elements of the display.

23. The plurality of sensing pixels of the optical sensor are regularly arranged in the active area of ​​the optical sensor, and the plurality of display elements are regularly arranged in the display. The center points of the plurality of sensing pixels are regularly spaced apart, The system according to claim 22, wherein the geometric centers of the plurality of display elements of the display are regularly spaced apart.

24. The plurality of sensing pixels of the optical sensor are regularly arranged in the active area of ​​the optical sensor, and the plurality of display elements are irregularly arranged in the display. The center points of the plurality of sensing pixels are regularly spaced apart, The system according to claim 22 or 23, wherein the geometric centers of the plurality of display elements of the display are irregularly spaced apart.

25. The system according to claim 24, wherein the computing device is configured to resample the regularly spaced center points of the plurality of sensing pixels to match the irregularly spaced geometric centers of the plurality of display elements by determining the position of each geometric center of the plurality of display elements based on a weighted sum of adjacent center points around the geometric center using one or more weighting algorithms.

26. The system according to claim 24 or 25, wherein the plurality of display elements form a Voronoi pattern.

27. The plurality of sensing pixels of the optical sensor are irregularly arranged in the active area of ​​the optical sensor, and the plurality of display elements are irregularly arranged in the display. The center points of the plurality of sensing pixels are irregularly spaced apart. The system according to any one of claims 22 to 26, wherein the geometric centers of the plurality of display elements of the display are irregularly spaced apart.

28. The system according to claim 27, wherein the irregular pattern formed by the plurality of sensing pixels matches the irregular pattern formed by the plurality of display elements.

29. The optical system includes an interferometer, and the optical hologram includes an interference pattern of an object beam that interacts with the live scene and a reference beam that interferes with the object beam by the interferometer. The system according to any one of claims 1 to 28.

30. The aforementioned optical system, A coherent light source configured to emit a coherent light beam, The system according to claim 29, further comprising: a beam splitter configured to separate the coherent light beam from the coherent light source into the object beam and the reference beam.

31. The system according to claim 30, wherein the optical system further comprises a beam combiner, the live scene is located on the optical path of the object beam upstream of the beam combiner, and the beam combiner is configured to superimpose the reference beam and the object beam to form the interference pattern.

32. The system according to claim 31, wherein the optical sensor is located downstream of the beam combiner and is configured to directly capture the interference pattern on the active area of ​​the optical sensor.

33. The system according to claim 32, wherein there is no optical lens between the beam combiner and the optical sensor.

34. The system according to any one of claims 31 to 33, wherein there is no optical lens in the optical path of the reference beam between the beam splitter and the beam combiner.

35. The system according to any one of claims 31 to 34, wherein the live scene is transparent and the object beam passes through the live scene and is incident on the beam combiner.

36. The system according to any one of claims 31 to 34, wherein the live scene is reflective, and the optical system comprises one or more reflective mirrors configured to guide the object beam toward the live scene at an angle such that the object beam is reflected or scattered from the live scene and incident upon the beam combiner.

37. The system according to any one of claims 31 to 36, wherein the optical system further comprises one or more optical lenses configured on the optical path of the object beam between the live scene and the beam combiner to enlarge or reduce the object beam after it has interacted with the live scene to fit the active area of ​​the optical sensor.

38. The system according to any one of claims 31 to 37, wherein the optical system further comprises an absorber positioned on the side of the beam combiner and configured to absorb another portion of the reference beam that propagates away from the interference pattern.

39. The system according to any one of claims 30 to 38, wherein the optical system further comprises one or more optical lenses positioned between the beam splitter and the live scene and configured to enlarge or reduce the object beam to fit the active area of ​​the optical sensor.

40. The system according to any one of claims 30 to 39, wherein the optical system comprises a collimator positioned upstream of the beam splitter and configured to collimate the coherent light beam from the coherent light source.

41. The system according to any one of claims 29 to 40, wherein the optical system comprises a phase adjuster positioned on the optical path of one of the object beam and the reference beam and configured to dynamically adjust the phase shift of one of the object beam and the reference beam before the interference pattern is formed, and the phase adjuster comprises a phase shifter or a dynamic retarder.

42. The system according to claim 41, wherein the phase adjuster is configured to sequentially adjust the phase shift to a series of predetermined values ​​within the time period such that the optical sensor captures a corresponding sequential optical hologram of the live scene over a time period.

43. The system further comprises a computing device coupled between the holographic capture system and the holographic display system, The system according to claim 42, wherein the computing device is configured to generate a noise-reduced digital hologram of the live scene based on the corresponding sequential optical hologram.

44. The system according to claim 43, wherein the phase adjuster comprises a liquid crystal cell configured to adjust the phase shift by corresponding voltages so that it behaves in a series of predetermined values.

45. The system according to claim 44, wherein the liquid crystal cell is a single cell having a size greater than or equal to the size of either the object beam or the reference beam.

46. The system according to claim 44 or 45, wherein the correspondence between the predetermined value of the phase shift and the corresponding voltage is pre-calibrated and predetermined.

47. The computing device is configured to process the corresponding sequential optical hologram in order to obtain the corresponding raw digital hologram. The system according to any one of claims 44 to 46, wherein the computing device is configured to perform one or more mathematical operations on the corresponding raw digital hologram in order to generate the digital hologram of the live scene.

48. The series of predetermined values ​​for the phase shift include 0, pi / 2, pi, and 3pi / 2, and the digital hologram is calculated based on the following formula: final_hologram=(hologram_0-hologram_pi) / (hologram_pi / 2)-hologram_3pi / 2, In the formula, final_hologram represents the digital hologram, hologram_0 represents the first corresponding raw digital hologram based on the first corresponding optical hologram with a phase shift of 0. hologram_pi / 2 represents a second corresponding raw digital hologram based on a second corresponding optical hologram with a phase shift of pi / 2. hologram_pi represents a third corresponding raw digital hologram based on a third corresponding optical hologram with a phase shift of pi, The system according to claim 47, wherein hologram_3pi / 2 represents a fourth corresponding raw digital hologram based on a fourth corresponding optical hologram with a phase shift of 3pi / 2.

49. The system according to any one of claims 30 to 48, wherein the coherent light source comprises a plurality of optical elements, each of which emits a separate color.

50. The plurality of coherent optical elements are configured to sequentially and alternately emit light having each of the respective colors. The system according to claim 49, wherein the holographic capture system further comprises a corresponding color filter positioned upstream of the optical sensor for each of the plurality of coherent optical elements, the corresponding color filter being configured to transmit light of the corresponding color from the coherent optical elements and to block light of other colors reaching the optical sensor.

51. The plurality of coherent optical elements are configured to simultaneously emit light having each of the respective colors. The system according to claim 49, wherein the holographic capture system further comprises a color filter array having groups of different color filters on a plurality of sensing pixels of the optical sensor, the different color filters being associated with the respective colors, and each group of the different color filters being arranged on a corresponding group of adjacent sensing pixels of the plurality of sensing pixels.

52. The system according to claim 51, wherein the optical sensor is configured to determine holographic data for each of the colors based on captured optical holograms captured by corresponding groups of adjacent sensing pixels of the plurality of sensing pixels.

53. The system further comprises a computing device coupled between the holographic capture system and the holographic display system, The system according to claim 51 or 52, wherein the computing device is configured to generate a digital hologram for each of the colors based on a captured optical hologram captured by a corresponding group of adjacent sensing pixels of the plurality of sensing pixels.

54. The aforementioned holographic display system A display including multiple display elements, The system comprises a drive device coupled to the display, The aforementioned drive device Based on the digital hologram associated with the live scene, control signals are generated for the plurality of display elements of the display. The system according to any one of claims 1 to 53, configured to transmit the control signal to the display in order to modulate the plurality of display elements of the display based on the control signal.

55. The system further comprises a computing device coupled between the holographic capture system and the holographic display system, The computing device is configured to receive at least a portion of the sequential hologram data from the optical sensor and to generate a digital hologram associated with the live scene based on at least a portion of the sequential hologram data. The system according to claim 54, wherein the holographic display system is configured to receive the digital hologram associated with the live scene from the computing device and to reconstruct the live scene in the 3D space based on the digital hologram.

56. The digital hologram comprises a group of digital holograms for multiple colors, and the holographic display system further comprises an illuminator having multiple coherent light elements for the multiple colors. The system according to claim 55.

57. The aforementioned drive device The display is modulated sequentially with a first digital hologram for a first color during a first time period, and then with a second digital hologram for a second color during a subsequent second time period. The system according to claim 56, wherein the illuminator is configured to control the illuminator so as to sequentially light up a first coherent light element so as to emit light of a first color during the first time period, and a second coherent light element so as to emit light of a second color during the subsequent second time period.

58. The aforementioned drive device A display driver coupled to the aforementioned display, A lighting driver coupled to the aforementioned lighting device, or The system according to any one of claims 55 to 57, comprising at least one of the following: a memory coupled to at least one of the display driver or the illuminator driver.

59. The system according to any one of claims 1 to 58, wherein the reconstructed live scene in the 3D space has one or more holographic properties comprising occlusion, parallax, and accommodation.

60. The steps include optically generating an optical hologram of a live scene comprising one or more three-dimensional (3D) objects, The steps include capturing a successive optical hologram of the live scene and generating successive hologram data associated with the successive optical hologram, wherein each optical hologram is associated with individual hologram data. The process includes the step of reconstructing the live scene in 3D space based on at least a portion of the sequential hologram data. method.

61. The further step involves processing at least a portion of the sequential hologram data in order to generate a digital hologram associated with the live scene, The step of reconstructing the live scene in 3D space based on at least a portion of the hologram data is: The process includes the step of reconstructing the live scene in the 3D space based on the digital hologram. The method according to claim 60.

62. Each of the aforementioned digital holograms comprises an amplitude-like hologram, The step of reconstructing the live scene in the 3D space based on the digital hologram is: The method according to claim 61, further comprising the step of directly modulating a phase-modulating display with the digital hologram.

63. The method according to claim 61 or 62, wherein the sequential hologram data comprises a stream of digital data, and the digital data comprises an array of data bits.

64. The step further comprises selecting individual hologram data of one or more optical holograms from the aforementioned sequential optical holograms using a frame grabber, The step of processing at least a portion of the sequential hologram data in order to generate the digital hologram associated with the live scene is: The step of generating the digital hologram associated with the live scene based on the selected individual hologram data of the one or more optical holograms, The method according to any one of claims 61 to 63.

65. The aforementioned frame grabber, A frame buffer-based grabber configured to deposit the individual hologram data into the frame buffer of the frame grabber before transmitting it to generate the digital hologram, or The method according to claim 64, further comprising one of a first-in, first-out (FIFO) based grabber configured to directly transmit the individual hologram data in order to generate the digital hologram.

66. The step of processing at least a portion of the sequential hologram data in order to generate the digital hologram associated with the live scene is: The method according to any one of claims 61 to 65, comprising the step of processing at least a portion of the sequential hologram data based on at least one of the pitch of sensing pixels of an optical sensor, the pitch of display elements of a display, the size of the active area of ​​the optical sensor, or the size of the display, in order to generate the digital hologram associated with the live scene.

67. The pitch of the sensing pixels is associated with the resolution of the captured optical hologram and the size of the scene that can be captured. The method according to claim 66, wherein the pitch of the display elements is associated with the acceptable viewing angle of the scene to be reconstructed and the size of the display.

68. The step of processing at least a portion of the sequential hologram data in order to generate the digital hologram associated with the live scene is: The method includes a step of suppressing the mismatch between the captured optical hologram of the live scene and the reconstruction of the live scene, The aforementioned mismatch, The difference between the pitch of the sensing pixel and the pitch of the display element, or The difference between the size of the active area of ​​the optical sensor and the size of the display is associated with at least one of the following: The method according to claim 66 or 67.

69. The step of processing at least a portion of the sequential hologram data in order to generate the digital hologram associated with the live scene is: A step of scaling a first digital hologram associated with a captured optical hologram based on the ratio between the pitch of the sensing pixel and the pitch of the display element, or The method according to any one of claims 66 to 68, comprising at least one of the steps of cropping the scaled first digital hologram based on the size of the display and the size of the hologram data in order to generate a second digital hologram to be modulated on the display.

70. The step of scaling the size of the first digital hologram associated with the captured optical hologram comprises using at least one of one interpolation algorithms, including linear interpolation, nearest neighbor interpolation, cubic spline interpolation, shape-preserving interpolation, biharmonic interpolation, and thin-plate spline interpolation. The method according to claim 69.

71. The step of processing at least a portion of the sequential hologram data in order to generate the digital hologram associated with the live scene is: The method according to any one of claims 66 to 70, comprising the step of resampling a first digital hologram associated with a captured optical hologram, the first digital hologram associated with the pitch of the sensing pixel, to a second digital hologram modulated on the display, the second digital hologram associated with the pitch of the display element, using a Fourier transform and an inverse Fourier transform.

72. The step of resampling the first digital hologram associated with the captured optical hologram into the second digital hologram modulated on the display is: To generate the converted first digital hologram, the Fourier transform is performed on the first digital hologram. In response to determining that the pitch of the sensing pixel is smaller than the pitch of the display element, Based on the ratio between the pitch of the sensing pixel and the pitch of the display element, the converted first digital hologram is cropped, and then, The method according to claim 71, further comprising the step of performing the inverse Fourier transform on the cropped transformed first digital hologram in order to obtain the second digital hologram.

73. The step of resampling the first digital hologram associated with the captured optical hologram into the second digital hologram modulated on the display is: To generate the converted first digital hologram, the Fourier transform is performed on the first digital hologram. If the pitch of the sensing pixel is larger than the pitch of the display element, Based on the ratio between the pitch of the sensing pixel and the pitch of the display element, one or more zero-paddings are added to the converted first digital hologram, and then, The method according to claim 71 or 72, further comprising the step of performing the inverse Fourier transform on the converted first digital hologram with the added one or more zero paddings in order to obtain the second digital hologram.

74. The step of resampling the first digital hologram associated with the captured optical hologram into the second digital hologram modulated on the display is: The method according to any one of claims 71 to 73, further comprising the step of resampling the first digital hologram into a second digital hologram by resampling the center points of the plurality of sensing pixels of the optical sensor so as to match the geometric centers of the plurality of display elements of the display.

75. The plurality of sensing pixels of the optical sensor are regularly arranged in the active area of ​​the optical sensor, and the plurality of display elements are regularly arranged in the display. The center points of the plurality of sensing pixels are regularly spaced apart, The method according to claim 74, wherein the geometric centers of the plurality of display elements of the display are regularly spaced apart.

76. The plurality of sensing pixels of the optical sensor are regularly arranged in the active area of ​​the optical sensor, and the plurality of display elements are irregularly arranged in the display. The center points of the plurality of sensing pixels are regularly spaced apart, The geometric centers of the plurality of display elements of the aforementioned display are irregularly spaced apart. The method according to claim 74 or 75.

77. The step of resampling the first digital hologram associated with the captured optical hologram into the second digital hologram modulated on the display is: The method comprises the step of resampling the regularly spaced center points of the plurality of sensing pixels to match the irregularly spaced geometric centers of the plurality of display elements by determining the position of each geometric center of the plurality of display elements based on a weighted sum of adjacent center points around the geometric center using one or more weighting algorithms. The method according to claim 76.

78. The plurality of sensing pixels of the optical sensor are irregularly arranged in the active area of ​​the optical sensor, and the plurality of display elements are irregularly arranged in the display. The center points of the plurality of sensing pixels are irregularly spaced apart. The geometric centers of the plurality of display elements of the aforementioned display are irregularly spaced apart. The method according to any one of claims 74 to 77, wherein the irregular pattern formed by the plurality of sensing pixels matches the irregular pattern formed by the plurality of display elements.

79. The step of optically generating the optical hologram of the live scene is The method according to any one of claims 60 to 78, comprising the step of forming an interference pattern by interfering an object beam interacting with the live scene with a reference beam, wherein the object beam and the reference beam are coherent light beams, and the optical hologram comprises the interference pattern.

80. The step of capturing the sequential optical hologram of the live scene is The method according to claim 79, further comprising the step of directly capturing the interference pattern on the active area of ​​an optical sensor.

81. The further step includes expanding or contracting the object beam, after it has interacted with the live scene, to fit the active area of ​​the optical sensor. The method according to claim 80.

82. The method further comprises the step of dynamically adjusting the phase shift of either the object beam or the reference beam before the interference pattern is formed. The method according to any one of claims 79 to 81.

83. The step of dynamically adjusting the phase shift of one of the object beam and the reference beam is: The process includes a step of sequentially adjusting the phase shift to a series of predetermined values ​​within a certain time period. The step of capturing the sequential optical hologram of the live scene comprises the step of capturing the corresponding sequential optical hologram of the live scene over the specified time period. The method according to claim 82.

84. The method further comprises the step of generating a noise-suppressed digital hologram of the live scene based on the corresponding sequential optical hologram. The method according to claim 83.

85. The step of generating a digital hologram of the live scene with suppressed noise based on the corresponding sequential optical hologram is: To obtain the corresponding raw digital hologram, the corresponding sequential optical hologram is processed, The method according to claim 84, further comprising the step of performing one or more mathematical operations on the corresponding raw digital hologram in order to generate the digital hologram of the live scene.

86. The series of predetermined values ​​for the phase shift include 0, pi / 2, pi, and 3pi / 2, and the digital hologram is calculated based on the following formula: final_hologram=(hologram_0-hologram_pi) / (hologram_pi / 2-hologram_3pi / 2), In the formula, final_hologram represents the digital hologram, hologram_0 represents the first corresponding raw digital hologram based on the first corresponding optical hologram with a phase shift of 0. hologram_pi / 2 represents a second corresponding raw digital hologram based on a second corresponding optical hologram with a phase shift of pi / 2. hologram_pi represents a third corresponding raw digital hologram based on a third corresponding optical hologram with a phase shift of pi, hologram_3pi / 2 represents the fourth corresponding raw digital hologram based on the fourth corresponding optical hologram with a phase shift of 3pi / 2. The method according to claim 85.

87. The step of optically generating the optical hologram of the live scene is The process includes the step of sequentially and alternately emitting light having multiple colors in order to sequentially and alternately generate optical holograms for multiple colors, The step of capturing the sequential optical hologram of the live scene is The method according to any one of claims 60 to 86, comprising the step of sequentially transmitting only light of individual colors and blocking light of other colors.

88. The step of optically generating the optical hologram of the live scene is It includes a step that simultaneously emits light of multiple colors, The step of capturing the sequential optical hologram of the live scene is The method according to any one of claims 60 to 86, comprising the step of capturing an optical hologram by corresponding groups of adjacent sensing pixels of a plurality of sensing pixels of an optical sensor using a color filter array disposed on the optical sensor, wherein the color filter array includes groups of different color filters on the plurality of sensing pixels of the optical sensor, the different color filters are associated with the plurality of colors, and each group of the different color filters is disposed on corresponding groups of adjacent sensing pixels of the plurality of sensing pixels.

89. The step of generating sequential hologram data associated with the sequential optical hologram of the live scene is: The method according to claim 88, further comprising the step of determining holographic data for each of the plurality of colors based on the optical hologram.

90. The further step is to generate digital holograms for the plurality of colors based on the optical hologram, The method according to claim 88 or 89, wherein the step of reconstructing the live scene in 3D space based on at least a portion of the hologram data comprises the step of reconstructing the live scene in 3D space based on the digital hologram.

91. The step of reconstructing the live scene in 3D space based on at least a portion of the hologram data is: Based on the digital hologram associated with the live scene, control signals are generated for multiple display elements of the display. The method according to any one of claims 59 to 90, further comprising the step of modulating the plurality of display elements of the display based on the control signal.

92. The further step is to generate a digital hologram associated with the live scene based on at least a portion of the sequential hologram data, The digital hologram comprises a group of digital holograms for multiple colors. The method according to claim 91.

93. The step of reconstructing the live scene in 3D space based on at least a portion of the hologram data is: The display is modulated sequentially with a first digital hologram for a first color during a first time period, and then with a second digital hologram for a second color during a subsequent second time period. The method according to claim 92, further comprising the step of sequentially lighting up a first coherent light element so as to emit light of the first color during the first time period, and a second coherent light element so as to emit light of the second color during the subsequent second time period.

94. A hologram generation system configured to generate one or more digital holograms corresponding to a live scene comprising one or more three-dimensional (3D) objects, A holographic display system configured to reconstruct the live scene in 3D space based on one or more digital holograms, The hologram generation system, One or more scene acquisition devices configured to capture visual data of the live scene from one or more views, which includes at least one of one or more images or one or more videos, A computing system, Based on the captured visual data of the live scene, primitive data associated with the live scene is obtained. A system comprising: a computing system configured to generate a digital hologram corresponding to the live scene based on the primitive data associated with the live scene and the display element information of the display of the holographic display system.

95. The computing system is a computing device coupled to one or more scene acquisition devices, Based on the captured visual data of the live scene, a 3D representation of the live scene is generated. The system according to claim 94, comprising a computing device configured to acquire primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, wherein the primitive data associated with the live scene comprises the primitive data of the 3D representation of the live scene.

96. The system according to claim 95, wherein the computing device is configured to use a 3D rendering algorithm to generate a 3D representation of the live scene based on the captured visual data of the live scene.

97. The one or more scene acquisition devices are configured to transmit sequential visual data to the computing device, which is sequential visual data of the live scene over a certain period of time, comprising first visual data and second visual data following the first visual data. The computing device, Using the 3D rendering algorithm, a first 3D representation of the live scene is generated based on the first visual data of the live scene. The system according to claim 96, configured to generate a second 3D representation of the live scene by updating the first 3D representation of the live scene based on the difference between the first visual data and the second visual data using the 3D rendering algorithm.

98. The computing device, The 3D representation of the live scene is loaded into a 3D simulation application. The system according to any one of claims 95 to 97, configured to acquire primitive data of the 3D representation of the live scene based on the output associated with the 3D representation of the live scene of the 3D simulation application.

99. The primitive data of the 3D representation of the live scene is The system according to claim 98, comprising data of a plurality of primitives corresponding to the 3D representation of the live scene, wherein the data comprises primitive data of each of the plurality of primitives, the primitive comprises at least one vertex, and the primitive data of the primitive comprises the data of the at least one vertex.

100. The TH primitive data of the aforementioned primitive is The primitive identifier of the primitive, The at least one vertex identifier of the at least one vertex, The coordinate information of the primitive in the 3D coordinate system, The color information of the primitive, Texture coordinate information of the primitive, Shading information for the aforementioned primitive, Viewpoint-dependent shading information associated with the primitive, or The system according to claim 99, comprising at least one of the occlusion information of the primitive.

101. The computing system is a processing device coupled to the computing device, For each of the plurality of primitives, the contribution of the electromagnetic (EM) field to each of the plurality of display elements of the display is determined based on the primitive data of the primitive. Each of the plurality of display elements of the display is configured to generate the sum of the contributions of the plurality of primitives' EM fields to the display element. The digital hologram comprises a processing device that includes the sum of the contributions of the EM field to the plurality of display elements of the display, The system according to claim 99 or 100.

102. The computing device, It is configured to generate the primitive data of the 3D representation of the live scene based on the output of the 3D simulation application using an Application Programming Interface (API), The computing device, For each of the multiple vertices of the multiple primitives, the individual vertex identifier of the vertex is associated with the individual vertex data of the vertex, and the association between the individual vertex identifier and the individual vertex data of the vertex is stored in the memory of the computing device. The API is configured to perform the following actions for each of the plurality of primitives: associating the individual primitive identifier of the primitive with one or more vertex identifiers of the primitive in memory; and storing the association between the individual primitive identifier and the one or more vertex identifiers of the primitive in memory. The system according to claim 101.

103. The computing device, Determine the primitive identifiers of multiple primitives associated with a command instruction. Determine the vertex identifier associated with the primitive identifier, The API is configured to execute a command including the command instruction, the vertex identifier associated with the primitive identifier, and the primitive identifiers of the plurality of primitives, without the primitive data of the plurality of primitives, to send the command to the processing device. The command instructs the program to draw the plurality of primitives based on at least one of the primitive identifiers of the plurality of primitives or the vertex identifiers associated with the primitive identifiers, in accordance with the command instruction. The system according to claim 102.

104. The processing device, The computer device receives the command, A command processor configured to process the command in order to obtain primitive data of the plurality of primitives from the computing device based on the command, A plurality of computing units configured to calculate the contribution of each of the plurality of primitives' electromagnetic (EM) fields to each of the plurality of display elements based on the primitive data of the plurality of primitives, The system comprises an accumulator configured to accumulate the contributions of the EM fields of the plurality of primitives to the plurality of display elements, The system according to claim 103.

105. The system according to claim 104, wherein the command processor, the plurality of computing units, and the accumulator are connected in series, and the plurality of computing units are connected in parallel between the command processor and the accumulator.

106. The system according to any one of claims 101 to 105, wherein the digital hologram is a complex value hologram, a phase hologram, or an amplitude hologram.

107. The holographic display system is a drive device coupled to the display, The system according to any one of claims 95 to 106, further comprising a drive device configured to generate modulation control signals for a plurality of display elements of the display based on the digital hologram corresponding to the live scene.

108. The digital hologram is a complex value hologram, and the driving device is The aforementioned complex value hologram is converted into a hologram showing only the phase, Based on the phase-only hologram, it is configured to generate the respective modulation control signals for the plurality of display elements. The system according to claim 107.

109. The holographic display system further includes an illuminator, The driving device is configured to transmit an illumination control signal to the illuminator to operate the illuminator to illuminate the display such that the light forms a volume light field corresponding to the live scene by the modulated display elements of the display, The aforementioned drive device The system according to claim 107 or 108, configured to output the individual modulation control signals to each of the plurality of display elements in coordination with transmitting the lighting control signal to the lighting device.

110. The aforementioned drive device A first modulation control signal is output sequentially for modulating the display with a first digital hologram associated with a first color during a first time period, and a second modulation control signal is output for modulating the display with a second digital hologram associated with a second color during a subsequent second time period. The system according to claim 109, configured to sequentially output a first illumination control signal for operating the illuminator to light up a first coherent light element so that it emits light of a first color during a first time period, and a second illumination control signal for operating the illuminator to light up a second coherent light element so that it emits light of a second color during a second time period.

111. The hologram generation system is configured to generate sequential digital holograms corresponding to the live scene. The system according to any one of claims 94 to 110, wherein the holographic display system is configured to continuously reconstruct the live scene in the 3D space based on the sequential digital hologram.

112. The steps of capturing visual data of a live scene from one or more views, wherein the live scene comprises one or more three-dimensional (3D) objects, and the visual data includes at least one of one or more images or one or more videos. The steps include obtaining primitive data associated with the live scene based on the captured visual data of the live scene, A step of generating one or more digital holograms corresponding to the live scene based on the primitive data associated with the live scene and the display element information of the display, A method comprising the step of reconstructing the live scene in 3D space by modulating the display with one or more of the aforementioned digital holograms.

113. The step of obtaining primitive data associated with the live scene based on the captured visual data of the live scene is: Based on the captured visual data of the live scene, a 3D representation of the live scene is generated. The method according to claim 112, comprising the step of obtaining primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, wherein the primitive data associated with the live scene comprises the primitive data of the 3D representation of the live scene.

114. The step of generating a 3D representation of the live scene based on the captured visual data of the live scene is: The method according to claim 113, further comprising the step of processing the captured visual data of the live scene using a 3D rendering algorithm to generate the 3D representation of the live scene.

115. A step of generating sequential visual data of the live scene over a certain period of time, which includes a first visual data and a second visual data following the first visual data; The steps include generating a first 3D representation of the live scene based on the first visual data of the live scene using the 3D rendering algorithm, The method comprises the step of generating a second 3D representation of the live scene by updating the first 3D representation of the live scene based on the difference between the first visual data and the second visual data using the 3D rendering algorithm, The method according to claim 114.

116. The step of obtaining the primitive data of the 3D representation of the live scene based on the 3D representation of the live scene is: The 3D representation of the live scene is loaded into a 3D simulation application. The method according to any one of claims 113 to 115, further comprising the step of obtaining primitive data of the 3D representation of the live scene based on the output associated with the 3D representation of the live scene of the 3D simulation application.

117. The primitive data of the 3D representation of the live scene is The method according to any one of claims 113 to 116, comprising data of a plurality of primitives corresponding to the 3D representation of the live scene, wherein the data comprises primitive data of each of the plurality of primitives, the primitive comprises at least one vertex, and the primitive data of the primitive comprises the data of the at least one vertex.

118. The TH primitive data of the aforementioned primitive is The primitive identifier of the primitive, The at least one vertex identifier of the at least one vertex, The coordinate information of the primitive in the 3D coordinate system, The color information of the primitive, Texture coordinate information of the primitive, Shading information for the aforementioned primitive, Viewpoint-dependent shading information associated with the primitive, or the method according to claim 117, comprising at least one of the occlusion information of the primitive.

119. The step of generating one or more digital holograms corresponding to the live scene is: For each of the plurality of primitives, the step of determining the contribution of the electromagnetic (EM) field to each of the plurality of display elements of the display based on the primitive data of the primitive, The method comprises the step of generating the sum of the contributions of the plurality of primitives to the EM field for each of the plurality of display elements of the display, The method according to claim 117 or 118, wherein the digital hologram comprises the sum of the contributions of the EM field to the plurality of display elements of the display.

120. For each of the multiple vertices of the multiple primitives, the steps include associating the individual vertex identifier of the vertex with the individual vertex data of the vertex, and storing the association between the individual vertex identifier and the individual vertex data of the vertex in memory. The method according to claim 119, further comprising the steps of: for each of the plurality of primitives, associating an individual primitive identifier of the primitive with one or more vertex identifiers of one or more vertices of the primitive in the memory; and storing the association between the individual primitive identifier and the one or more vertex identifiers of the primitive in the memory.

121. Determine the primitive identifiers of multiple primitives associated with a command instruction. Determine the vertex identifier associated with the primitive identifier, The method according to claim 120, further comprising the step of generating a command including the command instruction, the vertex identifier associated with the primitive identifier, and the primitive identifier of the plurality of primitives, wherein the command instructs to draw the plurality of primitives based on at least one of the primitive identifier of the plurality of primitives or the vertex identifier associated with the primitive identifier, in accordance with the command instruction.

122. The steps include processing the command to obtain primitive data of the plurality of primitives based on the command, A step of calculating the contribution of the electromagnetic (EM) field of the plurality of primitives to each of the plurality of display elements based on the primitive data of the plurality of primitives, The method according to claim 121, comprising the step of accumulating the contributions of the EM fields of the plurality of primitives to each of the plurality of display elements.

123. The step of reconstructing the live scene in 3D space by modulating the display with one or more of the aforementioned digital holograms is: The method according to any one of claims 112 to 122, comprising the step of generating modulation control signals for a plurality of display elements of the display based on a digital hologram corresponding to the live scene.

124. The digital hologram is a complex value hologram, and the method is The aforementioned complex value hologram is converted into a hologram showing only the phase, The method according to claim 123, further comprising the step of generating the respective modulation control signals for the plurality of display elements based on the phase-only hologram.

125. The step of reconstructing the live scene in 3D space by modulating the display with one or more of the aforementioned digital holograms is: The steps include: transmitting an illumination control signal to an illumination device to operate the illumination device to irradiate the display with light such that the light forms a volume light field corresponding to the live scene by the modulated display elements of the display; The method according to claim 123 or 124, comprising the step of transmitting the lighting control signal to the lighting device, in coordination with the step of outputting the individual modulation control signal to each of the plurality of display elements.

126. The step of reconstructing the live scene in 3D space by modulating the display with one or more of the aforementioned digital holograms is: Steps include sequentially outputting a first modulation control signal for modulating the display with information associated with a first color during a first time period, and a second modulation control signal for modulating the display with information associated with a second color during a subsequent second time period, The method according to claim 125, comprising the step of sequentially outputting a first illumination control signal for operating the illuminator to light up a first coherent light element so that it emits light of a first color during the first time period, and a second illumination control signal for operating the illuminator to light up a second coherent light element so that it emits light of a second color during the second time period.

127. The steps include generating a sequential digital hologram corresponding to the live scene based on the sequential visual data captured from the live scene, The method according to any one of claims 112 to 126, comprising the step of sequentially reconstructing the live scene in the 3D space based on the sequential digital hologram.

128. A system for generating digital holograms that correspond to live scenes, One or more scene acquisition devices configured to capture visual data of a live scene from one or more views, wherein the live scene includes one or more three-dimensional (3D) objects, and the visual data includes at least one of one or more images or one or more videos. A computing system, Based on the captured visual data of the live scene, primitive data associated with the live scene is obtained. A system comprising: a computing system configured to generate a digital hologram corresponding to the live scene based on the primitive data associated with the live scene and the display element information of the display.

129. The system according to claim 128, wherein the computing system is implemented by the computing system described in any one of claims 94 to 111.

130. A method performed by the system according to claim 128 or 129.

Images