3D display system using electromagnetic field calculations

Abstract

Methods, apparatuses, devices, and systems for three-dimensional (3D) display objects are provided. In one aspect, a method includes: obtaining data including corresponding primitive data of primitives corresponding to an object, determining an electromagnetic field contribution of each primitive to each display element of a display screen by calculating an electromagnetic field propagation from each primitive to each display element of the display screen, generating a sum of electromagnetic field contributions of multiple ones of the primitives to each display element, sending a corresponding control signal to each display element to modulate at least one characteristic of the display element based on the sum of electromagnetic field contributions, sending a timing control signal to an illuminator to activate the illuminator to irradiate light on the display screen, such that the modulated display elements of the display screen cause the light to form a volume light field corresponding to the object.

Description

[0001] This patent application is a divisional application of a patent application with an application date of January 16, 2019, application number 2019800085091, and invention name “Three-dimensional display method using electromagnetic field calculation”. Technical Field

[0002] The present disclosure relates to three-dimensional (3D) displays, and more particularly to 3D displays using computing techniques. Background Art

[0003] Advances in traditional two-dimensional (2D) projection and 3D rendering have led to new approaches for 3D displays, including many hybrid technologies that mix head and eye tracking with traditional display devices for virtual reality (VR), augmented reality (AR), and mixed reality (MR). These technologies attempt to replicate the experience of holographic images by combining tracking and measurement-based calculations to simulate stereoscopic or intraocular light fields that can be represented by actual holograms. Summary of the invention

[0004] The present disclosure describes methods, apparatus, devices, and systems for using electromagnetic (EM) field calculations for three-dimensional (3D) displays.

[0005] The present disclosure provides techniques that can overcome limitations present in known technologies. As an example, the techniques disclosed herein can be implemented without the use of bulky wearable devices such as "3D glasses". As another example, the techniques disclosed herein can be selectively implemented without being limited by: tracking mechanism accuracy, quality of display devices, relatively long processing times and / or relatively high computing requirements, and / or the inability to display objects to multiple viewers at the same time. As another example, the technology can be implemented without dedicated tools and software, thereby developing content that extends above and beyond the tools and software used in conventional 3D content creation. Various different embodiments can exhibit one or more of the aforementioned advantages. For example, certain embodiments of the present disclosure can produce real-time, full-color, true 3D images that appear to be real 3D objects in the real world and can be viewed simultaneously by multiple viewers from different points without hindrance.

[0006] One aspect of the present disclosure is characterized by a method comprising: for each of a plurality of primitives corresponding to an object in a three-dimensional (3D) space, determining an electromagnetic field contribution to the element by calculating electromagnetic field propagation from the primitive to each of a plurality of elements of a display screen in a 3D coordinate system; and for each of the plurality of elements, generating a sum of electromagnetic field contributions of the plurality of primitives to the element.

[0007] The electromagnetic field contribution may include at least one item selected from the group consisting of a phase contribution and an amplitude contribution. The primitive may include at least one item selected from the group consisting of a point primitive, a line primitive, and a polygon primitive. The primitive may include a line primitive having information including at least one item selected from the group consisting of a gradient color, a textured color, and a shading effect. The primitive may also include a polygon primitive having information including at least one item selected from the group consisting of a gradient color, a textured color, and a shading effect. The plurality of primitives may be indexed in a particular order.

[0008] In some embodiments, the method further includes obtaining corresponding primitive data for each of the plurality of primitives. The corresponding primitive data for each of the plurality of primitives may include corresponding color information of the primitive, and the determined electromagnetic field contribution to each of the elements includes information corresponding to the corresponding color information of the primitive. The color information may include at least one item selected from the group consisting of textured color and gradient color. The corresponding primitive data for each of the plurality of primitives may include texture information of the primitive. The corresponding primitive data for each of the plurality of primitives may include coloring information on one or more surfaces of the primitive. The coloring information may include modulation of at least one item selected from the group consisting of color on the one or more surfaces of the primitive and brightness on the one or more surfaces of the primitive.

[0009] In some embodiments, the corresponding primitive data of each primitive in the plurality of primitives includes corresponding coordinate information of the primitive in the 3D coordinate system. The corresponding coordinate information of each element in the plurality of elements in the 3D coordinate system may be determined based on the corresponding coordinate information of the plurality of primitives in the 3D coordinate system. The corresponding coordinate information of each element may correspond to a logical storage address for the element stored in a memory.

[0010] Determining the electromagnetic field contribution of each of the plurality of primitives to each of the plurality of elements may include: determining at least one distance between the element and the primitive based on the corresponding coordinate information of the element and the corresponding coordinate information of the primitive in the 3D coordinate system. Determining the electromagnetic field contribution of each of the plurality of primitives to each of the plurality of elements includes: determining a first distance between a first primitive of the plurality of primitives and the first element based on the corresponding coordinate information of the first primitive and the corresponding coordinate information of the first element of the plurality of elements; and determining a second distance between the first primitive and the second element based on the first distance and the distance between the first element and the second element of the plurality of elements. The distance between the first element and the second element may be predetermined based on the pitch of the plurality of elements of the display screen.

[0011] In some examples, at least one of the plurality of primitives is a line primitive including a first endpoint and a second endpoint, and determining at least one distance between the element and the primitive includes: determining a first distance between the element and the first endpoint of the line primitive; and determining a second distance between the element and the second endpoint of the line primitive. At least one of the plurality of primitives is a triangle primitive including a first endpoint, a second endpoint, and a third endpoint, and determining at least one distance between the element and the primitive includes: determining a first distance between the element and the first endpoint of the triangle primitive; determining a second distance between the element and the second end of the triangle primitive; and determining a third distance between the element and the third endpoint of the triangle primitive.

[0012] In some embodiments, determining the electromagnetic field contribution of each of the plurality of primitives to each of the plurality of elements comprises: determining the electromagnetic field contribution of the primitive to the element based on a predetermined expression of the primitive and the at least one distance. The predetermined expression is determined by analytically calculating the electromagnetic field propagation from the primitive to the element. The predetermined expression is determined by solving Maxwell's equations. The Maxwell equations can be solved by providing boundary conditions defined at the surface of the display screen. The boundary conditions may include Dirichlet boundary conditions or Cauchy boundary conditions. The plurality of primitives and the plurality of elements may be in the 3D space, and the surface of the display screen may form a portion of a boundary surface of the 3D space. The predetermined expression includes at least one item selected from the group consisting of a function including a sine function, a function including a cosine function, and a function including an exponential function, and determining the electromagnetic field contribution comprises: identifying a value of at least one of the functions in a table stored in a memory.

[0013] In some embodiments, determining the electromagnetic field contribution of each of the plurality of primitives to each of the plurality of elements and generating the sum of the electromagnetic field contributions to each of the plurality of elements includes: determining a first electromagnetic field contribution of the plurality of primitives to a first element of the plurality of elements and summing the first electromagnetic field contributions for the first element; and determining a second electromagnetic field contribution of the plurality of primitives to a second element of the plurality of elements and summing the second electromagnetic field contributions for the second element. Determining the first electromagnetic field contribution of the plurality of primitives to the first element may include: determining the electromagnetic field contribution of the first primitive of the plurality of primitives to the first element in parallel with determining the electromagnetic field contribution of the second primitive of the plurality of primitives to the first element.

[0014] In some embodiments, determining the electromagnetic field contribution of each of the plurality of primitives to each of the plurality of elements includes: determining the corresponding first electromagnetic field contribution of a first primitive of the plurality of primitives to each of the plurality of elements; and determining the corresponding second electromagnetic field contribution of a second primitive of the plurality of primitives to each of the plurality of elements, and generating the sum of the electromagnetic field contributions to each of the plurality of elements may include: accumulating the electromagnetic field contribution to the element by adding the corresponding second electromagnetic field contribution to the element to the corresponding first electromagnetic field contribution. Determining the corresponding first electromagnetic field contribution of the first primitive to each of the plurality of elements may be performed in parallel with determining the corresponding second electromagnetic field contribution of the second primitive to each of the plurality of elements.

[0015] Determining the electromagnetic field contribution of each of the plurality of primitives to each of the plurality of elements may include determining a first electromagnetic field contribution of a first primitive of the plurality of primitives to a first element of the plurality of elements in parallel with determining a second electromagnetic field contribution of a second primitive of the plurality of primitives to the first element.

[0016] In some embodiments, the method further comprises: for each of the plurality of elements, generating a corresponding control signal based on the sum of the electromagnetic field contributions of the plurality of primitives to the element, the corresponding control signal being used to modulate at least one characteristic of the element based on the sum of the electromagnetic field contributions of the plurality of primitives to the element. The at least one characteristic of the element may include at least one item selected from the group consisting of a refractive index, an amplitude index, a birefringence, and a hysteresis. The corresponding control signal may include an electrical signal, an optical signal, a magnetic signal, or an acoustic signal. In some cases, the method further comprises: multiplying a scaling factor by the sum of the electromagnetic field contributions to each of the elements to obtain a scaled sum of the electromagnetic field contributions, and wherein the corresponding control signal is generated based on the scaled sum of the electromagnetic field contributions to the element. The method may further comprise: normalizing the sum of the electromagnetic field contributions to each of the elements, wherein the corresponding control signal is based on the normalized sum of the electromagnetic field contributions to the element. The method may further comprise: sending the corresponding control signal to the element.

[0017] In some embodiments, the method further comprises: sending a control signal to an illuminator, the control signal instructing to turn on the illuminator so that the illuminator emits light onto the display screen. The control signal may be sent in response to determining that a sum of electromagnetic field contributions to each of the plurality of elements has been obtained. The modulated elements of the display screen may cause the light to propagate in different directions to form a volumetric light field corresponding to the object in the 3D space. The volumetric light field may correspond to a solution of Maxwell's equations with boundary conditions defined by the modulated elements of the display screen. The light may include white light, and the display screen may be configured to diffract the white light into light of different colors.

[0018] In some embodiments, the method further comprises: representing the values ​​using fixed point representation during the calculation. Each of the values ​​may be represented as an integer with an implicit scaling factor.

[0019] In some embodiments, the method further comprises: performing a mathematical function using a fixed-point number representation. The mathematical function may include at least one item selected from the group consisting of sine, cosine, and arctangent. Executing the mathematical function may include: receiving an expression in a first fixed-point format; and outputting a value in a second fixed-point format having a different level of precision than the first fixed-point format. Executing the mathematical function may include: looking up a table for calculation of the mathematical function, wherein the table includes at least one item selected from the group consisting of a fully enumerated lookup table, an interpolation table, a semi-table-based polynomial function, and a semi-table-based complete minimum maximum polynomial. Executing the mathematical function may include: applying a dedicated range reduction to the input. Executing the mathematical function includes: transforming the trigonometric calculation from the range [-π,π] to a signed 2's complementary representation in the range [-1,1].

[0020] Another aspect of the present disclosure is characterized by a method, comprising: obtaining primitive data of each of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; calculating a corresponding first electromagnetic field contribution of a first primitive of the plurality of primitives to each of a plurality of elements of a display screen; and calculating a corresponding second electromagnetic field contribution of a second primitive of the plurality of primitives to each of the plurality of elements of the display screen, wherein calculating the corresponding first electromagnetic field contribution from the first primitive is performed at least partially in parallel with calculating the corresponding second electromagnetic field contribution from the second primitive.

[0021] In some embodiments, calculating the first electromagnetic field contribution of the first primitive to the first element of the plurality of elements is performed in parallel with calculating the second electromagnetic field contribution of the second primitive of the plurality of primitives to the first element. The method may include: calculating the corresponding electromagnetic field contribution of each primitive of the plurality of primitives to each element of the plurality of elements. The calculation of the corresponding electromagnetic field contribution may be performed without at least one selected from the group consisting of: extending the geometry of the object to the plurality of elements, applying a visibility test before wrapping the wavefront, and decision or communication between parallel calculations for different primitives. Calculating the corresponding electromagnetic field contribution may be configured to facilitate at least one selected from the group consisting of: adjusting the parallel calculations for different primitives to achieve speed, cost, size or energy optimization, reducing the delay between initiating the drawing and the result being ready for display, using fixed-point representation to increase precision, and optimizing the calculation speed by optimizing mathematical functions.

[0022] In some implementations, the method further includes: representing the value using a fixed point representation during the computation. Representing the value using the fixed point representation may be performed without at least one selected from the group consisting of: floating point denormalization for gradual underflow, handling non-numeric values ​​resulting from operations including division by zero, changing a floating point rounding mode, and causing an operating system floating point exception.

[0023] In some embodiments, the method further comprises, for each element of the plurality of elements, accumulating the electromagnetic field contribution to the element by adding the corresponding second electromagnetic field contribution to the element to the corresponding first electromagnetic field contribution to the element.

[0024] In some embodiments, the method further includes: for each of the multiple elements, generating a corresponding control signal based on the sum of the electromagnetic field contributions of the multiple primitives to the element, the corresponding control signal being used to modulate at least one characteristic of the element based on the sum of the electromagnetic field contributions of the multiple primitives to the element.

[0025] In some embodiments, the method further includes scaling a first primitive adjacent to a second primitive by a predetermined factor so that a reconstruction of the first primitive does not overlap with a reconstruction of the second primitive. The predetermined factor may be determined at least in part based on a resolution of the display screen. The method may further include obtaining corresponding primitive data for each of the plurality of primitives, wherein the corresponding primitive data for each of the plurality of primitives includes corresponding coordinate information of the primitive in the 3D coordinate system; and determining new corresponding coordinate information of the first primitive based on the corresponding coordinate information of the first primitive and the predetermined factor. The method may further include determining an electromagnetic field contribution of the first primitive to each of the plurality of elements based on the new corresponding coordinate information of the first primitive. The method may further include scaling the second primitive by the predetermined factor. The first primitive and the second primitive may share a common portion, and scaling the first primitive includes scaling the common portion of the first primitive. Scaling the first primitive may include scaling the first primitive in a predetermined direction.

[0026] Another aspect of the present disclosure is characterized by a method comprising: obtaining primitive data of each of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; scaling the first primitive by a predetermined factor using corresponding primitive data of a first primitive adjacent to a second primitive and corresponding primitive data of the second primitive; and updating the corresponding primitive data of the first primitive based on a result of the scaling.

[0027] In some embodiments, the corresponding primitive data of each primitive among the multiple primitives includes corresponding coordinate information of the primitive in a 3D coordinate system, and updating the corresponding primitive data includes: determining new corresponding coordinate information of the first primitive based on the corresponding coordinate information of the first primitive and the predetermined factor.

[0028] In some embodiments, the predetermined factor is determined such that a reconstruction of the first primitive does not overlap with a reconstruction of the second primitive in the 3D space.

[0029] In some embodiments, the scaling is performed such that a gap between reconstructions of the first primitive and the second primitive in the 3D space is large enough to separate the first primitive and the second primitive to minimize overlapping effects, and small enough to make the reconstruction appear seamless.

[0030] In some implementations, the predetermined factor is determined based at least in part on a resolution of the display screen.

[0031] In some implementations, the method further comprises storing updated primitive data for the first primitive in a buffer.

[0032] In some implementations, the scaling is performed during a rendering process of the object to obtain primitive data for each of the plurality of primitives.

[0033] In some embodiments, the method further includes sending updated primitive data of the plurality of primitives to a controller, wherein the controller is configured to determine a corresponding electromagnetic field contribution of each of the plurality of primitives to each of a plurality of elements of a display screen based on the updated primitive data of the plurality of primitives.

[0034] In some embodiments, the method further includes determining an electromagnetic field contribution of the first primitive to each of a plurality of elements of a display screen based on the updated primitive data of the first primitive.

[0035] In some embodiments, the method further comprises scaling the second primitive by the predetermined factor.

[0036] In some embodiments, the first primitive and the second primitive share a common portion, and scaling the first primitive includes scaling the common portion of the first primitive.

[0037] In some implementations, scaling the first primitive includes scaling the first primitive in a predetermined direction.

[0038] In some embodiments, scaling the first primitive includes scaling a first portion of the first primitive by a first predetermined factor, and scaling a second portion of the second primitive by a second predetermined factor, wherein the first predetermined factor is different from the second predetermined factor.

[0039] Another aspect of the present disclosure is characterized by a method comprising: obtaining multiple discrete cosine transform (DCT) weights of an image on a specified surface of a specific primitive among multiple primitives corresponding to an object in a three-dimensional (3D) space to be mapped; and determining a corresponding electromagnetic field contribution of the specific primitive to each of multiple elements of a display screen by considering the influence of the multiple DCT weights of the image.

[0040] In some embodiments, the method further comprises: determining a resolution of the image to be mapped onto the designated surface of the particular primitive; and determining the plurality of DCT weights of the image based on the resolution.

[0041] In some embodiments, the method further comprises: decoding the DCT weights of the image to obtain a corresponding DCT magnitude of each pixel of the image.

[0042] In some embodiments, the method further comprises storing values ​​associated with corresponding DCT magnitudes of pixels of the image together with primitive data of the particular primitive. Determining the corresponding electromagnetic field contribution may comprise calculating the corresponding electromagnetic field contribution of the particular primitive to each of the plurality of elements using values ​​associated with corresponding DCT magnitudes of pixels of the image.

[0043] In some embodiments, the method further comprises selecting specific DCT terms to be included in determining the respective electromagnetic field contributions, each of the specific DCT terms having a respective DCT weight above a predetermined threshold.

[0044] Another aspect of the present disclosure is characterized by a method comprising: obtaining information of a given primitive and an occluder of the given primitive, wherein the given primitive belongs to one of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; and determining one or more specific elements among a plurality of elements of a display screen that do not contribute to the reconstruction of the given primitive due to the influence of the occluder.

[0045] In some implementations, the method further includes: storing information about the specific element and information about the given primitive and the occluder.

[0046] In some implementations, the determining is performed during a rendering process of the object for obtaining primitive data for the plurality of primitives.

[0047] In some embodiments, the method further comprises sending the stored information of the specific element and the information of the given primitive and the obstruction to a controller, the controller being configured to calculate electromagnetic field contributions of the plurality of primitives to the plurality of elements of the display screen.

[0048] In some embodiments, the method further comprises: for each of the specific elements, for each of the specific components, generating a sum of electromagnetic field contributions of the multiple primitives to the specific element by excluding the electromagnetic field contribution of the given primitive to the specific element.

[0049] In some embodiments, the method further comprises: for each element of the plurality of elements except the particular element, generating a sum of respective electromagnetic field contributions of the plurality of primitives to the element.

[0050] In some embodiments, the method further comprises: masking a contribution of the specific element to the reconstruction of the given primitive.

[0051] In some embodiments, determining the one or more specific elements includes: connecting the given primitive to an endpoint of the occluder; extending the connection to the display screen to determine an intersection between the connection and the display screen; and determining the elements within a specific range defined by the intersection as the specific elements that do not contribute to the reconstruction of the given primitive due to the influence of the occluder.

[0052] Another aspect of the present disclosure is characterized by a method comprising: obtaining information about a given primitive and an obstruction of the given primitive, wherein the given primitive belongs to one of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; and determining, for each of a plurality of elements of a display screen, a corresponding portion of the given primitive that does not contribute an electromagnetic field to the element due to the influence of the obstruction.

[0053] In some embodiments, the method further comprises: storing information of the corresponding portion of the given primitive and information of the given primitive and the occluder.

[0054] In some implementations, the determining is performed during a rendering process of the object for obtaining primitive data of the plurality of primitives.

[0055] In some embodiments, the method further comprises sending stored information of the corresponding portion of the given primitive and information of the given primitive and the obstruction to a controller, the controller being configured to calculate electromagnetic field contributions of the plurality of primitives to the plurality of elements of the display screen.

[0056] In some embodiments, the method further comprises masking an electromagnetic field contribution of the corresponding portion of the given primitive to each of the plurality of elements.

[0057] In some embodiments, the method further includes: for each element of the plurality of elements, generating a sum of electromagnetic field contributions of the plurality of primitives to the element by excluding the electromagnetic field contribution of the corresponding part of the given primitive to the element. Generating the sum of electromagnetic field contributions of the plurality of primitives to the element may include: subtracting the electromagnetic field contribution of the corresponding part of the given primitive to the element from the sum of electromagnetic field contributions of the plurality of primitives to the element without the influence of the obstruction. Generating the sum of electromagnetic field contributions of the plurality of primitives to the element may include: summing the electromagnetic field contributions of one or more other parts of the given primitive to the element, the corresponding part and the one or more other parts forming the given primitive.

[0058] In some embodiments, determining the corresponding portion of the given primitive that does not contribute to the electromagnetic field of the element due to the influence of the obstruction includes: connecting the element to an endpoint of the obstruction; determining an intersection between the connection and the primitive; and determining a specific portion of the given primitive surrounded by the intersection as the corresponding portion of the given primitive that does not contribute to the electromagnetic field of the element due to the influence of the obstruction.

[0059] Another aspect of the present disclosure is characterized by a method comprising: obtaining corresponding primitive data for each of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; obtaining corresponding geometric specular highlight information for each of the plurality of primitives; and storing the corresponding geometric specular highlight information and the corresponding primitive data for each of the plurality of primitives.

[0060] In some embodiments, the corresponding geometric specular highlight information of each primitive in the plurality of primitives includes: a reflectivity of a surface where the primitive is located according to a viewing angle.

[0061] In some embodiments, the method further comprises determining a respective electromagnetic field contribution of each of the plurality of primitives to each of a plurality of elements of a display screen by considering respective geometric specular highlight information of the primitives.

[0062] Another aspect of the present disclosure is characterized by a method, which includes: obtaining graphic data, the graphic data including corresponding primitive data of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; for each of the plurality of primitives, determining an electromagnetic field contribution to the element by calculating electromagnetic field propagation from the primitive to each of a plurality of elements of a display screen in a 3D coordinate system; for each of the plurality of elements, generating a sum of electromagnetic field contributions of the plurality of primitives to the element; for each of the plurality of elements, sending a corresponding control signal to the element, the control signal being used to modulate at least one characteristic of the element based on the sum of electromagnetic field contributions to the element; and sending a timing control signal to an illuminator to activate the illuminator so as to irradiate light onto the display screen, so that the modulated elements of the display screen cause the light to form a volumetric light field corresponding to the object.

[0063] Another aspect of the present disclosure is characterized by a method comprising: for each of a plurality of elements of a display screen, modifying a corresponding control signal using a predetermined calibration value; applying the modified corresponding control signal to each of the plurality of elements of the display screen; measuring the output of light incident on the display screen; and evaluating the predetermined calibration value based on the measurement of the output of the light.

[0064] In some embodiments, the predetermined calibration value is the same for each element of the plurality of elements.

[0065] In some embodiments, the method further includes: converting corresponding control signals of each of the plurality of components through a digital-to-analog converter (DAC), wherein changing the control signals of each of the plurality of components includes: changing the digital signals of the corresponding control signals using the predetermined calibration value.

[0066] In some implementations, the predetermined calibration value includes a plurality of bits.

[0067] In some embodiments, the method further comprises: adjusting the predetermined calibration value based on the result of the evaluation. Adjusting the predetermined calibration value may include: modifying one or more values ​​in the plurality of bits. Adjusting the predetermined calibration value may include: determining a combination of values ​​in the plurality of bits based on the predetermined calibration value and another calibration value determined according to a previous evaluation.

[0068] In some embodiments, the light output comprises a phase change of the light or an intensity difference between the light output and a background.

[0069] In some embodiments, the corresponding control signal of the element is determined based on the sum of electromagnetic field contributions to the element by a plurality of primitives corresponding to the object in the 3D space.

[0070] Another aspect of the present disclosure is characterized by a method comprising: obtaining, for each of a plurality of elements of a display screen, a sum of corresponding electromagnetic field contributions from a plurality of primitives in a three-dimensional (3D) space, the plurality of primitives corresponding to objects in the 3D space; applying a corresponding mathematical transformation to the sum of corresponding electromagnetic field contributions to the elements to obtain a sum of transformed corresponding electromagnetic field contributions to the elements; determining a corresponding control signal based on the sum of transformed corresponding electromagnetic field contributions to the elements; and modulating a characteristic of the element based on the corresponding control signal determined for the element.

[0071] In some embodiments, the method further includes: introducing light incident on the plurality of elements of the display screen; measuring a first output of the light; and adjusting one or more coefficients of a mathematical transformation of each of the plurality of elements based on the measurement of the first output of the light. The method may also include: changing the depth of the holographic pattern corresponding to the object according to the viewing angle of the display screen; measuring a second output of the light; and adjusting the one or more coefficients of the corresponding mathematical transformation based on the first output and the second output. The method may also include: changing the plurality of primitives corresponding to the first holographic pattern to a plurality of second primitives corresponding to the second holographic pattern; measuring the second output of the light; and adjusting the one or more coefficients of the corresponding mathematical transformation based on the first output and the second output. The first holographic pattern and the second holographic pattern may correspond to the object. The second holographic pattern may correspond to a second object that is different from the object associated with the first holographic pattern. The first output of the light may be measured by an imaging sensor. The imaging sensor may be configured to use a machine vision algorithm to determine what is being displayed and calculate a fitness parameter. The first holographic pattern and the second holographic pattern may each comprise a grid of dots, and wherein the fitness parameter is at least one selected from the group consisting of: how close the dots are, how centered the dots are, and how much distortion the dots have.

[0072] In some embodiments, the mathematical transformation is derived from Zernike polynomials.

[0073] In some embodiments, the mathematical transformation of the plurality of elements varies on an element-by-element basis.

[0074] In some embodiments, the method further includes: reproducing a sample set of known colors and intensities by illuminating the display screen; measuring the output light using a colorimeter device calibrated to an International Commission (CIE) standard observation curve; and defining the output light of the display screen in the International Commission CIE XYZ color space. The method may also include: determining the deviation of the defined output light value from the known standard value; and adjusting the output color on the display screen to bring it back into alignment.

[0075] Another aspect of the disclosure features a method that includes determining a cell gap of a liquid crystal (LC) display based on a pitch of display elements of the liquid crystal display; and calculating a minimum value of birefringence of a liquid crystal mixture based on the cell gap and a predetermined retardation of the liquid crystal display.

[0076] In some embodiments, the method further comprises: increasing a switching speed of the liquid crystal display while maintaining the birefringence of the liquid crystal mixture above a minimum value. Increasing the switching speed may include at least one item selected from the group consisting of: increasing the dielectric anisotropy of the liquid crystal mixture; and decreasing the rotational viscosity of the liquid crystal mixture.

[0077] In some embodiments, the liquid crystal display comprises a liquid crystal on silicon (LCOS) device having a silicon backplane.

[0078] In some embodiments, the liquid crystal display includes: a liquid crystal layer; a transparent conductive layer located on top of the liquid crystal layer and serving as a common electrode; and a backplane including a plurality of metal electrodes located at the bottom of the liquid crystal layer, wherein each of the plurality of metal electrodes is isolated from each other, and the backplane is configured to control the voltage of each of the plurality of metal electrodes.

[0079] Another aspect of the disclosure features a display screen that includes: a backplane; and a plurality of display elements on the backplane, wherein at least two of the plurality of display elements are of different sizes.

[0080] In some embodiments, the larger display element of the at least two display elements includes a buffer, and the smaller display element of the at least two display elements does not include a buffer. The larger display element can be connected to the first number of display elements through a wire, and the buffer is configured to cache a voltage applied to the wire so that the voltage is only applied to a second number of display elements within the first number of display elements, the number of the second number of display elements being less than the number of the first number of display elements.

[0081] In some embodiments, the buffer includes an analog circuit in the form of transistors or a digital circuit in the form of logic gates.

[0082] In some embodiments, the size distribution of the plurality of display elements is substantially equal to the size of the smaller display element of the at least two display elements.

[0083] In some embodiments, the display screen is configured as a liquid crystal on silicon device.

[0084] Another aspect of the disclosure features a display screen that includes: a backplane; and a plurality of display elements on the backplane, wherein at least two of the plurality of display elements have different shapes.

[0085] In some embodiments, the backplane includes a respective circuit for each display element, and a shape of the respective circuit for each of the at least two display elements corresponds to the different shapes of the at least two display elements.

[0086] In some embodiments, the size distribution of the plurality of display elements is substantially equal to the predetermined size.

[0087] In some embodiments, the display screen is configured as a liquid crystal on silicon device.

[0088] Another aspect of the present disclosure is characterized by a method, which includes: obtaining graphic data, the graphic data including corresponding primitive data of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; for each of the plurality of primitives, determining an electromagnetic field contribution to the element by calculating electromagnetic field propagation from the primitive to each of a plurality of elements of a display screen in a 3D coordinate system; for each of the plurality of elements, generating a sum of electromagnetic field contributions of the plurality of primitives to the element; for each of the plurality of elements, sending a corresponding control signal to the element, the control signal being used to modulate at least one characteristic of the element based on the sum of electromagnetic field contributions to the element; and sending a timing control signal to an illuminator to activate the illuminator so as to irradiate light onto the display screen, so that the modulated elements of the display screen cause the light to form a volumetric light field corresponding to the object.

[0089] Other embodiments of the aspects include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, all of which are configured to perform the actions of the method. One or more computer systems are configured to perform specific operations or actions, which means that software, firmware, hardware, or a combination thereof is installed on the system, which causes the system to perform the operation or action when they are run. One or more computer programs are configured to perform specific operations or actions, which means that the one or more programs include instructions that cause the device to perform the operation or action when executed by the data processing device.

[0090] Another aspect of the present disclosure is characterized by a device comprising: one or more processors; and a non-transitory computer-readable storage medium that communicates with the one or more processors and stores instructions that can be executed by the one or more processors, and the instructions, when executed, cause the one or more processors to perform one or more methods disclosed herein.

[0091] Another aspect of the disclosure features a non-transitory computer-readable storage medium storing instructions that are executable by one or more processors and that, when executed, cause the one or more processors to perform one or more methods disclosed herein.

[0092] Another aspect of the present disclosure is characterized by a display screen including a plurality of elements; and a controller coupled to the display screen and configured to perform one or more methods disclosed herein. The controller may include a plurality of computing units, each of which is configured to operate on one or more primitives of a plurality of primitives corresponding to an object in a three-dimensional (3D) space. In some embodiments, the controller is locally coupled to the display screen, and each of the computing units is coupled to one or more corresponding elements of the display screen and is configured to send a corresponding control signal to each of the one or more corresponding elements. The computing units may be configured to operate in parallel.

[0093] The controller may include at least one item selected from the group consisting of an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable gate array (PGA), a central processing unit (CPU), a graphics processing unit (GPU), and a standard computing unit. The display screen may include a spatial light modulator (SLM), the spatial light modulator including a digital micromirror device (DMD) or a liquid crystal on silicon (LCOS) device. The display screen may be configured for phase modulation, amplitude modulation, or phase and amplitude modulation. The controller may be coupled to the display screen via a storage buffer.

[0094] In some embodiments, the system includes: an illuminator disposed adjacent to the display screen and configured to emit light onto the display screen. The illuminator may be coupled to the controller and configured to be turned on / off based on a control signal from the controller.

[0095] In some cases, the luminaire is coupled to the controller via a memory buffer configured to control the amplitude or brightness of one or more light emitting elements in the luminaire. The size of the memory buffer for the luminaire can be smaller than the size of the memory buffer for the display screen. The number of light emitting elements in the luminaire can be smaller than the number of elements of the display screen. The controller can be configured to activate the one or more light emitting elements of the luminaire simultaneously.

[0096] The illuminator may be a coherent light source, a semi-coherent light source, or an incoherent light source. In some embodiments, the illuminator is configured to emit white light, and the display screen is configured to diffract the white light into light of different colors. In some embodiments, the illuminator includes two or more light-emitting elements, each configured to emit light of different colors. The controller may be configured to sequentially modulate the display screen using information associated with a first color during a first time period and information associated with a second color during a sequential second time period, and the controller may be configured to control the illuminator to sequentially turn on the first light-emitting element during the first time period to emit light of the first color and turn on the second light-emitting element during the second time period to emit light of the second color.

[0097] In some embodiments, the illuminator is arranged in front of the surface of the display screen and is configured to emit the light onto the surface of the display screen at an incident angle in a range between 0 degrees and 90 degrees, and the emitted light is reflected from the surface of the display screen. In some cases, the light emitted from the illuminator includes collimated light. In some cases, the light emitted from the illuminator includes divergent light. The light emitted by the illuminator includes semi-collimated light.

[0098] In some embodiments, the illuminator is disposed behind a rear surface of the display screen and is configured to emit divergent light onto the rear surface of the display screen, and the emitted light is transmitted through the display screen and out of the display screen from a front surface of the display screen.

[0099] In some embodiments, the illuminator includes: a light source configured to emit the light; and a waveguide coupled to the light source and arranged adjacent to the display screen, the waveguide configured to receive the light emitted from the light source and guide the emitted light to the display screen. In some cases, the light from the light source is coupled to the waveguide from a side cross section of the waveguide through an optical coupler. In some cases, the light source and the waveguide are integrated in a planar form and positioned on a surface of the display screen. The waveguide can be configured to guide the light to uniformly illuminate the display screen.

[0100] In some cases, the waveguide is positioned on a rear surface of the display screen, and the light is guided to be transmitted through the display screen and diffracted out of the display screen from a front surface of the display screen. The controller may be positioned on a rear surface of the waveguide. In some cases, the waveguide is positioned on a front surface of the display screen, and the light is guided to be incident on and reflected from the front surface of the display screen.

[0101] Another aspect of the present disclosure is characterized by a system, comprising: a display screen, comprising an array of elements; and an integrated circuit, comprising an array of computing units, each computing unit coupled to one or more corresponding elements of the display screen and configured to: calculate an electromagnetic field contribution of at least one primitive from a plurality of primitives to each element in the array of elements; and generate, for each of the one or more corresponding elements, a sum of the corresponding electromagnetic field contributions of the plurality of primitives to the element.

[0102] Each of the computing units can be configured to: receive the calculated electromagnetic field contributions of other primitives in the multiple primitives to each of the one or more corresponding elements from other computing units in the computing unit array; and for each of the one or more corresponding elements, generate the sum of the corresponding electromagnetic field contributions by adding the received calculated electromagnetic field contributions of the other primitives to the element.

[0103] Each of the computational units may be configured to generate a respective control signal for each of the one or more respective elements to modulate at least one property of the element based on a sum of respective electromagnetic field contributions to the element.

[0104] In some embodiments, the integrated circuit includes a corresponding accumulator configured to store the accumulated results of the calculated electromagnetic field contributions of the plurality of primitives to each element of the display screen. The integrated circuit can be configured to clear the accumulator at the beginning of the calculation operation. In some examples, the integrated circuit includes a corresponding storage buffer for each of the elements, and the integrated circuit can be configured to accumulate the calculated electromagnetic field contributions of the plurality of primitives to the element to obtain the sum of the corresponding electromagnetic field contributions as the final accumulated result in the corresponding accumulator, and pass the final accumulated result from the corresponding accumulator to the corresponding storage buffer for the element.

[0105] In some embodiments, the system further includes an illuminator positioned between the integrated circuit and the display screen and configured to receive a control signal from the integrated circuit and to illuminate light onto the display screen based on the control signal, wherein the integrated circuit, the illuminator, and the display screen may be integrated into a single unit.

[0106] Another aspect of the disclosure features a system comprising: a computing device configured to generate data including corresponding primitive data of a plurality of primitives corresponding to an object in a three-dimensional (3D) space; and a system as disclosed herein. The system is configured to receive the graphics data from the computing device and process the graphics data to render the object in the 3D space. The computing device may include an application programming interface (API) configured to create the primitive using the corresponding primitive data by rendering a computer generated (CG) model of the object.

[0107] In this disclosure, the term "primitive" refers to a basic indivisible element used for input and output within a computing system. The element can be a geometric element or a graphical element. The term "hologram" refers to a pattern displayed on a display screen that contains amplitude information or phase information about an object, or a combination thereof. The term "holographic reconstruction" refers to a volumetric light field (e.g., a holographic light field) from an illuminated display screen.

[0108] The details of one or more implementations of the subject matter herein are set forth in the accompanying drawings and the associated description. Other features, aspects, and advantages of the subject matter will be apparent from the description, drawings, and claims.

[0109] It should be understood that various aspects of the embodiments may be combined in different ways. As an example, features of certain methods may be combined with features of other methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1A A schematic diagram of an exemplary system including a holographic display is shown.

[0111] Figure 1B A schematic diagram of an exemplary holographic display is shown.

[0112] Figure 1C An exemplary system for 3D display is shown.

[0113] Figure 2 An exemplary configuration for electromagnetic (EM) propagation calculations is shown.

[0114] Figure 3A Exemplary electromagnetic propagation of a point primitive relative to a display element is shown.

[0115] Figure 3B Exemplary electromagnetic propagation of line primitives relative to display elements is shown.

[0116] Figure 3C Exemplary electromagnetic propagation of a triangle primitive relative to display elements is shown.

[0117] Figure 3D An exemplary implementation of Maxwell holographic occlusion using point primitives with line primitives as occluders is shown.

[0118] Figure 3E An exemplary implementation of Maxwell holographic occlusion in which a line primitive uses another line primitive as an occluder is shown.

[0119] Figure 3F An exemplary implementation of Maxwell holographic occlusion of triangle primitives with line primitives as occluders is shown.

[0120] Figure 3G An exemplary implementation of a Maxwell holographic bond is shown.

[0121] Figure 4 is a flow chart of an exemplary process for displaying a 3D object.

[0122] FIG. 5A to FIG. 5F An embodiment of an exemplary system for 3D display is shown.

[0123] Fig. 6A An exemplary display screen having display elements of non-uniform shapes is shown.

[0124] Figure 6B Exemplary display screens having display elements of varying sizes are shown. DETAILED DESCRIPTION

[0125] Embodiments of the present disclosure are characterized by a technique for enabling 3D display of complex computer-generated scenes as real holograms. The technique provides a new and deterministic solution to real-time dynamic computational holography based on Maxwell's equations for electromagnetic fields, which can be expressed as Maxwell holography. The calculations in Maxwell holography can be expressed as Maxwell holographic calculations. In an embodiment, the present disclosure utilizes tools including field theory, topological structures, analytical continuation, and / or symmetry groups to treat holograms as Dirichlet or Cauchy boundary condition problems for general electric fields, which enables real-time solution of holograms without the limitations of traditional holographic systems. In an embodiment, the technique can be used to produce phase-only, amplitude-only, or phase and amplitude holograms using a spatial light modulator (SLM) or any other holographic device.

[0126] Embodiments of the present disclosure may provide: 1) a mechanism for approximating holograms as electromagnetic boundary conditions using field theory and contact geometry rather than classical optics; 2) computer code and application programming interface (API) for deriving and implementing the electromagnetic boundary condition method for computational holography, i.e., implementing hologram computations as 2D analytical functions of the hologram plane and subsequently discretizing into a parallel algorithm; and / or 3) implementing a complete set of full 3D holographic versions of standard computer graphics primitives (e.g., points, lines, triangles, and texture triangles), which may enable full compatibility with standard existing computer graphics tools and techniques. The techniques may enable devices to display existing general-purpose content that was not created specifically for holography, and at the same time allow existing content creators to create holographic works without having to learn special techniques or use special tools.

[0127] Specifically, the technology may involve using mathematical formulas (or expressions) of light as electromagnetic (EM) phenomena to replace mathematical formulas of classical optics commonly used in computational holography, such as the Gerchberg-Saxton (GS) model. The mathematical formulas disclosed herein are derived from Maxwell's equations. In an embodiment, the technology disclosed herein involves treating the displayed image as an electromagnetic field and treating the hologram as a boundary value condition for generating the electromagnetic field (e.g., the Dirichlet problem). In addition, the desired image can be constructed using primitive paradigms that are ubiquitous in computer graphics, allowing, for example, the technology to be used to display any 3D image as a holographic reconstruction (e.g., a holographic light field) rather than a projected image on a 2D screen. Compared to deep point cloud technology that suffers from bandwidth limitations, the technology can avoid these limitations and use any suitable type of primitives, such as point primitives, line primitives, or polygonal primitives (such as triangle primitives). In addition, the primitives can be rendered using color information, texture information, and / or shading information. This can help implement recording and compression schemes for CG holographic content including live holographic video.

[0128] In an embodiment, the technique uses Maxwell's equations to compute the generated hologram as a boundary condition problem of electromagnetic field modeling, which can eliminate the reliance on fast Fourier transforms (FFTs) and their inherent limitations, eliminate the reliance on collimated light sources and lasers, and / or eliminate the limitations of previous methods and non-deterministic solutions of computational holography.

[0129] In an embodiment, the technique may be optimized for computational simplicity and speed through a mathematical optimization process that constrains independent inputs to the surface of the hologram, depending on the parameters of the computer generated (CG) primitives needed to build the scene. This allows the work to be performed in a highly parallel and highly optimal manner in a computing architecture (e.g., application specific integrated circuits (ASICs) and multi-core architectures). The processing of the computational hologram may be considered as a single instruction executed on the input data in the form of a computer generated image (CGI) scene, and may theoretically be completed in a single clock cycle of each CGI primitive.

[0130] In an embodiment, the technique views the holographic scene as components of full 3D holographic primitive apertures that are functionally compatible with standard primitives for conventional 3D graphics, as employed, for example, in video games, movies, television, computer display screens, or any other computational display technology. The technique may enable efficient implementation of these aperture primitives in hardware and software without the limitations inherent in standard implementations of computational holography. The amplitude and color of the primitives may be calculated automatically. The computational complexity may increase linearly with the number of phase elements n, compared to n^2 or n*log(n) in standard computational holography. The created image is fully 3D and not a collection of planar images, and the technique does not require iterative amplitude corrections with an unknown number of steps. In addition, the generated hologram does not have a "conjugate" image occupying space on the holographic device.

[0131] Because holographic primitives are part of a special set of mathematical objects, they are relatively simple and relatively fast to compute, and they are uniquely suited for parallel, distributed computing methods. The computability and parallelism can allow interactive computation of large holograms, thereby designing large-area holographic devices of theoretically unlimited size that can act as holographic computer display screens, phone display screens, home theaters, and even holographic rooms. In addition, holograms can fill large areas with light, for example, rendering large colored areas in 3D, without the limitations associated with conventional holographic computing methods, which can cause elements to appear as outlines rather than solid. Moreover, the relatively simple and relatively fast computations allow real-time holograms to be displayed at interactive speeds that are not constrained by n^2 computational loads and iterative amplitude corrections.

[0132] In embodiments, the technology can achieve natural computability on modern ASIC and multi-core architectures, and can achieve full compatibility with modern graphics hardware, modern graphics software, and / or modern graphics tools and tool chains. For example, the technology can achieve a clear and simple holographic API, and use common standard 3D content creation tools (e.g., or Unity3D) to achieve high-performance rendering of arbitrary CG models. The API enables developers or users to interact with holographic devices (e.g., light modulators or holographic systems). The holographic API can create computer graphics primitives as discrete holographic scene primitives, allowing rich holographic content to be generated using general and specially designed holographic computing hardware. The creation of mathematical and computational architectures allows holograms to be rendered using tools and techniques used to produce conventional 3D content and software applications. Optimization of mathematical and computational architectures allows executable embodiments of conventional graphics and rendering to be displayed as holographic reconstructions.

[0133] The algorithms in the described techniques are relatively easy to implement in hardware. This not only allows the computational speeds required for high-quality, modern rendering that users expect, but also allows the algorithms to be implemented in relatively simple circuits (e.g., ASIC gate structures that are part of a holographic device). As a result, bandwidth issues that can plague high-density display screens may become irrelevant because the calculation of the scene can be distributed in the computing architecture built into the display device (e.g., built-in computing), rather than having to be calculated remotely and then written to each pixel of the display screen for each frame of content. This also means that the number of display elements, and therefore the size of the holographic display screen, can be relatively unconstrained by the constraints that limit other technologies.

[0134] The technology can enable a variety of interactive technologies using structured light to be relatively simple and relatively cheap to implement in different applications, including, for example, solid-state light detection and ranging (LiDar) devices, 3D printing, smart illuminators, smart microdisplays, or any other application requiring structured light. The technology can also be used for optical simulations, such as for grating simulations.

[0135] Figure 1AA schematic diagram of an exemplary system 100 for 3D display is shown. The system 100 includes a computing device 102 and a holographic display device (or Maxwell holographic display device) 110. The computing device 102 is configured to prepare data of a primitive list corresponding to an object, such as a 3D object, and send the data to the holographic display device 110 via a wired or wireless connection (e.g., a USB-C connection or any other high-speed serial connection). The holographic display device 110 is configured to calculate the electromagnetic field contribution of the primitive list to the display element (e.g., a modulator) of the display screen in the holographic display device 110, modulate the display element using a pattern such as a hologram based on the calculated electromagnetic field contribution to the display screen, and display the light field corresponding to the object in 3D form under illumination, such as a holographic reconstruction. Here, a hologram refers to a pattern displayed on a display screen, the pattern containing amplitude information or phase information about the object, or a combination thereof. A holographic reconstruction refers to a volumetric light field (e.g., a holographic light field) from the display screen when the holographic screen is illuminated.

[0136] Computing device 102 may be any suitable type of device, such as a desktop computer, a personal computer, a notebook, a tablet computing device, a personal digital assistant (PDA), a network device, a smart mobile phone, a smart watch, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or any suitable combination of any two or more of these or other computing devices.

[0137] The computing device 102 includes an operating system (OS) 104, which may include a plurality of applications 106 as a graphics engine. The applications 106 may use standard 3D content creation tools (e.g., or Unity3D) to process or render a scene, for example, any arbitrary CG model. The scene may correspond to a 3D object. Applications 106 may operate in parallel to render the scene to obtain an OS graphics abstraction 101, which may be provided to a graphics processing unit (GPU) 108 for further processing. In some embodiments, the OS graphics abstraction 101 is provided to a holographic display device 110 for further processing.

[0138] The GPU 108 may include specialized electronic circuits designed to rapidly manipulate computer graphics and image processing. The GPU 108 may process the OS graphics abstraction 101 of the scene to obtain processed scene data 103, which may be used to obtain a primitive list 105 indexed in a specific order. The primitive may include at least one of a point primitive, a line primitive, or a polygon primitive. In some embodiments, the GPU 108 includes a video driver configured to generate the processed scene data 103 and the primitive list 105.

[0139] In some embodiments, GPU 108 includes a traditional renderer 120 that can render primitive list 105 as a list of items for drawing on a traditional monitor 124, such as a 2D display screen, using traditional rendering techniques such as culling and clipping. The list of items can be sent to traditional monitor 124 via screen buffer 122.

[0140] In some embodiments, the GPU 108 includes a holographic renderer 130 for rendering the primitive list 105 into graphic data to be displayed by the holographic display device 110. The graphic data may include a primitive list and corresponding primitive data. For example, the graphic data may include a hexadecimal code for each primitive.

[0141] In some embodiments, the GPU 108 includes both a traditional renderer 120 and a holographic renderer 130. In some embodiments, the GPU 108 includes a traditional renderer 120 and the holographic display device 110 includes the holographic renderer 130.

[0142] The corresponding primitive data of the primitive may also include color information (e.g., textured color, gradient color, or both), texture information, and / or shading information. The shading information may be obtained by any conventional CGI surface shading method involving modulating the color or brightness of the primitive surface.

[0143] The primitive data of the primitive may include coordinate information of the primitive in a 3D coordinate system such as a Cartesian coordinate system XYZ, a polar coordinate system, a cylindrical coordinate system, and a spherical coordinate system. As further described below, the display elements in the holographic display device 110 may also have corresponding coordinate information in the 3D coordinate system. The primitive at the coordinate position may represent a 3D object adjacent to the display element (e.g., in front of the display element).

[0144] As an example, a primitive is a shaded line, e.g., a straight line that changes smoothly from one color to another across its span. A primitive requires four data elements to render: two endpoints and color information (e.g., RGB color values) at each endpoint. Assume that the hexadecimal code for the line is A0. In a 3D coordinate system, the line extends from a first endpoint (0.1, 0.1, 0.1) to a second endpoint (0.2, 0.2, 0.2). The color at the first endpoint is 1 / 2 blue: RGB = (0, 0, 128), and the color at the second endpoint is full red: RGB = (255, 0, 0). The holographic renderer determines how much data and what type of data to expect for each primitive. For the line, the primitive data for the shaded line in the primitive stream may be the following set of instructions:

[0145] 0xA0 / / hex code for the shaded line

[0146] 0x3dcccccd / / first vertex at(0.1,0.1,0.1)float(single)

[0147] 0x3dcccccd

[0148] 0x3dcccccd

[0149] 0x000080 / / first vertex color is(0,0,128)

[0150] 0x3e4ccccd / / second vertex at(0.2,0.2,0.2)float(single)

[0151] 0x3e4ccccd

[0152] 0x3e4ccccd

[0153] 0xff0000 / / second vertex color is(255,0,0)

[0154] There are a total of 31 hexadecimal words in the primitive data for the shaded line primitive. Therefore, this can be an extremely efficient way to transmit complex scenes, and the primitive data can be further compressed. Because each primitive is a deterministic Turing step, no terminator is required. Unlike traditional models that simply draw line primitives on a 2D display, the primitive data for the line is sent to the holographic display device 110, which can calculate the hologram and display the corresponding holographic reconstruction that presents the line floating in space.

[0155] In some embodiments, the computing device 102 sends non-primitive-based data, such as recorded light field video, to the holographic display device 110. The holographic display device 110 can calculate sequential holograms to display the video as a sequential holographic reconstruction in space. In some embodiments, the computing device 102 sends the CG holographic content to the holographic display device 110 simultaneously with the live holographic content. The holographic display device 110 can also calculate corresponding holograms to display the content as corresponding holographic reconstructions.

[0156] like Figure 1AAs shown, the holographic display device 110 includes a controller 112 and a display screen 114. The controller 112 may include a plurality of computing units or processing units. In some embodiments, the controller 112 includes an ASIC, a field programmable gate array (FPGA) or a GPU, or any combination thereof. In some embodiments, the controller 112 includes a holographic renderer 130 for rendering the primitive list 105 as graphic data to be calculated by the computing unit. In some embodiments, the controller 112 receives the OS graphic abstraction 101 from the computing device 102 for further processing. The display screen 114 may include a plurality of display elements. In some embodiments, the display screen 114 includes a spatial light modulator (SLM). The SLM may be a phase SLM, an amplitude SLM, or a phase and amplitude SLM. In some examples, the display screen 114 is a digital micromirror device (DMD) or a liquid crystal on silicon (LCOS) device. In some embodiments, the holographic display device 110 includes an illuminator 116 adjacent to the display screen 114 and configured to emit light toward the display screen 114. The illuminator 116 may be a coherent light source such as a laser, a semi-coherent light source such as an LED (light emitting diode), or an incoherent light source.

[0157] Unlike conventional 3D graphics systems that take a 3D scene and project it onto a 2D display device, the holographic display device 110 is configured to produce a 3D output, such as a holographic reconstruction 117 in the form of a light field, e.g., a color 3D volume. In a hologram, each display element contributes to each portion of the scene. That is, for the holographic display device 110, each display element needs to be modulated for each portion of the scene (e.g., each primitive in a primitive list generated by the GPU 108) for a complete reproduction of the scene. In some embodiments, modulation of certain display elements may be omitted based on, for example, an acceptable level of accuracy in reproducing the scene.

[0158] In some embodiments, the controller 112 is configured to calculate the electromagnetic field contribution (e.g., phase, amplitude, or both) of each primitive to each display element, and generate a list of primitives for each display element to sum the electromagnetic field contributions of the display element. This can be done by traversing each primitive for a given display element and accumulating its contribution to the given display element, or by traversing each display element for each primitive.

[0159] The controller 112 may calculate the electromagnetic field contribution of each primitive to each display element based on a predetermined expression for the primitive. Different primitives may have corresponding expressions. In some cases, the predetermined expression is an analytical expression, as follows FIG. 3A to FIG. 3CDetailed further herein. In some cases, the predetermined expression is determined by solving Maxwell's equations using boundary conditions defined at display screen 114. The boundary conditions may include a Dirichlet boundary condition or a Cauchy boundary condition. The display element may then be modulated based on the sum of the electromagnetic field contributions, for example, by modulating at least one of a refractive index, an amplitude index, a birefringence, or a hysteresis of the display element.

[0160] If the value of the electromagnetic field at each point on the surface bounding the field is known (e.g., a solution to Maxwell's equations), then an accurate, unique configuration of the electromagnetic field within the volume bounded by the boundary surface can be determined. The list of primitives (or a holographic reconstruction of the corresponding hologram) and the display screen 114 define a 3D space, and the surface of the display screen 114 forms part of the boundary surface of the 3D space. By setting an electromagnetic field state (e.g., a phase or a phase state and an amplitude state) on the surface of the display screen 114, for example by shining light on the display screen surface, the boundary conditions of the electromagnetic field can be determined. Due to the time symmetry of Maxwell's equations, when the display elements are modulated based on the electromagnetic field contribution from the primitives corresponding to the hologram, a volumetric light field corresponding to the hologram as a holographic reconstruction can be obtained.

[0161] For example, a line primitive illuminated with a specific color may be provided in front of the display screen 114. Figure 3B As further detailed, the analytical expression of the linear aperture can be written as a function in space. The electromagnetic field contribution from the line primitive on the boundary surface including the display screen 114 can then be determined. If the electromagnetic field value corresponding to the calculated electromagnetic field contribution is set on the display screen 114, then due to the time symmetry of Maxwell's equations, the same linear aperture used in the calculation can appear at the corresponding position, such as the coordinate position of the line primitive in the 3D coordinate system.

[0162] In some examples, the following Figure 3BFurther details, assume that there is a line of light between two points A and B in 3D space. The light is emitted uniformly and the light intensity per line length l is I. At each differential dl along the line from A to B, an amount of light proportional to I*dl is emitted. The infinitesimal dl can act as a delta (point) source, and the electromagnetic field contribution of the differential dl to any point on the boundary surface around the scene corresponding to the primitive list can be determined. Therefore, for any display element on the display screen 114, an analytical equation representing the electromagnetic field contribution of the differential segment of the line to the display element can be determined. A specific type of summation / integration of the electromagnetic field contribution of the electromagnetic field at the display element on the display screen that travels along the line and accumulates the entire line can be determined as an expression. The value corresponding to the expression at the display element can be set, for example, by modulating the display element and irradiating the display element. Then, by time reversal and correction constants, the line can be created at the same position defined by point A and point B in 3D space.

[0163] In some embodiments, the controller 112 is coupled to the display screen 114 via a storage buffer. The control signal 112 can generate a corresponding control signal based on the sum of the electromagnetic field contributions to each display element. The control signal is used to modulate the display element based on the sum of the electromagnetic field contributions. The corresponding control signal is sent to the corresponding display element via the storage buffer.

[0164] In some embodiments, the controller 112 is integrated with and locally coupled to the display screen 114. Figure 1B In further detail, the controller 112 may include a plurality of computing units, each coupled to one or more corresponding display elements and configured to send a corresponding control signal to each of the one or more corresponding display elements. Each computing unit may be configured to perform a calculation on one or more primitives in the primitive list. The computing units may operate in parallel.

[0165] In some embodiments, the illuminator 116 is coupled to the controller 112 and is configured to be turned on / off based on a control signal from the controller 112. For example, the controller 112 may activate the illuminator 116 to turn on in response to the controller 112 completing a calculation (e.g., obtaining the sum of all electromagnetic field contributions to the display elements). As described above, when the illuminator 116 emits light onto the display screen 114, the modulated display elements cause the light to propagate in different directions to form a volumetric light field corresponding to the primitive list, where the primitive list corresponds to the 3D object. The resulting volumetric light field corresponds to a solution to Maxwell's equations with boundary conditions defined by the modulated display elements of the display screen 114.

[0166] In some embodiments, the controller 112 is coupled to the illuminator 116 via a storage buffer. The storage buffer can be configured to control the amplitude or brightness of the light-emitting elements in the illuminator. The storage buffer for the illuminator 116 can be smaller in size than the storage buffer for the display screen 114. The number of light-emitting elements in the illuminator 116 can be less than the number of display elements of the display screen 114, as long as the light from the light-emitting elements can be illuminated on the entire surface of the display screen 114. For example, an illuminator with 64x64 OLEDs (organic light-emitting diodes) can be used for a display screen with 1024x1024 elements. The controller 112 can be configured to activate multiple light-emitting elements of the illuminator 116 simultaneously.

[0167] In some embodiments, the illuminator 116 is a monochromatic light source, which is configured to emit monochromatic light, such as red light, green light, or blue light. In some embodiments, the illuminator 116 includes two or more light-emitting elements, each of which is configured to emit light with different colors. For example, the illuminator 116 may include red, green, and blue light-emitting elements. In order to display a full-color 3D object, three separate red, green, and blue holograms can be calculated. That is, three electromagnetic field contributions of the corresponding primitives to the display elements can be obtained. The display elements can be modulated sequentially based on the three electromagnetic field contributions, and the illuminator 116 can be controlled to sequentially turn on the red, green, and blue light-emitting elements. Relying on the temporal coherence of the visual effects in the viewer's eyes, the three colors can be combined in the eyes to present full color. In some cases, the illuminator 116 is turned off during the state change of the display image (or holographic reconstruction) and turned on when the effective image (or holographic reconstruction) is presented for a period of time. This can also rely on the temporal coherence of vision to make the image (or holographic reconstruction) appear stable.

[0168] In some embodiments, display screen 114 has a resolution small enough to diffract visible light, for example, having a resolution on the order of 0.5 μm or less. Illuminator 116 may include a single white light source, and the emitted white light may be diffracted by display screen 114 into different colors for holographic reconstruction.

[0169] As follows FIG. 5A to FIG. 5FAs further detailed, there may be different configurations for the system 100. The display screen 114 may be reflective or transmissive. The display screen 114 may have a variety of sizes ranging from small scales (e.g., 1-10 cm on a side) to large scales (e.g., 100-1000 cm on a side). The illumination from the illuminator 116 may come from the front surface of the display screen 114 (e.g., for a reflective display screen) or from the back surface of the display screen 114 (e.g., for a transmissive display screen). A planar waveguide may be used to uniformly illuminate the surface of the display screen 114. In some embodiments, the controller 112, the illuminator 116, and the display screen 114 may be integrated together into a single unit. The integrated single unit may include, for example, a holographic renderer 130 in the controller 112.

[0170] Figure 1B Schematic diagram of an exemplary holographic display device 150 is shown. The holographic display device 150 may be similar to Figure 1A The holographic display device 110 includes a computing architecture 152 and a display screen 156. The computing architecture 152 may be similar to Figure 1A The computing architecture 152 may include an array of parallel computing cores 154. The computing cores may be connected to adjacent computing cores via communication connections 159 (e.g., USB-C connections or any other high-speed serial connections). The communication connections 159 may be included in a data distribution network through which scene data 151 (e.g., scene primitives) may be distributed between the computing cores 154.

[0171] The display screen 156 may be similar to Figure 1A The display screen 114 may include an array of display elements 160 located on a backplane 158. The display elements 160 may be arranged on the front surface of the backplane 158, and the computing core 154 may be arranged on the rear surface of the backplane 158. The backplane 158 may be a substrate, such as a wafer. The computing core 154 may be on the same substrate as the display screen 156 or connected to the rear surface of the display screen 156.

[0172] Each computing core 154 may be connected to a corresponding tile (or array) of display elements 160. Each computing core 154 is configured to perform calculations on each of the plurality of primitives in the scene data 151 in parallel with each other. In some examples, the computing core 154 is configured to calculate the electromagnetic field contribution of each corresponding primitive to each display element in the array of display elements 160, and to generate a sum of the electromagnetic field contributions of the plurality of primitives to each display element in the corresponding tile of the display elements 160. The computing core 154 may receive the calculated electromagnetic field contributions of the other primitives in the plurality of primitives to each display element in the corresponding tile of the display elements 160 from the other computing cores in the array of computing cores 154, and generate the sum of the electromagnetic field contributions based on the received calculated electromagnetic field contributions. The computing core 154 may generate a control signal for each display element in the corresponding tile of the display elements to modulate at least one characteristic of each display element in the corresponding tile of the display elements 160 based on the sum of the electromagnetic field contributions to the display elements.

[0173] As described above, the computing architecture 152 can also generate a control signal to the illuminator 162, for example, in response to determining that the calculation of the sum of the electromagnetic field contributions of multiple primitives to each display element has been completed. The illuminator 162 emits input light 153 to illuminate the modulated display element 160, and the input light 153 is reflected by the modulated display element 160 to form a volumetric light field corresponding to the scene data 151, for example, a holographic light field 155.

[0174] like Figure 1B As shown, the tiles of display elements 160 can be interconnected to form a larger display screen. Accordingly, the computing cores 154 can be interconnected for data communication and distribution. Note that the parameter that varies in the holographic calculation between any given two display elements is their physical location. Therefore, the task of calculating the hologram can be evenly shared between the corresponding computing cores 154, and the entire display 150 can operate at the same speed as a single tile, that is, regardless of the number of tiles.

[0175] Figure 1C An exemplary system 170 for displaying objects in 3D space is shown. System 170 may include, for example Figure 1A The computing device 102 and the holographic display device 172, for example, Figure 1A Holographic display 110 or Figure 1B The user may use an input device (e.g., keyboard 174 and / or mouse 176) to operate system 170. For example, the user may create CG models for 2D object 178 and 3D object 180 through the computing device. The computing device or holographic display device 172 may include a holographic renderer, for example, Figure 1AThe holographic renderer 130 of the embodiment of the present invention is used to render the CG model to generate corresponding graphic data for the 2D object 178 and the 3D object 180. The graphic data may include corresponding primitive data of the primitive list corresponding to the objects 178 and 180.

[0176] The holographic display device 172 may include a controller, such as Figure 1A The controller 112 or Figure 1B The controller 152, and the display screen 173, for example Figure 1A Display screen 114 or Figure 1B The controller may calculate the sum of the corresponding electromagnetic field contributions of the primitives to each display element of the display screen 173, and generate a control signal for modulating each display element based on the sum of the corresponding electromagnetic field contributions. The holographic display device 172 may also include an illuminator, for example, Figure 1A Illuminator 116 or Figure 1B The controller may generate a timing control signal to activate the illuminator. When light from the illuminator is irradiated on the surface of the display screen 173, the modulated display elements may cause the light to propagate in the 3D space to form a volumetric light field corresponding to the holographic reconstruction for the 2D object 178 and the holographic reconstruction for the 3D object 180. Therefore, the 2D object 178 and the 3D object 180 are displayed as corresponding holographic reconstructions floating in the 3D space in front of the display screen 173.

[0177] In some embodiments, the computing device sends non-primitive-based data (e.g., recorded light field video) to the holographic display device 172. The holographic display device 172 can calculate and generate corresponding holograms (e.g., a series of sequential holograms) to display as corresponding holographic reconstructions in 3D space. In some embodiments, the computing device sends CG holographic content to the holographic display device 172 simultaneously with the live holographic content. The holographic display device 172 can also calculate and generate corresponding holograms to display the content as corresponding holographic reconstructions in 3D space.

[0178] Figure 2An exemplary configuration 200 for electromagnetic (EM) field calculations is shown. A display screen 202 (e.g., an LCOS device) having an array of display elements 204 and a primitive list including point primitives 206 are provided in a 3D space 208. The 3D space 208 includes a boundary surface 210. The point primitives 206 have coordinate information (x, y, z) in a 3D coordinate system XYZ. Each display element 204 is located in a plane relative to other display elements 204 and has a 2D position (u, v). The display element 204 also has a position in the 3D space. By mathematical point transformation, the 2D position (u, v) can be transformed into six coordinates 250 in the 3D coordinate system. That is, the surface of the display screen 202 forms a portion of the boundary surface 210. Therefore, the electromagnetic field contribution of the primitive list calculated by defining the boundary conditions at the surface of the display screen 202 represents a portion of the total electromagnetic field contribution of the primitive to the display element. A scaling factor (eg, six) may be multiplied by the sum of electromagnetic field contributions for each display element to obtain a scaled sum of electromagnetic field contributions, and the display elements may be modulated based on the scaled sum of electromagnetic field contributions.

[0179] Example electromagnetic field contributions of primitives

[0180] Primitives can be used in standard computer graphics rendering. Each type of primitive in standard computer graphics corresponds in this process to a discrete mathematical function that defines a single holographic primitive that is added to the graphical elements of the hologram. Each type of primitive can correspond to an expression for calculating the electromagnetic field contribution to the display element. The primitive can be a point primitive, a line primitive, or a polygon (e.g., a triangle) primitive. As shown below, an analytical expression can be derived by calculating the electromagnetic field propagation from the corresponding primitive to the display element of the display screen.

[0181] Figure 3A An exemplary electromagnetic propagation from a point primitive 304 to a display element 302 of a display screen 300 is shown. In a 3D coordinate system XYZ, the z coordinate is assumed to be 0 on the display screen 300, which means that z values ​​are negative behind the display screen 300 and positive in front of the display screen 300. The coordinates of the point primitive 304 are (x, y, z), and the coordinates of the display element 302 are (u, v, 0). The distance d between the point primitive 304 and the display element 302 is uv can be determined based on their coordinates.

[0182] The point element 304 can be regarded as a point charge with a time-varying amplitude. According to electromagnetic theory, the electric field E generated by such a point charge can be expressed as:

[0183]

[0184] Among them, λ represents the wavelength of the electromagnetic wave, and d represents the distance from the point charge.

[0185] Therefore, the electric field E at the element (u, v) is shown u,v It can be expressed as:

[0186]

[0187] Where I represents the relative strength of the holographic primitive electric field contributed by the dot primitive 304 at the display element.

[0188] As described above with respect to FIG. 1 , the surface of display screen 300 forms only a portion of the boundary surface of the electromagnetic field. A scaling factor may be applied to the electric field E uv To obtain the scaled electric field at the display element adjusted for the portion boundary

[0189]

[0190] in

[0191] Figure 3B An example of electromagnetic propagation from a line primitive 306 to a display element 302 of a display screen 300 in a 3D coordinate system XYZ is shown. As described above, the coordinates of the display element 302 may be (u, v, 0), where z = 0. The two endpoints P of the line primitive 306 are 0 and P 1 The coordinates of are (x0, y0, z0) and (x1, y1, z1). The endpoint P 0 The distance d from the display element 0 can be determined based on their coordinates. Similarly, the endpoint P 1 The distance d from the display element 1 The two endpoints P can be determined based on their coordinates. 0 With P 1 The distance between 01 It can also be determined, for example, that d 01 =d 1 -d 0 .

[0192] As described above, the line primitives can be viewed as superpositions or linear deformations, and the corresponding analytical expression of the line primitive as a linear aperture can be obtained as a distributed incremental function in space. The analytical expression can be a closed expression of a continuous 3D line segment as a hologram.

[0193] Figure 3C An exemplary electromagnetic propagation from a triangle primitive 308 to a display element 302 of a display screen 300 in a 3D coordinate system XYZ is shown. As described above, the coordinates of the display element 302 may be (u, v, 0), where z=0. The triangle primitive 308 has three endpoints: P0 (x 0 ,y 0 ,z 0 ), P 1 (x 1 ,y 1 ,z 1 ) and P 2 (x 2 ,y 2 ,z 2 ). Display elements and endpoints P 0 , P 1 and P 2 The distance between 0 d 1 and d 2 can be determined based on their coordinates respectively.

[0194] Similar to Figure 3B The line primitives in , the triangle primitives can be viewed as continuous apertures in space, and the analytical expression for the electromagnetic field contribution of the triangle primitives to the display element can be obtained by integration. This can be simplified to obtain an expression for efficient computation.

[0195] Example calculation of primitives

[0196] As described above, a controller, such as controller 112 of Figure 1, may calculate the electromagnetic field contribution of a primitive to a display element based on an analytical expression that may be determined as shown above. For example, the electromagnetic field contribution of a line primitive is calculated as follows.

[0197] Each display element in a display screen has a physical location in space, and each display element lies in a plane relative to the other display elements. Assuming that the display elements and their controllers are arranged as is customary in display and storage devices, a simple mathematical point transformation can be used to transform the logical location of a given display element to the actual physical location of the display element in space based on the logical storage address of the display element in the processor. Thus, when the logical storage address of a display element is cycled through the logical storage space of the processor, the corresponding actual physical location in the surface space of the display screen can be identified.

[0198] As an example, if the pitch of the display screen is 5 μm, each logical address increment may move 5 μm in the x direction, and when the x resolution limit of the display screen is reached, the next increment will move back to the initial x physical position and increase the y physical position by 5 μm. The third spatial coordinate z may be assumed to be zero on the display screen surface, meaning that negative z values ​​are behind the display screen and positive z values ​​are in front of the display screen.

[0199] To begin line calculation, a scaled physical distance between the current display element and each of the two points of the line primitive may be determined as d0 and d 1 In fact, 0 and d 1 It can be calculated once per primitive, since each subsequent calculation of the distance for all display elements is a small change of the initial value. In this way, the calculation is performed in one dimension.

[0200] An exemplary calculation process for each primitive may include the following calculation code:

[0201] DD=f(d 1 ,d 0 ),

[0202] iscale=SS*COLOR*Alpha1,

[0203] C1=-2*iscale*sin(DD / 2)*sin((Alpha2)*cos(Alpha3),

[0204] C2=-2*iscale*sin(DD / 2)*sin(Alpha2)*sin(Alpha4),

[0205] Where SS, Alpha1, Alpha2, Alpha3, and Alpha4 are pre-computed constants, COLOR is the RGB color value passed with the primitive, and all values ​​are scalar, single-precision floating point numbers. Both the sine and cosine functions can be looked up in a table stored in the controller to improve computational efficiency.

[0206] Then, at each display element, the results of C1 and C2 are accumulated for each primitive, for example in an accumulator for the display element, and may be normalized once at the end of the calculation for the display element. At this point, as described above, the controller may send a first control signal to the display element to modulate the display element based on the calculated result, and send a second control signal to the illuminator to turn on the illuminator to emit light. Thus, the holographic reconstruction (or holographic light field) is visible to the viewer. When the modulated display element is illuminated, the light may be caused to produce a clear continuous color line in three-dimensional space.

[0207] In some embodiments, the calculation code includes a hexadecimal code for clearing a previous accumulation in an accumulator, for example at the beginning of the code. The calculation code may also include a hexadecimal code for storing the accumulator result in a corresponding storage buffer for each display element, for example at the end of the code. In some embodiments, for example Figure 1AThe computing device 102 of the computing device 102 sends a plurality of background or static primitive hexadecimal codes to the controller at application startup or in the interval between frames displayed without affecting the main display frame rate. Subsequently, the computing device can send one or more combinations of hexadecimal codes to the controller, potentially along with other foreground or dynamic primitives, at a higher rate, and the controller can form corresponding control signals to modulate the display elements of the display screen.

[0208] The computational processing can be simpler and several orders of magnitude faster than the most efficient line drawing routines in traditional 2D display technology. In addition, the computational algorithm scales linearly with the number of display elements. Therefore, scaling the controller's computational unit to a 2D networked processing system can keep up with the computational needs of the increased surface area of ​​the display screen.

[0209] Exemplary Computing Implementations

[0210] For example, Figure 1A The Maxwell holographic controller of the controller 112 may calculate the electromagnetic field contribution of the primitives to the display elements based on an analytical expression which may be determined as shown above. The controller may be implemented in, for example, an ASIC, an FPGA, or a GPU, or any combination thereof.

[0211] In a modern GPU pipeline, the GPU performs a description of geometry and vertex and fragment shading programs to produce color and depth pixel outputs to one or more output image surfaces (called render targets). This process involves an explosive fan-out of information, where the geometry is expanded to shaded fragments, followed by a visibility test to select whether work needs to be done on each of these fragments. A fragment is a record containing all the information involved in shading the sample point (e.g., the barycentric coordinates of a triangle, interpolated values ​​(such as color or texture coordinates), surface derivatives, etc.). The process of creating these records and then rejecting those records that do not contribute to the final image is called visibility testing. Fragments that pass the visibility test can be encapsulated into working groups called wavefronts or warps that are executed in parallel by the shader engine. These produce output values ​​that are written back to memory as pixel values, ready for display, or used as input textures for later rendering passes.

[0212] In Maxwell holography, the rendering process can be significantly simplified. In Maxwell holographic computing, each primitive contributes to each display element. There is no need to extend the geometry to pixels, and there is no need to apply visibility tests before packing wavefronts. This can also remove the need for decisions or communication between Maxwell holographic pipelines, and allow computations to become parallel problems with multiple possible solutions, each optimized for speed, cost, size, or energy. The graphics pipeline is significantly shorter, with fewer intermediate steps, no data copying or movement, and fewer decisions, resulting in lower latency between initiating a draw and the result being ready for display. This can allow Maxwell holographic rendering to create extremely low latency displays. As described below, this can allow Maxwell holographic computing to increase accuracy, for example, by using fixed-point numbers in the Maxwell holographic pipeline, and to optimize computation speed, for example, by optimizing mathematical functions.

[0213] Use fixed point numbers

[0214] When calculating the electromagnetic field contribution of each primitive at each display element (e.g., phase cell), the intermediate calculations involve the generation of very large numbers. These large numbers require special handling because they also require the preservation of decimal parts during calculation.

[0215] Floating point values ​​have the disadvantage that they are most accurate close to the origin (zero on the number axis) and lose one bit of precision for every power of two as you move away from the origin. For values ​​close to the range [-1,1], the precision of floating point values ​​can be high, but once you get into the tens of millions, for example, to the point where there are no decimal places left for a single-precision 32-bit IEEE-754 floating point value, the entire significand (also called the mantissa) is used to represent the integer portion of the value. However, what Maxwell holography is particularly interested in preserving is the fractional portion of large values.

[0216] In some cases, fixed-point numbers are used in Maxwell holographic calculations. A fixed-point representation is a numerical value where the decimal point does not change depending on the situation. By choosing the correct number of bits for the integer and decimal parts of the value, the same number of decimal bits can be obtained regardless of the size of the value. A fixed-point representation is an integer with an implicit scale factor, for example, in a 16-bit fixed-point value with 8 decimal bits, 14.375 can be represented as the value 3680 (negative binary is 000011100100000). This can also be expressed as an "unsigned 16.8" fixed-point number, or u16.8 for short. Negative numbers can have an additional sign bit and be stored in a "2s compliment" format. This can greatly improve the accuracy of the calculation.

[0217] Optimization of mathematical functions

[0218] As shown above, Maxwell holographic calculations involve the use of a priori mathematical functions, such as sine, cosine, arctangent, etc. In the CPU, these functions are implemented as floating-point library functions that can use dedicated CPU instructions, or on the GPU as floating-point units in the GPU. These functions are written to take arguments as floating-point numbers and return results in the same floating-point representation. These functions are constructed so that for the general case, they are accurate, correctly rounded, and handle every edge case in the floating-point representation (+ / - infinity, non-numeric values, signed zeros, and abnormal floating-point numbers) when floating-point numbers are accurate.

[0219] In Maxwell holographic computing, using fixed-point representation, there is no need to use denormalized floating point numbers for gradual underflow, no need to handle non-numeric values ​​from operations such as division by 0, no need to change the floating point rounding mode, and no need to cause floating point exceptions from the operating system. All of this allows simplification (and / or optimization) of prior math functions, for example, as described below.

[0220] In some cases, optimizations can be made to take the argument in a fixed point format and return the value to different levels of precision, e.g., input 28.12 and output 15.14. This may be particularly desirable when computing the sine of large values ​​in the tens of millions, where the input argument may be large but the output may only need to represent the value range [-1,1], or the inverse tangent of a value that takes any value but returns the value in the range [-π / 2,π / 2].

[0221] In some cases, depending on the input ranges involved, optimizations can be made to freely implement the a priori function as a fully enumerated lookup table, an interpolated table, or as a semi-table-based polynomial function or a semi-table-based full minimax polynomial. This optimization also allows the application of specific range reduction methods that handle larger inputs, which general-purpose GPU pipeline calculations may skip due to speed.

[0222] In some cases, another optimization may be to transform trigonometric calculations from the range [-π,π] into signed 2's representation in the range [-1,1], which has the advantage of not requiring the expensive divide by 2π modulo operation.

[0223] Example Implementation of Occlusion

[0224] Occlusion is often considered a significant problem in computer graphics, and even more so in computational holography. This is because, in at least some cases, while the occlusion problem in projected CGI is static, in a holographic system what is hidden and what is visible depends on the position and orientation of the viewer. Wave methods of GS holography or its derivatives have been developed to address holographic occlusion. However, in GS methods, masking or blocking contributions from parts that are behind other parts in the scene can be very complex and computationally expensive.

[0225] In Maxwell holography, the occlusion problem can be solved relatively easily because which display elements (e.g., phase cells) correspond to which primitives is completely deterministic and subtle. For example, when performing calculations for a given primitive, it can be determined whether the given display element contributes to the reconstruction of the given primitive. After determining that several display elements do not contribute to the given primitive due to occlusion, when calculating the sum of the electromagnetic field contributions to one of the several display elements, the electromagnetic field contribution from the given primitive is omitted from the calculation of the sum of the electromagnetic field contributions to one of the several display elements.

[0226] For illustration purposes only. Figures 3D to 3F shows the determination of not occluding a given primitive ( Figure 3D The point in Figure 3E The lines and Figure 3F The line primitive starts at O1 and ends at O2.

[0227] like Figure 3D As shown, the point primitive P0 is behind the occluder and closer to the display screen. By extending the line connecting O1-P0 and O2-P0, the range D1 to D2 of display elements in the display screen that do not contribute to the reconstruction of the point primitive P0 is determined.

[0228] In some examples, the coordinate information of O1, O2, and P0 is known, for example, stored in a GPU (e.g., Figure 1A The GPU 108 of the holographic controller (e.g., Figure 1A The coordinate information of D1 and D2 may be in the "Z" buffer previously calculated by the controller 112 of the controller 112. For example, in the XZ plane with y=0, the coordinate information may be O1(Ox1, Oz1), O2(Ox2, Oz2) and PO(Px, Pz), where Oz1=Oz2=Oz. Based on this coordinate information, the coordinate information of D1 and D2 may be determined as:

[0229] Dx1=Px+ρ(Px-Ox2), Dx2=Dx1+ρ(Ox2-Ox1) (4),

[0230] Where ρ=Pz / (Oz-Pz), and Dz1=Dz2=0.

[0231] In addition to the information in the Z buffer for the point primitive P0, the information of D1 and D2 can also be stored as additional information in the "S" buffer for the Maxwell holographic controller. In this way, the additional information can be used to simply mask the contribution of a specific display element (in the range from D1 to D2) to a specific primitive P0 in the indexed primitive list.

[0232] Figure 3E The determination of how a particular display element contributes to a line primitive in the case where an occluder is in front of the line primitive is shown. By connecting the particular display element D0 to the start point O1 and end point O2 of the occluder, two point primitives P1 and P2 on the line primitive are determined as intersection points. Therefore, the particular display element D0 does not contribute to the reconstruction of the portion of the line primitive from P1 to P2 on the line primitive. Thus, when calculating the sum of the electromagnetic field contributions to the particular display element D0, the electromagnetic field contribution from the P1-P2 portion of the line primitive is not calculated.

[0233] This can be achieved in two ways. In the first way, by considering the occlusion from the occluder, the electromagnetic field contributions of the P0-P1 part and the P2-Pn part to the specific display element D0 are added as the electromagnetic field contribution of the line primitive to the specific display element D0. In the second way, the electromagnetic field contribution from the entire line primitive P0-Pn and the electromagnetic field contribution from the P1-P2 part are calculated, and by considering the occlusion from the occluder, the difference between the two calculated electromagnetic field contributions can be regarded as the electromagnetic field contribution of the line primitive to the specific display element D0. The coordinate information of P1 and P2 or the P1-P2 part can be stored in the "S" buffer of the Maxwell holographic controller as the part of the line primitive that does not contribute to the specific display element D0, together with the information of the occluder and other information in the "Z" buffer of the GPU.

[0234] Figure 3F 1 shows the determination of how a particular display element contributes to a triangle primitive when an occluder is in front of the triangle primitive. By connecting the particular display element D0 to the start point O1 and the end point O2 of the occluder, four point primitives P1, P2, P3, and P4 on the edge of the triangle primitive are determined as intersection points. Therefore, the contribution of the particular display element D0 to the points P1, P2, P3, P4, P5, and P6 of the triangle primitive is determined to be the same as the contribution of the particular display element D0 to the triangle primitive. C Thus, when calculating the sum of the electromagnetic field contributions to a particular display element D0, the P1-P2-P3-P4-P C That is, by considering the shielding of the obstruction, only the free point P A , the first triangle formed by P1 and P2 and the point PB The electromagnetic field contributions of the second triangle formed by P3 and P4 are added as the triangle primitive P A -P B -P C The electromagnetic field contribution of P1, P2, P3 and P4 or the triangular primitive P A -P1-P2 and P B -The coordinate information of P3-P4 can be used as the triangle primitive P A -P B -P C The portion of the image that contributes to a particular display element D0 is stored in the "S" buffer of the Maxwell holographic controller along with information about occluders and other information in the "Z" buffer of the GPU.

[0235] The implementation of occlusion in Maxwell holography enables the conversion of a "Z" buffer in the GPU into an "S" buffer in the Maxwell holographic controller, and the contribution of a specific primitive (or a specific portion of a primitive) in an indexed primitive list to a specific display element can be masked. This not only provides accurate, physically correct occlusion, but also saves computation time because primitives that do not contribute to a given display element continue to be used in the calculation for the next display element. The "S" buffer can contain additional information related to the diffraction efficiency of the display screen.

[0236] The "S" buffer may also include rendering features such as holographic specular highlights, where the reflectivity of a surface depends on the viewing angle. In traditional CGI, specular highlights depend only on the orientation of the rendered object, whereas in the context of Maxwell holography, the direction from which the object is viewed also plays a role. Therefore, geometric specular information may be encoded in the "S" buffer as an additive (specular) rather than a subtractive (occlusion) contribution. In Maxwell holography, the mathematical operations for holographic specular highlights may be essentially the same as those for holographic occlusion.

[0237] Exemplary Embodiments of Engagement

[0238] When light is shone on a display screen modulated with electromagnetic field contributions from a list of primitives of a 3D object, the modulated display screen causes the light to propagate in different directions to form a volumetric light field corresponding to the primitives. The volumetric light field is a Maxwell holographic reconstruction. Two adjacent primitives (e.g., triangle primitives) in a 3D object have shared edges. During reconstruction, a joining problem may arise, where the light intensity of the shared edge may be doubled due to the separate reconstruction of the two adjacent primitives. This may affect the appearance of the reconstructed 3D object.

[0239] In order to solve the joint problem in Maxwell holography, such as Figure 3GAs shown, adjacent primitives may be scaled down by a predetermined factor so that a gap may be formed between adjacent primitives. In some cases, rather than scaling down two adjacent primitives, only one primitive or a portion of a primitive may be scaled down. For example, a line in a triangle primitive may be scaled down to be separated from another triangle primitive. In some cases, scaling may include scaling different portions of a primitive using different predetermined factors. Scaling may be designed so that the gap is large enough to minimize stitching issues with adjacent primitives, and small enough to make the reconstructed 3D object appear seamless. The predetermined factor may be determined based on information of the display screen (e.g., the maximum spatial resolution of the display screen).

[0240] In some cases, a scaling operation may be applied to a holographic renderer (e.g. Figure 1A The holographic renderer 130) obtains the primitive data of the primitive, and sends the scaled primitive data of the primitive to the Maxwell holographic controller, for example, Figure 1A Controller 112. In some cases, the controller may perform a scaling operation on primitive data obtained from a holographic renderer before calculating the electromagnetic field contribution of the primitive to a display element of a display screen.

[0241] Example Implementation of Texture Mapping

[0242] Texture mapping is a technique developed in computer graphics. The basic idea is to take a source image and apply it as a decal to a surface in a CGI system, allowing details to be rendered into a scene without adding complex geometry. Texture mapping can include techniques for creating realistic lighting and surface effects in CGI systems, and can broadly refer to the application of surface data to a triangle mesh.

[0243] In Maxwell holography, a flat-shaded and interpolated triangle mesh can be rendered in true 3D using an analytical relationship between arbitrary triangles in space and the phase map on the holographic device. However, to be compatible with modern rendering engines, the ability to map information on the surface of these triangles is involved. This can pose practical problems, as the speed of the method derives from the presence of an analytical mapping, which does not tolerate data-driven amplitude changes.

[0244] The Discrete Cosine Transform (DCT) is an image compression technique and can be considered as a real-valued version of the FFT (Fast Fourier Transform). The DCT depends on an encoding-decoding process that assigns weights to the cosine harmonics in a given image. The result of the encoding is a set of weights, where the number of weights is equal to the number of pixels in the original image, and if each weight is used to reconstruct the image, no information will be lost. However, in many images, an acceptable reconstruction can be made based on a small subset of the weights, thereby achieving a large compression ratio.

[0245] The decoding (rendering) process of the DCT in two dimensions involves a weighted floating point sum (double sum) of each DCT weight and each target pixel. This can be applied to Maxwell holography for texture mapping. In Maxwell holography, triangle rendering involves a "peaked" double integration in phase space to determine the phase contribution of any individual phase to the triangle in question. The integral can be converted into a floating point sum that reflects the integral in the DCT reconstruction, and then the analytical triangle expression is re-derived based on the DCT weights. This implementation of the DCT technology in Maxwell holographic calculations enables the drawing of complete texture mapped triangles, thereby adopting image compression on the data of the rendered texture triangles, and taking advantage of the existing tool set that uses DCT / JPEG to automatically compress texture and image data.

[0246] In some embodiments, to draw a Maxwell holographic textured triangle, the spatial resolution required for the mapping on a specified surface is first calculated. Then, a texture with that resolution is provided, and a DCT compressed with angle and origin information is obtained to correctly position it on the triangle. Then, a list of angles and DCT weights for the triangle is included in a list of indexed primitives and sent to the Maxwell holographic controller. The DCT weights may be included in the electromagnetic field contribution of the triangle primitive to each display element. Textured triangles may be n times slower than flat triangles, where n is the number of (non-zero) DCT weights sent with the primitive. Modern techniques for "fragment shading" can be implemented in a Maxwell holographic system, where the step of DCT encoding replaces the filtering step used for traditional projection rendering.

[0247] As an example, the following expression shows the DCT weight B of an image pq :

[0248]

[0249] in, M and N are the corners of the rectangular image and (p,q) are the DCT terms.

[0250] By decoding, the amplitude value Amn can be obtained as follows:

[0251]

[0252] in

[0253] When computing the electromagnetic field contribution of a textured triangle primitive to a display element (e.g., a phase cell), the corresponding DCT weights The DCT term of can be included in the calculation as follows:

[0254]

[0255] Where X, Y are the angles of the triangle in the coordinate system, T corresponds to the electromagnetic field contribution of the triangle primitive to the display element, and is the non-zero term B in DCT pq The number of (p,q)DCT terms can be chosen by considering both information loss and information compression in reconstruction.

[0256] Exemplary Processing

[0257] Figure 4 4 is a flow chart of an exemplary process 400 for displaying a 3D object. Process 400 may be performed by a controller for a display screen. The controller may be Figure 1A The controller 112 or Figure 1B The display screen may be the display screen 114 of FIG. 1A or Figure 1B Display screen 156.

[0258] Data including primitive data for each of a plurality of primitives corresponding to an object in a 3D space is obtained (402). The data may be obtained from, for example, Figure 1A The computing device 102 of the computing device 102 is obtained. The computing device can process the scene to generate primitives corresponding to the object. The computing device may include a renderer to generate primitive data of the primitive. In some embodiments, the controller itself generates data, for example, by rendering the scene.

[0259] The primitive may include at least one of a point primitive, a line primitive, or a polygon primitive. The primitive list is indexed in a specific order, for example, by which an object can be reconstructed. The primitive data may include color information having textured colors and / or gradient colors. For example, a line primitive may have gradient colors and / or textured colors. A polygon primitive may also have gradient colors and / or textured colors. The primitive data may also include texture information of the primitive and / or shading information on one or more surfaces (e.g., triangles) of the primitive. The shading information may include modulation of color and / or brightness on one or more surfaces of the primitive. The primitive data may also include corresponding coordinate information of the primitive in a 3D coordinate system.

[0260] The display screen may include a plurality of display elements, and the controller may include a plurality of computing units. The corresponding coordinate information of each display element in the 3D coordinate system may be determined based on the corresponding coordinate information of the primitive list in the 3D coordinate system. For example, the distance between the display screen and the object corresponding to the primitive may be predetermined. Based on the predetermined distance and the coordinate information of the primitive, the coordinate information of the display element may be determined. The corresponding coordinate information of each display element may correspond to a logical storage address for the display element stored in the memory. In this way, when the controller cycles through the logical storage addresses for the display elements in the logical storage space of the controller, the corresponding actual physical position of the display element in the space may be identified.

[0261] Electromagnetic field contributions of the plurality of primitives to each display element are determined by calculating electromagnetic field propagation from each primitive in the plurality of primitives to the display element in a 3D coordinate system (404). The electromagnetic field contributions may include phase contributions and / or amplitude contributions.

[0262] As mentioned above FIG. 3A to FIG. 3C As shown, at least one distance between a primitive and a display element may be determined based on corresponding coordinate information of the display element and corresponding coordinate information of the primitive. In some cases, the at least one distance may be calculated or calculated only once for each primitive. For example, the controller may determine a first distance between a first primitive in the primitives and a first element in the display elements based on corresponding coordinate information of the first primitive and corresponding coordinate information of the first element, and determine a second distance between the first primitive and a second element in the elements based on the first distance and the distance between the first element and the second element. The distance between the first element and the second element may be predetermined based on the pitch of a plurality of elements of the display screen.

[0263] The controller may determine the electromagnetic field contribution of the primitive to the display element based on a predetermined expression for the primitive and the at least one distance. FIG. 3A to FIG. 3CAs shown, the predetermined expression can be determined by analytically calculating the electromagnetic field propagation from the primitive to the display element. In some cases, the predetermined expression is determined by solving Maxwell's equations. Specifically, the Maxwell equations can be solved by providing boundary conditions defined at the surface of the display screen. The boundary conditions may include Dirichlet boundary conditions or Cauchy boundary conditions. The primitive and the display element are in 3D space, and the surface of the display screen forms a part of the boundary surface of the 3D space. The predetermined expression may include at least one of the functions including sine function, cosine function and exponential function. During calculation, the controller may identify the value of at least one function in a table stored in a memory, which may increase the calculation speed. The controller may determine the electromagnetic field contribution to each display element for each primitive in the following manner: determining the first electromagnetic field contribution of the first primitive to the display element and determining the second electromagnetic field contribution of the second primitive to the display element in parallel.

[0264] For each display element, a sum of the electromagnetic field contributions of the primitive list to the display element is generated (406).

[0265] In some embodiments, the controller determines a first electromagnetic field contribution of a plurality of primitives to a first display element and sums the first electromagnetic field contribution for the first display element, and determines a second electromagnetic field contribution of a plurality of primitives to a second display element and sums the second electromagnetic field contribution for the second display element. The controller may include a plurality of computing units. The controller may determine the electromagnetic field contribution of the first primitive to the first element by the first computing unit in parallel with determining the electromagnetic field contribution of the second primitive to the first element by the second computing unit.

[0266] In some embodiments, the controller determines the corresponding first electromagnetic field contribution of the first primitive to each display element and determines the corresponding second electromagnetic field contribution of the second primitive to each display element. Then, the controller accumulates the electromagnetic field contribution to the display element by adding the corresponding second electromagnetic field contribution to the display element to the corresponding first electromagnetic field contribution. Specifically, the controller may determine the corresponding first electromagnetic field contribution of the first primitive to each display element by using the first calculation unit and determine the corresponding second electromagnetic field contribution of the second primitive to each display element by using the second calculation unit in parallel.

[0267] A first control signal is sent to the display screen, the first control signal being used to modulate at least one characteristic of each display element based on the sum of the electromagnetic field contributions to the display element (408). The at least one characteristic of the display element includes at least one of a refractive index, an amplitude index, birefringence, or hysteresis.

[0268] The controller may generate a corresponding control signal for each display element based on the sum of electromagnetic field contributions of multiple primitives to the display element. The corresponding control signal is used to modulate at least one characteristic of the element based on the sum of electromagnetic field contributions of multiple primitives to the element. That is, the first control signal includes the corresponding control signal for the display element.

[0269] In some examples, the display screen is controlled by electrical signals. The corresponding control signal can then be an electrical signal. For example, an LCOS display screen includes an array of tiny electrodes that are individually controlled as element intensities using voltage. The LCOS display screen can be filled with a birefringent liquid crystal (LC) formulation that changes its refractive index. Thus, a corresponding control signal from a controller can control the relative refractive index of each display element, and accordingly the relative phase of light passing through the display screen.

[0270] As described above, the display screen surface forms part of the boundary surface. The controller may multiply the scale factor by the sum of the electromagnetic field contributions for each display element to obtain a scaled sum of the electromagnetic field contributions, and generate a corresponding control signal based on the scaled sum of the electromagnetic field contributions for the display element. In some cases, the controller may normalize the sum of the electromagnetic field contributions for each display element, for example, of all display elements, and generate a corresponding control signal based on the normalized sum of the electromagnetic field contributions for the display element.

[0271] A second control signal is sent to the illuminator, the second control signal being used to turn on the illuminator to illuminate the modulated display screen (410). In response to determining that the sum of the electromagnetic field contributions to each display element has been obtained, the controller may generate and send the second control signal. Due to time symmetry (or energy conservation), the modulated display elements of the display screen may cause light to propagate in different directions to form a volumetric light field corresponding to an object in 3D space. The volumetric light field may correspond to a solution of Maxwell's equations having boundary conditions defined by the modulated display elements of the display screen.

[0272] In some embodiments, the luminaire is coupled to the controller via a storage buffer configured to control the amplitude or brightness of one or more light emitting elements in the luminaire. The size of the storage buffer for the luminaire can be smaller than the storage buffer for the display screen. The number of light emitting elements in the luminaire can be less than the number of display elements of the display screen. The controller can be configured to activate one or more light emitting elements of the luminaire simultaneously.

[0273] In some examples, the illuminator includes two or more light emitting elements, each light emitting element being configured to emit light having a different color. The controller may be configured to sequentially modulate the display screen with information associated with a first color during a first time period and information associated with a second color during a sequential second time period, and control the illuminator to sequentially turn on the first light emitting element during the first time period to emit light having the first color and turn on the second light emitting element during the second time period to emit light having the second color. In this manner, a multi-color object may be displayed in a 3D space.

[0274] In some examples, the display screen has a resolution small enough to diffract light. The illuminator can emit white light onto the display screen, which can diffract the white light into light of different colors to display a multi-colored object.

[0275] Exemplary Systems

[0276] FIG. 5A to FIG. 5F An exemplary system for 3D display is shown. Any system may correspond to, for example Figure 1A system 100.

[0277] Figure 5A A system 500 having a reflective display screen is shown. The system 500 includes a computer 502, a controller 510 (eg, an ASIC), a display screen 512 (eg, an LCOS device), and an illuminator 514. The computer 502 may be Figure 1A The computing device 102, the controller 510 may be Figure 1A The display screen 512 may be Figure 1A The display screen 114, the illuminator 514 may be Figure 1A illuminator 116.

[0278] like Figure 5A As shown, a computer 502 includes an application 504 having a renderer 503 for rendering a scene of an object. The rendered scene data is processed sequentially by a video driver 505 and a GPU 506. The GPU 506 may be Figure 1A The GPU 108 may be configured to generate a primitive list and corresponding primitive data corresponding to the scene. For example, the video driver 505 may be configured to process the rendered scene data and generate a primitive list. As described above, the GPU 506 may include a conventional 2D renderer, such as, Figure 1A The conventional 2D renderer 120 of FIG. 500 may render the primitives as a list of items to be drawn on the 2D display screen 508. The GPU 506 or the controller 510 may include a holographic renderer, for example, Figure 1A The holographic renderer 130 is used to render the primitive list into graphic data to be displayed by the display screen 512.

[0279] The controller 510 is configured to receive the graphics data from the computer 502, calculate the electromagnetic field contribution of the primitive list to each display element of the display screen 512, and generate a corresponding sum of the electromagnetic field contributions of the primitive to each display element. The controller 510 may generate a corresponding control signal for each display element to modulate at least one characteristic of the display element. The controller may send the corresponding control signal to the display element of the display screen 512 through the storage buffer 511 for the display screen 512.

[0280] The controller 510 may also generate and send control signals, such as illumination timing signals, to activate the illuminator 514. For example, in response to determining that the sum of the electromagnetic field contributions of the primitives to the display elements is complete, the controller 510 may generate and send control signals. As described above, the controller 510 may send control signals to the illuminator 514 via a storage buffer. The storage buffer may be configured to control the amplitude or brightness of the light-emitting elements in the illuminator 514 and activate the light-emitting elements simultaneously.

[0281] like Figure 5A As shown, the illuminator 514 can emit a collimated light beam 516 that is incident on the front surface of the display screen 512 at an incident angle ranging between 0 and 90 degrees. The emitted light beam is reflected from the front surface of the display screen 512 to form a holographic light field 518, which corresponds to an object visible to a viewer.

[0282] Figure 5B Another system 520 is shown having another reflective display screen 524. Figure 5A Compared to system 500, system 520 has a larger reflective display screen 524. To accommodate this, display controller 522 is included in a wedge-shaped housing that can be a support for illuminator 526. Controller 522 is similar to Figure 5A The controller 510 can be configured to receive the graphic data from the computer 521, calculate the electromagnetic field contribution of the plurality of primitives to each display element of the display screen 524, and generate a corresponding sum of the electromagnetic field contributions of the plurality of primitives to each display element. The controller 522 then generates a corresponding control signal for each display element to modulate at least one characteristic of the display element, and sends the corresponding control signal to the display element of the display screen 524 through the storage buffer 523 for the display screen 524.

[0283] The controller 522 also sends a control signal to the illuminator 526 to activate the illuminator 526. The illuminator 526 emits a divergent or semi-collimated light beam 527 to cover the entire surface of the display screen 524. The light beam 527 is reflected by the modulated display screen 524 to form a holographic light field 528.

[0284] Figure 5C A system 530 is shown having a transmissive display screen 534. The transmissive display screen 534 can be, for example, a large display screen. The system 530 includes a controller 532, which can be similar to Figure 5A Controller 510 of the present invention. Controller 532 may be configured to receive graphics data from computer 531, calculate electromagnetic field contributions of multiple primitives to each display element of display screen 534, and generate a corresponding sum of electromagnetic field contributions of multiple primitives to each display element. Controller 532 then generates a corresponding control signal for each display element to modulate at least one characteristic of the display element, and sends the corresponding control signal to the display element of display screen 534 through storage buffer 533 for display screen 534.

[0285] The controller 532 also sends a control signal to the illuminator 536 to activate the illuminator 536. Figure 5A System 500 and Figure 5B In the system 520 of FIG. 5 , the illuminator 536 in the system 530 is positioned behind the rear surface of the display screen 534. In order to cover the large surface of the display screen 534, the illuminator 536 emits a divergent or semi-collimated light beam 535 onto the rear surface of the display screen 534. The light beam 535 is transmitted through the modulated display screen 534 to form a holographic light field 538.

[0286] Figure 5D Another system 540 is shown having a transmissive display screen 544. The system 540 also includes a controller 542 and an illuminator 546. The controller 542 may be similar to Figure 5A The controller 510 can be configured to receive graphic data from the computer 541, perform calculations on the graphic data, generate control signals for modulation and send them to the display screen 544, and generate and send timing signals to activate the illuminator 546.

[0287] The illuminator 546 may include a light source 545 and a waveguide 547. Light emitted from the light source 545 may be coupled to the waveguide 547, for example, from a side cross section of the waveguide. The waveguide 547 is configured to guide the light to uniformly illuminate the surface of the display screen 544. The light guided by the waveguide 547 is incident on the rear surface of the display screen 544 and is transmitted through the display screen 544 to form a holographic light field 548.

[0288] Different from Figure 5A System 500, Figure 5B System 520 and Figure 5CIn the system 530, the controller 542, the display screen 544 and the waveguide 547 are integrated together into a single unit 550. In some cases, the waveguide 547 and the light source 545 can be integrated in a planar form as an active waveguide illuminator, which can further increase the integration of the single unit 550. As described above, the single unit 500 can be connected with other similar units 550 to form a larger holographic display device.

[0289] Figure 5E Another system 560 is shown having another transmissive display screen 564. Compared to system 540, transmissive display screen 564 can potentially enable a display screen that is larger than transmissive display screen 544. For example, transmissive display screen 564 can have a larger area than controller 562, and to accommodate this, controller 562 can be located away from display screen 564. System 560 includes an illuminator 566 having a light source 565 and a waveguide 567. Waveguide 567 is integrated with display screen 564, for example, integrated into the back surface of display screen 564. In some embodiments, display screen 564 is constructed on the front surface of a substrate, and waveguide 567 can be constructed on the back surface of the substrate.

[0290] The controller 562 may be similar to Figure 1A The controller 510 is configured to receive graphic data from a computer 561, perform calculations on the graphic data, generate control signals and send them to a display screen 564 through a storage buffer 563, and generate and send timing signals to activate a light source 565. Light emitted from the light source 565 is guided in a waveguide 567 to illuminate the rear surface of the display screen 564 and is transmitted through the display screen 564 to form a holographic light field 568.

[0291] Fig. 5F Another system 570 is shown with a reflective display screen 574. The reflective display screen 574 can be, for example, a large display screen. The waveguide 577 of the illuminator 576 is positioned on the front surface of the reflective display screen 574. Similar to Figure 5E The controller 572 of the controller 562 can be configured to receive graphics data from the computer 571, perform calculations on the graphics data, generate control signals and send them to the display screen 574 through the storage buffer 573, and generate and send timing signals to activate the light source 575 of the illuminator 576. The light coupled from the light source 575 of the illuminator 576 is guided to be incident on the front surface of the display screen 574 and reflected by the front surface to form a holographic light field 578.

[0292] Exemplary Display Screen Implementations

[0293] As described above, the display screen in Maxwell holography can be a phase modulation device. The phase elements (or display elements) of the display screen can be represented as phase cells. For illustration purposes only, a liquid crystal on silicon (LCOS) device is discussed below as a phase modulation device. An LCOS device is a display screen that uses a liquid crystal (LC) layer on top of a silicon backplane. The LCOS device can be optimized to achieve the smallest possible phase pitch, the smallest crosstalk between phases, and / or the largest available phase modulation or hysteresis (e.g., at least 2π).

[0294] A list of parameters that can be controlled to optimize the performance of the LCOS device include the birefringence of the LC mixture (Δn), the cell gap (d), the dielectric anisotropy of the LC mixture (Δε), the rotational viscosity of the LC mixture (η), the maximum applied voltage between the silicon backplane and the common electrode on top of the LC layer (V).

[0295] There may be fundamental trade-offs between the parameters of liquid crystal materials. For example, a fundamental boundary parameter is the available phase modulation or retardation (Re), which can be expressed as:

[0296] Re=4π·Δn·d / λ (8),

[0297] Where λ is the wavelength of the input light. If the retardation Re needs to be at least 2π for red light with a wavelength of about 0.633 μm, then

[0298] Δn·d≥0.317μm (9).

[0299] The above expression implies that there is a direct trade-off between the cell gap (d) and the birefringence (Δn) of an LC mixture.

[0300] Another boundary parameter is the switching speed or switching time (T) taken for the liquid crystal molecules in the liquid crystal (LC) layer to reach the desired orientation after a voltage is applied. For example, for real-time video (about 60 Hz) using a 3-color field sequential color system, a minimum of 180 Hz modulation of the LC layer is involved, which puts the upper limit of the LC switching speed at 5.6 milliseconds (ms). The switching time (T) is related to multiple parameters including the liquid crystal, the cell gap, and the applied voltage. First, T is related to d 2 As the cell gap d decreases, the switching time decreases with the square of d. Secondly, the switching time is also related to the dielectric anisotropy (Δε) of the liquid crystal mixture, where higher dielectric anisotropy leads to shorter switching times, and lower viscosity also leads to shorter switching times.

[0301] The third boundary parameter can be the fringe field. Due to the high electron mobility of crystalline silicon, LCOS devices can be constructed with very small phase cell sizes (e.g., less than 10 μm) and sub-micron phase cell gaps. When adjacent phase cells are operated at different voltages, the liquid crystal directors near the edges of the phase cells are distorted by the lateral components of the fringe field, which significantly reduces the electro-optical performance of the device. In addition, as the phase cell gap becomes comparable to the wavelength of the incident light, diffraction effects can cause severe light loss. The phase cell gap needs to be kept less than or equal to the phase distance to keep the noise within acceptable levels.

[0302] In some examples, LCOS devices are designed to have a phase separation of 2 μm and also have a cell gap of about 2 μm if the fringe field boundary conditions are observed. According to the above expression Δn·d≥0.317 μm, Δn needs to be equal to 0.1585 or greater, which is achievable using existing liquid crystal technology. Once the minimum birefringence for a given phase separation is determined, the LC can be optimized for switching speed, for example by increasing dielectric anisotropy and / or reducing rotational viscosity.

[0303] Non-uniform phase unit implementation scheme for display screens

[0304] In an LCOS device, a circuit chip (e.g., a complementary metal oxide semiconductor (CMOS) chip or equivalent) controls the voltage on a reflective metal electrode buried below the chip surface, each of which controls a phase unit. The common electrode for all phase units is provided by a transparent conductive layer made of indium tin oxide on a cover glass. The phase units may have the same size and the same shape (e.g., square). For example, a chip may have 1024x768 plates, each with an independently addressable voltage. As described above, when the phase unit gap becomes comparable to the wavelength of the incident light, diffraction effects may occur in the periodic structure of the LCOS device, which may result in severe light loss.

[0305] In Maxwell holographic calculations, each phase unit receives the sum of the electromagnetic field contributions from each primitive and is relatively independent of each other. Therefore, the phase units of the LCOS device in Maxwell holography can be designed to be different from each other. For example, Fig. 6A As shown, the LCOS device 600 may be composed of a plurality of non-uniform (or irregular) phase cells 602. At least two phase cells 602 have different shapes. The non-uniform shape of the phase cell 602 can greatly reduce or eliminate diffraction aberrations and other effects, and thus improve image quality. Although the phase cell may have a non-uniform shape, the phase cell may be designed to have a size distribution that meets the desired spatial resolution (e.g., about 3 μm). The silicon backplane may be configured to provide a corresponding circuit (e.g., including a metal electrode) for each phase cell according to the shape of the phase cell.

[0306] In a phase cell array in an LCOS device, in order to select a specific phase cell, a first voltage is applied to a word line connecting a phase cell row including the specific phase cell, and a second voltage is applied to a bit line connecting a phase cell column including the specific phase cell. Since each phase cell has a resistance, the operating speed of the LCOS device is limited.

[0307] As mentioned above, in Maxwell holography, the phase units can have different sizes. Figure 6B As shown, the LCOS device 650 is designed to have one or more phase cells 654, the size of which is larger than the size of other phase cells 652. All phase cells can still have a size distribution that meets the desired resolution. For example, 99% of the phase cells have a size of 3μm, and only 1% of the phase cells have a size of 6μm. In addition to other circuits that are the same as those in the phase cell 652, the larger size of the phase cell 654 also allows at least one buffer 660 to be arranged in the phase cell 654. The buffer 660 is configured to buffer the applied voltage so that the voltage is only applied to a smaller number of phase cells within a phase cell row or phase cell column. The buffer 660 can be an analog circuit (e.g., composed of transistors) or a digital circuit (e.g., composed of multiple logic gates) or any combination thereof.

[0308] For example, Figure 6B As shown, a voltage is applied to word line 651 and another voltage is applied to bit line 653 to select a specific phase cell 652*. Phase cell 652* is in the same row as a larger phase cell 654 including a buffer 660. The voltage is mainly applied to the first plurality of phase cells before the larger phase cell 654 in the row and is blocked by the buffer 660 in the larger phase cell 654. In this way, the operating speed of LCOS device 650 can be increased. Taking advantage of the larger size of phase cell 654, other circuits can also be arranged in LCOS device 650 to further improve the performance of LCOS device 650. Although Figure 6B The phase unit 654 and the phase unit 652 are shown to have a square shape. The phase unit may also have a square shape. Fig. 6A The different shapes shown are provided so long as one or more phase cells 654 are larger than the other phase cells 652 .

[0309] Example calibration

[0310] The unique properties of Maxwell holography in this disclosure allow for the protection of calibration techniques that can yield significant competitive advantages in the actual production of high-quality display screens. A number of calibration techniques can be implemented to combine with Maxwell holographic computing techniques, including:

[0311] (i) using image sensors in combination with Dirichlet boundary condition modulators and / or in combination with mechanical and software diffraction and non-diffraction calibration techniques;

[0312] (ii) software alignment and software calibration, including separate color calibration and alignment using Dirichlet boundary conditions; and

[0313] (iii) Embedding silicon features in boundary condition modulators that allow light detection to be built directly into the modulator, creating a powerful and unique method that simplifies the manufacturing calibration process when combined with Maxwell holography.

[0314] In the following, for illustration purposes only, three types of calibration are implemented for a phase cell based display screen (eg, LCOS display screen).Each phase element may be denoted as a phase cell.

[0315] Phase calibration

[0316] The amount of phase added to the light impinging on the LCOS phase element (or phase cell) can be directly known by the voltage applied to the LCOS phase cell. This is because the birefringent liquid crystal (LC) rotates in the presence of an electric field and thus changes its refractive index, thereby slowing down the light to change its phase. The changed phase may depend on the electrical properties of the liquid crystal (LC) and the silicon device in which the LC is located. The digital signal sent to the LCOS needs to be converted into the correct analog voltage to achieve high-quality holographic images. LCOS devices involve phase calibration to ensure that the digital signal is properly converted into an analog signal applied to the LC so that it produces the maximum phase range. It is expected that this conversion results in linear behavior. That is, when the voltage changes in fixed increments, the phase also changes in fixed increments, regardless of the starting voltage value.

[0317] In some cases, the LCOS device allows the user to change the digital-to-analog converter (DAC) so that the user controls the amount of analog voltage output given a digital input signal. A digital potentiometer can be applied to each input bit. For example, if there are 8 input bits, there can be 8 digital potentiometers corresponding to each input bit. The same digital input from the digital potentiometer can be applied to all phase units of the LCOS device. The bit set to 1 activates the voltage, and the bit set to "0" does not activate the voltage. All voltages from such "1" bits are summed together to obtain the final voltage sent to each phase unit. It is also possible to apply a DC voltage in all cases so that all "0" bits result in a baseline non-zero voltage. Therefore, phase calibration of the LCOS device can be achieved by setting the value of the digital potentiometer used for the LCOS device. For example, as described above, the controller can calculate the electromagnetic field contribution of the primitive list to each phase unit of the display screen, generate the sum of the corresponding electromagnetic field contributions of multiple primitives to each phase unit, and generate a corresponding control signal to each phase unit to modulate the phase of the phase unit. The same digital input from the digital potentiometer can be applied to adjust the control signal for each of all phase cells of the LCOS device, as opposed to phase calibration on a phase cell-by-phase basis. The digital input can be set once during operation of the LCOS device, for example for displaying a hologram.

[0318] In order to determine a set of optimal phase calibration values ​​for a digital input, a genetic algorithm can be applied, in which multiple input values ​​lead to an output value, such as a phase range or a holographic image contrast. The output value can be simplified to a numerical value called fitness. The genetic algorithm can be configured to explore different combinations of input values ​​until the output with the highest fitness is achieved. In some cases, the algorithm may take two or more of the most suitable inputs and combine their multiple component values ​​together to create a new input that has the characteristics of the inputs adopted but is different from each of the inputs adopted. In some cases, the algorithm may change one of these component values ​​to something that is not from any of the suitable inputs adopted, which is represented as a "variation" and the variant can be added to the available suitable inputs. In some cases, while trying new values, one or more optimal values ​​can be found by utilizing knowledge obtained from previous measurements with good results, so the optimal value is not limited to a local maximum.

[0319] There are multiple ways to calculate the fitness output value. One way is to calculate the phase change of light given a set of digital inputs applied to all phase units on the LCOS. In this scheme, the incident light can be polarized. When irradiated on the LCOS, the polarization of the incident light can change according to the rotation of the LC. The incident light can be set to the same polarization as the original polarization or a polarization that differs 90 degrees from the original polarization, and then enters the photodetector. Therefore, when the rotation of the LC changes, the intensity observed from the photodetector can change. Therefore, the phase change of light can be indirectly perceived by the change in the intensity of the light. Another way to calculate the phase change is to measure the intensity difference between the Maxwell holographic reconstruction and the background. This is most effective in projection display screens. In this case, measuring the intensity may require the use of computer vision algorithms to identify the Maxwell holographic reconstruction and measure its intensity.

[0320] Alignment Calibration

[0321] There is no guarantee that the light source is aligned within the holographic device and therefore requires alignment. Different liquid crystals (LC) may also behave differently given the wavelength of the light source. Furthermore, both the LC and the light source may vary from device to device, giving the same input hologram different characteristics when shown in different base colors, such as object scaling. Furthermore, certain hardware features may apply different optical effects, such as lensing, to the output light that also requires correction.

[0322] In some embodiments, the above problem can be solved by applying a mathematical transformation to the phase calculated for the phase cells of the display screen. The phase is the sum of the corresponding electromagnetic field contributions of the primitive list to the phase cell. The mathematical transformation can be derived from a mathematical expression (e.g., Zernike polynomials) and can be changed by changing the polynomial coefficients or other varying input values. The mathematical transformation can be varied per phase cell and by color. For example, there are Zernike polynomial coefficients that correspond to the amount of tilt applied to the light after it is reflected from the display screen.

[0323] To determine these coefficients / input values, a hardware setup can be created where the camera is pointed at a reflective surface in the case of a projection display and directly at the LCOS in the case of a direct-view display. A series of holographic test patterns and objects can be sent to the display and viewed by the camera. The camera can use a machine vision algorithm to determine what is being displayed and then calculate its fitness. For example, if a grid of dots is the pattern being tested, the fitness is how close the dots are, how centered the dot positioning is, how much distortion the dots have (e.g., scaling or pincushion distortion), etc. There may be different fitness values ​​for different characteristics. Depending on these values, corrections may be applied to the Zernike polynomials, for example, in the form of changing coefficients, until the fitness reaches a predetermined level of satisfaction. These tested patterns can be rendered at different distances to ensure that the alignment of the object is consistent at all distances, not just at one point. Such a depth-based calibration involves an iterative process that involves changing the depth of the reflective surface and the holographic test pattern in the case of a projection display, where the previous calibration can be repeated until convergence to a solution that is feasible for both depths. Finally, a white dot can be displayed to show the effect of the calibration.

[0324] Color Calibration

[0325] In display screens, whether holographic or otherwise, when any two units render the same image, it is important that the colors match between the displays and additionally match the colors defined by television (TV) and computer display screen standards, such as the Rec.709 standard for high-definition television (HDTV) or the sRGB color space for computer monitors. Different batches of hardware components (e.g., LEDs and laser diodes) may exhibit different behaviors for the same input and may output different colors when perceived by the human eye. Therefore, it is important to have a color standard that can calibrate all display units.

[0326] In some embodiments, an objective measurement of color specified by measuring intensity and chromaticity can be obtained by measuring color intensity against the International Commission (Commission internationale de l'éclairage, CIE) Standard Observer curve. By requesting each display screen to reproduce a sample set of known colors and intensities, and then measuring the output light using a colorimeter device calibrated according to the CIE Standard Observer curve, the color output of the device in the CIE XYZ color space can be objectively defined. Any deviations of the measured values ​​from the known good values ​​can be used to adapt the output color on the display screen to bring it back into alignment, which can be implemented using an iterative measure-adapt-measure feedback loop. Once the Maxwell holographic device produces accurate output for a given set of inputs, the final adaptation can be encoded as a lookup table for the illuminator that maps input values ​​to output intensities, and as a color matrix transform that transforms input colors into output color space values. These calibration tables can be embedded in the device itself to produce reliable objective output colors.

[0327] Furthermore, given an LCOS device with sufficiently fine features to control diffraction with sub-wavelength precision, tristimulus illumination (e.g., a linear mix of red, green, and blue) may not be required, and the LCOS device may be illuminated with a single broad spectrum light source and the phase cell output selectively tuned to produce tristimulus, quad-stimulus, or even N-stimulus output colors that, in combination with a spatial dithering pattern, can reproduce a full spectrum output of colors rather than the common tristimulus approximation. Given a sufficiently broad spectrum illuminator, this allows Maxwell holography to produce any reflected color that lies within the spectral locus of the human visual system.

[0328] The embodiments of the subject matter and functional operations described herein may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described herein may be implemented as one or more computer programs, for example, one or more computer program instruction modules encoded on a tangible non-transitory computer storage medium, for being executed by a data processing or controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0329] The term "data processing apparatus", "computer" or "electronic computer equipment" (or equivalents understood by those of ordinary skill in the art) refers to data processing hardware, and includes apparatus, equipment and machines for processing all kinds of data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include a dedicated logic circuit, for example, a central processing unit (CPU), an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In some embodiments, the data processing apparatus and the dedicated logic circuit may be hardware-based and software-based. The apparatus may optionally include code to create an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. It is contemplated herein to use a data processing apparatus with or without a conventional operating system.

[0330] A computer program, which may also be referred to or described as a program, software, software application, module, software module, script or code, may be written in any form of programming language, including compiled or deductive languages, declarative or procedural languages, and the computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data, such as one or more scripts stored in a markup language document; a single file or multiple collaborative files dedicated to the program in question, such as a file storing one or more modules, subroutines or code portions. A computer program may be deployed to be executed on one computer, or on multiple computers located in one location or distributed in multiple locations and interconnected by a communication network. Although the various parts of the program shown in the various figures are shown as modules that implement various features and functions through various objects, methods or other processes, the program may appropriately include multiple submodules, third-party services, components, libraries, etc. On the contrary, the features and functions of various components may be appropriately combined into a single component.

[0331] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating input data and generating output. The processes and logic flows may also be performed by dedicated logic circuits, and the apparatus may also be implemented as dedicated logic circuits, such as CPUs, GPUs, FPGAs, or ASICs.

[0332] Computers suitable for executing computer programs can be based on general-purpose microprocessors and / or special-purpose microprocessors, or any other type of CPU. Typically, CPU will receive instructions and data from a read-only memory (ROM) and / or a random access memory (RAM). The main element of a computer is a CPU for carrying out or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes or is operably coupled to one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, to receive data from one or more large-capacity storage devices and / or to transmit data thereto. However, a computer does not need such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few examples.

[0333] Computer-readable media suitable for storing computer program instructions and data (either transient or non-transitory, as the case may be) include all forms of non-volatile memory, media and storage devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM, DVD-R, DVD-RAM and DVD-ROM disks. The memory may store a variety of objects or data, including caches, classes, frameworks, applications, backup data, jobs, web pages, web page templates, database tables, repositories for storing business and dynamic information, and any other appropriate information including any parameters, variables, algorithms, instructions, rules, constraints or references. In addition, the memory may include any other appropriate data, such as logs, policies, security or access data, report files and other data. The processor and memory may be supplemented with or integrated in dedicated logic circuits.

[0334] To provide interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having: a display device for displaying information to a user, such as a CRT (cathode ray tube), LCD (liquid crystal display), LED (light emitting diode), or plasma monitor; and a keyboard and a pointer device such as a mouse, trackball, or trackpad that a user can use to provide input to the computer. A touch screen may also be used to provide input to the computer, such as a tablet computer surface with pressure sensitivity, a multi-touch screen using capacitance or inductance, or other types of touch screens. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form of input from the user may be received, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending and receiving documents to and from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0335] The term "graphical user interface" or "GUI" may be used in the singular or plural to describe one or more graphical user interfaces and each display of a particular graphical user interface. Thus, a GUI may represent any graphical user interface, including but not limited to a web browser, a touch screen, or a command line interface (CLI) that processes information and efficiently presents the results of the information to a user. In general, a GUI may include a plurality of user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, drop-down lists, and buttons that can be operated by a business suite user. These and other UI elements may be related to or represent the functionality of a web browser.

[0336] The embodiments of the subject matter described herein may be implemented in a computing system including a back-end component such as a data server, or a computing system including a middleware component such as an application server, or a computing system including a front-end component such as a client computer with a graphical user interface or a web browser, or a computing system including any combination of one or more such back-end components, middleware components, or front-end components, wherein a user may interact with the embodiments of the subject matter described herein through a graphical user interface or a web browser. The components of the system may be interconnected by any form or medium (e.g., a communication network) of wired or wireless digital data communication. Examples of communication networks include local area networks (LANs), radio access networks (RANs), metropolitan area networks (MANs), wide area networks (WANs), Worldwide Interoperability for Microwave Access (WIMAX), wireless local area networks (WLANs) using, for example, 902.11a / b / g / n and 902.20, all or part of the Internet, and any other communication system or system at one or more locations. For example, a network may communicate with Internet Protocol (IP) packets, frame relay frames, asynchronous transfer mode (ATM) cells, voice, video, data, or other suitable information between network addresses.

[0337] A computing system may include clients and servers. Clients and servers are generally remote from each other and generally interact through a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other.

[0338] In some embodiments, any or all components of a computing system, including hardware and software, may interface or dock with each other using an application programming interface (API) or a service layer. An API may include specifications for routines, data structures, and object classes. An API may be independent or dependent on a computer language, and refers to a complete interface, a single function, or even a set of APIs. The service layer provides software services to a computing system. The functions of various components of a computing system may be accessible to all service consumers using the service layer. Software services provide reusable, defined business functions through defined interfaces. For example, an interface may be software written in any suitable language that provides data in any suitable format. The API and service layer may be components integrated with other components of a computing system or independent components. In addition, without departing from the scope of this document, any or all parts of the service layer may be implemented as child or sub-modules of another software module, enterprise application, or hardware module.

[0339] Although this article contains details of many specific embodiments, these details should not be interpreted as limitations on the scope of any invention or the scope of the claims, but should be interpreted as descriptions of features unique to specific embodiments of specific inventions. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described as working in certain combinations, and even initially claimed as such, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0340] Specific embodiments of the subject matter have been described. Other embodiments, variations and permutations of the described embodiments will be apparent to those skilled in the art and are within the scope of the appended claims. Although operations are described in the drawings or in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order or sequence shown, or that all of the illustrated operations (some operations may be considered optional) be performed to achieve the desired results. In some cases, multitasking or parallel processing may be advantageous and is performed as appropriate.

[0341] Therefore, the description of the exemplary embodiments provided above does not define or constrain this document. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this document.

Claims

1. A three-dimensional (3D) display system, comprising: a display screen including a plurality of display elements; and a controller coupled to the display screen and configured to: for each of a plurality of primitives corresponding to an object, determine an electromagnetic field contribution to each of the plurality of display elements on the display screen by calculating the electromagnetic field propagation from the primitive to each of the plurality of display elements in a 3D coordinate system, wherein the primitive refers to an indivisible geometric or graphical element for input or output within a computing system; and for each of the plurality of display elements, generate a sum of the electromagnetic field contributions of each of the plurality of primitives to the display element.

2. The system according to claim 1, wherein the plurality of primitives includes at least one of the following: a point primitive, a line primitive, or a polygon primitive.

3. The system according to claim 1 or 2, wherein the controller is configured to obtain corresponding primitive data for each of the plurality of primitives, wherein the corresponding primitive data for the primitive includes at least one of the following: texture information of the primitive; shading information on one or more surfaces of the primitive; color information of the primitive; corresponding coordinate information of the primitive in the 3D coordinate system.

4. The system according to claim 1 or 2, wherein the controller is configured to determine the electromagnetic field contribution of the primitive to each of the plurality of display elements on the display screen by determining at least one distance between the display element and the primitive based on the corresponding coordinate information of the display element and the corresponding coordinate information of the primitive in the 3D coordinate system, and determining the electromagnetic field contribution of the primitive to the display element based on a predetermined expression of the primitive and the at least one distance.

5. The system according to claim 4, wherein the predetermined expression is determined based on at least one of the following: analytically calculating the electromagnetic field propagation from the primitive to the display element; solving Maxwell's equations with boundary conditions defined by the display screen; or at least one selected from the group consisting of functions including sine function, cosine function, and exponential function, wherein determining the electromagnetic field contribution includes identifying a value of at least one of the functions in a table stored in a memory.

6. The system according to claim 1 or 2, wherein the controller is configured to: determine a corresponding first electromagnetic field contribution of a first primitive among the plurality of primitives to each of the plurality of display elements; determine a corresponding second electromagnetic field contribution of a second primitive among the plurality of primitives to each of the plurality of display elements; and accumulate the electromagnetic field contribution to each of the plurality of display elements by adding the corresponding second electromagnetic field contribution to the display element to the corresponding first electromagnetic field contribution.

7. The system according to claim 1 or 2, wherein the controller is configured to calculate mathematical functions using fixed-point representation and / or floating-point representation.

8. The system according to claim 1 or 2, wherein The controller is configured to calculate a respective electromagnetic field contribution of each of the plurality of primitives to each of the plurality of display elements, wherein the calculation of the respective electromagnetic field contribution is performed without at least one selected from the group consisting of: extending the geometry of the object to the plurality of display elements, applying a visibility test before wrapping the wavefront, and making a decision or communication between parallel calculations of different primitives of the plurality of primitives; Alternatively, wherein calculating the respective electromagnetic field contribution is configured to facilitate at least one of the following: adjusting parallel calculations for a plurality of primitives to achieve speed, cost, size, or energy optimization; reducing the latency between initiating drawing and the result being ready for display; using fixed-point representation to increase precision; and optimizing the calculation speed by optimizing mathematical functions.

9. The system according to claim 1 or 2, wherein, the controller is configured to: obtain respective primitive data corresponding to each of the plurality of primitives of the object; scale the first primitive by a predetermined factor using the respective primitive data of the first primitive adjacent to the second primitive and the respective primitive data of the second primitive; and update the respective primitive data of the first primitive based on the result of the scaling, wherein the scaling is performed such that the gap between the reconstruction of the first primitive and the second primitive in 3D space is large enough to separate the first primitive and the second primitive to minimize the overlapping effect, and the gap is small enough for the reconstruction to appear seamlessly.

10. The system according to claim 1 or 2, wherein, the controller is configured to: obtain a plurality of discrete cosine transform (DCT) weights of an image to be mapped onto a designated surface of a specific primitive among the plurality of primitives; determine a respective electromagnetic field contribution of each of the primitives in the specific primitive to each of the plurality of display elements by considering the influence of the plurality of discrete cosine transform (DCT) weights of the image; decode the discrete cosine transform (DCT) weights of the image to obtain respective discrete cosine transform (DCT) amplitudes of each pixel of the image; and store values associated with the respective discrete cosine transform (DCT) amplitudes of the pixels of the image together with the primitive data of the specific primitive, wherein the controller is configured to calculate the respective electromagnetic field contribution of each of the primitives in the specific primitive to each of the plurality of display elements using the values associated with the respective discrete cosine transform (DCT) amplitudes of the pixels of the image to determine the respective electromagnetic field contribution.

11. The system according to claim 1 or 2, wherein, the controller is configured to: obtain information about a given primitive and an occluder of the given primitive, wherein the given primitive belongs to one of the plurality of primitives corresponding to the object; determine one or more specific display elements that do not contribute to the reconstruction of the given primitive due to the influence of the occluder; and For each of the one or more specific display elements, a sum of the electromagnetic field contributions of the plurality of primitive elements to the specific display element is generated by excluding the electromagnetic field contribution of the given primitive element to the specific display element.

12. The system according to claim 1 or 2, wherein, the controller is configured to: obtain information about a given primitive element and an occluder of the given primitive element, wherein the given primitive element belongs to one of the plurality of primitive elements corresponding to the object; for each of the plurality of display elements, determine a corresponding portion of the given primitive element that does not make an electromagnetic field contribution to the display element due to the influence of the occluder; and for each of the plurality of display elements, generate a sum of the electromagnetic field contributions of the plurality of primitive elements to the display element by excluding the electromagnetic field contribution of the corresponding portion of the given primitive element to the display element.

13. The system according to claim 1 or 2, wherein, the controller is configured to: obtain corresponding primitive element data for each of the plurality of primitive elements corresponding to the object; obtain corresponding geometric specular highlight information for each of the plurality of primitive elements; store the corresponding geometric specular highlight information and the corresponding primitive element data for each of the plurality of primitive elements; and determine the corresponding electromagnetic field contribution of each of the plurality of primitive elements to each of the plurality of display elements by considering the corresponding geometric specular highlight information of the primitive elements.

14. The system according to claim 1 or 2, wherein, the controller is configured to: for each of the plurality of display elements of the display screen, change a corresponding control signal using a predetermined calibration value; apply the changed corresponding control signal to each display element of the display screen; measure the output of light incident on the display screen; and evaluate the predetermined calibration value based on the measurement of the output of the light.

15. The system according to claim 1 or 2, wherein, the controller is configured to: for each display element of the display screen, obtain a sum of the corresponding electromagnetic field contributions from the plurality of primitive elements corresponding to the object; apply a corresponding mathematical transform to the sum of the corresponding electromagnetic field contributions to the display element to obtain a sum of the transformed corresponding electromagnetic field contributions to the display element; determine a corresponding control signal based on the sum of the transformed corresponding electromagnetic field contributions to the display element; and modulate the characteristics of the display element based on the corresponding control signal determined for the display element; introduce light incident on the display screen; measure the output of the light; and adjust one or more coefficients of the corresponding mathematical transform of the plurality of display elements based on the measurement result of the output of the light.

16. The system according to claim 1, further comprising: an illuminator arranged adjacent to the display screen and configured to emit light onto the display screen.

17. The system according to claim 16, wherein, the illuminator includes one of: a coherent light source, a semi-coherent light source, or an incoherent light source.

18. The system according to claim 16, wherein, The light emitted from the illuminator includes one of the following: collimated light, divergent light, or semi-collimated light.

19. The system according to claim 16, wherein, the illuminator is coupled to the controller through a first storage buffer, and the first storage buffer is configured to control the amplitude or brightness of one or more light-emitting elements in the illuminator; wherein the controller is coupled to the display screen through a second storage buffer.

20. The system according to claim 19, wherein, the size of the first storage buffer of the illuminator is smaller than the size of the second storage buffer of the display screen.

21. The system according to any one of claims 16 to 20, wherein, the illuminator includes two or more light-emitting elements each configured to emit light having different colors.

22. The system according to claim 21, wherein, the number of the light-emitting elements in the illuminator is smaller than the number of the display elements of the display screen.

23. The system according to claim 21, wherein, the controller is configured to simultaneously activate the light-emitting elements of the illuminator.

24. The system according to claim 21, wherein, the controller is configured to: modulate the display screen by sequentially using the information associated with the first color during a first time period and the information associated with the second color during a sequential second time period, and control the illuminator to sequentially turn on a first light-emitting element during the first time period to emit light having the first color, and turn on a second light-emitting element during the second time period to emit light having the second color.

25. The system according to any one of claims 16 to 20, wherein, the illuminator is configured to emit white light, and the display screen is configured to diffract the white light into light having different colors.

26. The system according to any one of claims 16 to 20, wherein, the illuminator is arranged in front of the surface of the display screen and is configured to emit the light onto the surface of the display screen at an incident angle within the range between 0 degrees and 90 degrees, and the emitted light is reflected from the surface of the display screen; alternatively, the illuminator is arranged behind the rear surface of the display screen and is configured to emit divergent light onto the rear surface of the display screen, and the emitted light transmits through the display screen and exits the display screen from the front surface of the display screen.

27. The system according to any one of claims 16 to 20, wherein, the illuminator includes: a light source configured to emit the light; and a waveguide coupled to the light source and arranged adjacent to the display screen, the waveguide being configured to receive the light emitted from the light source and guide the emitted light to the display screen.

28. The system according to claim 27, wherein, the light from the light source is coupled into the waveguide from a side cross-section of the waveguide through an optical coupler.

29. The system according to claim 27, wherein, the light source and the waveguide are integrated in a planar form and positioned on the surface of the display screen.

30. The system according to claim 27, wherein, the waveguide is configured to guide the light to irradiate the display screen uniformly.

31. The system according to claim 27, wherein, the waveguide is positioned on the rear surface of the display screen, and the light is guided to transmit through the display screen and diffract out of the display screen from the front surface of the display screen; alternatively, the controller is positioned on the rear surface of the waveguide; alternatively, the waveguide is positioned on the front surface of the display screen, and the light is guided to be incident on the front surface of the display screen and reflected by the front surface.

32. The system according to any one of claims 16 to 20, wherein, the controller is configured to: for each of the plurality of display elements, send a corresponding control signal to the display element, the control signal being used to modulate at least one characteristic of the display element based on the total sum of the electromagnetic field contributions to the display element; and send a timing control signal to the illuminator to activate the illuminator, so as to irradiate the light on the display screen, such that the modulated display elements of the display screen cause the light to form a volume light field corresponding to the object in the three-dimensional (3D) space.

33. The system according to claim 1, wherein, the controller includes a plurality of computing units, and each of the plurality of computing units is configured to perform corresponding operations on one or more of the plurality of primitives corresponding to the object.

34. The system according to claim 33, wherein, the plurality of computing units are configured to operate in parallel.

35. The system according to claim 33, wherein, the controller is locally coupled to the display screen, and each of the plurality of computing units is coupled to one or more corresponding display elements of the display screen and is configured to send a corresponding control signal to each of the one or more corresponding display elements.

36. The system according to claim 33, wherein, each of the plurality of computing units is coupled to one or more corresponding display elements of the display screen and is configured to: calculate the electromagnetic field contribution of at least one of the plurality of primitives to each of the plurality of display elements, and for each of the one or more corresponding display elements, generate the total sum of the corresponding electromagnetic field contributions of the plurality of primitives to the display element; alternatively, each of the plurality of computing units is configured to: receive the calculated electromagnetic field contributions of other primitives among the plurality of primitives to each of the one or more corresponding display elements from other computing units of the plurality of computing units, and for each of the one or more corresponding display elements, generate the total sum of the corresponding electromagnetic field contributions by adding the received calculated electromagnetic field contributions of the other primitives to the display element; Alternatively, each of the plurality of computing units is configured to: for each of the one or more corresponding display elements, generate a corresponding control signal to modulate at least one characteristic of the display element based on the sum of the corresponding electromagnetic field contributions to the display element.

37. The system according to any one of claims 33 to 36, wherein, the plurality of computing units are integrated in an integrated circuit, wherein the system further includes an illuminator located between the integrated circuit and the display screen and configured to receive a control signal from the integrated circuit and irradiate light on the display screen based on the control signal.

38. The system according to claim 37, wherein, the integrated circuit, the illuminator, and the display screen are integrated into a single unit.

39. The system according to any one of claims 33 to 36, wherein, the controller includes corresponding accumulators configured to store the accumulated results of the electromagnetic field contributions calculated by the plurality of primitives for each of the plurality of display elements of the display screen.

40. The system according to claim 39, wherein, the controller is configured to clear the accumulators at the start of a calculation operation.

41. The system according to claim 39, wherein, the controller includes corresponding storage buffers for each of the plurality of display elements, and the controller is configured to accumulate the calculated electromagnetic field contributions of the plurality of primitives to the display element to obtain a corresponding sum of the electromagnetic field contributions as the final accumulated result in the corresponding accumulator, and transfer the final accumulated result from the corresponding accumulator to the corresponding storage buffer of the display element.

42. The system according to claim 1 or 2, wherein, the controller includes at least one of the following: application specific integrated circuit ASIC, field programmable gate array FPGA, programmable gate array PGA, central processing unit CPU, graphics processing unit GPU, and standard computing unit; alternatively, the display screen includes a spatial light modulator SLM, and the spatial light modulator includes a digital micromirror device DMD or a liquid crystal on silicon LCOS device; alternatively, the display screen is configured for phase modulation, amplitude modulation, or phase and amplitude modulation.

43. The system according to claim 1 or 2, further includes: a computing device configured to generate graphic data including corresponding primitive data of the plurality of primitives corresponding to the object, wherein the controller is configured to receive the graphic data from the computing device and process the graphic data to present the object in a three-dimensional 3D space.

44. The system according to claim 43, wherein, the computing device includes an application programming interface API configured to create the plurality of primitives using the corresponding primitive data by rendering a computer-generated CG model of the object.

Images