Multi-depth three-dimensional display method and three-dimensional display engine

Abstract

The invention discloses a multi-depth three-dimensional display method and a three-dimensional display engine, and belongs to the technical field of photoelectric display. The multi-depth three-dimensional display method comprises the following steps of: decomposing a scene image into a plurality of sub-regions by constructing an image acquisition model, an image decomposition model, an image transformation modulation model and an image filtering model; transforming and modulating the plurality of sub-regions to obtain a modulated and transformed image; the three-dimensional scene image can be obtained without holographic calculation and light field coding, the method has the advantages of being high in bandwidth utilization rate, high in universality and easy to implement, and the scheme is scientific, reasonable, practical and scientific. Furthermore, the three-dimensional display engine does not need laser illumination, array devices and motion devices, and is simple in system structure and easy to implement.

Description

Technical Field

[0001] The invention relates to a multi-depth three-dimensional display method and a three-dimensional display engine, and belongs to the technical field of optoelectronic display. Background Art

[0002] Three-dimensional display technology has been widely used in various fields such as daily life, industrial and agricultural production, education and medical treatment. Recently, driven by the needs of new applications such as in-vehicle head-up display (HUD), augmented reality (AR), and virtual reality (VR), three-dimensional display technology has developed rapidly. However, the depth information and display effects displayed by the current three-dimensional display technology have not yet been able to meet the needs of natural observation of the human eye. In theory, holographic display technology is the most ideal three-dimensional display technology, but its calculation amount is huge and the display effect is restricted by the bandwidth of the spatial light modulator. The light field display technology has a moderate amount of calculation and can obtain full parallax color three-dimensional display, but its resolution is low, and the depth of field and viewing angle are limited. Volume display technology constructs three-dimensional object points in space through optical methods, which can achieve three-dimensional images with large viewing angle and depth of field, but its system is complex, and the display depth is restricted by the system's temporal resolution, and it cannot be applied to wearable display and in-vehicle display systems. Therefore, the current three-dimensional display technology is still a long way from the three-dimensional display effect that people expect. How to develop efficient three-dimensional display methods and systems to obtain high-quality three-dimensional display effects is an important problem that needs to be solved in the field.

[0003] The information disclosed in this Background Art is only for understanding the background of the inventive concept and therefore it may include information that does not constitute the prior art. Summary of the invention

[0004] In response to the above problem or one of the above problems, an object of the present invention is to provide a multi-depth three-dimensional display method, by constructing an image acquisition model, an image decomposition model, an image transformation modulation model, and an image filtering model, the scene image is decomposed into multiple sub-regions; and the multiple sub-regions are transformed and modulated to obtain modulated transformation images; and then the modulated transformation images are spatially filtered to obtain the required multi-depth three-dimensional scene images, so that the three-dimensional scene images can be obtained without holographic calculation and light field encoding. The method has the advantages of high bandwidth utilization, strong universality, and easy implementation. The scheme is scientific, reasonable, and practical.

[0005] In response to the above problem or one of the above problems, the second object of the present invention is to provide a multi-depth three-dimensional display engine, which outputs the required multi-depth three-dimensional scene image by setting a light source subsystem, a color image processing and display subsystem, an image encoder, a Fourier transform subsystem, a wavefront modulator, a spatial filter and a coordination controller. The display engine does not require laser illumination, array devices, or motion devices, and has a simple system structure and is easy to implement.

[0006] In response to the above problem or one of the above problems, the third object of the present invention is to provide a multi-depth three-dimensional display method and a three-dimensional display engine that are easy to integrate and have strong universality, and can be applicable to small and micro system scenarios such as wearable display systems and vehicle-mounted display systems.

[0007] To achieve one of the above purposes, the first technical solution of the present invention is:

[0008] A multi-depth three-dimensional display method comprises the following steps:

[0009] Step 1: acquiring a scene image of a to-be-displayed three-dimensional scene through a pre-built image acquisition model;

[0010] Step 2: using a pre-built image decomposition model, decompose the scene image into multiple sub-areas according to the depth information of the depth plane to be displayed;

[0011] Step 3, using a pre-built image transformation modulation model to transform and modulate multiple sub-regions to obtain a modulated transformation image;

[0012] Step 4: Use the pre-built image filtering model to perform spatial filtering on the modulated transformed image to obtain the required multi-depth three-dimensional scene image.

[0013] After continuous exploration and experimentation, the present invention decomposes the scene image into multiple sub-areas by constructing an image acquisition model, an image decomposition model, an image transformation modulation model, and an image filtering model; and transforms and modulates the multiple sub-areas to obtain modulated transformation images; and then spatially filters the modulated transformation images to obtain the required multi-depth three-dimensional scene images. Therefore, the three-dimensional scene image can be obtained without holographic calculation and light field coding. The present invention has the advantages of high bandwidth utilization, strong universality, and easy implementation. The scheme is scientific, reasonable, and practical.

[0014] As the preferred technical measures:

[0015] Step 1: The method of obtaining the scene image of the three-dimensional scene to be displayed through the pre-built image acquisition model is as follows:

[0016] Use a depth camera to capture a scene image;

[0017] Or, a scene image is obtained by rendering using a virtual camera in a computer.

[0018] As the preferred technical measures:

[0019] Step 2: Using the pre-built image decomposition model, the method of decomposing the scene image into multiple sub-areas according to the depth information of the depth plane to be displayed is as follows:

[0020] The scene image includes a color image and a depth image;

[0021] Determine the number of depth planes to be displayed according to the color image and the depth image;

[0022] Based on the number of depth planes, the depth map is discretized to obtain discretized depth information;

[0023] According to the discretized depth information, images with the same or similar depth information in the color image are divided into a sub-region to obtain multiple sub-region images, thereby completing the decomposition of the color image.

[0024] As the preferred technical measures:

[0025] Step 3: Using the pre-built image transformation modulation model, transform and modulate multiple sub-regions to obtain the modulated transformation image as follows:

[0026] Different sub-regions of the color image are respectively encoded with gratings of different orientations and / or different frequencies to obtain a coded color image;

[0027] Performing Fourier transform on the encoded color image to obtain the frequency spectrum of the color image in different sub-regions;

[0028] The frequency spectra of the color images in different sub-regions are modulated by wavefront on the Fourier spectrum plane to obtain wavefront information;

[0029] The modulated wavefront information is subjected to a second Fourier transform to obtain a modulated transformed image.

[0030] As the preferred technical measures:

[0031] The grating includes but is not limited to a phase grating, an amplitude grating or a phase and amplitude combined modulation grating;

[0032] The encoding method includes but is not limited to the following method:

[0033] Or / and, (a) using a grating with a certain frequency and orientation to fill a specific sub-region, and using a grating with a different frequency or orientation to fill a different sub-region; (b) using a grating function with different frequencies or orientations to multiply the image information of different sub-regions respectively;

[0034] Or / and, the Fourier transform includes but is not limited to an accurate optical Fourier transform or a Fourier transform with an additional phase factor;

[0035] Or / and, the wavefront modulation includes multiplying the frequency spectrum of the color images of different regions by a different phase factor; the phase factor includes one or more of a primary phase factor, a secondary phase factor, and a higher-order phase factor; the phase factor matches the position and depth information of the color images of different regions;

[0036] Or / and, the wavefront modulation includes performing secondary phase modulation, linear phase modulation and / or high-order phase modulation on the wavefront;

[0037] Or / and, the second Fourier transform includes but is not limited to an accurate optical Fourier transform or a Fourier transform with an additional phase factor.

[0038] As the preferred technical measures:

[0039] Step 4: Use the pre-built image filtering model to perform spatial filtering on the modulated transformed image to obtain the required multi-depth three-dimensional scene image as follows:

[0040] The modulated transform image is filtered through a spatial filtering operation, and the filtered high-order diffraction light and conjugate diffraction light in the modulated transform image are removed to obtain the required multi-depth three-dimensional scene image.

[0041] To achieve one of the above purposes, the second technical solution of the present invention is:

[0042] A multi-depth three-dimensional display method, comprising the following contents:

[0043] (1) Obtaining a color image and a depth image of a 3D scene to be displayed;

[0044] (2) discretizing the depth map according to the number of depth planes to be displayed, and decomposing the color image into multiple sub-regions according to the discretized depth information;

[0045] (3) encoding the color images of different sub-regions with gratings of different orientations and / or frequencies;

[0046] (4) performing Fourier transform on the encoded color image;

[0047] (5) performing wavefront modulation on the spectrum of the color image in different sub-regions on the Fourier spectrum plane, wherein the wavefront modulation includes performing secondary phase modulation, linear phase modulation and / or high-order phase modulation on the wavefront;

[0048] (6) performing a second Fourier transform on the modulated wavefront information;

[0049] (7) Perform spatial filtering on the output plane of the second Fourier transform to output the required multi-depth three-dimensional scene image.

[0050] Compared with existing holographic display, light field display and volume display technologies, the present invention does not require holographic calculation, light field encoding, or complex mechanical and motion structures, and has the advantages of high bandwidth utilization, easy integration, and strong universality.

[0051] Furthermore, the present invention can be applied to small and micro system scenarios such as wearable display systems and vehicle-mounted display systems.

[0052] To achieve one of the above purposes, the third technical solution of the present invention is:

[0053] A multi-depth three-dimensional display engine includes a light source subsystem, a color image processing and display subsystem, an image encoder, a Fourier transform subsystem, a wavefront modulator, a spatial filter and a coordination controller;

[0054] The light source subsystem is used to emit directional illumination light to illuminate the color image processing and display subsystem;

[0055] The color image processing and display subsystem includes an image processor and an image display, which are used to process and display images;

[0056] An image encoder, used for encoding a plurality of sub-region images output by the image display;

[0057] The Fourier transform subsystem is used to perform Fourier transform on the encoded image;

[0058] The wavefront modulator is used to multiply the frequency spectrum of the color image in different regions by a different phase factor, and is arranged near the frequency spectrum plane of the Fourier transform subsystem, and can perform wavefront modulation on the frequency spectrum of the color image in different sub-regions;

[0059] The modulated wavefront information undergoes a second Fourier transform through the Fourier transform subsystem;

[0060] The spatial filter is used to block the high-order diffracted light and the conjugate diffracted light, and is arranged near the frequency spectrum plane of the secondary transformation of the Fourier transform subsystem, and outputs the required multi-depth three-dimensional scene image after spatial filtering;

[0061] The coordination controller is used to coordinate and control the light source subsystem, the color image processing and display subsystem, the image encoder and the wavefront modulator.

[0062] The present invention outputs the required multi-depth three-dimensional scene images by setting up a light source subsystem, a color image processing and display subsystem, an image encoder, a Fourier transform subsystem, a wavefront modulator, a spatial filter and a coordination controller. It does not require laser illumination, array devices, or motion devices, and the system structure is simple and easy to implement.

[0063] Furthermore, the three-dimensional display engine of the present invention has a compact structure, is easy to integrate, and has strong universality, and can be applied to small and micro system scenarios such as wearable display systems and vehicle-mounted display systems.

[0064] As the preferred technical measures:

[0065] The image processor is used to discretize the depth of the image and decompose the color image into multiple sub-areas according to the depth;

[0066] The image display is used to output and display color images;

[0067] Or / and, the phase factor includes one or more of a primary phase factor, a secondary phase factor, and a higher-order phase factor; and the phase factor matches the position and depth information of color images of different regions.

[0068] As the preferred technical measures:

[0069] The directional illumination light emitted by the light source subsystem is collimated light, divergent light or convergent light;

[0070] The direction of directional lighting light changes or remains fixed according to a specified timing;

[0071] Or / and, the image processor is a computer or dedicated image processing hardware or image processing software;

[0072] Or / and, the image display is a transmissive or reflective display;

[0073] Or / and, the image encoder is a fixed or real-time refresh encoder, which is a phase grating or an amplitude grating;

[0074] The image display and the image encoder are physically close to each other or the image of the image display and the encoder are overlapped by optical imaging;

[0075] Or / and, the Fourier transform subsystem comprises at least one Fourier transform lens or lens group;

[0076] Or / and, the wavefront modulator is a phase type device or an amplitude type device;

[0077] Or / and, the wavefront modulator is a fixed or real-time refreshed modulator.

[0078] To achieve one of the above purposes, the fourth technical solution of the present invention is:

[0079] An electronic device comprising:

[0080] one or more processors;

[0081] A storage device for storing one or more programs;

[0082] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned multi-depth three-dimensional display method.

[0083] To achieve one of the above purposes, the fifth technical solution of the present invention is:

[0084] A computer-readable storage medium stores a computer program, which implements the above-mentioned multi-depth three-dimensional display method when executed by a processor.

[0085] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0086] After continuous exploration and experimentation, the present invention decomposes the scene image into multiple sub-areas by constructing an image acquisition model, an image decomposition model, an image transformation modulation model, and an image filtering model; and transforms and modulates the multiple sub-areas to obtain modulated transformation images; and then spatially filters the modulated transformation images to obtain the required multi-depth three-dimensional scene images. Therefore, the three-dimensional scene image can be obtained without holographic calculation and light field coding. The present invention has the advantages of high bandwidth utilization, strong universality, and easy implementation. The scheme is scientific, reasonable, and practical.

[0087] The present invention outputs the required multi-depth three-dimensional scene images by setting up a light source subsystem, a color image processing and display subsystem, an image encoder, a Fourier transform subsystem, a wavefront modulator, a spatial filter and a coordination controller. It does not require laser illumination, array devices, or motion devices, and the system structure is simple and easy to implement.

[0088] Furthermore, the three-dimensional display engine of the present invention has a compact structure, is easy to integrate, and has strong universality, and can be applied to small and micro system scenarios such as wearable display systems and vehicle-mounted display systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 is the color image (RGB) and depth image (D) of a scene;

[0090] Figure 2 A filling effect diagram of each sub-area of ​​the present invention;

[0091] Figure 3 It is an effect diagram after the various sub-regions of the present invention are combined;

[0092] Figure 4 A schematic diagram of spectrum distribution on the spectrum plane of the present invention;

[0093] Figure 5 is a schematic diagram of wavefront modulation on the spectrum plane of the present invention;

[0094] Figure 6 Schematic diagram of image contours corresponding to six depth planes of the present invention;

[0095] Figure 7 A schematic diagram of a first structure of a three-dimensional display engine of the present invention;

[0096] Figure 8 A second structural diagram of the three-dimensional display engine of the present invention;

[0097] Fig. 9 A third structural schematic diagram of the three-dimensional display engine of the present invention;

[0098] Fig.10 This is a fourth structural schematic diagram of the three-dimensional display engine of the present invention.

[0099] Description of reference numerals:

[0100] 1. Light source subsystem; 2. Color image processing and display subsystem; 3. Image encoder; 4. Fourier transform subsystem; 5. Wavefront modulator; 6. Spatial filter; 7. Three-dimensional scene image; 8. Coordination controller; 9. Relay lens; 10. Total reflection prism; 11. Semi-transparent and semi-reflective mirror; 21. Image processor; 22. Image display; 41. First Fourier transform lens; 42. Second Fourier transform lens. DETAILED DESCRIPTION

[0101] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0102] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the following detailed description of the present invention. Those skilled in the art can fully understand the present invention without the description of these details.

[0103] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0104] The first specific embodiment of the multi-depth three-dimensional display method of the present invention:

[0105] A multi-depth three-dimensional display method comprises the following steps:

[0106] Step 1: acquiring a scene image of a to-be-displayed three-dimensional scene through a pre-built image acquisition model;

[0107] Step 2: using a pre-built image decomposition model, decompose the scene image into multiple sub-areas according to the depth information of the depth plane to be displayed;

[0108] Step 3, using a pre-built image transformation modulation model to transform and modulate multiple sub-regions to obtain a modulated transformation image;

[0109] Step 4: Use the pre-built image filtering model to perform spatial filtering on the modulated transformed image to obtain the required multi-depth three-dimensional scene image.

[0110] A second specific embodiment of the multi-depth three-dimensional display method of the present invention:

[0111] A multi-depth three-dimensional display method comprises the following steps:

[0112] Step 1: Obtain a color image and a depth image of the three-dimensional scene to be displayed through a pre-built image acquisition model;

[0113] Step 2, using a pre-built image decomposition model, decompose the color image into multiple sub-regions according to the depth information of the depth plane to be displayed;

[0114] Step 3, using a pre-built image transformation modulation model to transform and modulate multiple sub-regions to obtain a modulated transformation image;

[0115] Step 4: Use the pre-built image filtering model to perform spatial filtering on the modulated transformed image and output the required multi-depth three-dimensional scene image.

[0116] A third specific embodiment of the multi-depth three-dimensional display method of the present invention:

[0117] A multi-depth three-dimensional display method comprises at least the following steps:

[0118] Step (1) obtains a color image (RGB) and a depth image (D) of a three-dimensional scene to be displayed. Step (2) discretizes the depth image according to the number of depth planes to be displayed, and decomposes the RGB image into multiple sub-regions according to the discretized depth information. Step (3) encodes the RGB images of different sub-regions with gratings of different orientations and / or different frequencies. Step (4) performs Fourier transform on the encoded RGB image. Step (5) performs wavefront modulation on the spectrum of the RGB images of different sub-regions on the Fourier spectrum plane, and the wavefront modulation includes quadratic phase modulation, linear phase modulation and / or high-order phase modulation of the wavefront. Step (6) performs a second Fourier transform on the modulated wavefront information. Step (7) performs spatial filtering on the output plane of the second Fourier transform to output the required multi-depth three-dimensional scene image.

[0119] In this example, the color image and the depth image described in step (1) may be image information obtained by taking pictures with an actual depth camera, or may be image information obtained by rendering with a virtual camera in a computer.

[0120] In this example, the images with the same or similar depth information in the RGB images described in step (2) are divided into one sub-region.

[0121] In this example, the grating described in step (3) includes but is not limited to a phase grating, an amplitude grating, and a phase and amplitude combined modulation grating.

[0122] In this example, the encoding method described in step (3) includes but is not limited to the following methods: (a) using a grating with a certain frequency and orientation to fill a specific sub-region, and using a grating with a different frequency or orientation to fill a different sub-region. (b) using a grating function with different frequencies or orientations to multiply the image information of different sub-regions respectively.

[0123] In this example, the Fourier transform in step (4) includes but is not limited to an accurate optical Fourier transform or a Fourier transform with an additional phase factor.

[0124] In this example, the wavefront modulation in step (5) includes multiplying the spectrum of the RGB images of different regions by a different phase factor. The phase factor includes one or more of a primary phase factor, a secondary phase factor, and a higher-order phase factor. The phase factor matches the position and depth information of the RGB images of different regions.

[0125] In this example, the second Fourier transform in step (6) includes but is not limited to an accurate optical Fourier transform or a Fourier transform with an additional phase factor.

[0126] In this example, in step (7), the high diffraction order and the conjugate diffraction order images of the desired multi-depth images are filtered through the spatial filtering operation.

[0127] Different from holographic display, light field display and volume display technologies, the method of the present invention has the advantages of no need for holographic calculation, no need for light field encoding, no need for complex motion devices and mechanical structures, and high bandwidth utilization.

[0128] A specific embodiment of the multi-depth three-dimensional display engine of the present invention:

[0129] A multi-depth three-dimensional display engine includes: a light source subsystem, a color image processing and display subsystem, an image encoder, a Fourier transform subsystem, a wavefront modulator, a spatial filter, a coordination controller, etc.

[0130] The light source subsystem emits directional illumination light to illuminate the color image processing and display subsystem. The color image processing and display subsystem includes an image processor and an image display. The image processor discretizes the depth of the image and decomposes the RGB image into multiple sub-areas according to the depth. The image display outputs the RGB image for display. The image encoder encodes the multiple sub-area images output by the image display. The Fourier transform subsystem performs Fourier transform on the encoded image. The wavefront modulator is arranged near the spectrum plane of the Fourier transform lens. The wavefront modulator performs wavefront modulation on the spectrum of the RGB images of different sub-areas respectively.

[0131] The wavefront modulation includes multiplying the spectrum of the RGB images of different regions by a different phase factor. The phase factor includes one or more of a primary phase factor, a secondary phase factor, and a higher-order phase factor. The phase factor matches the position and depth information of the RGB images of different regions. The modulated wavefront information is subjected to a second Fourier transform through a Fourier transform lens. The spatial filter is set near the spectrum plane of the second Fourier transform, and the wavefront outputs the required multi-depth three-dimensional scene image after spatial filtering.

[0132] In this example, the directional illumination light emitted by the light source subsystem is preferably collimated light, divergent light or convergent light. The direction of the directional illumination light emitted by the light source is preferably changed according to a specified time sequence, and less preferably fixed.

[0133] In this example, the image processor may be a computer or dedicated image processing hardware. The image display may be transmissive or reflective.

[0134] The image encoder can be fixed or refreshed in real time. The image encoder is preferably a phase grating or an amplitude grating.

[0135] The image display and the image encoder may be physically close to each other or the image of the image display may be overlapped with the encoder through optical imaging.

[0136] In this example, the Fourier transform subsystem includes at least one Fourier transform lens or lens group.

[0137] In this example, the wavefront modulator is preferably a phase type device or an amplitude type device. The wavefront modulator can be fixed or refreshed in real time.

[0138] In this example, the spatial filter is used to block the higher-order diffraction light and the conjugate diffraction light.

[0139] In this example, the coordination controller coordinates and controls the light source subsystem, the color image processing and display subsystem, the image encoder and the wavefront modulator.

[0140] The present invention does not require laser illumination, array devices, or motion devices, and has a simple system structure and is easy to implement.

[0141] The three-dimensional display engine of the present invention has a compact structure, is easy to integrate, and has strong universality, and can be applied to small and micro system scenarios such as wearable display systems and vehicle-mounted display systems.

[0142] like Figure 1-Figure 6 As shown, a specific embodiment of the method of the present invention is applied:

[0143] A multi-depth three-dimensional display method comprises the following steps:

[0144] (1) Use a depth camera to capture the 3D scene to be displayed, thereby obtaining the color image RGB and depth image D of the scene. The color image is as follows: Figure 1 In (a), the depth map is as follows Figure 1 As shown in (b) in .

[0145] (2) Discretize the depth map according to the number of depth planes to be displayed, and decompose the RGB image into multiple sub-regions based on the discretized depth information. Figure 1 As shown in (b), in this embodiment, the depth is discretized into 6 depth planes, each of which is represented by a different grayscale value. The RGB image corresponding to the area with the same grayscale value in the depth map is defined as a sub-area.

[0146] (3) Encode the RGB images of different sub-regions with gratings of different orientations and / or frequencies. Figure 2 The following figure shows the filling effect of each sub-area. Figure 3 Shown is the effect of combining the various sub-areas.

[0147] (4) Perform Fourier transform on the encoded RGB image. Input the encoded RGB image to the input surface of the Fourier transform lens, and obtain the spectrum of the image at the output surface. Figure 4 It is a schematic diagram of spectrum distribution on the spectrum plane.

[0148] (5) Wavefront modulation is performed on the spectrum of the RGB image in different sub-regions on the Fourier spectrum plane. In this embodiment, wavefront modulation is achieved by multiplying the phase factor on the screen, such as Figure 5 Shown is a schematic diagram of wavefront modulation on the spectrum plane.

[0149] (6) Perform a second Fourier transform on the modulated wavefront information.

[0150] (7) Perform spatial filtering on the output plane of the second Fourier transform to output the required multi-depth three-dimensional scene image by filtering the higher-order diffraction light and the conjugate diffraction light. Figure 6 What is shown is a schematic diagram of image contours corresponding to six depth planes, where the Z axis represents the depth direction.

[0151] See attached Figure 7 , the first specific embodiment of the three-dimensional display engine of the present invention is applied:

[0152] A multi-depth three-dimensional display engine includes a light source subsystem 1, a color image processing and display subsystem 2, an image encoder 3, a Fourier transform subsystem 4, a wavefront modulator 5, a spatial filter 6, a coordination controller 8, etc.

[0153] The light source subsystem 1 emits directional collimated illumination light to illuminate the color image processing and display subsystem 2. The color image processing and display subsystem 2 includes an image processor 21 and an image display 22.

[0154] The image processor 21 discretizes the depth of the image and decomposes the RGB image into multiple sub-areas according to the depth. The RGB image and the depth map in this embodiment are obtained by rendering with a virtual camera in a computer.

[0155] The depth map is discretized into four depth planes. The corresponding RGB image is decomposed into four sub-areas, each of which displays a letter. The image display 22 described in this embodiment is a transmissive liquid crystal display, which outputs and displays the RGB image.

[0156] The image encoder 3 is composed of four sub-areas, each of which is filled with a phase-type grating of different frequencies and orientations. The sub-areas of the image encoder 3 correspond to the sub-areas of the RGB image one by one. The encoder encodes the four sub-area images output by the image display 22. The Fourier transform subsystem 4 is composed of a first Fourier transform lens 41 and a second Fourier transform lens 42. The first Fourier transform lens 41 performs an accurate Fourier transform on the encoded image. The wavefront modulator 5 is arranged near the spectrum plane of the Fourier transform lens. The wavefront modulator 5 is composed of phase plates in four different areas, and different areas of the phase plate contain different first-order phase factors and second-order phase factors. The first-order phase factor and the second-order phase factor correspond to the three-dimensional coordinates of the output image. The light wave information passing through the wavefront modulator 5 is transformed by the second Fourier transform lens 42, and the diffracted light of the higher order and the conjugate order is filtered by the spatial filter 6, and finally a three-dimensional scene image 7 of multiple depths is output. The three-dimensional scene image 7 of multiple depths is stacked by images on four depth planes.

[0157] See attached Figure 8 , a second specific embodiment of the three-dimensional display engine of the present invention is applied:

[0158] A multi-depth three-dimensional display engine includes a light source subsystem 1, a color image processing and display subsystem 2, an image encoder 3, a Fourier transform subsystem 4, a wavefront modulator 5, a spatial filter 6, a coordination controller 8, a relay lens 9, etc.

[0159] The light source subsystem 1 emits directional convergent illumination light to illuminate the color image processing and display subsystem 2. The color image processing and display subsystem 2 includes an image processor 21 and an image display 22. The image processor 21 discretizes the depth of the image and decomposes the RGB image into multiple sub-areas according to the depth.

[0160] The RGB image and depth map in this embodiment are obtained by taking pictures with a real depth camera. The depth map is discretized into four depth planes. The corresponding RGB image is decomposed into four sub-areas, each of which displays a letter.

[0161] The image display 22 described in this embodiment is a transmissive liquid crystal display, which outputs and displays an RGB image. The RGB image is imaged on the image encoder 3 through a relay lens 9. The image encoder 3 is a transmissive phase spatial light modulator that can be refreshed in real time. The encoder is composed of four sub-areas, each of which is filled with phase-type gratings of different frequencies and orientations. The sub-areas of the encoder correspond one-to-one to the sub-areas of the RGB image. The image encoder 3 encodes the four sub-area images output by the image display 22. The Fourier transform subsystem 4 is composed of a first Fourier transform lens 41 and a second Fourier transform lens 42. The first Fourier transform lens 41 performs an accurate Fourier transform on the encoded image. The wavefront modulator 5 is arranged near the spectrum plane of the Fourier transform lens. The wavefront modulator 5 is composed of phase plates in four different areas, and different areas of the phase plate contain different first-order phase factors, second-order phase factors, and higher-order phase factors. The first-order phase factor and the second-order phase factor correspond to the three-dimensional coordinates of the output image, and the higher-order phase factor is conjugate with the aberration of the Fourier transform system. The light wave information passing through the wavefront modulator 5 is transformed by the second Fourier transform lens 42, and the diffracted light of the higher order and the conjugate order is filtered by the spatial filter 6, and finally a multi-depth three-dimensional scene image 7 is output. The multi-depth three-dimensional scene image 7 is formed by stacking images on four depth planes.

[0162] See attached Fig. 9 , a third specific embodiment of the three-dimensional display engine of the present invention is applied:

[0163] A multi-depth three-dimensional display engine includes a light source subsystem 1, a color image processing and display subsystem 2, an image encoder 3, a Fourier transform subsystem 4, a wavefront modulator 5, a spatial filter 6, a coordination controller 8, a relay lens 9, a total reflection prism 10, etc.

[0164] The light source subsystem 1 emits directional divergent illumination light, which is synthesized by independently controllable R, G, and B three-color lights. The illumination light illuminates the color image processing and display subsystem 2 through a total reflection prism 10. The color image processing and display subsystem 2 includes an image processor 21 and an image display 22. The image processor 21 discretizes the depth of the image and decomposes the RGB image into multiple sub-areas according to the depth. The RGB image and depth map in this embodiment are obtained by taking a light field camera. The depth map is discretized into four depth planes. The corresponding RGB image is decomposed into four sub-areas, each of which displays a letter. The image display 22 in this embodiment is a reflective digital micromirror device (DMD) display, which outputs and displays R, G, and B images in time-sharing, and the timing of the image time-sharing output matches the R, G, and B colors of the illumination light source. The RGB image is imaged on the image encoder 3 through a relay lens 9. The image encoder 3 is composed of four fixed sub-areas, each of which is filled with amplitude gratings of different frequencies and orientations. The sub-regions of the image encoder 3 correspond one-to-one to the sub-regions of the RGB image.

[0165] The image encoder 3 encodes the four sub-region images output by the image display 22. The Fourier transform subsystem 4 is composed of a first Fourier transform lens 41 and a second Fourier transform lens 42. The first Fourier transform lens 41 performs a Fourier transform with a quadratic phase factor on the encoded image. The wavefront modulator 5 is arranged near the spectrum plane of the Fourier transform lens. The wavefront modulator 5 is composed of phase plates in four different regions, and different regions of the phase plate contain different first-order phase factors, second-order phase factors and higher-order phase factors. The first-order phase factor and the second-order phase factor correspond to the three-dimensional coordinates of the output image, and the higher-order phase factor is conjugate with the aberration of the Fourier transform system. The light wave information passing through the wavefront modulator 5 is transformed by the second Fourier transform lens 42, and the diffracted light of the higher order and the conjugate order is filtered by the spatial filter 6, and finally a multi-depth three-dimensional scene image 7 is output. The multi-depth three-dimensional scene image 7 is formed by stacking images on four depth planes.

[0166] See attached Fig.10 , a fourth specific embodiment of the three-dimensional display engine of the present invention is applied:

[0167] A multi-depth three-dimensional display engine includes a light source subsystem 1, a color image processing and display subsystem 2, an image encoder 3, a Fourier transform subsystem 4, a wavefront modulator 5, a spatial filter 6, a coordination controller 8, a relay lens 9, a total reflection prism 10, and a semi-transparent and semi-reflective mirror 11.

[0168] The light source subsystem 1 emits directional divergent illumination light, which is synthesized by independently controllable R, G, and B three-color lights. The divergent illumination light illuminates the color image processing and display subsystem 2 through a total reflection prism 10. The color image processing and display subsystem 2 includes an image processor 21 and an image display 22. The image processor 21 discretizes the depth of the image and decomposes the RGB image into multiple sub-areas according to the depth.

[0169] The RGB image and depth map in this embodiment are obtained by taking a light field camera. The depth map is discretized into four depth planes. The corresponding RGB image is decomposed into four sub-areas, each of which displays a letter.

[0170] The image display 22 described in this embodiment is a reflective digital micromirror device (DMD) display, and the DMD display outputs and displays the R, G, and B three-color images in time-sharing, and the timing of the image time-sharing output matches the R, G, and B colors of the illumination light source. The RGB image is imaged on the image encoder 3 through the relay lens 9. The image encoder 3 is composed of four fixed sub-areas, each of which is filled with amplitude-type gratings of different frequencies and orientations. The sub-areas of the image encoder 3 correspond one-to-one to the sub-areas of the RGB image.

[0171] The image encoder 3 encodes the four sub-region images output by the image display 22. The Fourier transform subsystem 4 is composed of only the first Fourier transform lens 41. The first Fourier transform lens 41 performs accurate Fourier transform on the encoded image. The wavefront modulator 5 is arranged near the spectrum plane of the Fourier transform lens. The wavefront modulator 5 is a pure phase reflective spatial light modulator (LCOS) that can be refreshed in real time. The spatial light modulator is divided into four phase modulation areas, and the different phase modulation areas are provided with different first-order phase factors, second-order phase factors and higher-order phase factors. The first-order phase factor and the second-order phase factor correspond to the three-dimensional coordinates of the output image, and the higher-order phase factor is conjugate with the aberration of the Fourier transform system. After being reflected, the light wave information of the wavefront modulator 5 is transformed again by the first Fourier transform lens 41, and the diffracted light of the higher order and the conjugate order is filtered by the reflection of the semi-transparent and semi-reflective mirror 11 and the spatial filter 6, and finally a three-dimensional scene image 7 of multiple depths is output. The multi-depth three-dimensional scene image 7 is formed by stacking images on four depth planes. The coordination controller 8 coordinates and controls the light source subsystem 1 , the color image processing and display subsystem 2 and the wavefront modulator 5 .

[0172] An embodiment of a device applying the method of the present invention:

[0173] An electronic device comprising:

[0174] one or more processors;

[0175] A storage device for storing one or more programs;

[0176] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned multi-depth three-dimensional display method.

[0177] A computer medium embodiment using the method of the present invention:

[0178] A computer-readable storage medium stores a computer program, which implements the above-mentioned multi-depth three-dimensional display method when executed by a processor.

[0179] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, and computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0180] The present application is described by flowcharts or / and block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process or / and block in the flowchart or / and block diagram and the combination of the processes or / and blocks in the flowchart or / and block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart or / and block diagram. Figure 1 Process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0181] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 Process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0182] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0183] The model in this application is an object that objectively describes the morphological structure with the help of physical or virtual representation. The object is not equal to the object and is not limited to physical and virtual. It can be a data processing function, software program, processing mode, usage method, operation method, workflow, application process, electronic hardware, circuit module, processing system, system imitation or simulation object.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field can still modify or replace the specific implementation methods of the present invention with equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A multi-depth three-dimensional display method, characterized in that: The following steps are involved: Step 1: acquiring a scene image of a to-be-displayed three-dimensional scene through a pre-built image acquisition model; Step 2: using a pre-built image decomposition model, decompose the scene image into multiple sub-areas according to the depth information of the depth plane to be displayed; Step 3, using a pre-built image transformation modulation model to transform and modulate multiple sub-regions to obtain a modulated transformation image; Step 4: Use the pre-built image filtering model to perform spatial filtering on the modulated transformed image to obtain the required multi-depth three-dimensional scene image.

2. A multi-depth three-dimensional display method as claimed in claim 1, characterized in that: Step 1: The method of obtaining the scene image of the three-dimensional scene to be displayed through the pre-built image acquisition model is as follows: Use a depth camera to capture a scene image; Or, a scene image is obtained by rendering using a virtual camera in a computer.

3. The multi-depth three-dimensional display method according to claim 1, characterized in that: Step 2: Using the pre-built image decomposition model, the method of decomposing the scene image into multiple sub-areas according to the depth information of the depth plane to be displayed is as follows: The scene image includes a color image and a depth image; Determine the number of depth planes to be displayed according to the color image and the depth image; Based on the number of depth planes, the depth map is discretized to obtain discretized depth information; According to the discretized depth information, images with the same or similar depth information in the color image are divided into a sub-region to obtain multiple sub-region images, thereby completing the decomposition of the color image.

4. The multi-depth three-dimensional display method according to claim 1, characterized in that: Step 3: Using the pre-built image transformation modulation model, transform and modulate multiple sub-regions to obtain the modulated transformation image as follows: Different sub-regions of the color image are respectively encoded with gratings of different orientations and / or different frequencies to obtain a coded color image; Performing Fourier transform on the encoded color image to obtain the frequency spectrum of the color image in different sub-regions; The frequency spectra of the color images in different sub-regions are modulated by wavefront on the Fourier spectrum plane to obtain wavefront information; The modulated wavefront information is subjected to a second Fourier transform to obtain a modulated transformed image.

5. The multi-depth three-dimensional display method according to claim 4, characterized in that: The grating includes but is not limited to a phase grating, an amplitude grating or a phase and amplitude combined modulation grating; The encoding method includes but is not limited to the following method: Or / and, (a) using a grating with a certain frequency and orientation to fill a specific sub-region, and using a grating with a different frequency or orientation to fill a different sub-region; (b) using a grating function with different frequencies or orientations to multiply the image information of different sub-regions respectively; Or / and, the Fourier transform includes but is not limited to an accurate optical Fourier transform or a Fourier transform with an additional phase factor; Or / and, the wavefront modulation includes multiplying the frequency spectrum of the color images of different regions by a different phase factor; the phase factor includes one or more of a primary phase factor, a secondary phase factor, and a higher-order phase factor; the phase factor matches the position and depth information of the color images of different regions; Or / and, the wavefront modulation includes performing secondary phase modulation, linear phase modulation and / or high-order phase modulation on the wavefront; Or / and, the second Fourier transform includes but is not limited to an accurate optical Fourier transform or a Fourier transform with an additional phase factor.

6. The multi-depth three-dimensional display method according to claim 1, characterized in that: Step 4: Use the pre-built image filtering model to perform spatial filtering on the modulated transformed image to obtain the required multi-depth three-dimensional scene image as follows: The modulated transform image is filtered through a spatial filtering operation, and the filtered high-order diffraction light and conjugate diffraction light in the modulated transform image are removed to obtain the required multi-depth three-dimensional scene image.

7. A multi-depth three-dimensional display method, characterized in that: It includes the following: (1) Obtaining a color image and a depth image of a 3D scene to be displayed; (2) discretizing the depth map according to the number of depth planes to be displayed, and decomposing the color image into multiple sub-regions according to the discretized depth information; (3) encoding the color images of different sub-regions with gratings of different orientations and / or frequencies; (4) performing Fourier transform on the encoded color image; (5) performing wavefront modulation on the frequency spectrum of the color image in different sub-regions on the Fourier spectrum plane, wherein the wavefront modulation includes performing secondary phase modulation, linear phase modulation and / or high-order phase modulation on the wavefront; (6) performing a second Fourier transform on the modulated wavefront information; (7) Perform spatial filtering on the output plane of the second Fourier transform to output the required multi-depth three-dimensional scene image.

8. A multi-depth three-dimensional display engine, characterized in that: It includes light source subsystem, color image processing and display subsystem, image encoder, Fourier transform subsystem, wavefront modulator, spatial filter and coordination controller; The light source subsystem is used to emit directional illumination light to illuminate the color image processing and display subsystem; The color image processing and display subsystem includes an image processor and an image display, which are used to process and display images; An image encoder, used for encoding a plurality of sub-region images output by the image display; The Fourier transform subsystem is used to perform Fourier transform on the encoded image; The wavefront modulator is used to multiply the frequency spectrum of the color image in different regions by a different phase factor, and is arranged near the frequency spectrum plane of the Fourier transform subsystem, and can perform wavefront modulation on the frequency spectrum of the color image in different sub-regions; The modulated wavefront information undergoes a second Fourier transform through the Fourier transform subsystem; The spatial filter is used to block the high-order diffracted light and the conjugate diffracted light, and is arranged near the frequency spectrum plane of the secondary transformation of the Fourier transform subsystem, and outputs the required multi-depth three-dimensional scene image after spatial filtering; The coordination controller is used to coordinate and control the light source subsystem, the color image processing and display subsystem, the image encoder and the wavefront modulator.

9. A multi-depth three-dimensional display engine as claimed in claim 8, characterized in that: The image processor is used to discretize the depth of the image and decompose the color image into multiple sub-areas according to the depth; The image display is used to output and display color images; Or / and, the phase factor includes one or more of a primary phase factor, a secondary phase factor, and a higher-order phase factor; and the phase factor matches the position and depth information of color images of different regions.

10. The multi-depth three-dimensional display engine according to claim 9, characterized in that: The directional illumination light emitted by the light source subsystem is collimated light, divergent light or convergent light; The direction of directional lighting light changes or remains fixed according to a specified timing; Or / and, the image processor is a computer or dedicated image processing hardware or image processing software; Or / and, the image display is a transmissive or reflective display; Or / and, the image encoder is a fixed or real-time refresh encoder, which is a phase grating or an amplitude grating; The image display and the image encoder are physically close to each other or the image of the image display and the encoder are overlapped by optical imaging; Or / and, the Fourier transform subsystem comprises at least one Fourier transform lens or lens group; Or / and, the wavefront modulator is a phase type device or an amplitude type device; Or / and, the wavefront modulator is a fixed or real-time refreshed modulator.

Images