Augmented reality near-to-eye display device and method based on multiple layers of light guide plates

Abstract

The invention discloses an augmented reality near-to-eye display device and method based on multiple layers of light guide plates. The augmented reality near-to-eye display device comprises a multi-layer light guide illumination device, a micro display device, a light source controller, an eye movement tracking device, a main control module and the like. The light sources in the double-layer or multi-layer light guide lighting device are sequentially lightened, the light guide plate receives light from the light sources and transmits the light, the light is emitted through the dot matrix lists on the light guide plate to form a point light source array and then illuminates the micro-display device, and the micro-display device synchronously displays two-dimensional images of the light guide plate on the corresponding layer and enters pupils of human eyes so that the corresponding images can be watched. A corresponding image sequence loaded to the micro-display device is calculated through the main control module, and images on different depth focusing planes are fused with a real scene, so that an augmented reality type three-dimensional image can be always seen in a human eye pupil movement process, and therefore, augmented reality near-to-eye display is realized; the problems of convergence adjustment conflicts and too small field angle in the near-to-eye display device are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of near-eye display technology, and more specifically, to a near-eye display device and a display method for providing augmented reality. Background Art

[0002] In recent years, near-eye display technology has received more and more attention and research. Near-eye display products are head-mounted devices. The distance between the human eye and the display module is relatively close. It uses optical technology to realize virtual and enlarged images in the user's near-eye range to provide users with intuitive images, videos or text information. Today, related applications have gradually been applied to film and television, games, medical and other industries.

[0003] At present, near-eye display devices used to achieve three-dimensional display usually use traditional near-eye display technology to present three-dimensional images through binocular vision, that is, different images are loaded in the left and right eyes using the convergence characteristics of the two eyes, and the three-dimensional images are observed by the human eye through the brain fusion using binocular parallax. However, based on this principle, the focus position of each eye is actually on the display screen, making it difficult to present monocular depth cues. At the same time, for this reason, the depth perception formed by human binocular parallax and the depth perception formed by monocular accommodation are different, and there is a conflict between convergence and accommodation. Long-term viewing can cause dizziness, nausea and other discomforts to the viewer.

[0004] The field of augmented reality shows the prospect of integrating display graphics and human vision. An important requirement in augmented reality applications is a larger field of view, which improves the user experience through a large field of view. However, since augmented reality wearable devices require small size and compact structure, the field of view is constrained by the volume of the system and is limited to a narrow field of view, which makes it difficult to meet the user's usage needs. Therefore, augmented reality near-eye display technology urgently needs to solve the two major problems of convergence accommodation conflict and small field of view.

[0005] Patent document US2015177514A1 proposes a near-eye display device based on a single-layer light guide plate. The device emits light through a light source on the side of the light guide plate, so that the light is transmitted in the light guide plate, and after being emitted, a point light source array is formed. After illuminating the LCD display device, the image is imaged near the pupil of the human eye, thereby realizing near-eye display. Since the field of view angle range of the near-eye display device is related to the size of the point array on the light guide plate and the LCD display device, the user can be given a suitable field of view angle range. However, the device can only display flat images, that is, the display effect is two-dimensional, and the convergence adjustment conflict cannot be resolved. Long-term viewing will still make viewers feel uncomfortable, affecting the normal and natural viewing experience. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a near-eye display device and display method that can realize augmented reality. Images on different depth focusing planes are integrated with real scenes, so that the human eye can always see augmented reality three-dimensional images during pupil movement, thereby realizing augmented reality near-eye display, and effectively solving the problems of convergence adjustment conflict and small field of view angle in near-eye display devices.

[0007] The technical solution of the present invention is as follows:

[0008] On the one hand, the present invention provides an augmented reality near-eye display device based on a multi-layer light guide plate, including an eye tracking device for obtaining position information of a human pupil, which is characterized in that it also includes:

[0009] A light guide lighting device, comprising at least two layers of light guide plates and light sources corresponding to each layer of light guide plates, wherein the surface of the light guide plates is provided with point units arranged in an array, and when the light emitted by the light source is transmitted to the light guide plates, it is scattered by each point unit to form an array of point light sources at different depths and positions from the human eye;

[0010] A light source controller, connected to the light source, and used to control the on and off of the light source;

[0011] The micro-display device is located between the light guide lighting device and the pupil of the human eye, loaded with a point light source array corresponding to the light guide plate and a time-series image corresponding to the pupil position, and is used to receive the light emitted by the point light source array and send it to the pupil position of the human eye after modulation;

[0012] The main control module is connected to the eye tracking device, the micro display device, and the light source controller, respectively, and is used to obtain the human eye pupil position information, layer the target three-dimensional image, calculate the target three-dimensional image to be loaded on the micro display device, and synchronously control the switching of the light source and the loading and refreshing of the image required by the micro display device by controlling the light source controller, so as to ensure that the light source corresponding to each layer of the light guide plate emits light in a certain sequence and only lights up once, so that each corresponding sequential image is loaded by the micro display device and only loaded once.

[0013] Furthermore, the light guide plate is a transparent non-luminous device, and the dot units on the light guide plate are concave dots or convex dots.

[0014] Furthermore, the concave or convex points are circular, triangular or square.

[0015] Furthermore, the dot units of each light guide plate form a dot unit array with equilateral triangles staggered and distributed, and a dot unit array with triangles spaced and distributed.

[0016] Furthermore, the dot units of each light guide plate form a dot unit array with a square staggered distribution, or a dot unit array with a square interval distribution.

[0017] Furthermore, the dot unit arrays may be staggered, spaced, partially overlapped, or completely overlapped.

[0018] Furthermore, the light source is an LED lamp, which is made by combining "blue light technology" with fluorescent powder, or by mixing multiple monochromatic lights.

[0019] Furthermore, the micro display device is a transmissive LCD display screen or a transmissive OLED display screen.

[0020] Furthermore, the main control module is a computer, a single chip microcomputer, an embedded system or a PLC.

[0021] On the other hand, the present invention further provides an augmented reality near-eye display method based on a multi-layer light guide plate, which is characterized in that it includes the following steps:

[0022] S1. Obtain the position information (x, y, z) of the pupil of the human eye through an eye tracking device;

[0023] S2. According to the number of light guide plates n and the focal plane depth corresponding to each layer of light guide plates, the target three-dimensional image is layered and decomposed to obtain n layers of two-dimensional images with depth information;

[0024] S3. Calculate the target image to be loaded into the micro display device according to the coordinates (x, y, z) of the pupil position of the human eye;

[0025] S4. The main control module controls the light source controller to light up and turn off the light source in sequence, and synchronously controls the micro display device to load the corresponding image;

[0026] S5.: Cycle the light source on and off and load the corresponding image on the micro-display device. Light is emitted from the multi-layer light-guiding lighting device and reaches the vicinity of the pupil of the human eye after being modulated by the micro-display device. The image is integrated with the real scene, so that the human eye can see the three-dimensional scene of augmented reality.

[0027] Further, the step S2. hierarchically decomposes the target three-dimensional image according to the number of layers n of the light guide plates and the depth of the focal plane corresponding to each layer of the light guide plates to obtain n layers of two-dimensional images with depth information, specifically comprising:

[0028] S2.1 scaling and adjusting the input three-dimensional image according to the focal planes corresponding to the light guide plates of each layer;

[0029] S2.2 determining that the three-dimensional image is to be divided into n layers according to the number of light guide plates n, determining the interval of the depth of each layer of the two-dimensional image according to the spacing between the light guide plates of each layer, and determining the plane depth corresponding to the n layers of the two-dimensional image obtained by decomposition according to the depth of the focusing plane corresponding to each layer of the light guide plates;

[0030] S2.3 projects different objects onto a plane close to the depth of the object according to the depth of each object in the three-dimensional image, thereby obtaining n layers of two-dimensional images with depth information.

[0031] Further, the step S3. calculating the target image to be loaded on the micro display device according to the human eye pupil position coordinates (x, y, z) specifically includes:

[0032] S3.1 moves the input two-dimensional image according to the coordinates of the pupil position of the human eye to ensure that any sub-image is aligned with the corresponding point and the pupil of the human eye;

[0033] S3.2 Determine the size of the sub-image according to the pupil size of the human eye and the dot spacing on the light guide plate;

[0034] S3.3 determining the position and arrangement of the sub-image in the target image according to the position and arrangement of the dot array on the light guide plate;

[0035] S3.4 processes the input two-dimensional image, fills the area without objects in the image with white, and then decomposes the image into tiled sub-images that meet the conditions, and flips each sub-image vertically and horizontally at the same time to obtain the target image required to be loaded on the micro display.

[0036] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages:

[0037] 1. The device of the present invention utilizes a multi-layer light guide lighting device, and sequentially illuminates each layer of light guide lighting device to illuminate the time-sequential image loaded on the micro-display device, and sequentially presents image sequences of different depths to the human eye. Light guide lighting devices at different distances from the micro-display device correspond to time-sequential images of a certain depth, so that after the time-sequential images are sent to the human eye, images of multiple depths can be viewed, achieving a three-dimensional display effect and solving the convergence adjustment conflict.

[0038] 2. The device of the present invention uses a multi-layer light guide lighting device to light up in time, and the micro display device loads a sequential image, that is, each time a light source in a multi-layer light guide lighting device is lit, the corresponding sequential image is loaded, and the switching frequency of the light source matches the refresh frequency of the image sequence. In a sequence cycle, each layer of light guide plates is lit and only lit once, and each sequential image is loaded and only loaded once, so that through the visual persistence effect of the human eye, multiple depth images can be seen at the same time, achieving a three-dimensional display effect.

[0039] 3. The dot arrays on the light guide plates in each light guide lighting device in the device of the present invention are arranged differently. The dot spacings of different dot arrays on each light guide plate image the corresponding time-series images to the focusing planes at different depths from the human eye. The arrangement positions and arrangement quantities of different dot arrays on each light guide plate correspond to the time-series images at different regional positions or different regional ranges on the microdisplay device. Multi-layer light guide lighting devices with the same dot array arrangement cannot meet the geometric relationship of each depth time-series image, and it is easy to see missing or aliased images, and the depth perception is confused. By arranging different dot arrays on different layers, it is effectively achieved that each time-series image appears in the correct position, ensuring that the human eye obtains a good viewing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Attached Figure 1 It is a structural schematic diagram of Embodiment 1 of an augmented reality near-eye display device based on a multi-layer light-guiding illumination device of the present invention;

[0041] Attached Figure 2 A cross-sectional schematic diagram of a concave array light guide plate provided in Embodiment 1 of the present invention;

[0042] Attached Figure 3 A cross-sectional schematic diagram of a convex dot array light guide plate provided in Embodiment 1 of the present invention;

[0043] Attached Figure 4 A schematic diagram of an equilateral triangle staggered point array light guide plate provided in Embodiment 1 of the present invention;

[0044] Attached Figure 5 A schematic diagram of a triangular spaced dot array light guide plate provided in Embodiment 1 of the present invention;

[0045] Attached Figure 6 A schematic diagram of a square staggered dot array light guide plate provided in Embodiment 1 of the present invention;

[0046] Attached Figure 7 A schematic diagram of a square-spaced-dot array light guide plate provided in Embodiment 1 of the present invention;

[0047] Attached Figure 8 A schematic diagram of the structure of an augmented reality near-eye display system based on a multi-layer light-guiding lighting device provided in Embodiment 2 of the present invention;

[0048] Attached Fig. 9 A flowchart providing an overview of the display method of the augmented reality near-eye display system provided in Embodiments 1 and 2 of the present invention.

[0049] In the figure: 100 is a double-layer light guide lighting device, 101 is a first light guide lighting device, 102 is a second light guide lighting device, 110 is a micro display device, 120 is a light source controller, 130 is an eye tracking device, 131 is an eye tracking sensor, 132 is an eye tracking driver, 140 is a main control module, 201 is a first light source, 202 is a second light source, 211 is a first light guide plate, and 212 is a second light guide plate;

[0050] 300 is a multi-layer light guide lighting device, 301 is a light guide lighting device, 401 is a light source, 402 is a light guide plate, 901 is a first light guide plate dot, 902 is a second light guide plate dot, 911 is a first light guide plate triangular point, 912 is a second light guide plate triangular point, 921 is a first light guide plate square point, and 922 is a second light guide plate square point.

[0051] It should be understood that the above drawings are only schematic and are not drawn to scale. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The above scheme is further described below in conjunction with specific implementation examples. The preferred embodiments of the present invention are described in detail as follows:

[0054] Example 1

[0055] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the structure of an augmented reality near-eye display device according to an embodiment 1 of the present invention based on a multi-layer light guide lighting device. Figure 1 As shown, it includes a light-guiding lighting device 100 , a micro-display device 110 , a light source controller 120 , an eye-tracking device 130 , and a main control module 140 .

[0056] The double-layer light guide lighting device 100 is located behind the micro display device 110 and is used to provide an array of point light sources at different depths and positions from the human eye to provide illumination for the micro display device 110. The double-layer light guide lighting device 100 is composed of a first light guide lighting device 101 and a second light guide lighting device 102, which respectively correspond to the focal planes farther and closer to the human eye after imaging. The first light guide lighting device 101 and the second light guide lighting device 102 are stacked and placed in sequence. The spacing between the first light guide lighting device 101 and the second light guide lighting device 102 is determined by the depth of the three-dimensional scene to be presented, the distance from the human eye to the micro display device 110, and the distance from the human eye to the double-layer light guide lighting device 100.

[0057] The first light guide lighting device 101 is composed of a first light source 201 and a first light guide plate 202, and is used to create a point light source array far from the human eye. The second light guide lighting device 102 is composed of a second light source 211 and a second light guide plate 212, and is used to create a point light source array close to the human eye.

[0058] The first light source 201 and the second light source 211 can be LED lamps, or halogen lamps, or fluorescent lamps, etc., wherein the LED lamp can be a white LED lamp made by using a method of combining blue light technology with phosphors, or a white LED lamp made by a method of mixing multiple monochromatic lights, for providing white illumination light. The first light source 201 and the second light source 211 are respectively located on the sides of the first light guide plate 202 and the second light guide plate 212, and are connected to the light source controller 120. The first light source 201 illuminates the first light guide plate 202 from the side of the first light guide plate 202. The light emitted by the first light source 201 will be transmitted in the first light guide plate 202, and after encountering a point on the surface of the first light guide plate 102, the light will be scattered and form a small bright spot, thereby creating a point light source array. The second light source 211 illuminates the second light guide plate from the side of the second light guide plate 212. The light emitted by the second light source 211 will be transmitted in the second light guide plate 212, and after encountering a point on the surface of the second light guide plate 112, the light will be scattered and form a small bright spot, thereby creating a point light source array. The position of the light source in each light guide lighting device can be adjusted according to actual needs. It can be located only on one side of the corresponding light guide plate, or multiple light sources can be located on different sides. When lighting, the light guide plate can be illuminated from only one side, or multiple sides can be illuminated simultaneously.

[0059] The surfaces of the first light guide plate 202 and the second light guide plate 212 are both etched with a plurality of tiny dots to form an array. The first light guide plate 202 receives and propagates light from the first light source 201. The two-dimensional dot array on the surface of the first light guide plate 202 cooperates with the first light source 201, and light is emitted from the dots to produce an array of point light sources. The second light guide plate 212 receives and propagates light from the second light source 211. The two-dimensional dot array on the surface of the second light guide plate 212 cooperates with the second light source 211, and light is emitted from the dots to produce an array of point light sources. The first light guide plate 202 and the second light guide plate 212 are both transparent, and light from the first light guide lighting device 101 can pass through the second light guide plate 212 in the second light guide lighting device 102 to provide illumination for the micro display 110. The arrangement of the two-dimensional dot array on the light guide plate is related to the range of the field of view angle. For example, when the dot array is a 3×3 rectangular array, the 9 sub-images corresponding to the micro-display device 110 can be projected, and the human eye can see the corresponding 9 sub-images on the focusing plane corresponding to the light guide plate. Of course, the dot array on the surface of the light guide plate is far more than 3×3. When the positional relationship between each light guide plate and the micro-display device 110, and the micro-display device 110 and the human eye is certain, the number of dots is determined by the image size of the target image, and more dots can be prepared according to actual needs and processing accuracy. The shape of the dot array can be rectangular, circular or other shapes; the dot array can be located at any position of the light guide plate or on the entire surface of the light guide plate.

[0060] The first light guide lighting device 101 and the second light guide lighting device 102 are used to emit light from positions at different depths from the pupil of the human eye, so that the human eye can obtain corresponding images on focal planes at different depths. The distance between the first light guide lighting device 101, the second light guide lighting device 102 and the pupil of the human eye determines the depth position of the focal plane where the human eye can see a clear image.

[0061] The first light guide plate 202 in the first light guide lighting device 101 and the second light guide plate 212 in the second light guide lighting device 102 have different surface dot array distributions. The arrangement positions and arrangement quantities of the dot arrays on the two light guide plates can be adjusted according to actual needs, so as to realize projecting images of different regional positions or different regional ranges on the micro display device 110 onto focal planes of different depths. When the size and position of the pupil of the human eye are constant, the distance between the first light guide plate and the micro display device 110, the distance between the micro display device 110 and the human eye, and the spacing between the midpoints of the dot array on the first light guide plate 202 determine the depth position of the farther focal plane where the corresponding image is located; the distance between the second light guide plate 212 and the micro display device 110, the distance between the micro display device 110 and the human eye, and the spacing between the midpoints of the dot array on the second light guide plate determine the depth position of the closer focal plane where the corresponding image is located.

[0062] The micro display device 110 is a transmissive display device, which can be a transmissive LCD display screen, a transmissive LED display screen, a transmissive OLED display screen, etc. It is located between the human eye and the double-layer light guide lighting device, and is used to load the rendered image sequence, modulate the light from the first lighting device 101 and the second lighting device 102 respectively, and then the light forms an image near the pupil of the human eye. The distance between the micro display device 110 and the human eye is generally 20mm-50mm. The image loaded by the micro display device 110 is a decomposed sub-image of the target two-dimensional image after processing and tile distribution. Each sub-image corresponds to the light irradiated to different positions on the micro display device 110, that is, the light emitted from each point corresponds to a sub-image. Each image loaded in sequence on the micro display device 110 corresponds to a different light guide lighting device one by one, that is, when any light guide lighting device is lit, the micro display device 110 loads the corresponding image. The refresh rate of the micro display device 110 is determined by the refresh rate of the three-dimensional image that the human eye wants to see. For example, if the refresh rate of the three-dimensional image that the human eye wants to see is 30Hz, the refresh rate of the micro display device 110 needs to be 60Hz. The micro display device 110 is connected to the main control module 140, and its display image, display frame rate, resolution, etc. are controlled by the main control module 140. Its control mode is determined by the main control module 140. It is generally connected to the main control module 140 through video interfaces such as VGA, HDMI, DVI, DisplayPort, USB, serial port, and general I / O.

[0063] The light source controller 120 is connected to the first light source 201, the second light source 211 and the main control module 140, and is used to control the switching of the first light source 201 and the second light source 211. By lighting up the first light source 201 and the second light source 211 in sequence, the first light guide plate 202 and the second light guide plate 212 are lit in sequence, changing the direction of the light incident on the micro display device 110, cooperating with the micro display device 110 to display the corresponding images in sequence in a time-sharing manner, so that the human eye can obtain the corresponding images on the focusing planes at different depths.

[0064] The eye tracking system 130 is connected to the main control module 140 and is composed of an eye tracking driver 132 and an eye tracking sensor 131, and is used to track the spatial position of the pupil of the human eye and the spatial position of the gaze point. The eye tracking sensor 131 can be a CCD or CMOS camera module, or a combination of multiple single-point detectors.

[0065] The main control module 140 can be a computer, a single chip microcomputer, an embedded system, a PLC, etc. The main control module 140 has three functions. On the one hand, the target three-dimensional image is input, and the three-dimensional image is calculated and layered according to the depths of the focal planes corresponding to the two light guide plates to obtain two two-dimensional images of corresponding depths. On the other hand, the position information is processed according to the position of the human eye. The processing is mainly to calculate the target image to be loaded according to the position of the human eye pupil, and then the corresponding image is loaded onto the micro display device 110 to ensure that the light emitted from the double-layer light guide lighting device 100 can be transmitted to the position of the human eye pupil after being modulated by the micro display device 110. On the other hand, the main control module 140 controls the corresponding first light source 201 or the second light source 211 in the double-layer light guide lighting device 100 to emit light through the light source controller 120, and at the same time controls the micro display device 110 to load the corresponding image, ensuring that the light emitted from the double-layer light guide lighting device 100 can be modulated by the micro display device 110 so that the human eye can see the image on the corresponding focal plane, and the image is combined with the real scene to realize an augmented reality three-dimensional image display with two depths.

[0066] The points on the surfaces of the first light guide plate 202 and the second light guide plate 212 may be concave points, which are lower than the surfaces of the light guide plates, such as Figure 2 As shown in the cross-sectional view of the light guide plate; it can also be a convex point, which is higher than the surface of the light guide plate, such as Figure 3 As shown in the cross-sectional view of the light guide plate. At the same time, the shape of the dots can be circular, square, triangular or any other shape, as long as the two-dimensional dot array can form a point light source array after the light guide plate is illuminated from the side. The dot arrays on the surfaces of the two light guide plates can be staggered, partially overlapped or completely overlapped.

[0067] The first light guide plate 202 and the second light guide plate 212 may have equilateral triangle dot arrays arranged alternately on the two light guide plates. The distance between two adjacent dots in each direction of a single light guide plate is equal, and the dot arrays on the two light guide plates do not overlap. Figure 4 As shown. Among them, the dot array formed by the first light guide plate dots 901 is located on the surface of the first light guide plate 202, and the dot array formed by the second light guide plate dots 902 is located on the surface of the second light guide plate 212. The main control module 140 lights up the first light source 201 through the light source controller 120, and the point light source emitted from the first light guide plate dots 901 can be obtained. The main control module 140 lights up the second light source 211 through the light source controller 120, and the point light source emitted from the second light guide plate dots 902 can be obtained. The light emitted by the point light source is modulated by the micro display device 110 and reaches the pupil position of the human eye.

[0068] The first light guide plate 202 and the second light guide plate 212 may be arranged with triangular dot arrays at intervals, and the distance between two obliquely adjacent points of the two light guide plates is equal, and the dot arrays on the two light guide plates do not overlap. Figure 5 As shown. Among them, the dot array formed by the triangular points 911 of the first light guide plate is located on the surface of the first light guide plate 202, and the dot array formed by the triangular points 912 of the second light guide plate is located on the surface of the second light guide plate 212. The main control module 140 lights up the first light source 201 through the light source controller 120, and the point light source emitted from the triangular point 911 of the first light guide plate can be obtained. The main control module 140 lights up the second light source 211 through the light source controller 120, and the point light source emitted from the triangular point 912 of the second light guide plate can be obtained. The light emitted by the point light source is modulated by the micro display device 110 and reaches the pupil position of the human eye.

[0069] The first light guide plate 202 and the second light guide plate 212 may have square dot arrays arranged alternately on the two light guide plates. The distance between two adjacent dots in the horizontal and vertical directions of a single light guide plate is equal, and the dot arrays on the two light guide plates do not overlap. Figure 6 As shown. Among them, the dot array formed by the square dots 921 of the first light guide plate is located on the surface of the first light guide plate 202, and the dot array formed by the square dots 922 of the second light guide plate is located on the surface of the second light guide plate 212. The main control module 140 lights up the first light source 201 through the light source controller 120, and the point light source emitted from the square dots 921 of the first light guide plate can be obtained. The main control module 140 lights up the second light source 211 through the light source controller 120, and the point light source emitted from the square dots 922 of the second light guide plate can be obtained. The light emitted by the point light source is modulated by the micro display device 110 and reaches the pupil position of the human eye.

[0070] The first light guide plate 202 and the second light guide plate 212 may be arranged with square dot arrays at intervals, and the distances between two adjacent dots in the horizontal and vertical directions of the two light guide plates are equal, and the dot arrays on the two light guide plates do not overlap. Figure 7 As shown. Among them, the dot array formed by the square dots 921 of the first light guide plate is located on the surface of the first light guide plate 202, and the dot array formed by the square dots 922 of the second light guide plate is located on the surface of the second light guide plate 212. The main control module 140 lights up the first light source 201 through the light source controller 120, and the point light source emitted from the square dots 921 of the first light guide plate can be obtained. The main control module 140 lights up the second light source 211 through the light source controller 120, and the point light source emitted from the square dots 922 of the second light guide plate can be obtained. The light emitted by the point light source is modulated by the micro display device 110 and reaches the pupil position of the human eye.

[0071] In this embodiment, the light source in the double-layer light guide lighting device 100 and the image sequence to be loaded on the micro display device 110 are lit and loaded in time sequence. That is, in each sequence, it is ensured that the light source corresponding to each layer of the light guide plate is lit and only lit once, and each corresponding time sequence image is loaded and only loaded once. At the same time, it should be noted that the dot arrays on the two light guide plates in the double-layer light guide lighting device 100 correspond to the time sequence images one by one, and each dot in the dot array corresponds to a sub-image.

[0072] Assume that there are two two-dimensional images obtained by layering three-dimensional images, the first image corresponds to a distant object, and the second image corresponds to a nearby object, and the distant object is located in the middle of the original three-dimensional image, and the nearby object is located around the original three-dimensional image. According to the corresponding relationship, the first light guide plate 211 farther from the micro display device 110 corresponds to the first sequential image, and the second light guide plate 212 closer to the micro display device corresponds to the second sequential image. Due to the different distances, the dot array spacing on the first light guide plate 211 is larger than that on the second light guide plate 212. At the same time, due to the different positions of the objects in the sequential images, the dot arrays on the two light guide plates are arranged differently. In order to avoid image aliasing observed by the human eye, the dot arrays can be selected to be spaced or staggered. According to the dot array arrangement, the main control module 140 processes the two two-dimensional images to obtain two sequential images, and each sequential image has U tiled sub-images. Taking the occlusion relationship into consideration, the first image has a V-width sub-image decomposed from a distant object in the middle, and is surrounded by UV-width sub-images that are all pure white, while the second image has a V-width sub-image decomposed from a nearby object in the middle. Since the micro-display device 110 is a transmissive display device, the non-pure white image is seen when the pure white image is superimposed on the non-pure white image. Then, under the control of the main control module 140, the light source in the double-layer light-guiding lighting device 100 is sequentially lit and the sequential image is synchronously loaded on the micro-display device 110, so that the human eye can observe the three-dimensional image, and the image is integrated with the real scene, so that the human eye can see the augmented reality three-dimensional scene.

[0073] Example 2

[0074] An embodiment of an augmented reality near-eye display device based on a multi-layer light guide plate of the present invention is as follows: Figure 6 As shown, the augmented reality near-eye display device based on the multi-layer light guide plate includes a multi-layer light guide lighting device 300, a micro display device 110, a light source controller 120, an eye tracking device 130, and a main control module 140.

[0075] The multi-layer light guide lighting device 300 is located behind the micro display device 110 and is used to provide an array of point light sources at different depths and positions from the human eye to provide illumination for the micro display device 110. The multi-layer light guide lighting device 300 is a one-dimensional array (N≥3) composed of N light guide lighting devices 301, which are stacked and placed in sequence, respectively corresponding to the focal planes at different depths from the human eye after imaging. The spacing between adjacent light guide lighting devices 301 is determined by the depth of the required three-dimensional scene, the distance from the human eye to the micro display device 110, and the distance from the human eye to the multi-layer light guide lighting device 300.

[0076] The light guide lighting device 301 , which is composed of a light source 401 and a light guide plate 402 , is used to create a point light source array to provide lighting for the micro display device 110 .

[0077] The light source 401 can be an LED lamp, or a halogen lamp, or a fluorescent lamp, etc., wherein the LED lamp can be a white LED lamp made by using a method of combining blue light technology with a phosphor, or a white LED lamp made by a method of mixing multiple monochromatic lights, for providing white illumination light. The light source 401 is located on the side of the light guide plate 402 and is connected to the light source controller 120. The light source 401 illuminates the light guide plate 402 from the side of the light guide plate 402. The light emitted by the light source 401 will be transmitted in the light guide plate 402, and after encountering a point on the surface of the light guide plate 402, the light will be scattered and form a small bright spot, thereby creating a point light source array. The position of the light source can be adjusted according to actual needs. It can be located only on one side of the light guide plate, or multiple light sources can be located on different sides. When illuminating, the light guide plate can be illuminated from only one side, or multiple sides can be illuminated simultaneously. The surface of the light guide plate 402 is etched with multiple tiny points to form an array. The light guide plate 402 receives light from the light source 401 and transmits it. The two-dimensional dot array on the surface of the light guide plate 402 cooperates with the light source 401, and the light is emitted from the point to produce a point light source array. All the stacked light guide plates 402 in the multi-layer light guide lighting device 300 are transparent, and the light emitted from the light guide lighting device far away from the human eye can pass through the light guide plate in the nearby light guide lighting device to provide illumination for the micro display 110. The arrangement of the two-dimensional dot array is related to the range of the field of view. For example, when the dot array is a 3×3 rectangular array, the 9 sub-images corresponding to the micro display device 110 can be projected, and the human eye can also see the corresponding 9 sub-images on the focusing plane corresponding to the light guide plate. When the positional relationship between each light guide plate 402 and the micro display device 110, and the micro display device 110 and the human eye is certain, the number of points is determined by the image size of the target image, and more points can be prepared according to actual needs and processing accuracy. The shape of the dot array can be rectangular, circular or other shapes; the dot array can be located at any position of the light guide plate 402 or on the entire surface of the light guide plate 402. The spacing between the dots in the dot array can be adjusted according to actual needs, so as to control the depth of the focal plane where the human eye can see a clear image. The dots on the surface of the light guide plate 402 can be concave and lower than the surface of the light guide plate 402, or convex and higher than the surface of the light guide plate 402; at the same time, the shape of the dots can be circular, square, triangular or any other shape, as long as the light guide plate 402 is illuminated from the side by the light source, the dot array can form a point light source array.

[0078] The micro display device 110 is located between the human eye and the multi-layer light guide lighting device 300. It is a transmissive display device, which can be a transmissive LCD display screen, a transmissive LED display screen, a transmissive OLED display screen, etc. It is used to load the rendered image, modulate the light from the multi-layer multi-light lighting device 300, and then the light forms an image near the pupil of the human eye. The distance between the micro display device 110 and the human eye is generally 20mm-50mm. The image loaded by the micro display device 110 is a decomposed sub-image of the target two-dimensional image after processing and tile distribution. Each sub-image corresponds to the light irradiated to different positions on the micro display device 110, that is, the light emitted from each point corresponds to a sub-image. Each image loaded sequentially on the micro display device 110 corresponds to a stacked arrangement of different light guide lighting devices 301, that is, any light guide lighting device 301 in the multi-layer light guide lighting device 300 is lit, and the micro display device 110 loads the corresponding image. The refresh rate of the micro display device 110 is determined by the refresh rate of the three-dimensional image that the human eye wants to see. For example, if the refresh rate of the three-dimensional image that the human eye wants to see is 30Hz, and the number of light guide lighting devices 301 is N=4, then the refresh rate of the micro display device 110 needs to be 120NHz. The micro display device 110 is connected to the main control module 140, and its display image, display frame rate, resolution, etc. are controlled by the main control module 140. Its control mode is determined by the main control module 140. It is generally connected to the main control module 140 through video interfaces such as VGA, HDMI, DVI, DisplayPort, USB, serial port, and general I / O.

[0079] The light source controller 120 is connected to all stacked light sources 401 and the main control module 140 in the multi-layer light guide lighting device 300, and is used to control the switching of the light sources 401 of the stacked light guide lighting device 301 in the multi-layer light guide lighting device 300. By sequentially lighting up the stacked light sources 401, the corresponding light guide plates 402 are sequentially lit, changing the direction of the light incident on the micro display device 110, and cooperating with the micro display device 110 to sequentially display the corresponding images in time division, so that the human eye can obtain the corresponding images on the focusing planes at different depths.

[0080] The eye tracking system 130 is connected to the main control module 140 and is composed of an eye tracking driver 132 and an eye tracking sensor 131, and is used to track the spatial position of the pupil of the human eye and the spatial position of the gaze point. The eye tracking sensor 131 can be a CCD or CMOS camera module, or a combination of multiple single-point detectors.

[0081] The main control module 140 can be a computer, a single-chip microcomputer, a PLC, etc. The main control module 140 has three functions. On the one hand, the target three-dimensional image is input, and the three-dimensional image is calculated and layered according to the focal plane depths corresponding to the stacked light guide plates 301 in the multi-layer light guide lighting device 300 to obtain a plurality of two-dimensional images of corresponding depths; on the other hand, the position information is processed after the gaze observation area is determined according to the position of the human eye. The processing is mainly to calculate the target image to be loaded according to the position of the human eye pupil, and then load the corresponding image onto the micro display device 110 to ensure that the light emitted from the multi-layer light guide lighting device 300 passes through the micro display device 110. The modulation of the display device 110 can be transmitted to the pupil position of the human eye; on the other hand, the main control module 140 controls the stacked light sources 401 in the multi-layer light guide lighting device 300 through the light source controller 120 to emit light in sequence, illuminate the corresponding light guide plate 402 to form a point light source, and at the same time control the micro display device 110 to load the corresponding image, to ensure that the light emitted from the multi-layer light guide lighting device 300 can be modulated by the micro display device 110 so that the human eye can obtain an image on the corresponding focusing plane, and the image is combined with the real scene, thereby realizing multiple depths of augmented reality three-dimensional image display.

[0082] In the stacked array of light guide lighting devices 301 in the multi-layer light guide lighting device 300, the dot arrays on the surface of the light guide plates 402 at different positions in the array are arranged differently. The arrangement position and number of the dot arrays on each light guide plate 402 can be adjusted according to actual needs, so as to realize the projection of images of different regional positions or different regional ranges on the micro display device 110 onto focal planes of different depths. When the size and position of the pupil of the human eye are constant, the distance between the light guide plate 402 and the micro display device 110, the distance between the micro display device 110 and the human eye, and the spacing between the midpoints of the dot array on the light guide plate 402 determine the depth position of the focal plane where the corresponding image is located. When the position of the human eye changes, the eye tracking device 130 tracks the position of the human eye and transmits the signal to the main control module 140, and the main control module 140 calculates the image corresponding to the dot array on each stacked light guide plate 402 according to the size and position signal of the pupil of the human eye. Then the main control module 140 controls the light source controller 120 to light up the stacked light guide lighting devices 401 in sequence, and at the same time controls the micro display device 110 to load the corresponding image, and sequentially and time-dividedly incident light near the pupil of the human eye. The light guide plates 402 at different depths from the human eye correspond to focal planes at different depths after imaging. The images at the focal planes at different depths are combined with the real scene, thereby realizing three-dimensional near-eye display of augmented reality.

[0083] In this embodiment, each light source in the multi-layer light guide lighting device 300 and the image sequence to be loaded on the micro display device 110 are lit and loaded in time sequence. That is, in each cycle, it is ensured that the light source corresponding to each layer of light guide plate is lit and only lit once, and each corresponding time sequence image is loaded and only loaded once. At the same time, it should be noted that the dot array on each light guide plate in the multi-layer light guide lighting device 300 corresponds to the time sequence image one by one, and each dot in the dot array corresponds to a sub-image.

[0084] Assume that there are N=3 light guide lighting devices 301 at this time, and three two-dimensional images obtained by layering three-dimensional images, the first one corresponds to a distant object, the second one corresponds to an object in the middle, and the third one corresponds to a near object, and the distant object is located at the left position of the original three-dimensional image, the object in the middle is located at the middle position of the original three-dimensional image, and the near object is located at the right position of the original three-dimensional image. According to the corresponding relationship, as the light guide plate 402 is farther away from the micro display 110, the depth position of the corresponding time-sequential image is also farther. Due to the different distances, the farther the light guide plate 402 is from the micro display 110, the larger the point spacing in the point array. At the same time, due to the different positions of the objects in the time-sequential image, the point arrays on each light guide plate are arranged differently. In order to avoid image aliasing observed by the human eye, the point arrays can be selected to be spaced or staggered. According to the arrangement of the point arrays, the main control module 140 processes the three two-dimensional images to obtain three time-sequential images, and each time-sequential image has U tiled sub-images. Taking the occlusion relationship into consideration, the left side of the first image is the V sub-images obtained by decomposing the distant object, and the rest are pure white sub-images of UV. The middle position of the second image is the W sub-image obtained by decomposing the middle object, and the two sides are UW pure white sub-images. The right side of the third image is the UVW sub-image obtained by decomposing the nearby object, and the rest are V+W pure white sub-images. Since the micro-display device 110 is a transmissive display device, the superposition of multiple pure white images and non-pure white images is a non-pure white image. Then, under the control of the main control module 140, the light source of each layer of the light guide plate in the multi-layer light guide lighting device 300 is sequentially lit and the sequential image is synchronously loaded on the micro-display device 110, so that the human eye can observe the three-dimensional image, and the image is integrated with the real scene, so that the human eye can see the augmented reality three-dimensional scene.

[0085] A schematic flow chart of a display method of a three-dimensional near-eye display system with enhanced display provided in embodiments 1 and 2 of the present invention is shown in FIG. Figure 7 As shown, the method includes:

[0086] Step 1: Obtain the position information (x, y, z) and pupil size of the human eye through an eye tracking device;

[0087] Step 2: The main control module calculates the target 3D image, decomposes the 3D image according to the number of light guide plates N and the focal plane depth corresponding to each layer of light guide plates, and calculates the 2D image of the corresponding depth of layer i (1≤i≤N). (N≥2)

[0088] Step 3: The main control module calculates the target image to be loaded on the micro-display device according to the human eye pupil position coordinates (x, y, z) and pupil size, wherein the size, position and arrangement of the sub-image of the target image are determined by the dot array position and arrangement on the surface of the light guide plate, and the image sequence displayed by the micro-display device corresponding to each layer of the light guide plate is obtained [H 1 ,…,H N ];

[0089] Step 4: The main control module sequentially lights up the light source required to illuminate each layer of the light guide plate through the light source controller [m 1 ,…,m N ], and control the micro display device to load the corresponding image of each layer of light guide plate through the main control module [H 1 ,…,H N The light sources lit in sequence need to correspond to the image loaded by the micro display device. Each time a light source m in the multi-layer light guide lighting device is lit i , load the corresponding image H i , ensuring that the switching frequency of the light source matches the refresh frequency of the image sequence. The light source controller controls the switching of the light source and the micro display device loads the corresponding image synchronously under the control of the main control module;

[0090] Step 5: Cycle the light source and load the corresponding image on the micro-display device. The light emitted from the multi-layer light guide lighting device is modulated by the micro-display device and the image [H 1 ,…,H N ] are imaged near the pupil of the human eye in sequence, and the human eye can always observe the corresponding image on the focusing plane at different depths. At the same time, combined with the real scene, the human eye can see the augmented reality-style three-dimensional scene;

[0091] Step 6: When the size and position of the pupil of the human eye changes, jump to the first step and continue the above steps.

[0092] In the N-layer light guide plate described in the above display method, N depends on the screen refresh rate of the micro display device and the refresh rate of the three-dimensional image expected to be observed by the human eye, and generally N is 2 to 6. For example, if the screen refresh rate of the micro display device is 120 Hz and the refresh rate of the three-dimensional image expected to be observed by the human eye is 30 Hz, then the number of light guide plate layers N=4, and the depth of the three-dimensional image that can be observed is also 4 layers.

[0093] The above display method describes cyclically lighting up the light source and loading the corresponding image on the microdisplay device, wherein the lighting order can be from near to far, or other lighting orders. As long as each light source is lit and only lit once in a cycle, and each corresponding image required to be loaded on the microdisplay device is loaded and only loaded once, the human eye can always see the three-dimensional image.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An augmented reality near-eye display device based on a multi-layer light guide plate, comprising an eye tracking device for obtaining position information of a human pupil, characterized in that: Also includes: A light guide lighting device, comprising at least two layers of light guide plates and light sources corresponding to each layer of light guide plates, wherein the surface of the light guide plates is provided with point units arranged in an array, and when the light emitted by the light source is transmitted to the light guide plates, it is scattered by each point unit to form an array of point light sources at different depths and positions from the human eye; A light source controller, connected to the light source, and used to control the on and off of the light source; The micro-display device is located between the light guide lighting device and the pupil of the human eye, loaded with a point light source array corresponding to the light guide plate and a time-series image corresponding to the pupil position, and is used to receive the light emitted by the point light source array and send it to the pupil position of the human eye after modulation; The main control module is connected to the eye tracking device, the micro display device, and the light source controller, respectively, and is used to obtain the human eye pupil position information, layer the target three-dimensional image, calculate the target three-dimensional image to be loaded on the micro display device, and synchronously control the switching of the light source and the loading and refreshing of the image required by the micro display device by controlling the light source controller, so as to ensure that the light source corresponding to each layer of the light guide plate emits light in a certain sequence and only lights up once, so that each corresponding sequential image is loaded by the micro display device and only loaded once.

2. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 1, characterized in that: The dot unit is a concave dot or a convex dot.

3. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 2, characterized in that: The concave or convex points are circular, triangular or square.

4. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 1, characterized in that: The dot units of each light guide plate form an equilateral triangle staggered dot unit array and a triangle alternately distributed dot unit array.

5. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 1, characterized in that: The dot units of each light guide plate form a dot unit array with a square staggered distribution, or a dot unit array with a square interval distribution.

6. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 5, characterized in that: The dot unit arrays may be staggered, spaced, partially overlapped, or completely overlapped.

7. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 1, characterized in that: The light source is an LED lamp, which is made by combining "blue light technology" with fluorescent powder, or by mixing multiple monochromatic lights.

8. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 1, characterized in that: The micro display device is a transmissive LCD display screen or a transmissive OLED display screen.

9. The augmented reality near-eye display device based on a multi-layer light guide plate according to claim 1, characterized in that: The main control module is a computer, a single chip microcomputer, an embedded system or a PLC.

10. An augmented reality near-eye display method based on a multi-layer light guide plate, characterized in that: The steps include: S1. Obtain the position information (x, y, z) of the pupil of the human eye through an eye tracking device; S2. According to the number of light guide plates n and the focal plane depth corresponding to each layer of light guide plates, the target three-dimensional image is layered and decomposed to obtain n layers of two-dimensional images with depth information; S3. Calculate the target image to be loaded into the micro display device according to the coordinates (x, y, z) of the pupil position of the human eye; S4. The main control module controls the light source controller to light up and turn off the light source in sequence, and synchronously controls the micro display device to load the corresponding image; S5. Cycle the light source on and off and load the corresponding image on the micro-display device. Light is emitted from the multi-layer light-guiding lighting device and reaches the vicinity of the pupil of the human eye after being modulated by the micro-display device. The image is integrated with the real scene, so that the human eye can see the three-dimensional scene of augmented reality.

11. The method for augmented reality near-eye display based on a multi-layer light guide plate according to claim 10, characterized in that: The step S2. according to the number of layers n of light guide plates and the depth of the focal plane corresponding to each layer of light guide plates, the target three-dimensional image is decomposed in layers to obtain n layers of two-dimensional images with depth information, specifically comprising: S2.1 scaling and adjusting the input three-dimensional image according to the focal planes corresponding to the light guide plates of each layer; S2.2 determining that the three-dimensional image is to be divided into n layers according to the number of light guide plates n, determining the interval of the depth of each layer of the two-dimensional image according to the spacing between the light guide plates of each layer, and determining the plane depth corresponding to the n layers of the two-dimensional image obtained by decomposition according to the depth of the focusing plane corresponding to each layer of the light guide plates; S2.3 projects different objects onto a plane close to the depth of the object according to the depth of each object in the three-dimensional image, thereby obtaining n layers of two-dimensional images with depth information.

12. The method for augmented reality near-eye display based on a multi-layer light guide plate according to claim 10, characterized in that: The step S3. calculating the target image to be loaded on the micro display device according to the human eye pupil position coordinates (x, y, z) specifically includes: S3.1 moves the input two-dimensional image according to the coordinates of the pupil position of the human eye to ensure that any sub-image is aligned with the corresponding point and the pupil of the human eye; S3.2 Determine the size of the sub-image according to the pupil size of the human eye and the dot spacing on the light guide plate; S3.3 determining the position and arrangement of the sub-image in the target image according to the position and arrangement of the dot array on the light guide plate; S3.4 processes the input two-dimensional image, fills the area without objects in the image with white, and then decomposes the image into tiled sub-images that meet the conditions, and flips each sub-image vertically and horizontally at the same time to obtain the target image required to be loaded on the micro display.

Images