Image processing method and image processing device based on lens array

Abstract

The present invention discloses an image processing method based on a lens array, comprising the following steps: Step 1, obtaining an image as the original image of a display screen; determining the position of each pixel on the display screen; obtaining the current pupil size and the position of its center point; Step 2, determining the position of the imaging point of any pixel on the display screen on the virtual image plane based on the current pupil position and size; Step 3, calculating the weight of each imaging point on the virtual image plane for any pixel on the display screen; Step 4, calculating the display value of the pixel based on the weight of each imaging point on the virtual image plane; Step 5, displaying the calculated display value of the pixel on the display screen, and the image displayed on the display screen is a rendered image. The present invention can effectively eliminate image noise. The present invention also discloses an image processing device based on a lens array.

Description

Technical Field

[0001] The present invention relates to a VR / AR image processing method, and more particularly to an image processing method based on a lens array. The present invention also relates to an image processing method device based on a lens array. Background Art

[0002] VR (Virtual Reality) technology is a computer simulation system that allows users to create and experience virtual worlds. It uses computers to generate a simulated environment, integrating multi-source information, interactively simulating three-dimensional dynamic visuals and physical behaviors, allowing users to immerse themselves in this simulated environment. Currently, VR devices, specifically VR glasses, are too thick and heavy to wear for extended periods, a significant drawback. Furthermore, existing VR glasses suffer from severe image distortion, causing discomfort and significantly impacting the user experience and mood. This is because existing VR glasses utilize virtual reality display technology that uses binocular parallax to create a 3D depth of field effect. This results in unrealistic imaging and fails to achieve the effect of seeing distant objects clearly while near objects are blurred, or vice versa. This does not conform to the natural way the human eye perceives images. Therefore, wearing these VR glasses for extended periods not only lacks immersion but can also cause confusion in the brain, leading to feelings of fatigue and dizziness. Some people even experience symptoms such as nausea and vomiting.

[0003] AR (Augmented Reality) technology is a new technology that seamlessly integrates real-world and virtual-world information. It uses computers and other scientific technologies to simulate and superimpose physical information (visual information, sound, taste, touch, etc.) that is difficult to experience within a certain time and space in the real world. This technology applies virtual information to the real world and is perceived by human senses, thereby achieving a sensory experience beyond reality. In other words, the real environment and virtual objects are superimposed on the same screen or space in real time and coexist simultaneously. Currently, AR devices on the market that experience AR technology, namely AR glasses, generally use optical waveguide technology. This technology has many advantages, such as being lightweight and highly transparent, but its disadvantages are also obvious. Due to the low optical utilization, high-order light transmission losses are severe. Geometric waveguide optical components are expensive, complex to manufacture, and have low yields. During the imaging process, diffraction waveguides generate dispersion, leading to rainbow and haze effects that affect image quality. The eye relief (the distance between the eye and the optical module) is relatively large, typically exceeding 25mm. The field of view (FOV) of the waveguide is too small, limited by the angle of total internal reflection of the transmitted light, typically only around 50 degrees or less. Moreover, waveguide components are extremely wavelength-sensitive, requiring monochromatic light sources for imaging. Full-color imaging is technically difficult and expensive to install. Nreal's product features a removable shield in front of the field of view to separate the virtual and real images, but this is not a true AR device. AR devices require a wide FOV to provide an immersive user experience. However, current waveguide technology is completely inadequate.

[0004] Since 2010, near-eye light field display technology has emerged, which uses Fresnel lens arrays to achieve 3D display. Figure 1The figure shows a VR / AR lens using a Fresnel lens array, comprising an LCD display panel 1, on which are sequentially stacked a quarter-wavelength wave plate (QWP) 2, a beam splitter (BS) 3, and a polarization beam splitter (PBS) 4. The surface of the polarization beam splitter 4 is fixedly mounted with n Fresnel lenses 5 arranged in a rectangular array. The total area of ​​the n Fresnel lenses matches the area of ​​the polarization beam splitter 4. The operating principle of this lens is as follows: light emitted from a single pixel of the LCD display panel 1 is linearly polarized. After passing through the wave plate (QWP) 2 and the beam splitter (BS) 3, the polarization state of this light returns to its initial linear polarization. After reaching the polarization beam splitter (PBS) 4, the light is first reflected by the polarization beam splitter (PBS) 4 back to the beam splitter (BS) 3. After being reflected by the beam splitter (BS) 3, it is incident on the polarization beam splitter (PBS) 4 again. Finally, it passes through the polarization beam splitter (PBS) 4 and propagates toward the small lens array. Since the light from the display panel 1 is repeatedly transmitted three times between the beam splitter (BS) 3 and the polarization beam splitter (PBS) 4, an effective optical path three times longer than the physical system distance between the display panel 1 and the lens array can be obtained optically, that is, the physically required system distance can be reduced to 1 / 3 of the required optical system distance. This technology is based on polarized optical folding (Pancake lens), which allows light to reflect back and forth between interfaces, making the effective optical path longer than the physical distance. This is very beneficial for reducing the thickness of VR / AR lenses, which can further reduce the physical system distance to 1 / 3 of the required optical system distance.

[0005] This technology can present a more realistic 3D experience than the Oculus Rift (head-mounted display), projecting virtual images directly onto the human retina in the form of a light field. In addition to being able to see the virtual image, the user can also perceive its position. This experience is like being built on the real world and does not cause any discomfort. This solves some of the technical difficulties currently faced by virtual reality imaging. This technology allows users to achieve close to realistic stereoscopic vision, allowing the eyes to more comfortably experience high-definition, wide-field-of-view virtual reality presentation effects, avoiding dizziness, vomiting and other discomforts during the wearing process, and bringing a better experience to virtual reality users.

[0006] However, existing Fresnel lens array light field display technology still has many drawbacks, such as a small eyebox (the conical area between the near-eye display optical module and the eyeball, where the display content is clearest) and a large exit pupil distance. From a display perspective, since the image generated by each Fresnel lens has a clear boundary, when the human pupil exceeds the eyebox range, image noise caused by image crosstalk is clearly visible, which reduces image quality and causes the use of this technology to fall short of market expectations. Furthermore, due to the large amount of data transmitted and the long rendering time during the application process, the frame rate does not meet the requirements, resulting in video freezes. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide an image processing method based on a lens array, which can eliminate image noise caused by image crosstalk in the displayed image during a 3D display process.

[0008] To solve the above technical problems, the technical solution of the image processing method based on the lens array of the present invention is as follows:

[0009] Step 1: Obtain an image as the original image of the display screen; determine the position of each pixel on the display screen; obtain the size of the current human pupil and the position of its center point;

[0010] Step 2: Determine the position of the imaging point of any pixel point A on the display screen on the virtual image plane based on the current pupil position and size;

[0011] In another embodiment, the step 2 includes:

[0012] Step 2.1, based on the current position and size of the pupil, determine the lens in the lens array between the display screen and the pupil that actually participates in transmitting light;

[0013] Step 2.2: Determine the position of the imaging point of the pixel point A on the virtual image plane based on the line connecting the center of the lens actually participating in the transmission of the light and the pixel point A.

[0014] Step 3: Calculate the weights of each imaging point on the virtual image plane for any pixel point A on the display screen;

[0015] In another embodiment, the step three includes:

[0016] Step 3.1, based on the position of the lens actually involved in transmitting the light, calculate the area S of the intersection between the transmitted light column of the lens corresponding to each imaging point and the pupil;

[0017] Step 3.2: Calculate the weight of each imaging point using the following formula based on the intersection area S of the lens's transmitted light column and the pupil corresponding to each imaging point.

[0018]

[0019] Among them, W n It refers to the weight of any imaging point n on the virtual image plane;

[0020] S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil;

[0021] S n It refers to the area where the light column emitted by the current imaging point n intersects with the pupil.

[0022] Step 4: Calculate the display value a of pixel point A according to the weight of each imaging point on the virtual image plane;

[0023] In another embodiment, the step 4 uses the following formula to calculate the display value a of pixel point A:

[0024]

[0025] Where a refers to the display value of pixel A;

[0026] Z refers to the number of imaging points of pixel A on the virtual image plane;

[0027] v i It refers to the original pixel value of any imaging point of pixel A on the virtual image plane;

[0028] W i It refers to the weight of the imaging point on the virtual image plane;

[0029] S i It refers to the area where the light beam emitted from the imaging point intersects with the pupil.

[0030] Step 5: Display the calculated display value a of the pixel point A on the display screen, and the image displayed on the display screen is the rendered image.

[0031] In another embodiment, the area S where the transmitted light column of the lens corresponding to each imaging point intersects with the pupil is the intersection area of ​​the light emitted by the pixel point A of the display screen after passing through the lens array and the pupil and its expanded part.

[0032] In another embodiment, the value of the area S where the transmitted light column of the lens corresponding to each imaging point intersects with the pupil is the area of ​​the intersection area of ​​the line connecting the upper and lower edges of the lens corresponding to the pixel point A and the imaging point and the pupil and its expanded part.

[0033] In another embodiment, the area S is determined by the position of the pixel point A, the area of ​​the small lens in the lens array, the size of the pupil and the position of the center point thereof.

[0034] In another embodiment, after step 3.1, step 3.1.1 is performed: sort the areas S where each imaging point intersects with the pupil from large to small, and then select the first two or several data as S topi Calculate; then

[0035] In the fourth step, the value of pixel A is calculated and displayed as a according to the weight of each imaging point on the virtual image plane using the following formula;

[0036]

[0037] Where a refers to the display value of pixel A;

[0038] R refers to the number of imaging points with the highest weight value of pixel A on the virtual image plane;

[0039] v topi It refers to the original pixel value of the imaging point with the highest weight value of pixel A on the virtual image plane;

[0040] W topi It refers to the weight of the imaging point with the highest weight value on the virtual image plane;

[0041] S topi It refers to the area where the light column of the imaging point with the highest weight value on the virtual image plane intersects with the pupil.

[0042] The present invention also provides an image processing device based on a lens array, the technical solution of which includes:

[0043] The data acquisition module is configured to acquire an image as an original image of the display screen; determine the position of each pixel on the display screen; and obtain the size of the current human pupil and the position of its center point;

[0044] an imaging point positioning module configured to determine the position of an imaging point of any pixel point A on the display screen on the virtual image plane based on the current pupil position and size obtained by the data acquisition module;

[0045] An imaging point weight calculation module is configured to calculate the weights of each imaging point on the virtual image plane for any pixel point A on the display screen;

[0046] The pixel value calculation module is configured to calculate the value of pixel point A and display a according to the weight of each imaging point on the virtual image plane using the following formula;

[0047]

[0048] Where a refers to the display value of pixel A;

[0049] Z refers to the number of imaging points of pixel A on the virtual image plane;

[0050] v i It refers to the original pixel value of any imaging point of pixel A on the virtual image plane;

[0051] W i It refers to the weight of the imaging point on the virtual image plane;

[0052] S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil;

[0053] The display module is configured to display the calculated display value a of the pixel point A on a display screen, and the image displayed on the display screen is a rendered image.

[0054] In another embodiment, the imaging point positioning module includes:

[0055] a lens determination module configured to determine, based on the current pupil position and size obtained by the data acquisition module, a lens in the lens array between the display screen and the pupil that actually participates in transmitting light;

[0056] an imaging point position determination module configured to determine the position of the imaging point of the pixel point A on the virtual image plane based on the line between the center of the lens actually participating in the transmitted light and the pixel point A obtained by the lens determination module;

[0057] In another embodiment, the imaging point weight calculation module includes:

[0058] an area calculation module configured to calculate the area S of the intersection between the transmitted light column of the lens corresponding to each imaging point and the pupil based on the position of the lens actually participating in the transmitted light obtained by the lens determination module;

[0059] A weight calculation module is configured to calculate the weight of each imaging point using the following formula based on the area S of the intersection of the transmitted light column of the lens and the pupil corresponding to each imaging point obtained by the area calculation module;

[0060]

[0061] Among them, W n It refers to the weight of any imaging point n on the virtual image plane;

[0062] S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil;

[0063] S n It refers to the area where the light column emitted by the current imaging point n intersects with the pupil.

[0064] The technical effects that can be achieved by the present invention are:

[0065] The present invention can solve the problem of image noise caused by changes in the human pupil and the position of the human eye, thereby effectively eliminating image noise caused by the pupil exceeding the eyebox range due to pupil dilation or eye rotation during lens array AR / VR imaging, thereby improving imaging quality.

[0066] The present invention can reduce the usage of memory to the greatest extent and is suitable for on-board systems.

[0067] The present invention takes pupil size as a basis during image processing and can obtain different images according to different pupil sizes, thus having extremely strong adaptive capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Those skilled in the art will appreciate that the following description is merely illustrative of the principles of the present invention, which can be applied in a variety of ways to achieve many different alternative embodiments. These descriptions are intended only to illustrate the general principles of the teachings of the present invention and are not intended to limit the inventive concepts disclosed herein.

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the general description above and the detailed description of the drawings that follow, serve to explain the principles of the invention.

[0070] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0071] Figure 1 Schematic diagram of a VR / AR lens using a Fresnel lens array in the prior art;

[0072] Figure 2 Schematic diagram of the optical path of the lens array lens of the present invention; wherein, for the eye located at the top, the intersection area S6 of the light with the pupil and the dilated portion of the pupil is the third largest, the intersection area S7 of the light with the pupil and the dilated portion of the pupil is the largest, and the intersection area S8 of the light with the pupil and the dilated portion of the pupil is the second largest; for the eye located at the bottom, the intersection area S9 of the light with the pupil and the dilated portion of the pupil is the largest, and the intersection area S10 of the light with the pupil and the dilated portion of the pupil is the second largest;

[0073] Figure 3 1 is a flow chart of an image processing method based on a lens array according to the present invention;

[0074] Figure 4 This is an imaging photograph without the lens array lens of the present invention;

[0075] Figure 5 This is an imaging photograph using the lens array lens of the present invention;

[0076] Figure 6This is an imaging photograph without the lens array lens of the present invention;

[0077] Figure 7 This is an imaging photograph using the lens array lens of the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in this article do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0079] The image processing method based on the lens array of the present invention comprises the following steps:

[0080] Step 1: Obtain each frame of the video file to be processed as the original image of the display screen; traverse the display screen to determine the position of each pixel on the display screen; use human eye tracking technology to obtain the size of the current human eye pupil and the position of its center point;

[0081] Step 2: Determine the position of the image point of any pixel A on the display screen on the virtual image plane based on the position and size of the pupil (i.e., the position of the pupil center, the same below);

[0082] Step 2.1, based on the position and size of the pupil, determine the lens in the lens array between the display screen and the pupil that actually participates in transmitting light;

[0083] Step 2.2, based on the line connecting the center of the lens actually participating in the transmission of the light and the pixel point A, determine the position of the imaging point of the pixel point A on the virtual image plane;

[0084] Step 3: Calculate the weights of each imaging point on the virtual image plane for any pixel point A on the display screen;

[0085] Step 3.1, based on the position of the lens actually involved in transmitting the light, calculate the area S of the intersection between the transmitted light column of the lens corresponding to each imaging point and the pupil;

[0086] Then the area S is the intersection area between the light emitted from the pixel A of the display screen and the pupil and its expanded part after passing through the lens array;

[0087] The area S is the area of ​​the intersection of the line connecting the upper and lower edges of the lens corresponding to the pixel point A and the imaging point, and the pupil and its expanded part;

[0088] Pixel A is used as the light-emitting point. The line connecting pixel A and the upper and lower edges of the lens corresponding to the imaging point is the light column emitted by the imaging point. The intersection of this light column with the pupil and its dilated portion is the amount of light emitted by the imaging point that can enter the human eye. Therefore, the area of ​​this intersection can reflect the weight of the imaging point.

[0089] Obviously, the area S of the intersection region is determined by the position of the pixel point A, the area of ​​the small lens in the lens array, the size of the pupil and the position of its center point;

[0090] Step 3.2: Calculate the weight of each imaging point using formula (1) based on the intersection area S of the lens's transmitted light column and the pupil corresponding to each imaging point.

[0091]

[0092] Among them, W n It refers to the weight of any imaging point n on the virtual image plane;

[0093] S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil;

[0094] S n It refers to the area where the light column emitted by the current imaging point n intersects with the pupil;

[0095] Step three of the present invention can obtain the weights of all imaging points related to pixel point A.

[0096] Step 4: Using formula (2), calculate the display value a of pixel point A according to the weight of each imaging point on the virtual image plane;

[0097]

[0098] Where a refers to the display value of pixel A;

[0099] v i It refers to the original pixel value of any imaging point of pixel point A on the virtual image plane (that is, pixel point A corresponds to a pixel value of the original image);

[0100] W i It refers to the weight of the imaging point on the virtual image plane;

[0101] Z refers to the number of imaging points of pixel A on the virtual image plane;

[0102] S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil;

[0103] As a preferred embodiment, after step 3.1, step 3.1.1 can be performed to sort the areas S where each imaging point intersects with the pupil from large to small, and then select the first two or several data as S topi Calculate the rest and ignore them.

[0104] but

[0105]

[0106] Where a refers to the display value of pixel A;

[0107] R refers to the number of imaging points with the highest weight value of pixel A on the virtual image plane;

[0108] W topi It refers to the weight of the imaging point with the highest weight value on the virtual image plane;

[0109] S topi It refers to the area where the light column of the imaging point with the highest weight value on the virtual image plane intersects with the pupil;

[0110] v topi It refers to the original pixel value of the imaging point with the highest weight value of pixel A on the virtual image plane;

[0111] Step 5: Display the calculated display value a of pixel point A on the display screen. The image displayed on the display screen is a rendered image, and each pixel value of the rendered image is a value corrected according to the weight of each pixel point on the display screen.

[0112] As a specific embodiment, Figure 2 As shown, the display screen is divided into upper and lower parts based on the horizontal center axis of the display screen. When the human eye is located in the upper half of the display screen, the lenses 0-18 in the lens array between the display screen and the pupil that actually participate in transmitting light are lens 6, lens 7, and lens 8. The light transmitted by the other lenses 0-5 and 9-18 does not enter the human eye and is therefore negligible.

[0113] The position of imaging point V6 is determined by the intersection of the extended line connecting the center point of lens 6 and pixel point A with the virtual image plane. Similarly, the positions of imaging points V7 and V8 are determined by the lines connecting the center points of lenses 7 and 8 and pixel point A. The positions of the imaging points of pixel point A on the display screen on the virtual image plane are then pixel points V6, V7, and V8.

[0114] Since the lens array is fixed on the display screen, once the positions of the pupil and pixel A are determined, the lens that actually participates in transmitting light can be determined. Figure 2 As shown, when the human eye is located in the upper half of the display screen, although the light emitted from point A will pass through all lenses 0-18 in the lens array, only the transmitted light beams of lenses 6, 7 and 8 can enter the pupil of the human eye. Therefore, the present invention only considers the lenses corresponding to the light that can enter the pupil (that is, the lenses that actually participate in transmitting the light), and does not consider the lenses corresponding to the light that does not enter the pupil.

[0115] The areas of intersections formed by the three pixels V6, V7, and V8 with the pupil after passing through the display screen and lenses 6, 7, and 8 are S6, S7, and S8 respectively;

[0116] The weight W6 of pixel V6 is equal to S6 divided by the sum of S6, S7 and S8, that is, Z = 3 at this time; the weight W7 of pixel V7 and the weight W8 of pixel V8 are calculated in the same way; thus, the weights of all imaging points of any pixel A on the virtual image plane can be calculated;

[0117] Obviously, the value of the area S where each pixel intersects with the pupil is positively correlated with the area of ​​the small lens in the lens array and the size of the pupil. The larger the area of ​​the small lens, the larger the value of area S. When the user's pupil gradually enlarges or the user moves their eyes, causing the pupil to exceed the eyebox, the value of area S will change accordingly. The weight of each pixel on the virtual image plane will also change, and thus each display value a on the display screen will also change, and the rendered image will also change accordingly.

[0118] Similarly, when the human eye moves from the upper half of the display screen to the lower half of the display screen, the position of the imaging point of pixel point A on the display screen on the virtual image plane will change. At this time, the lenses actually participating in the transmission of light are lens 9 and lens 10, and the positions of the imaging points of pixel point A on the display screen on the virtual image plane are two pixel points V9 and V10.

[0119] When the position of the human eye changes, the position of the imaging point changes accordingly. Therefore, the present invention can render different images according to the different positions of the human eye, thereby solving the problem of image noise caused by pupil enlargement or when the position of the human eye changes.

[0120] In step 4 of the present invention, the sum of the product of the weight obtained in step 3 and the pixel value of the corresponding original imaging point is used as the display value a of the pixel point A; that is, according to formula 2,

[0121] a=v6×W6+v7×W7+v8×W8.

[0122] The present invention can obtain the weight of a certain pixel point through step 3.2, that is, calculate the ratio of the area of ​​the intersection area formed by the intersection of the pixel point and the pupil to the sum of the areas of the intersection areas formed by the intersection of all pixels and the pupil, and use the weight to correct the displayed display value a.

[0123] The present invention loads the weights of all components related to pixel point A on the basis of the pixel values ​​of each imaging point on the virtual image plane. Compared with the original image, the new displayed image obtained eliminates the visual impact of image noise; compared with traditional large lens imaging, the image distortion caused by each small lens in lens array imaging is almost negligible, so this method can significantly improve imaging quality.

[0124] This embodiment uses a lens array with 19 rows as an example. The actual lens array may have far more than 19 rows, which means that more than three rows of lenses actually participate in transmitting light, and therefore more than three imaging points on the virtual image plane corresponding to pixel A. By sorting the area S of the intersection of each imaging point with the pupil from largest to smallest in step 3.1.1, the present invention can ignore pixels with small weights that do not contribute to the value of pixel A but seriously affect computational efficiency, thereby saving computation time and memory.

[0125] like Figure 4 The following images show images of a lens array lens without the present invention. As can be seen from the images, significant noise appears in the image when the pupil diameter increases from 4.4 mm to 6.2 mm. Significant noise also appears when the distance from the pupil to the lens decreases from 15 mm to 10 mm. Clearly, without the present invention, noise would appear as the pupil increases.

[0126] like Figure 5 The images shown here are of a lens array lens using the present invention. As can be seen from the images, when the pupil diameter increases from 4.4 mm to 6.2 mm, no noise is present in the image; and when the distance from the pupil to the lens decreases from 15 mm to 10 mm, no noise is present in the image either. Clearly, with the present invention, noise is eliminated even when the pupil is enlarged.

[0127] Because the display value a of pixel A obtained in the present invention includes the area factor of the small lens in the lens array, that is, the display value a of pixel A can change according to the area of ​​the small lens in the lens array, the present invention can avoid the occurrence of image noise when the user's pupil becomes larger or the pupil exceeds the eyebox due to eye rotation, thereby significantly improving imaging quality.

[0128] Since the display value a of pixel point A of the present invention takes into account the size of the pupil and the position of its center point, that is, the display value a of pixel point A can change according to the size and position of the pupil, the present invention can accommodate sufficiently large pupil changes and large rotation angles. Even if the pupil diameter and position change during use, a noise-free image can still be obtained.

[0129] like Figure 6 The image shown is a photograph of an image obtained without the lens array lens of the present invention. As can be seen from the figure, when the eyeball center coordinates are (0.8mm, 0), slight noise appears in the image; when the eyeball center coordinates increase to (1.2mm, 0), the noise becomes more noticeable; and when the eyeball center coordinates increase to (1.7mm, 0), significant noise appears in the image. Clearly, without the present invention, noise would appear as the eye moves.

[0130] like Figure 7 Shown are images of a lens array lens using the present invention. As can be seen from the figure, when the eyeball center coordinates are (0.8mm, 0), (1.2mm, 0), and (1.7mm, 0), no noise is present in the image. Clearly, with the present invention, when the human eye rotates, noise is not present or is not noticeable. The present invention can meet the maximum human eye rotation speed without perceiving image noise.

[0131] The present invention is applicable to VR / AR optical devices using a small lens array (such as a Fresnel small lens array) and / or a focusing lens.

[0132] Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if such modifications of the present invention fall within the scope of the claims and their equivalents, the present invention is intended to include such modifications and variations.

Claims

1. An image processing method based on a lens array, characterized in that , including the following steps: Step 1: Obtain an image as the original image of the display screen; determine the position of each pixel on the display screen; obtain the size of the current human pupil and the position of its center point; Step 2: Determine the position of the imaging point of any pixel point A on the display screen on the virtual image plane based on the current pupil position and size; Step 3: Calculate the weights of each imaging point on the virtual image plane for any pixel point A on the display screen; The weight of the imaging point is calculated using the following formula: Among them, W n It refers to the weight of any imaging point n on the virtual image plane; S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil; S n It refers to the area where the light column emitted by the current imaging point n intersects with the pupil; Step 4: Calculate the display value a of pixel point A according to the weight of each imaging point on the virtual image plane; The display value a of the pixel point A is calculated using the following formula: Where a refers to the display value of pixel A; Z refers to the number of imaging points of pixel A on the virtual image plane; v i It refers to the original pixel value of any imaging point of pixel A on the virtual image plane; W i It refers to the weight of the imaging point on the virtual image plane; S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil; Step 5: Display the calculated display value a of the pixel point A on the display screen.

2. The image processing method based on lens array according to claim 1, characterized in that , the step 2 includes: Step 2.1, based on the current position and size of the pupil, determine the lens in the lens array between the display screen and the pupil that actually participates in transmitting light; Step 2.2: Determine the position of the imaging point of the pixel point A on the virtual image plane based on the line connecting the center of the lens actually participating in the transmission of the light and the pixel point A.

3. The image processing method based on lens array according to claim 2, characterized in that , the step three includes: Step 3.1, based on the position of the lens actually involved in transmitting the light, calculate the area S where the light column of the lens corresponding to each imaging point intersects with the pupil; Step 3.2: Calculate the weight of each imaging point based on the intersection area S of the lens's light column and the pupil corresponding to each imaging point.

4. The image processing method based on lens array according to claim 3, characterized in that The area S where the light column of the lens corresponding to each imaging point intersects with the pupil is the intersection area of ​​the light emitted by the pixel point A of the display screen after passing through the lens array and the pupil and its expanded part.

5. The image processing method based on lens array according to claim 3, characterized in that The value of the area S where the light column of the lens corresponding to each imaging point intersects with the pupil is the area of ​​the intersection of the line connecting the upper and lower edges of the lens corresponding to the pixel point A and the imaging point and the pupil and its expanded part.

6. The image processing method based on lens array according to claim 3, characterized in that ,The area S is determined by the position of the pixel point A, the area of ​​the small lens in the lens array, the size of the pupil and the position of its center.

7. The image processing method based on lens array according to claim 3, characterized in that After step 3.1, execute step 3.1.1: sort the area S where each imaging point intersects with the pupil from large to small, and then select the first M bits of data as S topi Calculate, where M = 2; then In the fourth step, the display value a of the pixel point A is calculated according to the weight of each imaging point on the virtual image plane using the following formula; Where a refers to the display value of pixel A; R refers to the number of imaging points with the highest weight value of pixel A on the virtual image plane; v topi It refers to the original pixel value of the imaging point with the highest weight value of pixel A on the virtual image plane; W topi It refers to the weight of the imaging point with the highest weight value on the virtual image plane; S topi It refers to the area where the light column of the imaging point with the highest weight value on the virtual image plane intersects with the pupil.

8. An image processing device based on a lens array, characterized in that ,include: The data acquisition module is configured to acquire an image as an original image of the display screen; determine the position of each pixel on the display screen; and obtain the size of the current human pupil and the position of its center point; an imaging point positioning module configured to determine the position of an imaging point of any pixel point A on the display screen on the virtual image plane based on the current pupil position and size obtained by the data acquisition module; The imaging point positioning module includes: a lens determination module configured to determine, based on the current pupil position and size obtained by the data acquisition module, a lens in the lens array between the display screen and the pupil that actually participates in transmitting light; an imaging point position determination module configured to determine the position of the imaging point of the pixel point A on the virtual image plane based on the line between the center of the lens actually participating in the transmitted light and the pixel point A obtained by the lens determination module; An imaging point weight calculation module is configured to calculate the weights of each imaging point on the virtual image plane for any pixel point A on the display screen; The imaging point weight calculation module includes: an area calculation module configured to calculate the area S of the intersection between the light column of the lens corresponding to each imaging point and the pupil based on the position of the lens actually participating in the transmission of the light obtained by the lens determination module; A weight calculation module is configured to calculate the weight of each imaging point using the following formula based on the area S of the intersection of the lens's light column and the pupil corresponding to each imaging point obtained by the area calculation module; Among them, W n It refers to the weight of any imaging point n on the virtual image plane; S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil; S n It refers to the area where the light column emitted by the current imaging point n intersects with the pupil The pixel value calculation module is configured to calculate the display value a of the pixel point A according to the weight of each imaging point on the virtual image plane using the following formula; Where a refers to the display value of pixel A; Z refers to the number of imaging points of pixel A on the virtual image plane; v i It refers to the original pixel value of any imaging point of pixel A on the virtual image plane; W i It refers to the weight of the imaging point on the virtual image plane; S i It refers to the area where the beam of light emitted from the imaging point intersects with the pupil; The display module is configured to display the calculated display value a of the pixel point A on a display screen, and the image displayed on the display screen is a rendered image.

Images