Human eye tracking and image fusion device and light field display processing method

Abstract

The invention discloses a human eye tracking and image fusion device and a light field display processing method.The device is used for a light field displayer and comprises a human eye tracking unit, an image fusion unit and a three-dimensional display unit, and the human eye tracking unit obtains binocular images and conducts reasoning to obtain human eye three-dimensional coordinate information and mode information; the image fusion unit obtains left and right views and performs image fusion processing and light field rendering processing in combination with human eye three-dimensional coordinate information to obtain a light field image, and the stereoscopic display unit regulates and controls the light field image to perform light field display on the light field display according to mode information. According to the technical scheme, a human eye tracking function is introduced, and a viewpoint light field rendering technology is combined, so that a 3D stereoscopic viewing angle can be greatly expanded, real and stable light field space reproduction is realized, and a user can obtain more comfortable and natural impression experience.

Description

Technical Field

[0001] The present application relates to the field of light field display technology, and in particular to a device for human eye tracking and image fusion and a light field display processing method. Background Art

[0002] At present, with the development of light field display technology, the traditional way of wearing 3D glasses to obtain stereoscopic imaging can no longer meet new needs, and it has been replaced by naked-eye 3D. The core of stereoscopic imaging lies in binocular parallax imaging, and binocular parallax imaging technology based on various principles is developing rapidly.

[0003] Traditional light field display technology, such as 3D glasses, mostly uses the difference in the design of left and right lenses. For example, the left and right lenses are designed with different polarization effects for visible light in different bands, so that the two eyes can obtain different color parts of a special picture, thereby realizing binocular parallax imaging.

[0004] Light field displays are now being promoted in the market. They aim to render the visual perception of three-dimensional scenes by reconstructing the geometric light rays emitted by the three-dimensional scenes in different directions, providing a natural and comfortable visual experience for the human visual system, and solving the convergence and accommodation conflict problem in traditional flat light field displays.

[0005] For example, to achieve light field display, light barrier technology can be used, using liquid crystal layer and polarization film to generate a series of stripes with a direction of 90 degrees. When light passes through, a stripe grid is formed. The stripe grid divides the line of sight of the two eyes, so that they can only see different pixel positions of the display screen behind the stripe grid. The left and right parallax images are arranged according to the positions of the pixels seen by the left and right eyes, thereby achieving a light field display effect. The resolution of this method is inversely proportional to the number of viewpoints, the viewpoint is fixed, and the viewing brightness is darker.

[0006] There is also a cylindrical lens technology, which adds a layer of micro-cylindrical lenses in front of the LCD screen. The image pixels under each cylindrical lens are divided into R, G, and B sub-pixels. Each sub-pixel is projected in different directions through the lens, so that the audience can see different views from different directions. However, this method has strict limitations on the viewing angle. Although there have been examples of extended viewing angles combined with human eye tracking in recent years, the dizziness caused by movement is relatively severe, and the crosstalk between different sub-pixels is serious.

[0007] The other is to use directional light source, also called directional backlight technology. The principle is to place two background light sources on the left and right sides, and then use optical film, LCD panel and corresponding driver to combine together. By controlling the backlight unit, different display images are projected into the left and right eyes of the person, and the corresponding 3D images can be observed by using the visual persistence of the human eye. However, this method has high frame rate requirements, easily causes eye fatigue, and has complex and immature design.

[0008] It can be seen that the prior art has the following shortcomings:

[0009] (1) Traditional 3D glasses can no longer provide three-dimensional viewing effects that meet people's current needs, and a light field display method with smaller viewing restrictions is needed.

[0010] (2) The viewing angle is narrow, and the movement of people in front of the display will have a negative impact on the final imaging effect.

[0011] (3) The viewing effect is affected. The brightness and frame rate of the final imaging result will be affected, and long-term viewing can easily cause dizziness.

[0012] (4) The circuit and optical structure designs are complex. Summary of the invention

[0013] The technical problem actually solved by this application is: how to achieve stereoscopic imaging of a light field display and ensure the stability of stereoscopic imaging. To solve the above problems, this application provides a human eye tracking and image fusion device and a light field display processing method.

[0014] According to a first aspect, an embodiment of the present application provides a device for human eye tracking and image fusion, which is used for use with a light field display, and the device includes a human eye tracking unit, an image fusion unit and a stereo display unit; the human eye tracking unit is used to obtain a binocular image and infer three-dimensional coordinate information and mode information of the human eye; the binocular image is an image of the environment in front of the light field display acquired by binocular photography; the image fusion unit is communicated with the human eye tracking unit to obtain left and right views, and performs image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image; the left and right views are images to be displayed from left and right viewing angles; the stereo display unit is communicated with the human eye tracking unit and the image fusion unit to control the light field image to perform light field display on the light field display according to the mode information.

[0015] Furthermore, the human eye data tracking unit includes a binocular camera module and an intelligent processing module; the binocular camera module is arranged to face the front of the light field display, and is used to collect the environmental image in front of the light field display through binocular camera, and generate the binocular image; the intelligent processing module obtains the binocular image from the binocular camera module, and is used to perform data splitting and data correction on the binocular image to generate a corrected image, and perform human eye feature inference on the corrected image, normalize and dimensionally convert the inference result, and obtain the three-dimensional coordinate information of the human eye; the intelligent processing module is also used to select a display mode from a preset mode configuration table, and generate the mode information based on the display mode.

[0016] Furthermore, the intelligent processing module performs data splitting and data correction on the binocular image to generate a corrected image, and performs human eye feature inference on the corrected image, normalizes and dimensionally transforms the inference results, and obtains the three-dimensional coordinate information of the human eye, specifically including: the intelligent processing module splits the binocular image into first imaging data and second imaging data at different viewpoints, performs distortion correction on the first imaging data and the second imaging data, and then converts them to the RGB color domain and scales them to a preset image size to obtain the corrected images corresponding to the first imaging data and the second imaging data respectively; the intelligent processing module inputs the corrected images corresponding to the first imaging data and the second imaging data respectively into a preset human eye feature inference model, and processes to obtain the inference results of three sets of coordinates; the human eye feature inference model is configured as a yoloV5-face model; the intelligent processing module numerically normalizes the inference results of the three sets of coordinates, generates corresponding grids using the normalized values, and performs numerical conversions of multiple dimensions respectively, obtains tensors of multiple dimensions by activation and splicing, and obtains the three-dimensional coordinate information of the human eye by mixed operations of tensors of multiple dimensions.

[0017] Furthermore, after obtaining the three-dimensional coordinate information of the human eye, the intelligent processing module also has a jitter judgment processing process: the intelligent processing module calculates the coordinate difference between the three-dimensional coordinate information of the human eye and the coordinate information obtained last time, and determines whether the coordinate difference is within a preset range. If so, the three-dimensional coordinate information of the human eye is retained; otherwise, the three-dimensional coordinate information of the human eye is abandoned and re-acquired.

[0018] Furthermore, the image fusion unit includes a preprocessing module, an image fusion module, a brightness compensation module, and an imaging rendering module; the preprocessing module is used to convert the data format of the left and right views to obtain the data to be fused corresponding to the left view and the right view respectively; the image fusion module is used to configure the calculation weights according to the three-dimensional coordinate information of the human eye, and perform weighted fusion calculation on the data to be fused based on the calculation weights to obtain a fused image from the human eye perspective; the brightness compensation module is used to compensate for the brightness of the fused image according to the relationship between the preset image grayscale and imaging brightness to obtain a compensated image; the imaging rendering module is used to sort the color channels of the compensated image to generate the light field image in the RBG color domain.

[0019] Furthermore, the image fusion module configures calculation weights according to the three-dimensional coordinate information of the human eye, performs data fusion calculations on the data to be fused based on the calculation weights, and obtains a fused image from the perspective of the human eye, specifically including: the image fusion module performs statistics on the projection distribution of the exit pupil according to the three-dimensional coordinate information of the human eye, obtains the distribution function of the exit pupil by fitting, and determines the calculation weights using the distribution function; the image fusion module performs data operations on the data to be fused corresponding to the left view and the data to be fused corresponding to the right view based on the calculation weights, and fuses them to obtain the fused image.

[0020] According to the second aspect, an embodiment of the present application provides a light field display processing method, including: acquiring a binocular image and inferring three-dimensional coordinate information and mode information of a human eye; the binocular image is an image of the environment in front of the light field display acquired by binocular photography; acquiring left and right views, and performing image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image; the left and right views are images to be displayed from left and right perspectives; and controlling the light field image to perform light field display on the light field display according to the mode information.

[0021] The method of acquiring a binocular image and inferring the three-dimensional coordinate information and mode information of a human eye specifically includes: obtaining the binocular image from a binocular camera module; the binocular camera module is arranged facing the front of the light field display, and is used to collect the environment image in front of the light field display through binocular camera, and generate the binocular image; performing data splitting and data correction on the binocular image to generate a corrected image, and performing human eye feature inference on the corrected image, normalizing and dimensionally converting the inference result, and obtaining the three-dimensional coordinate information of the human eye; selecting a display mode from a preset mode configuration table, and generating the mode information based on the display mode.

[0022] After obtaining the three-dimensional coordinate information of the human eye, there is also a jitter judgment processing process: calculating the coordinate difference between the three-dimensional coordinate information of the human eye and the coordinate information obtained last time, and judging whether the coordinate difference is within a preset range, if so, retaining the three-dimensional coordinate information of the human eye, otherwise abandoning the three-dimensional coordinate information of the human eye and re-acquiring it.

[0023] The obtaining of the left and right views, and performing image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain the light field image specifically includes: converting the left and right views into data formats to obtain the data to be fused corresponding to the left view and the right view respectively; configuring calculation weights according to the three-dimensional coordinate information of the human eye, and performing weighted fusion calculation on the data to be fused based on the calculation weights to obtain a fused image from the perspective of the human eye; compensating the brightness of the fused image according to the relationship between the preset image grayscale and imaging brightness to obtain a compensated image; and sorting the color channels of the compensated image to generate the light field image in the RBG color domain.

[0024] The beneficial effects of this application are:

[0025] According to the above-mentioned embodiment, a device for eye tracking and image fusion and a light field display processing method are provided. The device is used for light field display and includes an eye tracking unit, an image fusion unit and a stereo display unit. The eye tracking unit obtains binocular images and infers the three-dimensional coordinate information and mode information of the eye. The image fusion unit obtains the left and right views and performs image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the eye to obtain the light field image. The stereo display unit controls the light field image according to the mode information to perform light field display on the light field display. The technical solution introduces the eye tracking function and combines the viewpoint light field rendering technology, which can greatly expand the viewing angle of 3D stereo, realize the real and stable light field space reproduction, and allow users to obtain a more comfortable and natural viewing experience; moreover, the technical solution can configure the viewing mode to correspond to different binocular parallax imaging situations in different viewing directions, significantly reduce the picture flickering problem caused by the movement of the eye, can reduce the fatigue of viewing, and expand the viewing angle. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a structural diagram of a device for eye tracking and image fusion for light field display in this application;

[0027] Figure 2 This is a schematic diagram of an interface of a human eye tracking unit in one embodiment of the present application;

[0028] Figure 3 This is a schematic diagram of an interface of an image fusion unit in one embodiment of the present application;

[0029] Figure 4 A schematic diagram of the processing logic of a human eye tracking unit and an image fusion unit in one embodiment of the present application;

[0030] Figure 5 A flowchart of a stereoscopic display processing method in this application;

[0031] Figure 6A flowchart of a human eye tracking process in one embodiment of the present application;

[0032] Figure 7 The figure is a flowchart of the image fusion processing process in one embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, features and advantages of the present invention more obvious and understandable, the embodiments of the present invention are clearly and completely described below in conjunction with the specific implementation methods involved in the accompanying drawings. In the description of the present invention, it should be understood that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without performing creative work, such as embodiments that only change the purpose without changing the basic principles involved in the claims, belong to the scope of protection of the present invention.

[0034] Please refer to Figure 1 , discloses a device for eye tracking and image fusion, which is used for light field display, and includes an eye tracking unit M1, an image fusion unit M2, and a stereoscopic display unit M3, which are described below respectively.

[0035] The human eye tracking unit M1 is used to obtain binocular images and infer the three-dimensional coordinate information and pattern information of the human eye. It can be understood that the binocular image is an environmental image in front of the light field display collected by binocular photography. The light field display here includes but is not limited to a display screen using light barrier technology, cylindrical lens technology, and directional backlight technology. For example, a binocular image can be generated by a binocular camera module fixed on the light field display and facing the viewing area in front of the display.

[0036] The image fusion unit M2 is connected to the human eye tracking unit M1 for obtaining the left and right views, and performs image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image. It can be understood that the left and right views are images to be displayed from the left and right perspectives, such as pre-stored pictures and videos.

[0037] The stereo display unit M3 is connected to the human eye tracking unit M1 and the image fusion unit M2 for controlling the light field image to be displayed on the light field display according to the mode information. It can be understood that the stereo display unit M3 has a display control function for the light field display, and realizes the display effect of binocular parallax imaging on the light field display, so that the user can see the 3D stereoscopic display content.

[0038] In a specific embodiment, see Figure 1, the human eye data tracking unit M1 includes a binocular camera module 101 and an intelligent processing module 102. The binocular camera module 101 is arranged facing the front of the light field display, and is used to collect the environmental image in front of the light field display through binocular photography, that is, to shoot the environmental area of ​​the user's head in front of the screen, and generate a binocular image. The intelligent processing module 102 obtains the binocular image from the binocular camera module 101, and is used to perform data splitting and data correction on the binocular image to generate a corrected image, and perform human eye feature inference on the corrected image, normalize and dimensionally transform the inference result, and obtain three-dimensional coordinate information of the human eye; moreover, the intelligent processing module 102 is also used to select a display mode from a preset mode configuration table, and generate the mode information based on the display mode.

[0039] Since the binocular camera module 101 can capture the user's head image through the binocular camera, one of the functions of the intelligent processing module 102 is to obtain the three-dimensional coordinate information of the human eye based on the captured user's head image, such as performing data splitting and data correction on the binocular image to generate a corrected image, and performing human eye feature inference on the corrected image, normalizing and dimensionally converting the inference result to obtain the three-dimensional coordinate information of the human eye, which specifically includes the following processing steps.

[0040] (1) The intelligent processing module 102 splits the binocular image into first imaging data and second imaging data at different viewpoints, performs distortion correction on the first imaging data and the second imaging data, and then converts them into the RGB color domain and scales them to a preset image size to obtain corrected images corresponding to the first imaging data and the second imaging data, respectively.

[0041] (2) The intelligent processing module 102 inputs the corrected images corresponding to the first imaging data and the second imaging data into a preset human eye feature inference model, and processes them to obtain inference results of three sets of coordinates; the human eye feature inference model here is configured as a yoloV5-face model.

[0042] (3) The intelligent processing module 102 numerically normalizes the inference results of the three sets of coordinates, generates corresponding grids using the normalized values, and performs numerical transformations of multiple dimensions respectively, obtains tensors of multiple dimensions through activation and splicing, and obtains the three-dimensional coordinate information of the human eye through mixed operations of tensors of multiple dimensions.

[0043] It should be noted that the human eye tracking unit M1 is used to obtain the real-time data of the human eye currently required. The function is mainly realized by the binocular camera module 101 and the high-performance intelligent processing module 102 for human eye reasoning calculation. The intelligent processing module 102 mentioned above can adopt Rockchip RK3588, and the binocular camera module can adopt a 480p@30fps binocular camera. Please refer to Figure 2Rockchip RK3588 has USB3.0 interface, USB2.0 interface, CPU, GPU, NPU. Its USB3.0 interface 1021 is connected to the binocular camera module, and the CPU processor 1022 calls out the binocular image received by the interface through the built-in Opencv library. The GPU (reference numeral 1024) of Rockchip RK3588 is used for image processing, and the NPU (reference numeral 1025) is called to perform initialization. The inference weight and the processed image are sent to the NPU for weight inference, and three preliminary output results are obtained. The results are the position and confidence of the corresponding detection target obtained by retrieving the full image with a box size of 16×20, 32×40, and 64×80. Finally, the result data is output through the USB2.0 interface 1026. It can be understood that here a modern intelligent microprocessor chip is connected to a binocular camera to acquire images and process them to obtain the coordinates of the human eye. The NPU of the chip is used as a human eye tracking calculation and reasoning platform, and the output three-dimensional coordinate information of the human eye is sent to the image fusion unit M2, and the output mode information is sent to the stereo display unit M3.

[0044] It should be noted that the modern intelligent microprocessor chip (Rockchip RK3588) overcomes the functional limitations of traditional microprocessors, adopts 8nm process design, is equipped with an eight-core CPU of quad-core A76 + quad-core A55 and an Arm high-performance GPU, and has a built-in NPU with 6T computing power. It has powerful visual processing capabilities and can support a variety of fast face unlocking solutions such as structured light and TOF; it supports a variety of display interfaces and up to 8K display processing capabilities; it has strong scalability and supports high-speed interfaces such as PCIE3.0, SATA3.0, dual TypeC / USB3.1, and can be used for AI computing power, image data processing, etc. It is used in ARMPC, high-end tablets, edge computing servers, virtual reality, 8K TVs and other directions. In terms of image processing, modern intelligent microprocessor chips have built-in multiple powerful embedded hardware engines, supporting 8K@60fps H.265 and VP9 decoders, 8K@30fps H.264 decoders and 4K@60fps AV1 decoders; supporting 8K@30fps H.264 and H.265 encoders, high-quality JPEG encoders / decoders, and specialized image pre-processors and post-processors. Rockchip RK3588 also introduces a new generation of fully hardware-based maximum 48-megapixel ISP (image signal processor), and implements many algorithm accelerators, such as HDR, 3A, LSC, 3DNR, 2DNR, sharpening, dehaze, fisheye correction, gamma correction, etc., which are widely used in graphics post-processing. In terms of AI computing power, modern intelligent microprocessor chips bypass the traditional GPU-CUDA acceleration framework and use self-built third-generation NPU processors, which can support INT4 / INT8 / INT16 / FP16 mixed operations. Its strong compatibility can easily convert network models based on a series of frameworks such as TensorFlow / MXNet / PyTorch / Caffe. It can be understood that the use of modern intelligent microprocessor chips as the execution components of data processing has the benefits of realizing localization of components, high computing speed, and easy binocular parallax imaging installation, which is light and centralized, and does not require multiple components to be cascaded.

[0045] The intelligent processing module 102 uses binocular images for eye tracking analysis, mainly using yoloV5-face as the framework for eye tracking reasoning calculation, modifying its reasoning weight, splitting the original output into three outputs, and converting it into weights suitable for modern intelligent microprocessor chip reasoning. Perform additional preprocessing on the input binocular image, convert it into an input suitable for modern intelligent microprocessor chip reasoning, call NPU to accelerate the reasoning speed, add a post-processing module, and grid-weight the three outputs respectively and pass the activation function to obtain the final three-dimensional coordinate information of the human eye.

[0046] Of course, after obtaining the three-dimensional coordinate information of the human eye, the intelligent processing module 102 also has a jitter judgment processing process: the intelligent processing module 102 calculates the coordinate difference between the three-dimensional coordinate information of the human eye and the coordinate information obtained last time, and determines whether the coordinate difference is within a preset range. If so, the three-dimensional coordinate information of the human eye is retained; otherwise, the three-dimensional coordinate information of the human eye is abandoned and re-acquired.

[0047] It can be understood that the intelligent processing module 102 is used here to de-jitter the obtained three-dimensional coordinate information of the human eye to reduce data fluctuations. At the same time, the binocular coordinates are used to obtain depth information, and the corresponding mode table is calculated using the optical architecture principle to calculate the three-dimensional coordinates and the corresponding display mode. In addition, the use of multi-threaded accelerated computing can greatly improve the detection rate.

[0048] In a specific embodiment, see Figure 1 The image fusion unit M2 includes a preprocessing module 201, an image fusion module 202, a brightness compensation module 203, and an imaging rendering module 204.

[0049] The pre-processing module 201 is used to convert the data formats of the left and right views to obtain the data to be fused corresponding to the left view and the right view respectively.

[0050] The image fusion module 202 is used to configure calculation weights according to the three-dimensional coordinate information of the human eye, and perform weighted fusion calculation on the data to be fused based on the calculation weights to obtain a fused image from the perspective of the human eye.

[0051] The brightness compensation module 203 is used to compensate the brightness of the fused image according to the relationship between the preset image grayscale and imaging brightness to obtain a compensated image.

[0052] The imaging rendering module 204 is used to perform sorting processing on each color channel of the compensated image to generate a light field image in the RBG color domain.

[0053] It should be noted that the image fusion module 202 is used here to configure calculation weights according to the three-dimensional coordinate information of the human eye, and performs data fusion calculations on the data to be fused based on the calculated weights to obtain a fused image from the human eye perspective. The processing steps specifically include the following: (1) The image fusion module 202 performs statistics on the screen distribution of the exit pupil according to the three-dimensional coordinate information of the human eye, obtains the distribution function of the exit pupil by fitting, and determines the calculation weights using the distribution function; (2) The image fusion module 202 performs data operations on the data to be fused corresponding to the left view and the data to be fused corresponding to the right view based on the calculated weights, and fuses them to obtain the fused image.

[0054] It should be noted that Rockchip RK3588 can be used as the data processing platform of the image fusion unit M2. Please refer to Figure 3, the HDMI interface 2021 of Rockchip RK3588 receives the external left and right view information, and the CPU (reference numeral 2022) receives the three-dimensional coordinate information of the human eye through the USB2.0 interface 2023. After calculating the image arrangement order and weight, it is sent to the GPU (reference numeral 2024) for image fusion operation, and the obtained image is subjected to brightness compensation and imaging rendering. It can be understood that here the modern intelligent microprocessor chip obtains the binocular view input through HDMI, reads the pre-calculated weights, uses its GPU as an image fusion platform, and uses the light field image obtained after brightness compensation and imaging rendering to perform binocular parallax imaging through the stereo display unit M3.

[0055] It should be noted that the brightness compensation module 203 uses the exit pupil distribution to calculate the grayscale distribution, performs function fitting to analyze its distribution form, uses the conditional least squares method to optimize the weight, calculates the relationship between the exit pupil brightness and the viewing distance, and combines the two. In addition, the relationship between the image grayscale and the imaging brightness is configured using the imaging obtained by projecting different grayscale values ​​of the light field display screen, performs function fitting to analyze the relationship, and performs brightness compensation based on the results.

[0056] In a specific embodiment, the stereo display unit M3 includes a power supply module 301, a lower processing module 302 and a light field display module 303. The power supply module 301 can provide DC power to the lower processing module 302 and the intelligent processing module 102 (such as Rockchip RK3588) through the Type-C interface. The lower processing module 302 can obtain the mode information generated by the intelligent processing module 102, and control the light field display to execute the corresponding light field imaging mode based on the mode information. For example, the lower processing module 302 switches the grating according to the received mode information, and realizes stereo imaging through the light field display screen and the stereo imaging program; the light field display module 303 can control the light field image based on the mode information to perform light field display on the light field display.

[0057] It should be noted that the use of Figure 1 When the device performs eye tracking, image fusion and stereoscopic display control, the specific eye detection has high accuracy and fast speed, the stereoscopic imaging effect is good, the image switching is smooth, the viewing dizziness is reduced, and the components have the application advantage of national localization.

[0058] for Figure 1The device for eye tracking and image fusion disclosed in the invention can be summarized as follows: the binocular camera module 101 captures binocular images of the user's eyes, the intelligent processing module 102 calculates the three-dimensional coordinate information of the eyes and the corresponding mode information based on the binocular images, and monitors its changes in real time, the image fusion module 202 calculates the weight and arrangement order based on the three-dimensional coordinate information of the eyes, fuses the left and right views, and outputs the fused image to the brightness compensation module 203 for brightness compensation, and obtains the light field image through the processing of the imaging rendering module 204, and then realizes binocular parallax imaging through the regulation of the stereo display module 303. It can be understood that modern intelligent microprocessor chips are used for eye tracking, and their inherent NPU architecture can adapt to the deep learning reasoning requirements of eye tracking. Combined with the algorithm and reasoning framework of yoloV5-face, the accuracy of eye detection is extremely high; multi-threaded processing is adopted, which greatly utilizes the computing power of the single-chip microcomputer, and the final tracking rate reaches 60 frames per second, which is three times that of ordinary tracking algorithms. In addition, using modern intelligent microprocessor chips GPU, the left and right views can be fused by weight. Six modes can be calculated based on the weights and image layout according to the three-dimensional coordinate information of the human eye. A total of 108 sub-modes can be calculated based on the weight gradient of adjacent periods, corresponding to different binocular parallax imaging methods for different viewing directions, which significantly reduces image flickering caused by human eye movement, reduces viewing fatigue, and expands the viewing angle.

[0059] for Figure 1 The device for eye tracking and image fusion disclosed in the disclosure mainly involves the processing flow of eye tracking and image fusion, please refer to Figure 4 , the processing flow may include seven steps S001-S007. It should be noted that the environment used here is Ubuntu development on Rockchip RK3588, using C++ language, and the corresponding library functions need to be installed before use.

[0060] Step S001, the program starts the serial port and the computing framework initialization, reads the pre-calculated data, and creates multiple threads.

[0061] Step S002, reading the binocular image captured by the binocular camera module from the serial port through opencv.

[0062] Step S003, performing recognition of the user's eyes and space coordinate calculation processing on the binocular image to obtain three-dimensional coordinate information of the eyes.

[0063] Step S004, de-jittering the currently obtained data (three-dimensional coordinate information of the human eye) in combination with the prior data to reduce human motion and measurement noise.

[0064] Step S005 , outputting mode information from the serial port to the lower processing module 302 through the serial port, and outputting the three-dimensional coordinate information of the human eye to the image fusion module 202 .

[0065] Step S006 : The lower processing module 302 receives the mode information and switches the display mode according to the data content.

[0066] Step S007 , the image fusion module 202 receives the three-dimensional coordinate information of the human eye, calculates the image fusion weight and arrangement distribution according to the three-dimensional coordinate information of the human eye, completes the floating point image fusion, and is conducive to realizing active stereoscopic imaging of the light field display.

[0067] Please refer to Figure 5 , discloses a light field display processing method, including steps S100-S300, which are described below respectively.

[0068] Step S100, acquiring a binocular image and inferring the three-dimensional coordinate information and pattern information of the human eye; the binocular image here is an image of the environment in front of the light field display collected by binocular photography.

[0069] Step S200, obtaining left and right views, and performing image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image; the left and right views here are images to be displayed under the left and right viewing angles.

[0070] Step S300: adjusting the light field image according to the mode information to perform light field display on the light field display.

[0071] In a specific embodiment, the Figure 1 The human eye tracking unit M1 in the image acquisition binocular and infer the three-dimensional coordinate information and mode information of the human eye, which specifically includes the following processing steps:

[0072] (1) Obtaining a binocular image from the binocular camera module 101; the binocular camera module 101 is arranged to face the front of the light field display, and is used to collect the environment image in front of the light field display through binocular camera, and generate a binocular image.

[0073] (2) Data splitting and data correction are performed on the binocular image to generate a corrected image, and human eye feature inference is performed on the corrected image, and the inference result is normalized and dimensionally converted to obtain three-dimensional coordinate information of the human eye. For example, the binocular image is split into first imaging data and second imaging data at different viewpoints, and the first imaging data and the second imaging data are subjected to distortion correction, and then converted to the RGB color domain and scaled to a preset image size to obtain the corrected images corresponding to the first imaging data and the second imaging data respectively; the corrected images corresponding to the first imaging data and the second imaging data are input into a preset human eye feature inference model, and the inference results of three sets of coordinates are obtained by processing; the inference results of the three sets of coordinates are numerically normalized, and the normalized numerical values ​​are used to generate corresponding grids and numerical conversions of multiple dimensions are performed respectively, and tensors of multiple dimensions are obtained by activation and splicing, and the three-dimensional coordinate information of the human eye is obtained by mixed operations of tensors of multiple dimensions.

[0074] (3) Select a display mode from a preset mode configuration table and generate mode information based on the display mode.

[0075] In a specific embodiment, Figure 1 After the intelligent processing module 102 obtains the three-dimensional coordinate information of the human eye, it also has a jitter judgment processing process: calculating the coordinate difference between the three-dimensional coordinate information of the human eye and the coordinate information obtained last time, and judging whether the coordinate difference is within a preset range, if so, retaining the three-dimensional coordinate information of the human eye, otherwise abandoning the three-dimensional coordinate information of the human eye and re-acquiring it.

[0076] In a specific embodiment, the Figure 1 The image fusion unit M2 in the image fusion unit obtains the left and right views, and performs image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image, which specifically includes the following processing steps:

[0077] (1) Convert the data formats of the left and right views to obtain the data to be fused corresponding to the left and right views respectively.

[0078] (2) Calculation weights are configured according to the three-dimensional coordinate information of the human eye, and weighted fusion calculation is performed on the data to be fused based on the calculation weights to obtain a fused image from the human eye's perspective. For example, statistics are performed on the projection distribution of the exit pupil based on the three-dimensional coordinate information of the human eye, and the distribution function of the exit pupil is obtained by fitting, and the calculation weights are determined using the distribution function; data operations are performed on the data to be fused corresponding to the left view and the data to be fused corresponding to the right view based on the calculation weights, and the fused image is obtained by fusion.

[0079] (3) According to the preset relationship between the grayscale of the image and the imaging brightness, the brightness of the fused image is compensated to obtain a compensated image.

[0080] (4) The color channels of the compensated image are sorted to generate a light field image in the RBG color domain.

[0081] Please refer to Figure 6 , the following will combine Figure 6 The process of obtaining binocular images and inferring the three-dimensional coordinate information and mode information of the human eye in step S100 is described in detail. The main process has thirteen steps from S101 to S113. It should be noted that the weights used by Rockchip RK3588 are different from the ordinary yoloV5-face weights. After obtaining the inference weights on other platforms, they need to be converted into onnx weights, and then onnx_utils is used to disassemble the weights, and the original tail spliced ​​output is disassembled into three unactivated output interfaces, and then read with rknn_toolkit, and use rknn.build to build the model, cancel quantization, and finally get the weights suitable for Rockchip RK3588 inference. In addition, it is necessary to ensure that the weight version matches the library of the inference framework, otherwise an error will be reported.

[0082] Step S101, create multiple threads, initialize each thread, use rknnlite.load to read the weight, set the serial port baud rate according to the operating system, and read the pre-calculated mode configuration table.

[0083] Step S102, using loadstream to read the data (binocular image) generated by the binocular camera module, and perform data splitting, such as generating first imaging data and second imaging data at different viewpoint positions.

[0084] Step S103, perform distortion correction on the captured image according to the predicted measurement parameters, convert the original RBG image into a correct RGB image, and scale the image to the size of the inference input, so as to obtain the corrected images corresponding to the first imaging data and the second imaging data respectively by converting to the RGB color domain and scaling to the preset image size.

[0085] Step S104, using rknn.inference to perform inference, placing the three output results obtained by inference into an array, that is, inputting the corrected images corresponding to the first imaging data and the second imaging data into a preset human eye feature inference model, and processing to obtain the inference results of the three sets of coordinates.

[0086] Step S105, the three output results are first normalized, and then the corresponding grids are generated according to the size of the output results, and the dimension conversion is performed for each output, and activation and splicing are performed according to the last dimension. The 16-dimensional tensor outputted at last can obtain the coordinates of the human eye with respect to the image position by summing the anchors_grid and the product of the grid and the step size. It can be understood as using the normalized values ​​to generate the corresponding grids and perform numerical conversions of multiple dimensions respectively, obtaining tensors of multiple dimensions by activation and splicing, and obtaining the three-dimensional coordinate information of the human eye by mixed operations of tensors of multiple dimensions.

[0087] Step S106, performing non-maximum suppression on the inference results, and eliminating human eye results with confidence levels below a certain range.

[0088] Step S107, determine whether there is a human eye, if not, return to S101; if yes, send the human eye data (human eye three-dimensional coordinate information).

[0089] Step S108, judging which is the viewer's eye and which is the interfering eye based on the detected eye position. If there are multiple eyes on the screen at the same time, the eyes at the edge of the screen are judged as interfering eyes, and the distance between the eyes is calculated to measure the distance from the eyes to the screen. Eyes that are too far or too close are also judged as interfering eyes.

[0090] Step S109, inversely calculate the camera coordinates of the human eye according to the inverse matrix of the projection matrix. The position of the human eye in the three-dimensional scene is first transformed into the coordinate system of the camera through the camera's external parameter matrix change W, and then the camera coordinate system is projected into the two-dimensional coordinates of the human eye in the image through the camera matrix projection change K. The matrix corresponding to the above transformation is inverted and then multiplied by the two-dimensional coordinates to obtain the three-dimensional coordinate information of the human eye.

[0091] Step S110, recording the currently obtained coordinates.

[0092] Step S111, judging whether the current movement is jitter and measurement noise according to the current coordinates and the previously recorded coordinates, if not, sending data (three-dimensional coordinate information of the human eye); if yes, returning to S101. This step can be regarded as a jitter judgment process after obtaining the three-dimensional coordinate information of the human eye, calculating the coordinate difference between the three-dimensional coordinate information of the human eye and the coordinate information obtained last time, judging whether the coordinate difference is within a preset range, if yes, retaining the three-dimensional coordinate information of the human eye, otherwise abandoning the three-dimensional coordinate information of the human eye and re-acquiring it.

[0093] Step S112, searching the data (mode information) to be sent according to the mode calculation table.

[0094] Step S113, use serial.write to output to the serial port.

[0095] Please refer to Figure 7 , the following will combine Figure 7 The process of obtaining the left and right views in step S200 and performing image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain the light field image is described below. The main process includes eight steps S201-S208.

[0096] Step S201, initializing the program, reading pre-calculated row vectors and column vectors, and setting initial values.

[0097] Step S202, re-reading the calculated data to reduce the running time.

[0098] Step S203, calculating the masks corresponding to the left and right views, specifically, circularly shifting six bits to the left and right in the initial state, and circularly expanding in units of 6*6.

[0099] Step S204, the calculation of weights requires statistics on the projection distribution of the exit pupil, fitting the distribution function of the exit pupil with a function, and then calculating the weights using convex optimization. The specific optimization steps are: determine the center points of the two distribution functions based on the physical position of the sub-pixel, then normalize the function, fix the upper and lower bounds of the function, constrain the function maximum to be at the target position, and use the iterative method to optimize the square difference between the function size that takes the maximum value and the original function peak. This step can be regarded as configuring the calculation weights according to the three-dimensional coordinate information of the human eye, statistics on the projection distribution of the exit pupil according to the three-dimensional coordinate information of the human eye, obtain the distribution function of the exit pupil by fitting, and use the distribution function to determine the calculation weights.

[0100] Step S205, decomposing and expanding the RGB of the original function to achieve dimensional expansion.

[0101] Step S206, decompose the weights, fuse the weights with the function distribution, and fuse them into a general table, which specifically has 6 modes, each mode has 18 sub-modes, and a total of 108 sub-modes. It can be understood that based on the calculated weights, the weighted fusion calculation is performed on the data to be fused to obtain a fused image from the human eye's perspective.

[0102] Step S207, using the gray value projection image to obtain the relationship between the image grayscale and the imaging brightness, performing function fitting to analyze the relationship, and calculating the additional weight according to the result to perform brightness compensation to obtain a compensated image.

[0103] Step S208, finally rearrange the obtained results and reassemble them into RGB channel images, so as to obtain a light field image. It can be understood as sorting the color channels of the compensated image to generate a light field image in the RBG color domain.

[0104] It should be noted that the light field display solution provided in the above embodiment mainly introduces human eye tracking technology and viewpoint light field rendering technology, which can greatly expand the viewing angle of 3D stereoscopic, realize the real and stable light field space reproduction, and let users get a more comfortable and natural viewing experience. In addition, the stereoscopic display solution can configure the viewing mode, corresponding to different binocular parallax imaging situations in different viewing directions, significantly reducing the picture flickering problem caused by human eye movement, reducing viewing fatigue, and expanding the viewing angle.

[0105] The above contents are further detailed descriptions of the present application in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the inventive concept of the present application.

Claims

1. A device for eye tracking and image fusion, for use with a light field display, characterized in that: The device comprises a human eye tracking unit, an image fusion unit and a stereoscopic display unit; The human eye tracking unit is used to acquire binocular images and infer the three-dimensional coordinate information and pattern information of the human eye; The binocular image is an image of the environment in front of the light field display collected by binocular photography; The image fusion unit is in communication connection with the human eye tracking unit, and is used to obtain left and right views, and perform image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image; the left and right views are images to be displayed under left and right viewing angles; The stereoscopic display unit is communicatively connected with the human eye tracking unit and the image fusion unit, and is used for regulating the light field image to perform light field display on the light field display according to the mode information.

2. The device according to claim 1, characterized in that The human eye data tracking unit includes a binocular camera module and an intelligent processing module; The binocular camera module is arranged facing the front of the light field display, and is used to collect the environment image in front of the light field display through binocular camera, and generate the binocular image; The intelligent processing module obtains the binocular image from the binocular camera module, is used to perform data splitting and data correction on the binocular image to generate a corrected image, and performs human eye feature inference on the corrected image, normalizes and dimensionally transforms the inference result, and obtains the three-dimensional coordinate information of the human eye; The intelligent processing module is also used to select a display mode from a preset mode configuration table and generate the mode information based on the display mode.

3. The device according to claim 2, characterized in that The intelligent processing module performs data splitting and data correction on the binocular image to generate a corrected image, and performs human eye feature inference on the corrected image, normalizes and dimensionally transforms the inference result, and obtains the three-dimensional coordinate information of the human eye, specifically including: The intelligent processing module splits the binocular image into first imaging data and second imaging data at different viewpoints, performs distortion correction on the first imaging data and the second imaging data, and then converts them into an RGB color domain and scales them to a preset image size to obtain corrected images corresponding to the first imaging data and the second imaging data, respectively; The intelligent processing module inputs the corrected images corresponding to the first imaging data and the second imaging data into a preset human eye feature inference model, and processes to obtain inference results of three sets of coordinates; the human eye feature inference model is configured as a yoloV5-face model; The intelligent processing module numerically normalizes the inference results of the three sets of coordinates, generates corresponding grids using the normalized values ​​and performs numerical conversions of multiple dimensions respectively, obtains tensors of multiple dimensions through activation and splicing, and obtains the three-dimensional coordinate information of the human eye through mixed operations of tensors of multiple dimensions.

4. The device according to claim 3, characterized in that After obtaining the three-dimensional coordinate information of the human eye, the intelligent processing module also has a jitter determination processing process: The intelligent processing module calculates the coordinate difference between the three-dimensional coordinate information of the human eye and the coordinate information obtained last time, and determines whether the coordinate difference is within a preset range. If so, the three-dimensional coordinate information of the human eye is retained; otherwise, the three-dimensional coordinate information of the human eye is discarded and re-acquired.

5. The device according to claim 1, characterized in that The image fusion unit includes a preprocessing module, an image fusion module, a brightness compensation module, and an imaging rendering module; The preprocessing module is used to convert the data formats of the left and right views to obtain the data to be fused corresponding to the left view and the right view respectively; The image fusion module is used to configure calculation weights according to the three-dimensional coordinate information of the human eye, and perform weighted fusion calculation on the data to be fused based on the calculation weights to obtain a fused image from the perspective of the human eye; The brightness compensation module is used to compensate the brightness of the fused image according to the relationship between the preset image grayscale and imaging brightness to obtain a compensated image; The imaging rendering module is used to perform sorting processing on each color channel of the compensated image to generate the light field image in the RGB color domain.

6. The device according to claim 5, wherein the image fusion module configures calculation weights according to the three-dimensional coordinate information of the human eye, performs data fusion calculation on the data to be fused based on the calculation weights, and obtains a fused image from the perspective of the human eye, specifically comprising: The image fusion module performs statistics on the projection distribution of the exit pupil according to the three-dimensional coordinate information of the human eye, obtains the distribution function of the exit pupil by fitting, and determines the calculation weight by using the distribution function; The image fusion module performs data operation on the to-be-fused data corresponding to the left view and the to-be-fused data corresponding to the right view based on the calculation weights, and fuses them to obtain the fused image.

7. A light field display processing method, characterized in that: include: Acquire binocular images and infer the three-dimensional coordinate information and pattern information of the human eye; The binocular image is an image of the environment in front of the light field display collected by binocular photography; Obtaining left and right views, and performing image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image; the left and right views are images to be displayed under the left and right viewing angles; The light field image is regulated according to the mode information to perform light field display on the light field display.

8. The light field display processing method according to claim 7, characterized in that: The step of acquiring binocular images and inferring the three-dimensional coordinate information and pattern information of human eyes specifically includes: Obtaining the binocular image from a binocular camera module; the binocular camera module is arranged facing the front of the light field display, and is used to collect the environment image in front of the light field display through binocular camera, and generate the binocular image; Performing data splitting and data correction on the binocular image to generate a corrected image, and performing human eye feature inference on the corrected image, normalizing and dimensionally converting the inference result to obtain the three-dimensional coordinate information of the human eye; A display mode is selected from a preset mode configuration table, and the mode information is generated based on the display mode.

9. The light field display processing method according to claim 7, characterized in that: After obtaining the three-dimensional coordinate information of the human eye, a jitter determination process is also performed: Calculate the coordinate difference between the three-dimensional coordinate information of the human eye and the coordinate information obtained last time, and determine whether the coordinate difference is within a preset range. If so, retain the three-dimensional coordinate information of the human eye; otherwise, abandon the three-dimensional coordinate information of the human eye and obtain it again.

10. The light field display processing method according to claim 7, characterized in that: The acquiring of the left and right views and performing image fusion processing and light field rendering processing in combination with the three-dimensional coordinate information of the human eye to obtain a light field image specifically includes: Convert the left and right views into data formats to obtain the data to be fused corresponding to the left view and the right view respectively; Configuring calculation weights according to the three-dimensional coordinate information of the human eye, performing weighted fusion calculation on the data to be fused based on the calculation weights, and obtaining a fused image from the perspective of the human eye; According to the preset relationship between the grayscale of the picture and the imaging brightness, the brightness of the fused image is compensated to obtain a compensated image; The compensation image is subjected to a sorting process of each color channel to generate the light field image in an RBG color domain.

Images