Naked eye light field 3D display system and method based on human eye tracking, electronic equipment and storage medium

Abstract

The invention discloses a naked eye light field 3D display system and method based on human eye tracking, electronic equipment and a storage medium. The naked eye light field 3D display system comprises a human eye tracking module, a data processing and transmission module, a 3D content generation and rendering module and a naked eye light field 3D display system. The human eye tracking module is used for positioning and tracking human eye positions in real time; the data processing and transmission module is used for carrying out filtering and prediction processing on the human eye position coordinates in real time; the 3D content generation and rendering module is used for generating and rendering 3D content of a corresponding position according to the obtained human eye position coordinates; and the naked eye light field 3D display system is used for receiving the data sent by the 3D content generation and rendering module and displaying 3D scenes at different viewing positions in real time. According to the invention, a user can experience a light field 3D effect without wearing any auxiliary equipment while high-resolution display content is brought, and more comfortable and natural naked-eye light field 3D experience is brought to the user.

Description

Technical Field

[0001] The present invention belongs to the field of 3D display technology, and relates to a 3D display system, and in particular to a naked-eye light field 3D display system, method, electronic device and storage medium based on human eye tracking. Background Art

[0002] Traditional 3D display technology often requires the use of auxiliary equipment such as glasses or helmets to achieve a three-dimensional effect. Long-term use will cause severe discomfort and dizziness, and the user experience is poor. Naked-eye 3D technology allows users to watch the real 3D depth of field effect in real time without wearing any auxiliary equipment. To a certain extent, it solves the sense of restraint and dizziness brought by traditional virtual reality equipment to users. It has been widely used in exhibitions, medical imaging, engineering design, military and other fields.

[0003] Light field 3D display technology collects the light space seen by the human eye and reorganizes the light, and then simulates the effect of the human eye focusing and moving objects based on distance in virtual space, completely reproducing the 3D scene and providing users with a free stereoscopic effect with full parallax. This technology further enhances the naked eye 3D viewing experience, with advantages such as high resolution, free viewing angle, and no visual fatigue, and can provide a more realistic and natural 3D visual experience. However, how to ensure that users can obtain the best 3D visual experience at any position and angle is an important challenge.

[0004] In view of this, there is an urgent need to design a new light field 3D display system to overcome at least some of the above-mentioned defects of the existing light field 3D display system. Summary of the invention

[0005] In order to solve the dizziness and discomfort caused by the need to wear glasses or helmets for traditional 3D displays and to improve the user experience of watching 3D content, the present invention provides a naked-eye light field 3D display system, method, electronic device and storage medium based on human eye tracking, which can provide high-resolution display content while allowing users to experience the light field 3D effect without wearing any auxiliary equipment, thereby bringing users a more comfortable and natural naked-eye light field 3D experience.

[0006] To solve the above technical problem, according to one aspect of the present invention, the following technical solution is adopted:

[0007] A naked-eye light field 3D display system based on human eye tracking, the naked-eye light field 3D display system comprising:

[0008] The human eye tracking module is used to locate and track the human eye position in real time and output the spatial position coordinates of the human eye to the data processing and transmission module;

[0009] The data processing and transmission module is used to filter and predict the human eye position coordinates in real time and transmit the processed data to the 3D content generation and rendering module;

[0010] A 3D content generation and rendering module is used to generate and render 3D content at the corresponding position according to the acquired human eye position coordinates;

[0011] The naked eye light field 3D display system is used to receive the data sent by the 3D content generation and rendering module and display the 3D scenes at different viewing positions in real time.

[0012] As an embodiment of the present invention, the eye tracking module includes an industrial camera module and an end-to-end eye tracking algorithm, and the eye image is collected through the industrial camera module; the collection equipment includes but is not limited to a monocular camera, a binocular camera, etc.

[0013] The industrial camera module adopts a binocular real-sense depth camera to obtain the position information of the human eye; the binocular real-sense depth camera includes two infrared cameras and one RGB camera, which are used to capture the RGB image and infrared image of the human eye in real time;

[0014] The end-to-end human eye tracking algorithm processes the human eye RGB image and infrared image in real time to locate the human eye position, and finally outputs the viewer's binocular spatial position coordinates by fusing the RGB image with the depth image and converting the coordinate system.

[0015] The data processing and transmission module is used to filter, predict and transmit the binocular spatial position information captured by the human eye tracking module;

[0016] The human eye spatial coordinate data within a set time is processed through a filtering algorithm to remove data noise, and the human eye spatial position state is estimated and predicted in real time to improve data accuracy and stability. The human eye position data processed by the algorithm is transmitted through a connectionless transport layer protocol, greatly reducing data transmission delay.

[0017] The 3D content generation and rendering module generates images for 3D display through a 3D content production tool and outputs them in 3D format, and adjusts the position of the virtual camera in real time in the 3D content production tool according to the acquired human eye position coordinates, and renders and generates a synthetic image of the corresponding orientation; the 3D format includes but is not limited to left-right 3D format, top-bottom 3D format, and row-interleaved 3D format. The 3D content production tool includes but is not limited to Unity3D, Blender, Maya, etc.

[0018] The 3D content generation is to create a virtual scene in a 3D content production tool, add a virtual camera to simulate the human eye, set the virtual camera spacing to simulate the pupil distance of the human eye, shoot the virtual scene to generate an image for 3D display, and render it into a 3D format screen output.

[0019] The naked-eye light field 3D display system includes a naked-eye 3D display, a human eye tracking system and a computer system;

[0020] The eye tracking system obtains the position information of both eyes in real time, and the 3D content synthesis and rendering module renders the 3D virtual reality scene in 3D format, and at the same time receives the eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system;

[0021] The naked-eye 3D display and computer system are processed by algorithms to display the light field 3D images corresponding to the human eye position in real time, and the user can see the light field 3D effects at any angle without wearing any auxiliary equipment.

[0022] As an implementation mode of the present invention, the eye tracking module detects the working state of the camera through the camera management module. After the camera is started normally, the binocular real-sense depth camera captures RGB image data and infrared image data with relative parallax in real time; the eye tracking algorithm processes the RGB image data in real time, locates the region of interest (ROI) of the eye, and outputs the image coordinates m(x, y) of the eye;

[0023] From the acquired infrared disparity image, according to the geometric relationship of parallel binocular vision, the conversion formula between depth D and disparity P is as follows:

[0024] D=(f*B) / P

[0025] Among them, f is the normalized camera focal length, B is the distance between the optical centers of the two infrared cameras, and the depth value of the image can be calculated by the above formula;

[0026] By matching the acquired eye image coordinates with the depth map, after data fusion and coordinate mapping, the eye tracking module outputs the final eye spatial position coordinates M (x, y, z), and transmits the eye spatial position coordinates to the data transmission and processing module through the USB transmission interface.

[0027] After receiving the eye space coordinate data, the data processing and transmission module uses the Kalman filter algorithm to process the data. The Kalman filter algorithm estimates the state of the system through prediction and update. The prediction stage predicts the state at the current moment based on the state at the previous moment and the input control amount; the update stage corrects the prediction result according to the actual observation value to obtain a more accurate state estimation;

[0028] The system model is constructed based on the acquired human eye space coordinates. The system model is represented by the state equation and the observation equation. x represents the system vector. The state equation describes the state vector x at the previous moment. k-1 Mapped to the current system state x k ;

[0029] x k =F k x k-1 +B k u k +w k

[0030] Among them, F k is the transformation matrix, B k is the control matrix, w k is process noise, with mean 0 and covariance Q k Gaussian distribution, w k ~N(0,Q k );

[0031] Z k =H k x k +v k

[0032] Among them, H k is the observation matrix, v k is the observation noise, with a mean of 0 and a covariance of R k Gaussian distribution, v k ~N(0,R k );

[0033] The system model is predicted to generate the predicted value of the human eye spatial position coordinates at the current moment. The human eye spatial position state prediction equation at this time is:

[0034]

[0035] in, Indicates the predicted value of the eye coordinates at this time. At this time, w k The value is 0. Since the eye space coordinates are directly output by module S1, the system has no control input, so the control matrix B k is 0;

[0036] The prediction error covariance matrix at this time is It can be expressed by the following formula:

[0037]

[0038] According to the data obtained in the prediction process, the Kalman gain at this time can be calculated to complete the update of the spatial position coordinates of the human eye. At this time, the Kalman gain Kk It can be expressed by the following formula:

[0039]

[0040] After completing the prediction of the human eye data, the system model state is updated to output the final spatial coordinates of the human eye after Kalman filtering. The Kalman gain K obtained by the prediction process is k , we can get the updated and the updated It is expressed by the following formula:

[0041]

[0042] After the Kalman filter algorithm is updated and calculated, the data processing and transmission module outputs the spatial coordinates of the human eye at this time. That is, the spatial position coordinates of the human eye after filtering, denoising and prediction, and the data is transmitted to the 3D content generation and rendering module through the User Datagram Protocol (UDP). UDP is a connectionless network transmission protocol with the characteristics of fast speed, low latency and low resource consumption. By creating data packets and specifying transmission IP and ports, low-latency output of data streams can be achieved.

[0043] As an implementation mode of the present invention, the 3D content generation and rendering module receives the spatial position coordinates of the human eye after data processing, adjusts the virtual camera matrix in the virtual scene in real time according to the coordinate information, and renders to generate a synthetic image of the corresponding orientation; uses Unity3D to create a virtual scene, and sets the virtual camera in an off-axis manner, sets the virtual camera spacing to simulate the pupil distance of the human eye, shoots the virtual scene to generate an image for 3D display, and renders it into a left-right 3D format (side-by-side) screen output;

[0044] Setting the virtual camera in an off-axis manner means aligning the projection planes of multiple virtual cameras by cutting in Unity3D to ensure that the captured image has a reasonable three-dimensional effect and avoid the generation of vertical parallax in 3D display. This method can generate multiple image sequences with parallax at one time, and encode the image sequence for light field display synthesis, which is fast and accurate.

[0045] After the UDP communication module in the 3D content generation and rendering module obtains the coordinates of the human eye position, the position of the virtual camera is modified in real time according to the obtained human eye position coordinates, and the camera projection matrix is ​​modified according to the calculated camera offset value to achieve the effect that the virtual scene follows the movement of the human eye.

[0046] After receiving the 3D virtual reality scene rendered in 3D format by the 3D content synthesis and rendering module S3, the naked eye light field 3D display system simultaneously receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system;

[0047] When the user is in different viewing positions, the naked-eye 3D display and computer system are processed by algorithms to display the light field 3D virtual reality images corresponding to the human eye position in real time, so that users in different viewing zones can view the content of the corresponding viewing zones. After image algorithm processing and synthesis, the user can finally see the light field 3D effect without wearing any auxiliary equipment.

[0048] According to another aspect of the present invention, the following technical solution is adopted: a naked eye light field 3D display method based on human eye tracking, the naked eye light field 3D display method comprising:

[0049] The human eye tracking module locates and tracks the position of the human eye in real time, and outputs the spatial position coordinates of the human eye to the data processing and transmission module;

[0050] The data processing and transmission module filters and predicts the eye position coordinates in real time, and transmits the processed data to the 3D content generation and rendering module;

[0051] The 3D content generation and rendering module generates and renders the 3D content of the corresponding position according to the acquired human eye position coordinates;

[0052] The naked-eye light field 3D display system receives the data sent by the 3D content generation and rendering module, and displays the 3D scenes at different viewing positions in real time.

[0053] As an embodiment of the present invention, the eye tracking module includes an industrial camera module and an end-to-end eye tracking algorithm, and the eye image is collected through the industrial camera module; the collection equipment includes but is not limited to a monocular camera, a binocular camera, etc.

[0054] The industrial camera module adopts a binocular real-sense depth camera to obtain the position information of the human eye; the binocular real-sense depth camera includes two infrared cameras and one RGB camera, which are used to capture the RGB image and infrared image of the human eye in real time;

[0055] The end-to-end human eye tracking algorithm processes the human eye RGB image and infrared image in real time to locate the human eye position, and finally outputs the viewer's binocular spatial position coordinates by fusing the RGB image with the depth image and converting the coordinate system.

[0056] The data processing and transmission module filters, predicts and transmits the binocular spatial position information captured by the human eye tracking module; processes the human eye spatial coordinate data within a set time through a filtering algorithm to remove data noise, and estimates and predicts the state of the human eye spatial position in real time to improve the accuracy and stability of the data; and transmits the human eye position data after algorithm processing through a connectionless transport layer protocol, greatly reducing data transmission delay.

[0057] The 3D content generation and rendering module generates images for 3D display through a 3D content production tool and outputs them in 3D format, and adjusts the position of the virtual camera in real time in the 3D content production tool according to the acquired human eye position coordinates, and renders and generates a synthetic image of the corresponding orientation; the 3D format includes but is not limited to left-right 3D format, top-bottom 3D format, and row-interleaved 3D format. The 3D content production tool includes but is not limited to Unity3D, Blender, Maya, etc.

[0058] The 3D content generation is to create a virtual scene in a 3D content production tool, add a virtual camera to simulate the human eye, set the virtual camera spacing to simulate the pupil distance of the human eye, shoot the virtual scene to generate an image for 3D display, and render it into a 3D format screen output.

[0059] The naked-eye light field 3D display system includes a naked-eye 3D display, a human eye tracking system and a computer system;

[0060] The eye tracking system obtains the position information of both eyes in real time, and the 3D content synthesis and rendering module renders the 3D virtual reality scene in 3D format, and at the same time receives the eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system;

[0061] The naked-eye 3D display and computer system are processed by algorithms to display the light field 3D images corresponding to the human eye position in real time, and the user can see the light field 3D effects at any angle without wearing any auxiliary equipment.

[0062] As an implementation mode of the present invention, the eye tracking module detects the working state of the camera through the camera management module. After the camera is started normally, the binocular real-sense depth camera captures RGB image data and infrared image data with relative parallax in real time; the eye tracking algorithm processes the RGB image data in real time, locates the region of interest (ROI) of the eye, and outputs the image coordinates m(x, y) of the eye;

[0063] From the acquired infrared disparity image, according to the geometric relationship of parallel binocular vision, the conversion formula between depth D and disparity P is as follows:

[0064] D=(f*B) / P

[0065] Among them, f is the normalized camera focal length, B is the distance between the optical centers of the two infrared cameras, and the depth value of the image can be calculated by the above formula;

[0066] By matching the acquired eye image coordinates with the depth map, after data fusion and coordinate mapping, the eye tracking module outputs the final eye spatial position coordinates M (x, y, z), and transmits the eye spatial position coordinates to the data transmission and processing module through the USB transmission interface.

[0067] After receiving the eye space coordinate data, the data processing and transmission module uses the Kalman filter algorithm to process the data. The Kalman filter algorithm estimates the state of the system through prediction and update. The prediction stage predicts the state at the current moment based on the state at the previous moment and the input control amount; the update stage corrects the prediction result according to the actual observation value to obtain a more accurate state estimation;

[0068] The system model is constructed based on the acquired human eye space coordinates. The system model is represented by the state equation and the observation equation. x represents the system vector. The state equation describes the state vector x at the previous moment. k-1 Mapped to the current system state x k ;

[0069] x k =F k x k-1 +B k u k +w k

[0070] Among them, F k is the transformation matrix, B k is the control matrix, w k is process noise, with mean 0 and covariance Q k Gaussian distribution, w k ~N(0,Q k );

[0071] Z k =H k x k +v k

[0072] Among them, H k is the observation matrix, v k is the observation noise, with a mean of 0 and a covariance of R k Gaussian distribution, v k ~N(0,R k );

[0073] The system model is predicted to generate the predicted value of the human eye spatial position coordinates at the current moment. The human eye spatial position state prediction equation at this time is:

[0074]

[0075] in, Indicates the predicted value of the eye coordinates at this time. At this time, w k The value is 0. Since the eye space coordinates are directly output by module S1, the system has no control input, so the control matrix B k is 0;

[0076] The prediction error covariance matrix at this time is It can be expressed by the following formula:

[0077]

[0078] According to the data obtained in the prediction process, the Kalman gain at this time can be calculated to complete the update of the spatial position coordinates of the human eye. At this time, the Kalman gain K k It can be expressed by the following formula:

[0079]

[0080] After completing the prediction of the human eye data, the system model state is updated to output the final spatial coordinates of the human eye after Kalman filtering. The Kalman gain K obtained by the prediction process is k , we can get the updated and the updated It is expressed by the following formula:

[0081]

[0082] After the Kalman filter algorithm is updated and calculated, the data processing and transmission module outputs the spatial coordinates of the human eye at this time. That is, the spatial coordinates of the human eye after filtering, denoising and prediction, and the data is transmitted to the 3D content generation and rendering module through the User Datagram Protocol (UDP). UDP is a connectionless network transmission protocol with the characteristics of fast speed, low latency and low resource consumption. By creating data packets and specifying transmission IP and ports, low-latency output of data streams can be achieved.

[0083] As an implementation mode of the present invention, the 3D content generation and rendering module receives the coordinates of the human eye space position after data processing, adjusts the virtual camera matrix in the virtual scene in real time according to the coordinate information, and renders and generates a synthetic image of the corresponding orientation; uses Unity3D to create a virtual scene, and sets the virtual camera in an off-axis manner, sets the virtual camera spacing to simulate the pupil distance of the human eye, shoots the virtual scene to generate an image for 3D display, and renders it into a left-right 3D format (side-by-side) screen output;

[0084] Setting the virtual camera in an off-axis manner means aligning the projection planes of multiple virtual cameras by cutting in Unity3D to ensure that the captured image has a reasonable three-dimensional effect and avoid the generation of vertical parallax in 3D display. This method can generate multiple image sequences with parallax at one time, and encode the image sequence for light field display synthesis, which is fast and accurate.

[0085] After the UDP communication module in the 3D content generation and rendering module obtains the coordinates of the human eye position, the position of the virtual camera is modified in real time according to the obtained human eye position coordinates, and the camera projection matrix is ​​modified according to the calculated camera offset value to achieve the effect that the virtual scene follows the movement of the human eye.

[0086] After receiving the 3D virtual reality scene rendered in 3D format by the 3D content synthesis and rendering module S3, the naked eye light field 3D display system simultaneously receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system;

[0087] When the user is in different viewing positions, the naked-eye 3D display and computer system are processed by algorithms to display the light field 3D virtual reality images corresponding to the human eye position in real time, so that users in different viewing zones can view the content of the corresponding viewing zones. After image algorithm processing and synthesis, the user can finally see the light field 3D effect without wearing any auxiliary equipment.

[0088] According to another aspect of the present invention, the following technical solution is adopted: an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0089] According to another aspect of the present invention, the following technical solution is adopted: a storage medium stores computer program instructions, and the computer program instructions implement the steps of the above method when executed by a processor.

[0090] The beneficial effects of the present invention are as follows: the naked-eye light field 3D display system, method, electronic device and storage medium based on human eye tracking proposed in the present invention can provide high-resolution display content while allowing users to experience the light field 3D effect without wearing any auxiliary equipment, thereby bringing users a more comfortable and natural naked-eye light field 3D experience.

[0091] The present invention combines human eye tracking technology with naked-eye 3D light field display technology, realizes precise positioning of the human eye and real-time rendering of 3D content based on the position information of the human eye, so that users can watch real, continuous and natural naked-eye light field 3D content at different angles without the aid of any auxiliary equipment, thereby reducing the viewing burden and visual fatigue of users, improving the performance and user experience of the naked-eye light field 3D display system, and providing new ideas and methods for the development and application of light field 3D display technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 Schematic diagram of the composition of a naked-eye light field 3D display system based on human eye tracking in one embodiment of the present invention.

[0093] Figure 2 The figure is a flow chart of a naked-eye light field 3D display method based on human eye tracking in one embodiment of the present invention.

[0094] Figure 3 Schematic diagram of the composition of a naked-eye light field 3D display system in one embodiment of the present invention.

[0095] Figure 4 This is a flowchart of 3D content generation and rendering in one embodiment of the present invention.

[0096] Figure 5 Schematic diagram of the algorithm flow for filtering eye tracking coordinates in one embodiment of the present invention.

[0097] Figure 6 It is a schematic diagram of a user viewing different 3D contents in different viewing areas according to an embodiment of the present invention.

[0098] Figure 7 FIG. 4 is a schematic diagram of the composition of an electronic device in one embodiment of the present invention. DETAILED DESCRIPTION

[0099] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0100] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0101] The description in this section is only for several typical embodiments, and the present invention is not limited to the scope of the embodiments. The same or similar prior art means and some technical features in the embodiments are mutually replaced within the scope of the present invention.

[0102] The description of the steps in each embodiment in the specification is only for the convenience of explanation, and the implementation method of the present application is not limited by the order of implementation of the steps.

[0103] The term "connection" as used in the specification includes both direct connection and indirect connection, such as connection through some active devices, passive devices or electrically conductive media; it may also include connection through other active devices or passive devices known to those skilled in the art on the basis of achieving the same or similar functional purposes, such as connection through circuits or components such as switches and follower circuits.

[0104] The present invention discloses a naked eye light field 3D display system based on human eye tracking. Figure 1 FIG. 1 is a schematic diagram of the composition of a naked-eye light field 3D display system based on human eye tracking in one embodiment of the present invention; please refer to FIG. Figure 1 The naked-eye light field 3D display system includes: a human eye tracking module 1, a data processing and transmission module 2, a 3D content generation and rendering module 3 and a naked-eye light field 3D display system 4.

[0105] The human eye tracking module 1 is used to locate and track the human eye position in real time, and output the spatial position coordinates of the human eye to the data processing and transmission module; the data processing and transmission module 2 is used to filter and predict the human eye position coordinates in real time, and transmit the processed data to the 3D content generation and rendering module, so as to output more accurate and stable human eye spatial position coordinates; the 3D content generation and rendering module 3 is used to generate and render 3D content at the corresponding position according to the acquired human eye position coordinates; the naked eye light field 3D display system 4 is used to receive the data sent by the 3D content generation and rendering module, and display 3D scenes of different viewing positions (such as any angle) in real time.

[0106] In one embodiment of the present invention, the eye tracking module 1 includes an industrial camera module and an end-to-end eye tracking algorithm, and the eye image is collected through the industrial camera module; the collection equipment includes but is not limited to a monocular camera, a binocular camera, etc. The industrial camera module adopts a binocular real-sense depth camera to obtain the eye position information; the binocular real-sense depth camera includes two infrared cameras and one RGB camera, which is used to capture the eye RGB image and infrared image in real time. The end-to-end eye tracking algorithm processes the eye RGB image and infrared image in real time to locate the eye position, and finally outputs the viewer's binocular spatial position coordinates by fusing the RGB image with the depth image and converting the coordinate system.

[0107] The data processing and transmission module 2 is used to filter, predict and transmit the binocular spatial position information captured by the human eye tracking module. The human eye spatial coordinate data within a set time is processed by a filtering algorithm to remove data noise, and the state of the human eye spatial position is estimated and predicted in real time to improve the accuracy and stability of the data. The human eye position data after algorithm processing is transmitted through a connectionless transport layer protocol, which greatly reduces the data transmission delay.

[0108] The 3D content generation and rendering module 3 generates an image for 3D display through a 3D content production tool and outputs it in 3D format, and adjusts the position of the virtual camera in real time in the 3D content production tool according to the acquired human eye position coordinates, and renders and generates a synthetic image of the corresponding orientation; the 3D format includes but is not limited to left-right 3D format, top-bottom 3D format, and row-interleaved 3D format. The 3D content production tool includes but is not limited to Unity3D, Blender, Maya, etc. The 3D content generation is to create a virtual scene in the 3D content production tool, add a virtual camera to simulate the human eye, set the virtual camera spacing to simulate the pupil distance of the human eye, shoot the virtual scene to generate an image for 3D display, and render it into a 3D format image output.

[0109] Figure 3 FIG. 1 is a schematic diagram of the composition of a naked-eye light field 3D display system according to an embodiment of the present invention; please refer to FIG. Figure 3 In one embodiment of the present invention, the naked-eye light field 3D display system 4 includes a naked-eye 3D display 42, a human eye tracking system 41 and a computer system 43. The human eye tracking system 41 obtains the position information of both eyes in real time, and the 3D content synthesis and rendering module renders the 3D virtual reality scene in 3D format, and at the same time receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked-eye 3D display and the computer system. The naked-eye 3D display 42 and the computer system 43 are processed by the algorithm to display the light field 3D picture corresponding to the human eye position in real time, and the user can see the light field 3D effect at any angle without wearing any auxiliary equipment. Specifically, the computer system 43 processes the human eye data in real time and renders and generates 3D content, and finally displays the light field 3D scene at the corresponding position on the naked-eye 3D display 42.

[0110] In one embodiment of the present invention, the eye tracking module 1 detects the working status of the camera through the camera management module. After the camera is started normally, the binocular real-sense depth camera captures RGB image data and infrared image data with relative parallax in real time; the eye tracking algorithm processes the RGB image data in real time, locates the region of interest (ROI) of the eye, and outputs the image coordinates m (x, y) of the eye.

[0111] From the acquired infrared disparity image, according to the geometric relationship of parallel binocular vision, the conversion formula between depth D and disparity P is as follows:

[0112] D=(f*B) / P

[0113] Among them, f is the normalized camera focal length, B is the distance between the optical centers of the two infrared cameras, and the depth value of the image can be calculated by the above formula;

[0114] By matching the acquired eye image coordinates with the depth map, after data fusion and coordinate mapping, the eye tracking module outputs the final eye spatial position coordinates M (x, y, z), and transmits the eye spatial position coordinates to the data transmission and processing module 2 via the USB transmission interface (of course, other transmission methods can also be selected).

[0115] After receiving the human eye spatial coordinate data, the data processing and transmission module 2 uses the Kalman filter algorithm to process the data. The Kalman filter algorithm estimates the state of the system through prediction and update. The prediction stage predicts the state at the current moment based on the state and input control quantity at the previous moment; the update stage corrects the prediction result according to the actual observation value to obtain a more accurate state estimation.

[0116] Figure 5 This is a schematic diagram of an algorithm flow for filtering eye tracking coordinates in one embodiment of the present invention; please refer to Figure 5 In one embodiment of the present invention, a system model is constructed based on the acquired eye space coordinates. The system model is represented by a state equation and an observation equation. x represents the system vector. The state equation describes the state vector x at the previous moment. k-1 Mapped to the current system state x k ;

[0117] x k =F k x k-1 +B k u k +w k

[0118] Among them, F k is the transformation matrix, B k is the control matrix, w k is process noise, with mean 0 and covariance Q k Gaussian distribution, w k ~N(0,Q k );

[0119] Z k =H k x k +v k

[0120] Among them, H k is the observation matrix, v k is the observation noise, with a mean of 0 and a covariance of R k Gaussian distribution, v k ~N(0,R k );

[0121] The system model is predicted to generate the predicted value of the human eye spatial position coordinates at the current moment. The human eye spatial position state prediction equation at this time is:

[0122]

[0123] in, Indicates the predicted value of the eye coordinates at this time. At this time, w k The value is 0. Since the eye space coordinates are directly output by module S1, the system has no control input, so the control matrix B k is 0;

[0124] The prediction error covariance matrix at this time is It can be expressed by the following formula:

[0125]

[0126] According to the data obtained in the prediction process, the Kalman gain at this time can be calculated to complete the update of the spatial position coordinates of the human eye. At this time, the Kalman gain K k It can be expressed by the following formula:

[0127]

[0128] After completing the prediction of the human eye data, the system model state is updated to output the final spatial coordinates of the human eye after Kalman filtering. The Kalman gain K obtained by the prediction process is k , we can get the updated and the updated It is expressed by the following formula:

[0129]

[0130] After the Kalman filter algorithm is updated and calculated, the data processing and transmission module outputs the spatial coordinates of the human eye at this time. That is, the spatial position coordinates of the human eye after filtering, denoising and prediction, and the data is transmitted to the 3D content generation and rendering module through the User Datagram Protocol (UDP). UDP is a connectionless network transmission protocol with the characteristics of fast speed, low latency and low resource consumption. By creating data packets and specifying transmission IP and ports, low-latency output of data streams can be achieved.

[0131] The 3D content generation and rendering module 3 receives the human eye spatial position coordinates after data processing, adjusts the virtual camera matrix in the virtual scene in real time according to the coordinate information, and renders and generates a synthetic image of the corresponding orientation. Figure 4 As shown, a virtual scene is created using Unity3D (of course, it can also be created using other systems), and a virtual camera is set in an off-axis manner, and the virtual camera spacing is set to simulate the pupil distance of the human eye. The virtual scene is photographed to generate an image for 3D display, and the image is rendered into a side-by-side 3D format (of course, it can also be other 3D formats) for output.

[0132] Setting the virtual camera in an off-axis manner means aligning the projection planes of multiple virtual cameras in Unity3D by cutting, ensuring that the captured image has a reasonable three-dimensional effect and avoiding the generation of vertical parallax in 3D display. This method can generate multiple image sequences with parallax at one time, and encode the image sequence for light field display synthesis, which is fast and accurate.

[0133] After the UDP communication module in the 3D content generation and rendering module obtains the coordinates of the human eye position, the position of the virtual camera is modified in real time according to the obtained human eye position coordinates, and the camera projection matrix is ​​modified according to the calculated camera offset value to achieve the effect that the virtual scene follows the movement of the human eye.

[0134] After receiving the 3D virtual reality scene rendered in 3D format by the 3D content synthesis and rendering module S3, the naked-eye light field 3D display system 4 simultaneously receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked-eye 3D display and the computer system.

[0135] Figure 6 This is a schematic diagram of a user viewing different 3D contents in different viewing areas in one embodiment of the present invention; please refer to Figure 6 In one embodiment of the present invention, when the user is in different visual zone positions, the naked eye 3D display and the computer system are processed by algorithms to display the light field 3D virtual reality images corresponding to the human eye position in real time, so that users in different visual zones can view the contents of the corresponding visual zones. After image algorithm processing and synthesis, the user can finally see the light field 3D effect without wearing any auxiliary equipment.

[0136] The present invention further discloses a naked-eye light field 3D display method for human eye tracking. Figure 2 This is a flow chart of a naked-eye light field 3D display method based on human eye tracking in one embodiment of the present invention; please refer to Figure 2 , the naked eye light field 3D display method comprises:

[0137] [Step S1] The human eye tracking module locates and tracks the position of the human eye in real time, and outputs the spatial position coordinates of the human eye to the data processing and transmission module;

[0138] [Step S2] The data processing and transmission module filters and predicts the eye position coordinates in real time, and transmits the processed data to the 3D content generation and rendering module;

[0139] [Step S3] The 3D content generation and rendering module generates and renders the 3D content of the corresponding position according to the acquired human eye position coordinates;

[0140] [Step S4] The naked-eye light field 3D display system receives the data sent by the 3D content generation and rendering module, and displays the 3D scenes at different viewing positions in real time.

[0141] In one embodiment of the present invention, the eye tracking module 1 includes an industrial camera module and an end-to-end eye tracking algorithm, and the eye image is collected through the industrial camera module; the collection equipment includes but is not limited to a monocular camera, a binocular camera, etc. The industrial camera module uses a binocular real-sense depth camera to obtain the eye position information; the binocular real-sense depth camera includes two infrared cameras and one RGB camera, which is used to capture the eye RGB image and infrared image in real time; the end-to-end eye tracking algorithm processes the eye RGB image and infrared image in real time to locate the eye position, and finally outputs the viewer's binocular spatial position coordinates by fusing the RGB image with the depth image and converting the coordinate system.

[0142] The data processing and transmission module 2 filters, predicts and transmits the binocular spatial position information captured by the human eye tracking module; processes the human eye spatial coordinate data within a set time through a filtering algorithm to remove data noise, and estimates and predicts the state of the human eye spatial position in real time to improve the accuracy and stability of the data, and transmits the human eye position data after algorithm processing through a connectionless transport layer protocol, greatly reducing data transmission delay.

[0143] The 3D content generation and rendering module 3 generates an image for 3D display through a 3D content production tool and outputs it in 3D format, and adjusts the position of the virtual camera in real time in the 3D content production tool according to the acquired human eye position coordinates, and renders and generates a synthetic image of the corresponding orientation; the 3D format includes but is not limited to left-right 3D format, top-bottom 3D format, and row-interleaved 3D format. The 3D content production tool includes but is not limited to Unity3D, Blender, Maya, etc. The 3D content generation is to create a virtual scene in the 3D content production tool, add a virtual camera to simulate the human eye, set the virtual camera spacing to simulate the pupil distance of the human eye, shoot the virtual scene to generate an image for 3D display, and render it into a 3D format image output.

[0144] Figure 3 FIG. 1 is a schematic diagram of the composition of a naked-eye light field 3D display system according to an embodiment of the present invention; please refer to FIG. Figure 3 In one embodiment of the present invention, the naked-eye light field 3D display system 4 includes a naked-eye 3D display 42, a human eye tracking system 41, and a computer system 43. The human eye tracking system 41 obtains the position information of both eyes in real time, and the 3D content synthesis and rendering module renders the 3D virtual reality scene in 3D format, and at the same time receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked-eye 3D display and the computer system; the naked-eye 3D display 42 and the computer system 43 are processed by algorithms to display the light field 3D picture corresponding to the human eye position in real time, and the user can see the light field 3D effect at any angle without wearing any auxiliary equipment.

[0145] In one embodiment of the present invention, in step S1, the eye tracking module detects the working state of the camera through the camera management module. After the camera is started normally, the binocular real-sense depth camera captures RGB image data and infrared image data with relative parallax in real time; the eye tracking algorithm processes the RGB image data in real time, locates the region of interest (ROI) of the eye, and outputs the image coordinates m(x, y) of the eye;

[0146] From the acquired infrared disparity image, according to the geometric relationship of parallel binocular vision, the conversion formula between depth D and disparity P is as follows:

[0147] D=(f*B) / P

[0148] Among them, f is the normalized camera focal length, B is the distance between the optical centers of the two infrared cameras, and the depth value of the image can be calculated by the above formula;

[0149] By matching the acquired eye image coordinates with the depth map, after data fusion and coordinate mapping, the eye tracking module outputs the final eye spatial position coordinates M (x, y, z), and transmits the eye spatial position coordinates to the data transmission and processing module through the USB transmission interface.

[0150] In one embodiment of the present invention, in step S2, after receiving the eye space coordinate data, the data processing and transmission module uses the Kalman filter algorithm to process the data. The Kalman filter algorithm estimates the state of the system through prediction and update. The prediction stage predicts the state at the current moment based on the state at the previous moment and the input control amount; the update stage corrects the prediction result according to the actual observation value to obtain a more accurate state estimation;

[0151] Figure 5 This is a schematic diagram of an algorithm flow for filtering eye tracking coordinates in one embodiment of the present invention; please refer to Figure 5 In one embodiment of the present invention, a system model is constructed based on the acquired eye space coordinates. The system model is represented by a state equation and an observation equation. x represents the system vector. The state equation describes the state vector x at the previous moment. k-1 Mapped to the current system state x k ;

[0152] x k =F k x k-1 +B k u k +w k

[0153] Among them, F k is the transformation matrix, B k is the control matrix, w k is process noise, with mean 0 and covariance Q k Gaussian distribution, w k ~N(0,Q k );

[0154] Z k =H k x k +v k

[0155] Among them, H k is the observation matrix, v k is the observation noise, with a mean of 0 and a covariance of R k Gaussian distribution, v k ~N(0,R k );

[0156] The system model is predicted to generate the predicted value of the human eye spatial position coordinates at the current moment. The human eye spatial position state prediction equation at this time is:

[0157]

[0158] in, Indicates the predicted value of the eye coordinates at this time. At this time, w k The value is 0. Since the eye space coordinates are directly output by module S1, the system has no control input, so the control matrix B k is 0;

[0159] The prediction error covariance matrix at this time is It can be expressed by the following formula:

[0160]

[0161] According to the data obtained in the prediction process, the Kalman gain at this time can be calculated to complete the update of the spatial position coordinates of the human eye. At this time, the Kalman gain K k It can be expressed by the following formula:

[0162]

[0163] After completing the prediction of the human eye data, the system model state is updated to output the final spatial coordinates of the human eye after Kalman filtering. The Kalman gain K obtained by the prediction process is k , we can get the updated and the updated It is expressed by the following formula:

[0164]

[0165] After the Kalman filter algorithm is updated and calculated, the data processing and transmission module outputs the spatial coordinates of the human eye at this time. That is, the spatial position coordinates of the human eye after filtering, denoising and prediction, and the data is transmitted to the 3D content generation and rendering module through the User Datagram Protocol (UDP). UDP is a connectionless network transmission protocol with the characteristics of fast speed, low latency and low resource consumption. By creating data packets and specifying transmission IP and ports, low-latency output of data streams can be achieved.

[0166] In one embodiment of the present invention, in step S3, the 3D content generation and rendering module receives the processed human eye spatial position coordinates, adjusts the virtual camera matrix in the virtual scene in real time according to the coordinate information, and renders and generates a synthetic image of the corresponding orientation. Figure 4As shown, Unity3D can be used to create a virtual scene (of course, other systems can also be used to create it), and the virtual camera is set in an off-axis manner, and the virtual camera spacing is set to simulate the pupil distance of the human eye, and the virtual scene is photographed to generate an image for 3D display, and rendered into a left-right 3D format (side-by-side) screen output;

[0167] Setting the virtual camera in an off-axis manner means aligning the projection planes of multiple virtual cameras by cutting in Unity3D to ensure that the captured image has a reasonable three-dimensional effect and avoid the generation of vertical parallax in 3D display. This method can generate multiple image sequences with parallax at one time, and encode the image sequence for light field display synthesis, which is fast and accurate.

[0168] After the UDP communication module in the 3D content generation and rendering module obtains the coordinates of the human eye position, the position of the virtual camera is modified in real time according to the obtained human eye position coordinates, and the camera projection matrix is ​​modified according to the calculated camera offset value to achieve the effect that the virtual scene follows the movement of the human eye.

[0169] In one embodiment of the present invention, in step S4, after receiving the 3D virtual reality scene rendered in 3D format by the 3D content synthesis and rendering module S3, the naked eye light field 3D display system simultaneously receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system;

[0170] like Figure 6 As shown, when the user is in different viewing positions, the naked-eye 3D display and computer system are processed by algorithms to display the light field 3D virtual reality images corresponding to the human eye position in real time, so that users in different viewing zones can view the contents of the corresponding viewing zones. After image algorithm processing and synthesis, the user can finally see the light field 3D effect without wearing any auxiliary equipment.

[0171] The present invention also discloses an electronic device, Figure 7 FIG. 1 is a schematic diagram of the composition of an electronic device in an embodiment of the present invention; please refer to FIG. Figure 7 At the hardware level, the electronic device includes a memory, a processor and at least one network interface; the processor may be a microprocessor, and the memory may include a memory, such as a random access memory (RAM), or a non-volatile memory, etc. Of course, the electronic device may also be provided with other hardware as required.

[0172] The processor, network interface and memory may be interconnected via an internal bus, which may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus may include an address bus, a data bus, a control bus, etc. The memory is used to store programs (which may include operating system programs and application programs); the program may include program code, which may include computer operation instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0173] In one embodiment, the processor can read the corresponding program from the non-volatile memory into the memory and then run it; the processor can execute the program stored in the memory and is specifically used to perform the following operations (such as Figure 2 shown):

[0174] [Step S1] The human eye tracking module locates and tracks the position of the human eye in real time, and outputs the spatial position coordinates of the human eye to the data processing and transmission module;

[0175] [Step S2] The data processing and transmission module filters and predicts the eye position coordinates in real time, and transmits the processed data to the 3D content generation and rendering module;

[0176] [Step S3] The 3D content generation and rendering module generates and renders the 3D content of the corresponding position according to the acquired human eye position coordinates;

[0177] [Step S4] The naked-eye light field 3D display system receives the data sent by the 3D content generation and rendering module, and displays the 3D scenes at different viewing positions in real time.

[0178] The present invention further discloses a storage medium having computer program instructions stored thereon, which, when executed by a processor, implements the following steps of the method of the present invention (eg Figure 2 shown):

[0179] [Step S1] The human eye tracking module locates and tracks the position of the human eye in real time, and outputs the spatial position coordinates of the human eye to the data processing and transmission module;

[0180] [Step S2] The data processing and transmission module filters and predicts the eye position coordinates in real time, and transmits the processed data to the 3D content generation and rendering module;

[0181] [Step S3] The 3D content generation and rendering module generates and renders the 3D content of the corresponding position according to the acquired human eye position coordinates;

[0182] [Step S4] The naked-eye light field 3D display system receives the data sent by the 3D content generation and rendering module, and displays the 3D scenes at different viewing positions in real time.

[0183] In summary, the naked-eye light field 3D display system, method, electronic device and storage medium based on human eye tracking proposed in the present invention can provide high-resolution display content while allowing users to experience the light field 3D effect without wearing any auxiliary equipment, thereby bringing users a more comfortable and natural naked-eye light field 3D experience.

[0184] The present invention combines human eye tracking technology with naked-eye 3D light field display technology, realizes precise positioning of the human eye and real-time rendering of 3D content based on the position information of the human eye, so that users can watch real, continuous and natural naked-eye light field 3D content at different angles without the aid of any auxiliary equipment, thereby reducing the viewing burden and visual fatigue of users, improving the performance and user experience of the naked-eye light field 3D display system, and providing new ideas and methods for the development and application of light field 3D display technology.

[0185] It should be noted that the present application can be implemented in software and / or a combination of software and hardware; for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium; for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented in hardware; for example, as a circuit that cooperates with a processor to perform various steps or functions.

[0186] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0187] The description and application of the present invention here are illustrative, and it is not intended to limit the scope of the present invention to the above-mentioned embodiments. The effects or advantages involved in the embodiments may not be embodied in the embodiments due to interference from various factors, and the description of the effects or advantages is not used to limit the embodiments. The deformation and change of the embodiments disclosed here are possible, and the replacement of the embodiments and the various equivalent parts are well known to those of ordinary skill in the art. It should be clear to those skilled in the art that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential features of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. A naked-eye light field 3D display system based on human eye tracking, characterized in that: The naked eye light field 3D display system comprises: The human eye tracking module is used to locate and track the human eye position in real time and output the spatial position coordinates of the human eye to the data processing and transmission module; The data processing and transmission module is used to filter and predict the human eye position coordinates in real time and transmit the processed data to the 3D content generation and rendering module; A 3D content generation and rendering module is used to generate and render 3D content at the corresponding position according to the acquired human eye position coordinates; The naked eye light field 3D display system is used to receive the data sent by the 3D content generation and rendering module and display the 3D scenes at different viewing positions in real time.

2. The naked-eye light field 3D display system based on human eye tracking according to claim 1, characterized in that: The human eye tracking module includes an industrial camera module and an end-to-end human eye tracking algorithm, and collects human eye images through the industrial camera module; The industrial camera module uses a binocular real-sense depth camera to capture and obtain human eye position information; The binocular real-sense depth camera includes two infrared cameras and one RGB camera, which are used to capture human eye RGB images and infrared images in real time; The end-to-end human eye tracking algorithm processes the human eye RGB image and infrared image in real time to locate the human eye position, and finally outputs the viewer's binocular spatial position coordinates by fusing the RGB image with the depth image and converting the coordinate system; The data processing and transmission module is used to filter, predict and transmit the binocular spatial position information captured by the human eye tracking module; The human eye spatial coordinate data within a set time is processed through a filtering algorithm to remove data noise, and the human eye spatial position state is estimated and predicted in real time, and the human eye position data after algorithm processing is transmitted through a connectionless transport layer protocol; The 3D content generation and rendering module generates an image for 3D display through a 3D content production tool and outputs it in 3D format, and adjusts the position of the virtual camera in real time in the 3D content production tool according to the acquired human eye position coordinates, and renders and generates a synthetic image of the corresponding orientation; The 3D content generation is to create a virtual scene in a 3D content production tool, add a virtual camera to simulate the human eye, set the virtual camera distance to simulate the pupil distance of the human eye, shoot the virtual scene to generate an image for 3D display, and render it into a 3D format image output; The naked-eye light field 3D display system includes a naked-eye 3D display, a human eye tracking system and a computer system; The eye tracking system obtains the position information of both eyes in real time, and the 3D content synthesis and rendering module renders the 3D virtual reality scene in 3D format, and at the same time receives the eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system; The naked-eye 3D display and computer system are processed by algorithms to display the light field 3D images corresponding to the human eye position in real time, and the user can see the light field 3D effects at any angle without wearing any auxiliary equipment.

3. The naked-eye light field 3D display system based on human eye tracking according to claim 1, characterized in that: The eye tracking module detects the working status of the camera through the camera management module. After the camera is started normally, the binocular real-sense depth camera captures RGB image data and infrared image data with relative parallax in real time; The eye tracking algorithm processes the RGB image data in real time, locates the region of interest (ROI) of the eye, and outputs the image coordinates m(x,y) of the eye; From the acquired infrared disparity image, according to the geometric relationship of parallel binocular vision, the conversion formula between depth D and disparity P is as follows: D=(f*B) / P Among them, f is the normalized camera focal length, B is the distance between the optical centers of the two infrared cameras, and the depth value of the image can be calculated by the above formula; By matching the acquired eye image coordinates with the depth map, after data fusion and coordinate mapping, the eye tracking module outputs the final eye spatial position coordinates M (x, y, z), and transmits the eye spatial position coordinates to the data transmission and processing module through the USB transmission interface; After receiving the eye space coordinate data, the data processing and transmission module uses the Kalman filter algorithm to process the data. The Kalman filter algorithm estimates the state of the system through prediction and update. The prediction stage predicts the state at the current moment based on the state at the previous moment and the input control amount; the update stage corrects the prediction result according to the actual observation value to obtain a more accurate state estimation; The system model is constructed based on the acquired human eye space coordinates. The system model is represented by the state equation and the observation equation. x represents the system vector. The state equation describes the state vector x at the previous moment. k-1 Mapped to the current system state x k ; x k =F k x k-1 +B k u k +w k Among them, F k is the transformation matrix, B k is the control matrix, w k is process noise, with mean 0 and covariance Q k Gaussian distribution, w k ~N(0,Q k ); Z k =H k x k +v k Among them, H k is the observation matrix, v k is the observation noise, with a mean of 0 and a covariance of R k Gaussian distribution, v k ~N(0,R k ); The system model is predicted to generate the predicted value of the human eye spatial position coordinates at the current moment. The human eye spatial position state prediction equation at this time is: in, Indicates the predicted value of the eye coordinates at this time. At this time, w k The value is 0. Since the eye space coordinates are directly output by module S1, the system has no control input, so the control matrix B k is 0; The prediction error covariance matrix at this time is It can be expressed by the following formula: According to the data obtained in the prediction process, the Kalman gain at this time can be calculated to complete the update of the spatial position coordinates of the human eye. At this time, the Kalman gain K k It can be expressed by the following formula: After completing the prediction of the human eye data, the system model state is updated to output the final spatial position coordinates of the human eye after Kalman filtering; the Kalman gain K obtained by the prediction process k , we can get the updated and the updated It is expressed by the following formula: After the Kalman filter algorithm is updated and calculated, the data processing and transmission module outputs the spatial coordinates of the human eye at this time. That is, the spatial position coordinates of the human eye after filtering, denoising and prediction, and the data is transmitted to the 3D content generation and rendering module through the User Datagram Protocol (UDP).

4. The naked-eye light field 3D display system based on human eye tracking according to claim 1, characterized in that: The 3D content generation and rendering module receives the human eye spatial position coordinates after data processing, adjusts the virtual camera matrix in the virtual scene in real time according to the coordinate information, and renders and generates a synthetic image of the corresponding orientation; creates a virtual scene, sets the virtual camera in an off-axis manner, sets the virtual camera spacing to simulate the pupil distance of the human eye, shoots the virtual scene to generate an image for 3D display, and renders it into a 3D format screen output; The virtual camera is set up in an off-axis manner to align the projection planes of multiple virtual cameras through cropping, ensuring that the captured images have a reasonable three-dimensional effect and avoiding the generation of vertical parallax in 3D display; This method can generate multiple image sequences with parallax at one time, and encode the image sequences for light field display synthesis, with high speed and accuracy; After the UDP communication module in the 3D content generation and rendering module obtains the coordinates of the human eye position, the position of the virtual camera is modified in real time according to the obtained coordinates of the human eye position, and the camera projection matrix is ​​modified according to the calculated camera offset value to achieve the effect that the virtual scene moves with the human eye; After receiving the 3D virtual reality scene rendered in 3D format by the 3D content synthesis and rendering module S3, the naked eye light field 3D display system simultaneously receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system; When the user is in different viewing positions, the naked-eye 3D display and computer system are processed by algorithms to display the light field 3D virtual reality images corresponding to the human eye position in real time, so that users in different viewing zones can view the content of the corresponding viewing zones. After image algorithm processing and synthesis, the user can finally see the light field 3D effect without wearing any auxiliary equipment.

5. A naked eye light field 3D display method based on human eye tracking, characterized in that: The naked eye light field 3D display method comprises: The human eye tracking module locates and tracks the position of the human eye in real time, and outputs the spatial position coordinates of the human eye to the data processing and transmission module; The data processing and transmission module filters and predicts the eye position coordinates in real time, and transmits the processed data to the 3D content generation and rendering module; The 3D content generation and rendering module generates and renders the 3D content of the corresponding position according to the acquired human eye position coordinates; The naked-eye light field 3D display system receives the data sent by the 3D content generation and rendering module, and displays the 3D scenes at different viewing positions in real time.

6. The naked-eye light field 3D display method based on human eye tracking according to claim 5, characterized in that: The human eye tracking module includes an industrial camera module and an end-to-end human eye tracking algorithm, and collects human eye images through the industrial camera module; The industrial camera module uses a binocular real-sense depth camera to capture and obtain human eye position information; The binocular real-sense depth camera includes two infrared cameras and one RGB camera, which are used to capture human eye RGB images and infrared images in real time; The end-to-end human eye tracking algorithm processes the human eye RGB image and infrared image in real time to locate the human eye position, and finally outputs the viewer's binocular spatial position coordinates by fusing the RGB image with the depth image and converting the coordinate system; The data processing and transmission module filters, predicts and transmits the binocular spatial position information captured by the human eye tracking module; processes the human eye spatial coordinate data within a set time through a filtering algorithm to remove data noise, and estimates and predicts the state of the human eye spatial position in real time, and transmits the human eye position data after algorithm processing through a connectionless transport layer protocol; The 3D content generation and rendering module generates an image for 3D display through a 3D content production tool and outputs it in 3D format, and adjusts the position of the virtual camera in real time in the 3D content production tool according to the acquired human eye position coordinates, and renders and generates a synthetic image of the corresponding orientation; The 3D content generation is to create a virtual scene in a 3D content production tool, add a virtual camera to simulate the human eye, set the virtual camera distance to simulate the pupil distance of the human eye, shoot the virtual scene to generate an image for 3D display, and render it into a 3D format image output; The naked-eye light field 3D display system includes a naked-eye 3D display, a human eye tracking system and a computer system; The eye tracking system obtains the position information of both eyes in real time, and the 3D content synthesis and rendering module renders the 3D virtual reality scene in 3D format, and at the same time receives the eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system; The naked-eye 3D display and computer system are processed by algorithms to display the light field 3D images corresponding to the human eye position in real time, and the user can see the light field 3D effects at any angle without wearing any auxiliary equipment.

7. The naked-eye light field 3D display method based on human eye tracking according to claim 5, characterized in that: The eye tracking module detects the working status of the camera through the camera management module. After the camera is started normally, the binocular real-sense depth camera captures RGB image data and infrared image data with relative parallax in real time; The eye tracking algorithm processes the RGB image data in real time, locates the region of interest (ROI) of the eye, and outputs the image coordinates m(x,y) of the eye; From the acquired infrared disparity image, according to the geometric relationship of parallel binocular vision, the conversion formula between depth D and disparity P is as follows: D=(f*B) / P Among them, f is the normalized camera focal length, B is the distance between the optical centers of the two infrared cameras, and the depth value of the image can be calculated by the above formula; By matching the acquired eye image coordinates with the depth map, after data fusion and coordinate mapping, the eye tracking module outputs the final eye spatial position coordinates M (x, y, z), and transmits the eye spatial position coordinates to the data transmission and processing module through the USB transmission interface; After receiving the eye space coordinate data, the data processing and transmission module uses the Kalman filter algorithm to process the data. The Kalman filter algorithm estimates the state of the system through prediction and update. The prediction stage predicts the state at the current moment based on the state at the previous moment and the input control amount; the update stage corrects the prediction result according to the actual observation value to obtain a more accurate state estimation; The system model is constructed based on the acquired human eye space coordinates. The system model is represented by the state equation and the observation equation. x represents the system vector. The state equation describes the state vector x at the previous moment. k-1 Mapped to the current system state x k ; x k =F k x k-1 +B k u k +w k Among them, F k is the transformation matrix, B k is the control matrix, w k is process noise, with mean 0 and covariance Q k Gaussian distribution, w k ~N(0,Q k ); Z k =H k x k +v k Among them, H k is the observation matrix, v k is the observation noise, with a mean of 0 and a covariance of R k Gaussian distribution, v k ~N(0,R k ); The system model is predicted to generate the predicted value of the human eye spatial position coordinates at the current moment. The human eye spatial position state prediction equation at this time is: in, Indicates the predicted value of the eye coordinates at this time. At this time, w k The value is 0. Since the eye space coordinates are directly output by module S1, the system has no control input, so the control matrix B k is 0; The prediction error covariance matrix at this time is It can be expressed by the following formula: According to the data obtained in the prediction process, the Kalman gain at this time can be calculated to complete the update of the spatial position coordinates of the human eye. At this time, the Kalman gain K k It can be expressed by the following formula: After completing the prediction of the human eye data, the system model state is updated to output the final spatial position coordinates of the human eye after Kalman filtering; the Kalman gain K obtained by the prediction process k , we can get the updated and the updated It is expressed by the following formula: After the Kalman filter algorithm is updated and calculated, the data processing and transmission module outputs the spatial coordinates of the human eye at this time. That is, the spatial position coordinates of the human eye after filtering, denoising and prediction, and the data is transmitted to the 3D content generation and rendering module through the User Datagram Protocol (UDP).

8. The naked-eye light field 3D display method based on human eye tracking according to claim 5, characterized in that: The 3D content generation and rendering module receives the human eye spatial position coordinates after data processing, adjusts the virtual camera matrix in the virtual scene in real time according to the coordinate information, and renders and generates a synthetic image of the corresponding orientation; creates a virtual scene, sets the virtual camera in an off-axis manner, sets the virtual camera spacing to simulate the pupil distance of the human eye, shoots the virtual scene to generate an image for 3D display, and renders it into a 3D format screen output; The virtual camera is set up in an off-axis manner to align the projection planes of multiple virtual cameras through cropping, ensuring that the captured images have a reasonable three-dimensional effect and avoiding the generation of vertical parallax in 3D display; This method can generate multiple image sequences with parallax at one time, and encode the image sequences for light field display synthesis, with high speed and accuracy; After the UDP communication module in the 3D content generation and rendering module obtains the coordinates of the human eye position, the position of the virtual camera is modified in real time according to the obtained coordinates of the human eye position, and the camera projection matrix is ​​modified according to the calculated camera offset value to achieve the effect that the virtual scene moves with the human eye; After receiving the 3D virtual reality scene rendered in 3D format by the 3D content synthesis and rendering module S3, the naked eye light field 3D display system simultaneously receives the human eye position information to adjust the rendering picture, and transmits the image signal to the naked eye 3D display and the computer system; When the user is in different viewing positions, the naked-eye 3D display and computer system are processed by algorithms to display the light field 3D virtual reality images corresponding to the human eye position in real time, so that users in different viewing zones can view the content of the corresponding viewing zones. After image algorithm processing and synthesis, the user can finally see the light field 3D effect without wearing any auxiliary equipment.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 5 to 8 are implemented.

10. A storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 5 to 8 are implemented.

Images