Method and device for applying light field display and human eye detection integrated near-to-eye display

Abstract

The invention belongs to the technical field of near-to-eye display, and discloses an integrated near-to-eye display method and device applying lens array light field display and human eye detection, and the device comprises a VR light machine housing, an environment camera, a display screen, a human eye camera, and a splicing screen gap. The method is low in cost; the lens function is comprehensively utilized, and hardware requirements are reduced; one or more lenses in the lens array are used as an optical element for light field display imaging and an optical element for a human eye camera at the same time, so that multifunctionalization of equipment is realized, the requirement of an additional hardware sensor is reduced, and the hardware cost is reduced. Due to integrated design, installation and maintenance are simplified; the structure is simple; seamless splicing is achieved, and the optical design is optimized; the performance is good; high resolution and large field angle are achieved; the lens array light field display technology can significantly improve the display resolution and expand the field angle, and provides clearer and broader visual experience for users.

Description

Technical Field

[0001] The present invention belongs to the technical field of near-eye display, and in particular relates to a method and device for near-eye display that integrates lens array light field display and human eye detection. Background Art

[0002] Near-eye display devices are a crucial component of virtual reality (VR), augmented reality (AR), and extended reality (XR) technologies. These devices provide users with an immersive visual experience by displaying virtual images directly in front of their eyes. Traditional near-eye display devices rely primarily on a single screen or separate screens for both eyes. While these designs provide immersion, they also face challenges in resolution, field of view (FOV), and image distortion.

[0003] In recent years, lenticular array light field display technology has gradually emerged. This technology generates three-dimensional light field images through multiple microlens arrays, significantly improving display quality and visual experience. Lens array light field display technology can improve display resolution, expand the field of view, and reduce image distortion, allowing users to obtain clear and natural visual effects over a wider range.

[0004] Existing lenticular array light field displays typically use splicing technology to achieve a wide field of view. However, the splicing process often results in imaging discontinuities, which degrades image quality and impacts the user experience. Furthermore, traditional eye tracking is typically independent of the display, resulting in complex and costly installation and limited accuracy, making efficient and accurate eye tracking difficult to achieve in near-eye display devices.

[0005] Through the above analysis, the problems and defects of the existing technology are as follows:

[0006] (1) Imaging discontinuity: In lens array splicing screen technology, due to the gaps between the lenses, the image is prone to discontinuity at the splicing point. This phenomenon will lead to a decrease in image quality, and users may experience visual discomfort when watching, affecting the immersive experience.

[0007] (2) Complexity and cost of eye tracking: Existing eye tracking systems are usually independent of the display, complex to install, and expensive. These require additional sensors and processors to implement eye tracking, which not only increases the hardware cost of the device but also increases the complexity. In addition, the accuracy of traditional eye tracking is limited and it is difficult to meet the needs of high-precision eye detection.

[0008] (3) Limitations of display effects: Traditional near-eye display devices have limitations in terms of resolution and field of view. Although lens array light field display technology can improve display effects, in actual applications, due to the limitations of splicing screen technology, the display effect still needs to be improved. Summary of the Invention

[0009] In response to the problems existing in the prior art, the present invention provides a near-eye display device integrating lens array light field display and human eye detection.

[0010] The present invention is implemented as follows: a method for near-eye display using lens array light field display and human eye detection integration, comprising the following steps:

[0011] Step 1: Splice the display screen;

[0012] Multiple displays are stitched together through precise alignment and calibration to form a large, high-resolution display panel, ensuring that the edge gaps between each display are minimized.

[0013] Step 2: Install the lens array;

[0014] A lens array consisting of multiple optical lenses is installed at a certain distance in front of the spliced screen. The lens array optimizes the focus and projection of light through light field rendering, supporting the projection of images from different perspectives.

[0015] Step 3: Install the image sensor or camera;

[0016] Design appropriate space in the gaps between the spliced screens to install high-resolution image sensors or cameras to capture optical data from the user's eyes, ensuring that the lens can support both light field display and eye detection functions.

[0017] Step 4: Capture and analyze eye images;

[0018] Use an eye-tracking camera or sensor to capture images of the user's eyes. A high-precision image processing unit analyzes the deformation of structured light, infers the three-dimensional shape and movement of the eyes, and fuses this with image data captured by the surrounding camera. This allows for real-time monitoring of eye movement and pupil changes.

[0019] Step 5: Foveated rendering technology;

[0020] The detected eye state data is used to track the user's gaze point to implement foveated rendering technology; the user's eye state is accurately detected through image sensors or cameras;

[0021] Step 6: Environmental image fusion;

[0022] The external environment image is captured by the environmental camera and merged with the virtual image generated by the lens array; the brightness and color of the display are automatically adjusted using ambient light correction technology;

[0023] Step 7: Display the virtual image;

[0024] The virtual image is displayed on the display screen, and the lens array is responsible for light field display, supporting the projection of images from different perspectives. By optimizing the focusing and projection of light, 3D images with depth and parallax are generated.

[0025] Step 8: Multi-sensor fusion technology;

[0026] Utilizes data fusion technology from multiple sensors to combine data from environmental cameras, human eye cameras, and other sensors;

[0027] Step 9: Adaptive display technology;

[0028] Dynamically adjust the resolution and rendering strategy of displayed content based on the user's gaze point and eye movement information to optimize the use of system resources;

[0029] Step 10: User customized settings;

[0030] Provide user interface and configuration options to allow users to adjust display parameters and tracking accuracy according to personal preferences and usage scenarios;

[0031] Step 11: Algorithm optimization;

[0032] Use advanced machine learning and artificial intelligence algorithms to analyze and predict captured eye movement data;

[0033] Step 12: Hardware acceleration;

[0034] Utilize GPU and dedicated hardware accelerators to increase the speed of image processing and rendering;

[0035] Eye cameras or sensors are used to capture eye data, monitor eye movements and pupil changes in real time, provide high-precision eye tracking, and obtain eye movement and pupil size data;

[0036] Accurately detect the user's eye state through image sensors or cameras to achieve gaze point rendering and user interaction;

[0037] The detected eye state data is used to track the user's gaze point and implement foveated rendering technology;

[0038] Capturing the external environment image through the environment camera and fusing the image with the virtual image generated by the lens array;

[0039] The virtual image is displayed through the display screen, and the lens array is responsible for light field display, supporting the projection of images from different perspectives. By optimizing the focusing and projection of light, 3D images with depth and parallax are generated.

[0040] Furthermore, the method for near-eye display using lens array light field display and human eye detection integration includes:

[0041] The eye movement and pupil size detected by the human eye camera are used as user input to control interactive operations in the virtual environment;

[0042] The human eye camera is fixed in the center of the VR optical machine housing to accurately capture the user's eye status through the gaps in the spliced screen.

[0043] Furthermore, the method for near-eye display using lens array light field display and human eye detection integration includes:

[0044] A lens array is formed by using one or more layers of lens groups through an optical machine assembly within a VR optical machine housing;

[0045] The image displayed on the display forms a light field through the lens array, generating a more realistic 3D image and enhancing the user's sense of immersion.

[0046] Furthermore, the method for near-eye display using lens array light field display and human eye detection integration includes:

[0047] The light capture and image pickup part, together with the sub-lenses in the lens array, form a complete camera acquisition device, achieving high-precision capture of environmental images and user eye status;

[0048] Combining lens array light field display technology and human eye detection technology, it improves detection accuracy, optimizes device structure, reduces device weight, and provides a near-eye display experience with high immersion and natural interaction.

[0049] Furthermore, the method for near-eye display using lens array light field display and human eye detection integration includes:

[0050] The user's eye images are captured through a human eye camera, and the eye movement trajectory is analyzed using image processing algorithms;

[0051] Based on the analysis results, the rendering position of the virtual image is adjusted in real time to match the user's gaze point, thereby improving the user experience.

[0052] Furthermore, the method for near-eye display using lens array light field display and human eye detection integration includes:

[0053] The external environment images captured by the environment camera are pre-processed through image processing algorithms to remove noise and enhance details;

[0054] The pre-processed environment image is seamlessly integrated with the virtual image to provide a more realistic augmented reality effect.

[0055] Furthermore, the method for near-eye display using lens array light field display and human eye detection integration includes:

[0056] By adjusting the arrangement and focal length of each sub-lens in the lens array, the light field display effect is optimized;

[0057] Deep learning algorithms are used to intelligently analyze the user's eye status and environmental images, thereby dynamically adjusting the display content and improving the intelligence and adaptability of near-eye display devices.

[0058] Another object of the present invention is to provide a near-eye display device integrating lens array light field display and human eye detection, comprising:

[0059] VR optical machine housing, environmental camera, display screen, human eye camera, and splicing screen gaps;

[0060] An environmental camera is fixed to the right side of the VR optical machine housing by screws; a display screen is fixed to the left side of the VR optical machine housing by screws; a splicing screen gap is set in the center of the left side of the VR optical machine housing; a human eye camera is fixed to the splicing screen gap 5 by screws;

[0061] The human eye camera is used to detect the state of the human eye.

[0062] Furthermore, the environment camera is used to capture the environment and fuse the virtual image formed by the lens array;

[0063] Furthermore, inside the VR optical machine housing is a display screen and an optical machine assembly, and the optical machine assembly includes one or more layers of lens groups.

[0064] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the application method of the near-eye display device integrating lens array light field display and human eye detection.

[0065] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the application method of the near-eye display device integrating lens array light field display and human eye detection.

[0066] Another object of the present invention is to provide an information data processing terminal, which is used to realize the near-eye display device integrating lens array light field display and human eye detection.

[0067] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0068] First, the present invention proposes a novel near-eye display device that integrates lens array light field display and human eye detection, specifically solving the following technical problems:

[0069] Eliminate imaging discontinuities: By precisely calibrating the imaging area of the lens array, seamless stitching is achieved, eliminating imaging discontinuities and improving image quality and visual experience.

[0070] Integrated eye tracking: Integrate the eye camera into the gap of the lens array, and use one or more lenses in the lens array as optical elements to achieve an integrated design of eye detection and display, reducing hardware cost and complexity.

[0071] Improved display effects: By combining lens array light field display technology with human eye detection, the display resolution and field of view are improved, high-precision eye tracking is achieved, and a better immersive experience is provided.

[0072] This invention proposes a novel near-eye display device that integrates lens array light field display and eye detection. By utilizing splicing screen technology and innovative eye tracking, it significantly improves display quality and user experience. This invention addresses the issues of discontinuous imaging and low eye tracking accuracy during the splicing process, achieving a seamless integration of lens array light field display and eye detection.

[0073] 1. Splicing screen and lens array technology:

[0074] Splicing screen technology: By splicing multiple screens together to form a large display panel, however, the splicing process often causes imaging discontinuity.

[0075] Lens array: Used to realize light field display on the spliced screen. This array can present images at different viewing angles and provide an immersive visual experience.

[0076] 2. New eye tracking technology:

[0077] Human eye camera in the gap: One or more lenses are installed in the gap of the spliced screen. These lenses are not only used for light field display, but also serve as optical elements for eye tracking.

[0078] High-precision detection: This design allows for high-precision eye detection using only a minimal number of eye cameras or sensors.

[0079] Definition of integrated structure:

[0080] Lightfield display and eye detection integration: Lens arrays play two main roles in the lightfield display. First, they serve as the optical element for the lightfield display, generating immersive images. Second, they also serve as the optical element for the eye detection sensor, enabling eye tracking to be tightly integrated with the lightfield display.

[0081] Functional fusion: The combination of lens array and splicing screen achieves the perfect fusion of display and detection functions by placing eye detection sensors in the gaps.

[0082] Seamless integration of imaging and detection:

[0083] Reduced hardware requirements: This integrated design reduces additional hardware requirements and simplifies the structure, which not only reduces complexity but also improves overall performance.

[0084] Improve user experience: The integrated design can reduce the physical volume while providing higher display resolution and tracking accuracy, optimizing the user experience.

[0085] Technical Implementation

[0086] Processing of gaps in spliced screens: A structure suitable for placing lenses and sensors in the gaps of spliced screens is designed to ensure imaging continuity and detection accuracy.

[0087] The lens plays a dual role: it plays a role in both light field display and eye tracking, ensuring the effective combination of these two functions.

[0088] This design not only solves the imaging problem of traditional splicing screens, but also improves functionality and user experience through the integration of lenses and sensors.

[0089] The core advantage of integrated design.

[0090] 1. Functional integration

[0091] Improvements:

[0092] Lens arrays are not only used for light field displays, but also as optical elements for eye-tracking cameras or sensors.

[0093] Advantages:

[0094] Multi-Function Integration: The lens array plays a dual role in light field display and eye tracking, allowing a single system to accomplish both tasks simultaneously. This integrated design optimizes the device's functional combination and improves overall system efficiency.

[0095] 2. Reduce hardware requirements

[0096] Improvements:

[0097] By integrating eye-tracking cameras or sensors at the gaps in the lens array.

[0098] Advantages:

[0099] Reduced system complexity: Traditional systems typically require separate display and eye tracking devices, while the integrated design reduces the required hardware components, thereby simplifying the system structure.

[0100] Reduce costs: Reducing the amount of hardware and simplifying the design reduce production and maintenance costs, making the system more economical and practical.

[0101] 3. Optimize space utilization

[0102] Improvements:

[0103] Reasonably arrange lenses and sensors in the gaps of the spliced screen.

[0104] Advantages:

[0105] Compact design: The integration of lenses and cameras or sensors enables the system to implement light field display and eye tracking functions in a limited space, thereby improving space utilization efficiency.

[0106] 4. Improve display and tracking accuracy

[0107] Improvements:

[0108] The lens array simultaneously enhances light field display effects and eye tracking accuracy by optimizing the optical path.

[0109] Advantages:

[0110] Precise visual experience: Through integrated design, the lens can display images more accurately and effectively track eye position and movement, providing high-quality visual effects and user interaction experience.

[0111] 5. Simplify production and maintenance

[0112] Improvements:

[0113] By reducing independent components and integrating functionality.

[0114] Advantages:

[0115] Improved production quality: The integrated design reduces the complexity of the assembly and debugging process, thereby improving production quality and reliability.

[0116] Simplified maintenance: The integrated design makes system maintenance and upgrades easier, reducing long-term maintenance costs.

[0117] The integrated design of a lenticular array near-eye display and an eye-tracking camera or sensor overcomes multiple drawbacks of traditional designs by integrating display and eye-tracking functionality into a single system. This not only simplifies hardware structure and reduces costs, but also improves overall system performance and user experience. This design effectively combines the advanced technology of light-field displays with the requirements of high-precision eye tracking, representing a significant innovation in near-eye display technology.

[0118] The novel near-eye display device integrating lens array light field display and human eye detection created by this invention not only addresses the defects and shortcomings of existing technologies but also brings several significant advantages. These advantages are specifically manifested in the following aspects:

[0119] 1. Low cost

[0120] Comprehensively utilize lens functions to reduce hardware requirements

[0121] By using one or more lenses in the lens array as both optical elements for light field display imaging and optical elements for the human eye camera, the device is made multifunctional, reducing the need for additional hardware sensors and thus reducing hardware costs.

[0122] Integrated design simplifies installation and maintenance

[0123] The human eye camera is integrated into the gap of the splicing screen to form an integrated near-eye display device, which reduces the complexity of the equipment and reduces the installation and maintenance costs.

[0124] 2. Simple structure

[0125] Seamless splicing, optimized optical design

[0126] The present invention eliminates the phenomenon of imaging discontinuity during the splicing process by accurately calibrating the imaging area of each lens, making the entire lens array system more concise and efficient.

[0127] Reduce the number of components and simplify the system structure

[0128] The integrated human eye detection system and lens array light field display system reduces the number of independent components, simplifies the structure of the entire system, and makes the device more compact and lightweight.

[0129] 3. Good performance

[0130] High resolution and wide field of view

[0131] Lens array light field display technology can significantly improve display resolution and expand the field of view, providing users with a clearer and wider visual experience.

[0132] High-precision eye tracking

[0133] By combining a human eye camera with a lens array, the present invention achieves high-precision human eye tracking, can accurately detect the user's eye movement in real time, and improves interactivity and user experience.

[0134] Seamless image display

[0135] Through precise splicing calibration, seamless image display of the lens array is ensured, eliminating the imaging discontinuity in traditional splicing screen technology and providing consistent and clear visual effects.

[0136] 4. Easy to use

[0137] Integrated design, easy to wear and use

[0138] The integrated design of the present invention reduces the number of independent components, making the device easier to wear and use, and improving user convenience.

[0139] Real-time correction and feedback

[0140] The eye tracking system can analyze and provide feedback on the user's eye position and movement trajectory in real time, allowing the device to quickly respond to changes in the user's line of sight, improving flexibility and interactivity.

[0141] 5. High production quality

[0142] Optimize production processes and improve equipment reliability

[0143] By reducing independent components and simplifying the system structure, the present invention optimizes the production process, improves the reliability and consistency of the equipment, and ensures high-quality product output.

[0144] High reliability, low failure rate

[0145] The integrated design and high-precision manufacturing process make the equipment more stable during use, with a low failure rate, ensuring long-term reliable performance.

[0146] Advantages brought by specific improvements

[0147] 1. Lens array splicing screen technology: Precisely calibrate the imaging area of each lens to achieve seamless splicing, eliminate imaging discontinuity, and improve image quality and visual experience.

[0148] 2. Integrated eye tracking system: By integrating the eye camera in the gap of the splicing screen, high-precision eye tracking is achieved with minimal sensors, reducing hardware costs and system complexity.

[0149] 3. Dual-function lens: The lens serves as both an optical element for light field display imaging and an optical element for the human eye camera, reducing independent components, simplifying the system structure, and improving overall performance.

[0150] Through the above-mentioned specific improvements, the present invention effectively solves the imaging discontinuity phenomenon, complex human eye tracking system and display effect limitations in the prior art, and achieves multiple advantages of low cost, simple structure, good performance, easy use and high production quality, providing an innovative solution for the development of near-eye display devices.

[0151] Second, the inventive technical solution of the present invention and its advantages;

[0152] 1) Innovation in integrated design

[0153] Integrated Light Field Display and Eye Tracking: This invention integrates a lens array light field display and an eye tracking sensor into a single system. By combining the lens array and eye detection sensor in the gaps of the video wall, this not only simplifies the system structure but also improves the overall performance and cost-effectiveness of the device. This innovative integrated design solves the separation problem in traditional systems and achieves multifunctional integration. This design optimizes the device's functional combination and improves the efficiency and performance of the entire system.

[0154] 2) Seamless Splicing Technology: By precisely calibrating the imaging area of each lens, this invention achieves a completely seamless light field display, eliminating the imaging discontinuities found in traditional splicing screen technology. This innovation addresses a key challenge in display technology and provides users with a higher-quality visual experience.

[0155] 3) High-Precision Eye Tracking: This invention combines a high-precision human eye camera with a lens array to improve the accuracy and stability of eye tracking by optimizing the optical path and image capture. This technological breakthrough, previously unavailable in the market, fills a gap in the field of high-precision eye tracking systems.

[0156] Technological advantages

[0157] (1) Improve display effects and user experience

[0158] Clear images and wide viewing angles: By integrating lens arrays and splicing screen technology, it provides high-resolution, seamless light field display effects, significantly improving the user's visual experience.

[0159] High-precision eye tracking: Combining high-precision human eye cameras and optical correction technology to achieve high-precision eye tracking, enhancing the interactivity and immersion of VR, AR, and XR devices.

[0160] (2) Reduce production and maintenance costs

[0161] Integrated design: The integrated design of the lens array and eye detection sensor reduces the number of hardware components and system complexity, reduces the physical size and complexity of the equipment, and reduces production costs and maintenance difficulty.

[0162] High reliability and stability: Through optimized manufacturing and assembly processes, the high reliability and stability of the system are ensured, reducing failure rate and maintenance costs.

[0163] (3) Expanded application scenarios

[0164] Multi-field applications: The technical solution of the present invention can be widely used in VR, AR and XR devices, as well as other scenarios requiring high-precision display and eye tracking, such as medical imaging, education and training, and military simulation.

[0165] The technical solution of the present invention uses innovative integrated design, seamless splicing technology and high-precision eye tracking. Its unique integration method and optimized design not only solve the key problems in traditional technologies, but also provide a new direction for the development of related technical fields.

[0166] Second, as auxiliary evidence for the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0167] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0168] 1) Market competitive advantage

[0169] Technological leadership: This invention has significant advantages in display effects and eye tracking through an innovative integrated design of lens array light field display and human eye detection, which can attract more users and partners.

[0170] Brand enhancement: Devices using the technology of this invention can provide better user experience and performance, helping to enhance brand image and market awareness.

[0171] 2) New product development

[0172] Diversified product lines: Based on the technology of this invention, a series of near-eye display devices of different types and application scenarios can be developed to meet the needs of different users and expand market share.

[0173] Personalized customization: Through customized design and production, we can meet the special needs of specific industries or customers and provide differentiated products and services.

[0174] 3) Enhance user stickiness

[0175] High-quality user experience: High-resolution seamless splicing display and high-precision eye tracking technology can provide an excellent user experience, enhance user stickiness and satisfaction.

[0176] Rich interactive functions: The application of eye tracking technology can achieve more diverse interaction methods and enhance the interactivity and immersion between users and devices.

[0177] 4) Long-term development potential

[0178] Technology iteration and upgrade: By continuously optimizing and iterating the technical solutions of the present invention, we will continuously improve device performance and user experience and maintain our technological leadership.

[0179] Ecosystem construction: Based on the technology of this invention, a complete software and hardware ecosystem will be built to attract more developers and partners to jointly promote the development of the VR, AR and XR industries.

[0180] The technical solution of the present invention can not only significantly improve the display effect and user experience, but also has the advantages of higher production and maintenance costs, a wide range of application scenarios and market potential, and is expected to bring significant economic benefits and commercial value.

[0181] (2) Whether the technical solution of the present invention solves the technical problems that people have been eager to solve but have not been able to solve successfully:

[0182] 1) Imaging discontinuity: Traditional splicing screen technology suffers from imaging discontinuity in the splicing area, affecting image quality and user experience. Even in high-resolution display technology, this problem still exists and no effective solution has been found.

[0183] The solution of the present invention:

[0184] Precise Calibration: This invention achieves seamless light field display by precisely calibrating the imaging area of each lens, eliminating imaging discontinuities in spliced screens. This innovation significantly improves image quality and solves a long-standing imaging problem in spliced screens.

[0185] 2) Insufficient tracking accuracy: Existing eye tracking systems have limitations in terms of high precision and real-time response, especially in VR, AR and XR applications, and it is difficult to provide sufficiently accurate and stable eye tracking.

[0186] The solution of the present invention:

[0187] Optimizing the optical path: This invention combines a high-precision human eye camera with a lens array to improve the accuracy and stability of eye tracking by optimizing the optical path and image capture. This technological breakthrough, previously unattainable, fills a gap in high-precision eye tracking.

[0188] The technical solution of this invention not only solves a long-awaited technical problem but also achieves breakthroughs in multiple areas. Through integrated design, seamless splicing, high-precision tracking, and cost optimization, this invention fills a gap in light field display and eye tracking technology, providing an innovative solution for the development of related technologies.

[0189] (4) The technical solution of the present invention overcomes technical prejudice:

[0190] Biases in High-Precision Eye Tracking

[0191] Technical bias: When it comes to high-precision eye tracking, some believe it requires complex optical design and a large number of sensors, making it difficult and costly to implement.

[0192] Breakthrough of the present invention:

[0193] Optimizing the optical path: This invention improves the accuracy and stability of eye tracking by optimizing the optical path and utilizing an integrated eye detection system. This innovation demonstrates that high-precision eye tracking is not only achievable but also achieves breakthroughs in cost control and system complexity.

[0194] Lens array light field display and human eye detection integrated near-eye display device:

[0195] 1. System Integration

[0196] Integrate light field display technology and eye detection technology into a unified system. Specifically, this includes:

[0197] Light field display: The lens array is responsible for light field display, which generates 3D images with depth and parallax by optimizing the focusing and projection of light.

[0198] Eye detection: Eye cameras or sensors are used to capture optical data of the eyes, monitor eye movements and pupil changes in real time, and provide high-precision eye tracking.

[0199] This integration reduces the physical separation between components in the system, allowing light field display and eye detection to work seamlessly together within the same device.

[0200] 2. Functional integration

[0201] The all-in-one unit combines two different functions into one system:

[0202] Light field display function: supports the projection of images from different perspectives through a lens array to improve the display effect.

[0203] Eye detection function: Accurately detect the user's eye state through image sensors or cameras to achieve gaze point rendering and user interaction.

[0204] This not only improves the functionality of the system, but also optimizes the convenience of operation and user experience.

[0205] 3. Space optimization

[0206] In design, integration also means efficient use of space:

[0207] Gap Design: Lens arrays and eye detection sensors are installed in the gaps between the spliced screens to ensure maximum functional integration in the smallest space.

[0208] Compact structure: Integrating multiple functional components into one device reduces the physical volume of the system and reduces the complexity and cost of the equipment.

[0209] 4. Performance Improvement

[0210] Integrated technologies help improve overall system performance:

[0211] Consistency and coordination: Integrating light field display and eye detection allows for a more coordinated collaboration between the two functions, improving the system’s responsiveness and accuracy.

[0212] Simplified maintenance: The integrated design simplifies the system maintenance and upgrade process because all functional modules are in one system, reducing interface and connection issues.

[0213] 5. Cost-effectiveness

[0214] Cost-effectiveness can be achieved through integrated design:

[0215] Reduced hardware costs: Consolidating multiple functions into a single system reduces the number of components that need to be manufactured and maintained separately.

[0216] Simplified production and assembly: The integrated design simplifies the production and assembly process, improves production efficiency and reduces costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0217] Figure 1 This is a flow chart of a method for near-eye display using integrated lens array light field display and human eye detection, provided by an embodiment of the present invention.

[0218] Figure 2 This is a structural diagram of an integrated near-eye display device provided by an embodiment of the present invention.

[0219] Figure 3 This is a structural diagram of the internal components of an integrated near-eye display device provided by an embodiment of the present invention.

[0220] Figure 4 This is a structural diagram of a heterogeneous integrated near-eye display device provided by an embodiment of the present invention.

[0221] Figure 5 This is a diagram of the internal structure of the human eye camera of the integrated near-eye display device provided by an embodiment of the present invention.

[0222] Figure 6 This is a diagram of the internal structure of the human eye camera of the integrated near-eye display device provided by an embodiment of the present invention.

[0223] Figure 7 This is a diagram of the internal structure of the human eye camera of the integrated near-eye display device provided by an embodiment of the present invention.

[0224] Figure 8 This is a simplified structural diagram of an integrated near-eye display device provided by an embodiment of the present invention.

[0225] Figure 8 Middle: 1. VR optical machine housing; 2. Environmental camera; 3. Display screen; 4. Human eye camera; 5. Gap between spliced screens. DETAILED DESCRIPTION

[0226] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0227] Structured light eye tracking is a high-tech method that tracks eye movements by projecting a specific structured light pattern onto the eye and analyzing changes in the reflected light pattern. This technology consists of seven core components: a light source, projection optics, camera, image processing unit, control and interface unit, power supply and cooling system, and housing and bracket. These components work together to ensure accurate and stable eye tracking.

[0228] In practice, a light source projects a specific structured light pattern onto the eye through a projection optical system. The light pattern reflected from the eye's surface is then captured by a high-precision camera. Due to the shape and movement of the eye, the reflected light pattern is distorted accordingly.

[0229] The image processing unit is responsible for in-depth analysis of the captured image. By comparing the actual image with the original structured light pattern, the unit can accurately calculate the three-dimensional shape and movement trajectory of the eye.

[0230] Furthermore, the system can infer the actual position and movement of the eyes through image processing. This data has wide-ranging applications in user interface control, virtual reality, and augmented reality.

[0231] When structured light eye tracking technology is combined with a lenticular array video wall, its performance is further enhanced. The sub-lenses in the lenticular array play a crucial role in the imaging process. By optimizing the focus and capture of light, they significantly improve the clarity of eye features, thereby enhancing tracking accuracy and stability.

[0232] Structured light eye tracking technology, with its high precision and stability, has demonstrated tremendous potential for application in a variety of fields. As the technology continues to advance, it is expected to play an even more important role in future human-computer interaction.

[0233] Application Examples

[0234] In a near-eye display device integrating lens array light field display and human eye detection, the integration is achieved in the following aspects:

[0235] Inside the system: In the gaps between the spliced screens, a lens array and eye detection sensor are cleverly integrated, allowing the two functional modules to operate in the same location.

[0236] Functional coordination: Through the collaboration of light field display and eye detection, it provides higher-quality visual experience and user interaction, supporting applications in virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies.

[0237] In general, the concept of integration improves equipment performance, optimizes space utilization, reduces costs, and simplifies production and maintenance processes by integrating multiple functions and components into one system.

[0238] Example 1: Smart Glasses

[0239] 1. Application Scenario

[0240] Smart glasses integrate an integrated near-eye display device and are used for augmented reality (AR) experience in daily life, such as real-time information display, navigation, social interaction, etc.

[0241] 2. Implementation steps

[0242] Launching glasses: The user wears smart glasses and launches corresponding applications, such as navigation, real-time translation, or social interaction.

[0243] Environmental Perception: The environmental camera captures images of the surrounding environment, while the human eye camera performs eye tracking to understand the user's line of sight and gaze point.

[0244] Light Field Display: The lens array creates a light field display using virtual information on the display, projecting navigation instructions, notifications, or translation results directly into the user's field of view, enhancing the visualization of information.

[0245] Real-time interaction: Based on eye tracking data, the system can adjust the display position of virtual information in real time to ensure that the information is always within the user's line of sight. For example, navigation instructions will dynamically update based on the user's gaze direction.

[0246] Application functions: Users can operate smart glasses through gaze control, for example, by looking at a specific icon to launch an application, view messages, or adjust settings. Images from the ambient camera can also be used for augmented reality social functions, such as sharing views in real time or participating in virtual meetings.

[0247] Example 2: Medical auxiliary equipment

[0248] 1. Application Scenario

[0249] In the medical field, especially in ophthalmic surgery or vision testing, medical auxiliary equipment that integrates lens array light field display and human eye detection technology is used to provide accurate visual testing and surgical guidance.

[0250] 2. Implementation steps

[0251] Equipment preparation: The patient wears the medical assistive device, and the device enters the vision test or surgical guidance mode after startup.

[0252] Structured light projection: The device uses a light source and projection optics system to project a structured light pattern onto the patient's eye, allowing for precise measurement of the eye's shape and reflected light pattern.

[0253] Eye tracking: The human eye camera in the device captures the movement and status of the eye in real time, and the image processing unit analyzes the deformation of the structured light to generate three-dimensional shape data of the eye.

[0254] Light field display: The lens array projects the vision test pattern or surgical guide on the display screen into the patient's eyes through light field display technology, providing high-precision visual presentation.

[0255] Dynamic Adjustment: During vision tests, the system automatically adjusts the position and size of the test pattern based on eye movement and response for accurate vision assessment. During ophthalmic surgery, the system provides real-time surgical guidance information to help doctors perform precise operations.

[0256] Data Analysis and Feedback: The system collects and analyzes test data, providing detailed vision reports or surgical guidance. The device can record the eye's status and response, providing reference data for doctors or medical staff.

[0257] These two embodiments demonstrate the application of a near-eye display device integrating lens array light field display and human eye detection in smart glasses and medical auxiliary equipment, providing users with augmented reality experience and precise medical guidance, respectively.

[0258] like Figure 8 As shown, an embodiment of the present invention provides a near-eye display device integrating lens array light field display and human eye detection, comprising:

[0259] VR optical machine housing 1, environmental camera 2, display screen 3, human eye camera 4, splicing screen gap 5.

[0260] An environmental camera 2 is fixed to the right side of the VR optical machine housing 1 by screws; a display screen 3 is fixed to the left side of the VR optical machine housing 1 by screws; a splicing screen gap 5 is provided in the center of the left side of the VR optical machine housing 1; a human eye camera 4 is fixed to the splicing screen gap 5 by screws.

[0261] The human eye camera provided by the embodiment of the present invention is used to detect the state of the human eye.

[0262] The environmental camera provided in the embodiment of the present invention is used to fuse the captured environment with the virtual image formed by the lens array;

[0263] The VR optical machine housing provided by the embodiment of the present invention contains a display screen and an optical machine assembly, and the optical machine assembly includes one or more layers of lens groups.

[0264] An embodiment of the present invention provides a near-eye display device that integrates lens array light field display and human eye detection. The detailed working principle of the device is as follows:

[0265] 1) Overall structure:

[0266] The VR optical machine housing 1 serves as the protection and support structure of the entire device, and houses a display screen 3, an optical machine assembly (including one or more layers of lens assembly), an environmental camera 2 and a human eye camera 4.

[0267] The environment camera 2 is fixed on the right side of the housing and is used to capture external environment images.

[0268] The display screen 3 is located on the left side of the housing and is responsible for displaying virtual images.

[0269] The human eye camera 4 is fixed in the center of the housing through the gap 5 of the spliced screen and is used to detect the user's eye status.

[0270] 2) Human eye detection:

[0271] The human eye camera 4 can analyze the user's eye movement, pupil size and other conditions in real time by capturing images of the user's eyes.

[0272] This data can be used to track the user's gaze point, thereby achieving more accurate interaction and image rendering, such as implementing foveated rendering technology to improve rendering efficiency and image quality.

[0273] At the same time, human eye state can also be used as a form of user input to control interactions in virtual environments.

[0274] 3) Environmental integration:

[0275] The environment camera 2 captures images of the external environment.

[0276] These image data are fused with the virtual image formed by the lens array to achieve augmented reality (AR) effects.

[0277] In this way, users can see virtual images while also perceiving the real environment, achieving a more natural interactive experience.

[0278] 4) Lens array and light field display:

[0279] The optical machine assembly inside the VR optical machine housing includes one or more layers of lens groups, which form a lens array.

[0280] The image displayed on the display screen 3 forms a light field through the lens array, generating a 3D image with depth and parallax.

[0281] This light field display technology can provide more realistic 3D effects and enhance the user's sense of immersion.

[0282] 5) Splicing screen gap:

[0283] The gap 5 of the spliced screen is designed to meet the installation requirements of the human eye camera 4.

[0284] Through this gap, the human eye camera 4 can accurately capture the user's eye status without being blocked by the display screen 3 or other components.

[0285] In summary, the embodiments of the present invention integrate lens array light field display technology with human eye detection technology to provide a near-eye display device with high immersion, natural interaction, and efficient rendering. This device has broad application prospects in the fields of virtual reality (VR) and augmented reality (AR).

[0286] like Figure 1 As shown, an embodiment of the present invention provides an application method of a near-eye display device integrating lens array light field display and human eye detection, including:

[0287] S101: spliced display screen;

[0288] Multiple displays are stitched together through precise alignment and calibration to form a large, high-resolution display panel, ensuring that the edge gaps between each display are minimized.

[0289] S102: Installing the lens array;

[0290] A lens array consisting of multiple optical lenses is installed at a certain distance in front of the spliced screen. The lens array optimizes the focus and projection of light through light field rendering, supporting the projection of images from different perspectives.

[0291] S103: Installing an image sensor or camera;

[0292] Design appropriate space in the gaps between the spliced screens to install high-resolution image sensors or cameras to capture optical data from the user's eyes, ensuring that the lens can support both light field display and eye detection functions.

[0293] S104: Capturing and analyzing eye images;

[0294] Use an eye-tracking camera or sensor to capture images of the user's eyes. A high-precision image processing unit analyzes the deformation of structured light, infers the three-dimensional shape and movement of the eyes, and fuses this with image data captured by the surrounding camera. This allows for real-time monitoring of eye movement and pupil changes.

[0295] S105: Foveated rendering technology;

[0296] The detected eye state data is used to track the user's gaze point to implement foveated rendering technology; the user's eye state is accurately detected through image sensors or cameras;

[0297] S106: Environmental image fusion;

[0298] The external environment image is captured by the environmental camera and merged with the virtual image generated by the lens array; the brightness and color of the display are automatically adjusted using ambient light correction technology;

[0299] S107: Displaying a virtual image;

[0300] The virtual image is displayed on the display screen, and the lens array is responsible for light field display, supporting the projection of images from different perspectives. By optimizing the focusing and projection of light, 3D images with depth and parallax are generated.

[0301] S108: Multi-sensor fusion technology;

[0302] Utilizes data fusion technology from multiple sensors to combine data from environmental cameras, human eye cameras, and other sensors;

[0303] S109: Adaptive display technology;

[0304] Dynamically adjust the resolution and rendering strategy of displayed content based on the user's gaze point and eye movement information to optimize the use of system resources;

[0305] S110: User customized settings;

[0306] Provide user interface and configuration options to allow users to adjust display parameters and tracking accuracy according to personal preferences and usage scenarios;

[0307] S111: Algorithm optimization;

[0308] Use advanced machine learning and artificial intelligence algorithms to analyze and predict captured eye movement data;

[0309] S112: hardware acceleration;

[0310] Utilize GPU and dedicated hardware accelerators to increase the speed of image processing and rendering;

[0311] Eye cameras or sensors are used to capture eye data, monitor eye movements and pupil changes in real time, provide high-precision eye tracking, and obtain eye movement and pupil size data;

[0312] Accurately detect the user's eye state through image sensors or cameras to achieve gaze point rendering and user interaction;

[0313] The detected eye state data is used to track the user's gaze point and implement foveated rendering technology;

[0314] Capturing the external environment image through the environment camera and fusing the image with the virtual image generated by the lens array;

[0315] The virtual image is displayed through the display screen, and the lens array is responsible for light field display, supporting the projection of images from different perspectives. By optimizing the focusing and projection of light, 3D images with depth and parallax are generated.

[0316] Another object of the present invention is to provide a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the application method of the near-eye display device integrating lens array light field display and human eye detection.

[0317] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the application method of the near-eye display device integrating lens array light field display and human eye detection.

[0318] Another object of the present invention is to provide an information data processing terminal, which is used to realize the near-eye display device integrating lens array light field display and human eye detection.

[0319] There are many types of eye tracking devices, which are mainly classified according to their working principles, application scenarios and accuracy requirements. The following are common types of eye tracking devices and their characteristics:

[0320] 1. Infrared light-based eye tracking device

[0321] An infrared light source is used to illuminate the eye, and the infrared light is reflected on the surface of the eye (such as the cornea and pupil). The changes in the reflected light are captured by a camera or sensor to infer the movement and position of the eye.

[0322] type:

[0323] Eye-tracking cameras: Specially designed to capture infrared images of the eyes, commonly used in virtual reality (VR) and augmented reality (AR) systems.

[0324] Eye-tracking glasses: Glasses with integrated infrared light sources and sensors that can be used for various eye-tracking applications such as psychology research and user behavior analysis.

[0325] High precision: Able to provide high-precision eye position data.

[0326] Strong adaptability: can work in different lighting conditions, especially performs well in low-light environments.

[0327] 2. Vision-based eye tracking devices

[0328] It relies on an ordinary visible light camera to capture images of the eyes and uses image processing algorithms to analyze eye movements.

[0329] Monocular camera system: Use a camera to capture eye images and use image processing algorithms to determine the position and movement of the eyes.

[0330] Binocular camera system: Uses two cameras to capture eye images from different angles, improving accuracy through stereo vision technology.

[0331] Lower cost: Usually cheaper than infrared systems, but may be less accurate in environments with large lighting variations.

[0332] Wide applicability: can be integrated into various devices, such as computers, smartphones, etc.

[0333] 3. Optical-based eye tracking devices

[0334] How it works: It combines optical lenses and sensors to track eye movements by capturing the optical characteristics of the eye, such as pupil position and corneal reflection.

[0335] Pupil tracker: Focuses on tracking pupil position and is commonly used in medical and assistive technology.

[0336] Corneal reflection tracker: It determines the position of the eye by analyzing corneal reflection and is widely used in human-computer interaction systems.

[0337] High Precision: Capable of providing very accurate eye tracking data.

[0338] Optical adaptability: The system can be adjusted based on optical properties to suit different tracking needs.

[0339] 4. Laser-based eye tracking devices

[0340] The eye is scanned with a laser beam, and the reflection and refraction of the laser beam are analyzed to determine the position and movement of the eye.

[0341] Laser scanners: Used in applications requiring high precision, such as flight simulators and high-end VR systems.

[0342] Laser eye tracking device: An eye tracking device that integrates laser technology and can be used for high-precision eye movement research.

[0343] Ultra-high precision: Capable of providing extremely precise eye movement data.

[0344] Complex application: Usually used in high-demand application scenarios, with high system costs.

[0345] 5. Electrophysiology-based eye tracking devices

[0346] Eye position is tracked by measuring changes in electrical signals caused by eye movements.

[0347] Electrooculogram (EOG) devices measure electrooculogram (EOG) signals through electrodes attached around the eyes, and are used for research on eye movements and medical applications.

[0348] Suitable for medical use: Commonly used in medical research and diagnosis of eye movement disorders.

[0349] Complex signal processing: complex signal processing and analysis techniques are required.

[0350] 6. Integrated eye tracking device

[0351] Integrate eye tracking technology into other devices such as monitors, head-mounted displays (HMDs), and smartphones.

[0352] VR / AR headsets: Headsets with built-in eye tracking capabilities that provide an immersive experience.

[0353] Smart Displays: Displays with built-in eye-tracking capabilities for enhanced human-computer interaction.

[0354] High integration: Seamlessly integrate eye tracking functions with other devices to provide a convenient user experience.

[0355] Multifunctional: In addition to eye tracking, other interactive features may be included, such as gesture recognition.

[0356] Each of these eye tracking devices has advantages and disadvantages, and choosing the right one often depends on the specific application needs, accuracy requirements, and budget constraints.

[0357] 7. Structured Light Eye Tracking

[0358] Structured light is an eye-tracking technology that uses known optical patterns (usually lasers or projectors) to scan and capture the 3D shape and movement of the eye.

[0359] Figure 2 Structural diagram of the integrated near-eye display device.

[0360] The above figure shows the structure of the lens array near-eye light field display and the human eye camera. The human eye camera is used to detect the state of the human eye. The environmental camera is used to capture the environment and fuse the virtual image formed by the lens array (this is the implementation process of MR). Inside the VR optical machine housing is the display screen and the optical machine group, which includes one or more layers of lens groups (see Figure 8 and 3 ), but not limited to the two combinations mentioned above. Placing the eye camera in the gap between the spliced screens, facing the human eye position, can achieve more accurate eye detection with a minimum number of cameras (depending on the actual application, the eye camera can also be placed away from the human eye).

[0361] Figure 8 and 3 The internal components of the optomechanical housing are shown, including the light capture and image pickup components, which, along with the sub-lenses in the lens array, form a complete camera acquisition device. The image pickup component is not limited to the examples shown. All types of eye tracking devices described above can be applied here, but they are not limited to the examples shown.

[0362] Figure 4 A diagram of the structure of a heterogeneous integrated near-eye display device, where the light capture and image pickup components also appear in the gap between the spliced screens. To prevent the light capture and image pickup components from blocking the pixel light on the display, they are generally placed in alignment with the screen. Generally, the light capture and image pickup components cannot be closer to the optical engine (the optical engine refers to the optical component group, in this case, the lens array or a combination of the lens array and other lenses) than the display.

[0363] Figure 5 The internal structure of the human eye camera of the integrated near-eye display device, where the eyepiece is a sub-lens from the lens array. The rest is the light capture and image pickup part. Figure 6 The internal structure of the human eye camera of the integrated near-eye display device is shown in the diagram.

[0364] Figure 7 Diagram of the internal structure of the human eye camera in an integrated near-eye display device. The red arrow represents air, which allows pixel light from the display to pass through the optical engine and enter the human eye without obstructing vision.

[0365] The specific implementation of the near-eye display device integrating lens array light field display and human eye detection can be carried out according to the following steps and design scheme:

[0366] The lens array layout is mounted on the surface of the video wall. It consists of multiple optical lenses arranged in a specific structure to support light field display. Each lens is responsible for projecting images from different perspectives on the display into the user's eyes, creating a light field display effect. The video wall is composed of multiple displays joined together with tiny gaps to form a large display panel. The edges of each screen are precisely aligned and calibrated to minimize visual interference caused by the gaps.

[0367] Appropriate spaces are designed in the gaps between the tiled screens to accommodate lenses and eye detection cameras or sensors. The lenses are designed to support both light field display and eye detection. Image sensors are installed in each gap, and these sensors are combined with lenses to capture optical information from the eye. The lens array is designed to optimize the light field display effect, including improving viewing angle and image clarity. It also serves as an optical element for eye tracking, transmitting reflected or refracted optical eye data to the sensor for analysis.

[0368] Integrating the lens array and sensor at the gaps in the tiled screen makes the lens and sensor a single unit, simplifying the system architecture. This design reduces the physical size of the device while improving overall system performance. By precisely positioning the lens and optimizing the optical path to ensure the continuity of the light field display and the accuracy of the eye tracking data, the lens is calibrated to ensure optimal performance in both light field display and eye tracking.

[0369] During the production process, the lens array, video wall, and camera sensor are manufactured according to the design drawings, ensuring that all components meet the design specifications. During the manufacturing process, the lens array and sensor are precisely installed into the gaps in the video wall, using high-precision assembly processes to ensure the overall performance of the system. The integrated system is thoroughly tested, including light field display effects and eye tracking accuracy. Adjustments and optimizations are performed to ensure high system reliability and stability.

[0370] High-precision light field display technology and optimized lens design provide clear, continuous images and a wider field of view, enhancing user immersion. An integrated eye detection system enables high-precision eye tracking, enhancing the user's interactive experience with the system. This specific implementation integrates a lens array and eye detection sensors to achieve dual functions of light field display and eye tracking within the gaps between the video wall.

[0371] This design not only improves system performance but also reduces hardware complexity and cost, providing an innovative solution for near-eye display technology. Precise component design, optimized optical paths, and efficient manufacturing and assembly processes ensure system quality and stability, ultimately delivering an exceptional user experience.

[0372] Example 1: Lens array splicing screen design

[0373] Microlens: Made of high-refractive-index optical glass or plastic, with a diameter of approximately 15 mm and focal length customized according to design requirements.

[0374] Spliced screen: High-resolution OLED display with a resolution of 8K and above, high brightness and wide color gamut.

[0375] Lens array design and manufacturing: Design the lens array according to the target display area, determine the position and focal length of each lens, and use high-precision molds to manufacture microlenses.

[0376] Design and manufacturing of spliced screens: Select the appropriate OLED display and customize it according to the design of the lens array to ensure that the display matches the lens array.

[0377] Assembly of lens array and splicing screen: Fix the micro lens array in front of the splicing screen, and ensure the precise alignment of the lens array and the display screen through precise positioning and fixing devices.

[0378] Effects and Applications: Lenticular array splicing technology achieves seamless splicing by precisely calibrating the imaging area of each lens, eliminating imaging discontinuities and improving image quality and visual experience. It is suitable for high-resolution display requirements in VR, AR, and XR devices.

[0379] Example 2: Light Field Display Configuration

[0380] Light field display: High-resolution OLED display with resolution up to 8K and above.

[0381] Driving circuit and control chip: high-performance image processor that supports real-time light field image generation and display.

[0382] Light field display selection and installation: Select a high-resolution OLED display and install it behind the lens array, ensuring precise alignment of the display and the lens array.

[0383] Drive circuit and control chip configuration: According to the specifications and display requirements of the display, configure high-performance image processors and drive circuits to ensure the generation and display of real-time light field images.

[0384] Light field image generation: The input image data is processed by the control chip to generate a three-dimensional light field image, which is then displayed on the display.

[0385] Effects and Uses: The light field display configuration can generate high-resolution three-dimensional light field images, significantly improving the display effect and visual experience, and is suitable for high-quality display requirements in VR, AR, and XR devices.

[0386] Example 3: Eye Tracking System

[0387] Human eye camera (infrared camera or other detection sensor): a high-sensitivity, low-noise image sensor with high resolution and high frame rate characteristics.

[0388] Data processing module: high-performance processor, supporting real-time data processing and feedback.

[0389] Selection and installation of human eye camera: Select a highly sensitive infrared camera or other detection sensor and install it in the gap of the splicing screen. Through precise position and angle adjustment, high-precision detection of the human eye can be achieved.

[0390] Data processing module configuration: Based on the specifications and detection requirements of the human eye camera, a high-performance processor is configured to ensure real-time data processing and feedback.

[0391] Eye position and motion trajectory detection: The eye camera captures images of the user's eyes, and the data processing module performs real-time analysis and feedback to generate data on the user's eye position and motion trajectory.

[0392] Effects and Uses: The eye tracking system can achieve high-precision eye detection and tracking, improve the interactivity and user experience of VR, AR and XR devices, and is suitable for application scenarios that require high-precision eye tracking.

[0393] Example 4: Regional Correction Method

[0394] Calibration point measurement: Calibration points are pre-set in the lens array display system, and high-precision measuring equipment is used to measure the imaging range of each lens within the field of view.

[0395] Visible pixel optical correction: Optically correct the visible pixel area of each lens to ensure clear images entering the human eye.

[0396] Invisible pixel filling: The invisible pixel area of each lens is filled using an image fitting algorithm to ensure the integrity and consistency of the entire image area.

[0397] Effects and Uses: The regional correction method ensures image clarity and integrity and improves display effects by optically correcting and filling the visible and invisible pixels of each lens. It is suitable for application scenarios requiring high-precision image display.

[0398] Through the detailed description of the above specific implementation methods, the present invention provides a new near-eye display device that integrates lens array light field display and human eye detection. It has multiple advantages such as low cost, simple structure, good performance, easy use and high production quality, and provides an innovative solution for the development of VR, AR and XR devices.

[0399] As a specific implementation scheme of the embodiment of the present invention, it specifically includes:

[0400] The VR optical machine housing 1 serves as the protection and support structure of the entire device, and houses a display screen 3, an optical machine assembly (including one or more layers of lens assembly), an environmental camera 2 and a human eye camera 4.

[0401] The environment camera 2 is fixed on the right side of the housing and is used to capture external environment images.

[0402] The display screen 3 is located on the left side of the housing and is responsible for displaying virtual images.

[0403] The human eye camera 4 is fixed in the center of the housing through the gap 5 of the spliced screen and is used to detect the user's eye status.

[0404] Human eye detection:

[0405] The human eye camera 4 can analyze the user's eye movement, pupil size and other conditions in real time by capturing images of the user's eyes.

[0406] Image processing algorithms are used to analyze eye movement trajectories, and based on the analysis results, the rendering position of the virtual image is adjusted in real time to match the user's gaze point, achieving more accurate interaction and image rendering, such as implementing foveated rendering technology to improve rendering efficiency and image quality.

[0407] At the same time, human eye state can also be used as a form of user input to control interactions in virtual environments.

[0408] Environmental Integration:

[0409] The environment camera 2 captures images of the external environment.

[0410] These images are pre-processed using image processing algorithms to remove noise and enhance details.

[0411] The pre-processed environment image is seamlessly integrated with the virtual image formed by the lens array to achieve augmented reality (AR) effects.

[0412] In this way, users can see virtual images while also perceiving the real environment, achieving a more natural interactive experience.

[0413] Lens array and light field display:

[0414] The optical machine assembly inside the VR optical machine housing includes one or more layers of lens groups, which form a lens array.

[0415] The image displayed on the display screen 3 forms a light field through the lens array, generating a 3D image with depth and parallax.

[0416] By adjusting the arrangement and focal length of each sub-lens in the lens array, the light field display effect is optimized.

[0417] This light field display technology can provide more realistic 3D effects and enhance the user's sense of immersion.

[0418] Intelligent and adaptable:

[0419] Deep learning algorithms are used to intelligently analyze the user's eye status and environmental images, thereby dynamically adjusting the display content and improving the intelligence and adaptability of near-eye display devices.

[0420] Combining lens array light field display technology and human eye detection technology, it provides a near-eye display experience with high immersion and natural interaction.

[0421] Splicing screen gap:

[0422] The gap 5 of the spliced screen is designed to meet the installation requirements of the human eye camera 4.

[0423] Through this gap, the human eye camera 4 can accurately capture the user's eye status without being blocked by the display screen 3 or other components.

[0424] In summary, the embodiments of the present invention integrate lens array light field display technology with human eye detection technology to provide a near-eye display device with high immersion, natural interaction, and efficient rendering. This device has broad application prospects in the fields of virtual reality (VR) and augmented reality (AR).

[0425] The specific application fields or related products of the present invention.

[0426] 1. Virtual Reality (VR) Devices

[0427] application:

[0428] High-resolution display: Utilizing the lens array and light field display technology of the present invention, higher-resolution VR display can be achieved, improving the clarity and immersion of visual effects.

[0429] Precise eye tracking: The integrated eye tracking system can be used to enhance the user interaction experience in VR, providing more natural gaze tracking and real-time feedback.

[0430] Related Products:

[0431] VR headsets, such as Oculus Quest and HTC Vive.

[0432] High-end VR training and simulation system.

[0433] 2. Augmented Reality (AR) Devices

[0434] application:

[0435] Light field display: The technology of this invention can be used in AR glasses or head-mounted devices to achieve a more natural light field display effect.

[0436] Environmental fusion: The integrated eye tracking system allows for more precise fusion of virtual objects with the real environment.

[0437] Related Products:

[0438] AR smart glasses, such as Microsoft HoloLens.

[0439] Mobile AR devices and head-mounted displays.

[0440] 3. Mixed Reality (MR) Devices

[0441] application:

[0442] Seamless splicing: The seamless splicing technology of the present invention is suitable for large-size MR display screens or environments, providing a continuous visual effect.

[0443] Dual-function lens: Integrated light field display and eye tracking capabilities can enhance interactivity and user experience in MR.

[0444] Related Products:

[0445] MR headsets, such as Magic Leap.

[0446] MR environment experience equipment and applications.

[0447] 4. High-resolution display system

[0448] application:

[0449] Large display screens: Used for high-resolution displays in advertising, exhibitions, conferences, and other scenarios, providing clear images and viewing angles.

[0450] Medical imaging: Applied in medical image display to improve image quality and diagnostic accuracy.

[0451] Related Products:

[0452] Large screen display walls and digital signage.

[0453] Medical imaging equipment and monitors.

[0454] 5. Intelligent car display system

[0455] application:

[0456] Enhanced driving experience: Integrated light field display technology can be used in the car's head-up display (HUD) system to provide clearer driving information and augmented reality navigation.

[0457] Eye tracking: used to monitor the driver's eye status, improving driving safety and the responsiveness of the autonomous driving system.

[0458] Related Products:

[0459] Advanced Driver Assistance Systems (ADAS).

[0460] Automotive HUD displays and smart dashboards.

[0461] 6. Education and training system

[0462] application:

[0463] Immersive Learning: Using high-resolution light-field displays and precise eye-tracking technology in education and training to enhance learning outcomes and user engagement.

[0464] Simulation training: used to simulate complex environments and interactive scenarios to improve the actual effect and experience of training.

[0465] Related Products:

[0466] Educational VR / AR equipment.

[0467] Specialized training simulators and systems.

[0468] The technical solution of this invention has broad application potential in a variety of fields, including virtual reality, augmented reality, mixed reality, high-resolution displays, smart automotive display systems, and education and training. By improving display quality and eye-tracking accuracy, this invention provides innovative solutions for these fields, promoting the development and application of related products.

[0469] 1. High-resolution light field display

[0470] By using the lens array and light field display technology of this invention in a virtual reality headset, experiments have shown that image resolution has increased by approximately 30%. Users report that images are clearer and more detailed. A comparison of image resolution between traditional display technology and the technology of this invention reveals that, under the same hardware conditions, the resolution of traditional display technology is 4K, while using this technology, the resolution is increased to 5.2K.

[0471] 2. Seamless splicing display

[0472] In large-scale spliced screens, the lens array technology of this invention effectively eliminates the gap between the screens, improving visual continuity by 45%. The gap width of traditional spliced screens is about 1.2 mm, but this technology reduces the gap to 0.5 mm, making it almost imperceptible and improving the user's visual experience.

[0473] 3. Accurate eye tracking

[0474] By integrating the eye-tracking system of the present invention into augmented reality devices, the accuracy of eye position detection has increased by 35% and the response speed has increased by 20%. In a comparative experiment, the detection error of a traditional eye-tracking system was 0.8 degrees, while the technology of the present invention reduced the error to 0.52 degrees. Furthermore, the system is able to respond and adjust more quickly when the user moves their gaze quickly.

[0475] 4. Environmental integration effect

[0476] In mixed reality devices, the technology of this invention significantly improves the fusion of virtual objects and the real environment, reducing the fusion error by 50%. This technology reduces the fusion error between virtual objects and the real background from 2.0 cm to 1.0 cm, enhancing the user's immersion.

[0477] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated hardware. A person skilled in the art will understand that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in processor control code.

[0478] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for near-eye display using lens array light field display and integrated human eye detection, characterized in that: The following steps are involved: Step 1: Splice the display screen; Multiple displays are stitched together through precise alignment and calibration to form a large, high-resolution display panel; Ensure the edge gaps between each display are minimized; Step 2: Install the lens array; A lens array consisting of multiple optical lenses is installed at a certain distance in front of the spliced screen. The lens array optimizes the focus and projection of light through light field rendering, supporting the projection of images from different perspectives. Step 3: Install the image sensor or camera; Design appropriate space in the gaps between the spliced screens to install high-resolution image sensors or cameras to capture optical data from the user's eyes, ensuring that the lens can support both light field display and eye detection functions. Step 4: Capture and analyze eye images; using an eye-tracking camera or sensor to capture images of the user's eyes; The high-precision image processing unit analyzes the deformation of structured light, infers the three-dimensional shape and movement of the eye, and combines it with the image data captured by the environmental camera for fusion processing; Real-time monitoring of eye movements and pupil changes; Step 5: Foveated rendering technology; The detected eye state data is used to track the user's gaze point to implement foveated rendering technology; the user's eye state is accurately detected through image sensors or cameras; Step 6: Environmental image fusion; The external environment image is captured by the environmental camera and merged with the virtual image generated by the lens array; the brightness and color of the display are automatically adjusted using ambient light correction technology; Step 7: Display the virtual image; The virtual image is displayed on the display screen, and the lens array is responsible for light field display, supporting the projection of images from different perspectives. By optimizing the focusing and projection of light, 3D images with depth and parallax are generated. Step 8: Multi-sensor fusion technology; Utilizes data fusion technology from multiple sensors to combine data from environmental cameras, human eye cameras, and other sensors; Step 9: Adaptive display technology; Dynamically adjust the resolution and rendering strategy of displayed content based on the user's gaze point and eye movement information to optimize the use of system resources; Step 10: User customized settings; Provide user interface and configuration options to allow users to adjust display parameters and tracking accuracy according to personal preferences and usage scenarios; Step 11: Algorithm optimization; Use advanced machine learning and artificial intelligence algorithms to analyze and predict captured eye movement data; Step 12: Hardware acceleration; Utilize GPU and dedicated hardware accelerators to increase the speed of image processing and rendering; Eye cameras or sensors are used to capture eye data, monitor eye movements and pupil changes in real time, provide high-precision eye tracking, and obtain eye movement and pupil size data; Accurately detect the user's eye state through image sensors or cameras to achieve gaze point rendering and user interaction; The detected eye state data is used to track the user's gaze point and implement foveated rendering technology; Capturing the external environment image through the environment camera and fusing the image with the virtual image generated by the lens array; The virtual image is displayed through the display screen, and the lens array is responsible for light field display, supporting the projection of images from different perspectives. By optimizing the focusing and projection of light, 3D images with depth and parallax are generated.

2. The method for near-eye display using lens array light field display and integrated human eye detection as claimed in claim 1, characterized in that: Methods for near-eye display using integrated lens array light field display and human eye detection include: The eye movement and pupil size detected by the human eye camera are used as user input to control interactive operations in the virtual environment; The human eye camera is fixed in the center of the VR optical machine housing to accurately capture the user's eye status through the gaps in the spliced screen.

3. The method for near-eye display using lens array light field display and integrated human eye detection as claimed in claim 1, characterized in that: Methods for near-eye display using integrated lens array light field display and human eye detection include: A lens array is formed by using one or more layers of lens groups through an optical machine assembly within a VR optical machine housing; The image displayed on the display forms a light field through the lens array, generating a more realistic 3D image and enhancing the user's sense of immersion.

4. The method for near-eye display using lens array light field display and integrated human eye detection as claimed in claim 1, characterized in that: include: The light capture and image pickup part, together with the sub-lenses in the lens array, form a complete camera acquisition device, achieving high-precision capture of environmental images and user eye status; Combining lens array light field display technology and human eye detection technology, it provides a near-eye display experience with high immersion and natural interaction.

5. The method for near-eye display using lens array light field display and integrated human eye detection as claimed in claim 1, characterized in that: include: The user's eye images are captured through a human eye camera, and the eye movement trajectory is analyzed using image processing algorithms; Based on the analysis results, the rendering position of the virtual image is adjusted in real time to match the user's gaze point, thereby improving the user experience.

6. The method for near-eye display using lens array light field display and integrated human eye detection as claimed in claim 2, characterized in that: include: The external environment images captured by the environment camera are pre-processed through image processing algorithms to remove noise and enhance details; The pre-processed environment image is seamlessly integrated with the virtual image to provide a more realistic augmented reality effect.

7. The method for near-eye display using lens array light field display and integrated human eye detection as claimed in claim 3, characterized in that: Further including: By adjusting the arrangement and focal length of each sub-lens in the lens array, the light field display effect is optimized; Deep learning algorithms are used to intelligently analyze the user's eye status and environmental images, thereby dynamically adjusting the display content and improving the intelligence and adaptability of near-eye display devices.

8. A near-eye display device integrating lens array light field display and human eye detection, characterized in that: include: VR optical machine housing, environmental camera, display screen, human eye camera, and splicing screen gaps; An environmental camera is fixed to the right side of the VR optical machine housing by screws; a display screen is fixed to the left side of the VR optical machine housing by screws; a splicing screen gap is set in the center of the left side of the VR optical machine housing; a human eye camera is fixed to the splicing screen gap 5 by screws; The human eye camera is used to detect the state of the human eye.

9. The near-eye display device integrating lens array light field display and human eye detection according to claim 8, characterized in that: The environment camera is used to capture the environment and fuse the virtual image formed by the lens array.

10. The near-eye display device integrating lens array light field display and human eye detection according to claim 8, characterized in that: Inside the VR optical machine housing are a display screen and an optical machine assembly, which includes one or more layers of lens groups.

Images