Pseudo 3D display method and system based on eye movement tracking and affine transformation

Abstract

The invention discloses a pseudo 3D display method and system based on eye movement tracking and affine transformation, particularly relates to the technical field of display and man-machine interaction, and comprises the steps of 3D model preprocessing, real-time eye movement tracking, view angle parameter calculation, dynamic projection generation, affine transformation correction, display image adjustment and dynamic pseudo 3D display effect realization. The pseudo 3D effect that the visual angle is dynamically adjusted along with the eye position of the user can be achieved through a common 2D screen, special 3D display equipment is not needed, pixel-level perspective correction is achieved, the depth perception of a pseudo 3D image observed by the user is closer to a real three-dimensional space, the visual immersion and interaction naturalness are effectively improved, and the user experience is improved. The core problems that traditional pseudo 3D depth perception is fuzzy and distortion is easily generated due to view angle deviation are solved, the user immersion and interaction naturalness are remarkably improved, and the method can be widely applied to scenes such as virtual social contact, vehicle-mounted HMI, telemedicine and education simulation.

Description

Technical Field

[0001] This invention relates to the field of display and human-computer interaction technology, and more specifically, to a pseudo-3D display method and system based on eye tracking and affine transformation. Background Technology

[0002] With the rapid development of display technology and human-computer interaction technology, users' requirements for immersive visual experience are increasing day by day. The technological evolution from traditional 2D flat display to multi-dimensional visual presentation is accelerating. Users' needs for immersive visual experience and natural interaction have increased significantly. 3D display technology has become a research hotspot, involving a variety of scenarios such as virtual social interaction, in-vehicle HMI, telemedicine, and educational simulation.

[0003] Stereo glasses or polarized 3D require users to wear special equipment, which can easily cause dizziness and visual fatigue after prolonged use. In addition, the equipment is inconvenient to carry. On the other hand, naked-eye 3D has a narrow viewing angle, usually less than 30°. When the user deviates from the optimal position or when multiple people are watching, the 3D effect drops sharply. Holographic and light field display hardware structures are complex and costly, making it difficult to popularize them on consumer terminals such as mobile phones, tablets, and PCs. Most existing pseudo-3D solutions use fixed-viewpoint rendering and preset camera positions, which cannot adaptively adjust the viewpoint according to the user's head or eye movements, easily causing a sense of disorientation. Some solutions introduce head pose estimation, but can only obtain the overall position of the head and cannot accurately capture the three-dimensional coordinates of the eyes, making it difficult to achieve pixel-level perspective correction. Other solutions use eye tracking, but only to optimize rendering efficiency and do not solve the perspective distortion caused by viewpoint shift. There is a lack of pseudo-3D solutions with low hardware dependence, high real-time performance, and strong immersion. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a pseudo 3D display method and system based on eye tracking and affine transformation to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a pseudo-3D display method based on eye tracking and affine transformation, comprising the following steps: S1: 3D model preprocessing, loading the 3D model to be rendered, establishing the mapping relationship between the 3D model's world coordinate system and the screen's physical coordinate system based on the screen's physical dimensions, and defining the screen center as the origin. Where the x-axis represents the horizontal direction to the right of the screen, the y-axis represents the vertical direction upwards of the screen, and the z-axis represents the direction perpendicular to the screen facing outwards towards the user. The fixed coordinates of the front-facing camera in the world coordinate system are also calibrated as follows: ; S2: Real-time eye tracking, including the following steps: S21: Camera intrinsic parameter calibration, determining the intrinsic parameters of the front camera. ,in Focal length Principal point coordinates, in pixels; S22: Image Acquisition and Feature Detection. Captures user facial images via the front-facing camera at a frequency of at least 60fps, employs a lightweight CNN model to detect and output the pixel coordinates of the left eye. Right eye pixel coordinates Simultaneously, a monocular depth estimation algorithm or a binocular stereo vision algorithm is used to obtain the depth from the left eye to the front-facing camera. Depth from the right eye to the front-facing camera ; S23: Pixel coordinates back-projected to the camera coordinate system: Using the pinhole camera model, the pixel coordinates and depth are back-projected to the front camera coordinate system with the camera's optical center as the origin and the optical axis as +Z. Left eye 3D coordinates ,in , ; Right eye 3D coordinates ,in ; S24: Direction angle calculation, left eye horizontal direction angle : With the x-axis of the front camera coordinate system as the reference, the angle is positive when the line of sight shifts to the right; Left eye vertical angle : With the y-axis of the front camera coordinate system as the reference, the angle is positive when the line of sight shifts upward; if the v-axis of the pixel coordinate system is downward, the sign needs to be adjusted according to the actual situation. Right eye horizontal angle : ; Right eye vertical angle : ; Pupil center world coordinate system calculation: Based on the above orientation angle, depth, and front camera world coordinates Calculate the center of the left pupil Center of the right pupil Three-dimensional coordinates in the world coordinate system: Center of left pupil: ; ; ; Center of right pupil: ; ; ; S3: Visual angle parameter calculation. Based on the three-dimensional coordinates of the pupil centers obtained in S2, the position of the binocular center is calculated. The calculation formula is as follows: ; Combined with the coordinates of the center point of the screen And calculate the viewpoint offset angle. and The calculation formula is: ; S3: Visual angle parameter calculation. Based on the three-dimensional coordinates of the pupil centers obtained in S2, the position of the binocular center is calculated. The calculation formula is as follows: ; Combined with the coordinates of the center point of the screen And calculate the viewpoint offset angle. and The calculation formula is: ; S4: Affine transformation correction. A dynamic affine transformation matrix M is constructed to perform viewpoint alignment, size adaptation, and position correction on the 2D image after perspective projection of the 3D model. Specifically, the dynamic affine transformation matrix M is constructed based on the mathematical essence of 2D affine transformation. 2D affine transformation preserves the parallelism of points, lines, and lines within a plane, and can be represented as a composite of linear and translation transformations. Its homogeneous coordinate form is: ; The top left 2×2 submatrix It is responsible for linear transformations, including rotation and scaling; This is a translation compensation term, used to compensate for image shifts that may occur during affine transformations, ensuring that the corrected image is displayed completely within the physical area of ​​the screen. Last line To ensure that the coordinates remain in homogeneous form after transformation, the coordinates are standardized to 2D. in ; This is a depth-adaptive coefficient, based on the distance between the center of the user's eyes and the screen. Dynamic adjustment, with specific value selection rules as follows: ; The derivation and element calculation of matrix M are as follows: Matrix M is composed of rotation matrix, scaling matrix, and translation matrix in the order of rotation, scaling, and translation. The specific derivation and calculation method are as follows: Basic parameter definitions: Let the physical width of the screen be W and the height be H. The width of the 2D image after the perspective projection of the 3D model is w and the height is h; the viewing angle offset angle... , Calculated by S3, the distance between the center of the user's eyes and the screen is... ; Combining rotation and scaling matrices: Rotation matrix: for viewpoint offset angle and The rotation matrix is: Scaling matrix: Introducing depth adaptive coefficients ,according to The image size is dynamically adjusted using the scaling matrix: ; Composition of linear transformation terms: The composite result after rotation and scaling is , Right now: Translation compensation item Calculation: After rotation and scaling, the image center may deviate from the screen center. Translation compensation is needed to ensure the image is fully displayed on the screen. Calculate the image center offset caused by rotation: ; Translation compensation item: ; Through the above derivation, the dynamic affine transformation matrix M can realize the viewpoint alignment, size adaptation and position correction of 2D images, ensuring that the image under the user's current line of sight is distorted and displayed completely; S5: Dynamic projection generation. Using the viewing direction calculated in S3 as the camera orientation, the 3D model is projected through the calculated transformation matrix to obtain a 2D image in the viewing direction calculated in S3, and the 2D image in the viewing direction is stored in the 2D image storage module. S6: Repeat S2 to S5 at a frequency of no less than 30fps, and adjust the displayed 2D images with different viewing directions in real time according to the changes in the user's eye position to achieve a dynamic pseudo-3D display effect. When the viewing direction calculated by S3 is the same as the viewing direction of the 2D image that has been stored in the 2D image storage module, the pseudo-3D display effect of that viewing direction is achieved by directly calling the 2D image with the same viewing direction that has been stored in the 2D image storage module.

[0006] Preferably, the monocular depth estimation algorithm and the binocular stereo vision algorithm in S22 are used to obtain the depth from the left eye to the camera. Depth of right eye from camera .

[0007] Preferably, in step S5, the 2D image with the same viewing direction that has been stored in the 2D image storage module is directly called to save the steps of affine transformation calculation and dynamic projection generation, thereby improving projection efficiency.

[0008] Preferably, the end-to-end delay from acquiring an image in S2 to outputting a 2D image in S5 is defined as the total time elapsed from the change in the user's eye position to the display of the corresponding corrected image on the terminal device screen.

[0009] A pseudo-3D display system based on eye tracking and affine transformation for implementing the method described above, the system being integrated into a terminal device, including a 3D model storage module, an eye tracking module, a viewpoint parameter calculation module, an affine transformation module, a projection generation module, a 2D image storage module, and a display driving module; The 3D model storage module is used to store the 3D model to be rendered and establishes a mapping relationship between the 3D model world coordinate system and the screen physical coordinate system based on the screen physical size. The eye-tracking module includes a front-facing camera and a lightweight AI vision algorithm. The front-facing camera is used to capture user facial images at a frequency of no less than 60fps, and the lightweight AI vision algorithm is used to process the captured facial images. The viewpoint parameter calculation module is connected to the eye-tracking module and is used to calculate the position of the center of both eyes based on the acquired three-dimensional coordinates of the center of the pupils. And combined with the coordinates of the center point of the screen Calculate the viewpoint offset angle and ; The affine transformation module is connected to the viewpoint parameter calculation module and is used to calculate the viewpoint offset angle. , and depth adaptive coefficient Constructing dynamic affine transformation matrix The matrix It is a composite matrix consisting of a rotation matrix, a scaling matrix, and a translation matrix, including a translation compensation term. To compensate for image shift during affine transformation; where, The distance between the user's eye center and the screen is dynamically adjusted. The projection generation module is connected to the 3D model storage module, the viewpoint parameter calculation module, and the affine transformation module. It is used to perform perspective projection on the 3D model to generate an initial 2D image, using the viewing direction output by the viewpoint parameter calculation module as the camera's orientation, and then transforms the image using an affine transformation matrix. Correct the initial 2D image; The 2D image storage module is connected to the projection generation module and is used to store 2D images with different viewing directions after correction. When the current viewing direction is detected to be consistent with the viewing direction of the stored image, the stored image is directly called. The display driver module is connected to the 2D image storage module and is used to output the corrected or retrieved 2D image to the terminal device screen to achieve a dynamic pseudo-3D display effect at a frequency of not less than 30fps, and the system end-to-end latency is less than 20 milliseconds.

[0010] Preferably, the terminal device is a mobile phone, tablet computer, vehicle central control device or personal computer, and supports real-time rendering of 60fps or higher.

[0011] Preferably, the dynamic affine transformation matrix constructed by the affine transformation module includes depth adaptive coefficients. The depth adaptive coefficient Based on the distance between the user's eyes and the screen Dynamic adjustment, with specific value selection rules as follows: .

[0012] The technical effects and advantages of this invention are as follows: By using a conventional front-facing camera on a terminal device and employing a lightweight CNN model to assist in eye tracking, the system accurately calculates the three-dimensional coordinates of the center of the pupils in the world coordinate system. Based on the pupil center coordinates, it derives the position of the center of the eyes and the viewing angle offset, constructs a dynamic affine transformation matrix that integrates rotation, scaling, and translation compensation, and completes real-time projection. This eliminates the reliance on dedicated 3D devices such as stereoscopic glasses and polarized screens, and can achieve a pseudo-3D effect with the viewing angle dynamically adjusted according to the user's eye position using only a regular 2D screen. This completely solves the problems of traditional pseudo-3D fixed viewing angles that easily produce a sense of dislocation, and the fatigue and inconvenience of using dedicated 3D devices, significantly reducing the hardware threshold for pseudo-3D display. Unlike traditional solutions that only obtain a rough position through head pose estimation or optimize rendering efficiency through simple eye tracking, this approach uses a collaborative design of precise parameter calculation and dynamic matrix correction. First, it accurately calculates the position of the center of both eyes and the viewing angle offset using the three-dimensional coordinates of the center of the pupils, ensuring that the viewing parameters are highly matched with the user's actual line of sight. Then, in the dynamic affine transformation matrix, rotation and scaling are achieved through sub-matrices, while translation compensation terms correct the image offset during the transformation process, ensuring that the corrected image is distortion-free and fully displayed. This achieves pixel-level perspective correction, making the depth perception of the pseudo-3D image observed by the user closer to the real three-dimensional space, effectively improving visual immersion and natural interaction, and solving the core problems of blurred depth perception and distortion caused by viewing angle offset in traditional pseudo-3D. By storing 2D images with different viewing directions after affine transformation correction, when the current user's viewing direction is detected to be consistent with the viewing direction of the stored image, the stored image is directly called, eliminating the need for repeated affine transformation calculations and 3D model perspective projection steps. This mechanism, combined with a technical design that ensures a less than 20-millisecond end-to-end latency from image acquisition to 2D image output, significantly reduces the computational load on terminal devices while guaranteeing real-time performance. Furthermore, the entire system can be integrated into various terminals such as mobile phones, tablets, in-vehicle central control devices, and personal computers. The compatibility and efficiency of the core technical solution enable it to be widely adapted to diverse scenarios such as virtual social interaction, in-vehicle HMI, telemedicine, and educational simulation, breaking through the limitations of traditional pseudo-3D technology's narrow application scenarios and poor multi-terminal adaptability, thus expanding the practical value of pseudo-3D display technology. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the method flow structure of the present invention.

[0014] Figure 2 This is a schematic diagram of the system structure of the present invention.

[0015] Figure 3 This is a schematic diagram of the affine transformation matrix mapping of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: Interactive Scenario of Virtual Anchor on Mobile Terminal I. Basic Conditions for Implementation Hardware configuration: Adopts mainstream smartphone specifications; front-facing camera is 50 megapixels, supporting image acquisition at ≥60fps; calibration of the front-facing camera's world coordinate system is [fixed coordinates]. The screen shares the same X and Y axes with the center origin of the screen; the screen is a 6.5-inch OLED screen with a physical resolution of 2700×1224 pixels, and the measured physical width is... The device supports real-time rendering at 60fps or higher, and the GPU is an Adreno 750.

[0018] Software configuration: It runs on Android 13 operating system and uses Unity 2022.3.10f1 rendering engine; the lightweight CNN model uses MobileNetV3 combined with the pupil regression head model; the 3D model is a .fbx format virtual anchor animation model, containing 120 facial expressions and 50,000 vertices; the depth calculation adopts a monocular depth estimation algorithm based on 68 facial key points.

[0019] II. Detailed Implementation Steps S1: 3D Model Preprocessing Load the .fbx format virtual anchor model into the Unity engine and set the scaling factor to 0.01 to adapt to the screen's physical size; establish the mapping relationship between the 3D model's world coordinate system and the screen's physical coordinate system, with the X-axis horizontal to the right, the Y-axis vertically upward, and the Z-axis perpendicular to the screen and outward, in mm, and define the screen center as the origin. By binding the model's center point to the screen origin using Unity's "Transform" component, the initial rendering is centered.

[0020] S2: Real-time eye tracking, including the following steps: S21: Camera intrinsic parameter calibration, determining the front camera intrinsic parameters using Zhang Zhengyou's calibration method: , , Principal coordinates, corresponding to the center of the screen resolution.

[0021] S22: Image acquisition and feature detection. The front-facing camera is activated, configured with a frame rate of 60fps, exposure time of 1 / 120s, and ISO 100. The acquired facial image is preprocessed by grayscale conversion and 5×5 kernel Gaussian blur, then input into a lightweight CNN model. Output: Left eye pixel coordinates. Right eye pixel coordinates Monocular depth algorithm outputs left eye depth. Right eye depth .

[0022] S23: Pixel coordinates back-projected onto the camera coordinate system: Left eye 3D coordinates : ; ; Right eye 3D coordinates ; ; ; S24: Direction angle calculation: Left eye: ; ; Right eye: ; ; 5. Calculation of the pupil center in the world coordinate system: Left eye: ; ; ; Right eye: ; ; ; S3: Calculation of Viewpoint Parameters Center position of both eyes: ; ; ; Viewpoint offset angle: ; ; S4: Affine Transform Correction Basic parameter definition: Width of the 2D image after perspective projection of the 3D model ,high ; (30≤z_e≤60cm), depth adaptive coefficient k(z_e)=1.0.

[0023] 2. Matrix composition calculation: Rotation matrix ; Scaling matrix ; Linear transformation term: ; in: , $ , ; 3. Calculation of translation compensation item: Offset: ; ; Translation compensation item: ; ; 4. Dynamic affine transformation matrix: ; S5: Dynamic Projection Generation Create a virtual camera in the Unity engine, and set its orientation to... Internal parameters match the phone screen, focal length 1. Field of view 60°; Enable virtual anchor skeletal animation synchronization, perform perspective projection on the 3D model to generate an initial 2D image, and after matrix $M$ correction, store it in the 2D image storage module.

[0024] S6: Cyclic Update S2 to S5 are repeated at 30fps, and frame synchronization is ensured through the system's VSync mechanism; when a repeated gaze direction is detected, the stored image is directly called, saving the need for repeated calculations.

[0025] III. Implementation Results Real-time performance: 1000fps high-speed camera test, end-to-end latency 18.5ms.

[0026] Stability: After running continuously for 2 hours on 3 mainstream mobile phones, there was no lag or perspective distortion, and the average frame rate remained stable at 30fps.

[0027] Interactive experience: When the user moves their head, the virtual anchor's perspective responds synchronously, creating a natural 3D stereoscopic effect, with no perceptible delay in voice and perspective changes.

[0028] Example 2: Interactive Scenario of 3D Educational Courseware on a Tablet Terminal I. Basic Conditions for Implementation Hardware configuration: Utilizes mainstream tablet computers; front-facing camera is 12MP, supporting ≥60fps data capture; calibrates its world coordinate system with fixed coordinates. The screen is a 10.9-inch LCD screen with a physical resolution of 2560×1600 pixels. The measured physical width... ,high The GPU supports OpenGLES 3.2 and supports real-time rendering at 60fps or higher.

[0029] Software configuration: Running iPadOS 16, using Unreal Engine 5.1 for rendering; the lightweight CNN model is MobileNetV3 with a pupil regression head; the 3D model is a .obj format human heart anatomy model with blood vessel / valve annotations and 60,000 vertices; depth calculation uses a monocular depth estimation algorithm.

[0030] II. Detailed Implementation Steps S1: 3D Model Preprocessing Load the .obj format heart model into Unreal Engine, scaling it by 0.02 to fit the screen; establish the coordinate system mapping as in Example 1, defining the screen center origin. The engine tools bind the model's center point to the screen origin to ensure that the initial rendering is centered.

[0031] S2: Real-time eye tracking Camera intrinsic parameter calibration: Determine intrinsic parameters: , corresponding to the center of the screen resolution.

[0032] Image acquisition and feature detection: The camera is configured to acquire images at 60fps. After preprocessing, the images are input into a CNN model, and the output is the pixel coordinates of the left eye. Right eye pixel coordinates Monocular depth algorithm output , .

[0033] 3. Pixel coordinates are back-projected onto the camera coordinate system: Left eye: , , ; Right eye: , , ; 4. Direction angle calculation: Left eye: ; Right eye: ; 5. Calculation of the pupil center in the world coordinate system: Left eye: ; Right eye: S3: Calculation of Viewpoint Parameters Center of both eyes: ; Viewpoint offset angle: ; S4: Affine Transform Correction 1. Basic parameters: Width of the projected 2D image ,high ; , .

[0034] 2. Matrix calculations: Rotation matrix: ; Linear transformation term: ; 3. Translation compensation: ; 4. Matrix M: ; S5: Dynamic Projection Generation Virtual camera orientation set Internal parameters are matched to tablet screens, focal length 1. Field of view 55°; enable heart model annotation rendering, save the image after projection correction.

[0035] S6: Cyclic Update The process repeats at 30fps, and the stored image is directly called in the same line of sight.

[0036] III. Implementation Results Real-time performance: End-to-end latency 19.2ms.

[0037] Teaching experience: In the subjective evaluation of 30 science teachers, 93% believed that "the anatomical structure perspective is synchronized well and the observation of fine structures is intuitive", and 89% believed that it was "superior to traditional 2D courseware".

[0038] Stability: Runs continuously under high load for 2 hours without label misalignment or model flickering, suitable for long-term classroom use.

[0039] Pseudo-3D display system based on eye tracking and affine transformation I. System Architecture This system is integrated into terminal devices (mobile phones, tablets, in-vehicle central control devices, or personal computers, and supports real-time rendering at 60fps or higher). It includes seven functional modules, and the connection relationships and functions of each module are as follows: 1. 3D model storage module Core functions: Store 3D models to be rendered, such as virtual anchor models and human anatomical models, and establish a mapping relationship between the world coordinate system of the 3D model and the physical coordinate system of the screen based on the physical size of the terminal device screen, providing a coordinate basis for subsequent projection and correction.

[0040] 2. Eye-tracking module Composition: Includes a front-facing camera and lightweight AI vision algorithms, MobileNetV3 combined with a pupil regression head model; Function: The front-facing camera captures user facial images at a frequency of no less than 60fps; a lightweight AI vision algorithm preprocesses the captured images.

[0041] 3. Viewpoint Parameter Calculation Module Connection: Connected to the eye-tracking module; Function: Receives the three-dimensional coordinates of the pupil centers of both eyes from the eye-tracking module, and calculates the position of the pupil centers according to the formula defined in claim 1. And combined with the coordinates of the center point of the screen Calculate the viewpoint offset angle and .

[0042] 4. Affine Transformation Module Connection relationship: Connected to the view parameter calculation module; Function: Based on the viewpoint offset angle and depth adaptive coefficient Constructing dynamic affine transformation matrix ;matrix It is a composite matrix consisting of a rotation matrix, a scaling matrix, and a translation matrix, including a translation compensation term. To compensate for image shift during affine transformation; where the depth adaptive coefficients The value is dynamically adjusted based on the distance ze between the center of the user's eyes and the screen, and the value selection rules are as follows: .

[0043] 5. Projection Generation Module Connections: Connected to the 3D model storage module, viewpoint parameter calculation module, and affine transformation module; Function: Using the viewing direction output by the perspective parameter calculation module as the camera orientation, the 3D model in the 3D model storage module is subjected to perspective projection to generate an initial 2D image. Then, the initial 2D image is aligned in perspective, adapted in size, and corrected in position by the dynamic affine transformation matrix M output by the affine transformation module.

[0044] 6.2D Image Storage Module Connection relationship: Connected to the projection generation module; Function: Stores 2D images of different viewing directions after correction by the projection generation module; when the current viewing direction is detected to be consistent with the viewing direction of the stored image, the stored image is directly called, eliminating the repeated affine transformation and projection generation steps and improving the system's operating efficiency.

[0045] 7. Display driver module Connection relationship: Connected to the 2D image storage module; Function: Outputs the corrected image output from the 2D image storage module or the stored image to the terminal device screen at a frequency of no less than 30fps to achieve a dynamic pseudo-3D display effect; the system end-to-end latency is less than 20 milliseconds.

[0046] II. System Operation Logic 1. Initialization: The 3D model storage module loads the 3D model and establishes a coordinate system mapping, completing the coordinate calibration of the front camera; 2. Real-time tracking: The eye-tracking module continuously acquires facial images and outputs the coordinates of the pupil center; 3. Parameter Calculation: The viewing angle parameter calculation module calculates the position of the center of both eyes and the viewing angle offset angle in real time; 4. Image Processing: The affine transformation module constructs a dynamic matrix, and the projection generation module completes projection and correction; 5. Storage and Retrieval: The 2D image storage module stores the corrected image, which can be directly retrieved by repeating the gaze direction; 6. Display Output: The display driver module outputs images at a frequency of 30fps to achieve a dynamic pseudo-3D effect.

[0047] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A pseudo-3D display method based on eye tracking and affine transformation, characterized in that: Includes the following steps: S1: 3D model preprocessing, loading the 3D model to be rendered, establishing the mapping relationship between the 3D model's world coordinate system and the screen's physical coordinate system based on the screen's physical dimensions, and defining the screen center as the origin. Where the x-axis represents the horizontal direction to the right of the screen, the y-axis represents the vertical direction upwards of the screen, and the z-axis represents the direction perpendicular to the screen facing outwards towards the user. The fixed coordinates of the front-facing camera in the world coordinate system are also calibrated as follows: ; S2: Real-time eye tracking, including the following steps: S21: Camera intrinsic parameter calibration, determining the intrinsic parameters of the front camera. ,in Focal length Principal point coordinates, in pixels; S22: Image Acquisition and Feature Detection. Captures user facial images via the front-facing camera at a frequency of at least 60fps, employs a lightweight CNN model to detect and output the pixel coordinates of the left eye. Right eye pixel coordinates Simultaneously, a monocular depth estimation algorithm or a binocular stereo vision algorithm is used to obtain the depth from the left eye to the front-facing camera. Depth from the right eye to the front-facing camera ; S23: Pixel coordinates are back-projected onto the camera coordinate system. Using the pinhole camera model, the pixel coordinates and depth are back-projected onto the front camera coordinate system with the camera's optical center as the origin and the optical axis as +Z. Left eye 3D coordinates ,in , ; Right eye 3D coordinates ,in ; S24: Direction angle calculation, left eye horizontal direction angle : With the x-axis of the front camera coordinate system as the reference, the angle is positive when the line of sight shifts to the right; Left eye vertical angle : With the y-axis of the front camera coordinate system as the reference, the angle is positive when the line of sight shifts upward; if the v-axis of the pixel coordinate system is downward, the sign needs to be adjusted according to the actual situation. Right eye horizontal angle : ; Right eye vertical angle : ; Pupil center world coordinate system calculation: Based on the above orientation angle, depth, and front camera world coordinates Calculate the center of the left pupil Center of the right pupil Three-dimensional coordinates in the world coordinate system: Center of left pupil: ; ; ; Center of right pupil: ; ; ; S3: Calculation of visual angle parameters. Based on the three-dimensional coordinates of the pupil centers obtained in S2, the position of the binocular center is calculated using the following formula: ; Combined with the coordinates of the center point of the screen And calculate the viewpoint offset angle. and The calculation formula is: ； S4: Affine transformation correction. A dynamic affine transformation matrix M is constructed to perform viewpoint alignment, size adaptation, and position correction on the 2D image after perspective projection of the 3D model. Specifically, the dynamic affine transformation matrix M is constructed based on the mathematical essence of 2D affine transformation. 2D affine transformation preserves the parallelism of points, lines, and lines within a plane, and can be represented as a composite of linear and translation transformations. Its homogeneous coordinate form is: ; The top left 2×2 submatrix It is responsible for linear transformations, including rotation and scaling; This is a translation compensation term, used to compensate for image shifts that may occur during affine transformations, ensuring that the corrected image is displayed completely within the physical area of ​​the screen. Last line To ensure that the coordinates remain in homogeneous form after transformation, we can use a 2D image coordinate system. in ; This is a depth-adaptive coefficient, based on the distance between the center of the user's eyes and the screen. Dynamic adjustment, with specific value selection rules as follows: ； The derivation and element calculation of matrix M are as follows: Matrix M is composed of rotation matrix, scaling matrix, and translation matrix in the order of rotation, scaling, and translation. The specific derivation and calculation method are as follows: Basic parameter definitions: Let the physical width of the screen be W and the height be H. The width of the 2D image after the perspective projection of the 3D model is w and the height is h; the viewing angle offset angle... , Calculated by S3, the distance between the center of the user's eyes and the screen is... ; Combining rotation and scaling matrices: Rotation matrix: for viewpoint offset angle and The rotation matrix is: Scaling matrix: Introducing depth adaptive coefficients ,according to The image size is dynamically adjusted using the scaling matrix: ; Composition of linear transformation terms: The composite result after rotation and scaling is , Right now: Translation compensation item Calculation: After rotation and scaling, the image center may deviate from the screen center. Translation compensation is needed to ensure the image is fully displayed on the screen. The image center offset caused by rotation needs to be calculated. ; Translation compensation item: ; Through the above derivation, the dynamic affine transformation matrix M can realize the viewpoint alignment, size adaptation and position correction of 2D images, ensuring that the image under the user's current line of sight is distorted and displayed completely; S5: Dynamic projection generation. Using the viewing direction calculated in S3 as the camera orientation, the 3D model is projected through the calculated transformation matrix to obtain a 2D image in the viewing direction calculated in S3, and the 2D image in the viewing direction is stored in the 2D image storage module. S6: Repeat S2 to S5 at a frequency of no less than 30fps, and adjust the displayed 2D images with different viewing directions in real time according to the changes in the user's eye position to achieve a dynamic pseudo-3D display effect. When the viewing direction calculated by S3 is the same as the viewing direction of the 2D image that has been stored in the 2D image storage module, the pseudo-3D display effect of that viewing direction is achieved by directly calling the 2D image with the same viewing direction that has been stored in the 2D image storage module.

2. The pseudo-3D display method based on eye tracking and affine transformation according to claim 1, characterized in that: The monocular depth estimation algorithm and the binocular stereo vision algorithm in S22 are used to obtain the depth from the left eye to the camera. Depth of right eye from camera .

3. The pseudo-3D display method based on eye tracking and affine transformation according to claim 1, characterized in that: In step S5, the 2D image with the same viewing direction that has been stored in the 2D image storage module is directly called to save the steps of affine transformation calculation and dynamic projection generation, thereby improving projection efficiency.

4. The pseudo-3D display method based on eye tracking and affine transformation according to claim 1, characterized in that: The end-to-end delay from acquiring an image in S2 to outputting a 2D image in S5 is defined as the total time elapsed from the change in the user's eye position to the display of the corresponding corrected image on the terminal device screen.

5. A pseudo-3D display system based on eye tracking and affine transformation for implementing the method as described in any one of claims 1 to 4, characterized in that: The system is integrated into the terminal device and includes a 3D model storage module, an eye-tracking module, a viewpoint parameter calculation module, an affine transformation module, a projection generation module, a 2D image storage module, and a display driver module. The 3D model storage module is used to store the 3D model to be rendered and establishes a mapping relationship between the 3D model world coordinate system and the screen physical coordinate system based on the screen physical size. The eye-tracking module includes a front-facing camera and a lightweight AI vision algorithm. The front-facing camera is used to capture user facial images at a frequency of no less than 60fps, and the lightweight AI vision algorithm is used to process the captured facial images. The viewpoint parameter calculation module is connected to the eye-tracking module and is used to calculate the position of the center of both eyes based on the acquired three-dimensional coordinates of the center of the pupils. And combined with the coordinates of the center point of the screen Calculate the viewpoint offset angle and ; The affine transformation module is connected to the viewpoint parameter calculation module and is used to calculate the viewpoint offset angle. , and depth adaptive coefficient Constructing dynamic affine transformation matrix The matrix It is a composite matrix consisting of a rotation matrix, a scaling matrix, and a translation matrix, including a translation compensation term. To compensate for image shift during affine transformation; where, The distance between the user's eye center and the screen is dynamically adjusted. The projection generation module is connected to the 3D model storage module, the viewpoint parameter calculation module, and the affine transformation module. It is used to perform perspective projection on the 3D model to generate an initial 2D image, using the viewing direction output by the viewpoint parameter calculation module as the camera's orientation, and then transforms the image using an affine transformation matrix. Correct the initial 2D image; The 2D image storage module is connected to the projection generation module and is used to store 2D images with different viewing directions after correction. When the current viewing direction is detected to be consistent with the viewing direction of the stored image, the stored image is directly called. The display driver module is connected to the 2D image storage module and is used to output the corrected or retrieved 2D image to the terminal device screen to achieve a dynamic pseudo-3D display effect at a frequency of not less than 30fps, and the system end-to-end latency is less than 20 milliseconds.

6. The pseudo-3D display system according to claim 5, characterized in that: The terminal device can be a mobile phone, tablet computer, in-vehicle central control device or personal computer, and supports real-time rendering of 60fps or higher.

7. The pseudo-3D display system according to claim 5, characterized in that: The dynamic affine transformation matrix constructed by the affine transformation module contains depth adaptive coefficients. The depth adaptive coefficient Based on the distance between the user's eyes and the screen Dynamic adjustment, with specific value selection rules as follows: .

Images