Human eye tracking type low-crosstalk high-stability naked eye 3D playing method and system

Abstract

The invention relates to a human eye tracking type low-crosstalk high-stability naked eye 3D playing method and system, and belongs to the technical field of naked eye 3D display. The method comprises the following steps: decoding a binocular video and writing the binocular video into GPU external textures; performing human eye tracking through the front camera to obtain eye position data; performing stability enhancement processing on the eye position data to generate a stable eye position; generating stripe display control information according to the stable eye position; left and right views are sampled in a GPU fragment shader, and sub-pixel output is judged; and applying a brightness weight window function to the sub-pixels near the fringe phase boundary to perform energy shaping. The system comprises corresponding function modules. Picture flicker caused by eye position jitter and abrupt change is suppressed through stability enhancement processing, crosstalk is effectively reduced through phase boundary energy shaping, and high-stability, low-crosstalk and high-quality naked eye 3D playing is realized at a mobile terminal through cooperative work of the two.

Description

Technical Field

[0001] This invention belongs to the field of naked-eye 3D display technology, and relates to a naked-eye 3D playback method and system with low crosstalk and high stability based on human eye tracking. Background Technology

[0002] Naked-eye 3D display technology typically uses gratings, cylindrical lenses, or similar optical beam-splitting structures to distribute the sub-pixels of the display panel to different viewpoints according to a preset stripe pattern. This allows the observer's left and right eyes to receive images with parallax, thus creating a stereoscopic visual experience. For naked-eye 3D video playback on mobile devices, a common approach is to combine binocular video decoding with a graphics processing unit for stripe interlacing output. To enhance the viewing experience, this solution often incorporates eye-tracking technology, which adaptively adjusts display control information based on real-time changes in the viewer's position to achieve a more consistent and stable stereoscopic visual effect.

[0003] However, in practical applications, the viewing experience of existing media players still exhibits significant fluctuations, mainly in the following aspects:

[0004] First, the stability of the human eye tracking process is insufficient. Current human eye tracking results are easily affected by factors such as changes in ambient lighting, motion blur caused by rapid head movements, and fluctuations in the algorithm's inference frame rate. This directly manifests as jitter, short-term loss, or even abrupt changes in the coordinate data of the tracked key points. As input to display control, this unstable eye position data causes discontinuous jumps in stripe control information, ultimately producing an uncomfortable flickering phenomenon on the displayed screen, severely damaging the immersive experience and stability of the viewing experience.

[0005] Second, there is an inherent crosstalk problem in the display process. During the stripe interlacing and synthesis stage, the sub-pixels in the phase boundary region are most sensitive to their assigned view. Even a small deviation can lead to light leakage from an unexpected view. This energy leakage in the boundary region can cause image content from the opposite side view to be mixed in, resulting in ghosting and crosstalk, which significantly reduces image clarity and viewing comfort.

[0006] Third, there may be efficiency bottlenecks at the system architecture level. If the decoded video frame data needs to be frequently moved through memory on the central processing unit before being passed to the rendering layer, it will introduce unnecessary processing delays and potential timing inconsistencies, affecting the performance of real-time rendering.

[0007] Currently, existing technologies attempt to address the aforementioned issues from different perspectives. For example, some solutions focus on improving the human eye tracking model itself to enhance its robustness in complex environments, or attempt to mitigate visual abrupt changes when tracking is lost through methods such as image fading. To address crosstalk, some research employs pixel brightness weighting or grayscale modulation in boundary regions. However, these solutions often only address a single aspect and fail to perform collaborative optimization across the entire player system, resulting in either insufficient robustness or the introduction of new side effects in real-world, complex mobile application scenarios.

[0008] Therefore, there is an urgent need in this field for a technical solution that can systematically solve the above problems and still provide a high-stability, low-crosstalk, high-quality naked-eye 3D video playback experience under the condition of limited mobile resources. Summary of the Invention

[0009] In view of this, the purpose of this invention is to provide an eye-tracking, low-crosstalk, high-stability naked-eye 3D playback system suitable for mobile naked-eye 3D video players. Based on the complete playback chain, this method improves the stability and low-crosstalk display effect of mobile naked-eye 3D video playback through the synergy of adaptive stripe control driven by stable eye position and phase boundary energy shaping.

[0010] To achieve the above objectives, the present invention provides the following technical solution: A method for eye-tracking, low-crosstalk, and highly stable naked-eye 3D playback includes the following steps: S1: Decode the input stereo video and write the decoded frames to an external texture associated with the graphics processing unit (GPU) rendering thread; S2: Acquire images from the front-facing camera and perform eye tracking to obtain the position of the left eye. and the right eye position ; S3: Perform stability enhancement processing on the eye position data to generate a stable eye position, wherein when the eye position data is invalid, the previous stable eye position is maintained, and when the eye position data is valid, temporal smoothing is performed. S4: Generate stripe display control information based on the stable eye position, and transmit the control information to the GPU; S5: In the fragment shader, the left and right views are sampled from the stereo video frame texture respectively, and the stripe axis coordinates and sub-pixel phase within the period are constructed to determine whether each sub-pixel output comes from the left or right view. S6: After the decision is made, the stripe phase boundary region is determined based on the sub-pixel phase within the period, and the luminance weight window function is applied to the sub-pixel output near the stripe phase boundary to perform energy shaping according to the phase boundary energy shaping parameters generated in S3, so as to reduce crosstalk and flip flicker. The stable eye position output in S3 and the phase boundary energy shaping parameters are generated together in the same control period.

[0011] Furthermore, in S5, the texture coordinates are... The left and right view samplings satisfy: Left view texture coordinates

[0012] Right view texture coordinates

[0013] Get the color of the left view Color in the right view .

[0014] Furthermore, in S5, the screen pixel coordinates are... Constructing stripe axis coordinates satisfy:

[0015] in, s For the fringe tilt coefficient, For calibrating reference points, β The scaling factor is the ratio from pixel to sub-pixel. O This represents the phase offset.

[0016] Furthermore, in S5, for each sub-pixel channel Calculate sub-pixel phase within the calculation period ,in , c For stripe period control information; And based on r m Left and right view sampling decision: when At that time, sub-pixel output ;when At that time, sub-pixel output .

[0017] Furthermore, in S3, the stability enhancement process includes: S31: Construct a combined confidence level for the current frame eye position observation data, wherein the combined confidence level is based on at least one or more of the following: facial occlusion analysis and the distance of the change in eye position before and after; S32: When the combined confidence level is below the threshold, key points are missing, or the eye position displacement amplitude is greater than the jump threshold, the eye position data is deemed invalid and failure preservation is executed to satisfy the requirement. ; S33: When eye position data is valid, observe eye position Perform exponential smoothing to satisfy ,in To observe eye position vector, For the current stable eye position, To achieve a stable eye position, and For smoothing coefficients, .

[0018] Furthermore, in step S6, the half-cycle center is calculated. Phase distance ; when hour, ; when hour, ; .

[0019] Furthermore, in step S6, a luminance weighted window function is employed. for: when hour, ; when When defining normalized variables and adopt ; when hour, ; Modulate the sub-pixel output to ,in w This refers to the width parameter of the flat top.

[0020] Furthermore, in S1, the external texture is bound to the texture input object by the video decoding output, and the current decoded frame is provided to the fragment shader of S5 through frame-by-frame texture update.

[0021] A human eye-tracking, low-crosstalk, high-stability naked-eye 3D playback system includes: The video decoding and texture input module is used to decode stereo video and write the decoded frames into textures; The human eye tracking module is used to acquire images and track the human eye to obtain the eye position; The tracking stability enhancement module is used to perform stability enhancement processing on eye position data to generate stable eye positions; An adaptive control module is used to generate stripe display control information based on the stable eye position; The GPU stripe composition module is used to synthesize stripes in the fragment shader. The phase boundary energy shaping module, integrated in the GPU stripe synthesis module, is used to perform energy shaping on the sub-pixel output.

[0022] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method.

[0023] The beneficial effects of this invention are as follows: (1) This invention creatively introduces a stability enhancement processing mechanism for human eye tracking data in the playback link. This mechanism determines the validity of eye position data by constructing a combined credibility index, and can intelligently identify and filter tracking noise, short-term loss, or abnormal jumps caused by factors such as insufficient lighting and motion blur. When the data is invalid, the system adopts a failure-holding strategy to maintain the stable eye position of the previous moment; when the data is valid but there is jitter, a temporal smoothing algorithm is used for filtering. This makes the control information used to drive the display continuous and stable, fundamentally avoiding screen flipping and flickering caused by drastic fluctuations in eye position data, and providing viewers with a stable and smooth stereoscopic visual experience.

[0024] (2) In the final stage of subpixel-level stripe synthesis in the graphics processing unit, this invention adds a crucial step: phase boundary energy shaping. This step is not simply a hard decision between the left and right views. Instead, after the decision, for the stripe phase boundary—a region sensitive to errors and prone to light leakage—the phase distance of each subpixel is calculated, and a brightness weighted window function with a flat-top region and polynomial smooth roll-off characteristics is applied. This method can smoothly transition the brightness near the phase boundary while maintaining the brightness of the stripe center region, effectively suppressing light leakage to the side view, thereby significantly reducing ghosting and crosstalk, and improving image clarity and stereoscopic display fidelity.

[0025] (3) The core advantage of this invention lies in the fact that it does not improve a single link in isolation, but rather designs and controls the enhancement of eye tracking stability in conjunction with the pixel-level energy shaping of the final display. The stable eye position and the phase boundary energy shaping parameters are generated collaboratively within the same control cycle, ensuring that the stability of the eye position can be seamlessly transferred to the display link, while the shaping technology of the display link compensates for the physical optical crosstalk that may still exist even under a stable eye position. This closed-loop collaborative design enables the system to exhibit stronger robustness and overall performance advantages when facing the complex real-world usage environment and resource constraints of mobile terminals, avoiding the limitations or new side effects brought about by the "treating the symptoms but not the root cause" approach.

[0026] (4) In the video decoding and input stage, the present invention adopts the method of directly writing the decoded frame to the external texture associated with the rendering thread of the graphics processing unit. This design reduces the number of data transfers and memory copy overhead between the central processing unit and the graphics processing unit, reduces processing latency, and provides underlying protection for the real-time performance and timing consistency of the entire playback link, making high-quality, low-latency naked-eye 3D video playback possible.

[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a system structure block diagram according to an embodiment of the present invention; Figure 2 A simulation diagram illustrating the improvement of data stability for the human eye; Figure 3 Simulation diagram comparing the effects of windowing on crosstalk before and after. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0031] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0032] like Figure 1 As shown, the system in this embodiment includes: (1) Video Decoding and Texture Input Module: Decodes the stereo video and writes the decoded frames to the texture input object associated with the GPU rendering thread. The rendering side updates the texture frame by frame to obtain the current image. (2) Human eye tracking module: Collects frames from the front camera and calls visual algorithms to detect faces and extract key points of the eyes; (3) Tracking stability enhancement module: performs failure maintenance and exponential smoothing on eye position, and outputs a stable eye position; (4) Adaptive display control module: generates stripe period based on stable eye position. C With phase O Display control information; (5) GPU stripe composition module: Performs sub-pixel level stripe interlacing output on the left and right views in the fragment shader; (6) Phase boundary energy shaping module: Apply a brightness weight window function to the output sub-pixel in the stripe phase boundary region to make the boundary transition smooth, thereby reducing crosstalk and flip flicker.

[0033] Frame-by-frame execution process (decoding-tracking-control-compositing-shaping) The video playback thread continuously decodes the stereo video frames and writes the current frame into the texture input object; the rendering thread updates the texture content every frame and enters the stripe synthesis process to form a continuously playing rendering frame source.

[0034] The camera thread captures frames from the front-facing camera and triggers eye tracking; when a face is detected, it extracts... , Forming observation eye position And send it to the stability enhancement module; when it cannot be detected or the key point is unreliable, the stability enhancement module will... Maintain output to enhance the continuity of control information.

[0035] When the key points are valid, the stability enhancement module performs exponential smoothing on the observed eye position. Achieve stable eye position The adaptive display control module updates the fringe period C and phase O based on the stable eye position and passes them to the GPU rendering pipeline to drive the adaptive display of fringe synthesis.

[0036] GPU fragment shader combines frame textures according to , Sample the left and right views to obtain And based on screen coordinates Constructing stripe axis coordinates Calculate the sub-pixel phase within the period for RGB sub-pixel channels , in accordance with or Select the corresponding channel component from the right or left view as the base output. This completes subpixel-level interleaving output.

[0037] After completing the phase decision, the fragment shader further performs phase boundary pixel brightness adjustment. A half-cycle center is defined for each channel. Calculate phase distance Set the flat top width parameter w .when Seasonal brightness weight ;when At that time, calculate And a polynomial smoothing function is used. The final output is This results in a smooth roll-off of subpixels near the fringe phase boundary, making the energy distribution in the boundary region more continuous, improving low crosstalk performance and viewing comfort; at the same time, the parameters w This is used to adjust the energy shaping intensity, so that the brightness cost and the low crosstalk effect can be controlled and balanced.

[0038] Parameter and symbol explanation: Represents the observed eye position vector. ; Indicates a stable eye position; α Represents the exponential smoothing coefficient. ; C This indicates stripe period control information; , indicating the reciprocal of the period; O Indicates the phase offset; Represents screen pixel coordinates; Indicates the calibration reference point; β Indicates the scaling factor; s Indicates the fringe tilt coefficient; m Indicates the sub-pixel channel index. ; This represents the operator for retrieving the decimal part; Indicates the phase of a sub-pixel within a period; This represents the base output channel value after phase decision; Indicates the center of a half-cycle; Indicates phase distance; w Indicates the width of the flat top; This represents the brightness weighted window function; This represents the output after energy shaping.

[0039] Simulation effect description (corresponding) Figure 2 , Figure 3 ) (1) Simulation of improving the stability of human eye data ( Figure 2 ): Collect a sequence of the viewer's observed eye position It meets the characteristics of realistic viewing: most of the time there is slight shaking (natural head movement / slight hand shaking), and a few larger displacements (adjusting sitting posture / obvious head turning); at the same time, short-term detection failure or missing key points can be inserted to simulate low light, motion blur and other situations.

[0040] The comparison group was set up as follows: the comparison scheme directly used the observation eye position. Update stripe control information; the present invention first determines the failure of execution based on the combination confidence and effectiveness, and obtains a stable eye position through exponential smoothing. , and then Update control information. Figure 2 The comparison shows that the human eye data changes of the present invention are smoother, thereby suppressing flickering and improving stable viewing.

[0041] (2) Simulation comparison before and after crosstalk windowing ( Figure 3 ): By inputting a stereo video with a completely black left side and a completely white right side, the phase formula of sub-pixels within the period is used. Determine which view to choose between. (During simulation...) exist Scan within the range and calculate the output results. For each group Calculate two output schemes separately: the unwindowed scheme and the scheme based solely on... The phase decision threshold is used to perform hard switching between left and right views for output; windowing scheme: the half-cycle center is determined after the decision. Flat roof width parameters w Set it to 0.15 and calculate the phase distance. Brightness weight window function Applying energy to sub-pixels near the boundary yields an energy-shaped output. .

[0042] The crosstalk rate (CT) is calculated as follows: It is based on the average output brightness of the target view within the corresponding viewport. The average leakage brightness of the non-target view is ,but For each Calculate CT values ​​separately and plot them. Comparison curves. Figure 3 This indicates that, compared to the unwindowed scheme, the windowed scheme achieves a more continuous energy transition at the phase boundary and significantly reduces boundary energy leakage. Therefore, it can effectively reduce CT within one fringe cycle, thereby mitigating crosstalk and flip scintillation.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A human eye tracking low cross-talk high stability naked eye 3D playing method, characterized in that: Includes the following steps: S1: Decode the input stereo video and write the decoded frames to an external texture associated with the GPU rendering thread; S2: Acquire images from the front-facing camera and perform eye tracking to obtain the position of the left eye. and the right eye position ; S3: Perform stability enhancement processing on the eye position data to generate a stable eye position, wherein when the eye position data is invalid, the previous stable eye position is maintained, and when the eye position data is valid, temporal smoothing is performed. S4: Generate stripe display control information based on the stable eye position, and transmit the control information to the GPU; S5: In the fragment shader, the left and right views are sampled from the stereo video frame texture respectively, and the stripe axis coordinates and sub-pixel phase within the period are constructed to determine whether each sub-pixel output comes from the left or right view. S6: After the decision is made, the stripe phase boundary region is determined based on the sub-pixel phase within the period, and the luminance weight window function is applied to the sub-pixel output near the stripe phase boundary to perform energy shaping according to the phase boundary energy shaping parameters generated in S3, so as to reduce crosstalk and flip flicker. The stable eye position output in S3 and the phase boundary energy shaping parameters are generated together in the same control period.

2. The eye-tracking, low-crosstalk, high-stability naked-eye 3D playback method according to claim 1, characterized in that: In S5, for texture coordinates The left and right view samplings satisfy: Left view texture coordinates Right view texture coordinates Get the color of the left view Color in the right view .

3. The eye-tracking, low-crosstalk, high-stability naked-eye 3D playback method according to claim 1, characterized in that: In S5, the screen pixel coordinates Constructing stripe axis coordinates satisfy: in, s For the fringe tilt coefficient, For calibrating reference points, β The scaling factor is the ratio from pixel to sub-pixel. O This represents the phase offset.

4. The eye-tracking, low-crosstalk, high-stability naked-eye 3D playback method according to claim 3, characterized in that: In S5, for each sub-pixel channel Calculate sub-pixel phase within the period ,in , c For stripe period control information; And based on r m Perform left and right view sampling decision: when At that time, sub-pixel output ;when At that time, sub-pixel output .

5. The eye-tracking, low-crosstalk, high-stability naked-eye 3D playback method according to claim 1, characterized in that: In S3, the stability enhancement process includes: S31: Construct a combined confidence level for the current frame eye position observation data, wherein the combined confidence level is based on at least one or more of the following: facial occlusion analysis and the distance of the change in eye position before and after; S32: When the combined confidence level is below the threshold, key points are missing, or the eye position displacement amplitude is greater than the jump threshold, the eye position data is deemed invalid and failure preservation is executed to satisfy the requirement. ; S33: When eye position data is valid, observe eye position Perform exponential smoothing to satisfy ,in To observe eye position vector, For the current stable eye position, For the previous stable eye position, and For smoothing coefficients, .

6. The eye-tracking, low-crosstalk, high-stability naked-eye 3D playback method according to claim 4, characterized in that: In step S6, the half-cycle center is calculated. Phase distance ; when hour, ; when hour, ; 。 7. The eye-tracking, low-crosstalk, high-stability naked-eye 3D playback method according to claim 6, characterized in that: In step S6, a luminance weighted window function is used. for: when hour, ; when When defining normalized variables and adopt ; when hour, ; Modulate the sub-pixel output to ,in w This refers to the width parameter of the flat top.

8. The eye-tracking, low-crosstalk, high-stability naked-eye 3D playback method according to claim 1, characterized in that: In S1, the external texture is bound to the texture input object by the video decoding output, and the current decoded frame is provided to the fragment shader of S5 through frame-by-frame texture update.

9. A human eye-tracking, low-crosstalk, high-stability naked-eye 3D playback system, characterized in that: include: The video decoding and texture input module is used to decode stereo video and write the decoded frames into textures; The human eye tracking module is used to acquire images and track the human eye to obtain the eye position; The tracking stability enhancement module is used to perform stability enhancement processing on eye position data to generate stable eye positions; An adaptive control module is used to generate stripe display control information based on the stable eye position; The GPU stripe composition module is used to synthesize stripes in the fragment shader. The phase boundary energy shaping module, integrated in the GPU stripe synthesis module, is used to perform energy shaping on the sub-pixel output.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 8.

Images