Naked eye 3D real-time dynamic pupil tracking system based on double cameras

Through the dual camera system combined with multiple algorithms, high-precision pupil tracking and viewpoint prediction in complex environments is achieved, solving the problem of insufficient pupil tracking accuracy and stability in the existing technology, and providing a smooth naked-eye 3D visual experience.

CN119011806AInactive Publication Date: 2024-11-22SHENZHEN EASYQUICK TECH CO LTD

Patent Information

Application Number
CN202411503364.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-25
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In existing naked-eye 3D display systems, single cameras or low-resolution sensors are difficult to provide high-precision three-dimensional position and micro-movement data of the pupil, resulting in low pupil tracking accuracy and stability under fast motion or complex lighting conditions, which cannot meet high requirements. viewpoint tracking.

Method used

Using a dual camera real-time dynamic pupil tracking system, combined with a pupil detection module, feature enhancement recognition module, dual camera positioning module and eyeball micro-movement prediction and correction module, it accurately uses edge detection, convolutional neural network, binocular vision principles and recursive neural network. Extract pupil morphological characteristics, handle light interference and individual differences, predict viewpoint changes and dynamically adjust the display content.

Benefits of technology

Improves the accuracy and real-time response of pupil tracking, improves lag problems in complex environments, provides a smooth and realistic 3D visual experience, ensuring the stability and robustness of the system under different lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011806A_ABST
    Figure CN119011806A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of naked eye 3D, in particular to a naked eye 3D real-time dynamic pupil tracking system based on double cameras, which comprises a pupil detection module, a pupil feature enhancement recognition module, a double-camera positioning module, an eyeball micro-motion prediction and correction module and an image output module, the pupil detection module captures eye images of a user in real time through double cameras; the pupil feature enhancement recognition module performs feature enhancement on the extracted pupil morphological features; the dual-camera positioning module determines the three-dimensional position of the pupil of the user; the eyeball micro-motion prediction and correction module is used for predicting the change trend of a viewpoint; and the image output module outputs the corrected image to the naked eye 3D display equipment. According to the pupil tracking method and device, recognition errors caused by light interference and rapid movement of a user are effectively processed, the pupil tracking accuracy and the real-time response capability are improved, and the lag problem in a complex environment is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of naked-eye 3D technology, and in particular to a real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D. Background Art

[0002] With the rapid development of naked-eye 3D display technology, how to enhance users' natural and realistic 3D visual experience without wearing auxiliary equipment has become the focus of research and application. In naked-eye 3D display systems, viewpoint tracking technology is one of the keys. By capturing the user's viewpoint position in real time and dynamically adjusting the display content, the system can present a 3D effect that matches the user's viewing position based on the user's perspective. However, accurate and stable acquisition of the user's viewpoint information, especially the three-dimensional position and micro-motion data of the pupil, directly affects the accuracy of the displayed content and the user's visual comfort. Therefore, pupil tracking technology plays a vital role in naked-eye 3D display systems.

[0003] At present, pupil tracking technology on the market mainly relies on a single camera or a low-resolution sensor to obtain the pupil position. However, a single-camera system usually cannot provide high-precision depth information, resulting in low accuracy and stability of pupil tracking under fast motion or complex lighting conditions, especially in naked-eye 3D scenes. It is impossible to adjust the display content in real time and accurately. In addition, the existing technology is prone to recognition errors and lags when dealing with individual differences (such as pupil size, iris texture, etc.) and light interference (such as strong light or weak light environment), and cannot meet the high requirements of naked-eye 3D systems for viewpoint tracking. Therefore, improving the real-time and accuracy of pupil tracking, as well as its adaptability under different lighting conditions, has become a technical problem that needs to be solved urgently.

[0004] In view of the deficiencies of the above-mentioned prior art, the present invention provides a real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D, which can effectively improve the accuracy and stability of pupil recognition under different lighting conditions and individual differences of users, thereby providing users with a smooth and realistic naked-eye 3D visual experience. Summary of the invention

[0005] The invention provides a real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D.

[0006] A real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D includes a pupil detection module, a pupil feature enhancement recognition module, a dual-camera positioning module, an eyeball micro-movement prediction and correction module, and an image output module, wherein; The pupil detection module captures the user's eye images in real time through dual cameras, and extracts the morphological features of the pupil using edge detection and image processing algorithms; The pupil feature enhancement recognition module combines the multi-level features of the iris and the surrounding pupil (iris texture, orbital contour) to enhance the extracted pupil morphological features to solve the recognition errors caused by individual differences or light interference; The dual-camera positioning module is based on the binocular vision principle and determines the three-dimensional position of the user's pupil by calculating the images captured by the dual cameras at different viewing angles; The eyeball micro-movement prediction and correction module predicts the viewpoint change trend based on the user's eyeball micro-movement data, predicts the user's viewpoint change in advance, and performs dynamic correction by adjusting the naked eye 3D display content in real time, specifically including: Eye micro-movement data collection: dual cameras are used to capture the user's eye micro-movement data, including eye displacement, speed and acceleration; Viewpoint change trend prediction: The recurrent neural network (RNN) model is used to analyze the user's eye micro-movement data and historical viewpoint trajectory to predict the user's viewpoint change trend; Dynamic correction and content adjustment: adjust naked-eye 3D display content in real time based on the prediction results of viewpoint change trends; The image output module outputs the corrected image to the naked-eye 3D display device, and saves the display content and changes synchronously with the user's actual viewpoint, thereby achieving a seamless naked-eye 3D display effect.

[0007] Optionally, the pupil detection module includes: Image capture: Dual cameras synchronously capture the user's eye images, recording the visual information of the left eye and the right eye respectively; Edge detection: The captured eye image is processed using the Canny edge detection algorithm to identify the edge of the pupil; Morphological feature extraction: The morphological features of the pupil, including position, diameter and roundness, are extracted through the Hough circle transform algorithm.

[0008] Optionally, the pupil feature enhancement recognition module includes: Multi-level feature extraction: Convolutional neural network (CNN) is used to extract multi-level features around the iris and pupil, including iris texture and orbital contour; Feature fusion and enhancement: The extracted multi-level features and morphological features are fused into comprehensive image features, and an adaptive contrast enhancement algorithm is used to dynamically adjust the enhancement strength of the comprehensive image features according to different lighting conditions and individual user differences. Feature correction: Correct the comprehensive image features after intensity enhancement to correct potential errors caused by light interference or unclear pupil features.

[0009] Optionally, the multi-level feature extraction includes: Image input and convolution processing: The captured eye image is input into the convolutional neural network for processing. Each convolution layer extracts local features through filters. Feature pooling and dimensionality reduction: After the convolution operation, the extracted features are reduced in dimensionality through the maximum pooling layer; Multi-level feature output: The fully connected layer classifies the features after convolution and pooling through the Softmax function, outputs multi-level feature information, including the morphological characteristics of the pupil edge, iris texture and orbital contour, and uses the cross entropy loss function for optimization.

[0010] Optionally, the feature fusion and enhancement includes: Feature fusion: The extracted multi-level features and morphological characteristics Perform weighted fusion to form comprehensive image features ; Adaptive contrast enhancement: Based on the integrated image features after fusion, an adaptive contrast enhancement algorithm is used to enhance the contrast and clarity of the features; Dynamic adjustment: Dynamically adjust the contrast enhancement factor according to changes in lighting and differences in individual user pupil characteristics, including increasing the contrast enhancement factor in low light conditions (below 50 lux), reducing the contrast enhancement factor in strong light conditions (above 500 lux), and increasing the contrast enhancement factor when the pupil outline is blurred.

[0011] Optionally, the feature modification includes: Noise filtering and smoothing: Smoothing the enhanced image features through the median filtering algorithm; Local contrast correction: After image feature enhancement, correction is performed based on the local contrast of the image, and adaptive histogram equalization is used to dynamically adjust the contrast of different areas; Morphological correction: The image features after local contrast correction are corrected through morphological dilation and erosion operations.

[0012] Optionally, the dual-camera positioning module includes: Image acquisition: Dual cameras synchronously capture images of the user's pupils at different viewing angles, obtaining two-dimensional image data of the left and right eyes, including the position of the pupils at each viewing angle; Feature matching: Use the ORB feature matching algorithm to find the corresponding points of the pupil positions in the left and right view images; Depth calculation and three-dimensional reconstruction: Based on the principle of binocular vision, through triangulation, using the internal and external parameters of the camera and the parallax between the two perspectives (that is, the position offset of the pupil in different perspectives), the depth information of the pupil is calculated to determine its position in three-dimensional space.

[0013] Optionally, the eye micro-movement data collection includes: Displacement data collection: Use dual cameras to capture real-time images of the user's left and right eyes respectively, and use stereo vision technology to detect changes in the position of the pupil in the image; Speed ​​data calculation: The speed of the eyeball is obtained by calculating the displacement change per unit time. ; Acceleration data calculation: Calculate the acceleration of the eyeball by changing the speed over two consecutive time periods .

[0014] Optionally, the recurrent neural network (RNN) model includes: Input data construction: Integrate the historical viewpoint trajectory with the current eyeball displacement, velocity, and acceleration as input data; Hidden state update: During the hidden state update process, the recurrent neural network (RNN) model processes the input data of each time step, combines the current eye micro-movement data with the hidden state of the previous moment for update, and highlights the role of the current eye micro-movement data and the historical viewpoint trajectory by optimizing the weighting mechanism to capture small viewpoint changes; Dynamic learning rate adjustment: In order to adapt to the different stages of viewpoint change prediction, a dynamic learning rate adjustment mechanism is introduced; Output prediction: The output layer predicts the changing trend of the viewpoint position and combines the historical data of eye movement to predict the next time point. The viewpoint position; Error calculation and optimization: The mean square error (MSE) is used as the loss function for optimization to calculate the error between the predicted viewpoint position and the actual position.

[0015] Optionally, the dynamic correction and content adjustment include: Real-time data input: Determine the future viewpoint position and direction by obtaining the user's viewpoint change trend results; Display content adjustment: Based on the predicted viewpoint position, the naked-eye 3D display content is dynamically corrected. The correction process includes adjusting the image depth and viewing angle.

[0016] Beneficial effects of the present invention: The present invention uses dual cameras to capture images of the user's pupils in real time, and combines a pupil detection module and a feature enhancement recognition module to accurately extract pupil morphological features under different lighting conditions and individual differences. Through the Hough circle transform and convolutional neural network algorithm, it can effectively handle recognition errors caused by light interference and rapid movement of the user, thereby improving the accuracy and real-time response capability of pupil tracking and improving the lag problem in complex environments.

[0017] The present invention, by introducing a recursive neural network model and combining the user's eye micro-movement data and historical viewpoint trajectory, achieves high-precision viewpoint change trend prediction, and highlights the eye movement data at the most recent moment by optimizing the weighting mechanism, thereby accurately capturing the user's short-term viewpoint changes. At the same time, the dynamic correction and content adjustment module can automatically adjust the 3D display content according to the viewpoint change results, ensuring that when the user moves quickly, the angle and depth of the displayed content are adjusted synchronously, providing a seamless 3D visual experience.

[0018] The present invention can dynamically adjust the contrast and clarity of pupil features according to different lighting conditions and individual differences of users through adaptive contrast enhancement and feature correction, effectively solving the recognition error caused by uneven light or blurred pupil. In addition, the dual-camera positioning module combines parallax calculation and triangulation to accurately obtain the three-dimensional position of the user's pupil, ensuring that the system can quickly and accurately dynamically adjust the naked-eye 3D display content, thereby improving the overall visual experience and the robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of system function modules according to an embodiment of the present invention; Figure 2 Schematic diagram of an eye movement prediction and correction module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0022] It should be noted that the references to "one embodiment", "embodiment", "exemplary embodiments", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0023] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0024] like Figure 1-Figure 2 As shown, a real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D includes a pupil detection module, a pupil feature enhancement recognition module, a dual-camera positioning module, an eye micro-movement prediction and correction module, and an image output module, wherein; The pupil detection module uses dual cameras to capture the user's eye images in real time and uses edge detection and image processing algorithms to extract the morphological features of the pupils; The pupil feature enhancement recognition module combines the multi-level features of the iris and pupil surroundings (iris texture, orbital contour) to enhance the pupil morphological features extracted by the pupil detection module, thus solving the recognition errors caused by individual differences or light interference. The dual-camera positioning module is based on the principle of binocular vision. It determines the three-dimensional position of the user's pupil by calculating the images captured by the dual cameras at different viewing angles. The eye movement prediction and correction module predicts the viewpoint change trend based on the user's eye movement data, predicts the user's viewpoint change in advance, and performs dynamic correction by adjusting the naked eye 3D display content in real time. Specifically, it includes: Eye micro-movement data collection: dual cameras are used to capture the user's eye micro-movement data, including eye displacement, speed and acceleration; Viewpoint change trend prediction: The recurrent neural network (RNN) model is used to analyze the user's eye micro-movement data and historical viewpoint trajectory to predict the user's viewpoint change trend; Dynamic correction and content adjustment: adjust naked-eye 3D display content in real time based on the prediction results of viewpoint change trends; The image output module outputs the corrected image to the naked-eye 3D display device and saves the display content to synchronize with the user's actual viewpoint, achieving a seamless naked-eye 3D display effect; Through the above content, high-precision pupil tracking and 3D display synchronization adjustment can be achieved under complex lighting and fast user movement, which improves the system's real-time response capability and pupil recognition accuracy, ensuring that users can get a smooth and clear naked-eye 3D visual experience in different scenarios, and greatly improving the common lag and recognition error problems in traditional technologies.

[0025] The pupil detection module includes: Image capture: Dual cameras synchronously capture the user's eye images, recording the visual information of the left eye and the right eye respectively; Edge detection: The captured eye image is processed using the Canny edge detection algorithm to identify the edge of the pupil, expressed as: ; in, and Represent the gradient of the image in the horizontal and vertical directions respectively, Represents the comprehensive gradient magnitude, which is used to detect edge positions in images; Morphological feature extraction: The morphological features of the pupil are extracted using the Hough circle transform algorithm, including position, diameter, and roundness, expressed as: ; in, and Represents the coordinates of any pixel in the image. is the pupil center coordinate, is the pupil radius; Through the above content, the morphological features of the pupil, such as position, diameter and roundness, are effectively extracted, and high-precision pupil recognition can be maintained under different lighting conditions. Even if the user moves quickly or uses it in a complex environment, the real-time and stability of the system can be guaranteed, which greatly improves the accuracy of pupil tracking and the response speed of the system.

[0026] The pupil feature enhancement recognition module includes: Multi-level feature extraction: Convolutional neural network (CNN) is used to extract multi-level features around the iris and pupil, including iris texture and orbital contour; Feature fusion and enhancement: The extracted multi-level features and morphological features are fused into comprehensive image features, and an adaptive contrast enhancement algorithm is used to dynamically adjust the enhancement strength of the comprehensive image features according to different lighting conditions and individual user differences. Feature correction: Correct the comprehensive image features after intensity enhancement to correct potential errors caused by light interference or unclear pupil features; Through the above content, the recognition accuracy of the system in complex lighting environments and individual differences is improved. Through adaptive contrast enhancement and feature correction algorithms, the clarity of pupil features can be dynamically adjusted to effectively solve the recognition errors caused by light interference or unclear pupil features, thereby ensuring the stability and accuracy of pupil tracking.

[0027] Multi-level feature extraction includes: Image input and convolution processing: The captured eye image is input into the convolutional neural network for processing. Each convolution layer extracts local features through filters, which can be expressed as: in, is the input image at The pixel value of is the weight matrix of the convolution kernel, is the local eigenvalue extracted after convolution, and is the size of the convolution kernel, indicating the dimension of the convolution kernel (such as a 3x3 or 5x5 convolution kernel); Feature pooling and dimensionality reduction: After the convolution operation, the extracted features are reduced in dimensionality through the maximum pooling layer to retain key feature information and reduce the amount of data, which is expressed as: ; in, is the feature value after the pooling operation, indicating the maximum value extracted within the pooling window. After the convolution operation The eigenvalue at the position; Multi-level feature output: The fully connected layer classifies the features after convolution and pooling through the Softmax function, outputs multi-level feature information, including the morphological features of the pupil edge, iris texture, and orbital contour, and uses the cross entropy loss function for optimization, which is expressed as: ; in, Indicates The value of the output neuron (also called the activation value), which is the final classification score, Indicates The neuron and the input Features The weight between Indicates the input eigenvalues ​​(usually the output of the previous layer, the input vector after the flattening operation), Indicates The bias term of a neuron, is the number of input features; ; in, Indicates The predicted probability of the class, is the first fully connected layer output values, Indicates the total number of categories; ; in, is the loss value, which is used to measure the difference between the model prediction value and the true value. is the total number of samples, is the true label, indicating the sample The true classification label (usually 0 or 1), is the model prediction value, indicating the model's prediction of the sample The predicted probability of belonging to a certain category; Through the above content, we deeply explore the texture and contour features around the iris and pupil, and combine multi-layer convolution and pooling operations to not only capture local details but also retain global structural information. By using feature fusion and enhancement, the system can automatically adapt to individual differences under different lighting conditions, and further optimize the classification results through the fully connected layer, thereby improving the accuracy and robustness of pupil recognition, and effectively solving the recognition error problem of traditional methods in complex environments.

[0028] Feature fusion and enhancement include: Feature fusion: The extracted multi-level features and morphological characteristics Perform weighted fusion to form comprehensive image features , expressed as: ; in, and is the weight coefficient, which controls the proportion of multi-level features and morphological features in the comprehensive image features; Adaptive contrast enhancement: Based on the fused comprehensive image features, an adaptive contrast enhancement algorithm is used to enhance the contrast and clarity of the features, which is expressed as: ; in, Represents the image feature value after adaptive contrast enhancement, usually the enhanced pixel intensity or brightness value. is the fusion comprehensive image feature value, and are the minimum and maximum values ​​of the image, respectively. It is an adaptive contrast enhancement factor that can be adjusted according to different lighting conditions to improve the visual effect of the image; Dynamic adjustment: Dynamically adjust the contrast enhancement factor according to changes in lighting and individual user pupil characteristics, including increasing the contrast enhancement factor in low light conditions (below 50 lux), reducing the contrast enhancement factor in strong light conditions (above 500 lux), and increasing the contrast enhancement factor when the pupil outline is blurred; Through the above content, the clarity and contrast of the image can be effectively improved, especially in cases where the lighting conditions are complex or there are large individual differences among users. The contrast enhancement coefficient can be dynamically adjusted to ensure that the pupil features are accurately extracted in low-light or strong-light environments, thereby improving the recognition errors caused by light interference or pupil blur, and improving the robustness and accuracy of the system.

[0029] Feature modifications include: Noise filtering and smoothing: The enhanced image features are smoothed by the median filtering algorithm to eliminate the interference caused by noise and light in the image, which is expressed as: ; in, is the pixel value after median filtering. is the pixel value in the enhanced image, are the coordinates of the neighborhood pixels; Local contrast correction: After image feature enhancement, correction is performed based on the local contrast of the image. Adaptive histogram equalization is used to dynamically adjust the contrast of different areas to ensure that the pupil contour is more obvious, especially when the pupil edge is blurred or the lighting is uneven. By enhancing the contrast, the recognition accuracy of the pupil morphology is improved, which is expressed as: ; in, and are the minimum and maximum values ​​in the local area, respectively. is the image after local contrast correction; Morphological correction: Through morphological dilation and erosion operations, the image features after local contrast correction are corrected to eliminate the pseudo edges caused by uneven illumination or enhancement process, so that the final extracted pupil morphology is closer to the real features, expressed as: ; ; in, The image after the expansion operation is at position The pixel value at is a structural element, indicating the neighborhood range covered by the dilation or erosion operation. is the image after local contrast correction at position The pixel value at The image after the corrosion operation is at position The pixel value at ; Through the above content, the enhanced comprehensive image features are effectively corrected and the pseudo edges caused by uneven lighting or noise are eliminated, thereby ensuring that the pupil contour is clearer and more realistic. It can significantly improve the robustness of the system under complex lighting conditions, reduce recognition errors, enhance image stability and recognition accuracy, and make the extraction of pupil features more accurate, especially in the presence of interference or blurred edges, to optimize the final recognition effect.

[0030] The dual-camera positioning module includes: Image acquisition: Dual cameras synchronously capture images of the user's pupils at different viewing angles, obtaining two-dimensional image data of the left and right eyes, including the position of the pupils at each viewing angle; Feature matching: Use the ORB feature matching algorithm to find the corresponding points of the pupil positions in the left and right view images to ensure that the pupil positions in the two view images can be accurately aligned for depth calculation; Depth calculation and 3D reconstruction: Based on the principle of binocular vision, through triangulation, using the internal and external parameters of the camera and the parallax between the two perspectives (i.e. the position offset of the pupil in different perspectives), the depth information of the pupil is calculated to determine its position in 3D space, which is expressed as: ; in, It's parallax. and are the horizontal coordinate positions of the pupil in the left camera and right camera images respectively; ; in, is the depth of the pupil (distance from the camera), is the focal length of the camera, is the baseline distance between the two cameras (i.e. the physical distance between the two cameras), It’s parallax; ; ; in, and are the horizontal and vertical coordinates of the pupil in three-dimensional space, and is the 2D position of the pupil in the left camera image, and They are the optical center coordinates of the camera (the coordinates of the center point of the image), is the depth of the pupil; The ORB feature matching algorithm specifically includes: Feature detection (FAST corner detection): detect corners by comparing pixel values, with a pixel As the center, compare the 16 pixels around it (forming a circle), the detection conditions of the corner point are: ; in, is a user-defined threshold, is the gray value of the center pixel. If more than 12 adjacent pixels meet this condition, the pixel is considered a corner point. Feature Descriptor (BRIEF): Use randomly generated pixel pairs to construct binary descriptors, given two pixel locations and , the BRIEF descriptor is defined as: ; in, are two pixels in the image and The comparison results between Represents pixel The gray value of Represents pixel Gray value of Direction calculation: In order to ensure rotation invariance, ORB calculates the direction of each feature point so that the matching process is not affected by image rotation. The direction calculation uses the centroid formula, which is expressed as: ; in, is the angular direction, is the pixel grayscale value in the neighborhood of the feature point, and are the position coordinates of the neighborhood pixels, represents probability; Feature matching (Hamming distance): In the feature matching stage, ORB uses the Hamming distance to compare the BRIEF descriptors in the two images. The Hamming distance is expressed as: ; in, and is a binary vector of two descriptors, Represents the exclusive OR operation, is the length of the descriptor; User defined thresholds Adaptively set according to the lighting conditions of the image and the morphological characteristics of the pupil, including: Image grayscale mean: Calculate the grayscale mean of the entire eye image , which is used to reflect the overall brightness of the current image and is expressed as: ; in, is the total number of pixels in the image, Indicates the image The gray value of pixels; Image grayscale standard deviation: Calculate the grayscale standard deviation of the image , which is used to measure the discrete degree of image brightness distribution, is expressed as: ; Threshold setting: based on the calculated grayscale mean and standard deviation , set by empirical formula , expressed as: ; in, is a constant parameter used to adjust the influence of standard deviation on threshold setting. ), is a fixed offset used to ensure that the pupil edge can be correctly detected even in low-contrast images. It is set to a small positive value ( ); Through the above content, high-precision tracking of pupil position is ensured. Compared with a single-camera system, the dual camera can capture the position information of the user's pupil at different viewing angles, and obtain depth information through parallax and triangulation, which is crucial for improving naked-eye 3D display. It ensures that the system can quickly and accurately adjust the 3D display content and provide a more natural visual experience, especially in dynamic scenes, it can maintain real-time response and visual stability.

[0031] Eye movement data collection includes: Displacement data collection: The real-time images of the user's left and right eyes are captured by dual cameras, and the position change of the pupil in the image is detected using stereo vision technology, which is expressed as: ; ; in, and are the changes in the horizontal and vertical directions of the eye position per unit time, Indicates time The pupil position at the moment, Indicates time Pupil position at the moment; Speed ​​data calculation: The speed of the eyeball is obtained by calculating the displacement change per unit time. , expressed as: ; in, Indicates the time difference between two time points; Acceleration data calculation: Calculate the acceleration of the eyeball by changing the speed over two consecutive time periods , expressed as: ; in, and Respectively indicate time and time Eye speed at any moment, Indicates a time interval; Through the above content, the user's eye displacement, velocity and acceleration data can be accurately captured, and the tiny movements of the eye can be tracked in real time using stereo vision technology, ensuring that the system can accurately capture the user's viewpoint changes in various usage scenarios. Even in the case of fast movement or slight rotation, it can still provide stable tracking effects. By accurately calculating the movement characteristics of the eye, the system can better predict and adjust the display content, thereby improving the user's visual experience and tracking accuracy.

[0032] The Recurrent Neural Network (RNN) model consists of: Input data construction: The historical viewpoint trajectory and the current eyeball displacement, velocity, and acceleration are integrated as input data, expressed as: ; in, It's time Input features at each moment, including eye movement ,speed , acceleration , and history The viewpoint position at a moment ; Hidden state update: During the hidden state update process, the recurrent neural network (RNN) model processes the input data of each time step, combines the current eye micro-movement data with the hidden state of the previous moment for update, and optimizes the weighting mechanism to highlight the role of the current eye micro-movement data and the historical viewpoint trajectory, capturing small viewpoint changes, which is expressed as: ; in, For time The hidden state of the moment, is the weight matrix used to adjust the current input To weight, is the weight matrix used to hide the state at the previous moment To weight, represents the hidden state at the previous moment, is the bias term; Dynamic learning rate adjustment: In order to adapt to the different stages of viewpoint change prediction, a dynamic learning rate adjustment mechanism is introduced to help the model make corresponding learning rate adjustments at different training stages to avoid overfitting or slow learning speed, which is expressed as: ; in, It's time The learning rate at is the initial learning rate, is the attenuation factor; Output prediction: The output layer predicts the changing trend of the viewpoint position and combines the historical data of eye movement to predict the next time point. The viewpoint position is expressed as: ; in, represents the predicted viewpoint position, is the weight matrix of the output layer, is the hidden state at the current moment, is the bias term of the output layer; Error calculation and optimization: Use mean square error (MSE) as the loss function for optimization and calculate the error between the predicted viewpoint position and the actual position, expressed as: ; in, is the predicted viewpoint position, is the actual viewpoint position, is the number of time steps; Through the above content, the accuracy and response speed of viewpoint change trend prediction can be improved. By introducing the combination of historical viewpoint trajectory and current eye movement data, the user's viewpoint changes in the short term can be better captured. At the same time, it ensures that the viewpoint data at the most recent moment is given greater weight, thereby improving the accuracy of the prediction. It is more stable and flexible in complex naked-eye 3D scenes, can better adapt to the needs of real-time dynamic environments, and provide a smooth user experience.

[0033] Dynamic correction and content adjustment include: Real-time data input: Determine the future viewpoint position and direction by obtaining the user's viewpoint change trend results; Display content adjustment: Based on the predicted viewpoint position, the naked-eye 3D display content is dynamically corrected to ensure that the angle and position of the display content are consistent with the user's viewpoint. The correction process includes adjusting the image depth and viewing angle to ensure that the user sees the most natural and clear 3D effect from the current viewpoint; Through the above content, the viewing angle and depth of the naked-eye 3D display content are automatically adjusted to ensure that users can still obtain clear and realistic 3D visual effects when moving the viewpoint, and to provide a seamless display experience. Even in the case of fast movement or frequent switching of viewpoints, the 3D effect remains stable. This dynamic adjustment greatly enhances the user's immersion and visual comfort, and is particularly suitable for application scenarios with high requirements for precise viewpoint tracking.

[0034] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0035] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D. It is characterized in that It includes a pupil detection module, a pupil feature enhancement recognition module, a dual-camera positioning module, an eye micro-movement prediction and correction module, and an image output module, wherein; The pupil detection module captures the user's eye images in real time through dual cameras, and extracts the morphological features of the pupil using edge detection and image processing algorithms; The pupil feature enhancement recognition module combines the multi-level features of the iris and the pupil surrounding to enhance the extracted pupil morphological features, thereby solving the recognition errors caused by individual differences or light interference; The dual-camera positioning module is based on the binocular vision principle and determines the three-dimensional position of the user's pupil by calculating the images captured by the dual cameras at different viewing angles; The eyeball micro-movement prediction and correction module predicts the viewpoint change trend based on the user's eyeball micro-movement data, predicts the user's viewpoint change in advance, and performs dynamic correction by adjusting the naked eye 3D display content in real time, specifically including: Eye micro-movement data collection: dual cameras are used to capture the user's eye micro-movement data, including eye displacement, speed and acceleration; Viewpoint change trend prediction: The recursive neural network model is used to analyze the user's eye micro-movement data and historical viewpoint trajectory to predict the user's viewpoint change trend; Dynamic correction and content adjustment: adjust naked-eye 3D display content in real time based on the prediction results of viewpoint change trends; The image output module outputs the corrected image to the naked-eye 3D display device, and saves the display content and changes synchronously with the user's actual viewpoint, thereby achieving a seamless naked-eye 3D display effect.

2. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 1, It is characterized in that The pupil detection module comprises: Image capture: Dual cameras synchronously capture the user's eye images, recording the visual information of the left eye and the right eye respectively; Edge detection: The captured eye image is processed using the Canny edge detection algorithm to identify the edge of the pupil; Morphological feature extraction: The morphological features of the pupil, including position, diameter and roundness, are extracted through the Hough circle transform algorithm.

3. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 2, It is characterized in that The pupil feature enhancement recognition module comprises: Multi-level feature extraction: Convolutional neural network is used to extract multi-level features around the iris and pupil, including iris texture and orbital contour; Feature fusion and enhancement: The extracted multi-level features and morphological features are fused into comprehensive image features, and an adaptive contrast enhancement algorithm is used to dynamically adjust the enhancement strength of the comprehensive image features according to different lighting conditions and individual user differences. Feature correction: Correct the comprehensive image features after intensity enhancement to correct potential errors caused by light interference or unclear pupil features.

4. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 3, It is characterized in that The multi-level feature extraction includes: Image input and convolution processing: The captured eye image is input into the convolutional neural network for processing. Each convolution layer extracts local features through filters. Feature pooling and dimensionality reduction: After the convolution operation, the extracted features are reduced in dimensionality through the maximum pooling layer; Multi-level feature output: The fully connected layer classifies the features after convolution and pooling through the Softmax function, outputs multi-level feature information, including the morphological characteristics of the pupil edge, iris texture and orbital contour, and uses the cross entropy loss function for optimization.

5. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 4, It is characterized in that The feature fusion and enhancement include: Feature fusion: The extracted multi-level features and morphological characteristics Perform weighted fusion to form comprehensive image features ; Adaptive contrast enhancement: Based on the integrated image features after fusion, an adaptive contrast enhancement algorithm is used to enhance the contrast and clarity of the features; Dynamic adjustment: Dynamically adjust the contrast enhancement factor based on lighting changes and individual user pupil feature differences, including increasing the contrast enhancement factor in low-light conditions, reducing the contrast enhancement factor in strong light conditions, and increasing the contrast enhancement factor when the pupil outline is blurred.

6. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 5, It is characterized in that The feature modification includes: Noise filtering and smoothing: Smoothing the enhanced image features through the median filtering algorithm; Local contrast correction: After image feature enhancement, correction is performed based on the local contrast of the image, and adaptive histogram equalization is used to dynamically adjust the contrast of different areas; Morphological correction: The image features after local contrast correction are corrected through morphological dilation and erosion operations.

7. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 1, It is characterized in that The dual-camera positioning module includes: Image acquisition: Dual cameras synchronously capture images of the user's pupils at different viewing angles, obtaining two-dimensional image data of the left and right eyes, including the position of the pupils at each viewing angle; Feature matching: Use the ORB feature matching algorithm to find the corresponding points of the pupil positions in the left and right view images; Depth calculation and 3D reconstruction: Based on the principle of binocular vision, through triangulation, using the internal and external parameters of the camera and the parallax between the two perspectives, the depth information of the pupil is calculated to determine its position in 3D space.

8. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 1, It is characterized in that The eye micro-movement data collection includes: Displacement data collection: Use dual cameras to capture real-time images of the user's left and right eyes respectively, and use stereo vision technology to detect changes in the position of the pupil in the image; Speed ​​data calculation: The speed of the eyeball is obtained by calculating the displacement change per unit time. ; Acceleration data calculation: Calculate the acceleration of the eyeball by changing the speed over two consecutive time periods .

9. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 8, It is characterized in that The recursive neural network model includes: Input data construction: Integrate the historical viewpoint trajectory with the current eyeball displacement, velocity, and acceleration as input data; Hidden state update: During the hidden state update process, the recursive neural network model processes the input data at each time step, combines the current eye micro-movement data with the hidden state at the previous moment for update, and highlights the role of the current eye micro-movement data and the historical viewpoint trajectory by optimizing the weighting mechanism to capture small viewpoint changes; Dynamic learning rate adjustment: In order to adapt to the different stages of viewpoint change prediction, a dynamic learning rate adjustment mechanism is introduced; Output prediction: The output layer predicts the changing trend of the viewpoint position and combines the historical data of eye movement to predict the next time point. The viewpoint position; Error calculation and optimization: Use mean square error as the loss function for optimization to calculate the error between the predicted viewpoint position and the actual position.

10. The real-time dynamic pupil tracking system based on dual cameras for naked-eye 3D according to claim 9, It is characterized in that The dynamic correction and content adjustment include: Real-time data input: Determine the future viewpoint position and direction by obtaining the user's viewpoint change trend results; Display content adjustment: Based on the predicted viewpoint position, the naked-eye 3D display content is dynamically corrected. The correction process includes adjusting the image depth and viewing angle.

Citation Information

Patent Citations

  • Terminal backlight adjustment method and device

    CN105679253A

  • Eyeball protrusion measuring method and system, terminal and medium

    CN115984203A

  • Pupil dynamic tracking method and system based on naked eye 3D

    CN116723308A

  • Naked eye 3D display method and device based on eyeball tracking

    CN117075335A

Cited By

  • Naked eye 3D real-time human eye pupil distance measuring and adjusting method

    CN119316583A

  • Dynamic vision training method and system based on naked eye 3D imaging, electronic equipment and storage medium

    CN119970454A

  • A dynamic vision training method, system, electronic device, and storage medium based on naked-eye 3D imaging.

    CN119970454B

  • Display terminal system based on camera and control method

    CN120711163A

  • Anti-unmanned aerial vehicle intelligent identification and tracking system based on multi-source data fusion

    CN120850103A