Dynamic virtual image plane adjusting device and method based on multi-image plane eye movement tracking

Abstract

A dynamic virtual image plane adjusting device based on multi-image-plane eye movement tracking belongs to the technical field of visual training, and a multi-image-plane eye movement tracking system captures an eye image of a user by using an infrared camera, processes the eye image to obtain an accurate fixation point coordinate, and transmits data to a control system; the control system receives the coordinates of the fixation point, analyzes the currently required image plane depth, generates a lens adjustment instruction, and sends the lens adjustment instruction to the lens-screen adjustment system; the lens-screen adjusting system drives the lens and the screen to perform physical adjustment, and meanwhile, the adjustment precision is monitored in real time by utilizing a sensing feedback mechanism to ensure that the position of a virtual image is matched with a user watching focus; and the display system is combined with depth information provided by the control system to dynamically adjust the animation picture, so that the animation picture and the gazing depth of the user are synchronously changed. The invention further provides a dynamic virtual image plane adjusting method based on multi-image-plane eye movement tracking. According to the invention, real-time tracking and dynamic adjustment of the visual focus of the user are realized, and the precise visual interaction requirement in a multi-depth scene is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual training, and in particular provides a dynamic virtual image plane adjustment device and method based on multi-image plane eye tracking. Background Art

[0002] In recent years, virtual reality (VR) and augmented reality (AR) technologies have achieved rapid development and have been widely used in many fields such as medicine, education, and entertainment. However, the technical limitations of existing VR / AR devices in terms of display methods are still quite significant, especially in terms of focal plane adjustment, where there are many bottlenecks. Most VR / AR devices on the market currently use a fixed focal plane design, which causes the image plane that the user is looking at in the virtual environment to always be at the same depth, and the image plane position cannot be dynamically adjusted according to changes in the user's gaze point. This fixed focal length design not only limits the authenticity of the immersive experience, but may also cause visual fatigue after long-term use, affecting the user's visual health.

[0003] Although some devices have zoom functions, their adjustment methods often rely on manual operation by users, which is cumbersome and difficult to achieve real-time, intelligent focal length adjustment. For example, some zoom devices require users to adjust the lens position through mechanical knobs or buttons, while a few devices with electronic focus functions rely on preset focus modes, which not only lack active feedback mechanisms, but also lack dynamic adjustment functions, making it difficult to adapt to users' personalized needs in different scenarios.

[0004] At present, eye tracking technology is mainly used in the fields of gaze position analysis and interactive control, but has not been applied to dynamic focus adjustment. Mainstream eye tracking methods include pupil positioning technology based on infrared cameras and gaze prediction methods based on machine learning, both of which have made great progress in accuracy and real-time performance. However, these technologies have not yet been effectively combined, and their application in optimizing the visual experience of virtual reality devices, especially intelligent applications in focus adjustment, is still in its infancy.

[0005] On the other hand, multi-image display technology is a method proposed in recent years to relieve visual fatigue. Its core principle is to generate multiple virtual image planes at different depths, so that users can get a more realistic visual experience when viewing virtual objects at different depths. However, the implementation of this technology often relies on complex optical designs, such as multi-layer display screens, optical layering devices, or dynamic reconstruction systems based on light fields. The application of these solutions in existing devices is limited, mainly due to their complex hardware structure and high computational cost. At the same time, there is still a lot of room for optimization in terms of synchronization with changes in user vision and coordination with dynamic adjustment of the image plane.

[0006] The analysis of the existing relevant patents is as follows: Compared with patented technology CN209014753U (variable focus lens and VR device): it uses the change of liquid thickness in the elastic optical surface cavity to adjust the refractive index and realize the zoom function. However, its zoom method still relies on passive manual adjustment, requires a preset fixed focal length and focuses through analog circuits, and lacks active feedback based on the user's real-time gaze information.

[0007] Compared with the patented technology CN208823365U (a variable-focus VR eye vision instrument): it uses a motor drive to adjust the distance between the two lenses to change the optical path between the virtual image plane and the human eye. Although this method can achieve near and far vision adjustment to a certain extent, it still requires manual operation or relies on fixed time interval adjustment, and lacks adaptive adjustment to the user's real-time gaze needs.

[0008] Compared with patented technology CN114895793A (An active adaptive eye tracking method and AR glasses): it uses eye tracking to capture the user's gaze point, and combines the ToF (Time-of-Flight) sensor to detect the distance of the user's actual observation point to adjust the imaging position and focusing distance of the image module to avoid visual discomfort caused by frequent changes in virtual scenes. However, this method mainly focuses on the adjustment of the image projection position, and does not combine multi-image display technology to optimize the focal length adjustment.

[0009] Most devices on the market currently use a fixed focal plane display design, which can easily cause visual fatigue and discomfort when used for a long time. The main reason is that existing VR / AR devices usually only support a single fixed focal plane, and cannot dynamically adjust the position of the image plane according to the depth of the user's gaze point, or manually adjust the focus by the user. In this way, the image plane that the user is looking at in the virtual scene is always at the same depth, which cannot meet the natural focusing requirements of the line of sight. Long-term use will cause the lens to remain fixed continuously, increasing the visual burden and failing to meet the user's experience, thereby causing visual fatigue and adversely affecting eye health. Summary of the invention

[0010] In order to overcome the shortcomings of the existing technology, the present invention provides a dynamic virtual image plane adjustment device and method based on multi-image plane eye tracking, which realizes real-time tracking and dynamic adjustment of the user's visual focus through multi-system collaboration to meet the needs of precise visual interaction in multi-depth scenes, thereby dynamically adjusting the image plane position according to changes in the user's gaze point.

[0011] The technical solution adopted by the present invention to solve its technical problem is: A dynamic virtual image plane adjustment device based on multi-image plane eye tracking includes a multi-image plane eye tracking system, a lens-screen adjustment system, a control system and a display system. The multi-image eye tracking system uses an infrared camera to capture the user's eye image, and converts it into accurate gaze point coordinates through calculations by an image acquisition module, a pupil detection module, an ellipse fitting module, a calibration module, and a polynomial fitting module, and transmits the data to a control system; The control system receives the gaze point information, analyzes the currently required image plane depth, generates a lens adjustment instruction, and sends it to the lens-screen adjustment system; The lens-screen adjustment system drives the lens and the screen to make physical adjustments according to the control instructions, and uses a sensor feedback mechanism to monitor the adjustment accuracy in real time to ensure that the virtual image position matches the user's gaze focus; The display system dynamically adjusts the animation picture in combination with the depth information provided by the control system so that it changes synchronously with the user's gaze depth, thereby enhancing the visual immersion.

[0012] Furthermore, in the multi-image eye tracking system, the image acquisition module obtains the pupil image of the user's eye by calling an active infrared camera, the pupil detection module is used to calibrate the pupil contour, the ellipse fitting module is used to detect the pupil center position with the highest confidence, the calibration module performs coordinate conversion in combination with known reference points, the polynomial fitting module optimizes the data, and finally converts the pupil data into gaze point coordinates.

[0013] Preferably, the implementation process of the image acquisition module is as follows: Step 111, select a suitable camera such as a 940nm short-focus camera and infrared lamp beads; Step 112, connect the 940nm short-focus camera and the infrared lamp beads to facilitate subsequent collection; Step 113, adjust the positions of the 940nm short-focus camera and the infrared lamp beads to ensure that the eyes are fully illuminated to avoid pupil detection failure.

[0014] More preferably, the implementation process of the pupil detection module is as follows: Step 121, judging whether pupil detection fails according to the pupil threshold algorithm, if the detection fails, step 113 needs to be repeated to ensure the success of pupil detection; Step 122, convert the input video into video frames and process them frame by frame: The input original image is cropped and scaled to unify the image size and facilitate subsequent processing. By calculating the aspect ratio of the current image Current_ratio (current_width / current_height) and the target aspect ratio Desire_ratio (preset to width / height, for example, width = 580, height = 480), if Current_ratio>Desire_ratio, it indicates that the image is too wide, then calculate the new width New_width = int(Desire_ratio *Current_ratio), and crop it from the center offset loc_w = (current_width - New_width) / / 2 in the image width direction to obtain the cropped image cropped_img. If Current_ratio<=Desire_ratio, that is, the image is too high, calculate the new height New_height = int(current_width / Desire_ratio), and crop it from the center offset loc_h = (current_height - New_height) / / 2 in the image height direction to obtain cropped_img = image[loc_h: loc_h + New_height, :]; Finally, use the cv2.resize function to resize the cropped image to the target size (width, height); Step 123, storing similar binary images of three different regions A, B, and C in the image: Convert the cropped and scaled image to a grayscale image and apply a strict binarization threshold to the grayscale image; add the darkest pixel value Darkest_ value to the threshold increment , the calculated threshold is Threshold = Darkest_value + ; Use cv2.threshold function to binarize the image, set the pixels less than the threshold to white (255), and the rest to black (0), and get the image Thresholded_image_1 after strict threshold processing; then, take the darkest point Darkest_point as the center and the side length is Create a square mask and set the pixel values ​​outside the square area to , perform mask processing on Thresholded_image_1; Add the darkest pixel value Darkest_value to the threshold increment , calculate the threshold Threshold = Darkest_value + , binarize the grayscale image to get Thresholded_image_2, with the darkest point as the center and the side length Perform masking; Add the darkest pixel value Darkest_value to the threshold increment , calculate the threshold Threshold = Darkest_value + , binarize the grayscale image to get Thresholded_image_3; then take the darkest point as the center, and the side length Perform masking; And save Thresholded_image_1, Thresholded_image_2, and Thresholded_image_3; Step 124, detecting the color threshold ranges of three different areas A, B, and C; Step 125 , selecting the pupil image with the smallest fluctuation in average color threshold as the most credible pupil image.

[0015] More preferably, the implementation process of the ellipse fitting module is as follows: Step 131, dilation processing is performed on each binary image to enhance the target area: For the image after multi-threshold processing and masking, first use the morphological dilation operation (such as cv2.dilate function) to enhance the contour features in the image; then, use the cv2.findContours function to find the contours in the image; then, filter the found contours, traverse all contours, and calculate the area of ​​each contour area=cv2.contourArea(contour). If area>=pixel (for example, pixel=10), then further calculate the width w and height h of the circumscribed rectangle of the contour (through cv2.boundingRect function), and calculate the aspect ratio length = max(w, h), width=min(w, h), Current_ratio=max(length / width, width / length). If Current_ratio<= ratio_thresh (for example, ratio_thresh = 5), the contour meets the conditions, and the contour with the largest area that meets the conditions is returned; Step 132, extracting the outer contour in the dilated image; Step 133, filter the contours by area and quantity restrictions, and return the largest contour; Step 134, not searching within D pixels at the edge of the image; Step 135, checking the brightness every E pixels in the area; Step 136, along Axis and The axis is sampled every F pixels, ignoring the boundaries; Step 137, updating the pixel with the maximum threshold value between the 0-255 color channels; More preferably, the implementation process of the calibration module is as follows: Step 141, select a screen size as a calibration screen, input screen parameters into a control system and project them into human eyes through a lens; Step 142, selecting 9-12 points that evenly cover the screen as calibration points; Step 143, the user wears the device and adjusts the size of the eye image in the screen; Step 144, perform autonomous calibration through an external device such as a keyboard or a handle to ensure that the point being looked at in the current state is a calibration point: In calibration mode, the camera acquires images in real time, and the pupil center coordinates (x_eye, y_eye) are obtained using the pupil detection algorithm mentioned above; the current calibration target point is drawn on each frame (displayed in red); when the user presses the space bar, the current pupil coordinates (x_eye, y_eye) and the corresponding screen coordinates (i.e., the screen coordinates of the current calibration target point (x_screen, y_screen)) are recorded in the calibration data dictionary calibration_data. After all calibration points are recorded, calibration_data is saved in a file (such as calibration_data.json); Step 145, adjusting the distance between the lens and the screen, thereby changing the distance between the image plane and the human eye, and repeating steps 142 to 144; Step 146, saving the weight information calibrated at different depths.

[0016] Preferably, the implementation process of the polynomial fitting module is as follows: Step 151, obtaining pixel coordinates of 9-12 calibration points; Step 152, outputting the pixel coordinates of the pupil center from the video frame; Step 153: The pixel coordinates in step 151 and step 152 are and Split into One-dimensional data; Step 154, selecting a polynomial to perform one-dimensional data fitting on the coordinate points; Step 155, Fitting Polynomial coefficients in functions; Step 156, Fitting Polynomial coefficients in functions; Step 157, save the fitted Axis and Coefficients of the one-dimensional polynomial of the axis: Load pupil coordinate data eye_coords and screen coordinate data screen_coords from the saved calibration data file. Extract the x component eye_x and y component eye_y of the pupil coordinates, as well as the x component screen_x and y component screen_y of the screen coordinates. Use functions to perform polynomial fitting on the pupil x coordinate and the screen x coordinate, and the pupil y coordinate and the screen y coordinate, respectively, to obtain fitting coefficients coefficients_x and coefficients_y; Step 158, start predicting the gaze point coordinates by inputting the pupil center coordinates; Furthermore, the lens-screen adjustment system includes a lens, a screen, a linear slide rail, a micro-stepping motor and a driving circuit, wherein the driving circuit is connected to the micro-stepping motor, the action end of the micro-stepping motor is linked to the screen, the screen is slidably mounted on the linear slide rail, and the screen and the lens are arranged on the front and rear sides of the linear slide rail respectively; the components of the system are highly integrated, and when the gaze point enters the target area, the driving circuit sends a control signal to the micro-stepping motor, and the micro-drive motor drives the screen to perform a smooth and precise linear displacement along the linear slide rail, thereby adjusting the geometric distance between the lens and the screen. In this solution, the introduction of the micro-stepping motor realizes a lightweight design of the structure, and at the same time has a precise feedback adjustment capability. After receiving data from the multi-image plane eye tracking system, the system can drive the lens-screen unit through the micro-stepping motor for dynamic adjustment, thereby ensuring real-time matching of the user's visual focus and the image plane depth.

[0017] The lens-screen adjustment system also includes an adjustment module and a sensor feedback module. The adjustment module is used to select the distance to be adjusted in real time according to the predicted gaze point coordinates; the sensor feedback module is used to drive the micro stepping motor to adjust the distance between the lens and the screen.

[0018] Preferably, the implementation process of the adjustment module is as follows: Step 211, selecting a picture area according to the predicted gaze point coordinates outputted in step 158; Step 212, when the area is marked, adjust the travel range set in advance.

[0019] More preferably, the implementation process of the sensor feedback module is as follows: Step 221, select a micro stepper motor with 8mm stroke; Step 222: Apply PWM waves to control the travel of the micro-stepping motor according to different gaze areas.

[0020] The control system includes a data processing module, a calculation module and a control instruction module. The data processing module is used to receive the gaze point coordinates from the eye tracking system, the calculation module is used to parse the currently required image plane depth, and the control instruction module is used to send motion instructions to the lens-screen adjustment system.

[0021] Preferably, the implementation process of the data processing module is as follows: Step 311, receiving pupil center coordinates output by the eye tracking system; Step 312, receiving and parsing the calibrated gaze point coordinates; Step 313, based on the pupil center coordinates and the calibrated gaze point coordinates, coordinate correction and filtering are performed to remove noise and improve data accuracy; Step 314, storing the corrected gaze point coordinates and transmitting the data to the calculation module; More preferably, the implementation process of the calculation module is as follows: Step 321, receiving the gaze point coordinate information from the data processing module; Step 322, based on the gaze point-depth mapping model preset by the system, calculate the currently required image plane depth; Step 323, performing outlier detection on the calculation results and performing interpolation processing to optimize data smoothness; Step 324, storing the calculated depth information and sending it to the control instruction module; More preferably, the implementation process of the control instruction module is as follows: Step 331, receiving image plane depth data transmitted by a calculation module; Step 332, generating control instructions for the stepper motor according to the depth requirement, including forward, backward and stroke length; Step 333, using a PWM (pulse width modulation) signal to control the moving step length and speed of the micro stepping motor; Step 334, real-time monitoring of the stepper motor status to ensure accurate adjustment of the lens position; Step 335, feedback the execution result, and make secondary adjustments if necessary to improve the system response accuracy.

[0022] The display system includes an animation preprocessing module and a gaze point sensing module. The animation preprocessing module is used to set the virtual image depths corresponding to different areas in the picture in advance; the gaze point sensing module is used to ensure that the picture update and lens adjustment are carried out synchronously.

[0023] Preferably, the implementation process of the animation preprocessing module is as follows: Step 411, setting the depth information in the picture according to the depth information in real life; Step 412, presetting depth information for different areas of each frame of the animation; Step 413, presetting different depth information for images at different depths; Step 414, setting the preset depth information as a feedback terminal; Preferably, the implementation process of the gaze point sensing module is as follows: Step 421, setting the area threshold range; Step 422, presetting different area threshold ranges for images at different depths; Step 423, when the gaze point is transferred to the regional threshold, feedback information is generated to drive the micro-stepping motor of step 222 to achieve feeding.

[0024] A dynamic virtual image plane adjustment method based on multi-image plane eye tracking comprises the following steps: Step 1: The user wears the device: The user wears the eye tracking device correctly, ensures that the device fits the face firmly, and adjusts the position so that the camera can clearly capture the user's eye area; self-check whether the user's pupil is completely visible, and adjust the infrared light intensity; Step 2, eye movement calibration: Enter the calibration mode and guide the user to look at multiple preset calibration points in turn to establish the user's eye movement model; accurately fit the user's personalized eye movement parameters by calculating the mapping relationship between the pupil center offset and the screen coordinates; during the calibration process, dynamically adjust the parameters according to the sampling error; Step 3, gaze point prediction: In normal operation mode, the user's eye movement data, including pupil position and line of sight direction, is continuously collected; based on the established eye movement model, the user's current gaze point coordinates are calculated and predicted in real time to determine the specific target area of ​​their line of sight on the screen; this prediction data is used to drive the subsequent adaptive visual adjustment mechanism; Step 4: Gaze point sensing: further accurately measure the user's real-time gaze point and perform error correction on the predicted data; detect the user's eye movements and dynamically adjust the calculation accuracy of the gaze point to cope with the eye movement characteristics of different users; Step 5, lens-screen distance adjustment: When it is confirmed that the user's gaze point is within the threshold range of the area set by the gaze point sensing module, the drive circuit generates a drive signal and transmits it to the micro stepper motor. The micro stepper motor accurately moves the screen along the linear slide rail to change the geometric distance between the screen and the lens to adjust the visual focal length to match the user's current line of sight.

[0025] The technical concept of the present invention is as follows: the present invention proposes an intelligent, real-time responsive virtual image plane adjustment device by combining eye tracking technology with a multi-image plane dynamic adjustment scheme. The device can dynamically adjust the lens position according to the depth information of the user's gaze point, so that the virtual image plane can achieve intelligent focusing at different depths, thereby effectively alleviating visual fatigue, enhancing the immersive experience, and providing a more advanced solution for the focal length adjustment of future VR / AR devices. Compared with the prior art CN208823365U, the present invention combines eye tracking with multi-image plane dynamic adjustment technology, which not only makes the focal length adjustment process more natural and more intelligent, but also effectively alleviates visual fatigue; compared with the prior art CN114895793A, the present invention further proposes a virtual image plane dynamic adjustment scheme combined with multi-image plane eye tracking technology on the basis of the existing technology, so that the system can dynamically adjust the relative position of the lens and the screen according to the depth information of the user's gaze point, so as to achieve more accurate focal length control.

[0026] The present invention uses a multi-plane eye tracking module to perceive the user's line of sight in real time, combines with a lens-screen adjustment module to dynamically adjust the optical path length, cooperates with the efficient data processing of the control system module and the depth picture presentation of the display system module, thereby achieving the effect of dynamically adjusting the virtual image plane depth, providing users with a more comfortable and natural visual experience.

[0027] The beneficial effects of the present invention are mainly manifested in: 1) Innovation in the integration of optics and ophthalmology: This invention is based on the adjustment principle of the lens and integrates it with eye tracking. According to the requirements of ophthalmic medicine and basic knowledge of optics, it cleverly combines optics and ophthalmology efficiently, and independently designs a dynamic virtual image adjustment device and method based on multi-image eye tracking.

[0028] 2) Multi-plane eye tracking: Multi-plane eye tracking can satisfy users' need to realize eye tracking function at different plane depths. Compared with traditional VR / AR devices, which usually only perform eye tracking on a single plane, it is impossible to realize eye tracking on multiple planes.

[0029] 3) Animation depth setting: The depth information corresponding to different positions is set in advance in each frame of animation, and the feedback mechanism is implemented in conjunction with the multi-image eye tracking system to help users achieve a more realistic interactive scene.

[0030] 4) Active feedback: By judging the position of the eye's gaze point, the physical distance between the lens and the screen is changed in real time, thereby changing the distance between the virtual image and the human eye, without the need for manual adjustment, to achieve active eye movement feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a principle block diagram of a dynamic virtual image plane adjustment device based on multi-image plane eye tracking; Figure 2 It is a flow chart of a dynamic virtual image plane adjustment method based on multi-image plane eye tracking; Figure 3 It is a schematic diagram of the optical path of the multi-image eye tracking system; Figure 4 It is a schematic diagram of the preset depth of the display system; Figure 5 It is a front structural schematic diagram of a dynamic virtual image plane adjustment device based on multi-image plane eye tracking in the present invention; Figure 6 It is a schematic diagram of the back structure of the dynamic virtual image plane adjustment device based on multi-image plane eye tracking in the present invention.

[0032] Among them, 101 is a screen, 102 is a lens 102, 103 is a human eye, 104 is an active infrared camera, 105 is a partition, 106 is a micro stepping motor, 201 is a near-view cylindrical obstacle, 202 is a far-view cylindrical obstacle, 203 is the sun, 204 is a cloud, 301 is a shell, 302 is a linear slide rail, 303 is a lens frame, 304 is a card slot, 305 is an active infrared light source, and 401 is a screen frame. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings.

[0034] Reference Figure 1 to Figure 6 A dynamic virtual image plane adjustment device combined with multi-image plane eye tracking technology detects the user's gaze point position in real time by pre-setting the depth information of different positions in the displayed animation, dynamically adjusts the distance between the lens and the screen, changes the position of the virtual image plane, and realizes active feedback without the need for manual adjustment by the user, thereby alleviating visual fatigue and improving user experience.

[0035] Figure 1 is a principle block diagram of a dynamic virtual image plane adjustment device based on multi-image plane eye tracking in the present invention, such as Figure 1 As shown, the dynamic virtual image plane adjustment device based on multi-image plane eye tracking includes a multi-image plane eye tracking system, a control system, a lens-screen adjustment system and a display system.

[0036] The optical path of the multi-image eye tracking system is as follows: Figure 3 As shown, the light emitted by the screen 101 passes through the lens 102 and then enters the human eye 103, and the active infrared camera 104 collects the eye image to perform multi-plane eye tracking. The partition 105 is used to ensure that the two eyes can see separately, and the distance between the screen and the lens is changed by the micro stepping motor 106 according to the predicted position of the gaze point, thereby changing the distance between the image produced by the light emitted by the screen and the human eye, which can simulate the real scene more realistically.

[0037] The multi-image eye tracking system comprises an image acquisition module, a pupil detection module, an ellipse fitting module, a calibration module and a polynomial fitting module, wherein: The image acquisition module is used to obtain high-resolution images of the user's eyes to ensure the accuracy of subsequent processing. The implementation process is as follows: Step 111, selecting a suitable 940nm short-focus camera and infrared light source as data acquisition equipment to enhance the contrast of pupil detection; Step 112, correctly connect the 940nm short-focus camera to the infrared light source to ensure stable operation of the acquisition system; Step 113, optimizing the position of the camera and the infrared light source so that the eye area is fully illuminated, avoiding pupil detection failure, and improving the quality of data collection; The pupil detection module is responsible for identifying the pupil area from the collected image, extracting the pupil edge information, and preliminarily determining the pupil center position. The implementation process is as follows: Step 121, evaluate the validity of the current detection result based on the pupil threshold algorithm. If the detection fails, the camera and light source need to be adjusted (repeat the environment adjustment step 113) to ensure a stable pupil image; Step 122, converting the input video stream into independent video frames, and analyzing each frame frame by frame; Step 123, binarizing three different regions A, B, and C in the image, and storing the binary image to enhance pupil features; Step 124, detecting the color threshold ranges of regions A, B, and C, and analyzing their changing trends; Step 125, selecting the area with the smallest color threshold fluctuation in the A, B, and C areas as the most reliable pupil image to improve the stability of the detection; The ellipse fitting module is used to perform ellipse fitting on the pupil area based on the pupil detection results to optimize the positioning accuracy of the pupil center. The implementation process is as follows: Step 131, dilating the binarized image to enhance visibility of the target area; Step 132, extracting the outer contour from the expanded image to obtain a candidate pupil region; Step 133, based on the contour area and quantity characteristics, select the largest contour that best matches the pupil shape; Step 134, not performing pupil search in the area within D pixels of the edge of the image to reduce noise interference; Step 135, in a specific area, brightness detection is performed every E pixels to enhance the robustness of pupil detection; Step 136, sampling every F pixels along the X-axis and the Y-axis, ignoring boundary noise, and improving calculation efficiency; Step 137, updating the pixel with the largest threshold change in the color range of 0-255 to improve the accuracy of pupil detection; The calibration module is used to establish the mapping relationship between the pupil center and the screen gaze point to ensure the accuracy of eye tracking. The implementation process is as follows: Step 141, select a screen size as a calibration screen, and input its parameters into a control system to ensure that the lens projection imaging meets the user's visual requirements; Step 142, evenly arranging 9-12 calibration points on the screen to ensure that the data is evenly distributed to improve the calibration accuracy; Step 143, after the user wears the device, the size of the eye image on the screen is adjusted to meet the calibration requirements; Step 144, the user performs calibration through an external device (keyboard or handle) to ensure that the current gaze point corresponds correctly to the calibration point; Step 145, adjusting the distance between the lens and the screen, thereby changing the distance between the image plane and the human eye, and repeating steps 142 to 144; Step 146, recording calibration weight information under different depth conditions to provide reference data for subsequent eye tracking; The polynomial fitting module is responsible for fitting the mathematical relationship between the pupil center coordinates and the screen gaze point, and constructing a mathematical model for predicting the user's gaze position. The implementation process is as follows: Step 151, obtain the pixel coordinates of 9-12 calibration points ; Step 152: Output the pixel coordinates of the pupil center from the video frame ; Step 153: The pixel coordinates in step 151 and step 152 are and Split into One-dimensional data; Step 154, selecting the polynomial order and fitting the coordinate point data to construct an optimal mapping function; Step 155: training based on data The polynomial coefficients ensure the conversion accuracy in the X-axis direction; Step 156, calculate The polynomial coefficients of are used to ensure the mapping accuracy in the Y-axis direction; Step 157, save the fitted Axis and Coefficients of the one-dimensional polynomial of the axis; Step 158, during the actual operation, input the pupil center coordinates detected in real time, use the fitting function to predict the gaze point coordinates on the screen, and achieve high-precision eye tracking; The display system includes an animation preprocessing module and a gaze point sensing module. Figure 4 This is a schematic diagram of a preset depth picture in the display system, which simulates a scene on the road, including a near cylindrical obstacle 201, a far cylindrical obstacle 202, the sun 203 and clouds 204. According to the geometric perspective relationship, the human eye will spontaneously think that the near cylindrical obstacle 201 is closer than the far cylindrical obstacle 202, while the sun 203 and the clouds 204 can be regarded as infinitely far away. Therefore, when the predicted gaze point reaches a certain position among 201-204, a signal will be sent to the screen-lens adjustment system to change the distance between the two through a micro stepping motor.

[0038] The display system includes an animation preprocessing module and a gaze point sensing module, wherein: The animation preprocessing module is used to set the virtual image depth corresponding to different areas in the picture in advance. The implementation process is as follows: Step 211, setting the depth parameters in the virtual image according to the depth information of the real world; Step 212, allocating depth information of different regions to each frame of the animation; Step 213, setting corresponding depth parameters for different depth of field ranges; Step 214, transmitting the calculated depth information to the feedback terminal for dynamic adjustment in the visual display; The gaze point sensing module is used to ensure that the image update and lens adjustment are synchronized. The implementation process is as follows: Step 221, setting a screen area threshold range for gaze detection; Step 222, presetting different area thresholds according to different depth of field ranges; Step 223, when the user's gaze point enters the set area range, feedback information is triggered to drive the micro stepping motor to adjust the focal length to optimize the viewing experience.

[0039] According to Figure 2 As shown in the flowchart, the tester needs to Figure 5 The housing 301 shown in FIG. 1 is correctly worn on the head and can be seen through the lens 102 Figure 6 The screen 101 shown in the figure is installed in the screen frame 401, and the screen frame 401 is slidably installed on one side of the linear slide rail 302. The lens 102 is fixedly installed on the other side of the linear slide rail 302 through the lens frame 303, and is connected to the partition 105 through the card slot 304. When looking at the screen 101, the two eyes are separated by the partition 105 to ensure that binocular vision can be achieved, so that the pictures seen by the two eyes are relatively independent. After the test adjustment is completed, Figure 5 The active infrared light source 305 in the image illuminates the pupil, and the active infrared camera 104 takes a pupil image for eye movement calibration. When the calibration is completed, Figure 6 The micro-stepping motor 106 in the embodiment drives the screen frame 401 to move, thereby changing the distance between the screen and the lens. During the distance change, since the partition 105 is connected to the lens frame 303 through the card slot 304, the eyes are still independent of each other. After the distance change is completed, the screen frame 401 is moved according to the distance between the screen and the lens. Figure 2 The flowchart shown is cycled until the training is completed.

[0040] Furthermore, the micro-stepping motor 106 drives the screen frame 401 to move, which is completed by the lens-screen adjustment system.

[0041] The lens-screen adjustment system includes a lens 102, a screen 101, a linear slide 302, a micro-stepping motor 106 and a driving circuit. The driving circuit is connected to the micro-stepping motor 106. The action end of the micro-stepping motor 106 is linked to the screen 101. The screen 101 is slidably mounted on the linear slide 302. The screen 101 and the lens 102 are arranged on the front and rear sides of the linear slide 302 respectively. The components of the system are highly integrated. When the gaze point enters the target area, the driving circuit sends a control signal to the micro-stepping motor 106, and the driving motor drives the screen 101 to perform a smooth and precise linear displacement along the linear slide 302, thereby adjusting the geometric distance between the lens 102 and the screen 101. In this solution, the introduction of the micro-stepping motor 106 realizes a lightweight design of the structure, and at the same time has a precise feedback adjustment capability. After receiving data from the multi-image plane eye tracking system, the system can drive the lens-screen unit through the micro-stepping motor to perform dynamic adjustment, thereby ensuring real-time matching of the user's visual focus and the image plane depth.

[0042] The lens-screen adjustment system comprises an adjustment module and a sensor feedback module, wherein: The adjustment module is used to select the distance to be adjusted in real time according to the predicted gaze point coordinates. The implementation process is as follows: Step 311, selecting a picture area according to the output predicted gaze point coordinates; Step 312, when the gaze point falls into the area, the system adjusts the focal length according to the preset travel range to match the corresponding visual requirements; The sensor feedback module is used to drive the micro stepper motor to adjust the distance between the lens and the screen. The implementation process is as follows: Step 321, using a high-precision micro-stepping motor with a travel range of 8 mm to ensure the accuracy and stability of the adjustment; Step 322, according to the different gaze areas of the user, the system generates a PWM (pulse width modulation) control signal to drive the micro-stepping motor to perform corresponding displacement adjustment to achieve focal length optimization; The cycle process is implemented by a control system, which includes a data processing module, a calculation module, and a control instruction module, wherein: The data processing module is used to receive the gaze point coordinates from the eye tracking system. The implementation process is as follows: Step 411, receiving pupil center coordinates output by the eye tracking system; Step 412, receiving and parsing the calibrated gaze point coordinates; Step 413, based on the pupil center coordinates and the calibrated gaze point coordinates, coordinate correction and filtering are performed to remove noise and improve data accuracy; Step 414, storing the corrected gaze point coordinates and transmitting the data to the calculation module; The calculation module is used to analyze the current required image depth. The implementation process is as follows: Step 421, receiving the gaze point coordinate information from the data processing module; Step 422, based on the gaze point-depth mapping model preset by the system, calculate the currently required image plane depth; Step 423, performing outlier detection on the calculation results and performing interpolation processing to optimize data smoothness; Step 424, storing the calculated depth information and sending it to the control instruction module; The control command module is used to send motion commands to the lens-screen adjustment system. The implementation process is as follows: Step 431, receiving image plane depth data transmitted by a calculation module; Step 432, generating control instructions for the stepper motor according to the depth requirement, including forward, backward and stroke length; Step 433, using a PWM (pulse width modulation) signal to control the moving step length and speed of the micro stepping motor; Step 434, real-time monitoring of the stepper motor status to ensure accurate adjustment of the lens position; Step 435, feedback the execution result, and make secondary adjustments if necessary to improve the system response accuracy.

[0043] Reference Figure 2 ,A dynamic virtual image plane adjustment method based on multi-image plane eye tracking ,System startup, the system initializes and starts all core systems.,This stage ensures that all hardware works properly and the software enters the standby state to prepare for subsequent ,operations; The adjustment method comprises the following steps: Step 1. The user wears the device: The user wears the eye tracking device correctly, ensures that the device fits the face firmly, and adjusts it to the appropriate position so that the camera can clearly capture the user's eye area. The system self-checks whether the user's pupil is completely visible, and adjusts the infrared light intensity to optimize the pupil imaging quality and ensure the accuracy of subsequent data collection; Step 2: Eye movement calibration: The system enters the calibration mode and guides the user to look at multiple preset calibration points in turn to establish the user's eye movement model; by calculating the mapping relationship between the pupil center offset and the screen coordinates, the system accurately fits the user's personalized eye movement parameters to ensure that the subsequent gaze point prediction has high accuracy; during the calibration process, the parameters are dynamically adjusted according to the sampling error to improve tracking stability; Step 3: Prediction of gaze point: In normal operation mode, the system continuously collects the user's eye movement data, including characteristic information such as pupil position and line of sight direction. Based on the established eye movement model, the system calculates and predicts the user's current gaze point coordinates in real time to determine the specific target area of ​​the user's line of sight on the screen. This prediction data is used to drive the subsequent adaptive visual adjustment mechanism. Step 4: Gaze point sensing: The system further accurately measures the user's real-time gaze point and performs error correction on the predicted data. The system can detect the user's tiny eye movements and dynamically adjust the calculation accuracy of the gaze point to cope with the eye movement characteristics of different users. This stage ensures that the system has high stability and robustness in identifying the gaze target. Step 5, lens-screen distance adjustment: When the system confirms that the user's gaze point is within the threshold range of the area set by the gaze point sensing module, the drive circuit generates a drive signal and transmits it to the micro-stepping motor. The micro-stepping motor accurately moves the screen along the linear slide rail to change the geometric distance between the screen and the lens to adjust the visual focal length to match the user's current line of sight. The adjustment process adopts a closed-loop control mechanism to ensure the smoothness and accuracy of the screen movement, and dynamically adjusts according to the real-time changes of the user's gaze point. The system performs closing operations, records key data, and decides to enter the next operation cycle or safely shut down all devices based on user needs; the system releases hardware resources when exiting and saves the user's personalized calibration parameters so that they can be quickly loaded for subsequent use, improving system response efficiency; The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A dynamic virtual image plane adjustment device based on multi-image plane eye tracking, characterized in that: The adjustment device includes a multi-image plane eye tracking system, a lens-screen adjustment system, a control system and a display system. The multi-image eye tracking system uses an infrared camera to capture the user's eye image, and converts it into accurate gaze point coordinates through calculations by an image acquisition module, a pupil detection module, an ellipse fitting module, a calibration module, and a polynomial fitting module, and transmits the data to a control system; The control system receives the gaze point information, analyzes the currently required image plane depth, generates a lens adjustment instruction, and sends it to the lens-screen adjustment system; The lens-screen adjustment system drives the lens and the screen to make physical adjustments according to the control instructions, and uses a sensor feedback mechanism to monitor the adjustment accuracy in real time to ensure that the virtual image position matches the user's gaze focus; The display system dynamically adjusts the animation picture in combination with the depth information provided by the control system so that it changes synchronously with the user's gaze depth, thereby enhancing the visual immersion.

2. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to claim 1, characterized in that: In the multi-image eye tracking system, the image acquisition module acquires the pupil image of the user's eye by calling an active infrared camera, the pupil detection module is used to calibrate the pupil contour, the ellipse fitting module is used to detect the pupil center position with the highest confidence, the calibration module performs coordinate conversion in combination with known reference points, the polynomial fitting module optimizes the data, and finally converts the pupil data into gaze point coordinates.

3. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to claim 2, characterized in that: The implementation process of the image acquisition module is as follows: Step 111, select a short-focus camera and infrared lamp beads; Step 112, connecting the short-focus camera and the infrared lamp beads; Step 113, adjusting the position of the short-focus camera and the infrared lamp beads to ensure that the eyes are fully illuminated; The implementation process of the pupil detection module is as follows: Step 121, judging whether pupil detection fails according to the pupil threshold algorithm, if the detection fails, step 113 needs to be repeated to ensure the success of pupil detection; Step 122, convert the input video into video frames and process them frame by frame: Crop and scale the input original image to unify the image size and facilitate subsequent processing. By calculating the aspect ratio Current_ratio of the current image and the target aspect ratio Desire_ratio, if Current_ratio>Desire_ratio, it indicates that the image is too wide, then calculate the new width New_width = int(Desire_ratio *Current_ratio), and crop it from the center offset loc_w = (current_width - New_width) / / 2 in the image width direction to obtain the cropped image cropped_img. If Current_ratio<=Desire_ratio, that is, the image is too high, calculate the new height New_height = int(current_width / Desire_ratio), and crop it from the center offset loc_h = (current_height - New_height) / / 2 in the image height direction to obtain cropped_img = image[loc_h: loc_h + New_height, :]; Finally, use cv2.resize The function adjusts the cropped image to the target size (width, height); Step 123, storing similar binary images of three different regions A, B, and C in the image: Convert the cropped and scaled image to a grayscale image and apply a strict binarization threshold to the grayscale image; add the darkest pixel value Darkest_ value to the threshold increment , the calculated threshold is Threshold = Darkest_ value + ; Use cv2.threshold function to binarize the image, set the pixels less than the threshold to 255, and the rest to 0, and get the strictly thresholded image Thresholded_image_1, i.e. A; then, with the darkest point Darkest_point as the center and the side length Create a square mask and set the pixel values ​​outside the square area to , perform mask processing on Thresholded_image_1; Add the darkest pixel value Darkest_value to the threshold increment , calculate the threshold Threshold = Darkest_value + , the grayscale image is binarized to obtain Thresholded_image_2, i.e. B, with the darkest point as the center and the side length Perform masking; Add the darkest pixel value Darkest_value to the threshold increment , calculate the threshold Threshold = Darkest_value + , binarize the grayscale image to get Thresholded_image_3, i.e. C; then take the darkest point as the center and the side length Perform masking; And save Thresholded_image_1, Thresholded_image_2, and Thresholded_image_3; Step 124, detecting the color threshold ranges of three different areas A, B, and C; Step 125 , selecting the pupil image with the smallest fluctuation in average color threshold as the most credible pupil image.

4. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to claim 2, characterized in that: The implementation process of the ellipse fitting module is as follows: Step 131, dilation processing is performed on each binary image to enhance the target area: For the image after multi-threshold processing and masking, first use the morphological dilation operation to enhance the contour features in the image; then, use the cv2.findContours function to find the contours in the image; then, filter the found contours, traverse all contours, and calculate the area of ​​each contour area=cv2.contourArea(contour). If area>=pixel, pixel is the preset threshold, then further calculate the width w and height h of the circumscribed rectangle of the contour, and calculate the aspect ratio length =max(w, h), width=min(w, h), Current_ratio=max(length / width, width / length); if Current_ratio<= ratio_thresh, ratio_thresh is the threshold, then the contour meets the conditions, and the contour with the largest area that meets the conditions is returned; Step 132, extracting the outer contour in the dilated image; Step 133, filter the contours by area and quantity restrictions, and return the largest contour; Step 134, not searching within D pixels at the edge of the image; Step 135, checking the brightness every E pixels in the area; Step 136, along Axis and The axis is sampled every F pixels, ignoring the boundaries; Step 137, updating the pixel with the maximum threshold value between the 0-255 color channels.

5. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to claim 2, characterized in that: The implementation process of the calibration module is as follows: Step 141, select a screen size as a calibration screen, input screen parameters into a control system and project them into human eyes through a lens; Step 142, selecting 9-12 points that evenly cover the screen as calibration points; Step 143, the user wears the device and adjusts the size of the eye image in the screen; Step 144, perform autonomous calibration through an external device such as a keyboard or a handle to ensure that the point being looked at in the current state is a calibration point: In calibration mode, the camera acquires images in real time, and the pupil center coordinates (x_eye, y_eye) are obtained using the pupil detection algorithm. The current calibration target point is drawn on each frame of the image. When the user presses the space bar, the current pupil coordinates (x_eye, y_eye) and the corresponding screen coordinates are recorded in the calibration data dictionary calibration_data. After all calibration points are recorded, calibration_data is saved to a file. Step 145, adjusting the distance between the lens and the screen, thereby changing the distance between the image plane and the human eye, and repeating steps 142 to 144; Step 146, saving the weight information calibrated at different depths.

6. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to claim 2, characterized in that: The implementation process of the polynomial fitting module is as follows: Step 151, obtaining pixel coordinates of 9-12 calibration points; Step 152, outputting the pixel coordinates of the pupil center from the video frame; Step 153: The pixel coordinates in step 151 and step 152 are and Split into One-dimensional data; Step 154, selecting a polynomial to perform one-dimensional data fitting on the coordinate points; Step 155, Fitting Polynomial coefficients in functions; Step 156, Fitting Polynomial coefficients in functions; Step 157, save the fitted Axis and Coefficients of the one-dimensional polynomial of the axis: Load pupil coordinate data eye_coords and screen coordinate data screen_coords from the saved calibration data file, extract the x component eye_x and y component eye_y of the pupil coordinate, and the x component screen_x and y component screen_y of the screen coordinate, and use functions to perform polynomial fitting on the pupil x coordinate and the screen x coordinate, and the pupil y coordinate and the screen y coordinate, respectively, to obtain fitting coefficients coefficients_x and coefficients_y; Step 158, start predicting the gaze point coordinates by inputting the pupil center coordinates.

7. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to any one of claims 1 to 6, characterized in that: The lens-screen adjustment system comprises a lens, a screen, a linear slide rail, a micro-stepping motor and a driving circuit, wherein the driving circuit is connected to the micro-stepping motor, the action end of the micro-stepping motor is linked to the screen, the screen is slidably mounted on the linear slide rail, and the screen and the lens are arranged on the front and rear sides of the linear slide rail respectively; The lens-screen adjustment system further includes an adjustment module and a sensor feedback module, wherein the adjustment module is used to select the distance to be adjusted in real time according to the predicted gaze point coordinates; the sensor feedback module is used to drive the micro-stepping motor to adjust the distance between the lens and the screen; The implementation process of the adjustment module is as follows: Step 211, selecting a picture area according to the predicted gaze point coordinates output by the polynomial fitting module; Step 212, when the area is marked, adjust the travel range set in advance; The implementation process of the sensor feedback module is as follows: Step 221, select a micro stepper motor with 8mm stroke; Step 222: Apply PWM waves to control the travel of the micro-stepping motor according to different gaze areas.

8. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to any one of claims 1 to 6, characterized in that: The control system includes a data processing module, a calculation module and a control instruction module, wherein the data processing module is used to receive the gaze point coordinates from the eye tracking system, the calculation module is used to analyze the currently required image plane depth, and the control instruction module is used to send motion instructions to the lens-screen adjustment system; The implementation process of the data processing module is as follows: Step 311, receiving pupil center coordinates output by the eye tracking system; Step 312, receiving and parsing the calibrated gaze point coordinates; Step 313, based on the pupil center coordinates and the calibrated gaze point coordinates, coordinate correction and filtering are performed to remove noise and improve data accuracy; Step 314, storing the corrected gaze point coordinates and transmitting the data to the calculation module; The implementation process of the calculation module is as follows: Step 321, receiving the gaze point coordinate information from the data processing module; Step 322, based on the gaze point-depth mapping model preset by the system, calculate the currently required image plane depth; Step 323, performing outlier detection on the calculation results and performing interpolation processing to optimize data smoothness; Step 324, storing the calculated depth information and sending it to the control instruction module; The implementation process of the control instruction module is as follows: Step 331, receiving image plane depth data transmitted by a calculation module; Step 332, generating control instructions for the stepper motor according to the depth requirement, including forward, backward and stroke length; Step 333, using a pulse width modulation (PWM) signal to control the moving step length and speed of the micro-stepping motor; Step 334, real-time monitoring of the stepper motor status to ensure accurate adjustment of the lens position; Step 335, feedback the execution result, and make secondary adjustments if necessary to improve the system response accuracy.

9. The dynamic virtual image plane adjustment device based on multi-image plane eye tracking according to any one of claims 1 to 6, characterized in that: The display system includes an animation preprocessing module and a gaze point sensing module. The animation preprocessing module is used to set the virtual image depth corresponding to different areas in the picture in advance; the gaze point sensing module is used to ensure that the picture update and lens adjustment are synchronized; The implementation process of the animation preprocessing module is as follows: Step 411, setting the depth information in the picture according to the depth information in real life; Step 412, presetting depth information for different areas of each frame of the animation; Step 413, presetting different depth information for images at different depths; Step 414, setting the preset depth information as a feedback terminal; The implementation process of the gaze point sensing module is as follows: Step 421, setting the area threshold range; Step 422, presetting different area threshold ranges for images at different depths; Step 423, when the gaze point is transferred to the regional threshold, feedback information is generated to drive the micro-stepping motor of the lens-screen adjustment system to achieve feeding.

10. A method for implementing the dynamic virtual image plane adjustment device based on multi-image plane eye tracking as claimed in claim 1, characterized in that: The method comprises the following steps: Step 1: The user wears the device: The user wears the eye tracking device correctly, ensures that the device fits the face firmly, and adjusts the position so that the camera can clearly capture the user's eye area; self-check whether the user's pupil is completely visible, and adjust the infrared light intensity; Step 2, eye movement calibration: Enter the calibration mode and guide the user to look at multiple preset calibration points in turn to establish the user's eye movement model; accurately fit the user's personalized eye movement parameters by calculating the mapping relationship between the pupil center offset and the screen coordinates; during the calibration process, dynamically adjust the parameters according to the sampling error; Step 3, gaze point prediction: In normal operation mode, the user's eye movement data, including pupil position and line of sight direction, is continuously collected; based on the established eye movement model, the user's current gaze point coordinates are calculated and predicted in real time to determine the specific target area of ​​their line of sight on the screen; this prediction data is used to drive the subsequent adaptive visual adjustment mechanism; Step 4: Gaze point sensing: further accurately measure the user's real-time gaze point and perform error correction on the predicted data; detect the user's eye movements and dynamically adjust the calculation accuracy of the gaze point to cope with the eye movement characteristics of different users; Step 5, lens-screen distance adjustment: When it is confirmed that the user's gaze point is within the threshold range of the area set by the gaze point sensing module, the drive circuit generates a drive signal and transmits it to the micro stepper motor. The micro stepper motor accurately moves the screen along the linear slide rail to change the geometric distance between the screen and the lens to adjust the visual focal length to match the user's current line of sight.

Images