Eye tracking based human computer interaction method, system and apparatus

Abstract

The application provides a human-computer interaction method based on eye tracking, and has the characteristics that S1, an original image is acquired and preprocessed; S2, a depth information image of the preprocessed original image is generated, and face feature detection and interception are performed on the preprocessed original image to obtain a face feature image; S3, eye feature detection and interception are performed on the face feature image to obtain a left eye feature image and a right eye feature image respectively, and user feature information is acquired according to the face feature image; S4, the face feature image, the left eye feature image and the right eye feature image are processed by using a deep learning model, and a gaze point is calculated in combination with the depth information image; and S5, user features and gaze information are output in combination with the gaze point and the user feature information, and a corresponding program is executed in response to an interaction action. The application realizes accurate calculation of depth information of each pixel point in an image, and generates a depth information image completely corresponding to an input image.

Description

Technical Field

[0001] This application relates to the field of eye-tracking technology, specifically to human-computer interaction methods, systems, and devices based on eye-tracking. Background Technology

[0002] Deep learning is a machine learning method based on artificial neural networks. Its core lies in extracting features and modeling data through multi-layered neural networks. Machine learning mainly involves two steps: modeling and training. Modeling primarily involves building a neural network model to extract features from the input data, perform numerical calculations, and define the format of the output data. Training requires building a dataset. The dataset mainly contains two types of data: input example data and expected correct results. Desktop eye-tracking models are trained using this dataset. During training, these networks learn from a large amount of data, adjusting their internal connection weights, ultimately enabling the model to accurately complete tasks, such as determining whether there is a cat in an image or translating text. Simply put, deep learning allows computers to "learn" to summarize patterns from data, replacing traditional manual programming for problem-solving.

[0003] Desktop eye-tracking technology is a technique that monitors and analyzes a user's eye movements to obtain their gaze behavior. Its working principle primarily utilizes deep learning.

[0004] It mainly detects faces by inputting images. Once a face is detected, the face image is segmented. Segmentation refers to cutting the face image from the original input image. Subsequent image processing will only process the segmented image.

[0005] Desktop eye-tracking technology first detects and segments the face, then detects the eyes on the cropped face, cropping the eyes after detecting the left and right eyes. Next, features are extracted from the detected left and right eye images and the face images. Feature extraction refers to the process of performing multiple convolution operations on the cropped image using a multi-layer convolutional neural network, and then using fully connected operations to select relevant parameters and extract them.

[0006] The desktop eye-tracking device extracts features from the left and right eyes and the face separately, and then performs feature fusion. Feature fusion involves convolutional operations on the extracted features after concatenation, combining the features from the three cropped images. Finally, a fully connected operation is performed on the fused features to obtain the user's gaze point position on the screen.

[0007] However, most desktop eye-tracking technology is now being used on large screens. But currently, the large screen sizes using eye tracking are mainly 55 inches or smaller, and there is no precedent for its use on 77-inch screens. Summary of the Invention

[0008] To address the aforementioned technical problems, a human-computer interaction method based on eye tracking is proposed in the first aspect of this application, comprising: S1, acquiring an original image and preprocessing the original image;

[0009] S2. Generate a depth information image of the preprocessed original image, and perform facial feature detection and cropping on the preprocessed original image to obtain a facial feature image;

[0010] S3. Perform eye feature detection and cropping on the facial feature image to obtain the left eye feature image and the right eye feature image respectively, and obtain user feature information based on the facial feature image;

[0011] S4. Use a deep learning model to process the facial feature image, left eye feature image, and right eye feature image, and combine them with the depth information image to calculate the gaze point;

[0012] S5. Combine gaze point and user feature information to output user features and gaze information, and execute the corresponding program in response to interactive actions.

[0013] Preferably, S4 specifically includes:

[0014] S401. Calculate facial pose feature parameters of facial feature images using a deep learning model;

[0015] S402. Further extract feature parameters of the left and right eye feature images using a deep learning model;

[0016] S403. Combine the facial feature image with the depth information image to obtain the facial depth information corresponding to the facial feature image;

[0017] S404. Combine facial feature pose, feature parameters of the further extracted left and right eye feature images, and facial depth information to calculate the gaze point.

[0018] Preferably, obtaining user feature information based on facial feature images specifically includes: extracting feature vectors from facial feature images using the FaceNet model, and determining whether there is user information in the preset user information database that matches the extracted feature vectors. If so, the user ID number is output; otherwise, the user information database is updated and the newly created user ID number is output.

[0019] Preferably, the feature parameters of the left-eye and right-eye feature images further extracted using a deep learning model specifically include:

[0020] The first convolution operation is performed on the eye image to obtain the first convolution eye image data;

[0021] Max pooling is performed on the first convolutional eye image data to obtain pooled eye image data;

[0022] A second convolution operation is performed on the pooled eye image data to obtain the second convolution eye image data;

[0023] Dilation convolution is performed on the second convolutional eye image data to obtain the feature parameters of the eye image.

[0024] Preferably, combining facial feature pose, feature parameters of the further extracted left and right eye feature images, and facial depth information to calculate the gaze point specifically includes: stitching the left eye feature image, right eye feature image, and facial feature image to obtain a gaze point without 3D information; obtaining 3D position parameters from the facial depth information and combining them with the gaze point without 3D information to obtain the actual gaze point.

[0025] Preferably, generating the depth information image of the preprocessed original image specifically includes: the preprocessed original image is passed through 5 convolutional feature extraction layers to extract 5 layers of convolutional features, and the extracted 5 layers of convolutional features are input into the C3D model. The data output by the C3D model is passed through 3 deconvolutional layers and then superimposed to obtain the depth information image.

[0026] Preferably, the corresponding program executed in response to the interactive action includes:

[0027] Gaze selection: Determine whether the gaze point has been in a certain area for more than a preset time; if so, simulate a touch event.

[0028] Scrolling and swiping controls determine whether the gaze point has been in the edge area for more than a preset time; if so, the corresponding scrolling / swiping operation is triggered.

[0029] The system highlights the content being gazed upon, and determines whether the gaze point has been located in a specific text or image area for more than a preset time. If so, the area is highlighted.

[0030] The system uses gaze drawing to determine whether to enter the drawing application. If so, it records the coordinates of the gaze point and draws the gaze point trajectory in real time.

[0031] In a second aspect of this application, an eye-tracking-based human-computer interaction system is proposed to execute the aforementioned eye-tracking-based human-computer interaction method, comprising: a user data processing module and a user interaction action module; the user data processing module is configured to acquire image data, extract user feature information, and calculate user gaze points; the user interaction action module is configured to receive user gaze points and user information via TCP protocol, and then process the user gaze data.

[0032] Preferably, the user data processing module includes: an eye-tracking unit, a user identification unit, and a human 3D position detection unit; the eye-tracking unit is configured to extract gaze point information; the user identification unit is configured to identify the user's identity and distinguish between the user currently operating and the module to be operated; the human 3D position detection unit is configured to extract the detected user position and correct the gaze information of the eye-tracking.

[0033] In a third aspect of this application, a human-computer interaction device based on eye tracking is proposed to execute the aforementioned human-computer interaction method based on eye tracking. The device comprises: a first RGB camera, a second RGB camera, an eye-tracking data processing host computer, an Android operating system host computer, and a display. The first RGB camera and the second RGB camera are respectively connected to the eye-tracking data processing host computer. The eye-tracking data processing host computer communicates with the Android operating system host computer via TCP. The Android operating system host computer is connected to the display.

[0034] Compared with the prior art, the beneficial results of this application are as follows:

[0035] Compared to traditional two-dimensional position detection, the human body 3D position detection module of this system realizes the accurate calculation of the depth information of each pixel in the image through a deep learning model, thereby generating a depth information map that completely corresponds to the input image;

[0036] An improved multi-branch deep learning architecture was adopted, which significantly improved the accuracy and efficiency of feature extraction;

[0037] An improved deep learning-based identity information management module was introduced, which uses an optimized FaceNet model to achieve high-precision user identity recognition and dynamic management, ensuring that interactions in a multi-user environment do not conflict. Attached Figure Description

[0038] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of this application. Other embodiments and many anticipated advantages of these embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0039] Figure 1 This is a flowchart of a human-computer interaction method based on eye tracking according to an embodiment of this application;

[0040] Figure 2 This is a flowchart of facial deep learning feature extraction according to an eye-tracking-based human-computer interaction method in accordance with an embodiment of this application;

[0041] Figure 3 This is a flowchart of eye deep learning feature extraction according to the human-computer interaction method based on eye tracking in the embodiments of this application;

[0042] Figure 4 This is a flowchart of the gaze point generation process of the eye-tracking-based human-computer interaction method according to an embodiment of this application;

[0043] Figure 5 This is the process of adding 3D position parameters to gaze points without 3D information according to the eye-tracking-based human-computer interaction method in the embodiments of this application;

[0044] Figure 6 This is a flowchart of a 3D depth data extraction model based on an eye-tracking-based human-computer interaction method according to an embodiment of this application;

[0045] Figure 7 This is a flowchart of the user identification process based on the eye-tracking-based human-computer interaction method according to an embodiment of this application;

[0046] Figure 8 This is a flowchart of an interaction system based on an eye-tracking-based human-computer interaction method according to an embodiment of this application;

[0047] Figure 9 This is a system block diagram of an eye-tracking-based human-computer interaction system according to an embodiment of this application;

[0048] Figure 10 This is a schematic diagram of the structure of a human-computer interaction device based on eye tracking according to an embodiment of this application.

[0049] 1. 77-inch monitor; 2. Android system host computer; 3. Eye-tracking data processing module; 4. RGB camera 1; 5. RGB camera 2; 6. Eye-tracking module; 7. Human body 3D position detection module; 8. Identity information recognition module; 9. Eye-tracking data packet sending module; 10. Eye-tracking data packet receiving module. Detailed Implementation

[0050] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and illustrate illustrative specific embodiments in which the present application may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the present application. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present application is defined by the appended claims.

[0051] like Figure 1-8 As shown, a human-computer interaction method based on eye tracking includes: S1, acquiring the original image and preprocessing the original image;

[0052] S2. Generate a depth information image of the preprocessed original image, and perform facial feature detection and cropping on the preprocessed original image to obtain a facial feature image;

[0053] S3. Perform eye feature detection and cropping on the facial feature image to obtain the left eye feature image and the right eye feature image respectively, and obtain user feature information based on the facial feature image;

[0054] S4. Use a deep learning model to process the facial feature image, left eye feature image, and right eye feature image, and combine them with the depth information image to calculate the gaze point;

[0055] S5. Combine gaze point and user feature information to output user features and gaze information, and execute the corresponding program in response to interactive actions.

[0056] Preferably, S4 specifically includes:

[0057] S401. Calculate facial pose feature parameters of facial feature images using a deep learning model;

[0058] S402. Further extract feature parameters of the left and right eye feature images using a deep learning model;

[0059] S403. Combine the facial feature image with the depth information image to obtain the facial depth information corresponding to the facial feature image;

[0060] S404. Combine facial feature pose, feature parameters of the further extracted left and right eye feature images, and facial depth information to calculate the gaze point.

[0061] Preferably, obtaining user feature information based on facial feature images specifically includes: extracting feature vectors from facial feature images using the FaceNet model, and determining whether there is user information in the preset user information database that matches the extracted feature vectors. If so, the user ID number is output; otherwise, the user information database is updated and the newly created user ID number is output.

[0062] Preferably, the feature parameters of the left-eye and right-eye feature images further extracted using a deep learning model specifically include:

[0063] The first convolution operation is performed on the eye image to obtain the first convolution eye image data;

[0064] Max pooling is performed on the first convolutional eye image data to obtain pooled eye image data;

[0065] A second convolution operation is performed on the pooled eye image data to obtain the second convolution eye image data;

[0066] Dilation convolution is performed on the second convolutional eye image data to obtain the feature parameters of the eye image.

[0067] Preferably, combining facial feature pose, feature parameters of the further extracted left and right eye feature images, and facial depth information to calculate the gaze point specifically includes: stitching the left eye feature image, right eye feature image, and facial feature image to obtain a gaze point without 3D information; obtaining 3D position parameters from the facial depth information and combining them with the gaze point without 3D information to obtain the actual gaze point.

[0068] Preferably, generating the depth information image of the preprocessed original image specifically includes: the preprocessed original image is passed through 5 convolutional feature extraction layers to extract 5 layers of convolutional features, and the extracted 5 layers of convolutional features are input into the C3D model. The data output by the C3D model is passed through 3 deconvolutional layers and then superimposed to obtain the depth information image.

[0069] Preferably, the corresponding program executed in response to the interactive action includes:

[0070] Gaze selection: Determine whether the gaze point has been in a certain area for more than a preset time; if so, simulate a touch event.

[0071] Scrolling and swiping controls determine whether the gaze point has been in the edge area for more than a preset time; if so, the corresponding scrolling / swiping operation is triggered.

[0072] The system highlights the content being gazed upon, and determines whether the gaze point has been located in a specific text or image area for more than a preset time. If so, the area is highlighted.

[0073] The system uses gaze drawing to determine whether to enter the drawing application. If so, it records the coordinates of the gaze point and draws the gaze point trajectory in real time.

[0074] To better understand this application, let's take a specific embodiment as an example. In one specific embodiment, such as... Figure 1As shown, after acquiring the images, the system performs a series of preprocessing steps to improve the accuracy and efficiency of subsequent analysis. Preprocessing includes converting the image data into tensor types for easier processing by the deep learning model, followed by Gaussian filtering to denoise and smooth the images, reducing random noise and enhancing image quality. Furthermore, to accommodate subsequent processing, the system resizes the image from 1920x1080 to 640x480; this scaling reduces computational complexity, making the subsequent model run more efficiently. The preprocessed image is then loaded into the facial feature detection program. The system selects and loads a pre-trained face detection model trained on a large amount of data to efficiently identify and locate facial features in the image. Detected facial targets are cropped, then converted to grayscale to reduce computational complexity, and histogram equalization is used to enhance contrast, making the facial features clearer. Afterward, facial deep learning is used for feature extraction. The extracted facial feature parameters are saved, and the three are concatenated after the left and right eye feature parameters are calculated.

[0075] like Figure 2 As shown, the input image first undergoes two 3x3 convolution operations (both convolutional layers are set to 64 channels), followed by a 2x2 max pooling layer to reduce size and extract key features. Next, the image data undergoes two more 3x3 convolution operations (both convolutional layers are set to 128 channels) to enhance high-level feature representation. Then, a 1x1 convolution reduces the number of channels to 64, lowering computational complexity and introducing non-linear features. Following this, the data enters a series of dilated convolutional layers with dilation rates of 2, 3, 5, and 11. These dilated convolutions expand the receptive field, capturing more contextual information without increasing the number of parameters, thus enabling the model to better extract global and local facial features. The entire architecture is designed to efficiently extract multi-scale facial features for subsequent facial analysis tasks.

[0076] Next, the eye-tracking module will call the detected face data one by one and use OpenCV's built-in eye cascade classifier to extract the eye positions in the images, ensuring accurate identification of the eye region from the face images and cropping the region images to lay the foundation for subsequent eye-tracking analysis. The extracted eye images will undergo normalization processing, including uniformly adjusting the size to 64x96 and performing histogram equalization on each eye image to improve contrast and detail. After these processing steps, the system uses a deep learning model to extract features from the eye images. The left and right eyes cropped from the images are treated with the same feature extraction method. The extracted left and right eye feature data will be saved, awaiting concatenation of the three facial feature data.

[0077] like Figure 3As shown, the input eye image first passes through two 3x3 convolutional layers with 64 channels to extract low-level features. Next, the data is passed through a 2x2 max-pooling layer to reduce its size, extracting key spatial features and reducing computation. Then, the image data passes through two more 3x3 convolutional layers with 128 channels to further extract richer features. To further reduce computational complexity, a 1x1 convolutional layer is then used to reduce the number of channels to 64. Afterward, the data enters a series of dilated convolutional layers to expand the receptive field and better capture multi-scale feature information. The first layer is a dilated convolutional layer with a dilation rate of (2,2), followed by convolutional layers with dilation rates of (3,3), (4,5), and (5,11). These dilated convolutions allow the model to expand the receptive field without increasing the kernel size, thereby capturing more contextual information. The entire process aims to efficiently extract multi-scale features from the eye region for use in subsequent deep learning tasks such as gaze prediction or other eye-tracking analyses.

[0078] like Figure 4 As shown, the left and right eye images are processed by an eye feature extraction module, with the outputs passed through a fully connected layer (output size 256). Simultaneously, the facial image is also processed by a deep learning feature extraction module, then passed through two fully connected layers to obtain a 64-dimensional feature vector. These extracted features (left and right eye features and facial features) are then concatenated to form a feature vector containing 576 parameters. The concatenated features are then passed through a fully connected layer (output size 256) to generate a gaze point without 3D information.

[0079] like Figure 5 As shown, a deep learning model is used to calculate the depth information of each pixel in the entire image. This module ultimately generates a depth information map that perfectly corresponds to the depth information of the original input image. Then, the average depth information of the face location in the depth information map is calculated, and finally, the average depth information is added to the foveated points without 3D information.

[0080] like Figure 6As shown, an encoder-decoder scheme is used. The encoder consists of five connected modules, each with corresponding sub-modules including: batch normalization (BN) layers, rectified linear units (ReLU), and convolutional layers (Conv). Convolutional feature extraction layers 3, 4, and 5 employ residual structures. The decoder mainly consists of a deconvolution module and a coplanar-driven disparity derivation module (C3D module). C3D (Convolutional 3D) is a deep learning model primarily used for tasks such as video analysis and action recognition. Its core lies in using 3D convolutional operations to simultaneously process temporal and spatial information. By applying 3D convolutional kernels to consecutive frames of the input video, C3D can effectively extract spatiotemporal features, thereby capturing the dynamic changes of objects in different time frames. Specifically, the model first extracts features from video frames through multiple 3D convolutional layers, then gradually reduces the feature dimensionality through activation functions and pooling layers, strengthening important features while suppressing irrelevant information. Ultimately, C3D can extract rich spatial information, making it perform well in tasks such as disparity derivation and depth estimation. The final model deconvolves the generated spatial information to restore the original image size and then overlays the spatial information to obtain a depth map. The training dataset for this module is primarily based on the NYU DepthV2 dataset, a widely used RGB-D image dataset created by New York University specifically for depth estimation and indoor scene understanding. This dataset contains over 4,000 640x480 resolution RGB images and their corresponding depth maps, acquired using the Microsoft Kinect sensor, with depths ranging from 0.5 meters to 10 meters.

[0081] like Figure 7 As shown, the system compares user information in the database with the input facial image, determining similarity by calculating the Archid distance between their feature vectors. If the similarity reaches a threshold, the two images are considered to be the same user, and no new user is created. If the similarity does not reach the threshold, the newly identified image is identified as a new user, and new user information is created. The user identity database periodically deletes expired user data to prevent excessive storage resource consumption. The main function of this module is to identify users attempting to interact, preventing interaction conflicts between multiple users on the in-vehicle display and interaction system.

[0082] The FaceNet model's data processing steps are as follows: First, it starts with the input layer, receiving an input facial image. Then, the image passes through multiple convolutional layers (CONV1, CONV2, CONV3, CONV4). Each layer extracts facial features at different levels through convolution operations. These convolutional layers progressively capture local and global features from the face image. After the convolutional layers, the feature maps are fed into fully connected layers (FC1, FC2) to further integrate the feature information, outputting a fixed-length vector. After the second fully connected layer, L2 regularization (L2) is applied to normalize the embedding vector to a unit length for subsequent similarity calculation and classification. Finally, the normalized feature vector represents the unique features of the input face and can be used for identity recognition and verification.

[0083] like Figure 8 As shown, the eye-tracking interaction system mainly provides the following interactive actions: gaze selection, scrolling and swiping control, highlighting gaze content, and gaze mapping.

[0084] Gaze selection: Map the user's gaze point to screen coordinates. Then, by defining hot zones for UI elements, determine whether the user's gaze point is within a certain hot zone. If the gaze point remains in a specific area for more than a set time (e.g., 1-2 seconds), simulate a touch event to achieve the gaze selection function.

[0085] Scrolling and swiping controls: If a user gazes at the edge of the screen (such as the top or bottom) for a certain period of time, a corresponding scrolling action is triggered, such as scrolling up or down the page; if the user's gaze moves horizontally along the screen, swiping controls are triggered, allowing users to switch pages left or right. This method, by combining gaze position monitoring and touch event simulation, enables scrolling and swiping control functions, providing users with a more natural and intuitive interactive experience.

[0086] Gaze-based highlighting: When a user is detected to have lingered in a specific text or image area for more than a set time (e.g., 0.5-1 seconds), that area will be highlighted, for example by changing the background color or border to indicate the content the user is currently looking at.

[0087] Gaze-based drawing: In drawing applications, the coordinates of the user's gaze are recorded in real time and drawn on the screen, enabling drawing using eye movements. The user's gaze trajectory is directly converted into a drawing path, allowing users to draw graphics or write text simply by looking at different positions on the screen, without the need for fingers or a stylus. This provides a novel interaction method for users with special needs.

[0088] like Figure 9As shown, a 77-inch monitor 1 is connected to an Android system host computer 2; an eye-tracking data processing module 3 is connected to RGB camera 1 4 and RGB camera 2 5 respectively; the eye-tracking data processing module 3 consists of an eye-tracking module 6, a human body 3D position detection module 7, an identity information recognition module 8 and an eye-tracking data packet sending module 9; the Android system host computer consists of an eye-tracking data packet sending module 9 and an eye-tracking data packet receiving module 10.

[0089] like Figure 10 As shown, it consists of two wide-angle RGB camera modules, a high-performance eye-tracking data processing module, a host computer system running Android, and a 77-inch display. The dual cameras can ensure high-precision eye-tracking without blind spots within a 2m horizontal and 1-3m deep area inside the vehicle.

[0090] Specifically, the camera module uses a high-quality 2.8-inch CMOS sensor with a high-resolution 2-megapixel image capture capability, ensuring that every frame is delicate and clear. Its specially designed 67.6° viewing angle effectively reduces distortion in the output image, providing a natural visual experience suitable for applications such as video conferencing, live streaming, and surveillance. It can output 1920x1080 full HD images and supports a frame rate of up to 60Hz, ensuring smoothness and stability of dynamic images and greatly enhancing the user's viewing experience.

[0091] In one specific embodiment, the eye-tracking data processing module is primarily based on a high-performance Ubuntu system development board equipped with the domestically produced RK3588S chip. The RK3588S is a 64-bit ARM architecture general-purpose SoC that integrates an octa-core processor (quad-core Cortex-A76 and quad-core Cortex-A55), a high-performance quad-core Mali-G610 GPU, and a 6TAI NPU, capable of smoothly running multiple applications and meeting various computing needs. The development board is equipped with 32GB of memory and provides four USB protocol interfaces for efficient acquisition of camera data, ensuring the stability and reliability of real-time image transmission. It also features a WAN port for data communication with an Android operating system host computer, supporting multiple network protocols to adapt to different application scenarios.

[0092] The Mali-G610 not only supports advanced image processing features such as HDR rendering, anti-aliasing, and texture compression, but also boasts multi-threaded processing capabilities. This allows for efficient resource allocation when processing high-resolution images, ensuring both image quality and processing efficiency. These features enable the camera to capture more detail in complex scenes, enhancing the user experience. Furthermore, this module integrates deep learning inference algorithms, leveraging the powerful computing capabilities of the NPU to perform real-time image analysis, quickly adapting to diverse application needs such as security monitoring, intelligent transportation, and human-computer interaction.

[0093] The Android interactive system's host computer is primarily based on an Android development board equipped with the RK3566. The RK3566 is a 64-bit ARM architecture general-purpose SoC that integrates a quad-core Cortex-A55 processor with a clock speed of up to 1.8GHz, providing powerful computing capabilities. It also features a high-performance Mali-G52 GPU, supporting graphics processing and image rendering. This system can smoothly run multiple applications, meeting various computing needs. The development board is typically configured with 4GB or 8GB of LPDDR4 memory and supports up to 128GB of eMMC storage to meet the needs of large-capacity data storage. Furthermore, it supports MicroSD card expansion, further enhancing storage flexibility.

[0094] The development board offers a rich set of interfaces, including multiple USB 3.0 and USB 2.0 ports for efficient data reception from cameras, ensuring the stability and reliability of real-time image transmission. It also features a single Ethernet WAN port and a Wi-Fi module for data communication with a host computer, supporting various network protocols to adapt to different application scenarios. This diverse connectivity allows the module to be used flexibly in various environments.

[0095] In summary, ① compared to traditional 2D position detection, the human 3D position detection module of this system achieves accurate calculation of the depth information of each pixel in the image through a deep learning model, thereby generating a depth information map that completely corresponds to the input image. The module adopts an encoder-decoder structure. The encoder contains 5 convolutional modules, among which convolutions 3, 4, and 5 of the feature extraction layer introduce an improved residual structure, optimizing the skip connection method based on the standard residual module, further improving the feature representation ability and effectively alleviating the gradient vanishing problem; the decoder consists of a deconvolution module and a coplanar-driven disparity derivation module (C3D+ module). The C3D+ module not only processes temporal and spatial information simultaneously through 3D convolution operations, but also integrates an attention mechanism, assigning higher weights to key features, thereby more accurately capturing the spatiotemporal features in the image sequence, making it exhibit higher accuracy and robustness in disparity derivation and depth estimation. Specifically, C3D+ extracts spatiotemporal features through multi-layer 3D convolution, attention modules, and pooling operations, then restores the original image size through deconvolution, and combines multi-scale spatial information to generate the final depth information map.

[0096] The module's innovation is also reflected in the optimized processing after the depth information map is generated. The system calculates the weighted average depth value of the corresponding region in the depth information map based on the detected face pixel coordinates, and introduces a noise filtering strategy to improve the accuracy of the depth value calculation, ultimately serving as the user's 3D position parameter. When this 3D position parameter is fused with gaze point data without 3D information, a dynamic weight allocation strategy based on Kalman filtering is adopted, thereby achieving higher real-time performance and stability in the comprehensive analysis of spatial and gaze information.

[0097] The module's deep learning model is trained on the extended NYU DepthV2 dataset, which includes samples of various complex scenes and dynamic objects, further enhancing the model's generalization ability and stability. Through the combination of depth information maps and 3D position parameters, along with improved model design, this module significantly enhances the system's adaptability to complex scenes and dynamic environments while improving the accuracy and comprehensiveness of human position detection and spatial information.

[0098] ② Compared to traditional single-branch eye-tracking models, this system adopts an improved multi-branch deep learning architecture, significantly improving the accuracy and efficiency of feature extraction. Specifically, the model extracts features from the left eye, right eye, and face through a multi-branch structure. Each branch independently designs an optimized convolutional neural network module, further enhancing the ability to extract local feature information. The left and right eye branches share the same feature extraction module, containing multiple 3x3 convolutional layers and improved dilated convolutional layers (dilation rates of (2,2), (3,3), (4,5), (5,11)). A dynamic dilation rate adjustment mechanism is added to the dilated convolutional layers to adaptively adjust the dilation rate according to the distribution of input data, thereby more efficiently capturing multi-scale features of the eye region. The face branch introduces an attention mechanism to dynamically focus on key feature regions based on multiple convolutions and dilated convolutions, effectively fusing the global receptive field and detailed features of facial information.

[0099] Before the feature vectors output from the left and right eye branches and the face branch are converted into 256-dimensional embedding vectors through a fully connected layer, a feature normalization and denoising module is added to remove redundant information and enhance the discriminative power of the features. Finally, in the feature concatenation module, these embedding vectors are combined into a vector containing 576-dimensional features, and optimized through an adaptive feature weight allocation mechanism to improve the effectiveness of feature fusion and the accuracy of gaze point prediction. Through multi-branch feature extraction, dynamic adjustment, and optimized fusion, this system can more efficiently capture local and global contextual information in eye-tracking data, while exhibiting higher accuracy and robustness under complex lighting and angular conditions.

[0100] By training with the extended MPIIGaze dataset, this multi-branch architecture further enhances the model's generalization ability and stability by adding sample data under non-ideal conditions. Compared to ordinary multi-branch models, this improved architecture not only significantly improves the accuracy of fixation prediction but also strengthens the model's adaptability in real-world applications, especially performing better in dynamic scenes.

[0101] ③ Compared to traditional identity information management methods, this system introduces an improved deep learning-based identity information management module. Through an optimized FaceNet model, it achieves high-precision user identification and dynamic management, ensuring conflict-free interaction in multi-user environments. System functions include user identification, maintaining a user information database, dynamically adding new users, deleting expired users, and merging similar user data. The core FaceNet model incorporates an adaptive threshold adjustment mechanism when calculating the Euclidean distance between the input facial image and the user feature vectors in the database. This dynamically adjusts the similarity judgment criteria based on the current user data distribution, further improving the accuracy and robustness of identification. When the similarity exceeds the threshold, the system determines it to be the same user, avoiding duplicate user information creation.

[0102] The improved FaceNet model employs a hierarchical design. Building upon the traditional multi-convolutional layers (such as CONV1, CONV2, CONV3, and CONV4) that progressively extract local and global facial features, it introduces a multi-scale feature fusion module and an attention mechanism to dynamically enhance the expressive power of key feature regions. Subsequently, features are further integrated through fully connected layers (FC1 and FC2), and optimized L2 regularization is used to standardize the feature vectors, mapping them to a unit hypersphere. This significantly improves the consistency of feature representation and the accuracy of similarity calculation. Furthermore, a compression storage mechanism is introduced when storing feature vectors in the database, reducing storage footprint while maintaining efficient retrieval capabilities.

[0103] This module also optimizes the user state management process for multi-user interaction scenarios. When a user's gaze leaves the operating display for more than 5 seconds or they actively exit the system, the system will automatically release the current interaction permissions through an improved permission release strategy, providing an opportunity for the next user to operate. At the same time, by introducing a timeout user cleanup algorithm, user data that has not been active for a long time is periodically deleted, further reducing the consumption of storage resources.

[0104] By combining the improved FaceNet model with a dynamic database maintenance strategy, this system significantly enhances identity recognition accuracy and interaction management efficiency in multi-user environments, and demonstrates greater robustness and stability in complex scenarios. Compared to traditional models, this system not only achieves more accurate user recognition in in-vehicle display and interaction systems, but also effectively optimizes resource utilization and user experience in dynamic environments, providing a solid guarantee for the reliability of multi-user management.

[0105] It is obvious that those skilled in the art can make various modifications and alterations to the embodiments of this application without departing from the spirit and scope of this application. In this way, this application also aims to cover such modifications and alterations if they fall within the scope of the claims and their equivalents. The word "comprising" does not exclude the presence of other elements or steps not listed in the claims. The simple fact that certain measures are described in mutually different dependent claims does not indicate that a combination of these measures cannot be used for profit. Any reference numerals in the claims should not be considered limiting in scope.

Claims

1. A human-machine interaction method based on eye tracking, characterized in that, The method comprises the following steps: S1, obtaining an original image and preprocessing the original image; S2, generating a depth information image of the preprocessed original image, and performing face feature detection and extraction on the preprocessed original image to obtain a face feature image; S3, performing eye feature detection and extraction on the face feature image to obtain a left eye feature image and a right eye feature image, respectively, and obtaining user feature information according to the face feature image; S4, processing the face feature image, the left eye feature image and the right eye feature image by using a deep learning model, and calculating a gaze point in combination with the depth information image, specifically comprising: S401, calculating a face posture feature parameter of the face feature image by using the deep learning model; S402, further extracting feature parameters of the left eye feature image and the right eye feature image by using the deep learning model; S403, combining the face feature image and the depth information image to obtain face depth information corresponding to the face feature image; S404, combining the face feature posture, the further extracted feature parameters of the left eye feature image and the right eye feature image, and the face depth information to calculate the gaze point, specifically comprising: splicing the left eye feature image, the right eye feature image and the face feature image to obtain a 3D information-free gaze point; obtaining a 3D position parameter from the face depth information, and combining the 3D information-free gaze point with the 3D position parameter by using a dynamic weight distribution strategy based on Kalman filtering to obtain an actual gaze point; S5, outputting user features and gaze information in combination with the gaze point and the user feature information, and executing a corresponding program in response to an interactive action.

2. The eye movement tracking based human computer interaction method of claim 1, wherein, The user feature information obtained from the face feature image specifically comprises: extracting a feature vector of the face feature image by using a FaceNet model, and determining whether there is user information matching the extracted feature vector in a pre-set user information database, if yes, outputting a user ID number, if not, updating the user information database and outputting a newly created user ID number.

3. The eye movement tracking based human computer interaction method of claim 1, wherein, The further extraction of the feature parameters of the left eye feature image and the right eye feature image by using the deep learning model specifically comprises: performing a first convolution operation on the eye image to obtain first convolutional eye image data; performing maximum pooling on the first convolutional eye image data to obtain pooled eye image data; performing a second convolution operation on the pooled eye image data to obtain second convolutional eye image data; performing dilated convolution operation on the second convolutional eye image data to obtain the feature parameters of the eye image.

4. The eye movement tracking based human computer interaction method of claim 1, wherein, The generation of the depth information image of the preprocessed original image specifically comprises: the preprocessed original image respectively extracts 5 layers of convolution features through 5 layers of convolution feature extraction layers, and the extracted 5 layers of convolution features are input into a C3D model; the data output by the C3D model is superimposed after passing through 3 layers of deconvolution layers to obtain the depth information image.

5. The eye movement tracking based human computer interaction method of claim 1, wherein, The execution of the corresponding program in response to the interactive action comprises: gaze selection, determining whether the gaze point is located in a certain area for more than a pre-set time, if yes, simulating a touch event; scrolling and sliding control, determining whether the gaze point is located in an edge area for more than a pre-set time, if yes, triggering a corresponding scrolling / sliding operation; The gaze content is highlighted, and it is judged whether the gaze point is located in a specific text or image area for more than a preset time, and if so, the area is highlighted; The gaze drawing judges whether the drawing application is entered, and if so, the gaze point coordinates are recorded in real time and the gaze point trajectory is drawn.

6. An eye-tracking-based human-computer interaction system for performing the eye-tracking-based human-computer interaction method according to any one of claims 1-5, characterized in that, It comprises: A user data processing module and a user interaction action module; The user data processing module is configured to collect image data, extract user feature information, and calculate the user's gaze point; The user interaction action module receives the user's gaze point and user information through the TCP protocol, and then processes the user's gaze data.

7. The human-computer interaction system based on eye movement tracking according to claim 6, characterized in that, The user data processing module comprises an eye movement tracking unit, a user identification unit, and a human body 3D position detection unit; the eye movement tracking unit is configured to extract gaze point information; the user identification unit is configured to identify the identity of the user and distinguish between the operating user and the to-be-operated module; the human body 3D position detection unit is configured to extract the detected user position and correct the gaze information of the eye movement tracking.

8. An eye-tracking based human-computer interaction device for performing the eye-tracking based human-computer interaction method according to any one of claims 1-5, characterized in that, It comprises: A first RGB camera, a second RGB camera, an eye movement tracking data processing host computer, an Android operating system host computer, and a display; The first RGB camera and the second RGB camera are respectively connected to the eye movement tracking data processing host computer; the eye movement tracking data processing host computer communicates data with the Android operating system host computer through Tcp; the Android operating system host computer is connected to the display.

Images