Human-computer interaction method, system and device based on eye movement tracking

Abstract

The invention provides a human-computer interaction method based on eye movement tracking, and the method is characterized in that the method comprises the steps: S1, obtaining an original image, and carrying out the preprocessing of the original image; s2, generating a depth information image of the preprocessed original image, and performing facial feature detection and interception on the preprocessed original image to obtain a facial feature image; s3, performing eye feature detection and interception on the facial feature image to respectively obtain a left eye feature image and a right eye feature image, and obtaining user feature information according to the facial feature image; s4, processing the facial feature image, the left eye feature image and the right eye feature image by using a deep learning model, and calculating a fixation point in combination with the depth information image; and S5, outputting user features and gaze information by combining the gaze point and the user feature information, and executing a corresponding program in response to the interactive action. According to the invention, accurate calculation of the depth information of each pixel point in the image is realized, and the depth information graph completely corresponding to the input image is generated.

Description

Technical Field

[0001] The present application relates to the field of eye tracking technology, and in particular to a human-computer interaction method, system and device based on eye tracking. Background Art

[0002] Deep learning is a machine learning method based on artificial neural networks. Its core lies in feature extraction and modeling of data through multi-layer neural networks. Machine learning mainly consists of two steps: modeling and training. Modeling mainly involves building a neural network model to extract features and perform numerical operations on input data, and set the format of output data. Training requires the construction of a data set. The data set mainly contains two data: input sample data and expected correct results. The desktop eye tracking model is trained by using the data set. During the training process, these networks learn from a large amount of data, adjust the internal connection weights, and ultimately enable the model to accurately complete tasks, such as determining whether there is a cat in a picture or translating a text. In short, deep learning allows computers to "learn" to summarize patterns from data, replacing traditional manual programming to solve problems.

[0003] Desktop eye tracking technology is a technology that obtains users' gaze behavior by monitoring and analyzing their eye movements. Its working principle mainly uses deep learning.

[0004] It mainly detects faces through input images. After the face is detected, the face image is cut. Cutting means cutting the face image from the original input image. Subsequent image processing will only process the cut image.

[0005] Desktop eye tracking technology first detects and segments the face, then detects the eyes of the captured face, and continues to capture the eyes after the left and right eyes are detected. The detected left eye, right eye, and face images are then subjected to feature extraction. Feature extraction refers to the process of extracting relevant parameters by performing multiple convolution operations on the captured image through a multi-layer convolutional neural network and using a fully connected operation to select relevant parameters.

[0006] The desktop eye tracking device extracts features from the left and right eyes and the face respectively, and then performs feature fusion. Feature fusion refers to concatenating the extracted features and performing a convolution operation to fuse the features of the three captured images. Finally, the fused features are fully connected to obtain the user's gaze point position on the screen.

[0007] However, most desktop eye tracking technologies are currently being used on large screens. However, the screen sizes that are currently within the eye tracking range are mainly within 55 inches, and there is no precedent for using it on a 77-inch screen. Summary of the invention

[0008] In order to solve the above technical problems, a human-computer interaction method based on eye tracking is proposed in a first aspect of the present application, comprising: S1, obtaining an original image and preprocessing the original image;

[0009] S2, generating a depth information image of the preprocessed original image, and performing facial feature detection and interception on the preprocessed original image to obtain a facial feature image;

[0010] S3, performing eye feature detection and interception on the facial feature image to obtain a left eye feature image and a right eye feature image, respectively, and obtaining user feature information according to the facial feature image;

[0011] S4, using a deep learning model to process the facial feature image, the left eye feature image, and the right eye feature image, and calculating the gaze point in combination with the depth information image;

[0012] S5. Output the user characteristics and gaze information in combination with the gaze point and the user characteristic information, and execute a corresponding program in response to the interactive action.

[0013] Preferably, S4 specifically includes:

[0014] S401, calculating facial posture feature parameters of the facial feature image through a deep learning model;

[0015] S402, further extracting feature parameters of the left eye feature image and the right eye feature image through a deep learning model;

[0016] S403, combining the facial feature image and the depth information image to obtain facial depth information corresponding to the facial feature image;

[0017] S404: Combine the facial feature posture, the feature parameters of the further extracted left eye feature image and right eye feature image, and the facial depth information to calculate the gaze point.

[0018] Preferably, obtaining user feature information based on the facial feature image specifically includes: using the FaceNet model to extract the feature vector of the facial feature image, and determining whether there is user information matching the extracted feature vector in a preset user information database, if so, outputting the user ID number, if not, updating the user information database and outputting the newly created user ID number.

[0019] Preferably, further extracting feature parameters of the left eye feature image and the right eye feature image through the deep learning model specifically includes:

[0020] Performing a first convolution operation on the eye image to obtain first convolution eye image data;

[0021] Performing maximum pooling on the first convolution eye image data to obtain pooled eye image data;

[0022] Performing a second convolution operation on the pooled eye image data to obtain second convolution eye image data;

[0023] Perform a dilation convolution operation on the second convolution eye image data to obtain feature parameters of the eye image.

[0024] Preferably, combining the facial feature posture, the feature parameters of the further extracted left eye feature image and the right eye feature image, and the facial depth information to calculate the gaze point specifically includes: splicing the left eye feature image, the right eye feature image, and the facial feature image to obtain the gaze point without 3D information; obtaining 3D position parameters from the facial depth information, and combining them with the gaze point without 3D information to obtain the actual gaze point.

[0025] Preferably, generating a depth information image of the preprocessed original image specifically includes: the preprocessed original image is respectively passed through 5 layers of convolutional feature extraction layers to extract 5 layers of convolutional features, and the extracted 5 layers of convolutional features are input into the C3D model, and the data output by the C3D model is respectively passed through 3 layers of deconvolution layers and then superimposed to obtain a depth information image.

[0026] Preferably, executing a corresponding program in response to an interactive action includes:

[0027] Gaze selection: determine whether the gaze point has been in a certain area for more than a preset time. If so, simulate a touch event.

[0028] Scrolling and sliding control: determine whether the gaze point is in the edge area for more than a preset time, and if so, trigger the corresponding scrolling / sliding operation;

[0029] Highlight the gaze content, determine whether the gaze point is in a specific text or image area for more than a preset time, and if so, highlight the area;

[0030] Look at the drawing and determine whether to enter the drawing application. If so, record the gaze point coordinates in real time and draw the gaze point trajectory.

[0031] In the second aspect of the present application, a human-computer interaction system based on eye tracking is proposed to execute the aforementioned human-computer interaction method based on eye tracking, including: a user data processing module and a user interaction action module; the user data processing module is configured to collect image data, extract user feature information, and calculate user gaze points; the user interaction action module is configured to receive user gaze points and user information through a TCP protocol, and then process the user gaze data.

[0032] Preferably, the user data processing module includes: an eye tracking unit, a user identification unit and a human 3D position detection unit; the eye tracking unit is configured to extract gaze point information; the user identification unit is configured to identify the identity of the user and distinguish between the operating user and the module to be operated; the human 3D position detection unit is configured to extract the detected user position and correct the gaze information of the eye tracking.

[0033] In the third aspect of the present application, a human-computer interaction device based on eye tracking is proposed to execute the aforementioned human-computer interaction method based on eye tracking, which is characterized in that it includes: a first RGB camera, a second RGB camera, an eye tracking data processing host computer, an Android operating system host computer, and a display; the first RGB camera and the second RGB camera are respectively connected to the eye tracking data processing host computer; the eye tracking data processing host computer communicates data with the Android operating system host computer through TCP; the Android operating system host computer is connected to the display.

[0034] Compared with the prior art, the beneficial results of this application are:

[0035] Compared with traditional two-dimensional position detection, the human body 3D position detection module of this system realizes the accurate calculation of the depth information of each pixel in the image through the deep learning model, thereby generating a depth information map that completely corresponds to the input image;

[0036] An improved multi-branch deep learning architecture is adopted to significantly improve the accuracy and efficiency of feature extraction;

[0037] An improved deep learning-based identity information management module was introduced, which achieved high-precision identity recognition and dynamic management of users through the optimized FaceNet model, ensuring that interactions in a multi-user environment do not conflict. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present application. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0039] Figure 1 is a flow chart of a human-computer interaction method based on eye tracking according to an embodiment of the present application;

[0040] Figure 2 is a flowchart of facial deep learning feature extraction based on the human-computer interaction method of eye tracking according to an embodiment of the present application;

[0041] Figure 3 is a flow chart of eye deep learning feature extraction of a human-computer interaction method based on eye tracking according to an embodiment of the present application;

[0042] Figure 4 is a flow chart of gaze point generation according to the human-computer interaction method based on eye tracking according to an embodiment of the present application;

[0043] Figure 5 It is a process of adding 3D position parameters and gaze points without 3D information according to the human-computer interaction method based on eye tracking in an embodiment of the present application;

[0044] Figure 6 is a flow chart of a 3D depth data extraction model of a human-computer interaction method based on eye tracking according to an embodiment of the present application;

[0045] Figure 7 is a user identification flow chart of a human-computer interaction method based on eye tracking according to an embodiment of the present application;

[0046] Figure 8 is a flow chart of an interactive system of a human-computer interaction method based on eye tracking according to an embodiment of the present application;

[0047] Fig. 9 is a system block diagram of a human-computer interaction system based on eye tracking according to an embodiment of the present application;

[0048] Fig.10 It is a structural schematic diagram of a human-computer interaction device based on eye tracking according to an embodiment of the present application.

[0050] 1. 77-inch display; 2. Android system host computer; 3. Eye tracking data processing module; 4. RGB camera No. 1; 5. RGB camera No. 2; 6. Eye tracking module; 7. Human 3D position detection module; 8. Identity information recognition module; 9. Eye movement data packet sending module; 10. Eye movement data packet receiving module. DETAILED DESCRIPTION

[0051] In the following detailed description, reference is made to the accompanying drawings, which form a part of the detailed description and are shown by illustrative specific embodiments in which the present application can be practiced. To this end, directional terms, such as "top", "bottom", "left", "right", "up", "down", etc., are used with reference to the orientation of the figures described. Because the components of the embodiments can be positioned in several different orientations, directional terms are used for the purpose of illustration and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the present application. Therefore, the following detailed description should not be adopted in a limiting sense, and the scope of the present application is defined by the appended claims.

[0052] like Figure 1-8 As shown, a human-computer interaction method based on eye tracking includes: S1, obtaining an original image and preprocessing the original image;

[0053] S2, generating a depth information image of the preprocessed original image, and performing facial feature detection and interception on the preprocessed original image to obtain a facial feature image;

[0054] S3, performing eye feature detection and interception on the facial feature image to obtain a left eye feature image and a right eye feature image, respectively, and obtaining user feature information according to the facial feature image;

[0055] S4, using a deep learning model to process the facial feature image, the left eye feature image, and the right eye feature image, and calculating the gaze point in combination with the depth information image;

[0056] S5. Output the user characteristics and gaze information in combination with the gaze point and the user characteristic information, and execute a corresponding program in response to the interactive action.

[0057] Preferably, S4 specifically includes:

[0058] S401, calculating facial posture feature parameters of the facial feature image through a deep learning model;

[0059] S402, further extracting feature parameters of the left eye feature image and the right eye feature image through a deep learning model;

[0060] S403, combining the facial feature image and the depth information image to obtain facial depth information corresponding to the facial feature image;

[0061] S404: Combine the facial feature posture, the feature parameters of the further extracted left eye feature image and right eye feature image, and the facial depth information to calculate the gaze point.

[0062] Preferably, obtaining user feature information based on the facial feature image specifically includes: using the FaceNet model to extract the feature vector of the facial feature image, and determining whether there is user information matching the extracted feature vector in a preset user information database, if so, outputting the user ID number, if not, updating the user information database and outputting the newly created user ID number.

[0063] Preferably, further extracting feature parameters of the left eye feature image and the right eye feature image through the deep learning model specifically includes:

[0064] Performing a first convolution operation on the eye image to obtain first convolution eye image data;

[0065] Performing maximum pooling on the first convolution eye image data to obtain pooled eye image data;

[0066] Performing a second convolution operation on the pooled eye image data to obtain second convolution eye image data;

[0067] Perform a dilation convolution operation on the second convolution eye image data to obtain feature parameters of the eye image.

[0068] Preferably, combining the facial feature posture, the feature parameters of the further extracted left eye feature image and the right eye feature image, and the facial depth information to calculate the gaze point specifically includes: splicing the left eye feature image, the right eye feature image, and the facial feature image to obtain the gaze point without 3D information; obtaining 3D position parameters from the facial depth information, and combining them with the gaze point without 3D information to obtain the actual gaze point.

[0069] Preferably, generating a depth information image of the preprocessed original image specifically includes: the preprocessed original image is respectively passed through 5 layers of convolutional feature extraction layers to extract 5 layers of convolutional features, and the extracted 5 layers of convolutional features are input into the C3D model, and the data output by the C3D model is respectively passed through 3 layers of deconvolution layers and then superimposed to obtain a depth information image.

[0070] Preferably, executing a corresponding program in response to an interactive action includes:

[0071] Gaze selection: determine whether the gaze point has been in a certain area for more than a preset time. If so, simulate a touch event.

[0072] Scrolling and sliding control: determine whether the gaze point is in the edge area for more than a preset time, and if so, trigger the corresponding scrolling / sliding operation;

[0073] Highlight the gaze content, determine whether the gaze point is in a specific text or image area for more than a preset time, and if so, highlight the area;

[0074] Look at the drawing and determine whether to enter the drawing application. If so, record the gaze point coordinates in real time and draw the gaze point trajectory.

[0075] In order to better understand the present application, a specific embodiment is taken as an example. In a specific embodiment, Figure 1As shown in the figure, after acquiring the image, the system will perform a series of preprocessing on the data to improve the accuracy and efficiency of subsequent analysis. Preprocessing includes converting the image data into a tensor type to facilitate subsequent deep learning model processing, and then using Gaussian filtering technology to denoise and smooth the image to reduce random noise and enhance image quality. In addition, in order to meet the needs of subsequent processing, the system adjusts the image size from 1920x1080 to 640x480. This scaling process reduces the computational complexity and makes the subsequent model run more efficiently. The preprocessed image will be loaded into the facial feature detection program. The system selects and loads a pre-trained face detection model trained with a large amount of data to efficiently identify and locate facial features in the image. The detected face target will be intercepted, and then grayscaled to reduce the computational complexity, and the contrast will be enhanced by histogram equalization to make the facial features clearer. After that, facial deep learning is used for feature extraction, and the extracted facial feature parameters will be saved, and the three will be spliced ​​after the left eye feature parameters and the right eye feature parameters are calculated.

[0076] like Figure 2 As shown in the figure, first, the input image undergoes two 3x3 convolution operations (both convolution layers are set to 64 channels), and then enters a 2x2 maximum pooling layer to reduce the size and extract the main features. Next, the image data undergoes two 3x3 convolution operations (both convolution layers are set to 128 channels) to enhance the high-level feature expression. Then, the number of channels is reduced to 64 through 1x1 convolution, which reduces the computational complexity and introduces nonlinear features. Next, the data enters a series of dilated convolution layers with dilation rates of 2, 3, 5, and 11, respectively. These dilated convolutions can expand the receptive field and capture more contextual information without increasing the number of parameters, so that the model can better extract global and local features of the face. The entire structure is designed to efficiently extract multi-scale facial features for subsequent facial analysis tasks.

[0077] Next, the eye movement module will call the detected face data one by one, and use the eye cascade classifier built into OpenCV to extract the eye position in the image, ensuring that the eye area is accurately identified from the face image and the regional image is captured, laying the foundation for subsequent eye movement analysis. The extracted eye images will be normalized, including uniformly adjusting the size to 64x96, and performing histogram equalization on each eye image to improve contrast and details. After completing these processes, the system uses a deep learning model to extract features from the eye images. The same feature extraction method is used for the left and right eyes captured in the image. The extracted left eye feature data and right eye feature data will be saved, waiting for the facial feature data to splice the three.

[0078] like Figure 3As shown in the figure, first, the input eye image passes through two 3x3 convolution layers with the convolution layer channels set to 64 to extract low-level features. Next, the data is reduced in size through a 2x2 maximum pooling layer to extract the main spatial features and reduce the amount of computation. Then, the image data passes through two 3x3 convolution layers with the convolution layer channels set to 128 to further extract richer features. In order to reduce the computational complexity, a 1x1 convolution layer is then used to reduce the number of channels to 64. After that, the data enters a series of dilated convolution layers to expand the receptive field and better capture multi-scale feature information. The first is a dilated convolution layer with a dilation rate of (2,2), followed by convolution layers with dilation rates of (3,3), (4,5) and (5,11). These dilated convolutions allow the model to expand the receptive field without increasing the size of the convolution kernel, thereby obtaining more contextual information. The entire process aims to efficiently extract multi-scale features of the eye region so that these features can be used in subsequent deep learning tasks for gaze point prediction or other eye movement analysis.

[0079] like Figure 4 As shown in the figure, the left eye image and the right eye image are respectively subjected to feature extraction by the eye feature extraction module, and the output passes through the fully connected layer (the output size of the fully connected layer is 256). At the same time, the facial image is also processed by the deep learning feature extraction module, and then passes through two fully connected layers to finally obtain a 64-dimensional feature vector. These extracted features (left and right eye and facial features) are then spliced ​​together to form a feature vector containing 576 parameters. The spliced ​​features pass through the fully connected layer (the output size of the fully connected layer is 256) to generate a fixation point without 3D information.

[0080] like Figure 5 As shown in the figure, the depth information of each pixel of the entire image is calculated through the deep learning model. This module will eventually generate a depth information map that completely corresponds to the depth information of the input original image. Then the average depth information of the face position in the depth information map is calculated, and finally the average depth information is added to the fixation point without 3D information.

[0081] like Figure 6As shown in the figure, the encoder-decoder scheme is used. The encoder consists of 5 connected modules, each of which has corresponding submodules including batch normalization layer (BN), rectified linear unit (ReLU) and convolution layer (Conv). Among them, the convolution feature extraction layers 3, 4, and 5 adopt residual structure. The decoder includes a decoder mainly composed of a deconvolution module and a coplanar driven disparity inference module (C3D module). C3D (Convolutional 3D) is a deep learning model mainly used for tasks such as video analysis and action recognition. Its core is to use three-dimensional convolution operations to simultaneously process temporal and spatial information. C3D can effectively extract spatiotemporal features by applying three-dimensional convolution kernels to consecutive frames of the input video, thereby capturing the dynamic changes of objects in different time frames. Specifically, the model first extracts features from video frames through multiple three-dimensional convolution layers, and then gradually reduces the feature dimensions through activation functions and pooling layers, strengthens important features, and suppresses irrelevant information. Finally, C3D can extract rich spatial information, enabling it to perform well in tasks such as disparity inference and depth estimation. The final model deconvolves the generated spatial information to restore the original image size, and superimposes the spatial information to obtain the depth information map. The training dataset of this module is mainly based on the NYU DepthV2 dataset, which is a widely used RGB-D image dataset created by New York University and is specifically used for depth estimation and indoor scene understanding. The dataset contains more than 4,000 640x480 resolution RGB images and their corresponding depth maps. The depth information is obtained through the Microsoft Kinect sensor, with a depth range from 0.5 meters to 10 meters.

[0082] like Figure 7 As shown, the user information in the database is compared with the input facial image, and the similarity between the two is determined by calculating the Achilles distance of the feature vectors of the two. When the similarity reaches the threshold range, it is determined that the two are the same user, and there is no need to create a new user. When the similarity does not reach the threshold, the newly identified portrait is determined to be a new user, and new user information is created. The user identity database needs to delete the timed user data at regular intervals to prevent user data from occupying too much storage resources. The main function of this module is to identify the user who attempts to interact, so as to prevent multiple users from interacting with the in-vehicle display interaction system.

[0083] The FaceNet model data processing steps are as follows. First, it starts from the input layer and receives an input facial image. Then, the image passes through multiple convolutional layers (CONV1, CONV2, CONV3, CONV4). Each layer extracts facial features at different levels through convolution operations. The role of these convolutional layers is to gradually capture local and global features in the face image. After the convolutional layer, the feature map is input to the fully connected layer (FC1, FC2) to further integrate the feature information and output a fixed-length vector. After the second fully connected layer, L2 regularization (L2) is applied to normalize the embedded vector to unit length for subsequent similarity calculation and classification. Finally, the standardized feature vector is used to represent the unique features of the input face, which can be used for identity recognition and verification.

[0084] like Figure 8 As shown in the figure, the eye movement interaction system mainly provides the following interactive actions: gaze selection, scrolling and sliding control, gaze content highlighting, and line of sight drawing.

[0085] Gaze selection: Map the user's gaze point to the screen coordinates. Then, by defining the hotspots of UI elements, determine whether the user's gaze point is within a hotspot. If the gaze point stays in a specific area for more than a set time (such as 1-2 seconds), simulate a touch event to implement the gaze selection function.

[0086] Scrolling and sliding control: If the user gazes at the edge of the screen (such as the top or bottom) for a certain period of time, the corresponding scrolling operation is triggered, such as scrolling the page up and down; if the user's gaze point moves horizontally along the screen, the sliding control is triggered to switch pages left and right. This method combines the monitoring of the gaze point position with the simulation of touch events to achieve scrolling and sliding control functions, providing users with a more natural and intuitive interactive experience.

[0087] Highlighting gaze content: When it is detected that the user stays in a specific text or image area for more than a set time (such as 0.5-1 second), the area will be highlighted, for example, by changing the background color or border to prompt the user what they are currently gazing at.

[0088] Gaze drawing: In drawing applications, the coordinates of the user's gaze are recorded in real time and drawn on the screen, so that the function of drawing with sight is realized. The user's sight track is directly converted into a drawing path, so that the user can draw graphics or write by simply looking at different positions on the screen without using fingers or stylus, thus providing a novel interaction method for users with special needs.

[0089] like Fig. 9As shown, the 77-inch display 1 is connected to the Android system host computer 2; the eye tracking data processing module 3 is respectively connected to the RGB camera No. 1 4 and the RGB camera No. 2 5; wherein the eye tracking data processing module 3 is composed of: an eye tracking module 6, a human body 3D position detection module 7, an identity information recognition module 8 and an eye movement data packet sending module 9; the Android system host computer is composed of an eye movement data packet sending module 9 and an eye movement data packet receiving module 10.

[0090] like Fig.10 As shown in the figure, it consists of two wide-angle RGB camera modules, a high-performance eye tracking data processing module, a host computer system equipped with an Android operating system, and a 77-inch display. The dual cameras can ensure high-precision eye tracking without blind spots within the range of 2m horizontally and 1-3m deep in the car.

[0091] Specifically, the camera module uses a high-quality 2.8-inch CMOS sensor with a 2-megapixel high-resolution image capture capability, ensuring that every frame is delicate and clear. The specially designed 67.6° viewing angle effectively reduces the distortion of the output image, providing a natural visual experience, suitable for application scenarios such as video conferencing, live broadcasting and monitoring. It can output 1920x1080 full HD images and support a frame rate of up to 60Hz, ensuring the smoothness and stability of dynamic images, greatly improving the user's viewing experience.

[0092] In a specific embodiment, the eye tracking data processing module is mainly based on a high-performance Ubuntu system development board equipped with a domestic chip RK3588S. RK3588S is a general-purpose SoC based on a 64-bit ARM architecture, integrating quad-core Cortex-A76 and quad-core Cortex-A55 octa-core processors, Mali-G610 high-performance quad-core GPU, and 6TAI NPU computing power, which can smoothly run multiple applications and meet various computing needs. The development board is equipped with 32GB of memory and provides 4 USB protocol interfaces for efficient camera data collection to ensure the stability and reliability of real-time image transmission. It is also equipped with a WAN port for data communication with the Android operating system host computer and supports multiple network protocols to adapt to different application scenarios.

[0093] Mali-G610 not only supports advanced image processing features such as HDR rendering, anti-aliasing and texture compression, but also has multi-threaded processing capabilities, which can effectively allocate resources when processing high-resolution images to ensure the quality and processing efficiency of graphics. These features enable the camera to capture more details in complex scenes and improve the user experience. In addition, the module can also integrate deep learning inference algorithms, analyze images in real time through the powerful computing power of NPU, and quickly adapt to different application requirements, such as security monitoring, intelligent transportation and human-computer interaction.

[0094] The host computer of the Android interactive system is mainly based on the Android development board equipped with RK3566. RK3566 is a general-purpose SoC based on 64-bit ARM architecture, integrating a quad-core Cortex-A55 processor with a main frequency of up to 1.8GHz, providing powerful computing power, and equipped with a Mali-G52 high-performance GPU to support graphics processing and image rendering. The system can run multiple applications smoothly to meet various computing needs. The development board is usually equipped with 4GB or 8GB of LPDDR4 memory and supports up to 128GB of eMMC storage to meet the needs of large-capacity data storage. In addition, it also supports MicroSD card expansion to further enhance storage flexibility.

[0095] The development board provides a variety of interfaces, including multiple USB 3.0 and USB 2.0 interfaces, which are used to efficiently receive data from the camera and ensure the stability and reliability of real-time image transmission. It is also equipped with an Ethernet WAN port and a Wi-Fi module to communicate with the host computer and support a variety of network protocols to adapt to different application scenarios. This diverse connection method enables the module to be flexibly applied in different environments.

[0096] In summary, ① Compared with traditional two-dimensional position detection, the human body 3D position detection module of this system realizes the accurate calculation of the depth information of each pixel in the image through the deep learning model, thereby generating a depth information map that completely corresponds to the input image. The module adopts an encoder-decoder structure. The encoder contains 5 convolution modules, among which convolution 3, 4, and 5 of the feature extraction layer introduce an improved residual structure. On the basis of the standard residual module, the jump connection method is optimized to further improve the feature expression ability and effectively alleviate the gradient disappearance problem; the decoder consists of a deconvolution module and a coplanar driven disparity derivation module (C3D+ module). The C3D+ module not only processes temporal and spatial information simultaneously through three-dimensional convolution operations, but also integrates an attention mechanism to give higher weights to key features, thereby more accurately capturing the spatiotemporal features in the image sequence, so that it can show higher accuracy and robustness in disparity derivation and depth estimation. C3D+ specifically extracts spatiotemporal features through multi-layer three-dimensional convolution, attention modules, and pooling operations, and then restores it to the original image size through deconvolution, while combining multi-scale spatial information to generate the final depth information map.

[0097] The innovation of the module is also reflected in the optimization processing after the depth information map is generated. The system calculates the weighted average depth value of the corresponding area in the depth information map through the detected face pixel coordinates, and introduces a noise filtering strategy to improve the accuracy of the depth value calculation, which is finally used as the user's 3D position parameter. When the 3D position parameter is fused with the gaze point data without 3D information, a dynamic weight allocation strategy based on Kalman filtering is adopted, thereby achieving higher real-time and stability in the comprehensive analysis of space and gaze information.

[0098] The module's deep learning model is trained based on the expanded NYU DepthV2 dataset, which adds samples of various complex scenes and dynamic objects, further enhancing the model's generalization and stability. Through the combination of depth information maps and 3D position parameters, as well as improved model design, the module improves the accuracy of human position detection and the comprehensiveness of spatial information, while significantly enhancing the system's ability to adapt to complex scenes and dynamic environments.

[0099] ② Compared with the traditional single-branch eye tracking model, this system adopts an improved multi-branch deep learning architecture, which significantly improves the accuracy and efficiency of feature extraction. Specifically, the model extracts the features of the left eye, right eye and face through a multi-branch structure, and each branch independently designs an optimized convolutional neural network module to further enhance the ability to extract local feature information. The left and right eye branches share the same feature extraction module, which includes multiple 3x3 convolutional layers and improved dilated convolutional layers (with dilation rates of (2,2), (3,3), (4,5), (5,11)), and a dynamic dilation rate adjustment mechanism is added to the dilated convolutional layer to adaptively adjust the dilation rate according to the distribution of input data, so as to more efficiently capture the multi-scale features of the eye area; the facial branch introduces an attention mechanism to dynamically focus on the key feature area based on multiple convolutions and dilated convolutions, effectively integrating the global receptive field and detailed features of facial information.

[0100] Before the feature vectors output by the left and right eye branches and the face branch are converted into 256-dimensional embedding vectors through the fully connected layer, feature normalization and denoising modules are added to remove redundant information and enhance the distinguishability of features. Finally, in the feature concatenation module, these embedding vectors are combined into a vector containing 576-dimensional features and optimized through an adaptive feature weight allocation mechanism to improve the effectiveness of feature fusion and the accuracy of gaze point prediction. Through multi-branch feature extraction, dynamic adjustment and optimized fusion, this system can more efficiently capture local and global contextual information in eye movement data, while showing higher accuracy and robustness under complex lighting and angle conditions.

[0101] Combined with the expanded MPIIGaze dataset for training, the multi-branch structure further improves the generalization and stability of the model by adding sample data under non-ideal conditions. Compared with the ordinary multi-branch model, this improved architecture not only significantly improves the accuracy of gaze point prediction, but also enhances the adaptability of the model in practical applications, especially in dynamic scenes.

[0102] ③Compared with the traditional identity information management method, this system introduces an improved identity information management module based on deep learning. Through the optimized FaceNet model, it realizes high-precision identity recognition and dynamic management of users, ensuring that interactions in a multi-user environment do not conflict. The system functions cover identifying user identities, maintaining user information databases, dynamically adding new users, deleting expired users, and merging similar user data. The core FaceNet model adds an adaptive threshold adjustment mechanism when calculating the Euclidean distance between the input facial image and the user feature vector in the database. It dynamically adjusts the similarity judgment criteria according to the current user data distribution, thereby further improving the accuracy and robustness of recognition. When the similarity exceeds the threshold, the system determines it as the same user to avoid repeated creation of user information.

[0103] The improved FaceNet model adopts a hierarchical design. On the basis of gradually extracting local and global facial features through traditional multiple convolutional layers (such as CONV1, CONV2, CONV3, CONV4), a multi-scale feature fusion module and attention mechanism are introduced to dynamically enhance the expression ability of key feature areas. Subsequently, the features are further integrated through the fully connected layers (FC1, FC2), and the feature vectors are standardized through optimized L2 regularization, and the feature vectors are mapped to the unit hypersphere, thereby significantly improving the consistency of feature expression and the accuracy of similarity calculation. In addition, a compressed storage mechanism is introduced when storing feature vectors in the database to reduce storage occupancy while maintaining efficient retrieval capabilities.

[0104] The module also optimizes the user status management process for multi-user interaction scenarios. When a user's gaze is away from the operating display for more than 5 seconds or the user actively exits the system, the system will automatically release the current interaction permission through an improved permission release strategy, providing the next user with an opportunity to operate; at the same time, by introducing a timeout user cleanup algorithm, the user data that has not been active for a long time is regularly deleted, further reducing the usage of storage resources.

[0105] Combining the improved FaceNet model with a dynamic database maintenance strategy, the system significantly improves the accuracy of identity recognition and interaction management efficiency in multi-user environments, and demonstrates higher robustness and stability in complex scenarios. Compared with traditional models, the system not only achieves more accurate user recognition in the in-vehicle display interaction system, but also effectively optimizes resource utilization and user experience in dynamic environments, providing a solid guarantee for the reliability of multi-user management.

[0106] Obviously, those skilled in the art can make various modifications and changes to the embodiments of the present application without departing from the spirit and scope of the present application. In this way, if these modifications and changes are within the scope of the claims of the present application and their equivalents, the present application is also intended to cover these modifications and changes. The word "comprising" does not exclude the presence of other elements or steps not listed in the claims. The simple fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used to profit. Any figure mark in the claims should not be considered to limit the scope.

Claims

1. A human-computer interaction method based on eye tracking, characterized in that: include: S1, obtaining the original image and preprocessing the original image; S2, generating a depth information image of the preprocessed original image, and performing facial feature detection and interception on the preprocessed original image to obtain a facial feature image; S3, performing eye feature detection and interception on the facial feature image to obtain a left eye feature image and a right eye feature image, respectively, and obtaining user feature information according to the facial feature image; S4, using a deep learning model to process the facial feature image, the left eye feature image, and the right eye feature image, and calculating the gaze point in combination with the depth information image; S5. Output the user characteristics and gaze information in combination with the gaze point and the user characteristic information, and execute a corresponding program in response to the interactive action.

2. The human-computer interaction method based on eye tracking according to claim 1, characterized in that: The S4 specifically includes: S401, calculating facial posture feature parameters of the facial feature image through a deep learning model; S402, further extracting feature parameters of the left eye feature image and the right eye feature image through a deep learning model; S403, combining the facial feature image and the depth information image to obtain facial depth information corresponding to the facial feature image; S404: Combine the facial feature posture, the feature parameters of the further extracted left eye feature image and right eye feature image, and the facial depth information to calculate the gaze point.

3. The human-computer interaction method based on eye tracking according to claim 1, characterized in that: The method of obtaining user feature information based on the facial feature image specifically includes: extracting a feature vector of the facial feature image using the FaceNet model, and determining whether there is user information matching the extracted feature vector in a preset user information database, and if so, outputting a user ID number; if not, updating the user information database and outputting a newly created user ID number.

4. The human-computer interaction method based on eye tracking according to claim 2, characterized in that: The further extracting the feature parameters of the left eye feature image and the right eye feature image by the deep learning model specifically includes: Performing a first convolution operation on the eye image to obtain first convolution eye image data; Performing maximum pooling on the first convolution eye image data to obtain pooled eye image data; Performing a second convolution operation on the pooled eye image data to obtain second convolution eye image data; Perform a dilation convolution operation on the second convolution eye image data to obtain feature parameters of the eye image.

5. The human-computer interaction method based on eye tracking according to claim 2, characterized in that: The method of combining facial feature posture, feature parameters of the further extracted left eye feature image and right eye feature image, and facial depth information to calculate the gaze point specifically includes: splicing the left eye feature image, the right eye feature image, and the facial feature image to obtain the gaze point without 3D information; obtaining 3D position parameters from the facial depth information, and combining them with the gaze point without 3D information to obtain the actual gaze point.

6. The human-computer interaction method based on eye tracking according to claim 1, characterized in that: The method of generating a depth information image of the preprocessed original image specifically includes: the preprocessed original image is respectively subjected to 5 layers of convolutional feature extraction layers to extract 5 layers of convolutional features, and the extracted 5 layers of convolutional features are input into the C3D model, and the data output by the C3D model are respectively subjected to 3 layers of deconvolution layers and then superimposed to obtain a depth information image.

7. The human-computer interaction method based on eye tracking according to claim 1, characterized in that: The executing a corresponding program in response to the interactive action comprises: Gaze selection: determine whether the gaze point has been in a certain area for more than a preset time. If so, simulate a touch event. Scrolling and sliding control: determine whether the gaze point is in the edge area for more than a preset time, and if so, trigger the corresponding scrolling / sliding operation; Highlight the gaze content, determine whether the gaze point is in a specific text or image area for more than a preset time, and if so, highlight the area; Look at the drawing and determine whether to enter the drawing application. If so, record the gaze point coordinates in real time and draw the gaze point trajectory.

8. A human-computer interaction system based on eye tracking, used to execute the human-computer interaction method based on eye tracking according to any one of claims 1 to 7, characterized in that: include: User data processing module and user interaction action module; The user data processing module is configured to collect image data, extract user feature information, and calculate user gaze points; The user interaction action module receives the user gaze point and user information through the TCP protocol, and then processes the user gaze data.

9. The human-computer interaction system based on eye tracking according to claim 8, characterized in that: The user data processing module includes: an eye tracking unit, a user identification unit and a human 3D position detection unit; the eye tracking unit is configured to extract gaze point information; the user identification unit is configured to identify the identity of the user and distinguish between the operating user and the module to be operated; the human 3D position detection unit is configured to extract the detected user position and correct the gaze information of eye tracking.

10. A human-computer interaction device based on eye tracking, used to execute the human-computer interaction method based on eye tracking according to any one of claims 1 to 7, characterized in that: include: The first RGB camera, the second RGB camera, the eye tracking data processing host computer, the Android operating system host computer, and the display; The first RGB camera and the second RGB camera are respectively connected to the eye tracking data processing host computer; the eye tracking data processing host computer communicates data with the Android operating system host computer through TCP; and the Android operating system host computer is connected to the display.

Images