Brightness control method and display device based on ai and eye tracking

Abstract

Embodiments of the present application provide a brightness control method and display device based on AI and eye tracking, the method comprising: generating a pupil corneal reflection vector according to an eye image and a face image, in combination with a pupil center and a corneal reflection point; training a gaze recognition model using the image and the vector as training data; inputting a real-time eye image, a real-time face image, and a real-time pupil corneal reflection vector into the trained model to obtain a real-time gaze point; inputting the real-time gaze point into an AI decision model to demarcate a target pixel region that needs to be enhanced in brightness, generate voltage control instructions for each unit of a liquid crystal beam control layer, and control the light-emitting angle of a liquid crystal unit through the voltage instructions to guide light rays to converge on the real-time gaze position. The method can significantly improve the readability and comfort of the key area of the user's visual experience on the premise of reducing the overall power consumption of the display screen.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a brightness control method and display device based on AI and eye tracking. Background Technology

[0002] Currently, display screens have become the core information output carrier for human-computer interaction. Brightness control, as a key aspect of display technology that directly affects user visual experience and device energy consumption, has a significant impact on screen readability, comfort, and overall battery life. Existing display screen brightness control solutions mostly employ global backlight adjustment or local dimming technology. This involves uniformly adjusting the backlight brightness of the entire screen based on the external light intensity detected by an ambient light sensor, or dividing the screen into several fixed blocks and independently adjusting the brightness based on the average grayscale value of the image content within each block. However, conventional brightness control solutions are limited by factors such as the limited number of backlight zones, coarse dimming granularity, and a response mechanism that lags behind user eye movement. Existing solutions generally suffer from problems such as a mismatch between the bright areas and the user's actual gaze point, unnecessary power consumption waste in non-focused areas, and visual persistence and delayed perception during brightness switching in dynamic scenes. It is difficult to achieve precise local brightness enhancement without sacrificing display quality and user subjective clarity.

[0003] Therefore, how to significantly improve the readability and comfort of key areas of the user's visual experience while reducing the overall power consumption of the display screen is an urgent technical problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a brightness control method and display device based on AI and eye tracking, so as to significantly improve the readability and comfort of key areas of the user's visual experience while reducing the overall power consumption of the display screen.

[0005] In a first aspect, this application provides a brightness control method based on AI and eye tracking, comprising: Collect sets of eye images and face images when the user is looking at the display screen; Based on the pupil center and corneal reflection point of the eye image in the eye image set, the pupil-corneal reflection vector between the pupil center and the corneal reflection point is obtained; The eye image set, the face image set, and the pupil-corneal reflection vector corresponding to the eye image in the eye image set are used as training data to train the initial gaze point recognition model, thereby obtaining a trained gaze point recognition model. The real-time eye image, real-time face image, and real-time pupil-corneal reflection vector corresponding to the real-time eye image are input into the gaze point recognition model to obtain the real-time gaze point position when the user gazes at the display screen. The real-time gaze point position is input into the AI ​​decision model. The AI ​​decision module determines the target pixel area on the display screen that needs to be brightened based on the real-time gaze point position, and generates a voltage control signal for each liquid crystal unit in the liquid crystal beam control layer of the display screen. The voltage control signal is used to control each liquid crystal unit to make the emitted light converge toward the real-time gaze point position. By applying the voltage control signal to each liquid crystal cell in the liquid crystal beam control layer within the display screen, the light emission angle of each liquid crystal cell is adjusted to perform pixel-level display brightness adjustment on the display screen.

[0006] Secondly, this application provides a display device based on AI and eye tracking, including a display panel, an LED array, and a liquid crystal beam control layer located between the LED array and the display panel, and also includes an AI decision module connected to the liquid crystal beam control layer and a camera, the AI ​​decision module being used to execute the aforementioned brightness control method based on AI and eye tracking.

[0007] The brightness control method and display device based on AI and eye tracking provided in this application collect a set of eye images and a set of face images when the user gazes at the display screen. Using the pupil-corneal reflection vector between the pupil center and the corneal reflection point in the eye images, and combining the eye images and face images as training data, an initial gaze point recognition model is trained. This allows the trained gaze point recognition model to fully integrate the morphological features of the eyeball, the posture features of the face, and the pupil-corneal reflection vector reflecting the deviation between the optical axis and the visual axis of the eyeball, thereby accurately mapping the user's visual intent to the real-time gaze point position in the two-dimensional coordinate system of the screen. By inputting the real-time gaze point position into the AI ​​decision model, the AI ​​decision model determines the areas requiring enhancement based on this real-time gaze point position. The target pixel area for brightness is determined, and voltage control signals are generated for each liquid crystal cell in the liquid crystal beam control layer. These signals drive the liquid crystal cells to converge the emitted light towards the real-time gaze point, thereby achieving pixel-level local brightness enhancement at the precise location where the user is looking, rather than global brightness adjustment. By applying voltage control signals to each liquid crystal cell in the liquid crystal beam control layer and adjusting the light emission angle of each cell, the display screen can maintain normal brightness in non-gaze areas while increasing brightness only in the target pixel area currently being looked at by the user. This significantly improves the readability and comfort of key areas of the user's visual experience while reducing the overall power consumption of the display screen, achieving deep integration and synergistic optimization of eye tracking and display brightness control. Attached Figure Description

[0008] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0009] Figure 1 A flowchart illustrating a brightness control method based on AI and eye tracking, provided for an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a display device provided in an embodiment of this application.

[0010] The diagram shows: 1. Display panel; 2. LED array; 3. Liquid crystal beam control layer; 4. Camera; The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0011] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0012] Figure 1 This is a flowchart illustrating a brightness control method based on AI and eye tracking, provided as an embodiment of this application. It should be understood that in other embodiments, the order of some steps in the brightness control method based on AI and eye tracking in this embodiment can be shared according to actual needs, or some steps can be omitted or maintained. Figure 1 As shown, the method may include the following steps: S110. Collect a set of eye images and a set of face images when the user is looking at the display screen.

[0013] An eye image set refers to a series of image data containing the user's eye area, captured continuously or intermittently by an image acquisition device while the user is looking at a display screen. These images should clearly show the structural features of the eye, such as the pupil, iris, and cornea.

[0014] A facial image set is a series of images simultaneously collected, containing the complete facial region of the user, used to provide auxiliary information such as facial pose and expression. The acquisition process needs to ensure the quality and validity of the images to meet the requirements of subsequent feature extraction and model training.

[0015] This step can be accomplished using an image acquisition device that is integrated with the display screen or configured independently, such as an infrared camera to reduce ambient light interference. At the same time, it is necessary to consider the frame rate, resolution and other parameter settings to balance the amount of data and processing efficiency.

[0016] S111. Establish a two-dimensional coordinate system on the display screen.

[0017] Establishing a two-dimensional coordinate system on the display screen is to assign a unique positional identifier to each pixel on the screen, so as to accurately describe the position of the user's gaze point. The two-dimensional coordinate system consists of two mutually perpendicular coordinate axes, one of which extends along the horizontal direction of the screen, and the other extends along the vertical direction of the screen.

[0018] The origin of the coordinate system can be chosen based on actual needs; for example, the top-left or bottom-left corner of the screen can be used as the origin. The units of measurement for the coordinate axes typically correspond to the number of pixels on the screen, meaning each tick represents a pixel's distance. For example, if the horizontal pixel count of the display screen is X and the vertical pixel count is Y, then the horizontal coordinate can be set from 0 to X-1, and the vertical coordinate can be set from 0 to Y-1. In this way, each pixel on the screen can be represented by a unique (horizontal coordinate value, vertical coordinate value) coordinate pair.

[0019] S112. Acquire raw eye images and raw face images of the user when the user gazes at multiple different coordinate points in the two-dimensional coordinate system using the infrared camera corresponding to the display screen.

[0020] The infrared camera corresponding to the display screen refers to an infrared imaging device associated with the display screen, used to capture images of the user's eyes and face. Infrared cameras can emit and receive infrared light, thus clearly capturing eye and facial features under different lighting conditions and reducing the impact of ambient visible light on image quality.

[0021] Multiple different coordinate points refer to a series of points with different coordinate values ​​that are pre-set in the two-dimensional coordinate system established in step S111. These points should be distributed as evenly as possible across the entire display screen to cover various areas that the user may be looking at.

[0022] As the user gazes at these different coordinate points in sequence as instructed, the infrared camera simultaneously begins acquiring images, continuously or at regular time intervals, capturing raw images of the user's eyes and face. Raw eye images are unprocessed image data directly from the camera, containing the user's eye area. Raw face images are unprocessed image data directly capturing the user's entire face. During acquisition, it is crucial to ensure stability and image clarity when the user gazes at each coordinate point so that subsequent processing can accurately extract the desired features.

[0023] S113. Perform face region detection operation on the original eye image and the original face image to obtain the face region location.

[0024] Face region detection refers to the process of identifying and locating the face region from raw eye and face images using specific algorithms. Its purpose is to separate the face region from complex image backgrounds, enabling more accurate localization and processing of the eye region in subsequent steps.

[0025] Face detection algorithms can include feature-based algorithms (such as the Haar feature classifier) ​​and deep learning-based algorithms (such as the YOLO series, SSD, etc.). Face detection algorithms analyze pixel features, texture features, and other characteristics of an image to determine the presence of a face and the boundary location of the face region.

[0026] The location of a face region can be represented, for example, by a rectangular bounding box. The parameters of this bounding box include the coordinates of the top-left corner (x-coordinate and y-coordinate), the width, and the height of the rectangle. These four parameters allow for accurate selection of the face region within the original image. For instance, for an image with a resolution of A×B, the detected face region location might be represented as (x1, y1, w1, h1), where (x1, y1) are the coordinates of the top-left corner of the bounding box, w1 is the width of the bounding box, and h1 is the height of the bounding box.

[0027] S114. Perform binocular positioning operation based on the face region location to obtain the binocular region location.

[0028] After obtaining the location of the face region, a binocular positioning operation is performed based on that location. The purpose is to further and accurately determine the area where the eyes are located within the face region.

[0029] Binocular localization can be achieved by utilizing specific features of the eyes, such as the shape of the eyes and the color contrast between the iris and sclera. Specifically, feature point detection can be performed within the face region to identify eye-related features, such as the corners of the eyes and eyelid contours. Then, the approximate range of the binocular region can be determined based on these feature points.

[0030] The positions of the two eye regions can also be represented by rectangular bounding boxes, corresponding to the left and right eye regions respectively. For example, for the face region bounding box (x1, y1, w1, h1), after binocular positioning, we obtain the left eye region bounding box (x2, y2, w2, h2) and the right eye region bounding box (x3, y3, w3, h3), where (x2, y2) and (x3, y3) are the coordinates of the upper left corner of the left and right eye region bounding boxes, respectively, and w2, h2 and w3, h3 are the width and height of the left and right eye region bounding boxes, respectively.

[0031] S115. Based on the location of the two eye regions, crop out eye images containing one or both eyes from the original eye image to obtain an initial set of eye images.

[0032] Based on the binocular region positions obtained in step S114, image regions containing either one or both eyes are cropped from the original eye image. The purpose of cropping is to remove background regions unrelated to eye features from the original eye image, retaining only the effective regions containing either one or both eyes, thereby reducing the amount of data for subsequent processing and improving processing efficiency.

[0033] If the binocular region contains two separate regions, left and right, these two regions can be cropped separately to obtain an eye image containing the left eye and an eye image containing the right eye; alternatively, the binocular regions can be merged into a single region and cropped to obtain an eye image containing both eyes.

[0034] Combining all the cropped images, including those of one or both eyes, forms the initial set of eye images. Each image in the initial set of eye images corresponds to the user's eye state when looking at a specific coordinate point.

[0035] S116. Extract face images from the original face image to obtain an initial set of face images.

[0036] Extracting a face image from the original face image essentially involves separating the face region detected in step S113 from the original image. Similar to the cropping operation in step S115, this step involves cropping the corresponding face region from the original face image based on the bounding box parameters of the face region's location, resulting in an image containing only the face. The extracted face image can retain its original resolution or be resized as needed for subsequent processing.

[0037] By combining all the extracted face images, an initial set of face images is formed. Each image in the initial set of face images corresponds to an image in the initial set of eye images, together recording the facial and eye information of the user when gazing at different coordinate points.

[0038] S117. Perform data preprocessing operations on the initial eye image set and the initial face image set to obtain the eye image set and the face image set, wherein the preprocessing operations include at least one of the following: geometric normalization processing, grayscale normalization processing, and data augmentation processing.

[0039] Data preprocessing involves a series of processing and optimizations of the initial eye image set and the initial face image set to improve the quality and consistency of the image data, making it more suitable for subsequent model training.

[0040] Geometric normalization adjusts the size and proportions of an image to give all images a uniform size. For example, all eye and face images can be resized to a fixed width and height, such as adjusting eye images to 64×64 pixels and face images to 128×128 pixels. This can be achieved, for example, through image scaling algorithms (such as bilinear interpolation).

[0041] Gray-level normalization converts a color image to a grayscale image and standardizes the grayscale values, ensuring the image's grayscale distribution falls within a uniform range. For example, it normalizes grayscale values ​​to the range of 0-255 or 0-1. This helps eliminate the influence of different lighting conditions on the image's grayscale values.

[0042] Data augmentation involves performing a series of transformations on the original image to generate new training samples, thereby increasing the diversity of the training data and improving the model's generalization ability. For example, data augmentation methods can include random flipping (horizontal flipping, vertical flipping), random rotation (rotating the image within a certain angle range), random cropping (randomly cropping a portion of the image), and brightness and contrast adjustment (randomly changing the brightness and contrast of the image).

[0043] After performing at least one of the above preprocessing operations, the initial set of eye images and the initial set of face images are processed into a set of eye images and a set of face images of higher quality and more suitable for model training.

[0044] S120. Based on the pupil center and corneal reflection point of the eye image in the eye image set, obtain the pupil-corneal reflection vector between the pupil center and the corneal reflection point.

[0045] The pupil center refers to the geometric center of the pupil area in an image of the eye; it is an important reference point for reflecting eye movement. The corneal reflective point refers to the bright spot formed on the corneal surface when infrared light shines on it; its position changes as the eye moves.

[0046] The pupil-corneal reflection vector is a vector describing the relative positional relationship between the pupil center and the corneal reflection point. It is obtained by calculating the difference between the coordinates of the pupil center and the coordinates of the corneal reflection point. This vector plays a crucial role in eye-tracking technology, reflecting the movement state of the eyeball and providing key information for subsequently determining the user's fixation point.

[0047] S121. Perform threshold segmentation on the eye images in the eye image set to obtain binarized eye images.

[0048] Thresholding segmentation is a segmentation method based on the grayscale values ​​of image pixels. Its purpose is to separate the region of interest (such as the pupil) in an eye image from the background region.

[0049] For each eye image in the set of eye images, it first needs to be converted to a grayscale image (if the original image is a color image). Then, a suitable grayscale threshold is selected, and pixels with grayscale values ​​greater than the threshold are set to one color (e.g., white, grayscale value 255), while pixels with grayscale values ​​less than or equal to the threshold are set to another color (e.g., black, grayscale value 0), thus obtaining a binarized eye image. Specifically, a fixed threshold can be set manually, or an automatic thresholding method (such as the Otsu algorithm) can be used. This algorithm can automatically calculate the optimal threshold based on the image's grayscale histogram for more accurate segmentation.

[0050] For example, for an eye image with a grayscale distribution between 0 and 255, if the optimal threshold T is calculated using the Otsu algorithm, then after binarization, pixels with a grayscale value greater than T become 255, and the rest become 0. In this way, the pupil area (usually a darker area) will be segmented as black, while other areas (such as the sclera) will be segmented as white.

[0051] S122. Perform an ellipse fitting operation on the binarized eye image to generate a set of pupil center coordinates based on the pupil center coordinates in the obtained binarized eye image.

[0052] In a binarized eye image, the pupil region typically appears as an approximately elliptical black area. Ellipse fitting utilizes the coordinate data of pixels belonging to the pupil region in the binarized image to mathematically fit an elliptical curve that best represents the distribution of these pixels. For example, the least squares method can be used to implement the ellipse fitting algorithm. The least squares method determines the parameters of the ellipse, including the center coordinates, major axis length, minor axis length, and rotation angle, by minimizing the distance error between the fitted ellipse and the actual pixel values.

[0053] The coordinates of the center of the ellipse are considered to be the pupil center coordinates. This ellipse fitting operation is performed on each binarized eye image in the eye image set to obtain the pupil center coordinates for each image. Arranging these coordinates according to the order in which the images were acquired generates the set of pupil center coordinates.

[0054] For example, for a certain binary eye image, the pupil center coordinates are obtained as (cx1, cy1) by ellipse fitting, and (cx2, cy2) for another image, and so on, eventually forming a set of pupil center coordinates [(cx1, cy1), (cx2, cy2), ..., (cxn, cyn)], where n is the number of images in the eye image set.

[0055] S123. Detect corneal reflection bright spots generated by infrared supplementary light in the eye images in the eye image set, and generate a corneal reflection point coordinate set based on the obtained corneal reflection point coordinates in the eye images.

[0056] When an infrared supplemental light is working, it emits infrared light. When this light shines on the surface of the user's cornea, it produces a noticeable reflective bright spot, also known as a corneal reflective bright spot.

[0057] Detecting this corneal reflective spot in each eye image in a dataset involves pinpointing its exact location within the image. Detection methods can be based on the image's grayscale characteristics, as corneal reflective spots are typically among the areas with the highest grayscale values ​​in an image.

[0058] Specifically, by setting a high grayscale threshold, regions in the image with grayscale values ​​higher than this threshold are identified as potential reflective bright spots. These regions are then subjected to morphological processing (such as erosion and dilation) and connected component analysis to remove noise interference, ultimately determining the center coordinates of the corneal reflective bright spots, i.e., the corneal reflective point coordinates. This detection operation is performed on each eye image to obtain the corresponding corneal reflective point coordinates. These coordinates are then combined sequentially to generate a set of corneal reflective point coordinates.

[0059] For example, if the coordinates of the corneal reflection point detected in one eye image are (rx1, ry1) and in another image they are (rx2, ry2), then the set of corneal reflection point coordinates is [(rx1, ry1), (rx2, ry2), ..., (rxn, ryn)].

[0060] S124. Based on the difference in the horizontal and vertical coordinates between the pupil center coordinates and the corneal reflection point coordinates at the same acquisition time in the pupil center coordinate set and the corneal reflection point coordinate set, the pupil-corneal reflection vector corresponding to the eye image is obtained.

[0061] The coordinates in the set of pupil center coordinates and the set of corneal reflection point coordinates are in one-to-one correspondence, and each pair of coordinates comes from an eye image acquired at the same time.

[0062] For the pupil center coordinates (cx, cy) and corneal reflector coordinates (rx, ry) at the same acquisition time, calculate the difference between their horizontal coordinates (dx = cx - rx) and vertical coordinates (dy = cy - ry). These two differences constitute the pupil-corneal reflectance vector (dx, dy) corresponding to that eye image. The direction of the vector points from the corneal reflector to the pupil center, and its magnitude reflects the distance between the two. By performing this calculation on all corresponding time-time coordinate pairs in the sets of pupil center coordinates and corneal reflector coordinates, the pupil-corneal reflectance vector corresponding to each eye image can be obtained. These vectors collectively constitute important feature data for subsequent model training.

[0063] S130. Using the eye image set, the face image set, and the pupil-corneal reflection vector corresponding to the eye image in the eye image set as training data, the initial gaze point recognition model is trained to obtain a trained gaze point recognition model.

[0064] The initial gaze point recognition model is a machine learning model with a specific network structure that has not yet been trained. Its purpose is to establish a mapping relationship between eye features, facial features, and pupil-corneal reflection vectors to the user's gaze point position by learning from the input training data.

[0065] The training data consists of a set of eye images, a set of face images, and corresponding pupil-corneal reflection vectors. This data contains the visual and eye-movement features of the user when gazing at different screen coordinates. The training process involves continuously adjusting the model's parameters using this training data, enabling the model to accurately predict the user's gaze point position based on the input features. When the model's prediction accuracy reaches the preset requirements, training ends, and a trained gaze point recognition model is obtained.

[0066] S131. Obtain the initial gaze point recognition model, which includes an input layer, a feature extractor, a fully connected layer, and an output layer.

[0067] The initial gaze recognition model is a deep learning model built according to a pre-defined network structure. The input layer is the part of the model that receives input data. It is responsible for converting input data such as the set of eye images, the set of face images, and pupil-corneal reflection vectors into tensor forms that the model can process.

[0068] The feature extractor is the core part of the model and typically consists of multiple convolutional and pooling layers. Convolutional layers use different kernels to perform convolution operations on the input image to extract local features such as edges and textures; pooling layers downsample the feature maps extracted by the convolutional layers, reducing the amount of feature data while retaining important feature information.

[0069] The fully connected layer is located after the feature extractor and is used to map the high-dimensional feature vector output by the feature extractor to a low-dimensional space, integrating and non-linearly transforming the features through the connections between multiple neurons.

[0070] The output layer is the last layer of the model. The number of neurons in it is determined by the dimension of the target being predicted. In a gaze point recognition model, the output layer can contain two neurons, which are used to output the predicted values ​​of the horizontal and vertical coordinates of the gaze point, respectively.

[0071] S132. Input the set of eye images into the input layer to obtain eye image data to be processed, and input the set of face images into the input layer to obtain face image data to be processed.

[0072] Each image in the eye image set and the face image set is a matrix of data with a certain size and number of channels. When the eye image set is input into the input layer of the initial gaze point recognition model, the input layer performs format conversion and preprocessing on these image data. For example, it normalizes the pixel values ​​of the images to between 0 and 1 and converts them into tensor dimensions required by the model (such as batch_size×height×width×channels), thus obtaining the eye image data to be processed.

[0073] Similarly, after inputting the set of face images into the input layer, it undergoes similar processing to obtain the face image data to be processed. The eye image data and the face image data to be processed will be used as input to the feature extractor for subsequent feature extraction operations.

[0074] S133. Input the eye image data to be processed and the face image data to be processed into the feature extractor. Extract the eye morphology features and facial pose features through the convolutional layer and pooling layer of the feature extractor. Extract the pupil-corneal reflection vector corresponding to the eye image, the head rotation and translation vector estimated from the facial key points, and the gradient histogram features and scale-invariant feature transform operator features of the image.

[0075] After the eye image data to be processed is input into the feature extractor, the deep features of the eye image are extracted layer by layer through the convolution operation of the convolution layer and the downsampling operation of the pooling layer. These features can reflect the morphological information of the eye, such as the size and shape of the pupil, the state of the eyelid, etc., thus obtaining the eye morphological features.

[0076] Similarly, after the face image data to be processed is processed by the feature extractor, features that can reflect facial posture, such as the pitch angle, yaw angle and roll angle of the head, are extracted, i.e., facial posture features.

[0077] In addition to the features extracted through convolutional and pooling layers, other relevant features also need to be extracted. The pupil-corneal reflection vector is directly obtained from the results of step S124, and each eye image corresponds to a pupil-corneal reflection vector.

[0078] When estimating head rotation and translation vectors from facial landmarks, it is first necessary to detect key feature points in the facial image (such as the corners of the eyes, the tip of the nose, and the corners of the mouth). Then, based on the correspondence between the 3D coordinate information of these key points (obtained through a pre-defined 3D facial model or depth estimation method) and the 2D image coordinates, methods such as perspective transformation are used to estimate the rotation and translation vectors of the head relative to the camera coordinate system. The gradient histogram feature of an image is obtained by calculating the gradient intensity distribution of the image in different directions; it can describe the edge and texture features of the image. Scale-invariant feature transform operator features are local features with scale and rotation invariance, capable of stably extracting feature points and their descriptors of the image at different scales and rotation angles.

[0079] S134. Based on the eye morphology features, facial posture features, pupil-corneal reflection vector, head rotation and translation vector, gradient histogram features, and scale-invariant feature transformation operator features, a spliced ​​feature vector is obtained.

[0080] The extracted features have different dimensions and representations. To fuse these features together as input to the fully connected layer, a concatenation operation is required.

[0081] First, each feature is converted into a one-dimensional vector. For example, if eye shape features and facial pose features are in feature map form, they need to be flattened into one-dimensional vectors; the pupil and corneal reflection vector is itself a two-dimensional vector and can be directly used as part of a one-dimensional vector; head rotation and translation vectors are usually six-dimensional vectors (rotation vector is three-dimensional, translation vector is three-dimensional); gradient histogram features and scale-invariant feature transformation operator features also each have specific dimensions. Then, these one-dimensional vectors are concatenated in a certain order to form a higher-dimensional one-dimensional vector, i.e., feature vector concatenation.

[0082] The dimension of the concatenated feature vector is equal to the sum of the dimensions of all the feature vectors involved in the concatenation. For example, the flattened eye morphology features are 'a'-dimensional, the flattened facial pose features are 'b'-dimensional, the pupil-corneal reflection vector is 2-dimensional, the head rotation and translation vector is 6-dimensional, the gradient histogram features are 'c'-dimensional, and the scale-invariant feature transformation operator features are 'd'-dimensional. Therefore, the dimension of the concatenated feature vector is a + b + 2 + 6 + c + d.

[0083] S135. Input the spliced ​​feature vector into the fully connected layer, and obtain the flattened feature vector through the multi-layer fully connected operation of the fully connected layer.

[0084] After the concatenated feature vector is input into the fully connected layer, each neuron in the fully connected layer is connected to all neurons in the previous layer. Through multiple fully connected operations, the concatenated feature vector undergoes nonlinear transformation and feature integration. Each fully connected layer contains a certain number of neurons, and each neuron performs a weighted summation of the input vector. Nonlinear factors are introduced through activation functions (such as the ReLU function) to enhance the model's expressive power. After processing by multiple fully connected layers, a one-dimensional flattened feature vector is finally obtained. This vector contains deeply integrated feature information and can be used more effectively for subsequent gaze point coordinate prediction.

[0085] S136. Input the flattened feature vector into the output layer, and output normalized horizontal coordinate prediction value and normalized vertical coordinate prediction value through the output layer to obtain the set of predicted gaze point coordinates, wherein the output layer includes a linear layer with two neurons.

[0086] After flattening the feature vector and inputting it into the output layer, the output layer is a linear layer with two neurons, each corresponding to the predicted x and y coordinates of the gaze point. The linear layer performs a linear transformation on the flattened feature vector, resulting in two output values: a normalized predicted x-coordinate and a normalized predicted y-coordinate. Normalization typically maps the predicted coordinates to the range of 0-1 to facilitate comparison with the true coordinates (which are also normalized) and loss calculation. For each image in the eye image set, a normalized predicted gaze point coordinate (px, py) is obtained after model processing. Arranging all these coordinates in order yields the predicted gaze point coordinate set [(px1, py1), (px2, py2), ..., (pxn, pyn)].

[0087] S137. Calculate the mean square error loss value between each predicted gaze point coordinate in the predicted gaze point coordinate set and the corresponding real gaze point coordinate in the pre-calibrated real gaze point coordinate set.

[0088] The pre-calibrated set of true gaze coordinates is the set of coordinate points in the two-dimensional coordinate system where the user actually gazes during the acquisition of training data. These coordinate points are known and accurate. For each predicted coordinate (pxi, pyi) in the predicted gaze coordinate set, a corresponding true coordinate (txi, tyi) can be found in the true gaze coordinate set. They come from the image data acquired at the same time.

[0089] The mean squared error loss is calculated as follows: First, calculate the difference between the predicted coordinates and the true coordinates on the horizontal axis (pxi-txi) and the difference on the vertical axis (pyi-tyi). Then, square these two differences respectively and take the average value to obtain the mean squared error loss of a single sample.

[0090] The mean squared error loss is averaged across all samples in the predicted fixation point coordinate set to obtain the mean squared error loss value for the entire training batch. The calculation formula can be expressed as: Mean squared error loss value = (1 / n) × Σ[(pxi-txi)² + (pyi-tyi)²], where n is the number of samples, and Σ represents the summation over all samples.

[0091] S138. Based on the mean squared error loss value, update the weight parameters of the initial gaze point recognition model using the backpropagation algorithm until the mean squared error loss value meets the preset convergence condition, thereby obtaining the trained gaze point recognition model.

[0092] Backpropagation is a core algorithm in deep learning used to train neural network models. Based on the calculated mean squared error loss, it calculates the partial derivatives (gradients) of the loss function with respect to the weight parameters of each layer, starting from the output layer. Then, it updates the model's weight parameters using optimization algorithms such as gradient descent to reduce the loss value. This process is iterative, with each iteration using a batch of training data to calculate the loss, backpropagate gradients, and update parameters.

[0093] The preset convergence condition can be that the mean squared error loss value is less than a certain preset threshold, or that the loss value no longer decreases significantly within several consecutive iterations. When the convergence condition is met, model training stops, and the resulting model is the trained gaze point recognition model. This model can predict the user's gaze point position relatively accurately based on the input eye image, face image, and pupil-corneal reflection vector.

[0094] S140. Input the real-time eye image, real-time face image, and real-time pupil-corneal reflection vector corresponding to the real-time eye image into the gaze point recognition model to obtain the real-time gaze point position when the user gazes at the display screen.

[0095] In practical applications, it is necessary to acquire real-time images of the user's eyes and face, i.e., real-time eye images and real-time face images. Simultaneously, following steps S121 to S124, the real-time eye images are processed to obtain real-time pupil-corneal reflection vectors. These three real-time acquired feature data are input into a trained gaze point recognition model. The model undergoes internal feature extraction, integration, and prediction processes, outputting normalized real-time gaze point coordinate prediction values. Then, the normalized coordinate prediction values ​​are converted into actual coordinate values ​​in the two-dimensional coordinate system of the display screen through inverse normalization (i.e., multiplied by the number of horizontal and vertical pixels of the display screen, plus the offset of the origin). These coordinate values ​​represent the real-time gaze point position when the user gazes at the display screen.

[0096] S150. The real-time gaze point position is input into the AI ​​decision model. The AI ​​decision model determines the target pixel area on the display screen that needs to be brightened based on the real-time gaze point position, and generates a voltage control signal for each liquid crystal unit in the liquid crystal beam control layer of the display screen. The voltage control signal is used to control each liquid crystal unit to make the emitted light converge towards the real-time gaze point position.

[0097] The AI ​​decision model is specifically designed to determine the brightness adjustment strategy for the display screen based on the real-time gaze point location. After inputting the real-time gaze point location into the AI ​​decision model, the model first analyzes the specific area of ​​that location on the display screen, and then determines the target pixel area that needs to have its brightness enhanced based on preset brightness enhancement rules or decision logic learned through training.

[0098] The target pixel region can be, for example, a region of a certain size centered on the real-time gaze point, the size and shape of which can be adjusted according to actual needs. After determining the target pixel region, the AI ​​decision model needs to generate a corresponding voltage control signal. The liquid crystal beam control layer consists of many independent liquid crystal cells, and the transmittance and emission angle of each liquid crystal cell can be controlled by applying different voltages. Based on the position of each liquid crystal cell within the target pixel region and the real-time gaze point position, the AI ​​decision model calculates the voltage value to be applied to each liquid crystal cell. These voltage values ​​are combined to form the voltage control signal. After this voltage control signal is applied to the liquid crystal cell, it can change the alignment state of the liquid crystal molecules, thereby controlling the emission angle of each liquid crystal cell, causing the light to converge towards the real-time gaze point position, thus enhancing the brightness of that area.

[0099] S160. By applying the voltage control signal to each of the liquid crystal cells in the liquid crystal beam control layer within the display screen, the light emission angle of each of the liquid crystal cells is adjusted to perform pixel-level display brightness adjustment on the display screen.

[0100] The voltage control signals contain specific voltage commands for each liquid crystal cell in the liquid crystal beam control layer. These voltage control signals are applied to the corresponding liquid crystal cells via the display screen's driving circuitry. The liquid crystal molecules inside the cell deflect according to the magnitude and direction of the applied voltage. The deflection angle of the liquid crystal molecules determines the polarization state and emission angle of the light passing through that liquid crystal cell. By precisely controlling the voltage of each liquid crystal cell, its emission angle can be adjusted, allowing more light to escape from the target pixel area and converge towards the user's real-time viewing point, thereby increasing the brightness of that area. Because each liquid crystal cell can be controlled independently, pixel-level brightness adjustment of the display screen can be achieved. Different pixel areas can have different brightness levels to adapt to the user's viewing needs, improving display quality and user experience.

[0101] The method provided in this application collects a set of eye images and a set of face images when a user gazes at a display screen. Based on the pupil-corneal reflection vector between the pupil center and the corneal reflection point in the eye images, and using the eye and face images as training data, an initial gaze point recognition model is trained. This allows the trained gaze point recognition model to fully integrate the morphological features of the eyeball, the posture features of the face, and the pupil-corneal reflection vector reflecting the deviation between the optical axis and the visual axis of the eyeball. This accurately maps the user's visual intent to the real-time gaze point position in the two-dimensional coordinate system of the screen. By inputting the real-time gaze point position into an AI decision model, the AI ​​decision model determines the target pixel area requiring enhanced brightness based on this real-time gaze point position. The system generates voltage control signals for each liquid crystal cell in the liquid crystal beam control layer. These signals drive the liquid crystal cells to converge the emitted light towards the real-time gaze point, thereby achieving pixel-level local brightness enhancement at the precise location of the user's gaze, rather than global brightness adjustment. By applying voltage control signals to each liquid crystal cell in the liquid crystal beam control layer and adjusting the light emission angle of each cell, the display screen can maintain normal brightness in non-gaze areas while increasing brightness only in the target pixel area currently being gazed upon. This significantly improves the readability and comfort of key areas of the user's visual experience while reducing the overall power consumption of the display screen, achieving deep integration and synergistic optimization of eye tracking and display brightness control.

[0102] The training process of the AI ​​decision-making model includes: S210. Obtain a pre-constructed initial AI decision model and a training dataset of the initial AI decision model. The initial AI decision model includes a region mapping encoder and a voltage generation decoder. The training samples in the training dataset include training gaze point positions and corresponding training voltage control signals. The training voltage control signals are used to control each liquid crystal cell in the liquid crystal beam control layer within the display screen to converge the emitted light rays toward the training gaze point position.

[0103] The initial AI decision model is a pre-designed model with a specific network structure based on task requirements, consisting of two main parts: a region mapping encoder and a voltage generation decoder. The region mapping encoder is responsible for mapping the input gaze point position to pixel areas on the display screen, while the voltage generation decoder is responsible for generating corresponding voltage control signals based on the mapped region information.

[0104] The training dataset is a collection of data used to train the initial AI decision-making model. Each training sample consists of a training gaze point position and a corresponding training voltage control signal. The training gaze point position simulates various gaze positions of the user on the display screen, and the corresponding training voltage control signal is a voltage signal that enables the emitted light from the liquid crystal cell to converge towards the training gaze point position, obtained through pre-physical modeling or experimental measurement.

[0105] S220. Input the training gaze point position into the region mapping encoder, determine the training pixel coordinates of the training gaze point position in the pixel coordinate system of the display screen through the coordinate mapping operation of the region mapping encoder, and generate a training region growth seed point centered on the training pixel coordinates based on the training pixel coordinates.

[0106] The region-mapping encoder first receives the training gaze location, which is typically in an abstract coordinate space. A coordinate mapping operation transforms this abstract gaze location into the pixel coordinate system of the display screen, obtaining specific training pixel coordinates (tx, ty), which represent the position of the training gaze point within the screen's pixel grid. Then, seed points for training region growing are generated, centered on these training pixel coordinates. The seed points are the starting points of the region growing algorithm, defining the initial center of the region. For example, if the training pixel coordinates are (tx, ty), the seed points can be directly set to (tx, ty), or multiple points within a small region centered on these coordinates can be used as seed points to improve the robustness of region growing.

[0107] S230. The training region growth seed points are processed by the region growth operation of the region mapping encoder, and the training growth boundary is determined according to the brightness gradient value between adjacent pixels to generate a set of training boundary coordinates of the training target pixel region that needs to be enhanced in brightness.

[0108] Region growing begins by growing seed points from the training region. Following certain rules, pixels with similar characteristics are merged into the seed region, forming a continuous region. In this step, the rule for determining whether pixels belong to the same region is based on the brightness gradient value between adjacent pixels. The brightness gradient value reflects the rate of brightness change between adjacent pixels; the smaller the gradient value, the more similar the brightness of adjacent pixels.

[0109] A gradient threshold is set. When the brightness gradient value between adjacent pixels is less than this threshold, the two pixels are considered to belong to the same region and can be merged. By continuously adding adjacent pixels that meet the condition to the region, the region gradually grows until no more new pixels can be added. The boundary of this region is the training growth boundary. The training target pixel region is the set of pixels enclosed by the growth boundary. The coordinates of all pixels on the growth boundary are extracted to generate the training boundary coordinate set. This set describes the contour of the training target pixel region.

[0110] S240. Input the training boundary coordinate set into the voltage generation decoder, and map the training boundary coordinate set into the liquid crystal unit coordinate system of the liquid crystal beam control layer through the geometric transformation operation of the voltage generation decoder to obtain the training unit index set.

[0111] There may be a certain mapping relationship between the pixel coordinate system of the display screen and the liquid crystal unit coordinate system of the liquid crystal beam control layer. For example, pixels and liquid crystal units may correspond one-to-one, or there may be certain scaling, offset and other transformations.

[0112] The geometric transformation operation of the voltage generation decoder involves transforming the set of training boundary coordinates representing the contour of the training target pixel region from the pixel coordinate system to the liquid crystal cell coordinate system, based on this pre-defined mapping relationship. The transformed boundary coordinates correspond to the positions of the liquid crystal cells. By processing these coordinates, it is determined which liquid crystal cells belong to the training target pixel region. The training cell index set consists of the index numbers of these liquid crystal cells belonging to the training target pixel region, with each index number uniquely identifying a liquid crystal cell.

[0113] S250. The training unit index set is processed by the voltage distribution operation of the voltage generation decoder. The corresponding training voltage amplitude parameters are generated according to the distance parameters between each liquid crystal unit in the training unit index set and the training gaze point position. The training voltage control signal is obtained according to the training voltage amplitude parameters.

[0114] The voltage allocation operation determines the training voltage amplitude parameter for each liquid crystal cell (LCD) based on the distance parameter between the position of each LCD cell in the training cell index set and the training gaze point position. For example, LCD cells closer to the training gaze point need to have more of their emitted light converge towards the gaze point, thus potentially requiring a higher voltage amplitude; while LCD cells farther away can have a relatively lower voltage amplitude, or retain the default value. The distance parameter can be calculated using Euclidean distance, Manhattan distance, etc.

[0115] For example, a function can be set to change the voltage amplitude with distance, such as voltage amplitude = base voltage + (maximum additional voltage - (distance / maximum distance) × maximum additional voltage), where the base voltage is the default operating voltage of the liquid crystal cell, the maximum additional voltage is the maximum voltage value applied to enhance brightness, and the maximum distance is the maximum distance from the boundary of the training target pixel area to the training gaze point position.

[0116] After calculating the training voltage amplitude parameters for each liquid crystal cell using the above function, these parameters are arranged in the index order of the liquid crystal cells to obtain the training voltage control signal.

[0117] S260. Calculate the voltage deviation between the training voltage control signal and the pre-calibrated target voltage control signal, and update the network weight parameters of the initial AI decision model according to the voltage deviation value through the backpropagation algorithm until the voltage deviation value meets the preset training convergence condition, thereby obtaining the trained AI decision model.

[0118] The pre-calibrated target voltage control signal is obtained through precise physical experiments or simulations, and can ideally realize the voltage signal that causes light to converge towards the training gaze point. The voltage deviation between the training voltage control signal and the target voltage control signal is calculated, for example, by using a loss function such as mean square error, or by calculating the average of the sum of the squares of the differences in the voltage amplitude parameters of the corresponding liquid crystal cells in the two signals.

[0119] Then, using the backpropagation algorithm, gradients are backpropagated from the voltage generator decoder based on the voltage deviation value, updating the network weight parameters of each layer in the region mapping encoder and the voltage generator decoder. This process is iterated continuously, using a batch of training samples each time, until the voltage deviation value is less than the preset training convergence threshold, or the deviation value no longer decreases significantly within a certain number of iterations. At this point, the initial AI decision model training is complete, and the trained AI decision model is obtained.

[0120] Before acquiring a set of eye images and a set of face images when the user is looking at the display screen, the method further includes: S310. Multiple calibration markers with known coordinates are displayed sequentially on the display screen.

[0121] Calibration markers are reference points used to calibrate the gaze recognition model. Their coordinates in the two-dimensional coordinate system of the display screen are known and precise. Before image acquisition and model training, these calibration markers need to be displayed on the screen in a specific order. The display order can be random or arranged according to a certain pattern (such as from left to right or from top to bottom). The number of calibration markers should be sufficient to cover different areas of the screen; for example, 9, 16, or 25 markers can be selected to distribute them evenly across the entire screen. Each calibration marker can be displayed with a specific shape (such as a circle or cross) and color to attract the user's attention and ensure that the user can accurately gaze at the point.

[0122] S320. When displaying the calibration marker, collect the calibration eye features when the user gazes at the calibration marker. The calibration eye features include the calibration pupil-corneal reflection vector, the calibration eye image, and the calibration face image.

[0123] When a calibration marker is displayed on the screen, the user needs to gaze at the marker as instructed. During this gazing period, a calibration eye image and a calibration face image are simultaneously acquired via an image acquisition device. At the same time, the calibration eye image is processed using the same method as in steps S121 to S124 to extract the calibration pupil-corneal reflection vector. These calibration pupil-corneal reflection vectors, the calibration eye image, and the calibration face image together constitute the calibration eye features, recording the eye and facial feature information of the user while gazing at the calibration marker at known coordinates.

[0124] S330. Combine the known coordinates of the calibration markers with the corresponding calibration eye features to form an eye movement calibration association sample, thereby obtaining an eye movement calibration association sample set.

[0125] Each calibration marker has its known coordinates. These coordinates are mapped one-to-one with the calibration eye features collected when the user gazes at the marker, forming an eye-tracking calibration associated sample. For example, if the known coordinates of a calibration marker are (cx, cy), and the corresponding calibration eye features include the calibration pupil-corneal reflection vector (dvx, dvy), the calibration eye image Ie, and the calibration face image If, ​​then the eye-tracking calibration associated sample would be ((cx, cy), (dvx, dvy), Ie, If).

[0126] The eye-tracking calibration associated samples corresponding to all calibration markers are combined together to obtain the eye-tracking calibration associated sample set.

[0127] S340. Input the calibrated eye features in the eye-tracking calibration associated sample set into the gaze point recognition model to obtain the model prediction coordinates corresponding to the eye-tracking calibration associated samples.

[0128] Each calibrated eye feature (including the calibrated pupil-corneal reflection vector, calibrated eye image, and calibrated face image) from the eye-tracking calibration associated sample set is input into the current gaze point recognition model. The model processes and predicts these input features, outputting the model-predicted coordinates corresponding to each calibrated eye feature. The model-predicted coordinates represent the gaze point position that the model believes the user is gazing at that calibration marker.

[0129] S350. Based on the coordinate deviation between the known coordinates and the corresponding model predicted coordinates in the eye-tracking calibration associated samples, the total error value of the eye-tracking calibration associated sample set is obtained.

[0130] For each eye-tracking calibration associated sample, calculate the coordinate deviation between its known coordinates (true coordinates) and the model-predicted coordinates. The coordinate deviation can be obtained by calculating a combination of the differences between the two on the x and y axes, such as Euclidean distance deviation: deviation = sqrt((cx-px)²+(cy-py)²), where (cx, cy) are the known coordinates, (px, py) are the model-predicted coordinates, and sqrt represents the square root.

[0131] Then, the coordinate deviation values ​​of all samples in the eye-tracking calibration associated sample set are summarized to obtain the total error value. The total error value can be the average, sum, or root mean square of all sample deviation values, and is used to measure the current prediction accuracy of the model.

[0132] S360. Repeatedly adjust the model parameter set of the gaze point recognition model, and recalculate the total error value of the eye movement calibration associated sample set after each adjustment, until the total error value is less than or equal to the preset total error threshold, or the repeated adjustment operation reaches the preset number of repetitions, to obtain the target model parameter set.

[0133] The model parameter set includes various weight parameters and bias parameters in the gaze point recognition model. The model parameter set is adjusted using an optimization algorithm such as gradient descent or genetic algorithm.

[0134] After each adjustment, the total error value is recalculated using the eye-tracking calibration associated sample set. If the total error value is less than or equal to the preset total error threshold, the model's prediction accuracy meets the requirements; if the total error value is still greater than the threshold, the model parameter set is adjusted again. This process is repeated until the total error value meets the condition or the preset maximum number of repetitions is reached. The model parameter set obtained at this point is the target model parameter set.

[0135] S370. Update the gaze point recognition model using the target model parameter group.

[0136] The obtained target model parameter set replaces the original parameter set in the gaze point recognition model, completing the model update. The updated gaze point recognition model will improve the accuracy of predicting the user's gaze point position and can more accurately reflect the user's actual gaze.

[0137] The method provided in this application involves sequentially displaying multiple calibration markers with known coordinates on a screen before acquiring a set of eye images and a set of face images. When the user gazes at each calibration marker, calibration eye features, including the calibration pupil-corneal reflection vector, the calibration eye image, and the calibration face image, are acquired. These calibration eye features are then combined with the known coordinates of the calibration markers to form an eye-tracking calibration association sample set, thus establishing a calibration dataset containing a precise mapping relationship between the user's individualized eye-tracking features and the actual coordinates on the screen. The calibration eye features from the eye-tracking calibration association sample set are input into a gaze point recognition model to obtain the model's predicted coordinates. The coordinate deviation between the known coordinates and the model's predicted coordinates in each sample is then calculated for the entire sample set. The total error value quantifies the prediction bias of the current model for individual users. By repeatedly adjusting the model parameter set of the gaze point recognition model and recalculating the total error value after each adjustment until the total error value is less than or equal to the preset total error threshold or the adjustment operation reaches the preset number of repetitions, the target model parameter set is obtained. The gaze point recognition model is then updated using this target model parameter set, thereby achieving personalized model calibration for different users. This effectively eliminates the system prediction bias caused by differences in users' eye physiological structure, facial features, and gaze habits. The updated gaze point recognition model can predict the real-time gaze point position with higher accuracy in actual use, providing an accurate spatial positioning basis for subsequent pixel-level brightness adjustment based on the real-time gaze point position.

[0138] After inputting the real-time acquired eye images, real-time face images, and the real-time pupil-corneal reflection vectors corresponding to the real-time eye images into the gaze point recognition model, the method further includes: S410. Obtain the real-time gaze point position output by the gaze point recognition model and the acquisition timestamp corresponding to the real-time gaze point position.

[0139] The gaze point recognition model outputs the real-time gaze point location and simultaneously records the corresponding acquisition timestamp. The acquisition timestamp refers to the specific time when the real-time eye image and real-time face image were acquired, for example, in seconds or milliseconds. Obtaining the real-time gaze point location and its corresponding acquisition timestamp is for subsequent analysis and prediction of the gaze point's motion trajectory.

[0140] S420. Obtain a real-time gaze point position sequence according to the real-time gaze point positions and the acquisition timestamps.

[0141] The acquired real-time gaze point positions are arranged in the order of their corresponding acquisition timestamps to form the real-time gaze point position sequence. This real-time gaze point position sequence is an ordered list, where each element contains a real-time gaze point position and a corresponding timestamp.

[0142] For example, the real-time gaze point position sequence may be expressed as [(p1, t1), (p2, t2), ..., (pn, tn)], where pi is the i-th real-time gaze point position, ti is its corresponding acquisition timestamp, and t1<t2<...<tn. The real-time gaze point position sequence reflects the change of the user's gaze point over time.

[0143] S430. Input the real-time gaze point position sequence into a preset trajectory prediction filter, and process the real-time gaze point position sequence through a state transition equation of the trajectory prediction filter to obtain a filtered gaze point position sequence.

[0144] The trajectory prediction filter is an algorithm for estimating and predicting the state of a dynamic system, and common examples include a Kalman filter, a particle filter and the like. The preset trajectory prediction filter has a preset state transition equation, which describes the evolution rule of the system state (here, referring to the gaze point position, movement speed and the like) over time.

[0145] After the real-time gaze point position sequence is input into the trajectory prediction filter, the filter processes each real-time gaze point position in the sequence by using the state transition equation, and removes noise interference through prediction and update steps, so as to obtain a smoother and more accurate filtered gaze point position.

[0146] S431. Acquire an initial state vector of the trajectory prediction filter, where the initial state vector includes an initial gaze point position and an initial gaze point movement speed.

[0147] The initial state vector is the initial state estimation when the trajectory prediction filter starts to work, and includes an assumption for the initial state of the system. For gaze point trajectory prediction, the initial state vector may include, for example, two parts: the initial gaze point position and the initial gaze point movement speed. The initial gaze point position may be set as the first position in the real-time gaze point position sequence, or set in other manners (for example, the user initially gazes at the center of the screen). The initial gaze point movement speed may be set to a small initial value (such as a zero vector) based on experience, or obtained by estimating the change of the first several real-time gaze point positions.

[0148] S432. Assign the first real-time gaze point position in the real-time gaze point position sequence to the initial gaze point position in the initial state vector, and assign the preset initial velocity value to the initial gaze point movement velocity in the initial state vector to obtain the current state vector.

[0149] To initialize the trajectory prediction filter, the first real-time gaze position (p1) in the real-time gaze position sequence is assigned to the initial gaze position component of the initial state vector. Simultaneously, a preset initial velocity value (e.g., (0, 0), indicating that the gaze movement velocity is zero at the initial moment) is assigned to the initial gaze movement velocity component of the initial state vector. After this assignment, the initial state vector is updated to the current state vector, serving as the starting state for subsequent filter processing.

[0150] S433. Each real-time gaze point position in the real-time gaze point position sequence is sequentially input into the trajectory prediction filter. For the currently input real-time gaze point position, the predicted state vector at the current time is calculated based on the state vector at the previous time step using the state transition equation. The predicted state vector includes the predicted gaze point position and the predicted gaze point movement speed.

[0151] The trajectory prediction filter processes each real-time gaze position in the real-time gaze position sequence sequentially over time. For the current input real-time gaze position (let's say it's the k-th position pk), the filter first uses the state transition equation and the state vector from the previous time step (k-1 time step) (which includes the filtered gaze position and movement velocity from k-1 time step) to predict the state vector at the current time step (k time step), i.e., the predicted state vector.

[0152] State transition equations are typically constructed based on physical motion models (such as uniform motion models, uniformly accelerated motion models, etc.). For example, for a uniform motion model, the state transition equation can be expressed as: Predicted gaze position = Filtered gaze position at the previous moment + Filtered gaze movement velocity at the previous moment × Time interval; Predicted gaze movement velocity = Filtered gaze movement velocity at the previous moment. Here, the time interval is the difference between the current and previous acquisition timestamps.

[0153] S434. Calculate the Kalman gain based on the positional deviation between the currently input real-time gaze point position and the predicted gaze point position in the predicted state vector.

[0154] Kalman gain is an important parameter in the Kalman filter, used to balance the confidence of the predicted state vector and the current observation (i.e., the real-time gaze position of the current input).

[0155] Position bias is the difference between the current real-time gaze position and the predicted gaze position in the predicted state vector. The calculation of Kalman gain involves parameters such as the prediction error covariance matrix and the observation noise covariance matrix, which can be preset or estimated online based on the system's noise characteristics. The Kalman gain calculation formula results in a larger Kalman gain when the prediction error is large, as the filter relies more on the current observation to correct the prediction; conversely, a smaller Kalman gain when the observation noise is large, as the filter relies more on the predicted state.

[0156] S435. The predicted state vector is weighted and corrected according to the Kalman gain to obtain the filtered state vector at the current time. The filtered state vector includes the filtered gaze point position and the filtered gaze point movement speed.

[0157] The calculated Kalman gain is used to correct the predicted state vector. Specifically, the filtered state vector = predicted state vector + Kalman gain × position deviation. Through this weighted correction, the predicted and observed information are fused to obtain a more accurate system state estimate at the current moment, namely the filtered gaze point position and the filtered gaze point movement velocity.

[0158] S436. Arrange the filtered gaze point positions obtained at each time point in chronological order to obtain the filtered gaze point position sequence.

[0159] Steps S433 to S435 are performed on each real-time gaze point position in the real-time gaze point position sequence to obtain the filtered gaze point position corresponding to each time step. These filtered gaze point positions are arranged in chronological order to form a filtered gaze point position sequence. Compared with the original real-time gaze point position sequence, this filtered gaze point position sequence has less noise and a smoother trajectory.

[0160] S440. Based on the position difference and time difference between two adjacent filtered gaze point positions in the filtered gaze point position sequence, a gaze point movement speed sequence is obtained.

[0161] For two adjacent positions (pk, tk) and (pk+1, tk+1) in the filtered fixation position sequence, calculate the position difference (pk+1-pk) and the time difference (tk+1-tk). Then, divide the position difference by the time difference to obtain the average movement velocity between these two moments, i.e., the fixation velocity. Arrange the fixation velocities of all adjacent moments in chronological order to obtain the fixation velocity sequence. This fixation velocity sequence reflects the speed and direction of the user's fixation point movement.

[0162] S450. Input the filtered gaze point position sequence and the gaze point movement speed sequence into the observation update equation of the trajectory prediction filter to obtain the predicted gaze point position at future time.

[0163] In addition to the state transition equation, the trajectory prediction filter also has an observation update equation, which is used to predict the future state by combining the current state estimate and observation information. By inputting the filtered gaze position sequence and gaze velocity sequence into the observation update equation, the filter can extrapolate the predicted gaze position at one or more future moments based on historical position and velocity information. For example, it can predict the gaze position at moments 0.1 seconds, 0.2 seconds, etc.

[0164] S460. Input the predicted gaze point position as the updated real-time gaze point position into the AI ​​decision model.

[0165] The predicted gaze point position at future moments is used as the updated real-time gaze point position and input into the AI ​​decision-making model. This allows the AI ​​model to make brightness adjustment decisions in advance based on the predicted gaze point position, thereby reducing delays caused by data processing and control signal transmission, making brightness adjustments more timely and accurate, and improving the user experience.

[0166] The method provided in this application constructs a time-series sequence by acquiring the real-time gaze point position and acquisition timestamp. This sequence is then processed using the state transition equation of a trajectory prediction filter. The process sequentially completes initial state vector assignment, predicted state vector calculation, Kalman gain calculation, and weighted correction to obtain the filtered state vector. This filtered gaze point position sequence is then arranged in time sequence to filter out output noise. A gaze point movement speed sequence is calculated based on the filtered gaze point position sequence, and both sequences are input into the observation update equation to obtain the predicted gaze point position. This predicted position is then input into an AI decision model, enabling the AI ​​decision model to generate a voltage control signal in advance to compensate for system delays and ensure that the brightness enhancement area is aligned with the user's true gaze point position.

[0167] The method further includes, prior to applying the voltage control signal to each of the liquid crystal cells in the liquid crystal beam control layer within the display screen: S510. Obtain the real-time gaze point position sequence corresponding to the real-time gaze point position, and obtain the future gaze point prediction position sequence through trajectory prediction processing based on the real-time gaze point position sequence. The future gaze point prediction position sequence includes the predicted gaze point positions for multiple consecutive times after the current time.

[0168] The real-time gaze position sequence is a record of the user's gaze position changing over time, both currently and over a period of time. Trajectory prediction processing can employ trajectory prediction filters as described in steps S430 to S450, or other time series prediction algorithms (such as recurrent neural network models). By analyzing and learning the real-time gaze position sequence, the gaze position for multiple consecutive moments after the current moment (e.g., 5 future moments, each spaced 0.02 seconds apart) is predicted. These predicted positions, arranged chronologically, constitute the future gaze prediction position sequence.

[0169] S520. Based on the predicted gaze point positions in the predicted gaze point position sequence, generate future voltage control signals corresponding to the predicted gaze point positions to obtain a future voltage control signal sequence. Each future voltage control signal in the future voltage control signal sequence corresponds one-to-one with each predicted gaze point position in the predicted gaze point position sequence.

[0170] For each predicted gaze point position in the future gaze point prediction position sequence, a corresponding future voltage control signal is generated according to the method for generating voltage control signals by the AI ​​decision model in step S150. Each predicted gaze point position corresponds to one future voltage control signal. Arranging these future voltage control signals in chronological order yields the future voltage control signal sequence.

[0171] S530. Store the future voltage control signal sequence into the prefetch buffer in chronological order.

[0172] A prefetch buffer is a data storage area used to temporarily store future voltage control signal sequences. The future voltage control signal sequence is written to the prefetch buffer in chronological order, allowing for rapid retrieval when needed. The size of the prefetch buffer should be large enough to hold all signals in the future voltage control signal sequence to ensure that no data loss or delayed retrieval occurs during subsequent control processes.

[0173] S540. When the voltage control signal is applied to each liquid crystal cell of the liquid crystal beam control layer in the display screen, the future voltage control signal corresponding to the current time is read from the prefetch buffer as the voltage control signal at the current time, and the voltage control signal at the current time is applied to each liquid crystal cell of the liquid crystal beam control layer in the display screen.

[0174] When a voltage control signal is actually applied to the liquid crystal cell, the voltage control signal is no longer generated and applied in real time, but the future voltage control signal corresponding to the current moment is read from the prefetch buffer in chronological order. For example, if the current moment is t, future voltage control signals at moments t1, t2, t3... (t1<t2<t3...) are stored in the prefetch buffer. When the system time reaches t1, the voltage control signal at moment t1 is read and applied to the liquid crystal cell; when the system time reaches t2, the signal at moment t2 is read, and so on. This prefetching and buffering manner can effectively reduce the time delay between the generation and the application of the voltage control signal, and ensure that brightness adjustment can respond to changes in the user's gaze point in real time.

[0175] In the method provided by the embodiments of the present application, a real-time gaze point position sequence corresponding to real-time gaze point positions is acquired and subjected to trajectory prediction processing to obtain a future predicted gaze point position sequence that includes predicted gaze point positions at a plurality of consecutive moments after the current moment, which provides position predicted values at multiple time points for subsequent prefetch operations; corresponding future voltage control signals are respectively generated according to each predicted gaze point position in the future predicted gaze point position sequence, so as to obtain a future voltage control signal sequence in one-to-one correspondence with the future predicted gaze point position sequence, realizing batch pre-generation from predicted positions to control instructions; the future voltage control signal sequence is stored in the prefetch buffer in chronological order, and when the voltage control signal is actually applied, the future voltage control signal corresponding to the current moment is read from the prefetch buffer as the voltage control signal at the current moment and applied to the liquid crystal cell, so that the generation and application of the voltage control signal are decoupled in time, and the system can calculate control signals for a plurality of subsequent moments in advance during idle periods without performing real-time calculation at each moment, thereby effectively reducing the computational load and response delay of real-time processing, and ensuring smooth and timely brightness following adjustment for changes in the gaze point during rapid eye movement of the user.

[0176] S610. Inputting the real-time gaze point position into the AI decision model to obtain an initial target pixel area centered on the real-time gaze point position.

[0177] After receiving the real-time gaze point position, the AI decision model first determines an initial target pixel area centered on the position. The size and shape of the initial target pixel area may be determined according to a preset rule, for example, the initial target pixel area may be a square area with a side length of a fixed number of pixels, or a circular area with a radius of a fixed number of pixels. The setting of the initial target pixel area aims to preliminarily determine the approximate range where brightness needs to be enhanced.

[0178] S620. Expanding outward from the boundary of the initial target pixel area by a preset expansion step size to obtain an expanded target pixel area.

[0179] To avoid insufficient brightness at the edges of the user's gaze range due to an excessively small target pixel area, the initial target pixel area needs to be expanded. The expansion step size refers to the number of pixels to be expanded outwards, such as 5 or 10 pixels. Expanding the boundaries of the initial target pixel area outwards in both the horizontal and vertical directions by the preset expansion step size results in a larger expanded target pixel area. The expanded target pixel area can better cover the user's possible gaze range.

[0180] S630. Divide the expanded target pixel region into a central region and a surrounding region, wherein the central region contains the pixel unit where the real-time gaze point is located, and the surrounding region is the region in the expanded target pixel region other than the central region.

[0181] To achieve finer brightness control, the expanded target pixel region can be further divided into a central region and a surrounding region. The central region is a sub-region containing the pixel unit where the real-time gaze point is located, and its size is typically smaller than the expanded target pixel region. The surrounding region is the portion of the expanded target pixel region excluding the central region. For example, if the expanded target pixel region is a square with sides of 21 pixels, the central region could be a square with sides of 11 pixels, the center of which coincides with the real-time gaze point, and the surrounding region is the annular area outside the central region.

[0182] S640. Generate a first voltage amplitude parameter set for the liquid crystal cells in the central region, and generate a second voltage amplitude parameter set for the liquid crystal cells in the surrounding region, wherein the voltage amplitude parameter in the first voltage amplitude parameter set is greater than the voltage amplitude parameter in the second voltage amplitude parameter set.

[0183] The central region, being the core area of ​​the user's gaze, requires higher brightness. Therefore, a first set of voltage amplitude parameters is generated for each liquid crystal cell within the central region, with relatively high voltage amplitude parameters. The surrounding region, being the periphery of the core area of ​​gaze, has relatively lower brightness requirements. Therefore, a second set of voltage amplitude parameters is generated for the liquid crystal cells in the surrounding region, with voltage amplitude parameters lower than those in the first set. This setup achieves a gradient change in brightness, with the central region being the brightest and gradually darkening outwards, conforming to the visual characteristics of the human eye and improving visual comfort.

[0184] S650. Generate the voltage control signal based on the first voltage amplitude parameter set and the second voltage amplitude parameter set.

[0185] The first and second voltage amplitude parameter sets are combined according to the index order of the liquid crystal cells. That is, the liquid crystal cells belonging to the central region use the first voltage amplitude parameters, and the liquid crystal cells belonging to the surrounding regions use the second voltage amplitude parameters, thereby generating a complete voltage control signal. This voltage control signal enables differentiated brightness control of different sub-regions within the target pixel area.

[0186] The method provided in this application determines the basic range of brightness enhancement by inputting the real-time gaze point position into an AI decision model to obtain an initial target pixel region centered on it. It then expands the target pixel region outwards by a preset expansion step size based on the boundary of the initial target pixel region, avoiding insufficient brightness at the edge of the brightness enhancement range due to gaze point prediction errors or minor eye movements. By dividing the expanded target pixel region into a central region containing the pixel unit where the real-time gaze point is located and a surrounding region excluding the central region, it achieves separate control of the core and peripheral regions of the gaze point. Furthermore, by generating a first voltage amplitude parameter set with a larger amplitude for the liquid crystal units in the central region and a second voltage amplitude parameter set with a smaller amplitude for the liquid crystal units in the surrounding region, the light emission angle convergence of the central region is higher than that of the surrounding region. This results in a brightness peak at the real-time gaze point position that smoothly decays towards the surrounding area. This brightness distribution conforms to the visual physiological characteristic that the human eye's fovea is more sensitive to the central field of view than the peripheral field of view, effectively highlighting the user's current focus while reducing power consumption and visual interference in the peripheral region.

[0187] The eye-tracking method described in this invention, such as Figure 2 As shown, it can be implemented based on a display device containing a camera 4, including a display panel 1, an LED array 2, and a liquid crystal beam control layer 3 located between the LED array and the display panel, and also includes an AI decision module connected to the liquid crystal beam control layer and the camera, the AI ​​decision module being used to execute the eye-tracking method of the present invention.

[0188] Specifically, the display device in this invention can be implemented using LEDs, specifically by introducing a liquid crystal beam control layer 3 between the LED array 2 and the display panel 1. This liquid crystal beam control layer 3 can be an independent liquid crystal cell, comprising multiple liquid crystal units arranged in an array. Each liquid crystal unit, acting as a pixel, corresponds one-to-one with the pixels of the LEDs below, or they are grouped into smaller areas. Similar to a liquid crystal display, the voltage of each liquid crystal unit in the liquid crystal beam control layer can be independently controlled.

[0189] The LEDs in LED array 2 can adopt a MICROLED structure. A MICROLED is a PN junction diode made of direct bandgap semiconductor material. When the LED is in a forward bias state, electrons from the semiconductor conduction band and holes from the valence band recombine to emit monochromatic light photons.

[0190] By adjusting the voltage of each liquid crystal cell on the liquid crystal beam control layer, the light can be aligned with the coordinates of the real-time gaze point.

[0191] In the initial state, without applying voltage, the liquid crystal molecules in the liquid crystal cells of the liquid crystal beam control layer are arranged in a specific way, similar to a miniature waveplate or prism, causing the light passing through it to be deflected, forming a wider viewing angle.

[0192] When a voltage is applied, the electric field alters the alignment of the liquid crystal molecules, thereby changing their optical properties. Under a specific voltage, the light deflection angle is adjusted, and the beam is focused onto the coordinates of the real-time gaze point. By applying different voltages to each liquid crystal cell, independent and continuous control of the light emission angle of each pixel can be achieved. AI commands are ultimately translated into voltage signals applied to each liquid crystal cell.

[0193] The display device can also be implemented using a liquid crystal optical phase array (LCOPA). Based on the principle of light diffraction, LCOPA, by applying a precise voltage gradient to the liquid crystal pixels, can achieve continuous and precise control of the beam deflection angle and focusing range, similar to a phased array in radar. It can provide a high-quality initial light field for the upper-level dynamic beam deflector through chip-level beam collimation, reducing the difficulty of its control.

[0194] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0195] Finally, it should be noted that the above-disclosed embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A brightness control method based on AI and eye tracking, characterized in that, include: Collect sets of eye images and face images when the user is looking at the display screen; Based on the pupil center and corneal reflection point of the eye image in the eye image set, the pupil-corneal reflection vector between the pupil center and the corneal reflection point is obtained; The eye image set, the face image set, and the pupil-corneal reflection vector corresponding to the eye image in the eye image set are used as training data to train the initial gaze point recognition model, thereby obtaining a trained gaze point recognition model. The real-time eye image, real-time face image, and real-time pupil-corneal reflection vector corresponding to the real-time eye image are input into the gaze point recognition model to obtain the real-time gaze point position when the user gazes at the display screen. The real-time gaze point position is input into the AI ​​decision model, which determines the target pixel area on the display screen that needs to be brightened based on the real-time gaze point position, and generates a voltage control signal for each liquid crystal unit in the liquid crystal beam control layer of the display screen. The voltage control signal is used to control each liquid crystal unit to make the emitted light converge towards the real-time gaze point position. The real-time gaze point position sequence corresponding to the real-time gaze point position is obtained, and the future gaze point prediction position sequence is obtained by trajectory prediction processing based on the real-time gaze point position sequence. The future gaze point prediction position sequence includes the predicted gaze point positions for multiple consecutive times after the current time. Based on the predicted gaze point positions in the predicted gaze point position sequence, future voltage control signals corresponding to the predicted gaze point positions are generated respectively, resulting in a future voltage control signal sequence. Each future voltage control signal in the future voltage control signal sequence corresponds one-to-one with each predicted gaze point position in the predicted gaze point position sequence. The future voltage control signal sequence is stored in a prefetch buffer in chronological order. By applying the voltage control signal to each liquid crystal cell in the liquid crystal beam control layer within the display screen, adjusting the light emission angle of each liquid crystal cell to perform pixel-level display brightness adjustment of the display screen, the method includes: reading the future voltage control signal corresponding to the current moment from the prefetch buffer as the voltage control signal at the current moment, and applying the voltage control signal at the current moment to each liquid crystal cell in the liquid crystal beam control layer within the display screen.

2. The brightness control method based on AI and eye tracking according to claim 1, characterized in that, The collection of eye images and face images acquired when the user gazes at the display screen includes: A two-dimensional coordinate system is established on the display screen; The infrared camera corresponding to the display screen captures raw eye images and raw face images of the user when they gaze at multiple different coordinate points in the two-dimensional coordinate system. Perform face region detection on the original eye image and the original face image to obtain the face region location; Perform binocular localization based on the face region location to obtain the binocular region location; Based on the location of the two eye regions, eye images containing one or both eyes are cropped from the original eye image to obtain an initial set of eye images; Extract face images from the original face image to obtain an initial set of face images; Perform data preprocessing operations on the initial set of eye images and the initial set of face images to obtain the set of eye images and the set of face images, wherein the preprocessing operations include at least one of the following: geometric normalization processing, grayscale normalization processing, and data augmentation processing.

3. The brightness control method based on AI and eye tracking according to claim 1, characterized in that, The step of obtaining the pupil-corneal reflection vector between the pupil center and the corneal reflection point based on the pupil center and corneal reflection point of the eye images in the eye image set includes: A threshold segmentation operation is performed on the eye images in the set of eye images to obtain binarized eye images; An ellipse fitting operation is performed on the binarized eye image, and a set of pupil center coordinates is generated based on the pupil center coordinates in the obtained binarized eye image; Detect corneal reflection bright spots generated by infrared supplementary light in the eye images in the eye image set, and generate a corneal reflection point coordinate set based on the obtained corneal reflection point coordinates in the eye images; The pupil-corneal reflection vector corresponding to the eye image is obtained by calculating the difference in abscissa and ordinate between the pupil center coordinates and the corneal reflection point coordinates at the same acquisition time in the pupil center coordinate set and the corneal reflection point coordinate set.

4. The brightness control method based on AI and eye tracking according to claim 1, characterized in that, The step of using the set of eye images, the set of face images, and the pupil-corneal reflection vector corresponding to the eye images in the set of eye images as training data to train the initial gaze point recognition model, and obtaining the trained gaze point recognition model, includes: Obtain the initial gaze point recognition model, which includes an input layer, a feature extractor, a fully connected layer, and an output layer; The set of eye images is input into the input layer to obtain eye image data to be processed, and the set of face images is input into the input layer to obtain face image data to be processed. The eye image data to be processed and the face image data to be processed are input into the feature extractor. The convolutional layer and pooling layer of the feature extractor are used to extract the eye morphology features and facial pose features. The pupil-corneal reflection vector corresponding to the eye image, the head rotation and translation vector estimated from the facial key points, and the gradient histogram features and scale-invariant feature transform operator features of the image are also extracted. Based on the eye morphology features, facial posture features, pupil-corneal reflection vector, head rotation and translation vector, gradient histogram features, and scale-invariant feature transform operator features, a spliced ​​feature vector is obtained; The spliced ​​feature vector is input into the fully connected layer, and the flattened feature vector is obtained through the multi-layer fully connected operation of the fully connected layer. The flattened feature vector is input into the output layer, and the normalized predicted horizontal coordinate and predicted vertical coordinate are output through the output layer to obtain the set of predicted gaze point coordinates. The output layer includes a linear layer with two neurons. Calculate the mean squared error loss value between each predicted gaze point coordinate in the predicted gaze point coordinate set and the corresponding real gaze point coordinate in the pre-calibrated real gaze point coordinate set; Based on the mean squared error loss value, the weight parameters of the initial gaze recognition model are updated using the backpropagation algorithm until the mean squared error loss value meets the preset convergence condition, thus obtaining the trained gaze recognition model.

5. The brightness control method based on AI and eye tracking according to claim 1, characterized in that, The training process of the AI ​​decision-making model includes: Obtain a pre-constructed initial AI decision model and a training dataset of the initial AI decision model. The initial AI decision model includes a region mapping encoder and a voltage generation decoder. The training samples in the training dataset include training gaze point positions and corresponding training voltage control signals. The training voltage control signals are used to control each liquid crystal unit in the liquid crystal beam control layer of the display screen to converge the outgoing light rays toward the training gaze point position. The training gaze point position is input into the region mapping encoder. The coordinate mapping operation of the region mapping encoder determines the training pixel coordinates of the training gaze point position in the pixel coordinate system of the display screen. Based on the training pixel coordinates, a training region growth seed point centered on the training pixel coordinates is generated. The training region growth seed points are processed by the region growth operation of the region mapping encoder, and the training growth boundary is determined according to the brightness gradient value between adjacent pixels to generate a set of training boundary coordinates for the training target pixel region that needs to be enhanced in brightness. The training boundary coordinate set is input into the voltage generation decoder, and the training boundary coordinate set is mapped to the liquid crystal unit coordinate system of the liquid crystal beam control layer through the geometric transformation operation of the voltage generation decoder to obtain the training unit index set. The voltage generation decoder processes the training unit index set through voltage distribution operations, generates corresponding training voltage amplitude parameters based on the distance parameters between each liquid crystal unit in the training unit index set and the training gaze point position, and obtains a training voltage control signal based on the training voltage amplitude parameters. Calculate the voltage deviation between the training voltage control signal and the pre-calibrated target voltage control signal, and update the network weight parameters of the initial AI decision model according to the voltage deviation value through the backpropagation algorithm until the voltage deviation value meets the preset training convergence condition, thereby obtaining the trained AI decision model.

6. The brightness control method based on AI and eye tracking according to claim 1, characterized in that, Before acquiring a set of eye images and a set of face images when the user is looking at the display screen, the method further includes: Multiple calibration marker points with known coordinates are displayed sequentially on the display screen; When displaying the calibration marker, the calibration eye features are collected when the user gazes at the calibration marker. The calibration eye features include the calibration pupil-corneal reflection vector, the calibration eye image, and the calibration face image. The known coordinates of the calibration markers and the corresponding calibration eye features are combined to form an eye movement calibration association sample, thus obtaining an eye movement calibration association sample set. The calibration eye features in the eye-tracking calibration associated sample set are input into the gaze point recognition model to obtain the model prediction coordinates corresponding to the eye-tracking calibration associated samples. The total error value of the eye-tracking calibration associated sample set is obtained based on the coordinate deviation between the known coordinates and the corresponding model-predicted coordinates in the eye-tracking calibration associated samples. The model parameter set of the gaze point recognition model is repeatedly adjusted, and the total error value of the eye movement calibration associated sample set is recalculated after each adjustment until the total error value is less than or equal to a preset total error threshold, or the adjustment operation is repeated a preset number of times, to obtain the target model parameter set. The gaze recognition model is updated using the target model parameter set.

7. The brightness control method based on AI and eye tracking according to claim 1, characterized in that, After inputting the real-time acquired eye images, real-time face images, and the real-time pupil-corneal reflection vectors corresponding to the real-time eye images into the gaze point recognition model, the method further includes: Obtain the real-time gaze point position output by the gaze point recognition model and the acquisition timestamp corresponding to the real-time gaze point position; Based on the real-time gaze point position and the acquisition timestamp, a real-time gaze point position sequence is obtained; The real-time gaze position sequence is input into a preset trajectory prediction filter, and the real-time gaze position sequence is processed by the state transition equation of the trajectory prediction filter to obtain a filtered gaze position sequence. Based on the positional difference and time difference between two adjacent filtered fixation point positions in the filtered fixation point position sequence, a fixation point movement speed sequence is obtained; The filtered gaze position sequence and the gaze movement velocity sequence are input into the observation update equation of the trajectory prediction filter to obtain the predicted gaze position at future time. The predicted gaze position is then input into the AI ​​decision model as the updated real-time gaze position.

8. The brightness control method based on AI and eye tracking according to claim 7, characterized in that, The step of inputting the real-time gaze position sequence into a preset trajectory prediction filter and processing the real-time gaze position sequence through the state transition equation of the trajectory prediction filter to obtain a filtered gaze position sequence includes: Obtain the initial state vector of the trajectory prediction filter, the initial state vector including the initial gaze point position and the initial gaze point movement speed; Assign the first real-time gaze point position in the real-time gaze point position sequence to the initial gaze point position in the initial state vector, and assign the preset initial velocity value to the initial gaze point movement velocity in the initial state vector to obtain the current state vector; Each real-time gaze point position in the real-time gaze point position sequence is sequentially input into the trajectory prediction filter. For the currently input real-time gaze point position, the predicted state vector at the current time is calculated based on the state vector at the previous time step using the state transition equation. The predicted state vector includes the predicted gaze point position and the predicted gaze point movement speed. The Kalman gain is calculated based on the positional deviation between the current input real-time gaze point position and the predicted gaze point position in the predicted state vector. The predicted state vector is weighted and corrected according to the Kalman gain to obtain the filtered state vector at the current time. The filtered state vector includes the filtered gaze point position and the filtered gaze point movement speed. The filtered gaze point positions obtained at each time point are arranged in chronological order to obtain the filtered gaze point position sequence.

9. A display device based on AI and eye tracking, characterized in that, The device includes a display panel, an LED array, and a liquid crystal beam control layer located between the LED array and the display panel. It also includes an AI decision module connected to the liquid crystal beam control layer and a camera. The AI ​​decision module is used to execute the AI-based and eye-tracking brightness control method of any one of claims 1-8.

Images