Vehicle-mounted display control method and system based on human-computer interaction

Abstract

The present invention provides a vehicle display control method and system based on human-computer interaction, which relates to the field of display control technology, including collecting the driver's facial image through a vehicle camera, extracting eye movement features to calculate the real-time gaze point coordinates; matching the gaze point coordinates with a preset interactive area to trigger an activation state; generating interactive display content according to the activation state, and calculating the dynamic projection position; detecting the movement trajectory of the gaze point in the activation area, identifying the interactive gesture to trigger the control instruction; updating the display interface according to the control instruction and issuing a feedback prompt sound. The method improves the interaction efficiency and safety of the vehicle display system.

Description

Technical Field

[0001] The present invention relates to display control technology, and in particular to a vehicle-mounted display control method and system based on human-computer interaction. Background Art

[0002] Traditional in-vehicle display systems usually use a fixed position display method, which cannot be dynamically adjusted according to the driver's line of sight. This fixed display method can easily distract the driver's attention and affect driving safety.

[0003] Existing in-vehicle interactive systems mostly use touch screens or physical buttons for operation, requiring the driver to remove his hands from the steering wheel to operate, increasing the risk of distracted driving. The lack of a non-contact interaction method based on the driver's line of sight makes it impossible to achieve safer and more convenient human-computer interaction.

[0004] Current in-vehicle display systems often lack the ability to intelligently analyze and personalize responses to the driver's gaze behavior. They are unable to adaptively adjust display content and interaction methods based on the driver's real-time gaze, making it difficult to provide the driver with a more intuitive and efficient information acquisition and control experience. Summary of the invention

[0005] The embodiments of the present invention provide a vehicle-mounted display control method and system based on human-computer interaction, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] Provide a vehicle display control method based on human-computer interaction, including:

[0008] The driver's facial image is collected through the vehicle camera, the driver's eye movement features are extracted using a deep learning model, and the real-time gaze point coordinates of the driver are calculated based on the eye movement features; the real-time gaze point coordinates are matched with a preset vehicle-mounted augmented reality display interface interaction area, and when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold, the activation state of the interaction area is triggered;

[0009] According to the interactive area type corresponding to the activation state, the corresponding augmented reality display template is called, and the interactive display content is generated in combination with the vehicle driving state information, navigation information and environmental perception information; the dynamic projection position of the interactive display content is calculated based on the real-time gaze point coordinates, and the dynamic projection position is adjusted in real time with the change of the real-time gaze point coordinates to ensure that the interactive display content is always within the effective field of view of the driver;

[0010] Detect the movement trajectory of the real-time gaze point coordinates in the activated interactive area, and match the movement trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is recognized, trigger a corresponding control instruction according to the type of the interactive gesture, and the control instruction is used to adjust the display parameters of the interactive display content or perform functional operations; update the display interface based on the execution result of the control instruction, and emit an interactive feedback prompt sound through the vehicle-mounted speaker.

[0011] The driver's facial image is collected by the vehicle camera, the driver's eye movement features are extracted by the deep learning model, and the driver's real-time gaze point coordinates are calculated based on the eye movement features, including:

[0012] The driver's facial image is collected by a vehicle-mounted binocular infrared camera, the image resolution of the vehicle-mounted binocular infrared camera is 1920×1080 pixels, and the acquisition frame rate of the vehicle-mounted binocular infrared camera is 60fps; the vehicle-mounted near-infrared fill light is turned on, and the luminous intensity of the vehicle-mounted near-infrared fill light is adaptively adjusted according to the ambient light intensity, and the wavelength range of the vehicle-mounted near-infrared fill light is 850-940nm; the exposure parameters of the vehicle-mounted binocular infrared camera are adaptively adjusted according to the luminous intensity of the vehicle-mounted near-infrared fill light to generate a stable driver's facial image;

[0013] Inputting the driver's facial image into a preset adaptive enhancement network, wherein the adaptive enhancement network includes a brightness enhancement sub-network and a contrast enhancement sub-network, wherein the brightness enhancement sub-network uses a multi-scale convolution structure to extract image brightness and darkness features, and the contrast enhancement sub-network uses a residual connection structure to extract image texture features; fusing the output features of the brightness enhancement sub-network and the output features of the contrast enhancement sub-network to generate an enhanced driver's facial image;

[0014] The enhanced driver's facial image is input into a preset eye movement feature extraction network, which includes a face detection module, a key point positioning module and a feature extraction module. The face detection module detects the face area in the enhanced driver's facial image, the key point positioning module locates the key points of the eye area in the face area, and the feature extraction module extracts the eye movement features of the eye area; a driver's gaze point coordinate mapping model is established based on the eye movement features, and the driver's real-time gaze point coordinates are calculated according to the driver's gaze point coordinate mapping model.

[0015] Matching the real-time gaze point coordinates with a preset in-vehicle augmented reality display interface interaction area, and when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold, triggering the activation state of the interaction area includes:

[0016] A plurality of interaction areas are preset in the in-vehicle augmented reality display interface, each of the interaction areas includes a core interaction area, a secondary interaction area and an auxiliary interaction area, the core interaction area is located in the central area of ​​the in-vehicle augmented reality display interface and occupies 20% of the area of ​​the in-vehicle augmented reality display interface, the secondary interaction area is located in the peripheral area of ​​the core interaction area and occupies 40% of the area of ​​the in-vehicle augmented reality display interface, and the auxiliary interaction area is located in the peripheral area of ​​the secondary interaction area and occupies 40% of the area of ​​the in-vehicle augmented reality display interface;

[0017] Collecting vehicle driving state information and driver's perspective information, determining the priority of display content of the interactive area based on the vehicle driving state information, and establishing an adaptive layout model of the interactive area based on the driver's perspective information; dynamically adjusting the position coordinates and display size of the interactive area according to the adaptive layout model of the interactive area;

[0018] Convert the received real-time gaze point coordinates into standardized gaze point coordinates in the standardized coordinate system of the vehicle-mounted augmented reality display interface; construct a region index table of a quadtree structure, divide the vehicle-mounted augmented reality display interface into grid units of different scales, and establish a mapping relationship between the grid units and the interactive area;

[0019] A hierarchical search strategy is adopted based on the regional index table to determine the target grid unit where the standardized gaze point coordinates are located at a coarse-grained level, and to determine the target interaction area corresponding to the standardized gaze point coordinates at a fine-grained level; when the standardized gaze point coordinates stay continuously within the range of the target interaction area for a time that exceeds a preset activation time threshold, the activation state of the target interaction area is triggered.

[0020] Calculating the dynamic projection position of the interactive display content based on the real-time gaze point coordinates, and adjusting the dynamic projection position in real time as the real-time gaze point coordinates change, to ensure that the interactive display content is always within the effective field of view of the driver includes:

[0021] Establishing a driver's visual field model, the driver's visual field model includes a foveal visual area, a secondary foveal visual area and a peripheral visual area, the foveal visual area is an area of ​​2° to the left and right of the gaze point, the secondary foveal visual area is an area of ​​2° to 10° to the left and right of the gaze point, and the peripheral visual area is an area of ​​10° to 30° to the left and right of the gaze point; the foveal visual area is used to display key warning information, the secondary foveal visual area is used to display vehicle status information, and the peripheral visual area is used to display environmental perception information;

[0022] Collecting the driver's head posture information and the vehicle's driving state information, establishing a head feature point tracking model based on the driver's head posture information, and establishing a dynamic compensation model based on the vehicle's driving state information; converting the received real-time gaze point coordinates into three-dimensional space projection coordinates of the vehicle-mounted augmented reality display system through the head feature point tracking model and the dynamic compensation model;

[0023] A Kalman filter prediction model is established based on the historical movement trajectory of the real-time gaze point coordinates, the predicted gaze point coordinates at the next moment are calculated by the Kalman filter prediction model, and the projection position of the interactive display content is pre-calculated according to the predicted gaze point coordinates; when a new real-time gaze point coordinate is obtained, an error is calculated between the new real-time gaze point coordinates and the predicted gaze point coordinates, and the projection position of the interactive display content is corrected based on the error calculation result;

[0024] A buffer area is established around the real-time gaze point coordinates, and the range of the buffer area is positively correlated with the vehicle speed; when the real-time gaze point coordinates move within the buffer area, the projection position of the interactive display content remains unchanged; when the real-time gaze point coordinates move out of the buffer area, the projection position of the interactive display content is recalculated based on the real-time gaze point coordinates.

[0025] Establishing a Kalman filter prediction model based on the historical movement trajectory of the real-time gaze point coordinates, calculating the predicted gaze point coordinates at the next moment through the Kalman filter prediction model, and pre-calculating the projection position of the interactive display content according to the predicted gaze point coordinates includes:

[0026] Constructing a gaze point motion state vector, wherein the gaze point motion state vector includes a position coordinate, a velocity component, and an acceleration component of the gaze point; establishing a state transfer matrix based on the gaze point motion state vector, wherein the state transfer matrix describes a mapping relationship between gaze point motion states at adjacent moments;

[0027] Establishing an adaptive noise covariance matrix, the adaptive noise covariance matrix includes a process noise covariance matrix and a measurement noise covariance matrix, wherein the noise variance of the position component in the process noise covariance matrix is ​​0.01 square meters, the noise variance of the velocity component is 0.1 square meters per square second, and the noise variance of the acceleration component is 1.0 square meters per square second; the noise variance of the position measurement in the measurement noise covariance matrix is ​​0.005 square meters;

[0028] Predicting the current state according to the state transfer matrix to obtain a predicted state; calculating a Kalman gain based on the predicted state and the adaptive noise covariance matrix, wherein the Kalman gain is used to determine a weight distribution between a predicted value and a measured value; and updating the predicted state based on the Kalman gain to obtain an updated predicted state;

[0029] A projection transformation matrix is ​​established, wherein the projection transformation matrix includes the focal length, principal point coordinates, position and posture information of the display device; the updated prediction state is mapped to the display plane through the projection transformation matrix to obtain the projection position of the interactive display content; the prediction time span is determined based on the system processing delay and the display refresh cycle, and the prediction time span is 1.2 times the total system delay; and a Bezier curve interpolation algorithm is used to generate a smooth transition trajectory between adjacent prediction points.

[0030] Detecting the movement trajectory of the real-time gaze point coordinates in the activated interactive area, and matching the movement trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is recognized, triggering a corresponding control instruction according to the type of the interactive gesture includes:

[0031] Detecting the moving trajectory of the real-time gaze point coordinates in the activated interactive area, collecting a gaze point coordinate sequence, wherein the gaze point coordinate sequence includes a timestamp, a two-dimensional spatial position, and gaze duration information; when the gaze duration exceeds 100 milliseconds and the spatial jitter amplitude is less than 1 degree of viewing angle, it is determined to be a valid gaze point; and a linear interpolation method is used to correct an invalid gaze point;

[0032] Performing median filtering on the gaze point coordinate sequence, wherein the window size of the median filtering is 5 sampling points; simplifying the filtered gaze point coordinate sequence using the Douglas-Peucker algorithm, wherein the distance threshold of the Douglas-Peucker algorithm is 0.5 degree of visual angle; and normalizing the simplified gaze point coordinate sequence to obtain a standardized trajectory;

[0033] Constructing a hierarchical interactive gesture library, the interactive gesture library includes a basic layer gesture and a combined layer gesture, the basic layer gesture includes a single straight line gesture, an arc gesture and a broken line gesture, the combined layer gesture is formed by connecting the basic layer gestures in series; each of the interactive gestures includes a standard trajectory, a time constraint, a space constraint and a context constraint;

[0034] Calculating a characteristic distance matrix between the standardized trajectory and the interactive gestures in the interactive gesture library, wherein the characteristic distance matrix includes point-to-point Euclidean distance, tangent angle difference, and curvature difference; based on the characteristic distance matrix, a dynamic time warping algorithm is used to find an optimal matching path to obtain a regularized distance score;

[0035] Construct a state machine including an idle state, a ready state, a collection state, and a recognition state; when it is detected that the gaze point enters the interactive area and stays there for more than 200 milliseconds, the state machine switches from the idle state to the ready state; when the accumulated trajectory length reaches a preset length threshold or a direction change is detected, the state machine switches from the ready state to the collection state; when the gaze point leaves the interactive area or stays there for more than 2 seconds, the state machine switches from the collection state to the recognition state;

[0036] In the recognition state, the legality of the standardized trajectory is checked; a sliding time window is established, the sliding time window records the matching degree distribution of the most recent 50 recognitions, and the recognition threshold is dynamically adjusted based on the matching degree distribution; the regularized distance score is compared with the recognition threshold to obtain an interactive gesture recognition result;

[0037] Based on the interactive gesture recognition result, control parameters are extracted from a configuration matrix, wherein the rows of the configuration matrix represent interactive gesture types, the columns of the configuration matrix represent system functional modules, and the control parameters include function switch states, adjustment ranges, and execution priorities; and a control instruction sequence is generated according to the control parameters.

[0038] According to a second aspect of the embodiments of the present invention,

[0039] Provides an in-vehicle display control system based on human-computer interaction, including:

[0040] The first unit is used to collect a facial image of the driver through a vehicle-mounted camera, extract the driver's eye movement features using a deep learning model, and calculate the driver's real-time gaze point coordinates based on the eye movement features; match the real-time gaze point coordinates with a preset vehicle-mounted augmented reality display interface interaction area, and trigger the activation state of the interaction area when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold;

[0041] The second unit is used to call the corresponding augmented reality display template according to the interactive area type corresponding to the activation state, and generate interactive display content in combination with the vehicle driving state information, navigation information and environmental perception information; calculate the dynamic projection position of the interactive display content based on the real-time gaze point coordinates, and the dynamic projection position is adjusted in real time with the change of the real-time gaze point coordinates to ensure that the interactive display content is always within the effective field of view of the driver;

[0042] The third unit is used to detect the moving trajectory of the real-time gaze point coordinates in the activated interactive area, and match the moving trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is recognized, a corresponding control instruction is triggered according to the type of the interactive gesture, and the control instruction is used to adjust the display parameters of the interactive display content or perform functional operations; based on the execution result of the control instruction, the display interface is updated, and an interactive feedback prompt sound is issued through the vehicle-mounted speaker.

[0043] According to a third aspect of the embodiments of the present invention,

[0044] An electronic device is provided, comprising:

[0045] processor;

[0046] a memory for storing processor-executable instructions;

[0047] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0048] According to a fourth aspect of the embodiments of the present invention,

[0049] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0050] The beneficial effects of this application are as follows:

[0051] 1. Enhance driving safety and interactive convenience:

[0052] The present invention uses a vehicle-mounted camera to collect the driver's facial image in real time, uses a deep learning model to extract eye movement features, and calculates the real-time gaze point coordinates. This non-contact interaction method does not require manual operation by the driver, greatly reducing the risk of distracted driving. At the same time, the system can automatically activate the corresponding interaction area according to the driver's line of sight, improving the convenience and accuracy of human-computer interaction.

[0053] 2. Improve the intelligence and personalization of information display:

[0054] The present invention intelligently generates corresponding augmented reality display content based on the type of activated interactive area, combined with the vehicle's driving status, navigation information and environmental perception information. In addition, the system can dynamically adjust the projection position of the display content according to the driver's real-time gaze point to ensure that the information is always within the driver's effective field of view. This intelligent and personalized display method effectively improves the efficiency of information transmission and user experience.

[0055] 3. Realize intuitive and natural human-computer interaction control:

[0056] The present invention realizes natural interactive control based on eye movement by identifying the movement trajectory of the driver's line of sight in the interactive area and matching it with the pre-stored interactive gesture library. When a valid interactive gesture is identified, the system triggers the corresponding control command to adjust the display parameters or perform functional operations. This innovative interactive method is not only intuitive and natural to operate, but also effectively reduces the use of traditional physical buttons, simplifies the operating interface of the vehicle system, and improves the interaction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of a vehicle display control method based on human-computer interaction according to an embodiment of the present invention;

[0058] Figure 2 The figure is a schematic diagram of the structure of an in-vehicle display control system based on human-computer interaction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0061] Figure 1 FIG. 1 is a flow chart of a vehicle display control method based on human-computer interaction according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0062] S101. Capture the driver's facial image through the vehicle camera, extract the driver's eye movement features using the deep learning model, and calculate the driver's real-time gaze point coordinates based on the eye movement features; match the real-time gaze point coordinates with the preset vehicle-mounted augmented reality display interface interaction area, and when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold, trigger the activation state of the interaction area;

[0063] S102. According to the interactive area type corresponding to the activation state, the corresponding augmented reality display template is called, and the interactive display content is generated in combination with the vehicle driving state information, navigation information and environmental perception information; the dynamic projection position of the interactive display content is calculated based on the real-time gaze point coordinates, and the dynamic projection position is adjusted in real time with the change of the real-time gaze point coordinates to ensure that the interactive display content is always within the effective field of view of the driver;

[0064] S103. Detect the movement trajectory of the real-time gaze point coordinates in the activated interactive area, and match the movement trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is identified, trigger a corresponding control instruction according to the type of the interactive gesture, and the control instruction is used to adjust the display parameters of the interactive display content or perform functional operations; update the display interface based on the execution result of the control instruction, and emit an interactive feedback prompt sound through the vehicle-mounted speaker.

[0065] The present invention provides a vehicle-mounted display control method based on human-computer interaction, and the specific implementation methods are as follows:

[0066] First, the driver's facial image is collected in real time through the on-board camera. A high-resolution, high-frame-rate camera, such as a 1080p resolution, 60fps frame-rate camera, is used to ensure the clarity and real-time nature of the collected image. The camera is installed on the steering column or rearview mirror, and is aimed at the driver's face.

[0067] Next, a deep learning model is used to extract the driver's eye movement features from the collected facial images. Specifically, a convolutional neural network (CNN) model, such as the VGG16 or ResNet50 network structure, is used to extract features from facial images and locate the eye area. Then a recurrent neural network (RNN) model, such as the LSTM network, is used to perform time series analysis on the eye area and extract dynamic features such as eye movement trajectory and blinking frequency.

[0068] Based on the extracted eye movement features, the real-time coordinates of the driver's gaze point are calculated. Using a sight estimation algorithm, such as the pupil-corneal reflection method, the eye movement features are mapped to the coordinate system of the vehicle display screen to obtain the x and y coordinate values ​​of the gaze point on the screen. The coordinate calculation frequency is not less than 30Hz to ensure real-time performance.

[0069] The calculated real-time gaze point coordinates are matched with the preset in-vehicle augmented reality display interface interaction area. The display interface is pre-divided into multiple interaction areas, such as navigation area, music control area, vehicle status area, etc. Each area corresponds to a specific rectangular coordinate range. When the gaze point falls into the coordinate range of a certain area, the timing starts. If the gaze point stays in the area for more than a preset threshold (such as 1 second), the activation state of the interaction area is triggered.

[0070] According to the type of interactive area activated, the corresponding augmented reality display template is called. For example, when the navigation area is activated, the navigation information template is called, and when the music area is activated, the music playback control template is called. Combined with driving status information such as vehicle speed and direction, GPS navigation information, and environmental perception information such as roads and traffic signs obtained through vehicle sensors, rich interactive display content is generated.

[0071] The dynamic projection position of the interactive display content is calculated based on the real-time gaze point coordinates. The eye tracking algorithm is used to keep a certain offset between the center point coordinates of the display content and the gaze point coordinates (such as 20 pixels horizontally and 10 pixels vertically), and the display position is adjusted in real time as the gaze point moves. This ensures that the interactive content is always within the driver's effective field of view to avoid blocking important sightlines.

[0072] Continuously detect the moving trajectory of the real-time gaze point coordinates in the activated interactive area. Use the trajectory smoothing algorithm to reduce the noise of the original coordinate sequence and extract the effective moving trajectory. Match the processed trajectory with the pre-stored interactive gesture library. The gesture library contains standard trajectory templates for commonly used gestures such as sliding up, down, left, right, drawing a circle, and drawing a hook.

[0073] When a valid interactive gesture is recognized, the corresponding control command is triggered according to the gesture type. For example, sliding up triggers the volume increase command, drawing a circle triggers the loop play command, etc. The control command is used to adjust the display parameters of the interactive display content (such as brightness, contrast, font size, etc.) or perform functional operations (such as switching songs, setting destinations, etc.).

[0074] The display interface is updated in real time based on the execution results of the control instructions, such as volume bar changes, playlist updates, etc. At the same time, a short interactive feedback prompt tone, such as a "beep", is emitted through the vehicle speakers to enhance the realism and immediacy of the interactive experience.

[0075] In an optional implementation, the vehicle-mounted camera collects the driver's facial image, uses a deep learning model to extract the driver's eye movement features, and calculates the driver's real-time gaze point coordinates based on the eye movement features, including:

[0076] The driver's facial image is collected by a vehicle-mounted binocular infrared camera, the image resolution of the vehicle-mounted binocular infrared camera is 1920×1080 pixels, and the acquisition frame rate of the vehicle-mounted binocular infrared camera is 60fps; the vehicle-mounted near-infrared fill light is turned on, and the luminous intensity of the vehicle-mounted near-infrared fill light is adaptively adjusted according to the ambient light intensity, and the wavelength range of the vehicle-mounted near-infrared fill light is 850-940nm; the exposure parameters of the vehicle-mounted binocular infrared camera are adaptively adjusted according to the luminous intensity of the vehicle-mounted near-infrared fill light to generate a stable driver's facial image;

[0077] Inputting the driver's facial image into a preset adaptive enhancement network, wherein the adaptive enhancement network includes a brightness enhancement sub-network and a contrast enhancement sub-network, wherein the brightness enhancement sub-network uses a multi-scale convolution structure to extract image brightness and darkness features, and the contrast enhancement sub-network uses a residual connection structure to extract image texture features; fusing the output features of the brightness enhancement sub-network and the output features of the contrast enhancement sub-network to generate an enhanced driver's facial image;

[0078] The enhanced driver's facial image is input into a preset eye movement feature extraction network, which includes a face detection module, a key point positioning module and a feature extraction module. The face detection module detects the face area in the enhanced driver's facial image, the key point positioning module locates the key points of the eye area in the face area, and the feature extraction module extracts the eye movement features of the eye area; a driver's gaze point coordinate mapping model is established based on the eye movement features, and the driver's real-time gaze point coordinates are calculated according to the driver's gaze point coordinate mapping model.

[0079] This embodiment provides a method for collecting the driver's facial image through a vehicle-mounted camera and extracting the driver's eye movement features using a deep learning model, and then calculating the driver's real-time gaze point coordinates. The specific steps are as follows:

[0080] First, the driver's facial image is collected by the on-board binocular infrared camera. The image resolution of the on-board binocular infrared camera is 1920×1080 pixels, and the acquisition frame rate is 60fps. At the same time, the on-board near-infrared fill light is turned on, and the luminous intensity of the fill light is adaptively adjusted according to the ambient light intensity. The wavelength range of the on-board near-infrared fill light is 850-940nm. For example, when the ambient light is dark, the luminous intensity of the fill light can be increased to 80%; when the ambient light is bright, the luminous intensity of the fill light can be reduced to 20%. According to the luminous intensity of the fill light, the exposure parameters of the on-board binocular infrared camera are adaptively adjusted to generate a stable driver's facial image. Specifically, when the luminous intensity of the fill light is high, the exposure time of the camera can be appropriately shortened, such as setting it to 1 / 120 seconds; when the luminous intensity of the fill light is low, the exposure time of the camera can be appropriately extended, such as setting it to 1 / 60 seconds.

[0081] Next, the collected driver's facial image is input into the preset adaptive enhancement network. The adaptive enhancement network includes a brightness enhancement subnetwork and a contrast enhancement subnetwork. The brightness enhancement subnetwork uses a multi-scale convolution structure to extract the brightness and darkness features of the image. Specifically, three different sizes of convolution kernels, 3×3, 5×5 and 7×7, are used to perform convolution operations on the input image respectively, and then the feature maps of the three scales are fused to obtain multi-scale brightness features. The contrast enhancement subnetwork uses a residual connection structure to extract image texture features. Specifically, multiple residual blocks are used, each of which contains two convolutional layers and a jump connection, and the texture information of the image is extracted through residual learning. Finally, the output features of the brightness enhancement subnetwork are fused with the output features of the contrast enhancement subnetwork to generate an enhanced driver's facial image.

[0082] Then, the enhanced driver's facial image is input into the preset eye movement feature extraction network. The network includes a face detection module, a key point location module and a feature extraction module. The face detection module uses an improved YOLO algorithm to detect the face area in the enhanced driver's facial image. The key point location module locates the key points of the eye area in the detected face area, including the corner of the eye, eyelid and pupil center. The feature extraction module extracts the eye movement features of the eye area, including the pupil center coordinates, eyelid opening and closing degree, eyeball rotation angle, etc.

[0083] Finally, a mapping model of the driver's gaze point coordinates is established based on the extracted eye movement features. The model adopts a multi-layer perceptron structure, with eye movement features as input and the two-dimensional coordinates of the gaze point in the front field of view of the vehicle as output. Through training with a large amount of labeled data, the mapping relationship between eye movement features and gaze point coordinates is established. In practical applications, the real-time coordinates of the driver's gaze point are calculated using the trained mapping model based on the eye movement features extracted in real time.

[0084] The beneficial effects of this embodiment include the following three aspects:

[0085] 1. The vehicle-mounted binocular infrared camera and near-infrared fill light can stably capture the driver's facial image in various lighting environments, improving the reliability of subsequent processing.

[0086] 2. An adaptive enhancement network is introduced to pre-process the collected facial images, effectively improving the image quality and providing better input for subsequent feature extraction.

[0087] 3. A special eye movement feature extraction network and gaze point coordinate mapping model are designed, which can accurately extract the driver's eye movement features and calculate the gaze point coordinates in real time, providing an important basis for driver status monitoring.

[0088] In an optional implementation, the real-time gaze point coordinates are matched with a preset in-vehicle augmented reality display interface interaction area, and when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold, triggering the activation state of the interaction area includes:

[0089] A plurality of interaction areas are preset in the in-vehicle augmented reality display interface, each of the interaction areas includes a core interaction area, a secondary interaction area and an auxiliary interaction area, the core interaction area is located in the central area of ​​the in-vehicle augmented reality display interface and occupies 20% of the area of ​​the in-vehicle augmented reality display interface, the secondary interaction area is located in the peripheral area of ​​the core interaction area and occupies 40% of the area of ​​the in-vehicle augmented reality display interface, and the auxiliary interaction area is located in the peripheral area of ​​the secondary interaction area and occupies 40% of the area of ​​the in-vehicle augmented reality display interface;

[0090] Collecting vehicle driving state information and driver's perspective information, determining the priority of display content of the interactive area based on the vehicle driving state information, and establishing an adaptive layout model of the interactive area based on the driver's perspective information; dynamically adjusting the position coordinates and display size of the interactive area according to the adaptive layout model of the interactive area;

[0091] Convert the received real-time gaze point coordinates into standardized gaze point coordinates in the standardized coordinate system of the vehicle-mounted augmented reality display interface; construct a region index table of a quadtree structure, divide the vehicle-mounted augmented reality display interface into grid units of different scales, and establish a mapping relationship between the grid units and the interactive area;

[0092] A hierarchical search strategy is adopted based on the regional index table to determine the target grid unit where the standardized gaze point coordinates are located at a coarse-grained level, and to determine the target interaction area corresponding to the standardized gaze point coordinates at a fine-grained level; when the standardized gaze point coordinates stay continuously within the range of the target interaction area for a time that exceeds a preset activation time threshold, the activation state of the target interaction area is triggered.

[0093] When implementing the method, it is first necessary to preset multiple interaction areas in the vehicle-mounted augmented reality display interface. Each interaction area includes a core interaction area, a secondary interaction area, and an auxiliary interaction area. The core interaction area is located in the central area of ​​the vehicle-mounted augmented reality display interface, occupying 20% ​​of the area of ​​the display interface. The secondary interaction area is located in the peripheral area of ​​the core interaction area, occupying 40% of the area of ​​the display interface. The auxiliary interaction area is located in the peripheral area of ​​the secondary interaction area, occupying 40% of the area of ​​the display interface.

[0094] For example, for an in-vehicle augmented reality display interface with a resolution of 1920×1080 pixels, the core interaction area can be set to a rectangular area of ​​560×420 pixels in the center, the secondary interaction area can be set to a ring area of ​​840×630 pixels outside the core area, and the auxiliary interaction area is the outermost area.

[0095] Next, the system will collect vehicle driving status information and driver perspective information. Vehicle driving status information may include parameters such as vehicle speed, steering wheel angle, acceleration, etc. Driver perspective information can obtain driver head posture data through the in-car camera. Based on the vehicle driving status information, the system determines the priority of the interactive area display content. For example, when the vehicle speed is high, the priority of safety-related interactive areas will increase. Based on the driver's perspective information, the system establishes an adaptive layout model for the interactive area. The model can dynamically adjust the position and size of each interactive area according to the driver's line of sight to make it more in line with the driver's field of vision.

[0096] The system dynamically adjusts the position coordinates and display size of the interactive area based on the adaptive layout model. For example, when the driver's line of sight is to the right, the interactive area on the right can be appropriately enlarged and moved closer to the center, while the area on the left is reduced accordingly. This dynamic adjustment ensures that the interactive area is always within the driver's optimal field of vision.

[0097] In order to achieve efficient gaze point matching, the system converts the received real-time gaze point coordinates into standardized gaze point coordinates in the standardized coordinate system of the vehicle-mounted augmented reality display interface. For example, the original pixel coordinates can be converted into normalized coordinates in the range of [0,1].

[0098] Subsequently, the system constructs a regional index table with a quadtree structure. First, the vehicle-mounted augmented reality display interface is divided into grid units of different scales, starting from the top-level 1×1 grid, and then subdivided into 2×2, 4×4 and other finer-grained grids. Then, a mapping relationship between the grid unit and the interactive area is established, and the interactive area information contained or covered by each grid unit is recorded.

[0099] When performing gaze point matching, the system adopts a hierarchical search strategy based on the region index table. First, the target grid unit where the standardized gaze point coordinates are located is determined at the coarse-grained level. For example, for the standardized coordinates (0.3, 0.7), the system first locates the upper right corner unit in the 2×2 grid. Then the search continues at the finer-grained level corresponding to the unit, and finally the target interaction area corresponding to the standardized gaze point coordinates is determined.

[0100] When the target interaction area is determined, the system starts timing. If the standardized gaze point coordinates stay within the target interaction area for longer than the preset activation time threshold (e.g. 1.5 seconds), the activation state of the target interaction area is triggered. The activation state may include highlighting, popping up detailed information, or starting related functions.

[0101] The beneficial effects of this method are mainly reflected in three aspects:

[0102] First, by presetting the core, secondary and auxiliary three-layer interaction areas and dynamically adjusting them in combination with the vehicle status and driver's perspective information, the information display efficiency of the in-vehicle augmented reality interface and the accuracy of human-computer interaction are greatly improved. This layered design not only ensures the prominent display of important information, but also takes into account the reasonable presentation of secondary information, effectively balancing information density and readability.

[0103] Secondly, the use of a quadtree-structured regional index table and a hierarchical search strategy significantly improves the computational efficiency of gaze point matching. This method can quickly locate the target interaction area and maintain low computational complexity even in complex augmented reality scenes, which is conducive to achieving real-time response.

[0104] Finally, by introducing a continuous gaze time threshold as an activation condition, the possibility of misoperation is effectively reduced. This design not only takes into account the driver's intention confirmation, but also avoids responding to short, unconscious gazes, thereby improving the reliability and safety of the interaction. Overall, this method has significant advantages in improving the interactive experience and operational safety of the in-vehicle augmented reality system.

[0105] In an optional implementation, the real-time gaze point coordinates are used to calculate the dynamic projection position of the interactive display content, and the dynamic projection position is adjusted in real time as the real-time gaze point coordinates change, so as to ensure that the interactive display content is always within the effective field of view of the driver, including:

[0106] Establishing a driver's visual field model, the driver's visual field model includes a foveal visual area, a secondary foveal visual area and a peripheral visual area, the foveal visual area is an area of ​​2° to the left and right of the gaze point, the secondary foveal visual area is an area of ​​2° to 10° to the left and right of the gaze point, and the peripheral visual area is an area of ​​10° to 30° to the left and right of the gaze point; the foveal visual area is used to display key warning information, the secondary foveal visual area is used to display vehicle status information, and the peripheral visual area is used to display environmental perception information;

[0107] Collecting the driver's head posture information and the vehicle's driving state information, establishing a head feature point tracking model based on the driver's head posture information, and establishing a dynamic compensation model based on the vehicle's driving state information; converting the received real-time gaze point coordinates into three-dimensional space projection coordinates of the vehicle-mounted augmented reality display system through the head feature point tracking model and the dynamic compensation model;

[0108] A Kalman filter prediction model is established based on the historical movement trajectory of the real-time gaze point coordinates, the predicted gaze point coordinates at the next moment are calculated by the Kalman filter prediction model, and the projection position of the interactive display content is pre-calculated according to the predicted gaze point coordinates; when a new real-time gaze point coordinate is obtained, an error is calculated between the new real-time gaze point coordinates and the predicted gaze point coordinates, and the projection position of the interactive display content is corrected based on the error calculation result;

[0109] A buffer area is established around the real-time gaze point coordinates, and the range of the buffer area is positively correlated with the vehicle speed; when the real-time gaze point coordinates move within the buffer area, the projection position of the interactive display content remains unchanged; when the real-time gaze point coordinates move out of the buffer area, the projection position of the interactive display content is recalculated based on the real-time gaze point coordinates.

[0110] First, a model of the driver's field of vision is established. The model includes three parts: the foveal visual area, the parafoveal visual area, and the peripheral visual area. The foveal visual area is defined as an area of ​​2° to the left and right of the gaze point, which is used to display key warning information. The parafoveal visual area is defined as an area of ​​2° to 10° to the left and right of the gaze point, which is used to display vehicle status information. The peripheral visual area is defined as an area of ​​10° to 30° to the left and right of the gaze point, which is used to display environmental perception information. Through this partition design, different types of information can be reasonably allocated to different areas of the driver's field of vision according to the importance and urgency of the information, thereby improving the efficiency and accuracy of information transmission.

[0111] Next, the driver's head posture information and vehicle driving status information are collected. The head posture information can be captured in real time by the in-vehicle camera to obtain data such as the position and angle of the driver's head. The vehicle driving status information includes parameters such as vehicle speed, acceleration, and steering angle, which can be obtained from the vehicle CAN bus. Based on the head posture information, a head feature point tracking model is established, which can track the movement of the driver's head in real time through a feature point matching algorithm. At the same time, a dynamic compensation model is established based on the vehicle driving status information to compensate for the impact of vehicle movement on the position of the displayed content.

[0112] Then, the received real-time gaze point coordinates are converted through the head feature point tracking model and the dynamic compensation model to obtain the three-dimensional space projection coordinates of the in-vehicle augmented reality display system. This step ensures that the position of the displayed content is adjusted in real time with the driver's head movement and vehicle movement, and always remains in the correct field of view.

[0113] In order to improve the system response speed, a Kalman filter prediction model is established based on the historical movement trajectory of the real-time gaze point coordinates. The model predicts the gaze point coordinates at the next moment by analyzing the historical data of the driver's gaze point movement. According to the predicted gaze point coordinates, the system can calculate the projection position of the interactive display content in advance to reduce display delay. When the new real-time gaze point coordinates are obtained, they are compared with the predicted coordinates to calculate the error. According to the size of the error, the projection position of the display content is corrected accordingly to ensure the accuracy of the display position.

[0114] In order to avoid visual interference caused by frequent adjustment of the display position, a buffer area is established around the real-time gaze point coordinates. The range of the buffer area is positively correlated with the vehicle speed. For example, when the vehicle speed is 60km / h, the buffer area radius is 3°; when the vehicle speed is 120km / h, the buffer area radius is 5°. When the real-time gaze point coordinates move within the buffer area, the projection position of the interactive display content remains unchanged, reducing unnecessary frequent adjustments. The projection position of the display content is recalculated only when the gaze point coordinates move out of the buffer area.

[0115] In actual applications, the system can collect the driver's gaze point coordinates and head posture information every 20 milliseconds. Assume that at a certain moment, the driver's gaze point coordinates are (10°, 5°), the head position is (0.2m, 0.1m, 0.5m), and the vehicle speed is 80km / h. The system first determines that the gaze point is located in the secondary fovea area, so the vehicle status information is displayed in this area. Then, through the head feature point tracking model and dynamic compensation model, the gaze point coordinates are converted into the three-dimensional space projection coordinates of the vehicle-mounted AR display system, such as (1.2m, 0.8m, 2.5m).

[0116] Next, the Kalman filter prediction model predicts the next gaze point coordinates as (11°, 5.5°) based on historical data. The system pre-calculates the projection position of the displayed content based on this. 20 milliseconds later, the system obtains the new actual gaze point coordinates (10.5°, 5.2°), compares them with the predicted coordinates, and calculates the error to be (0.5°, 0.3°). Based on this error, the system fine-tunes the projection position of the displayed content.

[0117] At the same time, the system calculates the range of the buffer area. At a speed of 80 km / h, the radius of the buffer area can be set to 4°. Since the new gaze point coordinates (10.5°, 5.2°) are still within the buffer area with a radius of 4° centered at (10°, 5°), the system will not immediately adjust the projection position of the displayed content, but will keep the original position unchanged until the gaze point moves out of the buffer area.

[0118] Through the above steps, the system can adjust the projection position of the interactive display content in real time and accurately, ensuring that the information is always displayed within the driver's effective field of view, while avoiding visual interference caused by frequent adjustments.

[0119] Beneficial effects of this embodiment:

[0120] 1. Improve the pertinence and effectiveness of information display: By establishing a driver's field of vision model, different types of information are allocated to different areas of the field of vision, so that important information is always within the driver's optimal field of vision, improving the efficiency and accuracy of information transmission.

[0121] 2. Enhance the real-time performance and accuracy of the display system: The head feature point tracking model, dynamic compensation model and Kalman filter prediction model are used to achieve real-time adjustment and accurate prediction of the display content position, ensuring the accuracy of the information display position and the timeliness of the system response.

[0122] 3. Optimize user experience and reduce driving burden: By setting a buffer area, unnecessary frequent position adjustments are reduced and visual interference is avoided. At the same time, the vehicle speed factor is taken into consideration, making the display system more intelligent and user-friendly, effectively reducing the driver's visual burden and cognitive load.

[0123] In an optional implementation, the historical movement trajectory of the real-time gaze point coordinates establishes a Kalman filter prediction model, the predicted gaze point coordinates at the next moment are calculated by the Kalman filter prediction model, and the projection position of the interactive display content is pre-calculated according to the predicted gaze point coordinates, including:

[0124] Constructing a gaze point motion state vector, wherein the gaze point motion state vector includes a position coordinate, a velocity component, and an acceleration component of the gaze point; establishing a state transfer matrix based on the gaze point motion state vector, wherein the state transfer matrix describes a mapping relationship between gaze point motion states at adjacent moments;

[0125] Establishing an adaptive noise covariance matrix, the adaptive noise covariance matrix includes a process noise covariance matrix and a measurement noise covariance matrix, wherein the noise variance of the position component in the process noise covariance matrix is ​​0.01 square meters, the noise variance of the velocity component is 0.1 square meters per square second, and the noise variance of the acceleration component is 1.0 square meters per square second; the noise variance of the position measurement in the measurement noise covariance matrix is ​​0.005 square meters;

[0126] Predicting the current state according to the state transfer matrix to obtain a predicted state; calculating a Kalman gain based on the predicted state and the adaptive noise covariance matrix, wherein the Kalman gain is used to determine a weight distribution between a predicted value and a measured value; and updating the predicted state based on the Kalman gain to obtain an updated predicted state;

[0127] A projection transformation matrix is ​​established, wherein the projection transformation matrix includes the focal length, principal point coordinates, position and posture information of the display device; the updated prediction state is mapped to the display plane through the projection transformation matrix to obtain the projection position of the interactive display content; the prediction time span is determined based on the system processing delay and the display refresh cycle, and the prediction time span is 1.2 times the total system delay; and a Bezier curve interpolation algorithm is used to generate a smooth transition trajectory between adjacent prediction points.

[0128] This embodiment provides a method for predicting the projection position of interactive display content based on real-time gaze point coordinates. The method first obtains the real-time gaze point coordinates of the user, then establishes a Kalman filter prediction model based on the historical movement trajectory, calculates the predicted gaze point coordinates at the next moment through the model, and finally pre-calculates the projection position of the interactive display content based on the predicted coordinates.

[0129] The specific implementation steps are as follows:

[0130] First, construct the gaze point motion state vector. This vector contains the position coordinates, velocity components, and acceleration components of the gaze point. For example, for a two-dimensional plane, the state vector can be expressed as (x, y, vx, vy, ax, ay), where x and y are the position coordinates, vx and vy are the velocity components in the x and y directions, and ax and ay are the acceleration components in the x and y directions.

[0131] Next, a state transfer matrix is ​​established based on the gaze point motion state vector. This matrix describes the mapping relationship between the gaze point motion states at adjacent moments. The state transfer matrix reflects the relationship between position, velocity, and acceleration. For example, the position is equal to the position at the previous moment plus the velocity multiplied by the time interval, and the velocity is equal to the velocity at the previous moment plus the acceleration multiplied by the time interval.

[0132] Then, an adaptive noise covariance matrix is ​​established. This matrix includes the process noise covariance matrix and the measurement noise covariance matrix. In the process noise covariance matrix, the noise variance of the position component is set to 0.01 square meters, the noise variance of the velocity component is set to 0.1 square meters per square second, and the noise variance of the acceleration component is set to 1.0 square meters per square second. In the measurement noise covariance matrix, the noise variance of the position measurement is set to 0.005 square meters. These parameters can be adjusted according to the actual application scenario.

[0133] Next, the state transfer matrix is ​​used to predict the current state to obtain the predicted state. During the prediction process, the current state is multiplied by the state transfer matrix to obtain the predicted state at the next moment.

[0134] After that, the Kalman gain is calculated based on the predicted state and the adaptive noise covariance matrix. The Kalman gain is used to determine the weight distribution of the predicted value and the measured value, which determines the degree of trust the system has in the new measured value. The larger the gain value, the more trust the new measured value has; the smaller the gain value, the more trust the model predicts.

[0135] Subsequently, the predicted state is updated based on the Kalman gain to obtain an updated predicted state. During the updating process, the difference between the predicted state and the actual measured value is multiplied by the Kalman gain and then added to the predicted state to obtain an updated predicted state.

[0136] Next, build a projection transformation matrix. This matrix contains the focal length, principal point coordinates, position and attitude information of the display device. For example, for a display device with a focal length of 500mm and principal point coordinates of (320, 240), it is located at the origin of the coordinate system and faces the positive z-axis direction. The corresponding projection transformation matrix can be constructed accordingly.

[0137] Then, the updated prediction state is mapped to the display plane through the projection transformation matrix to obtain the projection position of the interactive display content. This step converts the predicted gaze point coordinates in three-dimensional space into coordinates on the two-dimensional display plane.

[0138] Next, the predicted time span is determined based on the system processing delay and the display refresh cycle. The predicted time span is set to 1.2 times the total system delay. For example, if the system processing delay is 20ms and the display refresh cycle is 16.7ms, the total delay is 36.7ms, and the predicted time span can be set to 44ms.

[0139] Finally, the Bezier curve interpolation algorithm is used to generate a smooth transition trajectory between adjacent prediction points. The Bezier curve can generate a smooth curve based on the control points, making the prediction trajectory more natural and smooth. For example, a cubic Bezier curve can be used, taking two adjacent prediction points as the starting point and the end point, selecting two control points according to their velocity direction, and then generating a smooth transition curve.

[0140] The beneficial effects of this method include:

[0141] 1. Improve prediction accuracy: By constructing the gaze point motion state vector and using the Kalman filter algorithm, the user's gaze point position can be predicted more accurately, thereby improving the positioning accuracy of the interactive display content.

[0142] 2. Reduce system latency: Determine the predicted time span based on system processing latency and display refresh cycle, and calculate the projection position in advance, effectively reducing the system's perceived latency and improving the user experience.

[0143] 3. Optimize display effects: Use Bezier curve interpolation algorithm to generate smooth transition trajectories, making the movement of interactive display content smoother and more natural, reducing screen jitter and jumps, and improving visual comfort.

[0144] In an optional implementation, the moving trajectory of the real-time gaze point coordinates in the activated interactive area, matching the moving trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is recognized, triggering a corresponding control instruction according to the type of the interactive gesture includes:

[0145] Detecting the moving trajectory of the real-time gaze point coordinates in the activated interactive area, collecting a gaze point coordinate sequence, wherein the gaze point coordinate sequence includes a timestamp, a two-dimensional spatial position, and gaze duration information; when the gaze duration exceeds 100 milliseconds and the spatial jitter amplitude is less than 1 degree of viewing angle, it is determined to be a valid gaze point; and a linear interpolation method is used to correct an invalid gaze point;

[0146] Performing median filtering on the gaze point coordinate sequence, wherein the window size of the median filtering is 5 sampling points; simplifying the filtered gaze point coordinate sequence using the Douglas-Peucker algorithm, wherein the distance threshold of the Douglas-Peucker algorithm is 0.5 degree of visual angle; and normalizing the simplified gaze point coordinate sequence to obtain a standardized trajectory;

[0147] Constructing a hierarchical interactive gesture library, the interactive gesture library includes a basic layer gesture and a combined layer gesture, the basic layer gesture includes a single straight line gesture, an arc gesture and a broken line gesture, the combined layer gesture is formed by connecting the basic layer gestures in series; each of the interactive gestures includes a standard trajectory, a time constraint, a space constraint and a context constraint;

[0148] Calculating a characteristic distance matrix between the standardized trajectory and the interactive gestures in the interactive gesture library, wherein the characteristic distance matrix includes point-to-point Euclidean distance, tangent angle difference, and curvature difference; based on the characteristic distance matrix, a dynamic time warping algorithm is used to find an optimal matching path to obtain a regularized distance score;

[0149] Construct a state machine including an idle state, a ready state, a collection state, and a recognition state; when it is detected that the gaze point enters the interactive area and stays there for more than 200 milliseconds, the state machine switches from the idle state to the ready state; when the accumulated trajectory length reaches a preset length threshold or a direction change is detected, the state machine switches from the ready state to the collection state; when the gaze point leaves the interactive area or stays there for more than 2 seconds, the state machine switches from the collection state to the recognition state;

[0150] In the recognition state, the legality of the standardized trajectory is checked; a sliding time window is established, the sliding time window records the matching degree distribution of the most recent 50 recognitions, and the recognition threshold is dynamically adjusted based on the matching degree distribution; the regularized distance score is compared with the recognition threshold to obtain an interactive gesture recognition result;

[0151] Based on the interactive gesture recognition result, control parameters are extracted from a configuration matrix, wherein the rows of the configuration matrix represent interactive gesture types, the columns of the configuration matrix represent system functional modules, and the control parameters include function switch states, adjustment ranges, and execution priorities; and a control instruction sequence is generated according to the control parameters.

[0152] According to the above technical solution, the specific implementation method includes the following steps:

[0153] First, detect the movement trajectory of the real-time gaze point coordinates in the activated interactive area. The user's gaze point coordinate sequence is collected through the eye tracking device. Each coordinate contains a timestamp, a two-dimensional spatial position (x, y), and a gaze duration. For example, the collected raw data may be: [(0, 100, 200, 50), (20, 105, 205, 60), (40,110, 210, 120), ...]. Each tuple represents the timestamp (ms), x coordinate, y coordinate, and gaze duration (ms) in turn.

[0154] Next, the validity of the collected fixation points is judged. When the fixation duration exceeds 100 milliseconds and the spatial jitter amplitude is less than 1 degree of visual angle, it is determined to be a valid fixation point. For example, assuming that 1 degree of visual angle corresponds to 10 pixels, then determine whether the Euclidean distance between two adjacent points is less than 10 pixels. For ineffective fixation points, linear interpolation method is used for correction. Assuming that point A (0, 100, 200) and point C (40, 110, 210) are valid fixation points, and the middle point B (20, 105, 205) is an ineffective point, then correct B to (20, 105, 205).

[0155] Then, median filter the corrected gaze point coordinate sequence to remove outliers. Use a sliding window of 5 sampling points and take the middle value in the window as the filtering result. For example, median filter the sequence [100, 102, 98, 103, 99, 101,97] to get [100, 99, 99, 99, 99].

[0156] Next, the Douglas-Peucker algorithm is used to simplify the filtered sequence and reduce redundant points. The distance threshold is set to 0.5 degrees of visual angle (i.e. 5 pixels). The algorithm starts from the line connecting the first and last points of the sequence, and gradually finds the point farthest from the line. If the distance from the point to the line is greater than the threshold, the point is retained and the recursive processing continues; otherwise, all the intermediate points in the line segment are deleted.

[0157] The simplified sequence is normalized and the trajectory is scaled to a square area of ​​fixed size (e.g. 100x100). This can eliminate the differences between trajectories of different sizes and facilitate subsequent matching. The normalized trajectory can be expressed as: [(0, 0), (20, 25), (50, 60), (80, 75), (100, 100)].

[0158] Construct a hierarchical interactive gesture library. The base layer contains three gestures: single straight line, arc, and broken line. For example, an upward straight line can be represented as [(0, 100), (0, 0)], and a clockwise quarter arc can be represented as [(0, 0), (50,0), (100, 50), (100, 100)]. The combination layer forms complex gestures by connecting the base layer gestures in series. For example, the "Z" shape can be represented as "a straight line to the right + a downward diagonal line to the left + a straight line to the right". Each gesture also contains time constraints (such as completion within 2 seconds), space constraints (such as the starting point is in the upper left corner), and context constraints (such as only valid in the menu interface).

[0159] Calculate the characteristic distance matrix between the standardized trajectory and each gesture in the gesture library. The matrix contains three dimensions: point-to-point Euclidean distance, tangent angle difference, and curvature difference. For example, for two trajectories A and B, the Euclidean distance between their corresponding points, the angle difference between the tangent directions formed by two adjacent points, and the local curvature difference can be calculated. The resulting characteristic distance matrix may be as follows: [

[0161] [5, 10, 0.2],

[0162] [8, 15, 0.3],

[0163] [3, 5, 0.1]

[0164] ].

[0165] Based on the feature distance matrix, the dynamic time warping (DTW) algorithm is used to find the optimal matching path. The DTW algorithm allows nonlinear scaling of trajectories through dynamic programming to find the best alignment between two trajectories. The algorithm outputs a regularized distance score, and the lower the score, the higher the matching degree.

[0166] Construct a state machine with four states: idle, ready, acquisition, and recognition. Initially, it is in the idle state. When the gaze point is detected to enter the interactive area and stay for more than 200ms, it switches to the ready state. In the ready state, when the cumulative trajectory length reaches a preset threshold (such as 50 pixels) or the direction change is detected to be greater than 45 degrees, it switches to the acquisition state. In the acquisition state, the trajectory is continuously recorded until the gaze point leaves the interactive area or stays for more than 2 seconds, and then it switches to the recognition state.

[0167] In the recognition state, firstly, the legality of the standardized trajectory is checked, such as whether the trajectory length is within the valid range, whether there is obvious noise or jitter, etc. Then a sliding time window of 50 samples is established to record the distribution of the most recent matching degree. Based on this distribution, the recognition threshold is dynamically adjusted, such as taking the mean value of the distribution minus one standard deviation as the threshold. The distance score obtained by the DTW algorithm is compared with the threshold. If it is less than the threshold, the recognition is considered successful.

[0168] Finally, the control parameters are extracted from the configuration matrix based on the recognition results. The rows of the configuration matrix represent different gestures, and the columns represent system functional modules. The matrix elements contain the function switch status (0 / 1), the adjustment range (-100%~100%), and the execution priority (1-10). For example, after recognizing the "upward straight line" gesture, searching the corresponding row may get the parameters: [1, 50%, 5], which means turning on a certain function, enhancing by 50%, and the priority is 5. Based on these parameters, a specific control instruction sequence is generated, such as "turn on volume control", "increase volume by 50%", etc.

[0169] The beneficial effects of this program are mainly reflected in three aspects:

[0170] 1. Improved the accuracy and robustness of eye movement interaction. Through multi-level filtering, trajectory simplification and standardization, the noise and jitter effects of eye movement data are effectively reduced. The DTW algorithm is used for flexible matching, which improves the tolerance to gesture deformation. The dynamic threshold mechanism can adapt to different users and scenarios.

[0171] 2. Enhanced flexibility and richness of interaction. The hierarchical gesture library supports basic gestures and complex combination gestures, greatly expanding the types of available interactions. The mapping relationship between gestures and functions can be flexibly defined by configuring the matrix, making it easy to quickly customize and adjust the interaction scheme.

[0172] 3. Optimized the real-time performance and user experience of the system. The state machine design rationally divides the interaction process and reduces false triggering. The sliding time window mechanism can adjust the system sensitivity in time and balance the recognition rate and response speed. Detailed control parameter settings make the system response more refined and intelligent.

[0173] Figure 2 FIG. 1 is a schematic diagram of the structure of a vehicle display control system based on human-computer interaction according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0174] The first unit is used to collect a facial image of the driver through a vehicle-mounted camera, extract the driver's eye movement features using a deep learning model, and calculate the driver's real-time gaze point coordinates based on the eye movement features; match the real-time gaze point coordinates with a preset vehicle-mounted augmented reality display interface interaction area, and trigger the activation state of the interaction area when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold;

[0175] The second unit is used to call the corresponding augmented reality display template according to the interactive area type corresponding to the activation state, and generate interactive display content in combination with the vehicle driving state information, navigation information and environmental perception information; calculate the dynamic projection position of the interactive display content based on the real-time gaze point coordinates, and the dynamic projection position is adjusted in real time with the change of the real-time gaze point coordinates to ensure that the interactive display content is always within the effective field of view of the driver;

[0176] The third unit is used to detect the moving trajectory of the real-time gaze point coordinates in the activated interactive area, and match the moving trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is recognized, a corresponding control instruction is triggered according to the type of the interactive gesture, and the control instruction is used to adjust the display parameters of the interactive display content or perform functional operations; based on the execution result of the control instruction, the display interface is updated, and an interactive feedback prompt sound is issued through the vehicle-mounted speaker.

[0177] According to a third aspect of the embodiments of the present invention,

[0178] An electronic device is provided, comprising:

[0179] processor;

[0180] a memory for storing processor-executable instructions;

[0181] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0182] According to a fourth aspect of the embodiments of the present invention,

[0183] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0184] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle display control method based on human-computer interaction, characterized in that: include: The driver's facial image is collected by a vehicle-mounted camera, the driver's eye movement features are extracted by a deep learning model, and the driver's real-time gaze point coordinates are calculated based on the eye movement features; Matching the real-time gaze point coordinates with a preset in-vehicle augmented reality display interface interaction area, and triggering the activation state of the interaction area when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold; Matching the real-time gaze point coordinates with a preset in-vehicle augmented reality display interface interaction area, and when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold, triggering the activation state of the interaction area includes: A plurality of interaction areas are preset in the vehicle-mounted augmented reality display interface, each of the interaction areas includes a core interaction area, a secondary interaction area and an auxiliary interaction area, the core interaction area is located in the central area of ​​the vehicle-mounted augmented reality display interface and occupies 20% of the area of ​​the vehicle-mounted augmented reality display interface, the secondary interaction area is located in the peripheral area of ​​the core interaction area and occupies 40% of the area of ​​the vehicle-mounted augmented reality display interface, and the auxiliary interaction area is located in the peripheral area of ​​the secondary interaction area and occupies 40% of the area of ​​the vehicle-mounted augmented reality display interface; Collecting vehicle driving state information and driver's perspective information, determining the priority of display content of the interactive area based on the vehicle driving state information, and establishing an adaptive layout model of the interactive area based on the driver's perspective information; dynamically adjusting the position coordinates and display size of the interactive area according to the adaptive layout model of the interactive area; Convert the received real-time gaze point coordinates into standardized gaze point coordinates in the standardized coordinate system of the vehicle-mounted augmented reality display interface; construct a region index table of a quadtree structure, divide the vehicle-mounted augmented reality display interface into grid units of different scales, and establish a mapping relationship between the grid units and the interactive area; A hierarchical search strategy is adopted based on the regional index table to determine the target grid unit where the standardized gaze point coordinates are located at a coarse-grained level, and to determine the target interaction area corresponding to the standardized gaze point coordinates at a fine-grained level; when the standardized gaze point coordinates stay within the range of the target interaction area for a period of time exceeding a preset activation time threshold, the activation state of the target interaction area is triggered; According to the interactive area type corresponding to the activation state, the corresponding augmented reality display template is called, and the interactive display content is generated in combination with the vehicle driving state information, navigation information and environmental perception information; the dynamic projection position of the interactive display content is calculated based on the real-time gaze point coordinates, and the dynamic projection position is adjusted in real time with the change of the real-time gaze point coordinates to ensure that the interactive display content is always within the effective field of view of the driver; Calculating the dynamic projection position of the interactive display content based on the real-time gaze point coordinates, and adjusting the dynamic projection position in real time as the real-time gaze point coordinates change, to ensure that the interactive display content is always within the effective field of view of the driver includes: Establishing a driver's visual field model, the driver's visual field model includes a foveal visual area, a secondary foveal visual area and a peripheral visual area, the foveal visual area is an area of ​​2° to the left and right of the gaze point, the secondary foveal visual area is an area of ​​2° to 10° to the left and right of the gaze point, and the peripheral visual area is an area of ​​10° to 30° to the left and right of the gaze point; the foveal visual area is used to display key warning information, the secondary foveal visual area is used to display vehicle status information, and the peripheral visual area is used to display environmental perception information; Collecting the driver's head posture information and the vehicle's driving state information, establishing a head feature point tracking model based on the driver's head posture information, and establishing a dynamic compensation model based on the vehicle's driving state information; converting the received real-time gaze point coordinates into three-dimensional space projection coordinates of the vehicle-mounted augmented reality display system through the head feature point tracking model and the dynamic compensation model; A Kalman filter prediction model is established based on the historical movement trajectory of the real-time gaze point coordinates, the predicted gaze point coordinates at the next moment are calculated by the Kalman filter prediction model, and the projection position of the interactive display content is pre-calculated according to the predicted gaze point coordinates; when a new real-time gaze point coordinate is obtained, an error is calculated between the new real-time gaze point coordinates and the predicted gaze point coordinates, and the projection position of the interactive display content is corrected based on the error calculation result; A buffer area is established around the real-time gaze point coordinates, wherein the range of the buffer area is positively correlated with the vehicle speed; when the real-time gaze point coordinates move within the buffer area, the projection position of the interactive display content remains unchanged; when the real-time gaze point coordinates move out of the buffer area, the projection position of the interactive display content is recalculated based on the real-time gaze point coordinates; Detecting the moving trajectory of the real-time gaze point coordinates in the activated interactive area, and matching the moving trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is identified, triggering a corresponding control instruction according to the type of the interactive gesture, the control instruction is used to adjust the display parameters of the interactive display content or perform a functional operation; updating the display interface based on the execution result of the control instruction, and issuing an interactive feedback prompt sound through the vehicle-mounted speaker; Detecting the movement trajectory of the real-time gaze point coordinates in the activated interactive area, and matching the movement trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is recognized, triggering a corresponding control instruction according to the type of the interactive gesture includes: Detecting the moving trajectory of the real-time gaze point coordinates in the activated interactive area, collecting a gaze point coordinate sequence, wherein the gaze point coordinate sequence includes a timestamp, a two-dimensional spatial position, and gaze duration information; when the gaze duration exceeds 100 milliseconds and the spatial jitter amplitude is less than 1 degree of viewing angle, it is determined to be a valid gaze point; and a linear interpolation method is used to correct an invalid gaze point; Performing median filtering on the gaze point coordinate sequence, wherein the window size of the median filtering is 5 sampling points; simplifying the filtered gaze point coordinate sequence using the Douglas-Peucker algorithm, wherein the distance threshold of the Douglas-Peucker algorithm is 0.5 degree of visual angle; and normalizing the simplified gaze point coordinate sequence to obtain a standardized trajectory; Constructing a hierarchical interactive gesture library, the interactive gesture library includes a basic layer gesture and a combined layer gesture, the basic layer gesture includes a single straight line gesture, an arc gesture and a broken line gesture, the combined layer gesture is formed by connecting the basic layer gestures in series; each of the interactive gestures includes a standard trajectory, a time constraint, a space constraint and a context constraint; Calculating a characteristic distance matrix between the standardized trajectory and the interactive gestures in the interactive gesture library, wherein the characteristic distance matrix includes point-to-point Euclidean distance, tangent angle difference, and curvature difference; based on the characteristic distance matrix, a dynamic time warping algorithm is used to find an optimal matching path to obtain a regularized distance score; Construct a state machine including an idle state, a ready state, a collection state, and a recognition state; when it is detected that the gaze point enters the interactive area and stays there for more than 200 milliseconds, the state machine switches from the idle state to the ready state; when the accumulated trajectory length reaches a preset length threshold or a direction change is detected, the state machine switches from the ready state to the collection state; when the gaze point leaves the interactive area or stays there for more than 2 seconds, the state machine switches from the collection state to the recognition state; In the recognition state, the legality of the standardized trajectory is checked; a sliding time window is established, the sliding time window records the matching degree distribution of the most recent 50 recognitions, and the recognition threshold is dynamically adjusted based on the matching degree distribution; the regularized distance score is compared with the recognition threshold to obtain an interactive gesture recognition result; Based on the interactive gesture recognition result, control parameters are extracted from a configuration matrix, wherein the rows of the configuration matrix represent interactive gesture types, the columns of the configuration matrix represent system functional modules, and the control parameters include function switch states, adjustment ranges, and execution priorities; and a control instruction sequence is generated according to the control parameters.

2. The method according to claim 1, characterized in that The driver's facial image is collected by the vehicle camera, the driver's eye movement features are extracted by the deep learning model, and the driver's real-time gaze point coordinates are calculated based on the eye movement features, including: The driver's facial image is collected by a vehicle-mounted binocular infrared camera, the image resolution of the vehicle-mounted binocular infrared camera is 1920×1080 pixels, and the acquisition frame rate of the vehicle-mounted binocular infrared camera is 60fps; the vehicle-mounted near-infrared fill light is turned on, and the luminous intensity of the vehicle-mounted near-infrared fill light is adaptively adjusted according to the ambient light intensity, and the wavelength range of the vehicle-mounted near-infrared fill light is 850-940nm; the exposure parameters of the vehicle-mounted binocular infrared camera are adaptively adjusted according to the luminous intensity of the vehicle-mounted near-infrared fill light to generate a stable driver's facial image; Inputting the driver's facial image into a preset adaptive enhancement network, wherein the adaptive enhancement network includes a brightness enhancement sub-network and a contrast enhancement sub-network, wherein the brightness enhancement sub-network uses a multi-scale convolution structure to extract image brightness and darkness features, and the contrast enhancement sub-network uses a residual connection structure to extract image texture features; fusing the output features of the brightness enhancement sub-network and the output features of the contrast enhancement sub-network to generate an enhanced driver's facial image; The enhanced driver's facial image is input into a preset eye movement feature extraction network, which includes a face detection module, a key point positioning module and a feature extraction module. The face detection module detects the face area in the enhanced driver's facial image, the key point positioning module locates the key points of the eye area in the face area, and the feature extraction module extracts the eye movement features of the eye area; a driver's gaze point coordinate mapping model is established based on the eye movement features, and the driver's real-time gaze point coordinates are calculated according to the driver's gaze point coordinate mapping model.

3. The method according to claim 1, characterized in that Establishing a Kalman filter prediction model based on the historical movement trajectory of the real-time gaze point coordinates, calculating the predicted gaze point coordinates at the next moment through the Kalman filter prediction model, and pre-calculating the projection position of the interactive display content according to the predicted gaze point coordinates includes: Constructing a gaze point motion state vector, wherein the gaze point motion state vector includes a position coordinate, a velocity component, and an acceleration component of the gaze point; establishing a state transfer matrix based on the gaze point motion state vector, wherein the state transfer matrix describes a mapping relationship between gaze point motion states at adjacent moments; Establishing an adaptive noise covariance matrix, the adaptive noise covariance matrix includes a process noise covariance matrix and a measurement noise covariance matrix, wherein the noise variance of the position component in the process noise covariance matrix is ​​0.01 square meters, the noise variance of the velocity component is 0.1 square meters per square second, and the noise variance of the acceleration component is 1.0 square meters per square second; the noise variance of the position measurement in the measurement noise covariance matrix is ​​0.005 square meters; Predicting the current state according to the state transfer matrix to obtain a predicted state; calculating a Kalman gain based on the predicted state and the adaptive noise covariance matrix, wherein the Kalman gain is used to determine a weight distribution between a predicted value and a measured value; and updating the predicted state based on the Kalman gain to obtain an updated predicted state; A projection transformation matrix is ​​established, wherein the projection transformation matrix includes the focal length, principal point coordinates, position and posture information of the display device; the updated prediction state is mapped to the display plane through the projection transformation matrix to obtain the projection position of the interactive display content; the prediction time span is determined based on the system processing delay and the display refresh cycle, and the prediction time span is 1.2 times the total system delay; and a Bezier curve interpolation algorithm is used to generate a smooth transition trajectory between adjacent prediction points.

4. A vehicle-mounted display control system based on human-computer interaction, used to implement the method according to any one of claims 1 to 3, characterized in that: include: The first unit is used to collect the driver's facial image through the vehicle-mounted camera, extract the driver's eye movement features using the deep learning model, and calculate the driver's real-time gaze point coordinates based on the eye movement features; Matching the real-time gaze point coordinates with a preset in-vehicle augmented reality display interface interaction area, and triggering the activation state of the interaction area when the real-time gaze point coordinates stay in the interaction area for a period of time exceeding a preset threshold; The second unit is used to call the corresponding augmented reality display template according to the interactive area type corresponding to the activation state, and generate interactive display content in combination with the vehicle driving state information, navigation information and environmental perception information; calculate the dynamic projection position of the interactive display content based on the real-time gaze point coordinates, and the dynamic projection position is adjusted in real time with the change of the real-time gaze point coordinates to ensure that the interactive display content is always within the effective field of view of the driver; The third unit is used to detect the moving trajectory of the real-time gaze point coordinates in the activated interactive area, and match the moving trajectory with a pre-stored interactive gesture library; when a valid interactive gesture is recognized, a corresponding control instruction is triggered according to the type of the interactive gesture, and the control instruction is used to adjust the display parameters of the interactive display content or perform functional operations; based on the execution result of the control instruction, the display interface is updated, and an interactive feedback prompt sound is issued through the vehicle-mounted speaker.

5. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 3.

6. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images