Vehicle-mounted head-up display visual focus prediction method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a vehicle-mounted head-up display visual focus prediction method and device, equipment and a medium, and the method comprises the steps: collecting an activity video of eyeballs of a user; determining a historical movement track of the sight focus based on the activity video, and calculating the delay time of the vehicle-mounted head-up display system according to the historical movement track; predicting a screen projection position corresponding to the sight line focus of the next frame according to the delay time to obtain predicted coordinate information; and controlling the vehicle-mounted head-up display system to execute projection operation according to the predicted coordinate information. The problem that the screen projection position is not matched with the actual visual focus position of the driver due to delay in the AR-HUD system in the existing eyeball tracking technology is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for predicting a visual focus of a vehicle-mounted head-up display. Background Art

[0002] With the continuous development of automobile technology, augmented reality head up display (AR-HUD) technology has gradually become an important part of intelligent driving systems. AR-HUD can project navigation, vehicle status and other information in the form of virtual images in front of the driver's line of sight, integrating with the real road to improve driving safety and convenience. For intelligent display control, the existing AR-HUD system combines eye tracking technology to dynamically adjust the display content according to the driver's eye position to ensure that the information is accurately and timely presented in the visual focus.

[0003] Although eye tracking technology has great potential in AR-HUD systems, there is a delay problem in actual application. Due to the lag in data processing and transmission, there is often a deviation between the projection screen position and the driver's actual focus position, which affects the accurate transmission of information and may interfere with driving or cause safety hazards. Especially when driving at high speeds, if the system cannot update the display position in real time, it will affect driving decisions. Therefore, solving the delay problem between eye tracking and AR-HUD projection has become an urgent technical challenge. Summary of the invention

[0004] In view of this, embodiments of the present invention provide a method, device, equipment and medium for predicting the visual focus of a vehicle-mounted head-up display, so as to solve the problem that the projection position of the screen does not match the actual visual focus position of the driver due to delay in the existing eye tracking technology in the AR-HUD system.

[0005] In a first aspect, an embodiment of the present invention provides a method for predicting a visual focus of a vehicle-mounted head-up display, the method comprising:

[0006] Collect videos of users’ eye movements;

[0007] Determine a historical movement trajectory of a sight focus based on the active video, and calculate a delay time of a vehicle head-up display system according to the historical movement trajectory;

[0008] Predicting the projection screen position corresponding to the sight focus of the next frame according to the delay time to obtain predicted coordinate information;

[0009] The vehicle-mounted head-up display system is controlled to perform a projection operation according to the predicted coordinate information.

[0010] Furthermore, the determining of the historical movement trajectory of the sight focus based on the activity video includes:

[0011] Extracting eyeball images of multiple time frames in the active video;

[0012] Identifying the position coordinates of the sight focus in the eyeball image;

[0013] A historical movement trajectory of the sight focus is generated according to the position coordinates.

[0014] Furthermore, the generating of the historical movement trajectory of the sight focus according to the position coordinates includes:

[0015] Acquire a mapping relationship between a collection area of ​​the eyeball image and a projection area of ​​the vehicle-mounted head-up display system;

[0016] Determine the mapping coordinates corresponding to the position coordinates in the projection area according to the mapping relationship;

[0017] The mapping coordinates corresponding to the eyeball images of each time frame are arranged to obtain the historical movement trajectory of the sight focus.

[0018] Furthermore, the calculating of the delay time of the vehicle head-up display system according to the historical movement trajectory includes:

[0019] Obtaining multiple historical projection instructions of the vehicle head-up display system;

[0020] Traversing the projection coordinates of the historical projection instructions, matching the mapping coordinates in the historical movement trajectory with the projection coordinates, and obtaining the projection coordinates of the hit target;

[0021] The delay time of the vehicle-mounted head-up display system is calculated according to the mapping coordinates and the target projection coordinates.

[0022] Further, the calculating the delay time of the vehicle head-up display system according to the mapping coordinates and the target projection coordinates includes:

[0023] Obtaining the execution time corresponding to the target projection coordinates and the timestamp corresponding to the mapping coordinates;

[0024] The delay time of the vehicle head-up display system is calculated according to the execution time and the timestamp.

[0025] Further, predicting the projection position corresponding to the sight focus of the next frame according to the delay time to obtain predicted coordinate information includes:

[0026] Get the total number of frames of the active video;

[0027] Calculate the delayed frame number of the sight focus according to the total frame number and the delay time;

[0028] The current position coordinates of the sight focus are obtained, and the projection position corresponding to the sight focus of the next frame is predicted according to the delayed frame number and the current position coordinates to obtain predicted coordinate information.

[0029] Further, after controlling the vehicle-mounted head-up display system to perform a projection operation according to the predicted coordinate information, the method further includes:

[0030] Collecting actual images of the user's eyes;

[0031] Determine the actual coordinate information of the current sight focus according to the actual image, and calculate the error value between the actual coordinate information and the predicted coordinate information;

[0032] It is determined whether the error value exceeds a preset threshold value. If the error value exceeds the preset threshold value, a correction mechanism is triggered to correct the delay time corresponding to the vehicle-mounted head-up display system according to the error value to obtain the corrected delay time.

[0033] Furthermore, the method further comprises:

[0034] Acquire the inertial sensor data of the target vehicle;

[0035] Calculating the relative displacement of the user's eyeball relative to the vehicle-mounted head-up display system based on the inertial sensor data;

[0036] The predicted coordinate information is adjusted according to the relative displacement to obtain adjusted predicted coordinate information.

[0037] Further, the adjusting the predicted coordinate information according to the relative displacement to obtain adjusted predicted coordinate information includes:

[0038] Determine the offset direction and offset amount of the sight focus according to the relative displacement;

[0039] Adjusting the predicted coordinate information based on the offset direction and the offset amount to obtain initial coordinate information;

[0040] Detecting whether the initial coordinate information is in the projection area of ​​the vehicle-mounted head-up display system;

[0041] If it is in the projection area, the initial coordinate information is used as the adjusted predicted coordinate information; or, if it is not in the projection area, the boundary processing mechanism is triggered and the preset coordinate information associated with the offset direction is used as the adjusted predicted coordinate information.

[0042] In a second aspect, an embodiment of the present invention provides a device for predicting a visual focus of a vehicle-mounted head-up display, the device comprising:

[0043] A collection module is used to collect videos of user eye movements;

[0044] A determination module, configured to determine a historical movement trajectory of a sight focus based on the active video, and calculate a delay time of an in-vehicle head-up display system according to the historical movement trajectory;

[0045] A prediction module, used to predict the projection position corresponding to the sight focus of the next frame according to the delay time to obtain predicted coordinate information;

[0046] A control module is used to control the vehicle-mounted head-up display system to perform a projection operation according to the predicted coordinate information.

[0047] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0049] In the fifth aspect, an embodiment of the present invention provides a vehicle, comprising: a visual focus prediction system and a controller; wherein the controller stores instructions for controlling the operation of the visual focus prediction system, and the visual focus prediction system is used to execute the method of the above-mentioned first aspect or any corresponding embodiment thereof according to the instructions of the controller.

[0050] The method provided in the embodiment of the present application has the following beneficial effects:

[0051] The method provided in the embodiment of the present application collects user eye movement videos through image acquisition equipment in the cockpit, and can obtain raw data of eye movement, providing a basis for subsequent analysis. Based on this, the historical movement trajectory of the line of sight focus is determined and the delay time of the vehicle-mounted head-up display system is calculated. The system delay can be accurately quantified, and then the projection position of the line of sight focus of the next frame is predicted according to the delay time and the projection operation is controlled. The problem of mismatch between the projection position and the actual visual focus position of the driver caused by delay in the existing eye tracking technology in the AR-HUD system is effectively solved, ensuring that information is accurately and timely presented at the driver's visual focus, thereby improving driving safety and convenience.

[0052] The method provided in the embodiment of the present application extracts eyeball images of multiple time frames in the active video, identifies the position coordinates of the eyeball focus in the eyeball image, obtains the mapping relationship between the acquisition area and the projection area and determines the mapping coordinates, and finally arranges the historical movement trajectory. This series of operations makes the determination of the historical movement trajectory of the sight focus more accurate and reliable. Each small step is like polishing the parts of a precision instrument, which together ensures the accurate restoration of the eyeball movement trajectory, and provides more accurate data support for the subsequent calculation of the delay time and the prediction of the projection position. In the step of calculating the delay time of the vehicle-mounted head-up display system, by obtaining multiple historical projection instructions, traversing the projection coordinates and matching the mapping coordinates in the historical movement trajectory to obtain the target projection coordinates, and then calculating the delay time according to the execution time and the timestamp corresponding to the mapping coordinates. This method comprehensively considers the correlation information between the system projection operation and the actual movement of the eyeball, makes the calculation of the delay time more scientific and reasonable, avoids the error of single factor judgment, and thus provides a more accurate time difference basis for the subsequent prediction of the projection position. In the step of predicting the projection position corresponding to the next frame of sight focus, the total number of frames of the active video is obtained, the number of delayed frames of the sight focus is calculated based on the total number of frames and the delay time, and then the prediction is made in combination with the current sight focus position coordinates. This makes the prediction process more detailed and targeted, fully considers the dynamic characteristics and delay factors of the video, further improves the accuracy of the predicted coordinate information, and makes the projection position more consistent with the changes in the driver's visual focus.

[0053] The method provided in the embodiment of the present application collects the actual image of the user's eyeball after the projection operation, determines the actual coordinate information and calculates the error value with the predicted coordinate information. When the error value exceeds the preset threshold, the correction mechanism is triggered to correct the delay time, and the accuracy of the prediction can be monitored in real time. When a large deviation occurs, the system parameters are adjusted in time to ensure that the system is always in a relatively accurate working state, and the stability and reliability of the entire prediction method in a complex environment are further improved. By obtaining the inertial sensor data of the target vehicle, the relative displacement of the user's eyeball relative to the vehicle-mounted head-up display system is calculated, and the predicted coordinate information is adjusted according to the relative displacement. This takes into account the dynamic changes in the vehicle during driving, such as the impact of vehicle acceleration, turning, etc. on the driver's eyeball position. Through this adjustment, the predicted coordinate information can be made more in line with the actual situation, effectively respond to the additional changes in the eyeball position caused by vehicle movement, ensure that the display content is always accurately projected on the driver's visual focus, and improve driving safety. Especially in complex road conditions, this adjustment mechanism can better adapt to environmental changes and reduce the interference of display deviations on driving decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 is a flow chart of a method for predicting a visual focus of a vehicle head-up display according to an embodiment of the present invention;

[0056] Figure 2 A schematic diagram of an actual sight focus and a predicted sight focus moving trajectory in a vehicle-mounted head-up display according to an embodiment of the present invention;

[0057] Figure 3 is a flow chart of another method for predicting a visual focus of a vehicle-mounted head-up display according to an embodiment of the present invention;

[0058] Figure 4 is a structural block diagram of a device for predicting a visual focus of a vehicle-mounted head-up display according to an embodiment of the present invention;

[0059] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0061] According to an embodiment of the present invention, a method, device, apparatus and medium for predicting the visual focus of a vehicle-mounted head-up display are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0062] In this embodiment, a method for predicting the visual focus of a vehicle head-up display is provided. Figure 1 is a flow chart of a method for predicting a visual focus of a vehicle head-up display according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0063] Step S11, collecting video of the user's eye movements.

[0064] In the embodiment of the present application, the visual focus prediction and delay problem solving need to be based on the user's eye movement information, and this information comes from the active video captured by the image acquisition device (such as a camera, etc.) in the cockpit. The position and parameters of the device are specially designed to accurately capture the movement of the driver's eyeballs. The captured active video is a continuous sequence of image frames, which contains various state information of the user's eyeballs at different times, such as the position and movement direction of the eyeballs.

[0065] Step S12, determining the historical movement trajectory of the visual focus based on the active video, and calculating the delay time of the vehicle-mounted head-up display system according to the historical movement trajectory.

[0066] In an embodiment of the present application, determining the historical movement trajectory of the sight focus based on the activity video includes the following steps A1-A3:

[0067] Step A1, extracting eyeball images of multiple time frames in the active video.

[0068] Specifically, image frames are extracted from the video of the user's eye activity according to a specific time interval or a specific algorithm. The time interval can be determined based on the frame rate of the video, the approximate speed of the eye movement, and the accuracy requirements of subsequent processing. For example, if the video frame rate is high and the eye movement is relatively slow, the time interval can be appropriately increased to reduce the amount of data while ensuring that key eye movement information is not lost. The extracted image frame should clearly contain the eye part, and the image needs to be pre-processed, such as denoising, contrast enhancement, etc., so that the eye features can be more accurately identified later.

[0069] Step A2, identifying the position coordinates of the sight focus in the eyeball image.

[0070] Specifically, image processing and computer vision technology are used to identify the position of the sight focus in the eye image. This includes: detecting feature points such as the pupil center and corneal reflection point of the eye, and calculating the coordinates of the sight focus through a specific geometric model or algorithm. For example, based on the relative position relationship between the pupil center and the corneal reflection point, combined with a known eye model (such as a spherical model, etc.), the sight direction can be determined by triangulation and other methods, and then the coordinates of the sight focus in the image can be determined. At the same time, in order to improve accuracy, it is necessary to calibrate and optimize the algorithm for images taken under different lighting conditions and at different angles.

[0071] Step A3, generating a historical movement trajectory of the sight focus according to the position coordinates.

[0072] It should be noted that since the line of sight focus coordinate position collected by the image acquisition device in the vehicle's cockpit belongs to a different coordinate system from the projection position of the vehicle's head-up display system, there is a natural difference between the two. This difference will affect the subsequent effective use of the driver's line of sight focus information, so it is necessary to convert the trajectory coordinates.

[0073] In the embodiment of the present application, step A3 includes the following steps A31-A33:

[0074] Step A31, obtaining a mapping relationship between the acquisition area of ​​the eyeball image and the projection area of ​​the vehicle-mounted head-up display system.

[0075] Specifically, during the system design and initialization phase, it is necessary to establish a mapping relationship between the acquisition area (i.e., the camera shooting range) and the projection area (AR-HUD display area). For example, when the vehicle is stationary, place some markers at known standard positions, observe the coordinates of these markers in the acquisition area through the camera, and record the corresponding coordinates of these markers in the projection area. Then use mathematical transformations (such as affine transformations, perspective transformations, etc.) to establish a mapping model between the two. This mapping relationship can be a two-dimensional coordinate transformation matrix or a set of mathematical formulas used to convert the coordinates in the acquisition area into coordinates in the projection area.

[0076] Step A32, determining the mapping coordinates corresponding to the position coordinates in the projection area according to the mapping relationship.

[0077] Specifically, after obtaining the mapping relationship, the coordinates of the identified sight focus in the acquisition area are substituted into the mapping relationship for calculation. For example, if the mapping relationship is a two-dimensional coordinate transformation matrix, the coordinate vector of the sight focus is multiplied by the matrix to obtain the corresponding coordinates in the projection area. This process actually converts the sight focus position in the eyeball image to the coordinate system of the AR-HUD projection area, so that the data can be analyzed and processed in the same coordinate system later.

[0078] Step A33, arranging the mapping coordinates corresponding to the eyeball images of each time frame to obtain the historical movement trajectory of the sight focus.

[0079] Specifically, the mapping coordinates of the sight focus corresponding to each time frame in the projection area are arranged in chronological order to form a coordinate sequence, which can be stored in a data structure, such as an array or a linked list. This coordinate sequence represents the movement path of the sight focus over a period of time. By analyzing this historical movement trajectory, the changing trend of the driver's sight focus can be understood, providing data support for subsequent calculations of delay time, prediction of projection screen positions, and other operations. At the same time, some preprocessing, such as smoothing, can be performed on this trajectory to reduce the impact of noise on the analysis results.

[0080] In the embodiment of the present application, the delay time of the vehicle head-up display system is calculated according to the historical movement trajectory, including the following steps B1-B3:

[0081] Step B1, obtaining multiple historical projection instructions of the vehicle head-up display system.

[0082] Specifically, the vehicle head-up display system generates a series of projection instructions during operation. These instructions include the projection content and the coordinates of each projection content in the projection area and other related information. The historical projection instructions are the collection of all projection instructions issued by the system in the past period of time. These instructions also include but are not limited to the projection time, projection content type, etc., but the focus here is on the projection coordinates.

[0083] Step B2, traverse the projection coordinates of the historical projection instructions, match the mapping coordinates in the historical movement trajectory with the projection coordinates, and obtain the projection coordinates of the hit target.

[0084] Specifically, the projection coordinates are taken in sequence from the structure storing the historical projection instructions. At the same time, the mapping coordinates in the historical movement trajectory of the sight focus are judged to be consistent using a matching algorithm designed based on coordinate similarity (such as setting a distance threshold, and determining that the two are matched when the Euclidean distance is less than the threshold). In this way, the projection instruction related to the historical position of the sight focus is found, and its projection coordinates are the target projection coordinates. In the process, the coordinate data can be pre-processed such as normalization to improve the matching accuracy.

[0085] Step B3, calculating the delay time of the vehicle head-up display system according to the mapping coordinates and the target projection coordinates.

[0086] It should be noted that the delay time directly affects the synchronization between the focus of vision and the projection content, which in turn affects the driver's information acquisition efficiency and driving safety. By carefully analyzing the difference between the execution time of the target projection coordinates and the timestamp of the mapping coordinates, we can more accurately understand the actual situation of the system response and provide data support for subsequent optimization.

[0087] In the embodiment of the present application, step B3 includes the following steps B31-B32:

[0088] Step B31, obtaining the execution time corresponding to the target projection coordinates and the timestamp corresponding to the mapping coordinates.

[0089] Specifically, each historical projection instruction corresponding to the target projection coordinate has an execution time, which records the moment when the instruction is executed in the vehicle head-up display system (that is, the information is projected in the projection area). For the mapping coordinates in the historical movement trajectory, since they are arranged in chronological order, each mapping coordinate has a corresponding timestamp, which indicates the moment when the line of sight is at that position. By obtaining these two sets of time information, the time difference between the two can be further analyzed to prepare for the calculation of the delay time.

[0090] Step B32, calculating the delay time of the vehicle head-up display system according to the execution time and the timestamp.

[0091] Specifically, the difference between the execution time and the timestamp of each set of matched target projection coordinates and mapping coordinates is calculated. This difference is the system delay time during the projection process. To estimate the delay time more accurately, the delay time of all matching coordinate pairs can be statistically analyzed, such as the mean, median, etc. If the delay time fluctuates greatly, further analysis of the cause is required, which may be caused by changes in system load or other factors (such as unstable network transmission, etc.). In this way, a stable and accurate delay time value can be obtained, providing a basis for subsequent operations.

[0092] Step S13, predicting the projection position corresponding to the line of sight focus of the next frame according to the delay time to obtain predicted coordinate information.

[0093] In the embodiment of the present application, step S13 includes the following steps C1-C3:

[0094] Step C1, obtaining the total number of frames of the active video.

[0095] Specifically, the active video is composed of continuous image frames. The total number of frames is the number of image frames, which can be obtained by reading video attributes or related video processing libraries. The total number of frames is included in the metadata of common video encoding formats. The total number of frames is critical to calculating the number of frames of line of sight focus delay and is an important benchmark for time-related calculations.

[0096] Step C2, calculating the delayed frame number of the sight focus according to the total frame number and the delay time.

[0097] Specifically, the delay time and the total number of frames are known. Since the image frames correspond to time points, the number of delayed frames can be calculated based on the frame rate (frame rate = 1000 / frame time, assuming that the frame time is fixed). For example, when the total number of frames is N, the frame rate is F frames / second (1000 / F milliseconds per frame), and the delay time is L milliseconds, the number of delayed frames = L / (1000 / F). Pay attention to the unit and data type when calculating. The number of delayed frames is the key to predicting the projection position of the line of sight focus, and reflects the number of frames that need to be predicted in advance for the system delay. The delay time can be dynamically calibrated, and the number of delayed frames determined by the total number of frames can be used in linear regression models to predict the future line of sight focus position. For example, when the delay time is 500 milliseconds and the number of video frames is 20 frames (50 milliseconds per frame), the number of delayed frames is 10 frames, that is, the position 10 frames after the current eye position is predicted.

[0098] Step C3, obtain the current position coordinates of the line of sight focus, and predict the projection position corresponding to the line of sight focus of the next frame according to the number of delayed frames and the current position coordinates to obtain predicted coordinate information.

[0099] Specifically, firstly, by detecting eye-related feature points such as pupil center and corneal reflection point in the currently processed image frame, a specific geometric model or algorithm such as triangulation is used to obtain the coordinates of the current sight focus. Since eye movement is usually continuous and regular in a short period of time, the future position can be predicted based on the change trend of the eye position coordinates in the previous frames. For example, if the sight focus coordinates in the previous frames change linearly, linear interpolation or linear regression methods can be used. The linear regression model formula is:

[0100] x=β0+β1x1+β2x2+…+βnxn+ε

[0101] Among them, x is the dependent variable, that is, the predicted coordinates of the next visual focus in the direction of the projection area (that is, the prediction result); x1, x2, ..., xn are independent variables, that is, the x coordinates of the eyeball at different times before; β0 is the intercept; β1, β2, ..., βn are the slopes; ε is the error term.

[0102] Substitute the current position coordinates and the coordinates of the previous frames within the time range determined by the number of delayed frames as input data into the model to calculate the predicted value of the coordinates. For the y coordinate, the same method is used to predict, and finally we get something like (x n+m +ε x ,y n+m +ε y ) in the form of predicted coordinate information. This process needs to take into account the complexity of eye movement and interference factors (such as the driver's sudden head turn, vehicle bumps, etc.) to improve prediction accuracy, thereby reducing the deviation between the projection position and the driver's actual line of sight.

[0103] It should be noted that the method of removing the complexity and interference factors of eye movement includes: establishing a dynamic threshold model, dynamically adjusting the threshold range according to the vehicle movement state and environmental information, and filtering the coordinate fluctuations caused by small tremors or short-term irregular movements of the eye. For example, the threshold range is increased when the vehicle is bumpy, and the threshold range is reduced when the vehicle is driving smoothly. The weighted moving average algorithm is used to assign different weights to the eye coordinate data at different times. Under normal circumstances, the weight of recent data is high. When interference (such as sudden acceleration and deceleration of the vehicle) is detected, the weight of recent data is reduced, and the weight of stable state data is increased to smooth the influence of coordinate mutation. Combined with head posture estimation, the driver's head posture information is obtained using the posture sensor at the headrest of the seat. When the head posture changes significantly, the line of sight shift is judged and the prediction model is adjusted accordingly. For example, if the head turns left, the attention and weight of the eye coordinate change trend in the left area of ​​the image frame are increased. Real-time outlier detection and correction, real-time monitoring of eye coordinate data outliers when processing data, by comparing with historical and adjacent data and combining the vehicle status and the normal range of eye movement, the detected outliers are corrected according to the surrounding normal data using interpolation (linear or spline interpolation).

[0104] Step S14, controlling the vehicle-mounted head-up display system to perform a projection operation according to the predicted coordinate information.

[0105] In an embodiment of the present application, after obtaining the predicted coordinate information, first verify its validity to ensure that it is within the effective projection area of ​​the vehicle head-up display system. If it is invalid, error handling is triggered. Subsequently, the visual and spatial properties of the projected content are adjusted according to the coordinates to ensure alignment with the driver's line of sight. The projected content is bound to the predicted coordinates for subsequent updates or adjustments. In addition, it can be integrated with other on-board modules (for example, vehicle navigation systems, driving assistance systems, and other modules), using more vehicle and driver information to optimize coordinate calculations and parameter adjustments, and improve the level of intelligence and user experience.

[0106] In the embodiment of the present application, after controlling the vehicle head-up display system to perform the projection operation according to the predicted coordinate information, the method further includes the following steps D1-D3:

[0107] Step D1, capturing the actual image of the user's eyeball.

[0108] Specifically, the camera in the cockpit is used to continue to collect images. This process is similar to the previous collection of eye movement videos, but the focus at this time is to obtain the image at the current moment to determine the actual focus position of the line of sight. In order to ensure the quality of the collected images, it may be necessary to adjust or optimize the parameters of the collection equipment in real time, such as automatically adjusting the exposure time, contrast and other parameters according to the changes in the light in the car to ensure that the characteristics of the eyeball can be clearly captured. At the same time, in order to reduce the amount of data processing and increase the processing speed, only the images of key areas can be collected or the collected images can be cropped to retain only the part containing the eyeball.

[0109] Step D2, determining the actual coordinate information of the current sight focus according to the actual image, and calculating the error value between the actual coordinate information and the predicted coordinate information.

[0110] Specifically, whether the moving tracks overlap is determined by comparing the error value between the actual coordinate information and the predicted coordinate information. Figure 2 As shown, the moving trajectory of the actual sight focus is basically consistent with the predicted moving trajectory, which indicates that the current prediction method has high accuracy. The process is as follows: using image processing and computer vision technology, the pupil center of the eyeball, corneal reflection point and other feature points in the actual image are detected, and the coordinates of the current sight focus in the image are calculated through a specific geometric model or algorithm (such as triangulation, etc.). Then, through the mapping relationship between the acquisition area and the projection area, it is converted into the coordinates of the projection area to obtain the actual coordinate information. Compare the actual coordinate information of the current sight focus with the predicted coordinate information. The error value can be calculated in a variety of ways. For example, when the coordinates are two-dimensional, the Euclidean distance formula can be used to calculate the straight-line distance between the two coordinate points on the two-dimensional plane to quantify the degree of deviation between the actual position and the predicted position.

[0111] Step D3, determining whether the error value exceeds a preset threshold value. If it exceeds the preset threshold value, triggering a correction mechanism, correcting the delay time corresponding to the vehicle head-up display system according to the error value, and obtaining a corrected delay time.

[0112] Specifically, the preset threshold is determined during the system design and debugging phase after comprehensively considering factors such as system performance, user experience, and actual operating environment. It is a standard value used to measure whether the prediction error is acceptable. Compare the error value between the actual and predicted coordinate information with the preset threshold. If the error value is less than or equal to the threshold, it indicates that the predicted coordinate information is relatively accurate and can be operated according to the current delay time; if the error value exceeds the preset threshold, it means that the delay time setting is unreasonable, and the predicted and actual sight focus positions do not match in time. It is necessary to trigger a correction mechanism to adjust the delay time of the vehicle head-up display system, such as using algorithms such as proportional relationship correction method. If the predicted position lags behind the actual position, the delay time can be reduced; if the predicted position is ahead, the delay time is increased. This adjustment is a continuous optimization process. The delay time should be further adjusted based on the new error value feedback in the future to improve the prediction accuracy, ensure that the system can adapt to complex changes, and enable the driver to receive the head-up display information at the appropriate position.

[0113] In the embodiment of the present application, the method further includes the following steps E1-E3:

[0114] Step E1, acquiring inertial sensor data of the target vehicle.

[0115] Specifically, inertial sensor data can include acceleration in all directions of the vehicle (such as front and back, left and right, up and down directions of linear acceleration) and angular velocity (such as roll, pitch, and yaw directions), which can reflect vehicle dynamics (acceleration, deceleration, turning, steady driving, etc.). These data are obtained to analyze the changes in the position of the user's eyeballs relative to the vehicle-mounted head-up display system. Because the vehicle's motion state will affect the driver's body posture and eyeball position, these data can be used to understand and compensate for the impact, more accurately predict the focus position of the line of sight, and improve the matching degree between the head-up display and the driver's line of sight.

[0116] Step E2: Calculate the relative displacement of the user's eyeball relative to the vehicle-mounted head-up display system based on the inertial sensor data.

[0117] Specifically, the displacement changes in each direction are estimated based on the vehicle's linear acceleration information, and the displacement of the vehicle within a certain period of time is obtained by integrating the time and other methods. Considering the fixed relative position of the driver and the vehicle, the relative displacement of the driver's body (including the head and eyeballs) when the vehicle accelerates or decelerates can be inferred. For example, when the vehicle accelerates, the driver leans back due to inertia, and the eyeballs have displacement changes relative to the vehicle's head-up display system. The rotation is determined by the vehicle's angular velocity information. When the vehicle turns, the displacement of the driver's head and eyeballs in different directions relative to the vehicle's head-up display system is calculated based on the turning angle, speed, and internal geometry of the vehicle, and considering factors such as wheelbase and driver's seat position.

[0118] Step E3, adjusting the predicted coordinate information according to the relative displacement to obtain adjusted predicted coordinate information.

[0119] In the embodiment of the present application, step E3 includes the following steps E31-E34:

[0120] Step E31, determining the offset direction and offset amount of the sight focus according to the relative displacement.

[0121] Specifically, the calculated relative displacement is a vector with directional information, which can be used to determine the offset direction of the sight focus in the vehicle head-up display system coordinate system (such as the horizontal positive direction is to the right, the vertical negative direction is downward, etc.). The magnitude of the relative displacement vector is the offset, which can quantify the degree of movement of the sight focus relative to the original predicted position. In a two-dimensional plane, it is determined by calculating its vector modulus. For example, when the relative displacement component in the X direction is x0 and the component in the Y direction is y0, the offset is The offset is used to adjust the predicted coordinate information.

[0122] Step E32, adjusting the predicted coordinate information based on the offset direction and the offset amount to obtain initial coordinate information.

[0123] Specifically, the predicted coordinate information is modified according to the determined offset direction and offset. If the offset direction is to the right, the corresponding offset is added to the x value of the predicted coordinate (assuming it is a two-dimensional coordinate); if it is upward, the y value is added. For example, the original predicted sight focus coordinate is (x1, y1), if the offset direction is to the right and the offset is Δx, downward and the offset is Δy, then the adjusted initial coordinate information is (x1+Δx, y1+Δy), this coordinate can better reflect the actual sight focus position of the driver.

[0124] Step E33, detecting whether the initial coordinate information is in the projection area of ​​the vehicle head-up display system.

[0125] Specifically, the projection area of ​​the vehicle head-up display system has a range, and the coordinate information within the range is the effective display position. By detecting whether the initial coordinate information is within the range, it can be determined whether the coordinates adjusted by relative displacement can be displayed normally in the head-up display system. This requires satisfying the boundary conditions of the projection area. For example, in a two-dimensional plane, the projection area is a rectangle with the upper left corner coordinate (x min ,y min ) and the lower right corner coordinate (x max ,y max ) and determines whether it is within the projection area by comparing the initial coordinates and the boundary value.

[0126] Step E34, if it is in the projection area, the initial coordinate information is used as the adjusted predicted coordinate information, or, if it is not in the projection area, the boundary processing mechanism is triggered and the preset coordinate information associated with the offset direction is used as the adjusted predicted coordinate information.

[0127] Specifically, if the initial coordinate information is within the projection area, the predicted coordinates are reasonable and can be directly used as the adjusted predicted coordinate information, that is, the coordinates adjusted for the eye displacement caused by vehicle movement can meet the head-up display requirements without further processing. If the initial coordinate information is not within the projection area, the boundary processing mechanism is triggered. Since different offset directions will cause the coordinates to exceed different boundaries, a preset coordinate information is associated with each offset direction. For example, if the coordinates exceed the right boundary, according to the rightward offset direction, the preset coordinates associated with the right boundary (such as the rightmost displayable position coordinates) are used as the adjusted predicted coordinate information. This ensures that under complex vehicle movements, the adjusted predicted coordinate information can be reasonably displayed within the projection area.

[0128] Figure 3 is a flow chart of another method for predicting the visual focus of a vehicle head-up display according to an embodiment of the present invention. Figure 3As shown in the figure, the process includes: first, the activity video of the user's eyeball is obtained through the image acquisition device in the cockpit, and then the historical movement trajectory of the line of sight focus is determined. Next, the on-board head-up display system obtains the historical projection instructions and matches them with the mapping coordinates of the line of sight focus to calculate the delay time. Then, the projection position of the line of sight focus of the next frame is predicted according to the delay time, and the AR-HUD is controlled to perform the projection operation. After that, the actual image of the user's eyeball is collected again, and the actual coordinates are compared with the predicted coordinates to calculate the error value. If the error exceeds the preset threshold, the delay time is corrected. At the same time, the vehicle inertial sensor data is obtained, and the relative displacement of the eyeball is calculated to adjust the predicted coordinate information to ensure that the head-up display better matches the driver's line of sight.

[0129] In this embodiment, a prediction device for the visual focus of a vehicle-mounted head-up display is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0130] This embodiment provides a prediction device for a vehicle-mounted head-up display visual focus, such as Figure 4 As shown, including:

[0131] A collection module 41 is used to collect the activity video of the user's eyeballs;

[0132] A determination module 42, configured to determine a historical movement trajectory of the sight focus based on the active video, and calculate a delay time of the vehicle head-up display system according to the historical movement trajectory;

[0133] A prediction module 43, used to predict the projection position corresponding to the sight focus of the next frame according to the delay time, and obtain predicted coordinate information;

[0134] The control module 44 is used to control the vehicle-mounted head-up display system to perform a projection operation according to the predicted coordinate information.

[0135] In an optional embodiment of the present application, the determination module 42 includes an identification submodule and a calculation submodule;

[0136] Specifically, the recognition submodule is used to extract eyeball images of multiple time frames in the activity video; identify the position coordinates of the sight focus in the eyeball image; and generate a historical movement trajectory of the sight focus according to the position coordinates.

[0137] Specifically, the identification submodule is used to obtain the mapping relationship between the acquisition area of ​​the eyeball image and the projection area of ​​the vehicle-mounted head-up display system; determine the mapping coordinates corresponding to the position coordinates in the projection area according to the mapping relationship; arrange the mapping coordinates corresponding to the eyeball image of each time frame to obtain the historical movement trajectory of the line of sight focus.

[0138] Specifically, the calculation submodule is used to obtain multiple historical projection instructions of the vehicle-mounted head-up display system; traverse the projection coordinates of the historical projection instructions, match the mapping coordinates in the historical movement trajectory with the projection coordinates, and obtain the hit target projection coordinates; calculate the delay time of the vehicle-mounted head-up display system according to the mapping coordinates and the target projection coordinates.

[0139] Specifically, the calculation submodule is used to obtain the execution time corresponding to the target projection coordinates and the timestamp corresponding to the mapping coordinates; and calculate the delay time of the vehicle head-up display system according to the execution time and the timestamp.

[0140] In an optional embodiment of the present application, the prediction module 43 is used to obtain the total number of frames of the active video; calculate the delayed frame number of the line of sight focus based on the total number of frames and the delay time; obtain the current position coordinates of the line of sight focus, and predict the projection position corresponding to the line of sight focus of the next frame based on the delayed frame number and the current position coordinates to obtain the predicted coordinate information.

[0141] In an optional embodiment of the present application, the device also includes: a judgment module, used to collect an actual image of the user's eyeball; determine the actual coordinate information of the current line of sight focus based on the actual image, and calculate the error value between the actual coordinate information and the predicted coordinate information; judge whether the error value exceeds a preset threshold, if it exceeds the preset threshold, trigger a correction mechanism, and correct the delay time corresponding to the vehicle-mounted head-up display system according to the error value to obtain the corrected delay time.

[0142] In an optional embodiment of the present application, the device also includes: an adjustment module for acquiring inertial sensor data of the target vehicle; calculating the relative displacement of the user's eyeball relative to the vehicle-mounted head-up display system based on the inertial sensor data; and adjusting the predicted coordinate information according to the relative displacement to obtain adjusted predicted coordinate information.

[0143] In an optional embodiment of the present application, an adjustment module is used to determine the offset direction and offset amount of the line of sight focus based on the relative displacement; adjust the predicted coordinate information based on the offset direction and offset amount to obtain initial coordinate information; detect whether the initial coordinate information is in the projection area of ​​the vehicle-mounted head-up display system; if it is in the projection area, use the initial coordinate information as the adjusted predicted coordinate information, or, if it is not in the projection area, trigger the boundary processing mechanism and use the preset coordinate information associated with the offset direction as the adjusted predicted coordinate information.

[0144] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0145] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0146] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0147] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0149] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0150] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0151] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for predicting the visual focus of a vehicle head-up display, characterized in that: The method comprises: Collect videos of users’ eye movements; Determine a historical movement trajectory of a sight focus based on the active video, and calculate a delay time of a vehicle head-up display system according to the historical movement trajectory; Predicting the projection screen position corresponding to the sight focus of the next frame according to the delay time to obtain predicted coordinate information; The vehicle-mounted head-up display system is controlled to perform a projection operation according to the predicted coordinate information.

2. The method according to claim 1, characterized in that: The determining the historical movement trajectory of the sight focus based on the activity video includes: Extracting eyeball images of multiple time frames in the active video; Identifying the position coordinates of the sight focus in the eyeball image; A historical movement trajectory of the sight focus is generated according to the position coordinates.

3. The method according to claim 2, characterized in that The generating the historical movement trajectory of the sight focus according to the position coordinates includes: Acquire a mapping relationship between a collection area of ​​the eyeball image and a projection area of ​​the vehicle-mounted head-up display system; Determine the mapping coordinates corresponding to the position coordinates in the projection area according to the mapping relationship; The mapping coordinates corresponding to the eyeball images of each time frame are arranged to obtain the historical movement trajectory of the sight focus.

4. The method according to claim 1, characterized in that The calculating the delay time of the vehicle head-up display system according to the historical movement trajectory includes: Obtaining multiple historical projection instructions of the vehicle head-up display system; Traversing the projection coordinates of the historical projection instructions, matching the mapping coordinates in the historical movement trajectory with the projection coordinates, and obtaining the projection coordinates of the hit target; The delay time of the vehicle-mounted head-up display system is calculated according to the mapping coordinates and the target projection coordinates.

5. The method according to claim 4, characterized in that The calculating the delay time of the vehicle head-up display system according to the mapping coordinates and the target projection coordinates includes: Obtaining the execution time corresponding to the target projection coordinates and the timestamp corresponding to the mapping coordinates; The delay time of the vehicle head-up display system is calculated according to the execution time and the timestamp.

6. The method according to claim 1, characterized in that The predicting the projection position corresponding to the sight focus of the next frame according to the delay time to obtain predicted coordinate information includes: Get the total number of frames of the active video; Calculate the delayed frame number of the sight focus according to the total frame number and the delay time; The current position coordinates of the sight focus are obtained, and the projection position corresponding to the sight focus of the next frame is predicted according to the delayed frame number and the current position coordinates to obtain predicted coordinate information.

7. The method according to claim 1, characterized in that After controlling the vehicle-mounted head-up display system to perform a projection operation according to the predicted coordinate information, the method further includes: Collecting actual images of the user's eyes; Determine the actual coordinate information of the current sight focus according to the actual image, and calculate the error value between the actual coordinate information and the predicted coordinate information; It is determined whether the error value exceeds a preset threshold value. If the error value exceeds the preset threshold value, a correction mechanism is triggered to correct the delay time corresponding to the vehicle-mounted head-up display system according to the error value to obtain the corrected delay time.

8. The method according to claim 1, characterized in that The method further comprises: Acquire the inertial sensor data of the target vehicle; Calculating the relative displacement of the user's eyeball relative to the vehicle-mounted head-up display system based on the inertial sensor data; The predicted coordinate information is adjusted according to the relative displacement to obtain adjusted predicted coordinate information.

9. The method according to claim 8, characterized in that The step of adjusting the predicted coordinate information according to the relative displacement to obtain adjusted predicted coordinate information includes: Determine the offset direction and offset amount of the sight focus according to the relative displacement; Adjusting the predicted coordinate information based on the offset direction and the offset amount to obtain initial coordinate information; Detecting whether the initial coordinate information is in the projection area of ​​the vehicle-mounted head-up display system; If it is in the projection area, the initial coordinate information is used as the adjusted predicted coordinate information; or, if it is not in the projection area, the boundary processing mechanism is triggered and the preset coordinate information associated with the offset direction is used as the adjusted predicted coordinate information.

10. A prediction device for visual focus of a vehicle head-up display, characterized in that: The device comprises: A collection module is used to collect videos of user eye movements; A determination module, configured to determine a historical movement trajectory of a sight focus based on the active video, and calculate a delay time of an in-vehicle head-up display system according to the historical movement trajectory; A prediction module, used to predict the projection position corresponding to the sight focus of the next frame according to the delay time to obtain predicted coordinate information; A control module is used to control the vehicle-mounted head-up display system to perform a projection operation according to the predicted coordinate information.

11. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 9.

13. A vehicle, characterized in that: The vehicle includes: a visual focus prediction system and a controller; wherein the controller stores instructions for controlling the operation of the visual focus prediction system, and the visual focus prediction system is used to execute the method for predicting the visual focus of the vehicle-mounted head-up display according to any one of claims 1 to 9 according to the instructions of the controller.

Images