Vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses

Abstract

The invention relates to the technical field of vehicle-mounted display systems, and discloses a vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses, and the system comprises AR glasses, a second high-frequency inertial measurement unit, a calibration module, a processing unit, an AR glasses display system, a system delay compensation module, and an in-vehicle map and vehicle coordinate system alignment module. According to the vehicle-mounted six-degree-of-freedom head-up display system and method based on the augmented reality glasses, the AR glasses are introduced into a vehicle-mounted HUD system, the limitation of a traditional fixed HUD is broken through, the system cost and space occupation are greatly reduced, real 6DoF tracking and display are achieved by integrating an AR glasses IMU, a vehicle-mounted IMU and an AR glasses camera, and the system and the method are suitable for popularization and application. According to the method, the space alignment precision of the AR content and the real world is remarkably improved, more visual and accurate information display can be provided for a driver, the man-machine interaction experience of an automatic driving system can be improved, and therefore the driving safety and the user experience are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle-mounted display systems, and in particular to a vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses. Background Art

[0002] With the rapid development of automotive multimedia, the layout of display screens in car cockpits is constantly innovating, and the display screens are getting larger and more numerous to meet the driving experience of occupants in different positions in the car cockpit. The driver needs to obtain more information, so the car's display system is also constantly upgrading, including instrument panel displays, head-up displays, augmented reality head-up displays, and so on.

[0003] Traditional head-up displays project key driving information onto the windshield or a dedicated transparent screen, allowing the driver to obtain information without shifting his or her sight. However, this information acquisition method can usually only display two-dimensional information and lacks a sense of depth and space. At the same time, the projection area is small, making it difficult to display complex or large amounts of information. Moreover, the information display is usually static and lacks real-time interaction with the driving environment. The degrees of freedom of augmented reality head-up displays are limited, and most systems only support 2D or 3D displays. At the same time, the display area is usually fixed in front of the driver and cannot be adjusted with the movement of the head. It also has a certain system delay and poor adaptability to the environment. In view of this, we propose an in-vehicle six-degree-of-freedom head-up display system and method based on augmented reality glasses. Summary of the invention

[0004] The object of the present invention is to provide a vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses to solve the problems raised in the above background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: an in-vehicle six-degree-of-freedom head-up display system based on augmented reality glasses, comprising AR glasses, a second high-frequency inertial measurement unit, a calibration module, a processing unit, an AR glasses display system, a system delay compensation module, and an in-vehicle map and vehicle coordinate system alignment module, wherein the system delay compensation module is used to comprehensively consider and predict the hardware, algorithm and display delays and compensate for them, and the in-vehicle map and vehicle coordinate system alignment module is used to ensure that the constructed map is correctly aligned with the vehicle coordinate system, the AR glasses include a monocular camera and a first high-frequency inertial measurement unit, and the second inertial measurement unit is installed on the vehicle, the processing unit includes a calibration module, and the calibration module is used to calibrate the first high-frequency inertial measurement unit and the second high-frequency inertial measurement unit, and the calibration module performs internal reference calibration and dual IMU dynamic calibration, and the AR glasses display system is used to display the generated AR information.

[0006] Preferably, the processing unit includes executing map construction optimization and visual-inertial fusion Kalman filter, wherein the executing map construction optimization includes map merging under multiple lighting conditions, inside-outside-vehicle segmentation, semantic segmentation, quality assessment and illumination invariance enhancement, and the visual-inertial fusion Kalman filter is used for posture smoothing and high-frequency output.

[0007] The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses comprises the following steps:

[0008] S1. System initialization and calibration: Install the second high-frequency inertial measurement unit and set the three-axis direction of the second high-frequency inertial measurement unit. Then calibrate the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit on the vehicle. Then calibrate the external parameters of the camera and the first high-frequency inertial measurement unit of the AR glasses. The external parameter data of the camera and the first high-frequency inertial measurement unit include the relative rotation matrix R and the relative translation vector t.

[0009] S2. Build and optimize the in-car scene map: Use AR glasses to collect in-car scene images and corresponding first high-frequency inertial measurement unit data under different lighting conditions, then use the IMU SLAM algorithm to extract image features and build maps under different scenes, and merge and optimize the maps built under different lighting conditions. Then, segment the in-car and out-car maps, remove the map points outside the car, and establish semantic indexes for different areas and objects in the car. Then, evaluate the quality of the map points to improve the stability and tracking quality of the map. Finally, improve the robustness of the map under different lighting conditions, so that it can be quickly relocated after tracking is lost.

[0010] S3. Aligning the in-car map with the vehicle coordinate system: performing coordinate system conversion calculations based on the data of the initial alignment position and the current position data of the AR glasses in the car, and comparing the coordinates of these points in the transformed map with the actual measured values ​​to ensure that the in-car map and the vehicle coordinate system are aligned;

[0011] S4, collecting data: collecting AR glasses camera frames, first inertial measurement unit data, and second inertial measurement unit data;

[0012] S5, calculate the relative movement of the head relative to the vehicle: calculate the difference IMU_diff between the first high-frequency inertial measurement unit data and the second high-frequency inertial measurement unit data according to the relative rotation matrix between the first inertial unit and the second relationship unit obtained in S3, that is, IMU diff =IMU glass -R×IMU car , thereby obtaining the relative motion value of the head relative to the vehicle;

[0013] S6. Based on the pre-built map, the improved monocular inertial SLAM algorithm is executed using the relative motion data and camera frames: in the in-vehicle scene map built and optimized in S2, the 6DoF pose data is calculated by monocular inertial SLAM;

[0014] S7, use visual-inertial fusion Kalman filter for pose smoothing: smooth and denoise the 6DoF pose data obtained in S6 through visual-inertial fusion Kalman filter;

[0015] S8, performing system delay compensation: predicting and compensating the 6DoF pose data that has been smoothed and denoised in S7 according to hardware delay, algorithm delay, and display system delay;

[0016] S9. Generate and display 6DoF AR information: finally generate and display 6DoF AR information through virtual object placement, occlusion processing and environment adaptive rendering, and display it.

[0017] Preferably, the merging of maps constructed under different lighting conditions in S2 includes the following steps:

[0018] S21, collecting in-vehicle scene images and IMU data under different lighting conditions;

[0019] S22, use the monocular-IMU SLAM algorithm to process the image sequence, extract features and build a map;

[0020] S23, merging and optimizing maps constructed under different lighting conditions;

[0021] S231, using feature descriptors to perform scene recognition and loop closure detection;

[0022] S232, using the iterative closest point (ICP) algorithm to align maps in different scenarios;

[0023] S233, optimizing the 3D position of feature points and key frame pose using global bundle adjustment (BA);

[0024] S24, performing map post-processing;

[0025] S241, vehicle interior and exterior map segmentation, using the RANSAC algorithm to fit the vehicle interior space geometry model;

[0026] S242, in-car scene semantic segmentation, using deep learning models to classify common in-car elements;

[0027] S243, map point quality assessment, based on indicators such as number of observations and reprojection error;

[0028] S244, illumination invariance is enhanced by storing multiple descriptors for each map point and performing fusion weighted processing.

[0029] Preferably, the virtual object placement, occlusion processing and environment adaptive rendering in S9 include the following steps:

[0030] S91, pose data smoothing and denoising, using visual-inertial fusion Kalman filter to fuse low-frequency SLAM pose estimation results and high-frequency IMU difference data, where the state vector of the visual-inertial fusion Kalman filter contains position, attitude, velocity and zero bias;

[0031] S92, dynamic delay compensation, including hardware delay, algorithm delay and display system delay, and using the prediction compensation formula to extrapolate the current moment posture, including translation and rotation compensation;

[0032] S93. Calculate the position of the virtual object in the vehicle coordinate system based on the compensated posture, and use the pre-built 3D model of the vehicle interior to perform occlusion calculation to achieve a semi-transparent effect. At the same time, adjust the brightness and contrast of the AR content according to the current lighting conditions to achieve a dynamic shadow effect.

[0033] Preferably, when the three-axis directions of the second high-frequency inertial measurement unit are set in S1, the X-axis is consistent with the direction of the vehicle head, and the Y-axis points to the right or left of the vehicle, that is, a right-hand coordinate system or a left-hand coordinate system, and the Z-axis is aligned with the direction of gravity, vertically downward in the right-hand coordinate system and vertically upward in the left-hand coordinate system, and the user chooses to use the left-hand coordinate system or the right-hand coordinate system before starting the system setting, and only one coordinate system can be used in the same period of time.

[0034] Preferably, the calibrating the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit on the vehicle in S1 includes the following steps:

[0035] S11. Fix the AR glasses at the driver's position to simulate the actual wearing height and angle;

[0036] S12. Ensure that the three-axis directions of the AR glasses IMU are consistent with the three-axis directions of the vehicle-mounted IMU;

[0037] S13. The acquisition frequency settings of the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit in the vehicle are consistent, and are set within the range of 200 Hz to 1000 Hz;

[0038] S14. The first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit in the vehicle are precisely synchronized using a hardware trigger signal or using the Network Time Protocol (NTP) for software-level synchronization to ensure timestamp accuracy;

[0039] S15. Obtain zero bias, scale factor, and axis offset of the first high-frequency inertial measurement unit and the second high-frequency inertial measurement unit to obtain calibrated IMU data.

[0040] Preferably, the AR glasses camera intrinsic parameter data in S1 includes distortion coefficients and camera matrix parameters, wherein the distortion coefficients include radial distortion (k1, k2, k3) and tangential distortion (p1, p2), and the camera matrix parameters include focal length (fx, fy) and principal point coordinates (cx, cy). The AR glasses camera intrinsic parameters execute a predefined motion sequence including translation and rotation when calibrating the camera and the first high-frequency inertial measurement unit extrinsic parameters, collect a video sequence containing a checkerboard, and record IMU data at the same time. Then, the camera calibration program provided by OpenCV or the camera calibration module in the Kalibr toolkit is used to calculate the AR glasses camera intrinsic parameters and the camera and the first high-frequency inertial measurement unit extrinsic parameters.

[0041] Preferably, the step of extracting image features using the IMU SLAM algorithm and constructing maps in different scenarios in S2 includes the following steps:

[0042] S201, using the monocular-IMU SLAM algorithm to process the image sequence, extract image features and construct maps under different scenarios;

[0043] S202, extracting key frame image features using feature descriptors such as ORB or BAD;

[0044] S203, using the image feature points on different key frames to perform feature point matching, performing triangulation calculation based on the matched feature points, and combining the data information of the first high-frequency inertial measurement unit to obtain 3D map points;

[0045] S204: reproject the calculated 3D map point back to the key frame and calculate the reprojection error. If the requirements are met, the map point is used to obtain the map.

[0046] Preferably, when establishing semantic indexes for different areas and objects in the car in S2, a pre-trained deep learning model, such as Segment Anything Model 2 (SAM 2), is first used to perform semantic segmentation, and semantic labels are associated with map points. Finally, common in-car elements are classified, such as steering wheels, dashboards, side windows, rearview mirrors, seats, and front windshields.

[0047] Compared with the prior art, the present invention provides a vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses, which has the following beneficial effects:

[0048] 1. The vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses introduces AR glasses into the vehicle-mounted HUD system, breaking through the limitations of traditional fixed HUD, greatly reducing system costs and space occupancy, and realizing true 6DoF tracking and display by integrating AR glasses IMU, vehicle-mounted IMU and AR glasses camera, significantly improving the spatial alignment accuracy between AR content and the real world, which can not only provide drivers with more intuitive and accurate information display, but also enhance the human-computer interaction experience of the autonomous driving system, thereby improving driving safety and user experience.

[0049] 2. The vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses, by providing a SLAM algorithm specifically for the vehicle-mounted environment, effectively solves problems such as multi-coordinate system conversion and complex lighting adaptation, ensuring the stability and accuracy of the system in various driving scenarios.

[0050] 3. The vehicle-mounted six-degree-of-freedom head-up display system and method based on augmented reality glasses uses an innovative sensor fusion algorithm to combine low-frequency SLAM pose estimation with high-frequency IMU data, achieving a system update frequency of up to 400 Hz, greatly improving the smoothness and real-time performance of AR display. At the same time, it introduces time delay compensation and motion prediction mechanisms, effectively reducing the impact of system delays on AR display effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a schematic diagram of the map merging and optimization process of the present invention;

[0052] Figure 2 This is a schematic diagram of alignment between the in-vehicle map and the vehicle coordinate system of the present invention;

[0053] Figure 3 This is a schematic diagram of the real-time 6DoF operation process of the present invention;

[0054] Figure 4 This is a flow chart of HUD information generation and display according to the present invention. DETAILED DESCRIPTION

[0055] like Figure 1-Figure 4As shown, the present invention provides a technical solution: an on-vehicle six-degree-of-freedom head-up display system based on augmented reality glasses, AR glasses, a second high-frequency inertial measurement unit, a calibration module, a processing unit, an AR glasses display system, a system delay compensation module, and an in-vehicle map and vehicle coordinate system alignment module, the system delay compensation module is used to comprehensively consider and predict the hardware, algorithm and display delays, and the in-vehicle map and vehicle coordinate system alignment module is used to ensure that the constructed map is correctly aligned with the vehicle coordinate system, the AR glasses include a monocular camera and a first high-frequency inertial measurement unit, the second inertial measurement unit is installed on the vehicle, the processing unit includes a calibration module, the calibration module is used to calibrate the first high-frequency inertial measurement unit and the second high-frequency inertial measurement unit, the calibration module performs internal reference calibration and dual IMU dynamic calibration, and the AR glasses display system is used to display the generated AR information.

[0056] In addition, when installing the vehicle-mounted IMU, shock-absorbing measures are adopted, such as using rubber shock-absorbing pads, sponge shock-absorbing pads or spring shock absorbers, to reduce the impact of road bumps on the data.

[0057] In one embodiment of the present invention, the processing unit includes performing map construction optimization and visual-inertial fusion Kalman filter, the map construction optimization includes map merging under multiple lighting conditions, inside-outside-vehicle segmentation, semantic segmentation, quality assessment and illumination invariance enhancement, and the visual-inertial fusion Kalman filter is used for posture smoothing and high-frequency output.

[0058] In addition, the present invention also includes a display method of a vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses, which includes the following steps:

[0059] S1. System initialization and calibration: Install the second high-frequency inertial measurement unit and set the three-axis direction of the second high-frequency inertial measurement unit. Then calibrate the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit on the vehicle. Then calibrate the external parameters of the camera and the first high-frequency inertial measurement unit of the AR glasses. The external parameter data of the camera and the first high-frequency inertial measurement unit include the relative rotation matrix R and the relative translation vector t.

[0060] In S1, calibrating the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit on the vehicle includes the following steps:

[0061] S11. Fix the AR glasses at the driver's position to simulate the actual wearing height and angle;

[0062] S12. Ensure that the three-axis directions of the AR glasses IMU are consistent with the three-axis directions of the vehicle-mounted IMU;

[0063] S13. The acquisition frequency settings of the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit in the vehicle are consistent, and are set within the range of 200 Hz to 1000 Hz;

[0064] S14. The first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit in the vehicle are precisely synchronized using a hardware trigger signal or using the Network Time Protocol (NTP) for software-level synchronization to ensure timestamp accuracy;

[0065] S15, acquiring the zero bias, scale factor and axis deviation of the first high-frequency inertial measurement unit and the second high-frequency inertial measurement unit to obtain calibrated IMU data;

[0066] S2. Build and optimize the in-car scene map: Use AR glasses to collect in-car scene images and corresponding first high-frequency inertial measurement unit data under different lighting conditions, then use the IMU SLAM algorithm to extract image features and build maps under different scenes, and merge and optimize the maps built under different lighting conditions. Then, segment the in-car and out-car maps, remove the map points outside the car, and establish semantic indexes for different areas and objects in the car. Then, evaluate the quality of the map points to improve the stability and tracking quality of the map. Finally, improve the robustness of the map under different lighting conditions, so that it can be quickly relocated after tracking is lost.

[0067] Using the IMU SLAM algorithm in S2 to extract image features and build maps for different scenarios includes the following steps:

[0068] S201, using the monocular-IMU SLAM algorithm to process the image sequence, extract image features and construct maps under different scenarios;

[0069] S202, extracting key frame image features using feature descriptors such as ORB or BAD;

[0070] S203, using the image feature points on different key frames to perform feature point matching, performing triangulation calculation based on the matched feature points, and combining the data information of the first high-frequency inertial measurement unit to obtain 3D map points;

[0071] S204, reprojecting the calculated 3D map point back to the key frame and calculating the reprojection error. If the requirements are met, the map point is used to obtain the map;

[0072] The merging of maps constructed under different lighting conditions in S2 includes the following steps:

[0073] S21, collecting in-vehicle scene images and IMU data under different lighting conditions;

[0074] S22, use the monocular-IMU SLAM algorithm to process the image sequence, extract features and build a map;

[0075] S23, merging and optimizing maps constructed under different lighting conditions;

[0076] S231, using feature descriptors to perform scene recognition and loop closure detection;

[0077] S232, using the iterative closest point (ICP) algorithm to align maps in different scenarios;

[0078] S233, optimizing the 3D position of feature points and key frame pose using global bundle adjustment (BA);

[0079] S24, performing map post-processing;

[0080] S241, vehicle interior and exterior map segmentation, using the RANSAC algorithm to fit the vehicle interior space geometry model;

[0081] S242, in-car scene semantic segmentation, using deep learning models to classify common in-car elements;

[0082] S243, map point quality assessment, based on indicators such as number of observations and reprojection error;

[0083] S244, illumination invariance enhancement, storing multiple descriptors for each map point and performing fusion weighted processing;

[0084] S3. Aligning the in-car map with the vehicle coordinate system: performing coordinate system conversion calculations based on the data of the initial alignment position and the current position data of the AR glasses in the car, and comparing the coordinates of these points in the transformed map with the actual measured values ​​to ensure that the in-car map and the vehicle coordinate system are aligned;

[0085] S4, collecting data: collecting AR glasses camera frames, first inertial measurement unit data, and second inertial measurement unit data;

[0086] S5, calculate the relative movement of the head relative to the vehicle: calculate the difference IMU_diff between the first high-frequency inertial measurement unit data and the second high-frequency inertial measurement unit data according to the relative rotation matrix between the first inertial unit and the second relationship unit obtained in S3, that is, IMU diff =IMU glass -R×IMU car , thereby obtaining the relative motion value of the head relative to the vehicle;

[0087] S6. Based on the pre-built map, the improved monocular inertial SLAM algorithm is executed using the relative motion data and camera frames: in the in-vehicle scene map built and optimized in S2, the 6DoF pose data is calculated by monocular inertial SLAM;

[0088] S7, use visual-inertial fusion Kalman filter for pose smoothing: smooth and denoise the 6DoF pose data obtained in S6 through visual-inertial fusion Kalman filter;

[0089] S8, performing system delay compensation: predicting and compensating the 6DoF pose data that has been smoothed and denoised in S7 according to hardware delay, algorithm delay, and display system delay;

[0090] S9, generating and displaying 6DoF AR information: finally generating and displaying 6DoF AR information through virtual object placement, occlusion processing, and environment adaptive rendering, and displaying it;

[0091] Virtual object placement, occlusion processing, and environment adaptive rendering in S9 include the following steps:

[0092] S91, pose data smoothing and denoising, using visual-inertial fusion Kalman filter to fuse low-frequency SLAM pose estimation results and high-frequency IMU difference data, where the state vector of the visual-inertial fusion Kalman filter contains position, attitude, velocity and zero bias;

[0093] S92, dynamic delay compensation, including hardware delay, algorithm delay and display system delay, and using the prediction compensation formula to extrapolate the current moment posture, including translation and rotation compensation;

[0094] S93. Calculate the position of the virtual object in the vehicle coordinate system based on the compensated posture, and use the pre-built 3D model of the vehicle interior to perform occlusion calculation to achieve a semi-transparent effect. At the same time, adjust the brightness and contrast of the AR content according to the current lighting conditions to achieve a dynamic shadow effect.

[0095] In an embodiment of the present invention, when the three-axis directions of the second high-frequency inertial measurement unit are set in S1, the X-axis is consistent with the direction of the front of the vehicle, and the Y-axis points to the right or left of the vehicle, that is, a right-hand coordinate system or a left-hand coordinate system, and the Z-axis is aligned with the direction of gravity, vertically downward in the right-hand coordinate system and vertically upward in the left-hand coordinate system. Before the system starts the setting, the user can choose to use the left-hand coordinate system or the right-hand coordinate system, and only one coordinate system can be used in the same period of time.

[0096] In the present invention, the AR glasses camera intrinsic parameter data in S1 includes distortion coefficients and camera matrix parameters, wherein the distortion coefficients include radial distortion (k1, k2, k3) and tangential distortion (p1, p2), and the camera matrix parameters include focal length (fx, fy) and principal point coordinates (cx, cy). The AR glasses camera intrinsic parameters execute a predefined motion sequence including translation and rotation when calibrating the camera and the first high-frequency inertial measurement unit extrinsic parameters, collect a video sequence containing a checkerboard, and record IMU data at the same time. Then, the camera calibration program provided by OpenCV or the camera calibration module in the Kalibr toolkit is used to calculate the AR glasses camera intrinsic parameters and the camera and the first high-frequency inertial measurement unit extrinsic parameters.

[0097] Furthermore, when S2 establishes semantic indexes for different areas and objects in the car, it first uses pre-trained deep learning models, such as Segment Anything Model 2 (SAM 2), to perform semantic segmentation, and associates semantic labels with map points. Finally, it classifies common in-car elements, such as steering wheels, dashboards, side windows, rearview mirrors, seats, and front windshields. When S2 merges and optimizes maps built under different lighting conditions, it first uses the front-end feature descriptor (egORB or BAD) feature matching method to perform scene recognition and loop detection. Then, it uses the iterative closest point (ICP) algorithm based on the co-view between different maps to align maps in different scenes, merge duplicate map points, update descriptors and observation information, and finally uses the g2o optimizer to perform global bundle adjustment (BA) to optimize the 3D position of feature points and keyframe poses.

[0098] In addition, in S2, the segmentation of the inside and outside maps of the car needs to first use the depth information and spatial position information, and then use the RANSAC algorithm to fit the geometric model of the space inside the car, and finally mark the points outside the geometric model as outside points and remove them, so as to finally achieve the inside and outside map segmentation of the car. Specifically, when evaluating the quality of map points, first analyze the number of observations of each map point, reprojection error and other indicators, then assign a quality score to each map point, and finally remove or reduce the weight of low-quality points, so as to improve the stability and tracking quality of the map. In addition, in order to enhance the illumination invariance, first store multiple descriptors (egORB, BAD, BEBLID, etc.) for each map point to correspond to different illumination conditions, and then perform fusion weighted processing on the multiple feature descriptors corresponding to each map point. After the tracking is lost, extract different feature descriptors of the same feature point in the video, and match them with the fused feature descriptors stored in the map after fusion, so as to enhance the repositioning capability.

[0099] Furthermore, when performing coordinate system conversion calculation in S3, since the AR glasses are aligned with the front of the vehicle, the rotation matrix value calculated by the SLAM algorithm It is the rotation from the map coordinate system to the vehicle coordinate system, that is

[0100] It is worth noting that when calculating 6DoF pose for monocular SLAM in S6, it includes visual feature tracking, local map tracking, keyframe selection and insertion, local optimization and map update. Specifically, visual feature tracking first extracts the visual features of the current frame and matches them with the features in the previous frame or local map, and then uses the optical flow method to track the feature points of the previous frame to improve continuity. Local map tracking requires projecting local map points to the current frame according to the estimated pose, and then performing feature matching to increase the robustness of tracking. Finally, the dynamic threshold method is used to judge the tracking quality. At the same time, keyframe selection and The insertion is specifically based on the change of viewpoint, the number of feature tracking, the distance to the previous keyframe, etc., and then the IMU integration error is considered. The keyframe is forced to be inserted when there is no keyframe for a long time. The local optimization is achieved by locally optimizing the most recent set of keyframes and observed 3D points, and then minimizing the reprojection error, while optimizing the camera pose and 3D point position. The last map update is to insert the new keyframe into the pre-built map, and then update the map data structures such as the co-viewing area and keyframe data set. Finally, the triangulated new map points are inserted into the map, and map operations such as adding and merging are performed.

[0101] Furthermore, in S7, the visual-inertial fusion Kalman filter includes a state vector, an observation, a prediction step, and an update step, wherein the state vector includes position, attitude, velocity, and zero bias, and the observation includes low-frequency SLAM pose estimation results (eg30hz) and high-frequency IMU difference data (eg400hz). The prediction step predicts the state at the next moment based on the motion model, and the update step predicts when the IMU data arrives and updates when the SLAM results arrive.

[0102] At the same time, the hardware delay in 92 includes camera exposure time, IMU data transmission delay, algorithm delay includes SLAM processing time, filter processing time, and display system delay includes rendering and display delay. When predicting compensation, the received state and speed information is used to extrapolate the current position, and then the translation and rotation values ​​are entered into the compensation formula.

[0103] P * =P+v*Δt+0.5*a*Δt 2

[0104] Q * =Q·Δq

[0105] Δq=[cos(||ω||*Δt / 2),sin(||ω||*Δt / 2)*ω / ||ω||]

[0106] Where P* is the compensated position, P is the algorithm output position, v is the velocity, a is the acceleration, Δt is the estimated total delay time, Q* is the compensated quaternion, Q is the algorithm output quaternion, Δq is the predicted rotation increment (in quaternion form), ω is the angular velocity, ||ω|| is the modulus of the angular velocity, and * represents quaternion multiplication.

[0107] The above generally describes the present invention in detail, but it is obvious to a person skilled in the art that some modifications or improvements can be made to the present invention. Therefore, modifications or improvements that do not depart from the spirit of the present invention are within the scope of protection of the present invention.

Claims

1. A vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses, characterized in that: It includes AR glasses, a second high-frequency inertial measurement unit, a calibration module, a processing unit, an AR glasses display system, a system delay compensation module, and an in-vehicle map and vehicle coordinate system alignment module. The AR glasses include a monocular camera and a first high-frequency inertial measurement unit. The second inertial measurement unit is installed on the vehicle. The processing unit includes a calibration module. The calibration module is used to calibrate the first high-frequency inertial measurement unit and the second high-frequency inertial measurement unit. The calibration module performs internal reference calibration and dual IMU dynamic calibration. The AR glasses display system is used to display the generated AR information.

2. The vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 1, characterized in that: The processing unit includes executing map construction optimization and visual-inertial fusion Kalman filter, wherein the executing map construction optimization includes map merging under multiple lighting conditions, vehicle interior and exterior segmentation, semantic segmentation, quality assessment and illumination invariance enhancement, and the visual-inertial fusion Kalman filter is used for posture smoothing and high-frequency output.

3. A display method of a vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses as described in claims 1-2, characterized in that: The following steps are involved: S1. System initialization and calibration: Install the second high-frequency inertial measurement unit and set the three-axis direction of the second high-frequency inertial measurement unit. Then calibrate the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit on the vehicle. Then calibrate the external parameters of the camera and the first high-frequency inertial measurement unit of the AR glasses. The external parameter data of the camera and the first high-frequency inertial measurement unit include the relative rotation matrix R and the relative translation vector t. S2. Build and optimize the in-car scene map: Use AR glasses to collect in-car scene images and corresponding first high-frequency inertial measurement unit data under different lighting conditions, then use the IMU SLAM algorithm to extract image features and build maps under different scenes, and merge and optimize the maps built under different lighting conditions. Then, segment the in-car and out-car maps, remove the map points outside the car, and establish semantic indexes for different areas and objects in the car. Then, evaluate the quality of the map points, and finally improve the robustness of the map under different lighting conditions. S3. Aligning the in-car map with the vehicle coordinate system: performing coordinate system conversion calculations based on the data of the initial alignment position and the current position data of the AR glasses in the car, and comparing the coordinates of these points in the transformed map with the actual measured values ​​to ensure that the in-car map and the vehicle coordinate system are aligned; S4, collecting data: collecting AR glasses camera frames, first inertial measurement unit data, and second inertial measurement unit data; S5, calculate the relative movement of the head relative to the vehicle: calculate the difference IMU_diff between the first high-frequency inertial measurement unit data and the second high-frequency inertial measurement unit data according to the relative rotation matrix between the first inertial unit and the second relationship unit obtained in S3, that is, IMU diff =IMU glass -R×IMU car , thereby obtaining the relative motion value of the head relative to the vehicle; S6. Based on the pre-built map, the improved monocular inertial SLAM algorithm is executed using the relative motion data and camera frames: in the in-vehicle scene map built and optimized in S2, the 6DoF pose data is calculated by monocular inertial SLAM; S7, use visual-inertial fusion Kalman filter for pose smoothing: smooth and denoise the 6DoF pose data obtained in S6 through visual-inertial fusion Kalman filter; S8, performing system delay compensation: predicting and compensating the smoothed and denoised 6DoF pose data in S7 according to hardware delay, algorithm delay, and display system delay; S9. Generate and display 6DoF AR information: finally generate and display 6DoF AR information through virtual object placement, occlusion processing and environment adaptive rendering, and display it.

4. The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 3, characterized in that: The merging of maps constructed under different lighting conditions in S2 includes the following steps: S21, collecting in-vehicle scene images and IMU data under different lighting conditions; S22, use the monocular-IMU SLAM algorithm to process the image sequence, extract features and build a map; S23, merging and optimizing maps constructed under different lighting conditions; S231, using feature descriptors to perform scene recognition and loop closure detection; S232, using the iterative closest point (ICP) algorithm to align maps in different scenarios; S233, optimizing the 3D position of feature points and key frame pose using global bundle adjustment (BA); S24, performing map post-processing; S241, vehicle interior and exterior map segmentation, using RANSAC algorithm to fit the vehicle interior space geometry model; S242, in-car scene semantic segmentation, using deep learning models to classify common in-car elements; S243, map point quality assessment, based on indicators such as number of observations and reprojection error; S244, illumination invariance is enhanced by storing multiple descriptors for each map point and performing fusion weighted processing.

5. The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 1, characterized in that: The virtual object placement, occlusion processing and environment adaptive rendering in S9 include the following steps: S91, pose data smoothing and denoising, using visual-inertial fusion Kalman filter to fuse low-frequency SLAM pose estimation results and high-frequency IMU difference data, where the state vector of the visual-inertial fusion Kalman filter contains position, attitude, velocity and zero bias; S92, dynamic delay compensation, including hardware delay, algorithm delay and display system delay, and using the prediction compensation formula to extrapolate the current moment posture, including translation and rotation compensation; S93. Calculate the position of the virtual object in the vehicle coordinate system based on the compensated posture, and use the pre-built 3D model of the vehicle interior to perform occlusion calculation to achieve a semi-transparent effect. At the same time, adjust the brightness and contrast of the AR content according to the current lighting conditions to achieve a dynamic shadow effect.

6. The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 3, characterized in that: When setting the three-axis direction of the second high-frequency inertial measurement unit in S1, the X-axis is consistent with the direction of the vehicle head, and the Y-axis points to the right or left of the vehicle, that is, the right-hand coordinate system or the left-hand coordinate system, and the Z-axis is aligned with the direction of gravity, vertically downward in the right-hand coordinate system and vertically upward in the left-hand coordinate system. Before starting the setting, the user can choose to use the left-hand coordinate system or the right-hand coordinate system, and only one coordinate system can be used in the same period of time.

7. The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 3, characterized in that: The step of calibrating the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit on the vehicle in S1 includes the following steps: S11. Fix the AR glasses at the driver's position to simulate the actual wearing height and angle; S12. Ensure that the three-axis directions of the AR glasses IMU are consistent with the three-axis directions of the vehicle-mounted IMU; S13. The acquisition frequency settings of the first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit in the vehicle are consistent, and are set within the range of 200 Hz to 1000 Hz; S14. The first high-frequency inertial measurement unit in the AR glasses and the second high-frequency inertial measurement unit on the vehicle are precisely synchronized using a hardware trigger signal or using the Network Time Protocol (NTP) for software-level synchronization to ensure timestamp accuracy; S15. Obtain zero bias, scale factor, and axis offset of the first high-frequency inertial measurement unit and the second high-frequency inertial measurement unit to obtain calibrated IMU data.

8. The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 3, characterized in that: The AR glasses camera intrinsic parameter data in S1 includes distortion coefficients and camera matrix parameters, wherein the distortion coefficients include radial distortion (k1, k2, k3) and tangential distortion (p1, p2), and the camera matrix parameters include focal length (fx, fy) and principal point coordinates (cx, cy). The AR glasses camera intrinsic parameters execute a predefined motion sequence including translation and rotation when calibrating the camera and the first high-frequency inertial measurement unit extrinsic parameters, collect a video sequence containing a checkerboard, and record IMU data at the same time. Then, the camera calibration program provided by OpenCV or the camera calibration module in the Kalibr toolkit is used to calculate the AR glasses camera intrinsic parameters and the camera and the first high-frequency inertial measurement unit extrinsic parameters.

9. The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 3, characterized in that: The S2 uses the IMU SLAM algorithm to extract image features and construct maps in different scenarios, including the following steps: S201, using the monocular-IMU SLAM algorithm to process the image sequence, extract image features and construct maps under different scenarios; S202, extracting key frame image features using feature descriptors such as ORB or BAD; S203, using the image feature points on different key frames to perform feature point matching, performing triangulation calculation based on the matched feature points, and combining the data information of the first high-frequency inertial measurement unit to obtain 3D map points; S204: reproject the calculated 3D map point back to the key frame and calculate the reprojection error. If the requirements are met, the map point is used to obtain the map.

10. The display method of the vehicle-mounted six-degree-of-freedom head-up display system based on augmented reality glasses according to claim 3, characterized in that: When establishing semantic indexes for different areas and objects in the car in S2, a pre-trained deep learning model, such as Segment Anything Model 2 (SAM 2), is first used to perform semantic segmentation, and semantic labels are associated with map points. Finally, common elements in the car are classified.

Images