Vehicle-mounted AR-HUD real-time alignment system and method based on multi-modal fusion

Abstract

The invention discloses a vehicle-mounted AR-HUD real-time alignment system and method based on multi-modal fusion. The system comprises an image sensor module, an inertial measurement unit IMU, a high-performance calculation unit and an HUD display unit. The image sensor module is installed on the front portion of a vehicle, the output end of the image sensor module and the output end of the IMU are both connected with the input end of the high-performance calculation unit, and the output end of the high-performance calculation unit is connected with the input end of the HUD display unit. The method solves the problems that in the prior art, alignment stability is poor in a vehicle-mounted dynamic environment, alignment robustness is insufficient in a complex environment, long-term alignment drifting is caused by accumulated errors in an existing real-time alignment system, shielding processing is inaccurate due to limited three-dimensional sensing capacity, and alignment accuracy is poor. And when high-precision and high-robustness alignment is realized, calculation efficiency and real-time performance are difficult to consider at the same time.

Description

Technical Field

[0001] This invention belongs to the field of real-time alignment of in-vehicle AR-HUD based on multimodal fusion, specifically relating to a real-time alignment system and method for in-vehicle AR-HUD based on multimodal fusion. Background Technology

[0002] With the rapid development of the traditional automotive industry towards intelligent and connected vehicles, in-vehicle display systems are undergoing profound changes. Early traditional car dashboards mainly displayed basic information such as vehicle speed, RPM, and fuel level in mechanical or simple digital form. Drivers needed to frequently look down to check these key data while driving, which led to brief "windows" in their observation of the road ahead, easily distracting them and increasing the risk of traffic accidents.

[0003] To address this safety hazard and enhance the driving experience, the Windshield Head-Up Display (W-HUD) system was developed. W-HUDs project information such as vehicle speed, navigation direction, and warnings directly onto the windshield in front of the driver, allowing them to access crucial information without looking down, significantly reducing eye-shift time and initially improving driving safety. However, traditional W-HUDs typically display relatively simple information, limited to a flat display in a fixed area of ​​the windshield. Their content lacks deep integration with the actual road conditions, failing to provide more intuitive and contextualized driving assistance. For example, a navigation arrow might simply be a floating symbol in the field of vision, unable to be accurately superimposed on the actual turning point.

[0004] Against this backdrop, Augmented Reality Head-Up Displays (AR-HUDs), representing the future development trend, have emerged. AR-HUDs represent a qualitative leap forward from W-HUDs. They are no longer simply projections of information, but rather, through advanced computer vision, sensor fusion, and image processing algorithms, precisely overlay and fuse virtual information (such as virtual navigation lane lines, highlighted obstacles ahead, distance warnings, and real-time POI (Point of Interest) information) onto the real road surface or objects seen by the driver in a high-precision, real-time, and three-dimensional manner. With AR-HUDs, drivers can intuitively see virtual navigation arrows directly "laid" on the road before a turn, or see dangerous vehicles ahead "locked" in real-time by virtual boxes, greatly improving the efficiency of information transmission, the accuracy of driver judgment, and driving safety, truly ushering in a new era of immersive intelligent driving.

[0005] However, the core of an AR-HUD system lies in the precise alignment between virtual images and real-world scenes. To achieve image registration, current research primarily employs image feature extraction and matching methods, among which the Scale Invariant Feature Transform (SIFT) algorithm is widely used. This method extracts key feature points from images and uses feature descriptors to achieve matching and transform estimation, thereby establishing a spatial mapping relationship between two frames. Although SIFT exhibits certain stability in static environments, the algorithm is computationally intensive and time-consuming, making it difficult to meet the real-time and low-latency requirements of in-vehicle AR-HUDs. Furthermore, it is unstable in complex environments such as image blurring, strong light interference, and occlusion.

[0006] To overcome the aforementioned shortcomings, a registration scheme based on Simultaneous Localization and Mapping (SLAM) is further proposed. This type of method can build scene maps in unknown environments and estimate camera pose in real time, thereby achieving dynamic registration. Typical schemes include PTAM, LSD-SLAM, and ORB-SLAM, among others. ORB-SLAM, in particular, has become one of the commonly used techniques in vehicle-mounted visual registration due to its strong robustness, low computational overhead, and fast operating speed.

[0007] However, SLAM-based methods also have certain limitations. Their system stability is easily affected by external environmental factors such as changes in lighting and dynamic target occlusion. Furthermore, they still face computational challenges when running on resource-constrained in-vehicle terminals, making it difficult to achieve high frame rates and low latency real-time registration. Currently, registration accuracy is still difficult to maintain at the sub-pixel level, and virtual images are prone to drift or misalignment, limiting the reliability and user experience of AR-HUD systems in actual driving.

[0008] In conclusion, achieving robust, high-precision, and low-latency image registration in dynamic driving scenarios remains a key challenge for the practical application of AR-HUD systems, and there is an urgent need to explore new technical solutions that combine multimodal fusion, lightweight computing, and efficient matching algorithms.

[0009] In existing technology one, traditional AR-HUD image registration methods often use the SIFT (Scale Invariant Feature Transform) algorithm to extract image feature points for registration. This method can extract local image features that are scale and rotation invariant, calculate the correspondence between images through feature descriptors, and further estimate the image transformation matrix to complete image alignment.

[0010] The general process of this technology is as follows: feature extraction of the input image → feature description → matching → transformation model estimation. It is widely used for image registration and tracking in static environments.

[0011] The disadvantages of the existing technology are: the algorithm has a large amount of computation and a long processing time, which cannot meet the low latency requirements of in-vehicle AR-HUD; under the interference of image blurring, occlusion, strong light and other factors in actual driving, feature point detection is prone to failure; the registration results are not stable enough and the accuracy is limited in dynamic environments, which affects the virtual-real fusion effect.

[0012] Existing technology two proposes a large-scale tracking and mapping method based on visual SLAM to address the registration problem in unknown scenes. Typical methods, such as PTAM and LSD-SLAM, can simultaneously complete environmental map construction and camera pose estimation for visual registration in dynamic environments.

[0013] These methods emphasize the continuity between image sequences, combining keyframe selection and feature tracking to achieve simultaneous registration and localization, and possess a certain degree of environmental adaptability.

[0014] The disadvantages of the existing technology 2 are: high computational resource consumption, making it unsuitable for embedded or resource-constrained vehicle terminals; weak robustness to environmental interference such as changes in lighting, occlusion, and motion blur; difficulty in suppressing accumulated errors, and easy registration drift during long-term operation; difficulty in balancing real-time performance and stability, making it unsuitable for AR displays in demanding dynamic scenes.

[0015] This invention aims to address the following technical shortcomings in existing real-time alignment technology for in-vehicle augmented reality head-up displays (AR-HUDs):

[0016] The technical shortcomings of existing technologies in terms of poor alignment stability in dynamic in-vehicle environments: Existing AR-HUD alignment algorithms, such as those based on a single visual sensor or unoptimized feature processing, cannot consistently output stable, jitter-free alignment results when the vehicle is moving at high speed, the road surface is bumpy, there is rapid acceleration or deceleration, steering, or the in-vehicle camera itself shakes. This leads to pixel-level drift, jitter, and jumps between the virtual image and the real road conditions, directly affecting the accurate matching of virtual information and the real scene. This is a dynamic error that cannot be overcome at the technical level.

[0017] Existing technologies suffer from insufficient robustness in alignment under complex environments: Current AR-HUD real-time alignment algorithms, including some traditional visual odometry (VO) or visual SLAM schemes, exhibit a sharp decline or even failure in accuracy for feature extraction, matching, and tracking used for pose estimation in challenging environments such as direct sunlight, rapid shadow transitions, low light at night, rain, snow, fog, and situations with missing or repetitive road textures. This results in unstable alignment performance under non-ideal conditions, failing to provide a reliable real-time augmented reality experience.

[0018] The technical drawback of existing real-time alignment systems is the accumulation of errors leading to long-term alignment drift: Most vision-based real-time alignment technologies, lacking effective global optimization or closed-loop detection mechanisms, inevitably accumulate errors during continuous operation. These tiny inter-frame pose estimation errors accumulate as the vehicle travels further, resulting in persistent nonlinear alignment drift between virtual information and the real scene. This severely impacts the long-term stability and usability of the system, requiring frequent manual calibration.

[0019] Existing alignment technologies suffer from limitations in 3D perception capabilities, leading to inaccurate occlusion handling: Traditional AR-HUD alignment methods, especially those relying on monocular vision, inherently lack the ability to accurately acquire real-time depth information of the real scene. This directly results in the inability to accurately determine the spatial occlusion relationship between virtual and real objects during virtual information rendering. Consequently, virtual content may be incorrectly displayed in areas occluded by real objects, or may incorrectly occlude real objects, affecting the realism of the fused display and the accuracy of information transmission.

[0020] Existing technologies suffer from a technical limitation in achieving high-precision and robust alignment while simultaneously compromising computational efficiency and real-time performance: Some existing alignment algorithms that prioritize high precision or global consistency, such as some complex SLAM schemes, often come with high computational complexity. This creates a technical conflict with the stringent requirements of automotive embedded systems for low latency, high real-time performance, and low power consumption. Under limited automotive computing resources, it becomes difficult to simultaneously meet the performance and computational efficiency demands of real-time alignment, leading to system lag. Summary of the Invention

[0021] To address the aforementioned technical problems, this invention proposes a vehicle-mounted AR-HUD real-time alignment system and method based on multimodal fusion. This solves the problems of poor alignment stability in vehicle dynamic environments, insufficient alignment robustness in complex environments, long-term alignment drift caused by accumulated errors in existing real-time alignment systems, inaccurate occlusion handling due to limited 3D perception capabilities, and the difficulty in balancing computational efficiency and real-time performance when achieving high-precision and high-robust alignment.

[0022] The technical solution provided by this invention is as follows:

[0023] A vehicle-mounted AR-HUD real-time alignment system based on multimodal fusion includes an image sensor module, an inertial measurement unit (IMU), a high-performance computing unit, and a HUD display unit.

[0024] The image sensor module is installed at the front of the vehicle. The output of the image sensor module and the output of the inertial measurement unit (IMU) are both connected to the input of the high-performance computing unit (HUD). The output of the high-performance computing unit is connected to the input of the HUD display unit.

[0025] Preferably, the image sensor module is used for parallel transmission of real-time bus data and synchronous capture of frame sequence image data of the environment;

[0026] The inertial measurement unit (IMU) includes a three-axis accelerometer and a three-axis gyroscope. The IMU is used to acquire the vehicle's angular velocity and acceleration information in real time, and to provide high-frequency, short-term robust attitude information for pose estimation when pure visual information is insufficient or the vehicle is moving rapidly, so as to compensate for the limitations of visual algorithms and smooth the trajectory.

[0027] The high-performance computing unit is responsible for executing all algorithm modules such as image preprocessing, feature extraction and matching, pose estimation, map building and optimization, and virtual content rendering at high speed, ensuring high-efficiency and low-latency real-time computing;

[0028] The HUD display unit is used to receive virtual image data rendered by the computing unit and to accurately project the virtual image onto the windshield in front of the driver, thereby achieving a fusion display of virtual information and real scene.

[0029] Preferably, a real-time alignment method for an in-vehicle AR-HUD based on multimodal fusion includes the following steps:

[0030] Step S1: Initialize the system and establish a high-precision geometric calibration algorithm;

[0031] Step S2: Based on the high-precision geometric calibration algorithm, integrate real-time multimodal visual perception and stable attitude tracking algorithm;

[0032] Step S3: Based on the real-time multimodal visual perception and stable pose tracking algorithm, render virtual content with high precision and integrate with the real scene stabilization algorithm.

[0033] Preferably, step S1 includes the following steps:

[0034] Step S11: Calibrate the high precision of the intrinsic and extrinsic parameters of multiple sensors;

[0035] Step S12: Calibrate the HUD display and driver's eye point with high precision;

[0036] Step S13: Based on steps S11 and S12, construct a multi-coordinate system transformation model.

[0037] Preferably, step S2 includes the following steps:

[0038] Step S21: Algorithm for preprocessing multimodal image data;

[0039] Step S22: Based on the multimodal image data algorithm, extract and describe a real-time, efficient, and interference-resistant feature algorithm;

[0040] Step S23: Based on the real-time, efficient, and interference-resistant feature algorithm, match robust multimodal features with a high-precision 3D reconstruction algorithm;

[0041] Step S24: Based on robust multimodal features and high-precision 3D reconstruction algorithms, construct a real-time, high-precision, drift-free sensor attitude estimation and smooth tracking algorithm.

[0042] Preferably, S3 includes the following steps:

[0043] Step S31: Generate virtual content and management algorithm;

[0044] Step S32: Based on the virtual content and management algorithm, form a high-precision projection and anti-jitter rendering algorithm for virtual content.

[0045] Preferably, step S11 includes the following steps:

[0046] Step S111: Calibrate the intrinsic parameters of the left and right eye image sensors;

[0047] Step S112: Calibrate the extrinsic parameters of the left and right eye image sensors;

[0048] Step S113: Jointly calibrate multiple sensors and the vehicle system;

[0049] Step S114: Jointly calibrate the internal and external participating eye points of the AR glasses sensor module.

[0050] Preferably, step S23 includes the following steps:

[0051] Step S231: Quickly match stereo features;

[0052] Step S232: Verify RANSAC robustness;

[0053] Step S233: Based on steps S231 and S232, reconstruct a high-precision 3D model;

[0054] Step S234: Based on step S233, match inter-frame features with predicted motion;

[0055] Step S24 includes the following steps:

[0056] Step S241: Construct and optimize the local 3D map;

[0057] Step S242: Solve the real-time attitude using the PnP algorithm;

[0058] Step S243: Optimize the ICP algorithm and detect loop closures based on the real-time attitude of the PnP algorithm;

[0059] Step S244: Based on the ICP algorithm and loop closure detection, smooth the pose and apply filtering;

[0060] Step S245: Calculate the real-time transformation matrix based on the smoothed pose and filtering.

[0061] Preferably, step S32 includes the following steps:

[0062] Step S321: Using the rendering pipeline algorithm of OpenGL or a high-performance graphics rendering library, render the virtual graphics model into a two-dimensional image through this projection matrix;

[0063] Step S322: Based on the two-dimensional image, form a depth testing and precise occlusion processing algorithm;

[0064] Step S323: The rendered virtual image data stream is transmitted to the HUD display unit, which projects it onto the windshield, achieving high-precision, low-latency, and jitter-free visual overlay with the real scene observed by the driver through the windshield.

[0065] The beneficial effects of the multimodal fusion-based vehicle-mounted AR-HUD real-time alignment system and method of the present invention are as follows:

[0066] 1. This invention achieves high-precision real-time human eye positioning and ultimate alignment stability: This invention utilizes the OV9282 camera and IMU integrated into AR glasses for multi-sensor fusion processing. Employing pose smoothing and filtering algorithms, it effectively suppresses jitter caused by driver head movements, vehicle dynamics, and sensor noise. The system achieves positioning accuracy within 1ms, and within a positioning distance of 50cm, the positioning field of view (FOV) reaches -30 to 30 degrees, ensuring that virtual information highly matches the human eye's viewpoint, providing a drift-free and jitter-free ultimate real-time alignment effect. This greatly improves the stability, visual comfort, and information reliability of AR fusion displays.

[0067] 2. This invention achieves ultra-low latency AR fusion display: Through the tight integration of AR glasses sensors such as the OV9282 camera and IMU, and an efficient fusion positioning algorithm, this invention precisely controls the fusion positioning latency to within 3ms, which is significantly lower than the 25ms latency of traditional visual positioning. It also successfully achieves MTP (Motion-to-Photon) latency within 20ms, which greatly shortens the lag time from driver head movement to virtual image response.

[0068] 3. This invention effectively suppresses accumulated errors and achieves long-term drift-free human eye positioning: By optimizing the fusion positioning algorithm and supplementing it with possible local optimization and error correction mechanisms, this invention can effectively suppress the accumulated errors in human eye / head posture estimation. This ensures that AR glasses can continuously provide high-precision, drift-free human eye positioning information during long-term wear and use, thereby guaranteeing the long-term stability and accuracy of AR fusion display content and reducing the need for alignment calibration.

[0069] 4. This invention optimizes system integration and efficient computing, while balancing comfort and high performance: This invention directly integrates the key sensor camera OV9282 and IMU into the AR glasses, realizing direct and accurate perception of the driver's head movements and simplifying the layout of some complex sensors in the vehicle.

[0070] 5. The efficient fusion algorithm of this invention meets the requirements of ultra-low latency and achieves high-precision and high-robust alignment with limited computing resources, improving the overall integration efficiency, computing efficiency and user wearing comfort of the system, and achieving a perfect balance between high performance and practicality. Attached Figure Description

[0071] To more clearly illustrate the purpose, design concept, and innovation of the vehicle-mounted AR-HUD real-time alignment system and method based on multimodal fusion proposed in this invention, the invention will be described in detail below with reference to the accompanying drawings and tables.

[0072] Figure 1 This is a system composition diagram of the present invention.

[0073] Figure 2 This is the overall flowchart of the present invention. Detailed Implementation

[0074] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0075] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0076] This invention provides a real-time alignment system and method for in-vehicle AR-HUD based on multimodal fusion. This method fuses visual data from left and right eye image sensors and multimodal data from the in-vehicle IMU, and employs an optimized, efficient, robust, and stable algorithm to achieve accurate perception of real-world driving scenarios and real-time, high-precision estimation of sensor (camera) pose. This ensures that virtual information can be accurately superimposed and fused with the real scene with high precision, without jitter or drift.

[0077] System hardware composition

[0078] The system described in this invention (e.g.) Figure 1 The system architecture diagram shown includes the following main hardware modules, which work together to provide the necessary data input, computing platform, and output display for the operation of the algorithm of this invention:

[0079] Image sensor module: Contains at least one pair of high-resolution left and right eye image sensors (binocular cameras), such as CMOS image sensors. This module is mounted at the front of the vehicle (e.g., in front of the rearview mirror, above the windshield) and captures frame sequence image data (such as color or grayscale images) of the driving environment in real time and synchronously via a parallel data bus, serving as the main data input source for the core multimodal vision algorithm of this invention. Each camera has an internal clock synchronization mechanism to ensure strict synchronization of the left and right image frames.

[0080] Inertial Measurement Unit (IMU): Contains a three-axis accelerometer and a three-axis gyroscope. This module connects to the high-performance computing unit to acquire the vehicle's angular velocity and acceleration information in real time. Its data serves as auxiliary input to multimodal fusion algorithms, providing high-frequency, short-term robust attitude information for pose estimation when purely visual information is insufficient (e.g., missing texture, drastic lighting changes) or when the vehicle is moving rapidly. This compensates for the limitations of visual algorithms and smooths the trajectory.

[0081] High-performance computing unit: This module is the execution carrier for all the complex algorithms of this invention. It integrates a high-performance processor and a large-capacity high-speed memory, and connects to the sensor module and display unit through a high-speed interface. This unit has powerful parallel computing and data processing capabilities, and is responsible for executing all the algorithm modules described in this invention at high speed, including image preprocessing, feature extraction and matching, pose estimation, map construction and optimization, and virtual content rendering, ensuring high-efficiency, low-latency real-time computing.

[0082] HUD Display Unit: Serves as the final output platform for the real-time alignment results of this invention. This module receives virtual image data rendered by the computing unit via a high-speed video interface (such as DisplayPort, HDMI, or a dedicated LVDS interface). Internally, it includes a projection module (such as a Digital Micromirror Device (DMD) or a Liquid Crystal Display (LCoS)) and an optical reflection system, accurately projecting the virtual image onto the windshield in front of the driver, achieving a fusion display of virtual information and the real scene.

[0083] Core Algorithm Implementation Process

[0084] The real-time alignment method for vehicle-mounted AR-HUD based on multimodal fusion described in this invention has the following overall process: Figure 2 As shown, this is mainly achieved through the following three closely connected and collaborative algorithmic stages:

[0085] Phase 1: System Initialization and High-Precision Geometric Calibration Algorithm. This phase aims to establish precise and stable geometric and coordinate system relationships between the various components of the system, providing a high-precision reference for subsequent real-time alignment algorithms. This phase includes:

[0086] 1. High-precision calibration of internal and external parameters of multiple sensors:

[0087] (1) Intrinsic parameter calibration of left and right eye image sensors: Accurate intrinsic parameter calibration of the left and right eye cameras is performed using an offline calibration program and standard calibration boards based on optimization theory, such as checkerboard patterns or dot arrays. This algorithm solves for the intrinsic parameter matrix, including camera focal length, principal point coordinates, and radial and tangential distortion coefficients. and This step, achieved by minimizing reprojection error, is the first step in ensuring the geometric accuracy of all subsequent image-based computations and suppressing sources of distortion error.

[0088] (2) Calibration of extrinsic parameters of left and right eye image sensors: Similarly, through an offline calibration program, the relative rotation matrix between the left and right cameras is accurately solved using a binocular camera calibration algorithm. Translation vector These precise stereo baseline parameters are crucial inputs for high-precision depth recovery from subsequent multimodal visual data (binocular images).

[0089] (3) Joint calibration of multiple sensors and vehicle system: Using hand-eye calibration or multi-source sensor fusion calibration algorithm, the coordinates of the image sensor module and IMU module relative to the vehicle body coordinate system are accurately determined. The external attitude transformation matrix.

[0090] (4) Relative extrinsic parameter calibration and eye-point joint calibration within the AR glasses sensor module: Through an offline calibration program, the intrinsic parameter matrix and distortion coefficients of the internal camera (such as OV9282) of the AR glasses are accurately obtained, and the relative pose relationship between the internal camera of the AR glasses and the IMU sensor is accurately calibrated to form a high-precision, integrated head posture perception unit. Based on this, the precise geometric relationship between the internal sensor coordinate system of the AR glasses and the driver's eye-point and head coordinate system is established after the AR glasses are worn. This calibration determines the conversion method from the sensor data of the AR glasses themselves to the actual eye-point position of the driver, which is the key to achieving accurate correspondence and fixation between AR images and eye positions.

[0091] 2. High-precision calibration of HUD display and driver eye point: Using specialized calibration equipment (such as a laser tracker or a specific pattern projection and image acquisition system), the projection geometry model of the HUD display (including its position and orientation in the vehicle coordinate system, projection parameters, etc.) is precisely calibrated. Simultaneously, by establishing a dynamic driver eye point model based on ergonomics and the driver's seat position, a real-time reference benchmark is provided for the accurate projection of virtual content.

[0092] 3. Multi-coordinate system transformation model construction: Based on all the high-precision calibration results mentioned above, a multi-coordinate system transformation model is constructed and fixed during system initialization, transitioning from the world coordinate system. To vehicle coordinate system Then to the camera coordinate system and HUD display coordinate system and driver's eye coordinates A series of accurate and robust coordinate system transformation matrix algorithm models are used. These models are stored in matrix form and multiplied at runtime to achieve data transformation between different coordinate systems.

[0093] Phase Two: Real-time Multimodal Visual Perception and Stable Pose Tracking Algorithm. This phase is the core innovation of this invention in achieving real-time alignment and stable display. It fuses left and right eye image stream data and IMU data, and applies a series of efficient, robust, and stable algorithms optimized for the in-vehicle environment to perceive the environment in real time and accurately, without drift or jitter, estimate the system's pose in the world coordinate system. This phase includes:

[0094] 1. Multimodal image data preprocessing algorithm:

[0095] Image distortion correction: Correcting the distortion of raw left and right eye image frames acquired in real time. and Using the intrinsic parameter matrix obtained in stage one , Based on the distortion coefficients, an inverse mapping interpolation algorithm is used for efficient real-time image distortion correction, generating distortion-free images. and This eliminates the geometric distortion of the image itself, ensuring the accuracy of subsequent feature processing.

[0096] 2. Real-time, efficient, and interference-resistant feature extraction and description algorithm:

[0097] (1) Parallel feature detection: in the corrected left and right eye images and In this study, the FAST (Features from Accelerated Segment Test) algorithm is used in parallel to quickly detect key points in images. Due to its strategy of comparing pixel neighborhood gray values, the FAST algorithm is computationally efficient and robust to image noise and blur, making it very suitable for real-time in-vehicle environments.

[0098] (2) Feature descriptor generation: For the detected keypoints, the BRIEF (Binary Robust Independent Elementary Features) algorithm is used to generate compact and discriminative binary descriptors. The binary nature of BRIEF makes it fast in matching and also has a certain degree of robustness to changes in viewpoint.

[0099] 3. Robust multimodal feature matching and high-precision 3D reconstruction algorithm

[0100] (1) Fast stereo feature matching: FLANN (Fast Library for Approximate Nearest Neighbors) is used to match the left and right eye images. and The extracted feature descriptors are used for fast, near-nearest neighbor matching to obtain an initial set of corresponding points. FLANN accelerates the search through spatial partitioning structures such as kd-trees or k-means trees, ensuring real-time performance.

[0101] (2) RANSAC Robustness Verification: The key technology lies in introducing the RANSAC (Random Sample Consensus) algorithm to verify the geometric consistency of the initial matching results (e.g., based on epipolar geometric constraints or homography constraints). RANSAC, through iterative random sampling and model fitting, can effectively identify and eliminate most mismatched points caused by vehicle vibration, environmental noise, occlusion, or motion blur, ensuring the acquisition of high-confidence, accurate stereo matching pairs. This greatly improves the robustness of subsequent 3D reconstruction and pose estimation, and is the key to suppressing alignment jitter.

[0102] (3) High-precision 3D reconstruction: Using the calibrated intrinsic and extrinsic parameters of the left and right eye cameras, combined with the precise stereo matching pairs filtered by RANSAC, the 3D coordinates of these matching points in the camera coordinate system are accurately calculated through the triangulation algorithm, thereby constructing and updating the local sparse point cloud map in real time. This step is the core embodiment of the "multimodal fusion" of this invention, providing high-precision and highly robust scene depth information.

[0103] (4) Inter-frame feature matching and motion prediction: The current frame Features and the previous frame The features are matched (using both FLANN and RANSAC) to robustly estimate the relative motion of the camera between two frames.

[0104] 4. Real-time, high-precision, drift-free sensor attitude estimation and smooth tracking algorithm:

[0105] (1) Local 3D map construction and optimization: The newly reconstructed 3D point cloud is continuously integrated into the local sparse 3D map maintained by the system. In order to suppress the cumulative error, optimization algorithms such as local Bundle Adjustment (Local BA) can be used to perform online nonlinear optimization of local map points and keyframe poses to improve the accuracy and consistency of the map.

[0106] (2) PnP (Perspective-n-Point) algorithm to solve pose in real time: It uses 2D feature points in the current frame image and corresponding 3D points in the local 3D map (associated through inter-frame tracking or feature reprojection), combined with the intrinsic parameters of the left eye camera. The left-eye camera's coordinates in the world coordinate system are solved using an iteratively optimized PnP algorithm. Real-time, high-precision pose positioning The PnP algorithm is characterized by high efficiency and high accuracy when reliable 3D-2D correspondence points are known, and it is the core algorithm for real-time alignment in this invention.

[0107] (3) ICP (Iterative Closest Point) Algorithm Optimization and Loop Closure Detection: If the system can generate a relatively dense local point cloud, or detects visited areas (loop closures) during long-term operation, the ICP algorithm can be further used to align the current point cloud with the existing local map or keyframe point cloud with high precision. By detecting loop closures and performing global optimization (such as attitude map optimization), accumulated drift can be fundamentally eliminated, ensuring the drift-free operation of the system over a long period of time.

[0108] (4) Pose smoothing and filtering algorithm: For the pose sequence estimated in real time The algorithm performs time-domain smoothing filtering (e.g., using a low-pass filter, Kalman filter, or a more complex nonlinear smoother). This effectively suppresses high-frequency noise and occasional jitter, ensuring a smooth and continuous output attitude trajectory. This completely eliminates the jitter phenomenon of virtual information at the display end from the algorithmic level, greatly improving display stability.

[0109] (5) Real-time transformation matrix calculation: Based on the real-time estimated and smoothed camera pose And the coordinate system transformation relationships established in Phase 1 calibration, enabling real-time and accurate calculation of the transition from the world coordinate system. To the driver's eye point coordinate system Transformation matrix This matrix serves as the real-time basis for high-precision projection of virtual content.

[0110] Phase 3: Algorithm for stable fusion of high-precision rendering of virtual content and real-world scenes

[0111] 1. This stage utilizes accurately estimated and smoothed real-time pose information to accurately and smoothly render and overlay virtual content. This stage includes:

[0112] Virtual content generation and management algorithm: Based on the requirements of AR-HUD applications (such as navigation routes, vehicle speed, lane departure warnings, obstacle markers, etc.), the program dynamically generates or loads the corresponding three-dimensional or two-dimensional virtual graphic model data, and performs real-time management and updates.

[0113] 2. High-precision projection and anti-jitter rendering algorithm for virtual content:

[0114] Based on the smoothed real-time transformation matrix at the current moment By combining the projection parameters of the HUD display and the driver's eye point model, a correct perspective projection matrix is ​​constructed.

[0115] (1) Using the rendering pipeline algorithm of OpenGL or other high-performance graphics rendering libraries, the virtual graphics model is rendered into a two-dimensional image through this projection matrix. This rendering process works closely with the pose smoothing algorithm to ensure that the final projected virtual content does not exhibit jitter caused by pose estimation jitter.

[0116] (2) Depth Testing and Precise Occlusion Handling Algorithm: Utilizing high-precision pixel-level depth information of the real scene obtained from binocular vision reconstruction, precise depth testing is performed during the rendering process. This algorithm ensures that virtual objects and real-world objects correctly occlude with each other (e.g., virtual navigation lines are occluded by real vehicles), greatly enhancing the realism and immersion of AR fusion display, while avoiding abrupt changes caused by inaccurate depth perception.

[0117] (3) The rendered virtual image data stream is transmitted to the HUD display unit, which projects it onto the windshield to achieve high-precision, low-latency, and jitter-free visual superposition with the real scene observed by the driver through the windshield.

[0118] Alternative technical solutions:

[0119] Alternative for in-vehicle environmental perception and vehicle pose estimation (replacing binocular vision + in-vehicle IMU):

[0120] Pure monocular visual SLAM / VO:

[0121] Alternative: Use only one onboard camera for visual odometry or SLAM.

[0122] Limitations: It cannot directly obtain precise pixel-level depth information, resulting in limited 3D perception capabilities and difficulty in achieving accurate occlusion handling. It suffers from scale estimation drift and exhibits poor robustness in low-texture, fast-moving scenarios. The problem of accumulated error is more pronounced.

[0123] LiDAR + IMU Fusion:

[0124] Alternative solution: Use vehicle-mounted LiDAR to acquire high-precision 3D point cloud and depth information, and combine it with IMU for pose estimation.

[0125] Limitations: LiDAR is relatively expensive, and its performance may be affected by inclement weather (rain, snow, fog). Point cloud data has less density and texture information than images, and it also places a greater burden on computation, potentially making it less effective than image-based methods for matching certain visual features.

[0126] Example 1

[0127] Example scenario: A vehicle is driving on a city road, the driver is wearing AR glasses, and the system provides navigation prompts for the curve ahead.

[0128] 1. System startup and multimodal data acquisition:

[0129] (1) The system completes initialization based on the pre-defined high-precision calibration results.

[0130] (2) The vehicle-mounted binocular image sensor synchronously and in real time acquires images of the environment ahead.

[0131] (3) The AR glasses worn by the driver have internal sensors (OV9282 camera and IMU) that acquire the driver’s precise eye / head pose data in real time.

[0132] 2. Real-time multimodal perception and stable pose estimation algorithm operation:

[0133] (1) After distortion correction, the image data is used to extract features through FAST and BRIEF, and robust matching is performed using FLANN and RANSAC to ensure the cleanliness and reliability of the data.

[0134] (2) Based on these matches, the system accurately reconstructs the local three-dimensional scene and obtains high-precision depth information.

[0135] (3) Combining vehicle sensor data (and optional vehicle IMU) and driver eye pose data provided by AR glasses, the system uses the PnP / ICP algorithm to estimate the vehicle and human eye pose with high accuracy.

[0136] (4) This process incorporates error elimination mechanisms such as pose smoothing filtering, local optimization and loop closure detection to ensure that the pose trajectory is free from jitter and drift.

[0137] (5) Finally, the system calculates a stable and accurate real-time transformation matrix.

[0138] 3. High-precision AR rendering is stably integrated with real-world scenes for display:

[0139] (1) The system dynamically generates a virtual right-turn arrow model based on the navigation instructions.

[0140] (2) Based on the accurate vehicle pose and the human eye pose provided by the AR glasses, the system constructs the correct perspective projection matrix and uses the OpenGL rendering pipeline to render virtual content with anti-jitter.

[0141] (3) The depth test algorithm uses three-dimensional depth information to ensure that the virtual arrow is correctly occluded with real objects such as vehicles in front, just like it is actually "laid" on the road.

[0142] (4) Finally, the driver sees a precisely anchored, vibration-free, and drift-free virtual indicator projected onto the windshield by the HUD display unit. This is perfectly integrated with the real road conditions, greatly improving the intuitiveness of driving, the accuracy of information, and driving safety.

[0143] This invention aims to protect a real-time alignment system and method for in-vehicle AR-HUD based on multimodal fusion. Its key technical points and the areas for which protection are intended are as follows:

[0144] 1. High-precision and robust 3D perception method for vehicle environment: Utilize binocular vision and robust feature processing algorithms (such as FLANN and RANSAC) to achieve high-precision acquisition of scene depth information.

[0145] 2. Anti-interference real-time vehicle attitude stabilization tracking method: High accuracy and smooth continuity of vehicle attitude are ensured by using efficient pose solving algorithms (such as PnP / ICP) and pose smoothing filtering.

[0146] 3. A method for achieving ultra-high precision real-time positioning of the human eye using AR glasses: Innovatively integrating the camera and IMU data inside the AR glasses to achieve real-time positioning of the driver's eye with ultra-low latency (within 3ms) and ultra-high precision (within 1ms).

[0147] 4. Long-term alignment stability by eliminating accumulated errors: By combining map optimization and (optional) loop closure detection, pose accumulation errors are completely eliminated, achieving long-term drift-free alignment of the system.

[0148] 5. Precise Occlusion Rendering Method Based on Depth Information: Utilizes depth information from vehicle-mounted binocular cameras to achieve accurate spatial occlusion between virtual content and real objects.

[0149] 6. System-level collaborative architecture that balances high performance and real-time performance: Through efficient algorithm selection and multimodal data collaboration between the vehicle-mounted end and the AR glasses end, the overall system achieves high precision, high robustness and ultra-low latency (MTP within 20ms).

[0150] The definitions of abbreviations and key terms are shown in Table 1.

[0151] Table 1

[0152] abbreviation Full name Chinese definition HUD Head-Up Display Heads-up display W-HUD Windshield Head-Up Display Windshield head-up display system AR-HUD Augmented Reality Head-Up Display Augmented Reality Head-Up Display System SLAM Simultaneous Localization and Mapping Simultaneous localization and mapping PTAM Parallel Tracking and Mapping Parallel tracking and mapping methods (a type of SLAM) LSD-SLAM Large-Scale Direct Monocular SLAM Large-scale direct monocular SLAM ORB-SLAM Oriented FAST and Rotated BRIEF SLAM ORB feature-based SLAM system SIFT Scale-Invariant Feature Transform Scale-invariant feature transformation POI Point of Interest Points of interest (POIs) (commonly used in navigation and AR annotation) IMU Inertial Measurement Unit Inertial Measurement Unit ICP Iterative Closest Point Iterative Closest Point Algorithm (Registration) PnP Perspective-n-Point Perspective n-point pose solution method FLANN Fast Library for Approximate Nearest Neighbors Fast approximate nearest neighbor search library (for feature matching) RANSAC Random Sample Consensus Random sample consensus algorithm (used to remove false matches) MTP Motion-to-Photon Dynamic delay VO Visual Odometry Visual Odometry

Claims

1. A multi-modal fusion-based real-time alignment system for vehicle-mounted AR-HUD, characterized in that, The image sensor module, the inertial measurement unit (IMU), the high-performance computing unit, and the HUD display unit are included. The image sensor module is installed at the front of the vehicle, and the output ends of the image sensor module and the inertial measurement unit (IMU) are connected to the input end of the high-performance computing unit, and the output end of the high-performance computing unit is connected to the input end of the HUD display unit.

2. The multi-modal fusion based real-time alignment system for vehicle AR-HUD according to claim 1, wherein, The image sensor module is used for parallel transmission of real-time bus data and synchronous capture of frame sequence image data of the environment. The inertial measurement unit (IMU) includes a three-axis accelerometer and a three-axis gyroscope, which is used to obtain the angular velocity and acceleration information of the vehicle in real time, and to provide high-frequency and short-time robust attitude information for pose estimation when pure visual information is insufficient or the vehicle is moving fast, so as to compensate for the limitations of visual algorithms and smooth the trajectory. The high-performance computing unit is used to be responsible for high-speed execution of all algorithm modules such as image preprocessing, feature extraction and matching, pose estimation, map construction and optimization, and virtual content rendering, to ensure efficient and low-delay real-time calculation. The HUD display unit is used to receive the virtual image data rendered from the computing unit and accurately project the virtual image onto the windshield in front of the driver, to realize the fusion display of virtual information and real scene.

3. A multi-modal fusion-based real-time alignment method for vehicle-mounted AR-HUD, characterized in that, The method includes the following steps: Step S1: initializing the system and establishing a high-precision geometric calibration algorithm; Step S2: fusing real-time multi-modal visual perception and stable attitude tracking algorithms according to the high-precision geometric calibration algorithm; Step S3: high-precision rendering of virtual content and fusion of real scene stability algorithms according to the real-time multi-modal visual perception and stable attitude tracking algorithms.

4. The multi-modal fusion based real-time alignment method for vehicle AR-HUD according to claim 3, characterized in that, The step S1 includes the following steps: Step S11: calibrating the multi-sensor internal and external parameters with high precision; Step S12: calibrating the HUD display and the driver's eye point with high precision; Step S13: constructing a multi-coordinate system conversion model according to steps S11 and S12.

5. The multi-modal fusion based real-time alignment method for vehicle AR-HUD according to claim 3, wherein, The step S2 includes the following steps: Step S21: preprocessing multi-modal image data algorithm; Step S22: extracting and describing real-time efficient and anti-interference feature algorithm according to multi-modal image data algorithm; Step S23: matching robust multi-modal features and reconstructing high-precision three-dimensional algorithm according to real-time efficient and anti-interference feature algorithm; Step S24: constructing real-time high-precision, drift-free sensor pose estimation and smooth tracking algorithm according to robust multi-modal feature and reconstruction high-precision three-dimensional algorithm.

6. The multi-modal fusion based real-time alignment method for vehicle AR-HUD according to claim 3, wherein, The S3 includes the following steps: Step S31: generating virtual content and management algorithm; Step S32: forming virtual content high-precision projection and anti-shake rendering algorithm according to virtual content and management algorithm.

7. The multi-modal fusion based real-time alignment method for vehicle AR-HUD according to claim 4, wherein, The step S11 includes the following steps: Step S111: calibrating the left and right eye image sensor internal parameters; Step S112: calibrating the left and right eye image sensor external parameters; Step S113: jointly calibrating the multi-sensor and vehicle system; Step S114: jointly calibrating the AR glasses sensor module internal and relative external parameters and eye point.

8. The multi-modal fusion based real-time alignment method for vehicle AR-HUD according to claim 5, wherein, The step S23 includes the following steps: Step S231: fast matching of stereo features; Step S232: verifying RANSAC robustness; Step S233: according to step S231 and step S232, reconstructing high-precision three-dimensional; Step S234: according to step S233, matching inter-frame features and predicted motion; The step S24 includes the following steps: Step S241: constructing and optimizing local three-dimensional maps; Step S242: solving PnP algorithm real-time pose; Step S243: according to PnP algorithm real-time pose, optimizing ICP algorithm and detecting loop; Step S244: according to ICP algorithm and detecting loop, smoothing pose and filtering; Step S245: according to the smoothing pose and filtering, calculating real-time transformation matrix.

9. The multi-modal fusion based real-time alignment method for vehicle AR-HUD according to claim 6, wherein, The step S32 includes the following steps: Step S321: using the rendering pipeline algorithm of OpenGL or high-performance graphics rendering library, rendering the virtual graphics model into a two-dimensional image through the projection matrix; Step S322: according to the two-dimensional image, forming a depth test and accurate occlusion processing algorithm; Step S323: transmitting the rendered virtual image data stream to the HUD display unit, and projecting it onto the windshield by the HUD display unit, so as to realize high-precision, low-delay and non-jitter visual superposition with the driver observing the real scene through the windshield.

Images