A method, apparatus, system and vehicle for ar-hud projection

Abstract

The application discloses a method, device, system and vehicle for AR-HUD projection. Lane line center line data is acquired based on an image of a road surface on which the vehicle is located, the real lane form and details of the vehicle in a close range are accurately restored, and the stability and reliability of underlying data are improved. The lane line center line data is fused with pre-acquired road shape data of a driving route to generate a complete lane line point set to be projected. The lane line center line on the visual image is relied on to ensure accuracy in a close range, and continuous geometric data support is provided by the road shape of the driving route in a far range, so that a coherent and non-offset lane line point set of a whole road section is generated. AR-HUD virtual guiding information is generated by using the lane line point set to be projected and is projected, the virtual image is accurately matched with the real road surface, full-scene lane-level guidance is realized, the guidance line offset and breakage are avoided, the driving risk caused by the driver's visual line switching is reduced, and the driving safety is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, system and vehicle for AR-HUD projection. Background Technology

[0002] Augmented Reality Head-Up Display (AR-HUD) devices can project virtual information onto the real road surface in front of the driver's field of vision, thereby reducing the need for drivers to look down at the device and improving driving safety.

[0003] In existing technologies, the projection results of AR-HUD depend on the accuracy of road surface data, but commercial high-precision road surface datasets are costly and have limited coverage. If the projection image is generated solely based on data collected by the vehicle itself, issues such as projection line misalignment and breakage can easily occur in road sections with sparse road features, making it difficult to meet the high-precision projection requirements of AR-HUD and thus posing driving safety hazards. Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method, apparatus, system, and vehicle for AR-HUD projection, enabling low-cost and high-precision AR-HUD projection.

[0005] This application discloses a method for AR-HUD projection, the method comprising: Lane centerline data is obtained based on road surface images; the road surface images are images of the road surface where the target vehicle is currently located. The lane centerline data is fused with the road shape data to obtain the set of lane line points to be projected; the road shape data is the shape data of the target vehicle's driving route that has been pre-acquired. Virtual guidance information is generated based on the set of lane line points to be projected, and then projected through an augmented reality head-up display (AR-HUD).

[0006] Optionally, acquiring lane centerline data based on road surface images includes: Extract the lane line feature point set from the road surface image; The lane centerline data is generated based on the lane feature point set.

[0007] Optionally, generating the lane centerline data based on the lane feature point set includes: Generate discrete points for the center line of the lane line based on each lane line feature point in the set of lane line feature points. Based on the clarity and continuity of each lane line feature point, calculate the confidence level of the discrete points of the lane line centerline generated by each lane line feature point; The lane centerline data is obtained by removing discrete points of the lane centerline with a confidence level lower than a preset value.

[0008] Optionally, fusing the lane centerline data with the road shape data to obtain the set of lane line points to be projected includes: Based on the curvature changes between road shape points in the road shape data, the road shape data is divided into straight road segments and turning segments; The straight road segment is fitted into a straight line, and the turning segment is fitted into multiple short straight lines; Obtain the distance from each discrete point of the lane centerline in the lane centerline data to the corresponding straight line; Using the distance as the registration error index, the discrete points of each lane centerline are registered with the straight line to obtain the set of lane line points to be projected.

[0009] Optionally, the step of using the distance as a registration error index to register the discrete points of each lane centerline with the straight line to obtain the set of lane line points to be projected includes: The weight of the discrete points of the lane centerline is obtained based on the confidence level of the discrete points and the distance attenuation term; the distance attenuation term is related to the distance of the discrete points of the lane centerline from the target vehicle. An error function is constructed based on the distance and the weights; With the goal of minimizing the error function, the discrete points of each lane centerline are registered with the straight line to obtain the set of lane line points to be projected.

[0010] Optionally, the step of registering the discrete points of each lane centerline with the straight line to obtain the set of lane line points to be projected, with the goal of minimizing the error function, includes: The rotation matrix and translation matrix are obtained by iterative calculation using singular value decomposition until the error function is minimized. The rotation matrix and the translation matrix are used to perform coordinate transformation on the lane centerline data to obtain the set of lane line points to be projected.

[0011] Optionally, generating virtual guidance information based on the set of lane line points to be projected for projection via an augmented reality head-up display (AR-HUD) includes: By combining the real-time location of the target vehicle and the turning points in the driving route, the set of lane line points to be projected is used to generate virtual guidance information corresponding to the real-time location; The virtual guidance information is aligned with the road surface in the real scene and projected using an AR-HUD.

[0012] Based on the above-mentioned AR-HUD projection method, this application also discloses an AR-HUD projection device, including: an acquisition unit, a fusion unit, and a generation unit; The acquisition unit is used to acquire lane centerline data based on a road surface image; the road surface image is an image of the road surface where the target vehicle is currently located. The fusion unit is used to fuse the lane centerline data with the road shape data to obtain a set of lane line points to be projected; the road shape data is the shape data of the target vehicle's driving route that has been pre-acquired. The generation unit is used to generate virtual guidance information based on the set of lane line points to be projected, so as to project it through an augmented reality head-up display (AR-HUD).

[0013] Optionally, the acquisition unit includes: An extraction subunit is used to extract the lane line feature point set from the road surface image; A sub-unit is generated to generate the lane centerline data based on the lane feature point set.

[0014] Optionally, the generating subunit includes: A discrete generation subunit is used to generate discrete points of the lane centerline based on each lane feature point in the lane feature point set. The calculation subunit is used to calculate the confidence level of the discrete points of the lane centerline generated by each lane feature point based on the clarity and continuity of each lane feature point. The elimination sub-unit is used to eliminate discrete points of the lane centerline with a confidence level lower than a preset value, thereby obtaining the lane centerline data.

[0015] Optionally, the fusion unit includes: The sub-unit is used to divide the road shape data into straight sections and turning sections based on the curvature changes between each road shape point in the road shape data. The fitting subunit is used to fit the straight road segment into a straight line and the turning segment into multiple short straight lines; The distance acquisition subunit is used to acquire the distance from each discrete point of the lane centerline in the lane centerline data to the corresponding straight line; The registration subunit is used to register the discrete points of each lane centerline with the straight line using the distance as the registration error index, so as to obtain the set of lane line points to be projected.

[0016] Optionally, the registration subunit includes: The weight acquisition subunit is used to acquire the weight of the discrete point of the lane centerline based on the confidence level of the discrete point of the lane centerline and the distance attenuation term; the distance attenuation term is related to the distance of the discrete point of the lane centerline from the target vehicle; Construct sub-units for building an error function based on the distance and the weights; The minimization subunit is used to register the discrete points of each lane centerline with the straight line with the goal of minimizing the error function, so as to obtain the set of lane line points to be projected.

[0017] Optionally, the minimized subunit includes: The iterative sub-unit is used to perform iterative calculations using the singular value decomposition method until the error function is minimized, thereby obtaining the rotation matrix and translation matrix. The transformation subunit is used to perform coordinate transformation on the lane centerline data using the rotation matrix and the translation matrix to obtain the set of lane line points to be projected.

[0018] Optionally, the generation unit includes: Combined with the sub-unit, it is used to combine the real-time position of the target vehicle and the turning points in the driving route to generate the virtual guidance information corresponding to the real-time position from the set of lane line points to be projected; The projection subunit is used to align the virtual guidance information with the road surface in the real scene and project it through AR-HUD.

[0019] Based on the above-mentioned AR-HUD projection method, this application also discloses an AR-HUD projection system for implementing the above method. The system is electrically connected to a visual recognition module, a data acquisition module, a point set fusion and registration module, and a guidance generation module. The visual recognition module is used to identify lane centerline data from a road surface image; the road surface image is an image of the road surface where the target vehicle is currently located. The data acquisition module is used to acquire road shape data; the road shape data is the shape data of the target vehicle's driving route. The point set fusion and registration module is used to fuse the lane centerline data with the road shape data to obtain the lane line point set to be projected. The guidance generation module is used to generate virtual guidance information based on the set of lane line points to be projected, and project it through an augmented reality head-up display (AR-HUD).

[0020] Based on the above-described AR-HUD projection method, this application also discloses a vehicle, including a processor and an augmented reality head-up display (AR-HUD) device, for implementing the above method.

[0021] This application discloses an AR-HUD projection method, device, system, and vehicle. Lane centerline data is acquired based on a road surface image of the vehicle's current location, accurately reproducing the real lane shape and details at close range, improving the stability and reliability of the underlying data. The lane centerline data is fused with pre-acquired road shape data of the driving route to generate a complete set of lane line points to be projected. This method does not rely on high-cost, limited-coverage high-precision lane data; it relies on lane centerlines in the visual image for accuracy at close range and on the road shape of the driving route for continuous geometric data support at distant range, enabling the generation of a coherent, unbiased set of lane line points across the entire road segment. The fused complete set of lane line points to be projected is used to generate and project AR-HUD virtual guidance information. The virtual guidance accurately matches the real road surface, achieving lane-level guidance across all scenarios, avoiding deviations and breaks in guidance lines, reducing the need for drivers to look down at the device, lowering the driving risk caused by changing gaze, and thus improving driving safety. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating an AR-HUD projection method disclosed in an embodiment of this application; Figure 2 This is a flowchart illustrating another AR-HUD projection method disclosed in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an AR-HUD projection device disclosed in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an AR-HUD projection system disclosed in an embodiment of this application. Detailed Implementation

[0024] AR-HUD relies on lane-level data to achieve virtual-real fusion lane guidance. Current mainstream solutions heavily depend on high-precision lane-level data, which is expensive to procure and only covers highways and urban expressways, making it unsuitable for ordinary urban and rural roads. Traditional multi-source point fusion uses the standard Iterative Closest Point (ICP) algorithm, which uses point-to-point distance as the registration error. In scenarios where road shape points are sparse, the registration deviation is large, and it cannot distinguish the reliability of visual recognition points, making it difficult to meet the needs of accurate near-field guidance and continuous information at a distance. If only vehicle-mounted vision is used to recognize lane lines, it is easily affected by lighting, lane marking wear, and occlusion interference, and the accuracy of long-distance lane recognition is insufficient, making it impossible to output a stable and continuous guidance benchmark. To overcome the aforementioned shortcomings, this application proposes a method for generating virtual guidance information that combines visual and driving route information. It acquires road surface images using an onboard forward-looking camera and extracts discrete points of lane centerlines with confidence levels. These points are then combined with a pre-acquired set of sparse shape points along the driving route. A dual optimization process of "point-to-line" registration and adaptive weighting is performed to accurately fuse the two types of data, resulting in a continuous and complete set of lane line points to be projected. Finally, virtual guidance information matching the real road is generated based on the fused points and projected onto an AR-HUD for display. This method achieves stable lane-level guidance across all scenarios without requiring high-precision lane data.

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Example 1: This application discloses a method for AR-HUD projection.

[0027] For details, please refer to Figure 1 The AR-HUD projection method disclosed in this embodiment includes the following steps: Step 101: Obtain lane centerline data based on road surface images.

[0028] As an feasible solution, the entity executing this step can be the visual recognition module in an AR-HUD projection system disclosed in this application, such as a vehicle-mounted forward-facing camera. In the method of this embodiment, this step is used to complete lane line feature extraction, lane line centerline point fitting, confidence assessment and invalid point filtering based on the original image of the road surface where the vehicle is currently located (i.e., the road surface in front of the vehicle) collected by the target vehicle, and output standardized lane line centerline data. This provides a near-range, high-precision visual data source for the subsequent point set fusion and registration process, and is a prerequisite step for realizing the fusion and registration of visual and road-level map information.

[0029] In this embodiment, the road surface image is defined as a real-time image of the road surface area in front of the vehicle, acquired under the current driving conditions of the target vehicle. The image acquisition area may include the left lane line, the right lane line, and the road surface base area of ​​the vehicle's current driving lane. After acquiring the road surface image, image preprocessing operations such as grayscale conversion, adaptive binarization, and Canny edge detection can be performed to weaken the influence of external interference factors such as ambient light, road stains, road shadows, and road water on the lane line edge features, strengthen the pixel gradient difference between the lane line and the road surface base, and improve the completeness and effectiveness of subsequent lane line feature point extraction.

[0030] In the method of this embodiment, after completing the road surface image preprocessing operation, the operation of extracting lane line feature point sets from the road surface image can be performed. As an feasible solution, based on the pixel gradient information of the preprocessed image, the coordinates of all edge pixels of the left and right lane lines in the image can be identified, and all identified lane line edge pixels can be uniformly collected to form a complete lane line feature point set. This lane line feature point set can include all pixels distributed on both sides of the lane line, and each lane line feature point carries basic attributes such as pixel clarity and cross-frame matching features. However, in actual working conditions, due to interference from road surface lane line wear, strong light backlight, obstruction by the vehicle in front, rain and snow, etc., the lane line feature point set contains a large number of blurred, broken, and abrupt invalid lane line feature points, which cannot be directly used for lane line centerline geometric fitting. Therefore, it is necessary to proceed to the subsequent fitting and filtering process.

[0031] In this embodiment, the method for generating lane centerline data based on lane feature point sets can specifically involve using lane feature point sets corresponding to the two edges of the lane line as input to solve for the lateral geometric center position of the lane line, thereby generating discrete lane centerline points. These discrete lane centerline points can fully characterize the lateral offset of the current driving lane, changes in road curvature, and other geometric features, serving as visual data for quantifying the lane's spatial position. Each discrete lane centerline point is associated with the image quality parameters of the corresponding original lane feature points, which are used for subsequent confidence quantification calculations.

[0032] As a feasible approach, in generating discrete points for lane centerlines, interpolation calculations can first be performed on each lane feature point in the feature point set to obtain the lateral geometric center position of the lane line as the discrete point for the lane centerline. Subsequently, two indices are obtained for each lane feature point: sharpness and continuity. These two indices are used to calculate the confidence level of the generated discrete points for the lane centerlines. Sharpness is the completeness and clarity of the lane markings quantified based on the pixel gradient magnitude of the lane feature points, while continuity is the stability quantified based on the matching and overlap of feature points at the same spatial location within multiple consecutive frames of road surface images.

[0033] As a feasible solution, the clarity and continuity indicators are weighted and fused according to preset fixed weights to obtain a confidence value ranging from 0 to 1. This allows for the assignment of confidence values ​​for discrete points on the lane centerlines, thus achieving a quantitative classification of the reliability of each lane centerline discrete point identification. Specifically, the confidence calculation formula is as follows: (1) In the formula, C represents the confidence level, S1 represents the sharpness score, and S2 represents the continuity score. 0.6 and 0.4 are the weights of the two indicators, which can be adjusted according to actual needs.

[0034] In the method of this embodiment, after obtaining the confidence values ​​of discrete points of each lane centerline, discrete points with confidence values ​​lower than a preset value can be removed to obtain the final lane centerline data. The preset value can be an internally pre-stored value determined through multi-scenario real-vehicle calibration, such as 0.5. As an feasible approach, all discrete points of the lane centerlines are traversed, and the confidence value corresponding to each point is compared with the preset value one by one. Discrete points of the lane centerlines with confidence values ​​lower than the preset value are identified as low-reliability recognition noise points and can be removed, not participating in subsequent calculations. These points typically correspond to lane line wear areas, obstacle occlusion areas, backlight distortion areas, and other recognition failure locations.

[0035] In addition to removing discrete lane centerline points with confidence levels below a preset value, the remaining points can be categorized according to their confidence levels. Specifically, points with confidence levels greater than or equal to a high confidence value (e.g., 0.8) are identified as high confidence points, which can serve as important benchmarks for lane centerline fitting and subsequent point set fusion and registration. Points with confidence levels greater than or equal to a preset value and simultaneously less than a high confidence value are identified as medium confidence points; these points only serve as auxiliary reference points in the fitting and registration calculations.

[0036] The lane centerline data output after this step retains only the stable and geometrically accurate discrete points of the lane centerlines. This effectively avoids visual recognition deviations caused by external environmental factors such as lighting, occlusion, and aging of road markings, forming a continuous and reliable close-range lane geometry data source. This provides standardized input for the subsequent point set fusion and registration module, meeting the requirements of AR-HUD lane-level guidance for accurate close-range alignment without geometric offset.

[0037] Step 102: Fuse the lane centerline data with the road shape data to obtain the lane line point set to be projected.

[0038] As an feasible solution, this step can be performed by the data acquisition module and the point set fusion registration module in the AR-HUD projection system disclosed in this application. In the method of this embodiment, this step segments and fits the pre-acquired road shape data, and performs point-to-line distance calculation, weighted registration, and coordinate transformation based on the lane centerline data output in step 101, finally outputting a continuous and complete set of lane line points to be projected, which supports the subsequent generation of AR-HUD lane-level guidance information, and solves the technical problem of poor fusion accuracy in traditional ICP algorithms that rely on point-to-point error registration and sparse map point set scenarios.

[0039] In the method of this embodiment, the road shape data used in this step consists of discrete points of road geometry that are pre-retrieved from a road-level map data source and match the current driving route of the target vehicle; these are known as road shape points. This road shape data contains only sparse road contour points and intersection turning points, eliminating the need for high-cost, high-precision lane-level map data with limited coverage. It can provide basic constraint information for long-distance road geometry trends, compensating for the insufficient accuracy of long-distance lane recognition in pure visual recognition.

[0040] In this embodiment, after inputting lane centerline data and road shape data, the curvature between every two road shape points in the road shape data can be obtained. Based on the change in curvature, the road shape data is divided into straight segments and turning segments. As one feasible approach, adjacent road shape points in the road shape data are traversed, and the curvature corresponding to the road segment between adjacent points is calculated group by group. The road segment type is then determined based on the change in curvature value. For example, for road segments between two adjacent road shape points, segments where the curvature value changes close to zero can be identified as straight segments, while segments where the curvature value changes beyond a calibrated threshold can be identified as turning segments. This segmentation operation enables differentiated processing of straight and curved road segments, adapting to complex road conditions such as urban roads and rural roads that include straight and turning intersections.

[0041] Subsequently, straight lines were fitted to each straight and turning segment separately. As a feasible approach, for the divided straight segments, the least squares method was used to uniformly fit all straight segments, generating a single complete straight line. For turning segments, due to the constraints of road curve geometry, a single straight line cannot reconstruct the curve's trajectory. Therefore, the entire turning segment was broken down into multiple short straight segments for segmented fitting. By piecing together these multiple straight segments, the road turning profile was approximately reconstructed, improving the accuracy of the road's geometric representation in the turning area and providing a standardized linear benchmark for subsequent point-to-line distance calculations.

[0042] In the method of this embodiment, after completing the straight line fitting operation, the distance from each discrete point of the lane centerline in the lane centerline data to the corresponding straight line is obtained. As an feasible solution, each discrete point of the lane centerline can be traversed, and the fitted straight line to which the spatial position of the discrete point belongs can be matched. The vertical distance from the discrete point of the lane centerline to the corresponding fitted straight line is solved based on the normalized general equation of the straight line. This vertical distance replaces the point-to-point Euclidean distance of the traditional ICP algorithm as the error index for multi-source point set registration, thereby solving the problem of large registration error caused by the sparse road shape point set.

[0043] The distance calculation formula can be expressed as follows: (2) In the formula, d i Let Ax be the distance from the i-th point to the fitted line. Ax + By + C = 0 is the equation of the fitted line in the world coordinate system, where A, B, and C are the coefficients of the line equation, satisfying A... 2 +B 2 =1. x i and y i Let be the coordinates of the i-th point in the world coordinate system.

[0044] In this embodiment, after obtaining the vertical distance from each lane centerline discrete point to the corresponding fitted straight line, the distance is used as the registration error index to register the lane centerline discrete points with the straight line. As an feasible approach, the weights of the lane centerline discrete points can be obtained based on their confidence level and a distance attenuation term. The distance attenuation term is related to the distance between the lane centerline discrete point and the target vehicle; for example, the closer the point is to the vehicle, the larger the distance attenuation term value, and the farther the point is from the vehicle, the smaller the distance attenuation term value. The distance attenuation term can be calculated based on the confidence level of each lane centerline discrete point.

[0045] In this embodiment, the registration weight of each lane centerline discrete point is calculated based on confidence level and distance attenuation. This weight is positively correlated with confidence level and negatively correlated with the distance from the lane centerline discrete point to the vehicle. This achieves an adaptive weighting mechanism that assigns higher weights to near-distance, high-confidence visual points and lower weights to far-distance, low-confidence points, meeting the requirements of accurate near-distance guidance and consistent long-distance guidance in AR-HUD. The formula for calculating this weight is as follows: (3) In the formula, ω i Let C be the weight of the i-th point. i Let be the confidence level of the i-th point, and e be the natural constant. λ is the distance decay constant, which can be set according to actual needs. e, λ, and d... i It constitutes the distance attenuation term.

[0046] As an example, for automotive-grade applications, λ is 50 meters. Therefore, when a point has a confidence level of 0.9 and is 20 meters from the vehicle, the distance decay term is 0.67, and the weight of that point is 0.60. When a point has a confidence level of 0.8 and is 40 meters from the vehicle, the distance decay term is 0.45, and the weight of that point is 0.36. When a point has a confidence level of 0.7 and is 60 meters from the vehicle, the distance decay term is 0.30, and the weight of that point is 0.21.

[0047] In the method of this embodiment, an error function can be constructed based on distance and weights. Specifically, it can be a weighted error function constructed using the perpendicular distance from the discrete points of each lane centerline to the corresponding fitted line and the weights calculated from the discrete points of each lane centerline as basic variables. This error function introduces rotation and translation matrices as transformation parameters to be solved. The distance errors of each point are weighted and corrected through weights, so that the errors of nearby high-confidence points have a higher influence in the error function, ensuring that the registration operation prioritizes the alignment with the lane geometry information recognized by near-field vision. The error function can be expressed as follows: (4) In the formula, E(R, t) is the error function introduced by the rotation matrix R and the translation matrix t, n is the number of points, and t is the error function introduced by the rotation matrix R and the translation matrix t. x and t y These are the components of the translation matrix in the x and y directions, respectively.

[0048] Subsequently, with the goal of minimizing the error function, the discrete points of the lane centerline are registered with the straight line. Specifically, this can be achieved by using the minimum value of the weighted error function as the constraint objective for iterative optimization, and continuously adjusting the parameters of the rotation matrix and translation matrix to complete the iterative registration of the visual point set with the fitted straight line of the road.

[0049] As a feasible approach, singular value decomposition (SVD) can be used for iterative calculations until the error function is minimized, yielding the rotation and translation matrices. Specifically, the weighted error function can be iteratively solved multiple times. In each iteration, the rotation and translation matrix parameters are updated, and the error function value is recalculated. This iterative process continues until the error function converges to a minimum. At this point, the optimal transformation parameters, i.e., the optimal rotation and translation matrices, are output.

[0050] Subsequently, the optimal rotation and translation matrices are used to transform the lane centerline data to obtain the set of lane line points to be projected. Specifically, the rotation and translation matrices obtained through iterative solutions are applied uniformly to all valid discrete lane centerline points within the lane centerline data, completing the translation and rotation of all discrete lane centerline points in the world coordinate system. After translation and rotation, the discrete lane centerline points are accurately aligned with the fitted straight lines in the road shape data, enabling smooth integration with the road shape data. This generates a complete and continuous set of lane line points to be projected, possessing both high-precision geometric features for near-field vision and constraints from long-distance road-level maps, serving as the standard input data source for the AR-HUD guidance generation module.

[0051] The lane line point set obtained after this step combines the advantages of high accuracy of vehicle vision recognition at close range with the advantages of long-distance road contour constraints of conventional road-level maps. Relying on point-to-line registration and adaptive weighted optimization algorithms, it significantly reduces the registration error of multi-source point sets. Even in scenarios such as ordinary urban roads and rural roads where there is no high-precision lane-level map coverage, it can still ensure the continuity and stability of lane geometry data, providing reliable underlying data support for virtual-real fusion AR-HUD lane-level guidance.

[0052] Step 103: Generate virtual guidance information based on the set of lane line points to be projected, and project it through an augmented reality head-up display (AR-HUD).

[0053] In the method of this embodiment, this step relies on the real-time location of the vehicle and the pre-stored turning points of the road-level map to complete the modeling of virtual guidance information. Then, through virtual-real fusion calibration, the virtual guidance is accurately matched with the real road scene. Finally, the image is projected and output through AR-HUD to realize lane-level driving assistance guidance with virtual-real fusion, thereby improving driving safety.

[0054] In this embodiment, by combining the real-time position of the target vehicle and the turning points in the driving route, virtual guidance information corresponding to the real-time position can be generated from the set of lane line points to be projected. Specifically, the real-time position coordinates of the target vehicle in the world coordinate system are obtained to calibrate the vehicle's current reference in the set of lane line points to be projected. At the same time, the coordinates of the turning points in the driving route from the road shape data are obtained. These turning points correspond to key road geometric nodes such as intersection turns and lane changes, and are the basis for generating turning guidance signs. Starting from the real-time position of the vehicle, continuous lane line points within the effective range in front of the vehicle in the set of lane line points to be projected are extracted, and a smooth and continuous dynamic lane guidance line is generated by fitting the coordinate sequence of the lane line points to be projected. At the same time, by combining the position and turning angle of each turning point in the driving route, matching turning arrows, lane change prompts, and other sign elements are superimposed at the corresponding spatial positions. The lane geometric contour and turning prompts are integrated and processed to generate standardized virtual guidance information that corresponds one-to-one with the vehicle's current real-time position and the direction of the road ahead, fully covering various road driving conditions such as going straight, turning left, and turning right.

[0055] Subsequently, as a feasible solution, the virtual guidance information is aligned with the road surface in the real scene and projected through an AR-HUD. Specifically, this can be achieved by establishing a virtual-real coordinate mapping relationship based on real-time images of the road surface captured by the vehicle's forward-facing camera, the vehicle's current attitude parameters, and the AR-HUD optical projection parameters. The generated virtual guidance information is then subjected to coordinate offset, scaling, and angle correction to eliminate the virtual-real misalignment problem caused by optical projection deviation. This ensures that elements such as virtual guide lines and turning arrows accurately conform to the lane positions on the real road surface, achieving spatial alignment between virtual guidance and the real road scene.

[0056] After completing the virtual-real calibration, the virtual guidance information is transmitted to the AR-HUD, which projects the image onto the driver's field of vision area on the windshield of the vehicle, forming an immersive virtual-real fusion display effect. The driver can directly observe continuous and unbiased lane-level guidance prompts within the real road field of vision without having to look down at the vehicle's instrument panel or central control screen, effectively reducing the burden caused by switching driving eyes.

[0057] In this embodiment, AR-HUD lane guidance can be achieved across all scenarios using only conventional maps and in-vehicle visual data, without being limited by the coverage of high-precision lane datasets. Simultaneously, the virtual-real alignment calibration mechanism ensures a high degree of matching between virtual guidance and the real road surface, with no deviation in guidance at close or long distances. This continuously provides drivers with stable and intuitive lane-level driving assistance prompts, comprehensively improving driving safety during vehicle operation.

[0058] The method described in this embodiment breaks through the reliance of traditional technologies on high-precision lane-level map data. It can complete the construction of lane-level data for all scenarios by relying only on road visual information combined with conventional road-level map data with wider coverage and lower cost. This expands the applicable scenarios of lane-level guidance from highways and urban expressways to ordinary urban roads and rural roads, significantly reducing the cost of implementing AR-HUD lane-level guidance functions. Furthermore, it adopts road segment fitting combined with the vertical distance from point to line as the registration error index to improve the registration accuracy of multi-source point sets in scenarios with sparse road shape point sets. At the same time, it introduces an adaptive weight allocation mechanism based on the confidence of discrete points on the visual lane line centerline and distance attenuation, prioritizing the role of close-range high-confidence visual points in the registration calculation. This adapts to the usage requirements of AR-HUD for accurate close-range guidance and no deviation in long-range guidance, significantly improving the overall accuracy and robustness of multi-source point set fusion.

[0059] Furthermore, the entire visual recognition module can complete lane line feature extraction using only a standard in-vehicle forward-facing camera, eliminating the need for expensive hardware such as LiDAR or high-precision positioning. This results in moderate performance requirements for the in-vehicle camera hardware, low deployment barriers, and excellent conditions for engineering implementation and mass production. Simultaneously, the computational logic within the solution, including road segmentation fitting, point-to-line error calculation, weight assignment, and singular value decomposition iterative solution, utilizes lightweight mathematical operations. The overall algorithm's computational load is controllable, allowing for rapid data processing on conventional in-vehicle embedded processors and high-performance chips. This enables real-time output of the lane line point set to be projected and virtual guidance information, meeting the real-time requirements of low-latency dynamic projection in AR-HUD. Finally, relying on the AR-HUD projection display method after virtual-real fusion calibration, lane-level virtual guidance information is superimposed onto the driver's real-world view. Drivers no longer need to frequently look down at the in-vehicle instrument panel or central navigation device, improving driving safety during vehicle operation.

[0060] Example 2: This application discloses another method for AR-HUD projection. Please refer to [link / reference]. Figure 2 This embodiment describes the process of generating virtual boot information.

[0061] Step 201: The target vehicle located on First Avenue acquires a road surface image in front of the vehicle and extracts a set of lane line feature points from the road surface image.

[0062] Step 202: Generate discrete points of lane centerlines based on the lane feature point set, and calculate the confidence level of each discrete point of lane centerlines.

[0063] Step 203: Determine if there are any discrete points on the lane centerline with a confidence level lower than a preset value. If yes, proceed to step 204. If no, proceed to step 205.

[0064] Step 204: Delete the discrete points of the lane centerline to obtain the lane centerline data. Proceed to Step 205.

[0065] Step 205: Calculate the weight of each lane centerline discrete point based on the confidence level and distance attenuation term. Proceed to Step 208.

[0066] Step 206: Obtain the driving route, which shows that the target vehicle goes straight on First Avenue, turns right onto Second Avenue after reaching the intersection of First Avenue and Second Avenue, and then goes straight on Second Avenue.

[0067] Step 207: Extract the road shape data of First Avenue, Second Avenue and their intersection, and perform straight line fitting on the straight and turning sections in the road shape data.

[0068] Step 208: Calculate the distance from each discrete point of the lane centerline in the lane centerline data to its corresponding straight line.

[0069] Step 209: Construct an error function based on the distance and weight of discrete points on the lane centerline.

[0070] Step 210: Minimize the error function to obtain the rotation matrix and translation matrix.

[0071] Step 211: Use rotation and translation matrices to translate and rotate the lane centerline data to obtain the set of lane line points to be projected.

[0072] Step 212: Generate virtual guidance information based on the set of lane line points to be projected, including virtual guidance information obtained by combining lane centerline data and road shape data on the first avenue, and virtual guidance information for intersections and the second avenue obtained mainly by road shape data.

[0073] Based on the AR-HUD projection method disclosed in the above embodiments, this embodiment correspondingly discloses an AR-HUD projection device. Please refer to... Figure 3 The AR-HUD projection device includes: an acquisition unit 301, a fusion unit 302, and a generation unit 303; The acquisition unit 301 is used to acquire lane centerline data based on a road surface image; the road surface image is an image of the road surface where the target vehicle is currently located. The fusion unit 302 is used to fuse the lane centerline data with the road shape data to obtain a set of lane line points to be projected; the road shape data is the shape data of the target vehicle's driving route that has been pre-acquired. The generation unit 303 is used to generate virtual guidance information based on the set of lane line points to be projected, so as to project it through an augmented reality head-up display (AR-HUD).

[0074] Optionally, the acquisition unit 301 includes: An extraction subunit is used to extract the lane line feature point set from the road surface image; A sub-unit is generated to generate the lane centerline data based on the lane feature point set.

[0075] Optionally, the generating subunit includes: A discrete generation subunit is used to generate discrete points of the lane centerline based on each lane feature point in the lane feature point set. The calculation subunit is used to calculate the confidence level of the discrete points of the lane centerline generated by each lane feature point based on the clarity and continuity of each lane feature point. The elimination sub-unit is used to eliminate discrete points of the lane centerline with a confidence level lower than a preset value, thereby obtaining the lane centerline data.

[0076] Optionally, the fusion unit 302 includes: The sub-unit is used to divide the road shape data into straight sections and turning sections based on the curvature changes between each road shape point in the road shape data. The fitting subunit is used to fit the straight road segment into a straight line and the turning segment into multiple short straight lines; The distance acquisition subunit is used to acquire the distance from each discrete point of the lane centerline in the lane centerline data to the corresponding straight line; The registration subunit is used to register the discrete points of each lane centerline with the straight line using the distance as the registration error index, so as to obtain the set of lane line points to be projected.

[0077] Optionally, the registration subunit includes: The weight acquisition subunit is used to acquire the weight of the discrete point of the lane centerline based on the confidence level of the discrete point of the lane centerline and the distance attenuation term; the distance attenuation term is related to the distance of the discrete point of the lane centerline from the target vehicle; Construct sub-units for building an error function based on the distance and the weights; The minimization subunit is used to register the discrete points of each lane centerline with the straight line with the goal of minimizing the error function, so as to obtain the set of lane line points to be projected.

[0078] Optionally, the minimized subunit includes: The iterative sub-unit is used to perform iterative calculations using the singular value decomposition method until the error function is minimized, thereby obtaining the rotation matrix and translation matrix. The transformation subunit is used to perform coordinate transformation on the lane centerline data using the rotation matrix and the translation matrix to obtain the set of lane line points to be projected.

[0079] Optionally, the generation unit 303 includes: Combined with the sub-unit, it is used to combine the real-time position of the target vehicle and the turning points in the driving route to generate the virtual guidance information corresponding to the real-time position from the set of lane line points to be projected; The projection subunit is used to align the virtual guidance information with the road surface in the real scene and project it through AR-HUD.

[0080] Based on the above-mentioned AR-HUD projection method, this application also discloses an AR-HUD projection system for implementing the above method. The system is composed of a visual recognition module 401, a data acquisition module 402, a point set fusion and registration module 403, and a guidance generation module 404 electrically connected together. The visual recognition module 401 is used to identify lane centerline data from a road surface image; the road surface image is an image of the road surface where the target vehicle is currently located. The data acquisition module 402 is used to acquire road shape data; the road shape data is the shape data of the driving route of the target vehicle. The point set fusion and registration module 403 is used to fuse the lane centerline data with the road shape data to obtain the lane line point set to be projected. The guidance generation module 404 is used to generate virtual guidance information based on the set of lane line points to be projected, and project it through an augmented reality head-up display (AR-HUD).

[0081] Based on the above-described AR-HUD projection method, this application also discloses a vehicle, including a processor and an augmented reality head-up display (AR-HUD) device, for implementing the above method.

[0082] The embodiments in this specification are described in a progressive manner. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant details can be found in the method section.

[0083] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0084] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0085] The features described in the embodiments of this specification can be substituted for or combined with each other, so that those skilled in the art can implement or use this application.

[0086] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for AR-HUD projection, characterized in that, include: Lane centerline data is obtained based on road surface images; the road surface images are images of the road surface where the target vehicle is currently located. The lane centerline data is fused with the road shape data to obtain the set of lane line points to be projected; the road shape data is the shape data of the target vehicle's driving route that has been pre-acquired. Virtual guidance information is generated based on the set of lane line points to be projected, and then projected through an augmented reality head-up display (AR-HUD).

2. The method according to claim 1, characterized in that, The process of acquiring lane centerline data based on road surface images includes: Extract the lane line feature point set from the road surface image; The lane centerline data is generated based on the lane feature point set.

3. The method according to claim 2, characterized in that, The process of generating the lane centerline data based on the lane feature point set includes: Generate discrete points for the center line of the lane line based on each lane line feature point in the set of lane line feature points. Based on the clarity and continuity of each lane line feature point, calculate the confidence level of the discrete points of the lane line centerline generated by each lane line feature point; The lane centerline data is obtained by removing discrete points of the lane centerline with a confidence level lower than a preset value.

4. The method according to claim 1, characterized in that, The process of fusing the lane centerline data with the road shape data to obtain the set of lane line points to be projected includes: Based on the curvature changes between road shape points in the road shape data, the road shape data is divided into straight road segments and turning segments; The straight road segment is fitted into a straight line, and the turning segment is fitted into multiple short straight lines; Obtain the distance from each discrete point of the lane centerline in the lane centerline data to the corresponding straight line; Using the distance as the registration error index, the discrete points of each lane centerline are registered with the straight line to obtain the set of lane line points to be projected.

5. The method according to claim 4, characterized in that, The process of registering discrete points of each lane centerline with the straight line using the distance as the registration error index to obtain the set of lane line points to be projected includes: The weight of the discrete points of the lane centerline is obtained based on the confidence level of the discrete points and the distance attenuation term; the distance attenuation term is related to the distance of the discrete points of the lane centerline from the target vehicle. An error function is constructed based on the distance and the weights; With the goal of minimizing the error function, the discrete points of each lane centerline are registered with the straight line to obtain the set of lane line points to be projected.

6. The method according to claim 5, characterized in that, The step of registering the discrete points of each lane centerline with the straight line to obtain the set of lane line points to be projected, with the goal of minimizing the error function, includes: The rotation matrix and translation matrix are obtained by iterative calculation using singular value decomposition until the error function is minimized. The rotation matrix and the translation matrix are used to perform coordinate transformation on the lane centerline data to obtain the set of lane line points to be projected.

7. The method according to claim 1, characterized in that, The step of generating virtual guidance information based on the set of lane line points to be projected, and projecting it through an augmented reality head-up display (AR-HUD), includes: By combining the real-time location of the target vehicle and the turning points in the driving route, the set of lane line points to be projected is used to generate virtual guidance information corresponding to the real-time location; The virtual guidance information is aligned with the road surface in the real scene and projected using an AR-HUD.

8. An AR-HUD projection device, characterized in that, include: Acquisition unit, fusion unit, and generation unit; The acquisition unit is used to acquire lane centerline data based on a road surface image; the road surface image is an image of the road surface where the target vehicle is currently located. The fusion unit is used to fuse the lane centerline data with the road shape data to obtain a set of lane line points to be projected; the road shape data is the shape data of the target vehicle's driving route that has been pre-acquired. The generation unit is used to generate virtual guidance information based on the set of lane line points to be projected, so as to project it through an augmented reality head-up display (AR-HUD).

9. A system for AR-HUD projection, characterized in that, For implementing the method of any one of claims 1-7, the system comprises a visual recognition module, a data acquisition module, a point set fusion and registration module, and a guidance generation module electrically connected together; The visual recognition module is used to identify lane centerline data from a road surface image; the road surface image is an image of the road surface where the target vehicle is currently located. The data acquisition module is used to acquire road shape data; the road shape data is the shape data of the target vehicle's driving route. The point set fusion and registration module is used to fuse the lane centerline data with the road shape data to obtain the lane line point set to be projected. The guidance generation module is used to generate virtual guidance information based on the set of lane line points to be projected, and project it through an augmented reality head-up display (AR-HUD).

10. A vehicle, characterized in that, Includes a processor and an augmented reality head-up display (AR-HUD) device for implementing the method of any one of claims 1-7.

Images