AR-HUD calibration method, system, device and automobile
By performing high-precision calibration of the projected virtual image and physical world coordinates in AR-HUD, and using eye-position camera extrinsic parameters and thin spline interpolation formulas to correct distortion, the problem of the projected virtual image and the real scene not being able to be accurately matched is solved, achieving accurate fusion of the projected virtual image and the real scene and improving driving safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN DESAY SV AUTOMOTIVE CO LTD
- Filing Date
- 2023-09-25
- Publication Date
- 2026-07-07
AI Technical Summary
Existing technologies cannot achieve high-precision calibration between the physical world coordinates of the real scene in AR-HUD and the virtual image coordinates of the HUD projection screen, resulting in the virtual image and the real scene not being accurately matched, increasing driving hazards.
By taking a single shot with the HUD on to obtain the key point coordinates of the distortion-free projected virtual image, and then taking a second shot with the HUD off to obtain the physical coordinates of the calibration board, the physical world coordinates are calculated using the eye-position camera extrinsic parameters and the key point coordinates of the distortion-free projected virtual image. The transformation relationship between the projected virtual image and the physical world is established, and the image distortion is corrected using a thin spline interpolation formula.
It achieves precise fusion of the projected virtual image and the real scene, avoids the hidden dangers of blind driving, and builds an efficient human-computer interaction bridge.
Smart Images
Figure CN117252932B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of automotive display technology, specifically relating to an AR-HUD calibration method, system, device, and automobile. Background Technology
[0002] As automobiles become increasingly intelligent, they are evolving into intelligent entities capable of independently sensing their external environment. The information they need to convey to drivers is becoming increasingly diverse, making traditional dashboards insufficient for meeting the multi-dimensional human-machine interaction needs. Drivers prioritize driving, and frequently shifting their focus to the dashboard, center console, and mobile phone information can lead to numerous blind driving hazards. This has given rise to AR-HUD (Augmented Reality Head-Up Display), which seamlessly integrates virtual images with the real-world environment. On one hand, it allows drivers and passengers to intuitively perceive driving status, navigation, and ADAS information (such as lane departure warning) without shifting their gaze, significantly improving safety. On the other hand, with the continuous expansion of the imaging area, HUDs can further display entertainment information, enabling functions such as movie viewing and map push notifications, providing passengers with an immersive interactive experience and becoming a bridge connecting the virtual and real worlds.
[0003] To ensure that real-world ADAS information aligns perfectly with virtual images, a one-to-one correspondence between the coordinates of the real-world physical world and the coordinates of the virtual image on the HUD projection screen needs to be established. However, current technologies cannot achieve this point-to-point, high-precision calibration. Therefore, how to establish this one-to-one correspondence between the coordinates of the real-world physical world and the coordinates of the virtual image on the HUD projection screen remains an unsolved technical problem in the field of AR-HUD technology. Summary of the Invention
[0004] To address the aforementioned technical issues, this application proposes an AR-HUD calibration method, system, device, and vehicle.
[0005] Specifically, this application provides an AR-HUD calibration method, including:
[0006] S1: Turn on the HUD, take a picture, process the projected virtual image obtained from the picture, and obtain the key point coordinates of the projected virtual image after distortion removal;
[0007] S2: Turn off HUD, take a second shot, and obtain the actual physical coordinates of key points on the calibration board;
[0008] S3: Calculate the physical world coordinates corresponding to the distortion-free projected virtual image based on the external parameters of the eye position camera and the key point coordinates of the distortion-free projected virtual image;
[0009] S4: Based on the key point coordinates of the distorted projected virtual image and the corresponding physical world coordinates, calculate the conversion relationship between the two, and calibrate the image coordinate position corresponding to the projection screen according to the conversion relationship;
[0010] S5: Display the pattern at the image coordinates.
[0011] By calculating the key point coordinates of the distorted HUD projection virtual image and its corresponding physical world coordinates, a one-to-one correspondence between the two is obtained. Based on this one-to-one correspondence, the position of the HUD projection virtual image is determined, ensuring that the projection virtual image accurately matches the real scene.
[0012] Step S1 includes:
[0013] S101: Fix the position of the eye-position camera and calibration plate, and turn on the HUD;
[0014] S102: Use the eye-position camera to capture the projected virtual image of the HUD onto the calibration plate;
[0015] S103: Obtain the key point coordinates of the original image and the projected virtual image;
[0016] S104: Calculate the mapping relationship between the original image and the projected virtual image to perform distortion correction on the projected virtual image;
[0017] S105: Obtain the key point coordinates of the projected virtual image after distortion correction.
[0018] A virtual image is captured using an eye-position camera. The coordinates of the virtual image are obtained and a distortion correction operation is performed on it based on the original projection image to obtain the key point coordinates of the distorted virtual image. The physical world coordinates corresponding to the distorted virtual image are then calculated in step S3.
[0019] The mapping relationship between the original image and the projected virtual image calculated in S104 includes:
[0020] By substituting the keypoint coordinates of the original image and the keypoint coordinates of the projected virtual image into the thin spline interpolation formula, the values of the parameters in the thin spline interpolation formula are obtained.
[0021] Thin spline interpolation is a mathematical method for curve interpolation. Using thin spline interpolation to correct image distortion can accurately fit data points and correct the distortion of the projected virtual image with high precision, so that the distorted projected virtual image can be accurately integrated with the real scene environment.
[0022] Step S2 includes:
[0023] S201: Turn off the HUD and use the eye-position camera to photograph the calibration board to obtain the coordinates of key points on the calibration board;
[0024] S202: Measure the physical world coordinates of key points on the calibration plate;
[0025] S203: Perform external parameter calibration on the eye-position camera based on the coordinates of the key points on the calibration board and the physical world coordinates.
[0026] An eye-position camera is used to photograph a calibration board to obtain the coordinates of key points on the calibration board. Then, combined with the measured physical world coordinates of the key points on the calibration board, the eye-position camera is calibrated using external parameters to obtain the first rotation matrix and the first translation matrix. In step S3, the physical world coordinates corresponding to the projected virtual image after distortion removal are calculated.
[0027] Step S203 includes:
[0028] Substituting the coordinates of the key points on the calibration board and the physical world coordinates into the extrinsic parameter calibration formula, the extrinsic parameters from the world coordinates to the eye position camera are obtained; wherein, the extrinsic parameters include a first rotation matrix and a first translation matrix.
[0029] Step S3 includes:
[0030] Based on the first rotation matrix, the first translation matrix, and the key point coordinates of the distortion-free projected virtual image, the physical world coordinates corresponding to the distortion-free projected virtual image are calculated.
[0031] Step S4 includes:
[0032] Substituting the keypoint coordinates and corresponding physical world coordinates of the distorted projected virtual image into the extrinsic parameter calibration formula yields the second rotation matrix and the second translation matrix, thus obtaining the transformation relationship between the keypoint coordinates and the corresponding physical world coordinates of the distorted projected virtual image.
[0033] Specifically, this application also discloses an AR-HUD calibration system according to any one of claims 1-7, comprising:
[0034] The first processing module is used to process the projected virtual image obtained after a single shot to obtain the key point coordinates of the distorted projected virtual image.
[0035] The second processing module is used to obtain the actual physical coordinates of key points on the calibration board after the second shooting.
[0036] The third processing module is used to calculate the physical world coordinates corresponding to the distortion-free projected virtual image based on the external parameters of the eye position camera and the key point coordinates of the distortion-free projected virtual image.
[0037] The fourth processing module is used to calculate the conversion relationship between the key point coordinates of the distorted projected virtual image and the corresponding physical world coordinates.
[0038] Specifically, this application also discloses an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the computer program, when executed by the processor, implements the AR-HUD calibration method.
[0039] Specifically, this application also discloses a vehicle that includes an AR-HUD calibration system for implementing the AR-HUD calibration method.
[0040] Compared with the prior art, this application has at least the following beneficial effects:
[0041] The AR-HUD fusion display calibration system proposed in this application effectively removes distortion generated during the projection process. By obtaining the key point coordinates of the HUD projected virtual image after distortion removal and the corresponding physical world coordinates, the relationship between the physical world coordinates and the HUD projected virtual image coordinates is effectively and accurately calibrated. This enables the perfect fusion of the projected virtual image and the real road environment on the windshield of a car, avoiding the dangers of blind driving and building a new bridge for human-computer interaction. Attached Figure Description
[0042] Figure 1 This is a schematic diagram of the AR-HUD calibration method shown in the embodiments of this application.
[0043] Figure 2 This is a schematic diagram illustrating the specific process of step S1 of the AR-HUD calibration method shown in the embodiments of this application.
[0044] Figure 3 This is a schematic diagram illustrating step S2 of the AR-HUD calibration method shown in the embodiments of this application.
[0045] Figure 4 This is a schematic diagram of the AR-HUD calibration system structure shown in the embodiments of this application.
[0046] Figure 5 This is a schematic diagram of the AR-HUD calibration system structure shown in the third embodiment of this application.
[0047] Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation
[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0049] Example 1:
[0050] Reference Figure 1 This is a flowchart illustrating the AR-HUD calibration method in an embodiment of this application.
[0051] Specifically, this application provides an AR-HUD calibration method, including:
[0052] S1: Turn on the HUD, take a picture, process the projected virtual image obtained from the picture, and obtain the key point coordinates of the projected virtual image after distortion removal;
[0053] Specifically, the positions of the eye-position camera and the calibration plate are fixed, the HUD is turned on, and the eye-position camera is used to capture the projected virtual image of the HUD onto the calibration plate;
[0054] Extract key point information from the original projection image and the captured virtual projection image, where pi(x,y) represents the key points of the original projection image, and qi(x) represents the key points of the virtual projection image. h_distort ,y h_distort ) represents the key point of the projected virtual image;
[0055] Distortion is corrected using thin spline interpolation of the corresponding keypoints, and a loss function is constructed as follows: Where U = r 2 logr 2 , w = (w1, w2, ..., w n ) T ,a=(a1,a x ,a y ) T ,w1,w2,…,w n and a1,a x ,a y As parameters; by solving the linear equation, we find the vectors w and a, and obtain pi(x,y) and qi(x). h_distort ,y h_distort The mapping relationship;
[0056] Key points are extracted from the distortion-free HUD projection virtual image to obtain the key point coordinates (x, y). hud ,y hud ).
[0057] S2: Turn off HUD, take a second shot, and obtain the actual physical coordinates of key points on the calibration board;
[0058] Specifically, the HUD is turned off, and the eye-position camera is used to photograph the calibration board to obtain the coordinates (x, y) of key points on the calibration board. board ,y board );
[0059] The physical world coordinates (X and Y) of the key points on the calibration plate were measured. board ,Y board Z board );
[0060] Specifically, based on the coordinates (x, y) of the key points on the calibration board obtained from the steps... board ,y board The physical world coordinates (X and X) of key points on the calibration board. board ,Y board Z board ), calibrate the extrinsic parameters R and T from the world coordinates to the eye-position camera, as shown in the formula: Where M is the intrinsic parameter matrix of the camera, S is the scaling factor, R is the rotation matrix, and T is the translation matrix.
[0061] S3: Calculate the physical world coordinates corresponding to the distortion-free projected virtual image based on the external parameters of the eye position camera and the key point coordinates of the distortion-free projected virtual image;
[0062] Specifically, based on R and T calculated in step S2 and the key point coordinates (x, y) of the distortion-free projected virtual image... hud ,y hud ), through the formula: Calculate key points (x) hud ,y hud The physical world coordinates (X) hud ,Y hud Z hud ).
[0063] S4: Based on the key point coordinates of the distorted projected virtual image and the corresponding physical world coordinates, calculate the conversion relationship between the two, and calibrate the image coordinate position corresponding to the projection screen according to the conversion relationship;
[0064] Specifically, based on the (x) obtained in step S1 hud ,y hud ) and (X) obtained in step S3 hud ,Y hud Z hud ), calculate the transformation relationship R between world physical coordinates and the HUD projection screen. hud ,T hud For example, in the formula:
[0065] S5: Display the pattern at the image coordinates.
[0066] By calculating the key point coordinates of the distorted HUD projection virtual image and its corresponding physical world coordinates, a one-to-one correspondence between the two is obtained. Based on this one-to-one correspondence, the position of the HUD projection virtual image is determined, thus achieving precise alignment between the projection virtual image and the real scene.
[0067] Reference Figure 2 This is a detailed flowchart of step S1 of the AR-HUD calibration method shown in the embodiments of this application.
[0068] Step S1 includes:
[0069] S101: Fix the position of the eye-position camera and calibration plate, and turn on the HUD;
[0070] S102: Use the eye-position camera to capture the projected virtual image of the HUD onto the calibration plate;
[0071] S103: Obtain the keypoint coordinates of the original image and the projected virtual image:
[0072] Extract key point information from the original projection image and the captured virtual projection image, where pi(x,y) represents the key points of the original projection image, and qi(x) represents the key points of the virtual projection image. h_distort ,y h_distort ) represents the key point of the projected virtual image;
[0073] S104: Calculate the mapping relationship between the original image and the projected virtual image to perform distortion correction on the projected virtual image:
[0074] Specifically, due to assembly errors and the influence of the windshield, the projected virtual image is distorted to a certain extent. Distortion is corrected using thin spline interpolation of the corresponding key points in step S103, and a loss function is constructed as follows: Where U = r 2 logr 2 , w = (w1, w2, ..., w n ) T ,a=(a1,a x ,a y ) T ,w1,w2,…,w n and a1,a x ,a y As parameters; by solving the linear equation, we can find the vectors w and a, and thus obtain pi(x,y) and qi(x). h_distort ,y h_distort The mapping relationship;
[0075] S105: Obtain the key point coordinates of the projected virtual image after distortion correction:
[0076] Key points are extracted from the distortion-reduced HUD projection virtual image in step S104, and the key point coordinates (x, y, z) are obtained. hud ,y hud ).
[0077] A virtual image is captured using an eye-position camera. The coordinates of the virtual image are obtained and a distortion correction operation is performed on it based on the original projection image to obtain the key point coordinates of the distorted virtual image. The physical world coordinates corresponding to the distorted virtual image are then calculated in step S3.
[0078] The mapping relationship between the original image and the projected virtual image calculated in S104 includes:
[0079] By substituting the keypoint coordinates of the original image and the keypoint coordinates of the projected virtual image into the thin spline interpolation formula, the values of the parameters in the thin spline interpolation formula are obtained.
[0080] Specifically, due to assembly errors and the influence of the windshield, the projected virtual image is distorted to a certain extent. Distortion is corrected using thin spline interpolation of the corresponding key points in step S103, and a loss function is constructed as follows: Where U = r 2 logr 2 , w = (w1, w2, ..., w n ) T ,a=(a1,a x ,a y ) T ,w1,w2,…,w n and a1,a x ,a y As parameters; by solving the linear equation, we find the vectors w and a, and obtain pi(x,y) and qi(x). h_distort ,y h_distort The mapping relationship;
[0081] Thin plate spline interpolation is a commonly used interpolation method for estimating continuous functions between irregular data points. It is based on thin plate spline function theory and fits the data by imposing smoothing and bending constraints on the data points.
[0082] The key to thin spline interpolation is determining a smoothing parameter to control the smoothness of the fitted function. Generally, a larger smoothing parameter results in a smoother fitted function, while a smaller smoothing parameter makes the fitted function closer to the original data.
[0083] Thin spline interpolation can be performed on two-dimensional or higher-dimensional data. In this application, thin spline interpolation is applied to two-dimensional data to calculate the mapping relationship between the original image and the projected virtual image.
[0084] Thin spline interpolation is a mathematical method for curve interpolation. Using thin spline interpolation to correct image distortion can accurately fit data points and correct the distortion of the projected virtual image with high precision, so that the distorted projected virtual image can be accurately integrated with the real scene environment.
[0085] Reference Figure 3 This is a detailed flowchart of step S2 of the AR-HUD calibration method shown in the embodiments of this application.
[0086] Step S2 includes:
[0087] S201: Turn off the HUD and use the eye-position camera to photograph the calibration board to obtain the coordinates (x, y) of key points on the calibration board. board ,y board );
[0088] S202: Measure the physical world coordinates (x, y) of the key points on the calibration plate. board ,Y board Z board );
[0089] S203: Perform extrinsic parameter calibration on the eye-position camera based on the coordinates of the key points on the calibration board and the physical world coordinates.
[0090] Specifically, based on the coordinates (x, y) of the key points on the calibration board obtained in step S201... board ,y board The physical world coordinates (X and X) of the key points on the calibration board obtained in step S202. board ,Y board Z board ), calibrate the extrinsic parameters R and T from the world coordinates to the eye-position camera, as shown in the formula: Where M is the intrinsic parameter matrix of the camera, S is the scaling factor, R is the rotation matrix, and T is the translation matrix.
[0091] An eye-position camera is used to photograph a calibration board to obtain the coordinates of key points on the calibration board. Then, combined with the measured physical world coordinates of the key points on the calibration board, the eye-position camera is calibrated using external parameters to obtain the first rotation matrix and the first translation matrix. In step S3, the physical world coordinates corresponding to the projected virtual image after distortion removal are calculated.
[0092] Step S203 includes:
[0093] Substituting the coordinates of the key points on the calibration board and the physical world coordinates into the extrinsic parameter calibration formula, the extrinsic parameters from the world coordinates to the eye position camera are obtained; wherein, the extrinsic parameters include a first rotation matrix and a first translation matrix.
[0094] Specifically, based on the key point coordinates on the calibration board obtained in step S201 and the physical world coordinates of the key points on the calibration board obtained in step S202, the extrinsic parameters R and T from the world coordinates to the eye-position camera are calibrated, as shown in the formula: Where M is the intrinsic parameter matrix of the camera, S is the scaling factor, R is the rotation matrix, and T is the translation matrix.
[0095] Step S3 includes:
[0096] Based on the first rotation matrix, the first translation matrix, and the key point coordinates of the distortion-free projected virtual image, the physical world coordinates corresponding to the distortion-free projected virtual image are calculated.
[0097] Specifically, based on R and T calculated in step S203 and the key points (x) of the projected virtual image obtained after distortion removal in step S104... hud ,y hud ), through the formula: Calculate key points (x) hud ,y hud The physical world coordinates (X) hud ,Y hud Z hud ).
[0098] Step S4 includes:
[0099] Substituting the keypoint coordinates and corresponding physical world coordinates of the distorted projected virtual image into the extrinsic parameter calibration formula yields the second rotation matrix and the second translation matrix, thus obtaining the transformation relationship between the keypoint coordinates and the corresponding physical world coordinates of the distorted projected virtual image.
[0100] According to the (x) obtained in step S105 hud ,y hud ) and (X) obtained in step S3 hud ,Y hud Z hud ), calculate the transformation relationship R between world physical coordinates and the HUD projection screen. hud ,T hud For example, in the formula:
[0101] Specifically, when the physical world coordinates of ADAS are obtained, based on R obtained in step S105... hud ,T hud The image coordinates (x, y) corresponding to the projection screen can be calculated.show ,y show ), as in the formula: Where (X) ADAS ,Y ADAS Z ADAS (This provides the world physical coordinates for ADAS.)
[0102] Step S5 includes:
[0103] The corresponding projection pattern is drawn using OpenGL, thereby achieving the alignment of the virtual image with the real scene.
[0104] In this application, OpenCV is used to implement calibration steps S1-S5, and OpenGL is used to implement the rendering of the graphical interface.
[0105] Example 2:
[0106] Reference Figure 4 This is a schematic diagram of the AR-HUD calibration system structure shown in the embodiments of this application.
[0107] Specifically, this application also discloses an AR-HUD calibration system according to any one of claims 1-7, comprising:
[0108] The first processing module is used to process the projected virtual image obtained after a single shot to obtain the key point coordinates of the distorted projected virtual image.
[0109] Specifically, this application provides an AR-HUD calibration method, including:
[0110] S1: Turn on the HUD, take a picture, process the projected virtual image obtained from the picture, and obtain the key point coordinates of the projected virtual image after distortion removal;
[0111] Specifically, the positions of the eye-position camera and the calibration plate are fixed, the HUD is turned on, and the eye-position camera is used to capture the projected virtual image of the HUD onto the calibration plate;
[0112] Extract key point information from the original projection image and the captured virtual projection image, where pi(x,y) represents the key points of the original projection image, and qi(x) represents the key points of the virtual projection image. h_distort ,y h_distort ) represents the key point of the projected virtual image;
[0113] Distortion is corrected using thin spline interpolation of the corresponding keypoints, and a loss function is constructed as follows: Where U = r 2 logr 2 , w = (w1, w2, ..., w n ) T ,a=(a1,ax ,a y ) T ,w1,w2,…,w n and a1,a x ,a y As parameters; by solving the linear equation, we find the vectors w and a, and obtain pi(x,y) and qi(x). h_distort ,y h_distort The mapping relationship;
[0114] Key points are extracted from the distortion-free HUD projection virtual image to obtain the key point coordinates (x, y). hud ,y hud ).
[0115] The second processing module is used to obtain the actual physical coordinates of key points on the calibration board after the second shooting.
[0116] Specifically, the HUD is turned off, and the eye-position camera is used to photograph the calibration board to obtain the coordinates (x, y) of key points on the calibration board. board ,y board );
[0117] The physical world coordinates (X and Y) of the key points on the calibration plate were measured. board ,Y board Z board );
[0118] Specifically, based on the coordinates (x, y) of the key points on the calibration board obtained from the steps... board ,y board The physical world coordinates (X and X) of key points on the calibration board. board ,Y board Z board ), calibrate the extrinsic parameters R and T from the world coordinates to the eye-position camera, as shown in the formula: Where M is the intrinsic parameter matrix of the camera, S is the scaling factor, R is the rotation matrix, and T is the translation matrix.
[0119] The third processing module is used to calculate the physical world coordinates corresponding to the distortion-free projected virtual image based on the external parameters of the eye position camera and the key point coordinates of the distortion-free projected virtual image.
[0120] Specifically, based on R and T calculated in step S2 and the key point coordinates (x, y) of the distortion-free projected virtual image... hud ,y hud ), through the formula: Calculate key points (x) hud ,y hud The physical world coordinates (X) hud ,Y hud Z hud ).
[0121] The fourth processing module is used to calculate the conversion relationship between the key point coordinates of the distortion-free projected virtual image and the corresponding physical world coordinates.
[0122] Specifically, based on the (x) obtained in step S1 hud ,y hud ) and (X) obtained in step S3 hud ,Y hud Z hud ), calculate the transformation relationship R between world physical coordinates and the HUD projection screen. hud ,T hud For example, in the formula:
[0123] Example 3:
[0124] Reference Figure 5 This is a schematic diagram of the AR-HUD calibration system structure shown in the third embodiment of this application.
[0125] In order to implement the AR-HUD calibration method provided in the application, this embodiment provides an AR-HUD calibration system.
[0126] Furthermore, the entire system includes a communication module (MCU), an image processing module (SOC), and an image imaging unit (PGU) module. The communication module reads vehicle status information, including vehicle speed, steering, navigation coordinates from the vehicle's infotainment system, and ADAS information, via the vehicle's CAN bus. It interacts with the SOC through a serial peripheral interface (SPI). The SOC analyzes and processes the data, generates a display image, and outputs the image to the PGU module after pre-distortion processing. The light emitted from the image source is reflected through the optical path to form a virtual image directly on the windshield in front of the driver.
[0127] Reference Figure 5 The positioning module provides map navigation information to the communication module, the vehicle-related module provides vehicle information including speed and steering to the communication module, and the ADAS module provides ADAS information to the communication module. The image processing module uses the AR-HUD calibration method provided in this application to calibrate the display. The image processing module communicates with the HUD driver via LVDS (Low Voltage Differential Signaling). The HUD driver sends display information to the PGU module, and the PGU module transmits optical information to the optical / structural components.
[0128] The specific steps by which the system implements the AR-HUD calibration method provided in this application are as follows:
[0129] The MCU receives detection results from the vehicle system and ADAS via CAN communication and interacts with the SoC via SPI.
[0130] The SoC is responsible for graphics and image analysis and processing, outputting light graphics, and rendering the graphical interface. The image analysis and processing is implemented by OpenCV, and the rendering of the graphical interface is handled by OpenGL. Specifically, the graphics and image analysis and processing includes all the steps in the AR-HUD calibration method provided in this application.
[0131] Example 4:
[0132] Reference Figure 6 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.
[0133] Specifically, this application also discloses an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the computer program, when executed by the processor, implements the AR-HUD calibration method.
[0134] like Figure 6 As shown, the electronic device 6 of this embodiment includes: at least one processor 60 ( Figure 6 (Only one is shown in the diagram), memory 61, and computer program 62 stored in said memory 61 and executable on said at least one processor 60, wherein said processor 60 executes said computer program 62 to implement the steps in any of the above method embodiments.
[0135] The electronic device 6 is generally an in-vehicle infotainment system, which may be a basic entertainment system, a multimedia entertainment system, a navigation and entertainment system, a rear-seat entertainment system, or an in-vehicle connected entertainment system, etc. This electronic device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of electronic device 6 and does not constitute a limitation on electronic device 6. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.
[0136] The processor 60 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0137] In some embodiments, the memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. In other embodiments, the memory 61 may be an external storage device of the electronic device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 6. Furthermore, the memory 61 may include both internal and external storage units of the electronic device 6. The memory 61 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 61 can also be used to temporarily store data that has been output or will be output.
[0138] Example 5:
[0139] Specifically, this application also discloses a vehicle that includes an AR-HUD calibration system for implementing the AR-HUD calibration method.
[0140] In summary, this application provides an AR-HUD calibration method. The method involves turning on the HUD, taking a single photograph, processing the resulting projected virtual image to obtain the key point coordinates of the distortion-free projected virtual image, turning off the HUD, taking a second photograph, and obtaining the actual physical coordinates of the key points on the calibration board. Based on the extrinsic parameters of the eye-position camera and the key point coordinates of the distortion-free projected virtual image, the corresponding physical world coordinates of the distortion-free projected virtual image are calculated. Based on the key point coordinates of the distortion-free projected virtual image and the corresponding physical world coordinates, the conversion relationship between the two is calculated, and the image coordinate position corresponding to the projection screen is calibrated based on this conversion relationship. A pattern is displayed at the image coordinate position, thereby achieving a perfect fusion of the projected virtual image and the real road environment on the car's windshield, avoiding blind driving hazards, and building a new bridge for human-computer interaction.
[0141] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.
[0142] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0143] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.
[0144] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.
[0145] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0146] Although the description of this application has been made in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.
Claims
1. An AR-HUD calibration method, characterized in that, include: S1: Turn on the HUD, take a picture, process the projected virtual image obtained from the picture, and obtain the key point coordinates of the projected virtual image after distortion removal; S2: Turn off HUD, take a second shot, and obtain the coordinates of key points on the calibration board and physical world coordinates to perform extrinsic parameter calibration on the eye-position camera. The extrinsic parameters include the first rotation matrix and the first translation matrix. S3: Based on the first rotation matrix, the first translation matrix, and the key point coordinates of the distortion-free projected virtual image, calculate the physical world coordinates corresponding to the distortion-free projected virtual image; S4: Based on the key point coordinates of the distorted projected virtual image and the corresponding physical world coordinates, calculate the conversion relationship between the two, and calibrate the image coordinate position corresponding to the projection screen according to the conversion relationship; S5: Display the pattern at the image coordinates.
2. The AR-HUD calibration method according to claim 1, characterized in that, Step S1 includes: S101: Fix the position of the eye-position camera and calibration plate, and turn on the HUD; S102: Use the eye-position camera to capture the projected virtual image of the HUD onto the calibration plate; S103: Obtain the key point coordinates of the original image and the projected virtual image; S104: Calculate the mapping relationship between the original image and the projected virtual image to perform distortion correction on the projected virtual image; S105: Obtain the key point coordinates of the projected virtual image after distortion correction.
3. The AR-HUD calibration method according to claim 2, characterized in that, The mapping relationship between the original image and the projected virtual image calculated in S104 includes: By substituting the keypoint coordinates of the original image and the keypoint coordinates of the projected virtual image into the thin spline interpolation formula, the values of the parameters in the thin spline interpolation formula are obtained.
4. The AR-HUD calibration method according to claim 3, characterized in that, The external parameter calibration of the eye-position camera includes: Substituting the coordinates of the key points on the calibration board and the physical world coordinates into the extrinsic parameter calibration formula, we obtain the extrinsic parameters from the world coordinates to the eye-position camera.
5. The AR-HUD calibration method according to claim 4, characterized in that, Step S4 includes: Substituting the keypoint coordinates and corresponding physical world coordinates of the distorted projected virtual image into the extrinsic parameter calibration formula yields the second rotation matrix and the second translation matrix, thus obtaining the transformation relationship between the keypoint coordinates and the corresponding physical world coordinates of the distorted projected virtual image.
6. A system for AR-HUD calibration according to any one of claims 1-5, characterized in that, include: The first processing module is used to process the projected virtual image obtained after a single shot to obtain the key point coordinates of the distorted projected virtual image. The second processing module is used to obtain the actual physical coordinates of key points on the calibration board after the second shooting. The third processing module is used to calculate the physical world coordinates corresponding to the distortion-free projected virtual image based on the external parameters of the eye position camera and the key point coordinates of the distortion-free projected virtual image. The fourth processing module is used to calculate the conversion relationship between the key point coordinates of the distorted projected virtual image and the corresponding physical world coordinates.
7. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program that, when executed by the processor, implements the AR-HUD calibration method as described in any one of claims 1-5.
8. A car, characterized in that, The vehicle includes an AR-HUD calibration system, which is used to implement the AR-HUD calibration method according to any one of claims 1-5.