HUD projection imaging distortion-oriented correction method, system, electronic equipment and device

Abstract

The invention discloses a correction method, system, electronic equipment and device for HUD projection imaging distortion, and belongs to the technical field of image imaging, and the method comprises the steps: determining a test camera, calibrating the test camera, and obtaining camera lens distortion data and camera internal reference data; determining a test plane and an HUD device, obtaining a calibration image projected by the HUD device on the test plane as a first image through the test camera, and obtaining HUD lens distortion data through the first image; acquiring a calibration image projected by the HUD equipment on the windshield of the target vehicle as a second image through the test camera; obtaining distortion data of the windshield through the second image, and obtaining a distortion correction function by combining the distortion data of the camera lens, the internal reference data of the camera and the distortion data of the HUD lens; and inputting to-be-output image data into the distortion correction function to obtain a corrected image, and outputting the corrected image through the HUD device. The problem of image distortion caused by distortion of the HUD lens and distortion of the front baffle panel glass is solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image imaging, and particularly relates to a correction method, system, electronic equipment and device for HUD projection imaging distortion. BACKGROUND

[0002] The HUD is a projection system which is fixed to the workbench of the automobile, projects onto the glass of the front panel of the automobile, and is reflected into the human eye to combine the information such as navigation to be displayed with the actual road outside to form an augmented reality AR effect, which is the principle of AR-HUD. The scheme can only be fixed on the workbench of the automobile during automobile production, and the projection direction needs to be fixed. The display parameters are manually adjusted by the debugging personnel to roughly correct the complex double distortion formed by the HUD lens and the front panel glass. However, the shortcomings are also obvious:

[0003] 1. The debugging personnel can only observe through the human eye and adjust the approximate shape parameters (such as pincushion distortion), so the distortion in shape cannot be finely corrected.

[0004] 2. The angle of the HUD projection onto the glass of the front panel of the automobile is very strict.

[0005] 3. At the same time, the HUD must be embedded and fixed during production, thereby limiting the wide application of the HUD, especially the automobile which is originally produced without the HUD cannot be remanufactured.

[0006] Therefore, people need a flexible AR-HUD and measurement scheme which can achieve the following advantages:

[0007] 1. The AR-HUD can be placed on the produced automobile flexibly and can be used after simple setting;

[0008] 2. The complex distortion formed by the irregular pose and the front panel glass can be automatically corrected, so that the angle deviation tolerance of the HUD projection onto the glass of the front panel of the automobile is large. SUMMARY

[0009] The application develops a correction method, system, electronic equipment and device for HUD projection imaging distortion, and aims to solve the technical problem of double fusion of the HUD lens distortion and the distortion of the front panel glass which causes the distortion of the image observed by the driver in the prior art.

[0010] Technical scheme: in the first aspect, the application provides a correction method for HUD projection imaging distortion, comprising:

[0011] determining a test camera, calibrating the test camera, and acquiring camera lens distortion data and camera intrinsic parameter data;

[0012] determining a test plane and a HUD device, acquiring, by the test camera, a calibration image of the HUD device projected on the test plane as a first image, and acquiring, by the first image, HUD lens distortion data;

[0013] acquiring, by the test camera, the calibration image of the HUD device projected on the windshield of a target vehicle as a second image;

[0014] acquiring, by the second image, windshield distortion data, and acquiring, in combination with the camera lens distortion data, camera intrinsic data, and HUD lens distortion data, a distortion correction function;

[0015] inputting image data to be output into the distortion correction function, acquiring a corrected image, and outputting by the HUD device.

[0016] In some embodiments, the step of acquiring, by the first image, HUD lens distortion data comprises:

[0017] acquiring a first relative pose of the test camera and the center position of the first image, and a second relative pose of the center position of the first image and the axis center of the HUD device;

[0018] acquiring, based on the camera lens distortion data, the camera intrinsic data, the first relative pose, and the second relative pose, HUD lens distortion data.

[0019] In some embodiments, the HUD lens distortion data is acquired using a homogeneous transformation method, and a representation formula thereof comprises:

[0020] Pfinal = DISTcamera(INM * EXM3 * EXM4 * DISThud(Pidea));

[0021] wherein Pfinal is feature data of the first image; DISTcamera is the camera lens distortion data; INM is the camera intrinsic data; EXM3 is the first relative pose; EXM4 is the second relative pose; and Pidea is feature data of an ideal image of the HUD device.

[0022] In some embodiments, further comprising:

[0023] acquiring, by the second image, extrinsic data of the HUD device relative to the second image and extrinsic data of the second image relative to the test camera.

[0024] In some embodiments, a representation formula of the distortion correction function comprises:

[0025] Pidea = invMd(Pfinal);

[0026] wherein, Pidea is the corrected HUD image; invMd is the inverse mapping of Md, Md = DISTcamera(INM*EXM2*DISTforeg(EXM1*DISThud)), DISTcamera is the camera lens distortion data, INM is the camera intrinsic data, EXM1 is the extrinsic data of the HUD device relative to the second image, EXM2 is the extrinsic data of the second image relative to the test camera, DISTforeg is the windshield distortion data, and DISThud is the HUD lens distortion data; Pfinal = DISTcamera(INM*EXM2*DISTforeg(EXM1*DISThud(Pidea))).

[0027] In some embodiments, the calibration is performed by Zhang Zhengyou chessboard correction method.

[0028] In some embodiments, the calibration image comprises a chessboard image of Zhang Zhengyou chessboard correction method, a red image, a blue image, and a green image.

[0029] In the second aspect, embodiments of the present application also provide a correction system for HUD projection imaging distortion, comprising:

[0030] a camera calibration module, configured to determine a test camera and calibrate the test camera to obtain camera lens distortion data and camera intrinsic data;

[0031] a HUD calibration module, configured to determine a test plane and a HUD device, obtain a calibration image as a first image by projecting the HUD device on the test plane through the test camera, and obtain HUD lens distortion data through the first image;

[0032] a calibration projection module, configured to obtain the calibration image as a second image by projecting the HUD device on a windshield of a target vehicle through the test camera;

[0033] a correction function obtaining module, configured to obtain windshield distortion data through the second image, and obtain a distortion correction function in combination with the camera lens distortion data, the camera intrinsic data, and the HUD lens distortion data;

[0034] a projection correction module, configured to input image data to be output into the distortion correction function, obtain a corrected image, and output through the HUD device.

[0035] In a third aspect, the embodiments of the present application further provide an electronic device comprising a memory and a processor;

[0036] The memory is configured to store a computer program.

[0037] The processor is configured to implement the method for correcting the HUD projection imaging distortion when executing the computer program.

[0038] In a fourth aspect, the embodiments of the present application further provide a correction device for HUD projection imaging distortion, which is applied to the method for correcting the HUD projection imaging distortion as described in any one of the first aspect, and the correction device comprises a test camera, an HUD device, a test platform and a controller.

[0039] The test camera, the HUD device and the test platform are configured to acquire a first image by the test camera, the first image being a calibration image projected by the HUD device on the test platform, and acquire HUD lens distortion data by the first image.

[0040] The test camera, the HUD device and the controller are configured, the controller being in communication connection with the test camera and the HUD device, to acquire a second image by the test camera, the second image being the calibration image projected by the HUD device on the windshield of a target vehicle, acquire windshield distortion data by the second image, and acquire a distortion correction function by combining the camera lens distortion data, the camera intrinsic parameter data and the HUD lens distortion data, the distortion correction function being stored in the controller.

[0041] The HUD device and the controller are configured to be connected, to input the to-be-output image data into the distortion correction function, acquire a corrected image, and output the corrected image by the HUD device.

[0042] Beneficial effects: compared with the prior art, the method for correcting the HUD projection imaging distortion provided by the embodiment of the application comprises the following steps: determining a test camera, calibrating the test camera, obtaining camera lens distortion data and camera intrinsic data; determining a test plane and a HUD device, obtaining a calibration image projected by the HUD device on the test plane as a first image through the test camera, and obtaining HUD lens distortion data through the first image; obtaining a calibration image projected by the HUD device on the windshield of a target vehicle as a second image through the test camera; obtaining windshield distortion data through the second image, and obtaining a distortion correction function in combination with the camera lens distortion data, the camera intrinsic data and the HUD lens distortion data; inputting to-be-output image data into the distortion correction function, obtaining a corrected image, and outputting through the HUD device. The windshield distortion data and the HUD lens distortion data are fused to obtain a distortion correction function for correcting the HUD output data, then the to-be-output image of the HUD is pre-corrected and adjusted, and finally the ideal image after correction is output through the HUD, so that the image distortion problem caused by the HUD lens distortion and the windshield distortion is solved. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0044] Figure 1 The step flow chart of the method for correcting the HUD projection imaging distortion provided by the embodiment of the application;

[0045] Figure 2 The step flow chart of the method for correcting the HUD projection imaging distortion provided by the embodiment of the application;

[0046] Figure 3 The module connection diagram of the correction system for the HUD projection imaging distortion provided by the embodiment of the application;

[0047] Figure 4 The structural schematic diagram of the electronic device provided by the embodiment of the application;

[0048] Figure 5 The structural diagram of the test plane 3, the HUD device 2 and the test camera 1 when the HUD lens distortion data is obtained in the method for correcting the HUD projection imaging distortion provided by the embodiment of the application;

[0049] Figure 6The structural diagram of windshield distortion is obtained in the method for correcting HUD projection imaging distortion provided in the embodiments of this application.

[0050] Reference numerals: 1. Test camera; 2. HUD device; 3. Test plane; 4. Windshield; 5. USB Bluetooth adapter; 10. Camera calibration module; 20. HUD calibration module; 30. Calibration projection module; 40. Correction function acquisition module; 50. Projection correction module. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0052] A head-up display (HUD) is a projection system fixed to a car's dashboard, projecting onto the windshield and reflecting into the viewer's eyes. It combines navigation and other information with actual road conditions to create an augmented reality (AR) effect – this is the principle behind AR-HUD. Currently, this approach requires fixing the HUD to the car's dashboard during production, maintaining a fixed projection direction, and then manually adjusting display parameters based on the viewer's perspective to roughly correct the complex double distortion caused by the HUD lens and the windshield. However, its drawbacks are also quite significant:

[0053] 1. Debugging personnel can only observe with the naked eye and adjust the general shape parameters (such as pincushion distortion), so they cannot finely correct the shape distortion.

[0054] 2. The angle requirements for the HUD projection onto the windshield of the car are very strict.

[0055] 3. At the same time, because it must be embedded and fixed during production, the widespread application of HUD is limited. In particular, cars that were not originally produced with HUD cannot be recycled and modified.

[0056] Therefore, a flexible AR-HUD and measurement solution is needed that can achieve the following advantages:

[0057] 1. It can be flexibly placed on a pre-produced car and can be used after simple setup;

[0058] 2. It can automatically correct the complex distortions formed by irregular postures and the windshield glass, thus having a large tolerance for the angle deviation of the HUD projection onto the windshield glass of the car.

[0059] Therefore, the embodiment of the present application provides a correction method for HUD projection imaging distortion, which comprises determining a test camera 1, calibrating the test camera 1, obtaining camera lens distortion data and camera intrinsic data; determining a test plane 3 and a HUD device 2, obtaining a calibration image projected by the HUD device 2 on the test plane 3 as a first image through the test camera 1, and obtaining HUD lens distortion data through the first image; obtaining a calibration image projected by the HUD device 2 on a windshield 4 of a target vehicle as a second image through the test camera 1; obtaining windshield distortion data through the second image, and obtaining a distortion correction function in combination with the camera lens distortion data, the camera intrinsic data and the HUD lens distortion data; inputting to-be-output image data into the distortion correction function, obtaining a corrected image, and outputting through the HUD device 2. The distortion data of the windshield 4 and the distortion data of the HUD lens are fused to obtain a distortion correction function for correcting the HUD output data, then the to-be-output image of the HUD is pre-corrected and adjusted, and finally the ideal image after correction is output through the HUD, thereby solving the image distortion problem caused by the HUD lens distortion and the windshield distortion.

[0060] The present application refers to the knowledge and professional translation vocabulary in the book "Robotics Machine Vision and Control MATLAB Algorithm Foundation" ((Australia) Corke, P. wrote; Liu Rong translated). According to the knowledge in the reference book, the main data modules in the process of classical camera correction include:

[0061] (1) Intrinsic matrix INM:

[0062]

[0063] INM represents the inherent parameters of the camera, which are fixed at the time of production. Among them, p w , p h are the width and height of the image element in the camera, f is the focal length of the camera, and (u0, v0) is the principal point coordinate - the principal point is the intersection of the image plane and the optical axis. By presetting the camera parameter matrix as K, the equation of the intrinsic matrix INM can be written as:

[0064]

[0065] In the embodiment of the present application, it is a camera for Bluetooth communication when testing, and it is replaced by the user's eye when normally used.

[0066] (2) Extrinsic matrix EXM:

[0067] EXM represents the position relationship between the camera and the target, which is the expression of the target deviating from the camera coordinate system. It mainly includes 3 variables of translation and 3 variables of rotation, totaling 6.

[0068] This matrix refers to the spatial relationship between the image (target) projected on the glass of the front panel of the car and the camera (camera) of the Bluetooth communication during the test; during normal use, it refers to the spatial relationship between the target and the user's eye (camera).

[0069] (3) Lens distortion DISTcamera:

[0070] Geometric distortion is usually the most important problem encountered in camera calibration, which includes two parts: radial and tangential. Radial distortion causes image points to shift along radial lines from the principal point.

[0071] Radial error can be expressed by a polynomial approximation:

[0072] δr=k1r 3 +k2r 5 +k3r 7 +…;

[0073] Or

[0074] δr=k1r 2 +k2r 4 +k3r 6 +…;

[0075] Where r is the distance between the image point and the principal point. In the specific embodiment of the present application, 2, 4, 6, and 3 parameters are used. There is no order after 8.

[0076] Tangential distortion occurs in the direction perpendicular to the radius, which is usually not as severe as radial distortion.

[0077] After distortion, a point (u, v) that should have been projected on the plane becomes (u d ,v d ) after distortion, and the distorted coordinates are given by

[0078] u d = u + δ u ;

[0079] v d = v + δ v ;

[0080]

[0081] For different (u, v) values, the displacement vector can be drawn, and the displacement required for different points in the image to correct the distortion, i.e. -δ u , -δ v . That is:

[0082] u = ud -δ u ;

[0083] v=v d -δ v ;

[0084] Common barrel distortion occurs when the magnification decreases as one moves away from the principal point, causing straight lines near the edge of the image to curve outward.

[0085] Pincushion distortion occurs when the magnification increases as one moves away from the principal point, causing straight lines near the edge of the image to curve inward.

[0086] In the classic image correction, the homogeneous transformation method such as Zhang Zhengyou chessboard correction method is adopted, and the camera distortion, external parameter and internal parameter are calculated through multiple image acquisition; the internal parameter can be provided by the supplier when the camera is purchased, or can be calculated by an algorithm. Therefore, the camera lens distortion DISTcamera and the internal parameter matrix INM can be calculated by the Zhang Zhengyou chessboard method, and generally will not change as long as the camera is not actively disassembled. The external parameter changes according to the scene, so the camera lens distortion DISTcamera and the internal parameter matrix INM can be obtained.

[0087] In the application scenario faced by the present application, an ideal point Pidea on a HUD is subjected to the distortion DISThud of the HUD lens to obtain Phud; and is projected onto an irregular car front panel glass to present Pforeg, so the distortion parameter DISThud of the HUD lens is added here. In addition, when reaching the camera with Bluetooth communication (actually used as the driver's eye) from the windshield, Pfinal is obtained, which also includes a distortion parameter DISTforeg of the windshield. In addition, the external parameter EXM1 of the HUD relative to the projection part of the windshield, and the external parameter EXM2 of the projection part relative to the camera (eye) part with Bluetooth communication.

[0088] Therefore, the overall formula is written as:

[0089] Phud=DISThud(Pidea);

[0090] Pforeg=DISTforeg(EXM1*Phud);

[0091] Pfinal=DISTcamera(INM*EXM2*Pforeg);

[0092] Among them, the camera lens distortion DISTcamera and the internal parameter matrix INM are obtained by the classic homogeneous transformation method such as Zhang Zhengyou chessboard correction method, and will not be described here.

[0093] Then, in the application scenario in the present application, we need to calculate is the distortion parameter DISTforeg of the windshield, the distortion parameter DISThud of the HUD lens, the external parameter EXM1 of the HUD relative to the projection part of the windshield, and the external parameter EXM2 of the projection part relative to the camera (human eye) part with Bluetooth communication.

[0094] Continue to analyze, EXM1 is to express the relative pose of HUD and the projected surface, and EXM2 is to express the relative pose of the projected surface and the camera (actually used as the driver's eye) with Bluetooth communication, that is, each of them is an external parameter matrix including three variables of translation and three variables of rotation.

[0095] The distortion DISTforeg of the glass of the front windshield of the car is essentially consistent with the principle of the camera lens distortion DISTcamera, but the main source is to reflect the tangential distortion at different angles. Similarly, the principle of the lens distortion DISThud of the HUD is consistent with the principle of the camera DISTcamera, so the writing method of the parameter matrix is also the same structure.

[0096] In some embodiments, please refer to Figure 1 , Figure 1 The step flow chart of the correction method for the HUD projection imaging distortion provided by the embodiments of the present application is implemented by steps 100 to 500.

[0097] Step 100: determine the test camera 1 and calibrate the test camera 1 to obtain camera lens distortion data and camera internal parameter data.

[0098] Specifically, the Zhang Zhengyou chessboard correction method of homogeneous transformation is used for the test camera 1 alone, the camera lens distortion data and the camera internal parameter data are calculated by multiple image acquisition of the Zhang Zhengyou chessboard picture.

[0099] Step 200: determine the test plane 3 and the HUD device 2, obtain the calibration image of the HUD device 2 projected on the test plane 3 as the first image by the test camera 1, and obtain the HUD lens distortion data by the first image.

[0100] In some embodiments, please refer to Figure 2 , Figure 2 The step flow chart of the correction method for the HUD projection imaging distortion provided by the embodiments of the present application is implemented by steps 100 to 500.

[0097] Step 100: determine the test camera 1 and calibrate the test camera 1 to obtain camera lens distortion data and camera internal parameter data.

[0098] Specifically, the Zhang Zhengyou chessboard correction method of homogeneous transformation is used for the test camera 1 alone, the camera lens distortion data and the camera internal parameter data are calculated by multiple image acquisition of the Zhang Zhengyou chessboard picture.

[0099] Step 200: determine the test plane 3 and the HUD device 2, obtain the calibration image of the HUD device 2 projected on the test plane 3 as the first image by the test camera 1, and obtain the HUD lens distortion data by the first image.

[0100] In some embodiments, please refer to Figure 2 , Figure 2 The step flow chart of the correction method for the HUD projection imaging distortion provided by the embodiments of the present application is implemented by steps 100 to 500.

[0101] Step 210: obtaining a first relative pose of the test camera 1 with respect to a first image center position, and a second relative pose of the first image center position with respect to an axis center of the HUD device 2.

[0102] Step 220: obtaining HUD lens distortion data based on the machine lens distortion data, the camera intrinsic parameter data, the first relative pose and the second relative pose.

[0103] Specifically, the HUD lens distortion data is obtained by using the homogeneous transformation method, and the representation formula thereof includes:

[0104] Pfinal=DISTcamera(INM*EXM3*EXM4*DISThud(Pidea));

[0105] wherein, Pfinal is the feature data of the first image; DISTcamera is the camera lens distortion data; INM is the camera intrinsic parameter data; EXM3 is the first relative pose; EXM4 is the second relative pose; and Pidea is the feature data of the ideal image of the HUD device.

[0106] Specifically, please refer to Figure 5 , Figure 5 The structure diagram of the test plane 3, the HUD device 2 and the test camera 1 when obtaining the HUD lens distortion data in the correction method for the HUD projection imaging distortion provided by the embodiment of the present application is shown in the figure. The Zhang Zhengyou chessboard picture is projected onto the HUD device 2 by using the HUD device 2, the image is collected by the test camera 1, a large number of matched Pfinal and Pidea feature points are obtained, and then the homogeneous transformation method of the Zhang Zhengyou chessboard correction method is used to calculate the HUD lens distortion. The test plane 3 can be a common office table.

[0107] Step 300: obtaining, by the test camera 1, a calibration image projected by the HUD device 2 on the windshield 4 of the target vehicle as a second image.

[0108] Step 400: obtaining windshield distortion data by the second image, and obtaining a distortion correction function in combination with the camera lens distortion data, the camera intrinsic parameter data and the HUD lens distortion data.

[0109] Further, the external parameter data of the HUD device 2 with respect to the second image and the external parameter data of the second image with respect to the test camera 1 are obtained by the second image, and then the representation formula of the distortion correction function includes:

[0110] Pidea=invMd(Pfinal);

[0111] Pfinal = DISTcamera(INM*EXM2*DISTforeg(EXM1*DISThud(Pidea))).

[0112] In detail, refer to Figure 6 , Figure 6 The structure diagram for obtaining windshield distortion in the correction method for HUD projection imaging distortion provided by the embodiment of the present application is shown in the following figure. In the automobile environment, based on the following complete formula, the same HUD device 2 is used to project Zhang Zhengyou chessboard pictures, and the same homogeneous transformation method is used to calculate the distortion DISTforeg of the front windshield of the automobile, as well as EXM1 and EXM2.

[0113] Phud = DISThud(Pidea).

[0114] Pforeg = DISTforeg(EXM1*Phud).

[0115] Pfinal = DISTcamera(INM*EXM2*Pforeg).

[0116] Through the above formula, it is obtained that

[0117] Pfinal = DISTcamera(INM*EXM2*DISTforeg(EXM1*DISThud(Pidea))).

[0118] Then, all the right side formulas are integrated to obtain Md, that is, there is

[0119] Pfinal = Md(Pidea).

[0120] wherein, Md = DISTcamera(INM*EXM2*DISTforeg(EXM1*DISThud)).

[0121] Then, the inverse mapping of Md is calculated to obtain invMd.

[0122] At this moment, the inverse is obtained as

[0123] Pidea = invMd(Pfinal);

[0124] Then, Pfinal can be adjusted so that Pidea is the ideal pattern.

[0125] Step 500: input the image data to be output into the distortion correction function, obtain a corrected image, and output the corrected image through the HUD device 2.

[0126] In some embodiments, the present application adopts Zhang Zhengyou chessboard correction method for calibration, and the calibration image includes a chessboard image, a red image, a blue image and a green image of the Zhang Zhengyou chessboard correction method.

[0127] It can be understood that the embodiments of the present application provide a correction method for HUD projection imaging distortion, which includes determining a test camera 1, calibrating the test camera 1, obtaining camera lens distortion data and camera intrinsic data; determining a test plane 3 and a HUD device 2, obtaining a calibration image projected by the HUD device 2 on the test plane 3 as a first image through the test camera 1, and obtaining HUD lens distortion data through the first image; obtaining a calibration image projected by the HUD device 2 on a windshield 4 of a target vehicle as a second image through the test camera 1; obtaining windshield distortion data through the second image, and obtaining a distortion correction function in combination with the camera lens distortion data, the camera intrinsic data and the HUD lens distortion data; inputting image data to be output into the distortion correction function, obtaining a corrected image, and outputting the corrected image through the HUD device 2. The present application fuses the distortion data of the windshield 4 and the distortion data of the HUD lens, obtains a distortion correction function for correcting the HUD output data, then pre-corrects and adjusts the image to be output by the HUD, and finally outputs the corrected ideal image through the HUD, thereby solving the image distortion problem caused by the distortion of the HUD lens and the windshield.

[0128] Correspondingly, the embodiments of the present application also provide a correction system for HUD projection imaging distortion. Please refer to Figure 3 , Figure 3 The module connection diagram of the correction system for HUD projection imaging distortion provided by the embodiments of the present application, the correction system for HUD projection imaging distortion provided by the embodiments of the present application includes:

[0129] A camera calibration module 10, which is used to determine a test camera 1, calibrate the test camera 1, obtain camera lens distortion data and camera intrinsic data;

[0130] The HUD calibration module 20 is configured to determine the test plane 3 and the HUD device 2, acquire, by the test camera 1, a calibration image projected by the HUD device 2 on the test plane 3 as a first image, and acquire, by the first image, the HUD lens distortion data;

[0131] The calibration projection module 30 is configured to acquire, by the test camera 1, a calibration image projected by the HUD device 2 on the windshield 4 of the target vehicle as a second image;

[0132] The correction function acquisition module 40 is configured to acquire, by the second image, the windshield distortion data, and acquire, in combination with the camera lens distortion data, the camera intrinsic parameter data and the HUD lens distortion data, the distortion correction function;

[0133] The projection correction module 50 is configured to input the to-be-output image data into the distortion correction function, acquire a corrected image, and output, by the HUD device 2.

[0134] Correspondingly, the embodiment of the present application further provides an electronic device, please refer to Figure 4 , Figure 4 The electronic device provided by the embodiment of the present application includes a memory and a processor, and a structure schematic diagram of the electronic device provided by the embodiment of the present application is shown in the figure.

[0135] The memory is configured to store a computer program.

[0136] The processor is configured to, when executing the computer program, implement the correction method for the HUD projection imaging distortion provided by the embodiment of the present application.

[0137] Correspondingly, the embodiment of the present application further provides a correction device for the HUD projection imaging distortion, which is applied to the correction method for the HUD projection imaging distortion provided by the embodiment of the present application.

[0138] The correction device for the HUD projection imaging distortion includes a test camera 1, a HUD device 2, a test platform and a controller.

[0139] The test camera 1, the HUD device 2 and the test platform are configured to acquire, by the test camera 1, a calibration image projected by the HUD device 2 on the test plane 3 as a first image, and acquire, by the first image, the HUD lens distortion data.

[0140] The test camera 1, the HUD device 2 and the controller are configured, the controller is in communication connection with the test camera 1 and the HUD device 2, and is used for acquiring a calibration image projected by the HUD device 2 on the windshield 4 of the target vehicle as a second image through the test camera 1; windshield distortion data is acquired through the second image, and a distortion correction function is acquired in combination with camera lens distortion data, camera intrinsic parameter data and HUD lens distortion data and is stored in the controller;

[0141] The HUD device 2 and the controller are configured, and the HUD device 2 and the controller are connected, and are used for inputting to-be-output image data into the distortion correction function, acquiring a corrected image, and outputting through the HUD device 2.

[0142] Specifically, the HUD device 2 has a Bluetooth communication device, which is used to project the content to be displayed onto the glass of the front windshield panel of the automobile to form the function of AR-HUD, and the data source can be provided by the central control of the automobile through the Bluetooth channel, or can be provided by the controller (as a user's mobile phone) in the application. The HUD device 2 is fixed on the support 1. The HUD device 2 is a customized product of Nanjing Bota Visual Technology Co., Ltd., and the data source is a mobile phone.

[0143] The support 1 is a 3-degree-of-freedom manual adjustment holder, one side of which can be stuck to the central control console of the automobile through an adhesive material, and the other side is used to fix the HUD device 2, thereby fixing the HUD device 2 on the central control console of the automobile and manually adjusting the pose of the HUD device 2 relative to the glass of the front windshield panel of the automobile. The support 1 is customized, has a PVC structure, and the minimum angle of adjustment is 1 degree. Meanwhile, the support 1 has a fixed-size two-dimensional code with laser marking, which faces away from the glass of the front windshield panel of the automobile and faces toward the driver, and can be observed by the test camera 1 (the test camera 1 in the application is a Bluetooth communication camera).

[0144] The function of the Bluetooth communication camera is to capture different patterns displayed by the Bluetooth communication HUD device 2 under the instruction of the mobile phone from the position of the driver's eyes to the glass of the front windshield panel of the automobile, and the patterns are reflected after being projected onto the glass of the front windshield panel of the automobile by the Bluetooth communication HUD device 2 and are observed by the Bluetooth communication camera. The Bluetooth communication camera is fixed on the support 2. The Bluetooth communication camera is a Huawei P70 mobile phone.

[0145] The support 2 is a 3-degree-of-freedom manual adjustment holder, one side of which can be fixed on the driving position through a structural member, and the other side is used to fix the Bluetooth communication camera, thereby fixing the Bluetooth communication camera on the driving position of the automobile and manually adjusting the pose of the Bluetooth communication camera relative to the glass of the front windshield panel of the automobile. The support 2 is customized, has a PVC structure, and the minimum angle of adjustment is 1 degree.

[0146] The application also provides a USB Bluetooth adapter 5, which has two functions, one is to solve the problem that the general mobile phone cannot control multiple Bluetooth devices in multiple channels when the application is calibrated, and the other is to solve the problem that the HUD occupies the only Bluetooth channel of the user's mobile phone when the AR-HUD is used. The USB Bluetooth adapter 5 is powered by the car's own power converter and can be placed in the glove compartment of the car's co-pilot. The application uses a common 4-channel Bluetooth adapter, which can be physically connected to the mobile phone (which can be the mobile phone used in daily life) through the USB port.

[0147] The mobile phone receives traffic data of remote software through traffic, such as prompt information that needs to be displayed by navigation software, and sends data to the HUD device 2 with Bluetooth communication through Bluetooth to control the display content. It receives the shooting data of the camera with Bluetooth communication and provides computing resources to the software part of the application. In the specific embodiment of the application, the algorithm is installed in the form of APP on the mobile phone, and the mobile phone uses Huawei P70.

[0148] During the test, the working steps are as follows:

[0149] Step A-1. The user places the car in a simulated night driving environment, because people rely more on HUD display at night than in the daytime.

[0150] Step A-2. Prompt the user to establish physical contact between the mobile phone and the USB Bluetooth adapter 5 through the USB port by manual operation. Then the mobile phone establishes contact with the HUD device 2 with Bluetooth communication and the camera with Bluetooth communication, and prompts the user with the contact information.

[0151] Step A-3. Prompt the user to install the support 1 on the car's center console and install the HUD device 2 on the support 1 by manual operation. Then manually adjust the support 1 so that the HUD device 2 is roughly opposite the glass of the car's front panel.

[0152] Step A-4. Prompt the user to install the support 2 on the driver's position and install the camera with Bluetooth communication on the support 2 by manual operation. Then manually adjust the support 2 so that the camera with Bluetooth communication is roughly opposite the glass of the car's front panel.

[0153] Step A-5. Wait until the user controls the test process to start. The mobile phone projects the Zhang Zhengyou chessboard correction method chessboard image, pure red, blue, green image on the glass of the car front panel through the HUD, and then instructs the camera with Bluetooth communication to take pictures and transmit the pictures to the mobile phone, thereby completing the shooting of a picture. At the same time, the laser coded fixed size two-dimensional code on the support 1 is also shot.

[0154] Among them, the Zhang Zhengyou chessboard correction method chessboard image is multiple, which will shoot pictures with the chessboard center in the image center but rotating 3 degrees from 0 to 360 degrees, a total of 120 pictures; In addition, a total of 100 pictures without rotation and equidistantly translated to the left and right edges of the image will also be shot; A total of 100 pictures without rotation and equidistantly translated to the top and bottom edges of the image will also be shot. Then the SIFT feature operator is used to extract the features, and the RANSAC algorithm is used for matching, etc. These are well-known contents, which can be found in Chapter 11 of the reference book "Robotics Machine Vision and Control MATLAB Algorithm Foundation" ((Australia) Corke, P. Written; Liu Rong translated). The present application does not repeat here.

[0155] Step A-6. Analyze each parameter using the algorithm module of the present application;

[0156] Step A-7. Report the parameters that are outside the range of plus or minus three times the variance from the average of the previous test data to the user.

[0157] In this way, after the user sets up the physical framework, the overall process is fully automatic, reducing the user's work. The hardware in the entire test process, except for the support 2, can be reused.

[0158] In use, the user disassembles the support 2 and the camera with Bluetooth communication in the test process. Then, the user's mobile phone receives the data that need to be displayed through the HUD, and after correction by the algorithm part of the present application, it is projected to the user for use.

[0159] The above describes the correction method, system, electronic device and apparatus for the HUD projection imaging distortion provided by the embodiment of the present application in detail, and the principles and implementation modes of the present application are described by applying specific examples in this paper. The above embodiment is only used to help understand the method and its core idea of the present application; At the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for correcting projection imaging distortion in a HUD, characterized in that, include: The test camera (1) is determined and calibrated to obtain camera lens distortion data and camera intrinsic parameter data; Determine the test plane (3) and the HUD device (2), obtain the calibration image projected by the HUD device (2) on the test plane (3) through the test camera (1) as the first image, and obtain the HUD lens distortion data through the first image; The calibration image projected by the HUD device (2) onto the windshield (4) of the target vehicle by the test camera (1) is the second image. The windshield distortion data is obtained through the second image, and the distortion correction function is obtained by combining the camera lens distortion data, camera intrinsic parameter data and HUD lens distortion data. The image data to be output is input into the distortion correction function to obtain the corrected image, and then output through the HUD device (2).

2. The method for correcting HUD projection imaging distortion according to claim 1, characterized in that, The step of acquiring HUD lens distortion data through the first image includes: Obtain the first relative pose of the test camera (1) and the center position of the first image, and the second relative pose of the center position of the first image and the axis of the HUD device (2); HUD lens distortion data is obtained based on the camera lens distortion data, the camera intrinsic parameter data, the first relative pose, and the second relative pose.

3. The method for correcting HUD projection imaging distortion according to claim 2, characterized in that, The homogeneous transform method is used to obtain the distortion data of the HUD lens, and its characterization formula includes: Pfinal=DISTcamera(INM*EXM3*EXM4*DISThud(Pidea)); Wherein, Pfinal is the feature data of the first image; DISTcamera is the camera lens distortion data; INM is the camera intrinsic parameter data; EXM3 is the first relative pose; EXM4 is the second relative pose; and Pidea is the feature data of the ideal image of the HUD device.

4. The method for correcting HUD projection imaging distortion according to claim 1, characterized in that, Also includes: The external parameter data of the HUD device (2) relative to the second image and the external parameter data of the second image relative to the test camera (1) are obtained through the second image.

5. The method for correcting HUD projection imaging distortion according to claim 4, characterized in that, The characterization formula of the distortion correction function includes: Pidea = invMd(Pfinal); Wherein, Pidea is the corrected HUD image; invMd is the inverse mapping of Md, Md = DISTcamera(INM*EXM2*DISTforeg(EXM1*DISThud)), DISTcamera is the camera lens distortion data, INM is the camera intrinsic data, EXM1 is the HUD device extrinsic data relative to the second image, EXM2 is the second image extrinsic data relative to the test camera (1), DISTforeg is the windshield distortion data, DISThud is the HUD lens distortion data; Pfinal = DISTcamera(INM*EXM2*DISTforeg(EXM1*DISThud(Pidea))).

6. The method for correcting HUD projection imaging distortion according to claim 1, characterized in that, The calibration was performed using Zhang Zhengyou's chessboard calibration method.

7. The method for correcting HUD projection imaging distortion according to claim 1, characterized in that, The calibration images include the chessboard image, red image, blue image, and green image of Zhang Zhengyou's chessboard calibration method.

8. A correction system for HUD projection imaging distortion, characterized in that, include: Camera calibration module (10), the camera calibration module (10) is used to determine the test camera (1), calibrate the test camera (1), and obtain camera lens distortion data and camera intrinsic parameter data; HUD calibration module (20), the HUD calibration module (20) is used to determine the test plane (3) and the HUD device (2), and to obtain the calibration image projected by the HUD device (2) on the test plane (3) through the test camera (1) as the first image, and to obtain HUD lens distortion data through the first image; The calibration projection module (30) is used to acquire the calibration image projected by the HUD device (2) on the windshield (4) of the target vehicle through the test camera (1) as a second image; The correction function acquisition module (40) is used to acquire windshield distortion data through the second image and acquire distortion correction function by combining the camera lens distortion data, camera intrinsic parameter data and HUD lens distortion data; The projection correction module (50) is used to input the image data to be output into the distortion correction function, obtain the corrected image, and output it through the HUD device (2).

9. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the method for correcting HUD projection imaging distortion as described in any one of claims 1 to 7.

10. A correction device for HUD projection imaging distortion, characterized in that, The correction device is applied to the correction method for HUD projection imaging distortion according to any one of claims 1 to 5. Includes a test camera (1), a HUD device (2), a test platform, and a controller; Configure the test camera (1), the HUD device (2) and the test platform to obtain the calibration image projected by the HUD device (2) on the test plane (3) through the test camera (1) as the first image, and obtain HUD lens distortion data through the first image; Configure the test camera (1), the HUD device (2), and the controller. The controller is communicatively connected to the test camera (1) and the HUD device (2) and is used to obtain the calibration image projected by the HUD device (2) onto the windshield (4) of the target vehicle as a second image through the test camera (1); obtain windshield distortion data through the second image, and obtain a distortion correction function by combining the camera lens distortion data, camera intrinsic parameter data, and HUD lens distortion data, and store it in the controller. Configure the HUD device (2) and the controller, the HUD device (2) and the controller are connected, and the HUD device (2) is configured to input the image data to be output into the distortion correction function, obtain the corrected image, and output it through the HUD device (2).

Images