Image calibration method and device for AR-HUD of vehicle, storage medium and program product

Abstract

One or more embodiments of the invention provide an image calibration method and apparatus for a vehicle AR-HUD, a storage medium and a program product. The method comprises the steps of obtaining a to-be-calibrated image; determining a coordinate deviation value between a to-be-calibrated feature point in the to-be-calibrated image and a corresponding standard feature point in a standard test image; based on the coordinate deviation value, determining a pixel transformation relation between the standard test image and the to-be-calibrated image by adopting a thin-plate spline interpolation algorithm; and performing reverse transformation operation on an original image projected by the AR-HUD system based on the pixel transformation relation.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of terminal, and in particular, to an image calibration method and device for vehicle AR-HUD, a storage medium and a program product. BACKGROUND

[0002] Augmented Reality Head-Up Display (AR-HUD) is a system that combines AR technology and vehicle HUD. It provides dynamic interactive experience by reflecting virtual information to the real driving environment through the windshield. The core technology of AR-HUD is Digital Light Processing (DLP), which relies on a Digital Micromirror Device (DMD) chip composed of millions of micromirrors. The chip generates high-resolution images by reflecting light sources and has advantages such as wide color gamut, strong temperature resistance, and compatibility with optical waveguides. However, due to the irregular curved surface of the automobile windshield and the complex optical structure of the HUD, the light reflection path is easily disturbed, resulting in geometric distortion of the projected virtual image, such as stretching or twisting. In order to improve the imaging authenticity and driving safety, it is necessary to pre-correct the projection content to offset the visual error caused by the optical path deviation and ensure the accurate fitting of virtual information and real scene.

[0003] In related technologies, the projection content is usually corrected for distortion based on the rational model of camera imaging. Specifically, it fits the radial distortion and tangential distortion through a polynomial function. Although this method can solve simple optical distortion, the corresponding fitting accuracy significantly decreases when facing the complex nonlinear distortion scenario of AR-HUD, resulting in pixel-level position deviation in the corrected image. Especially in large field of view or dynamic projection scenarios, error accumulation may cause virtual image drift or misplacement, thereby affecting the accuracy of the projection content. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide technical solutions as follows:

[0005] According to a first aspect of one or more embodiments of the present specification, an image calibration method for vehicle AR-HUD is provided, the method comprising:

[0006] obtaining a to-be-calibrated image;

[0007] determining a coordinate deviation value of a to-be-calibrated feature point in the to-be-calibrated image and a corresponding standard feature point in a standard test image;

[0008] determine a pixel transformation relationship between the standard test image and the image to be calibrated based on the coordinate deviation value by using a thin plate spline interpolation algorithm;

[0009] perform inverse transformation operation on the original image projected by the AR-HUD system based on the pixel transformation relationship.

[0010] According to a second aspect of one or more embodiments of the present specification, an image calibration device for a vehicle AR-HUD is provided, the device comprising:

[0011] an image acquisition unit configured to acquire an image to be calibrated;

[0012] a coordinate deviation value determination unit configured to determine a coordinate deviation value between a feature point to be calibrated in the image to be calibrated and a corresponding standard feature point in a standard test image;

[0013] a relationship determination and acquisition unit configured to determine a pixel transformation relationship between the standard test image and the image to be calibrated based on the coordinate deviation value by using a thin plate spline interpolation algorithm;

[0014] an inverse transformation operation execution unit configured to perform inverse transformation operation on the original image projected by the AR-HUD system based on the pixel transformation relationship.

[0015] According to a third aspect of the present specification, a computer readable storage medium is provided, which stores a computer program, the program being executed by a processor to implement the steps of the method of the first aspect.

[0016] According to a fourth aspect of the present specification, a computer program product is provided, comprising computer programs / instructions, which are executed by a processor to implement the steps of the method of the first aspect.

[0017] As can be seen from the above embodiments, after acquiring the image to be calibrated, the present specification can determine the coordinate deviation value between the feature point to be calibrated in the image to be calibrated and the corresponding standard feature point in the standard test image, and establish the pixel mapping relationship between the standard test image and the image to be calibrated by introducing the thin plate spline (TPS) algorithm, thereby realizing more accurate geometric distortion correction. The thin plate spline algorithm can accurately compensate for non-uniform deformation caused by irregular windshield surfaces such as vehicle windshields and complex light paths through the elastic surface modeling capability, effectively suppressing error accumulation. In summary, while maintaining the physical logic of optical correction, the pixel alignment accuracy of the projected image and the real scene is improved to the pixel level, enhancing the accuracy of the projected content and ensuring the spatial fit of the AR information and the real driving environment. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1is a schematic diagram of an image calibration system for a vehicle AR-HUD according to an embodiment of the present disclosure;

[0019] Figure 2 is a flowchart of an image calibration method for a vehicle AR-HUD according to an embodiment of the present disclosure;

[0020] Figure 3 is a schematic diagram of an image calibration scene for a vehicle AR-HUD according to an embodiment of the present disclosure;

[0021] Figure 4 is a flowchart of another image calibration method for a vehicle AR-HUD according to an embodiment of the present disclosure;

[0022] Figures 5a-5b is a schematic diagram of an image before and after inverse transformation according to an embodiment of the present disclosure;

[0023] Figure 6 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure;

[0024] Figure 7 is a block diagram of an image calibration device for a vehicle AR-HUD according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present specification are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0026] Figure 1 is a schematic diagram of an image calibration system for a vehicle AR-HUD according to an example embodiment. As shown in Figure 1 , the system can include an AR-HUD system 10 and a calibration device 12.

[0027] The AR-HUD system 10 is integrated in the vehicle cabin, including a DMD chip, a light source module, and an optical lens group. The system generates a high-resolution virtual image by dynamically reflecting the light source through the micro-mirror array of the DMD chip, and projects the virtual image to a specific reflection area of the vehicle windshield via the optical lens group. The windshield acts as an optical reflection surface, reflecting the virtual image light into the driver's field of view, realizing the dynamic superposition of AR content such as navigation guidance, vehicle speed information, lane markings, and real driving scenes. The system supports wide color gamut display, and its high temperature resistance and optical waveguide compatibility make it suitable for complex vehicle environments, while meeting the projection requirements of large viewing angles.

[0028] The calibration device 12 is a processing unit for performing the image calibration method, which can be communicatively connected with the AR-HUD system 10 and integrated with an optical camera module configured to capture a to-be-calibrated image projected by the AR-HUD system 10 onto the vehicle windshield in real time. Feature point matching is performed between the to-be-calibrated image and the original standard test image, and corresponding coordinate deviation values are extracted. Based on the deviation values, a pixel nonlinear transformation model is constructed using a thin plate spline interpolation algorithm. The camera module can be deployed at a simulated position of the driver's field of view to ensure that the imaging angle is consistent with the actual observation angle of the driver, thereby accurately reflecting the influence of optical distortion on the projected virtual image. At the same time, the device can be embedded in the controller of the AR-HUD system 10 or used as an independent external module to interact with the vehicle through, for example, a Controller Area Network (CAN) or Ethernet vehicle bus. In summary, whether integrated or deployed independently, the calibration device 12 can update the pixel transformation relationship in real time or periodically, and pre-correct the original projection image through inverse transformation to eliminate the influence of windshield curvature and optical distortion on AR information space positioning.

[0029] The embodiments of the image calibration method for vehicle AR-HUD of the present specification will be described in detail below with reference to the accompanying drawings.

[0030] Figure 2 is a flowchart of an image calibration method for vehicle AR-HUD according to an exemplary embodiment of the present specification. As shown in Figure 2 , the method can include the following steps:

[0031] Step S202, obtaining a to-be-calibrated image.

[0032] Firstly, the vehicle can project a predefined standard test image, such as a checkerboard or dot matrix, to the windshield of the vehicle through the AR-HUD system. The standard test image is reflected by the windshield and forms a to-be-calibrated image. Then, an optical camera module built-in the calibration device captures the to-be-calibrated image from the perspective of the driver, ensuring that the imaging range covers the entire projection area of the AR-HUD system, providing raw data input for subsequent distortion analysis. In addition, the standard test image can be pre-stored in the calibration device or provided by the vehicle to which the AR-HUD system belongs, which is not limited in the present specification.

[0033] For the to-be-calibrated image, the present application can provide two optional embodiments according to actual needs, taking into account standardization and scene adaptability while adapting to different calibration scene needs:

[0034] In an embodiment, an image captured by a camera mounted on a mechanical arm based on a preset eyebox region can be obtained as the to-be-calibrated image. With a mechanical arm carrying a high-precision optical camera, the to-be-calibrated image can be moved and captured within the preset driver eyebox region. The mechanical arm simulates the driver's line of sight range according to a pre-programmed path, ensuring that the camera's perspective covers the entire projection area of the AR-HUD system. This method is suitable for standardized calibration before the vehicle is shipped or during maintenance, and the stability and spatial consistency of image acquisition are ensured through the pose control of the mechanical arm, eliminating human operation errors. Further, for different driving habits of different users, corresponding eyebox regions can be allocated for different heights, so that the AR-HUD system is adapted to a larger group of drivers.

[0035] In another embodiment, an image captured by a camera built-in a user terminal based on a dynamically captured user viewpoint position can be obtained as the to-be-calibrated image. With a camera built-in a user terminal such as a vehicle central control device or a driver's mobile phone, the actual viewpoint position of the user can be captured in real time, and the camera angle can be adjusted based on the dynamically tracked head position to capture the to-be-calibrated image at the current viewpoint. Specifically, this method can update the calibration reference in real time through face recognition, eye tracking technology, or in combination with IMU data of the vehicle, adapt to the height and posture differences of different drivers, and realize personalized dynamic calibration, especially suitable for scenarios where the projection distortion parameters change due to seat adjustment or driver replacement during vehicle use.

[0036] In summary, the above two modes can be used independently or cooperatively according to needs, ensuring initial calibration accuracy through a mechanical arm, and realizing adaptive compensation in long-term use through a user terminal, thereby improving the spatial positioning robustness of the AR-HUD system.

[0037] Step S204, determine the coordinate deviation value of the to-be-calibrated feature point in the to-be-calibrated image and the corresponding standard feature point in the standard test image.

[0038] Based on the above to-be-calibrated image and the standard test image, the corresponding feature points of the two are extracted by a feature matching algorithm, and the coordinate deviation value of each feature point in the two-dimensional image coordinate system, i.e. the horizontal and vertical deviation values, is calculated. The above deviation value quantifies the actual path deviation of the projection light after being reflected by the windshield, reflecting the nonlinear distribution characteristics of the distortion of different regions. Among them, the feature matching algorithm can be changed according to the different styles of the standard test image, so as to enhance the discrimination degree and distortion quantization accuracy of feature matching. For example, when the standard test image is a checkerboard, the to-be-calibrated feature point is a corner point in the checkerboard, and the corresponding feature matching algorithm is, for example, a corner detection algorithm such as Harris corner detection. Or when the above standard test image is a dot array, the above to-be-calibrated feature point is a dot in the dot array, and the corresponding feature matching algorithm is a corresponding dot detection algorithm. Of course, even the same detection algorithm can be called the above corner detection algorithm or dot detection algorithm in different scenarios, such as Scale-Invariant Feature Transform (SIFT) or Binary Robust Independent Elementary Features (BRIEF) algorithm.

[0039] Based on the to-be-calibrated feature points of the to-be-calibrated image and the corresponding standard feature points of the standard test image, one-to-one comparison can be performed to determine the coordinate deviation value of the two, and then the accuracy and efficiency requirements of different calibration scenarios can be adapted.

[0040] In an embodiment, based on the physical calibration board installed outside the vehicle windshield and showing the above standard test image, the standard feature points on the calibration board can be mapped to the plane coordinate system to which the above to-be-calibrated image belongs, and then the coordinate deviation value of each standard feature point and the corresponding to-be-calibrated feature point is compared respectively. Specifically, the above physical calibration board is synchronized with the AR-HUD system for calibration, and the surface feature point position is known and strictly aligned with the feature points of the standard test image. After the to-be-calibrated image is captured by the camera, the actual projection point of the physical calibration board in the image is extracted as the to-be-calibrated feature point based on the image registration algorithm, and the theoretical coordinates of the standard feature points are compared point by point to calculate the horizontal and vertical pixel deviation values. This method ensures the absolute accuracy of feature point mapping through a physical reference, and is especially suitable for factory calibration or high-precision static calibration scenarios.

[0041] In another embodiment, based on the model matrix of the vehicle windshield, the view matrix of the camera used to obtain the to-be-calibrated image, and the projection matrix, the standard feature points in the standard test image can be determined to be mapped to the plane coordinate system to which the to-be-calibrated image belongs, and the coordinate deviation values of each standard feature point and the corresponding to-be-calibrated feature point are compared respectively. Wherein, through the MVP (Model View Projection) matrix, that is, the three-dimensional model matrix for representing the geometric parameters of the curved surface based on the vehicle windshield, the view matrix of the camera for representing the pose and angle of view of the camera, and the projection matrix of the AR-HUD system for representing the virtual image generation parameters, the theoretical mapping relationship of the feature points in the standard test image from the virtual coordinate system to the to-be-calibrated image plane coordinate system actually captured by the camera can be constructed. Through matrix transformation operation, the theoretical coordinates of the standard feature points can be projected to the to-be-calibrated image plane, and the pixel-by-pixel deviation calculation is performed with the actually detected to-be-calibrated feature point coordinates. Compared with the previous embodiment, this method does not need to rely on external physical calibration board, and through mathematical modeling, dynamic calibration can be realized, which adapts to the real-time distortion compensation demand caused by environmental temperature, mechanical vibration or driver's view change in vehicle use, has higher flexibility and scene adaptability.

[0042] Whether it is a physical calibration board mapping method or a multi-matrix mapping method, it can be applied independently or cooperatively: the former ensures the reliability of the initial calibration reference, and the latter supports dynamic error correction in long-term use, and together improves the calibration robustness and long-term stability of the AR-HUD system.

[0043] In step S206, based on the coordinate deviation values, a thin-plate spline interpolation algorithm is used to determine the pixel transformation relationship between the standard test image and the to-be-calibrated image.

[0044] After determining the feature point coordinate deviation values, a thin-plate spline interpolation algorithm can be used to construct the pixel mapping relationship between the standard test image and the to-be-calibrated image, that is, the pixel transformation relationship. This thin-plate spline interpolation algorithm can generate a continuous and smooth two-dimensional transformation surface by minimizing the elastic deformation energy function, accurately fitting the local deformation caused by irregular windshield curves and complex light paths, thereby overcoming the fitting limitations of high-order distortion of traditional polynomial models.

[0045] The essence of the above thin-plate spline interpolation algorithm is to construct a high-precision nonlinear mapping model to ensure the generation of accurate pixel mapping between the standard test image and the image to be calibrated, which is hereinafter referred to as a thin-plate spline interpolation model or a TPS model. First, the unique solution of the TPS model depends on the geometric distribution of the feature point pairs. Once all the feature points are collinear, i.e., located on the same straight line, the affine transformation coefficients (a0, a1, a2) and the radial basis weight (wi) in the mapping function cannot be uniquely determined, which further causes model underfitting or solution failure. Therefore, at least four non-collinear feature point pairs are required to be selected in this specification to ensure the stability and uniqueness of the mapping model. Of course, additional six to ten feature points can further improve the local nonlinear fitting accuracy. Each feature point pair includes a feature point to be calibrated in the image to be calibrated and its corresponding standard feature point in the standard test image. The above-mentioned feature points can be distributed at different positions of the projection area, such as the corners, center or key deformation area, to cover the complex distortion characteristics of the windshield surface. For example, if the standard test image is a checkerboard, the corners of the new selected checkerboard can be selected as feature points; if it is a dot matrix, uniformly distributed code points can be selected. At the same time, the extracted feature points to be calibrated and the theoretical feature point coordinates in the standard test image need to be strictly matched. Assuming that the standard test image is a checkerboard with known coordinates, the Perspective-n-Point (PnP) algorithm can be used to align the coordinate systems of the two, ensuring one-to-one correspondence of the physical positions of the feature point pairs.

[0046] In addition, for feature point mismatching caused by camera noise or occlusion, the Random Sample Consensus (RANSAC) algorithm or outlier detection (such as the 3σ criterion) can be used to remove abnormal point pairs to avoid interference with the construction of the TPS model.

[0047] Taking a standard checkerboard test image as an example, assuming that its corner coordinates are known theoretical values (pi). After projecting the checkerboard to the windshield through the AR-HUD system, the checkerboard in the image to be calibrated captured by the camera is deformed due to distortion. The corners of the image to be calibrated are extracted as feature points to be calibrated (qi), and are matched one by one with the theoretical coordinates. If four corners and a center are selected as five non-collinear feature point pairs, the TPS model can accurately fit the overall tilt of the projection area (i.e., the affine part) and the edge distortion (i.e., the radial basis part).

[0048] For the TPS model, the coordinate deviation values of each feature point pair, i.e., the horizontal and vertical coordinate difference values of the to-be-calibrated feature points and the standard feature points, can be input into a preset TPS model. The model takes minimizing bending energy as the core target. N sets of feature point pairs are input, wherein the feature points of the standard test image are control points pi=(xi, yi), and the feature points of the to-be-calibrated image are target points qi=(xi', yi'). TPS can solve a smooth mapping function f(x, y) to satisfy f(xi, yi)=xi' and f(xi, yi)=yi', while minimizing the bending energy functional:

[0049]

[0050] The so-called bending energy functional is derived from the physical deformation model of a thin metal plate under external force. The minimization process is equivalent to finding the smoothest deformation surface to avoid distortion caused by local severe deformation of the mapping function.

[0051] Meanwhile, the solution of TPS can be composed of a global affine transformation and a local nonlinear correction:

[0052]

[0053] where a0, a1, and a2 are affine transformation coefficients, representing overall translation, scaling, and rotation, respectively; wi is the weight coefficient of the radial basis function; which represents the Euclidean distance between the current point and the control point, which represents the radial basis function, used to describe local nonlinear distortion.

[0054] By solving the above equation, a thin plate spline transformation matrix containing the above affine transformation coefficients and radial basis weight coefficients can be obtained, and then the pixel transformation relationship between the standard test image and the to-be-calibrated image can be generated.

[0055] At this point, TPS accurately fits high-order distortion caused by windshield surface irregularity and optical path nonlinear deviation through elastic surface modeling capability, especially significantly reducing error accumulation under large field of view. Its strategy of combining affine transformation and radial basis function not only retains the stability of global linear correction, but also improves the pixel alignment accuracy in complex deformation areas through local correction.

[0056] Step S208: Perform inverse transformation operation on the original image projected by the AR-HUD system based on the pixel transformation relationship.

[0057] After the pixel transformation relationship is determined, the pixel transformation relationship can be reversely applied to the original image to be projected by the AR-HUD system to pre-distort the image pixel coordinates. After the corrected image is re-projected to the windshield by the AR-HUD system, the geometric distortion of the reflected virtual image is dynamically compensated, the spatial alignment of the virtual information and the real driving scene is realized, and the visual positioning accuracy and stability of the AR content are ensured.

[0058] For the above-mentioned reverse transformation operation, in this specification, an efficient and high-precision real-time image correction can be realized by a reverse distortion field construction and lookup table pre-computation mechanism.

[0059] In an embodiment, an inverse deformation field from the image to be calibrated to the standard test image can be first determined based on the pixel transformation relationship described above, and an inverse look-up table (LUT) for the original image is constructed according to the inverse deformation field. Meanwhile, real-time resampling is performed on the original image based on the look-up table record to generate a pre-distortion compensation image as a calibrated image. The deformation field is in the form of a two-dimensional vector field, describing the offset relationship between the theoretical position (u', v') of each pixel in the original image and the target position (u, v) in the corrected image. For example, if a pixel in the original image is offset by Δx to the right and Δy downward due to windshield surface reflection, the inverse deformation field will record that the pixel needs to be offset by Δx to the left and Δy upward to compensate for the distortion. The inverse deformation field is discretized into an inverse look-up table, which uses the original image pixel coordinates (u, v) as the index and stores the corresponding corrected coordinates (u', v') and bilinear interpolation weights. Specifically, the corrected coordinates (u', v') refer to the theoretical position of each pixel (u, v) in the original image in the standard test image coordinate system, calculated through the inverse deformation field. For the interpolation weights, assuming that the corrected coordinates (u', v') are non-integer coordinates, such as (10.3, 20.7), bilinear interpolation weights can be calculated based on the distance relationship of the four adjacent integer pixel points around them to ensure that the resampled image is smooth and free of jaggies. For real-time resampling, it refers to the operation performed on the original image based on the inverse look-up table when the AR-HUD system is running. The AR-HUD system needs to distribute the RGB values of each pixel (u, v) in the original image to the target pixel corresponding to the corrected coordinates (u', v') according to the coordinate mapping relationship recorded in the look-up table. At the same time, for non-integer coordinate target pixels, the pixel values are generated through, for example, weighted averaging of the RGB values of the four adjacent source pixels, to avoid image jaggies or holes caused by coordinate discretization. Finally, the pre-distortion compensation image is output, whose pixel distribution has been pre-corrected for windshield surface and optical path distortion. When this image is projected onto the windshield through the AR-HUD system, the geometric deformation of the reflected virtual image is dynamically offset, achieving sub-pixel level spatial alignment of virtual information and real scene.

[0060] Taking the projection of a navigation arrow as an example, assuming that the arrowhead coordinate in the original image is (100, 200), the inverse look-up table indicates that it needs to be offset to (98.5, 202.3). The corrected pixel is generated by calculating the weighted mixed value of the surrounding four pixels (98, 202), (98, 203), (99, 202), and (99, 203) through bilinear interpolation, so that the arrowhead of the reflected virtual image accurately coincides with the real scene lane line, with an error of less than 0.5 pixels.

[0061] It is worth mentioning that the projection content of AR-HUD is often divided into different regions according to different focal lengths, thereby showing different contents. The present specification provides a multi-focal-length regional calibration mechanism to improve the distortion correction accuracy in a similar complex optical scene.

[0062] In an embodiment, the standard test image can be composed of multiple groups of standard test sub-images with different focal lengths, at this time the above-mentioned to-be-calibrated image can be divided into to-be-calibrated sub-images corresponding to each standard test sub-image according to the spatial region, and for each group of standard test sub-images and the corresponding to-be-calibrated sub-images, the feature point coordinate deviation value calculation and the thin plate spline interpolation calculation operation can be performed respectively, thereby generating a regional pixel transformation relationship, and meanwhile, each regional pixel transformation relationship can be spliced according to the spatial position to form a global pixel transformation relationship for the reverse transformation operation.

[0063] Each group of sub-images covers a specific projection region and adapts to the multi-focal optical characteristics of the AR-HUD system, for example: the near-field sub-image adapts to the near-field region of the driver and is used to display images such as vehicle speed, navigation arrow, etc., and adopts a high-density feature point layout such as a fine-grained chessboard, which adapts to the short focal length projection requirement; the far-field sub-image adapts to the far-field region and can be used to display images such as lane line extension, front obstacle prompt, etc., and adopts a sparse but large-range feature point distribution such as a wide-angle dot matrix, which adapts to the long focal length projection requirement. In summary, based on the focal length distribution of the above standard test sub-images, the to-be-calibrated image can be divided into corresponding to-be-calibrated sub-images according to the spatial position, specifically, according to the optical parameters of the AR-HUD system such as the focal length of the lens group, the curvature of the windshield surface, and the driver's field of view partition, the projection region can be divided into a plurality of overlapping or non-overlapping sub-regions, each sub-region is aligned with the spatial range of a standard test sub-image. If the to-be-calibrated image captured by the camera has perspective deformation, each sub-region can be mapped to the coordinate system of the standard test sub-image through a perspective correction algorithm such as Homography transformation, to ensure spatial alignment consistency. Meanwhile, for each group of standard test sub-images and the corresponding to-be-calibrated sub-images, matching feature points in the sub-image pair can be extracted, thereby calculating the coordinate deviation value. For example, the near-field sub-image adopts Harris corner detection, and the far-field sub-image adopts SIFT feature matching, to adapt to different resolution requirements; and based on the feature point deviation value, a regional pixel transformation relationship is generated by using a thin plate spline interpolation algorithm, so that the TPS model of each sub-region supports independent optimization, and only local distortion characteristics such as fine distortion in the near-field region or overall shift in the far-field region are concerned, avoiding the smoothing effect of the global model on local details.

[0064] Finally, the sub-regional pixel transformation relationship of each sub-region can be spliced according to the spatial position to form a global pixel transformation relationship. For the overlapping pixels at the junction of the sub-regions, the transformation parameters of the adjacent regions can be fused by using a weighted average strategy. The specific weight is dynamically adjusted according to the distance of the pixel from the sub-region boundary, so as to ensure the transition smoothness. In addition, the spliced transformation field can be optimized by a global energy minimization algorithm such as Poisson fusion, so as to eliminate the abrupt change or discontinuity between the sub-regions and ensure the overall consistency of the corrected image.

[0065] The following will be described in combination with Figure 3 and Figure 4 , another image calibration method for vehicle AR-HUD is introduced. First, for Figure 3 , which describes an image calibration scene for vehicle AR-HUD, specifically including vehicle windshield 32, rack 30, mechanical arm camera 34. The rack 30 is used to support and fix the vehicle windshield 32, simulating the environment in which the vehicle actually installs the windshield; the vehicle windshield 32 serves as the reflecting surface of the projected image of the AR-HUD system, and its curved surface characteristics will cause optical distortion to the projected image; the mechanical arm camera 34 is equipped with a high-precision optical camera and can move within a preset eyebox 36 area. The eyebox 36 is an area simulating the position of the driver's eyes, and the mechanical arm camera 34 moves and shoots the to-be-calibrated image reflected by the AR-HUD system after being projected to the vehicle windshield 32 according to the preprogrammed path. In this way, it can be ensured that the camera view covers the entire projection area of the AR-HUD system, simulating the actual observation angle of the driver and providing accurate image data for subsequent image calibration. Secondly, as shown in Figure 4 , the above method includes the following steps:

[0066] Step S402, the calibration device acquires the to-be-calibrated image.

[0067] In an embodiment, in the vehicle factory test, the camera carried by the mechanical arm of the calibration device can shoot the black background white dot standard dot matrix projected by the AR-HUD from the preset eyebox position. The dot matrix is deformed due to the reflection of the windshield curved surface, forming a to-be-calibrated image (as shown in Figure 5a ). The camera ensures that the imaging range covers the vehicle speed display area in the near field and the lane extension area in the far field, providing complete data input for subsequent analysis.

[0068] Step S404, the calibration device acquires the to-be-calibrated image.

[0069] In an embodiment, the image processing module of the calibration device can assume that the calibration device detects 50 white point feature points for the near field sub-image of the high-density dot matrix, and aligns each point with the theoretical coordinates of the standard dot matrix diagram one by one, and calculates the offset of each point (such as Δx = 0.8 pixels, Δy = -1.1 pixels). The far field sub-image of the sparse dot matrix matches the feature points through the BRIEF algorithm, and records the large range offset trend, for example, the whole left offset is 2.5 pixels.

[0070] In step S406, the calibration device acquires the image to be calibrated.

[0071] In an embodiment, the thin plate spline interpolation module of the calibration device can select five groups of non-collinear white point feature pairs in the above-mentioned near field sub-image, and construct a TPS model. It is assumed that the model analysis shows that the center area is slightly depressed due to the curvature of the windshield, wherein the affine coefficient a1 = 0.97, a2 = 0.03, and the edge area presents nonlinear compression due to the light path inclination, wherein the radial basis weight w1 = 0.4, w2 = 0.6. The far field sub-image constructs a global tilt model through three groups of feature points, wherein a1 = 1.03, a2 = -0.15.

[0072] In step S408, the calibration device acquires the image to be calibrated.

[0073] In an embodiment, the image fusion module of the calibration device can splice the TPS transformation models of each sub-region according to the spatial position to form a global pixel transformation relationship. At the boundary of the sub-regions, the adjacent model parameters are fused through a weighted average strategy, and the weight is dynamically adjusted according to the distance of the pixel from the boundary, for example, the weight linearly decreases within 10 pixels from the boundary. Finally, the continuity of the global transformation field is optimized through the Poisson fusion algorithm, and the sudden change or inconsistency (such as Figure 5b

[0074] In step S410, the AR-HUD system acquires the image to be calibrated.

[0075] In an embodiment, for the original image of the navigation arrow, it is assumed that the tip coordinate is (200, 400). The LUT indicates that it needs to be offset to the right by 1.2 pixels, that is, the corrected coordinate is (201.2, 400). The AR-HUD system generates a smooth corrected image by blending the RGB values of the surrounding four pixels (201, 400), (201, 401), (202, 400), and (202, 401) through bilinear interpolation. After projection, the arrow tip is accurately aligned with the real scene lane line, and the error is less than 0.4 pixels.

[0076] Figure 6 FIG. 1 is a schematic structural diagram of a device provided by an exemplary embodiment. Please refer to Figure 6 ​At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and of course can also include other hardware required by functions. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 610 into the memory 608 by the processor 602 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0077] Please refer to Figure 7 The image calibration device for the vehicle AR-HUD can be applied to the device as shown in Figure 6 The image calibration device can include:

[0078] An image acquisition unit 702 is configured to acquire a to-be-calibrated image.

[0079] A coordinate deviation value determination unit 704 is configured to determine a coordinate deviation value of a to-be-calibrated feature point in the to-be-calibrated image and a corresponding standard feature point in a standard test image.

[0080] A relationship determination acquisition unit 706 is configured to determine a pixel transformation relationship between the standard test image and the to-be-calibrated image based on the coordinate deviation value by using a thin-plate spline interpolation algorithm.

[0081] An inverse transformation operation execution unit 708 is configured to perform an inverse transformation operation on an original image projected by an AR-HUD system based on the pixel transformation relationship.

[0082] Optionally, the coordinate deviation value determination unit 704 is specifically configured to:

[0083] Map the standard feature points on a physical calibration board on which the standard test image is displayed and which is installed outside the vehicle windshield to a plane coordinate system to which the to-be-calibrated image belongs, and compare the coordinate deviation value of each standard feature point and the corresponding to-be-calibrated feature point, respectively; or

[0084] Determine the mapping of the standard feature points in the standard test image to the plane coordinate system to which the to-be-calibrated image belongs based on a model matrix of the vehicle windshield, an observation matrix of a camera used to acquire the to-be-calibrated image, and a projection matrix, and compare the coordinate deviation value of each standard feature point and the corresponding to-be-calibrated feature point, respectively.

[0085] Optionally, the relationship determination acquisition unit 706 is specifically configured to:

[0086] selecting at least four groups of non-collinear feature point pairs, each group of feature point pairs containing any to-be-calibrated feature point in the to-be-calibrated image and a corresponding standard feature point in the standard test image;

[0087] inputting the coordinate deviation values of the feature point pairs into a preset thin plate spline interpolation model, and generating the pixel transformation relationship according to a thin plate spline transformation matrix output by the thin plate spline interpolation model.

[0088] Optionally, the inverse transformation operation execution unit 708 is specifically configured to:

[0089] determining an inverse deformation field from the to-be-calibrated image to the standard test image based on the pixel transformation relationship;

[0090] constructing an inverse lookup table for the original image according to the inverse deformation field, the inverse lookup table recording coordinate mapping relationships and bilinear interpolation weights of each pixel point of the original image;

[0091] performing real-time resampling on the original image based on the inverse lookup table record to generate a pre-distortion compensation image as a calibrated image.

[0092] Optionally, the standard test image contains multiple groups of standard test sub-images respectively corresponding to different focal lengths, and the device further includes:

[0093] a sub-image calibration unit configured to divide the to-be-calibrated image into to-be-calibrated sub-images corresponding to the standard test sub-images according to spatial regions;

[0094] performing feature point coordinate deviation value calculation and thin plate spline interpolation calculation operations respectively for each group of standard test sub-images and corresponding to-be-calibrated sub-images to generate sub-regional pixel transformation relationships;

[0095] splicing the sub-regional pixel transformation relationships according to spatial positions to form a global pixel transformation relationship for inverse transformation operation.

[0096] Optionally, the standard test image is a checkerboard, and the to-be-calibrated feature points are corner points in the checkerboard; or

[0097] the standard test image is a dot array, and the to-be-calibrated feature points are dots in the dot array.

[0098] Optionally, the image acquisition unit 702 is specifically configured to:

[0099] acquire an image captured by a camera carried by a mechanical arm based on a preset eyebox region as the to-be-calibrated image; or

[0100] The image captured by the camera built-in the user terminal based on the dynamically captured user viewpoint position is determined as the to-be-calibrated image.

[0101] The implementation process of the functions and roles of the units in the device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0102] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the scheme of the present specification according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0103] Based on the same idea as the above method, the present specification also provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method according to any one of the above embodiments by running the executable instructions.

[0104] Based on the same idea as the above method, the present specification also provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0105] Based on the same idea as the above method, the present specification also provides a computer program product, which comprises computer program / instructions, and the computer program / instructions are executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0106] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0107] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and the apparatus can be implemented as special purpose logic circuitry.

[0108] Computers suitable for the execution of a computer program include, by way of example, general and / or special purpose microprocessors, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory and / or a random access memory. The essential elements of a computer are a central processing unit for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a GPS receiver, a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0109] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0110] While the specification contains many specifics, these should not be construed as limiting the scope of any invention or of the required claims in any way. The specification and the described embodiments are merely illustrative of specific ways to make and use the many inventive features and the present application will be measured in terms of the claims which follow, rather than the ability to practice the specification. Certain features described in the specification in the context of one embodiment also can be implemented in other embodiments. Conversely, certain features of the described embodiments can also be implemented in a less than all of the embodiments described and can be practiced or carried out in various other ways. Furthermore, the described features can be combined in any suitable sub-combination or variations thereof. Additionally, features can be practiced in isolation from each other and the specification and claims are intended to cover any and all novel and non-obvious combinations of features. Similarly, although operations can be depicted in the drawings in a particular order, this should not be understood as requiring or implying that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0111] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring or implying that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0112] Accordingly, specific embodiments of the subject matter have been described. Further, the described process can not be required to be performed in the particular order described, or in sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0113] The above-described embodiments are merely meant to be illustrative of the present disclosure and should not be construed as limiting the scope of the disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present disclosure should be included in the scope of the present disclosure.

Claims

1. An image calibration method for vehicle AR-HUD, characterized in that: The method comprises: Get the image to be calibrated; Determining coordinate deviation values ​​between the feature points to be calibrated in the image to be calibrated and the corresponding standard feature points in the standard test image; Based on the coordinate deviation value, a thin plate spline interpolation algorithm is used to determine a pixel transformation relationship between the standard test image and the image to be calibrated; Based on the pixel transformation relationship, an inverse transformation operation is performed on the original image projected by the AR-HUD system.

2. The method according to claim 1, characterized in that The determining of the coordinate deviation value between the feature point to be calibrated in the image to be calibrated and the corresponding standard feature point in the standard test image includes: Based on a physical calibration plate mounted on the outside of the vehicle windshield and displaying the standard test image, the standard feature points on the physical calibration plate are mapped to the plane coordinate system to which the image to be calibrated belongs, and the coordinate deviation value of each standard feature point is compared with the corresponding feature point to be calibrated; or Based on the model matrix of the vehicle windshield, the observation matrix of the camera used to obtain the image to be calibrated, and the projection matrix, the standard feature points in the standard test image are mapped to the plane coordinate system to which the image to be calibrated belongs, and the coordinate deviation value of each standard feature point and the corresponding feature point to be calibrated are compared respectively.

3. The method according to claim 1, characterized in that The method of determining the pixel transformation relationship between the standard test image and the image to be calibrated based on the coordinate deviation value by using a thin plate spline interpolation algorithm includes: Selecting at least four groups of non-collinear feature point pairs, each group of feature point pairs includes any feature point to be calibrated in the image to be calibrated and its corresponding standard feature point in the standard test image; The coordinate deviation value of each feature point pair is input into a preset thin plate spline interpolation model, and the pixel transformation relationship is generated according to the thin plate spline transformation matrix output by the thin plate spline interpolation model.

4. The method according to claim 1, wherein The performing an inverse transformation operation on the original image projected by the AR-HUD system based on the pixel transformation relationship includes: Determining a reverse deformation field from the image to be calibrated to the standard test image based on the pixel transformation relationship; Constructing a reverse lookup table for the original image according to the reverse deformation field, wherein the reverse lookup table records the coordinate mapping relationship and bilinear interpolation weight of each pixel point of the original image; The original image is resampled in real time based on the inverse lookup table records to generate a pre-distortion compensated image as a calibrated image.

5. The method according to claim 1, wherein The standard test image includes a plurality of groups of standard test sub-images corresponding to different focal lengths. The method further includes: Dividing the image to be calibrated into sub-images to be calibrated corresponding to each standard test sub-image according to the spatial region; For each set of standard test sub-images and their corresponding sub-images to be calibrated, feature point coordinate deviation calculation and thin plate spline interpolation calculation operations are performed to generate pixel transformation relationships for each region. The pixel transformation relationships of each sub-region are spliced ​​according to spatial positions to form a global pixel transformation relationship for inverse transformation operation.

6. The method according to claim 1, characterized in that The standard test image is a checkerboard, and the feature points to be calibrated are corner points in the checkerboard; or The standard test image is a point array, and the feature points to be calibrated are points in the point array.

7. The method according to claim 1, characterized in that The step of obtaining the image to be calibrated includes: Obtain an image captured by a camera carried by the robotic arm based on a preset eye box area as the image to be calibrated; or, An image captured by a camera built into the user terminal based on a dynamically captured user viewpoint position is obtained and determined as the image to be calibrated.

8. An image calibration device for vehicle AR-HUD, characterized in that: The device comprises: An image acquisition unit, used to acquire an image to be calibrated; A coordinate deviation value determining unit, configured to determine a coordinate deviation value between a feature point to be calibrated in the image to be calibrated and a corresponding standard feature point in the standard test image; a relationship determination and acquisition unit, configured to determine, based on the coordinate deviation value, a pixel transformation relationship between the standard test image and the image to be calibrated using a thin plate spline interpolation algorithm; An inverse transformation operation execution unit is used to perform an inverse transformation operation on the original image projected by the AR-HUD system based on the pixel transformation relationship.

9. A computer-readable storage medium, characterized in that Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images