ARHUD imaging method and imaging system

Abstract

The invention provides an ARHUD imaging method and an ARHUD imaging system. Comprising the following steps: acquiring a conversion matrix between a world coordinate system and an eye position coordinate system under a plurality of visual angles based on a calibration object, and constructing a virtual image 3DGS model based on graph card data and the conversion matrix; fitting dynamic distortion of the HUD imaging model by using a spherical harmonic function based on the virtual image 3DGS model so as to obtain a virtual image 3DGS model comprising the spherical harmonic function; and further performing rendering based on the updated virtual image 3DGS model to obtain a target virtual image and performing projection display. According to the method, the HUD virtual image 3DGS model can be constructed according to different visual angles and eye positions, and the dynamic distortion of the HUD imaging model is fitted by using the spherical harmonic function, so that the dynamic distortion condition is fitted more accurately, image distortion caused by eye position change is reduced, the accuracy of distortion correction is effectively improved, and a clearer and more real imaging effect is provided for a user; meanwhile, exhaustion calculation of a large number of positions is avoided, the distortion correction process is simplified, the calculation amount is reduced, and therefore the processing efficiency is remarkably improved.

Description

Technical Field

[0001] This application relates to the field of head-up display technology, and in particular to an ARHUD imaging method and imaging system. Background Technology

[0002] Augmented Reality-Head-Up Display (AR-HUD) is an intelligent driving assistance device used in transportation vehicles such as airplanes, cars, and high-speed trains. It projects key data such as driving information, navigation instructions, and road condition warnings directly onto the windshield in front of the driver's line of sight through optical devices. This allows the driver to easily obtain the information they need while keeping their eyes focused on the road, thereby reducing eye shifts and distractions and effectively reducing the risk of accidents.

[0003] Errors in optical design and manufacturing assembly can easily cause significant dynamic distortion in the final image of an ARHUD. This means that the distortion changes considerably as the user's eye position changes. When the user moves their eyes, the image is distorted to varying degrees, which affects the visual effect and user experience.

[0004] Existing imaging methods typically employ an exhaustive approach to correct distortion in the ARHUD image before imaging. However, there may be omissions in the eyebox space, leading to deviations in the final projection result. Furthermore, the exhaustive approach is cumbersome and inefficient. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an ARHUD imaging method and imaging system.

[0006] Specifically, this application provides an ARHUD imaging method, comprising the following steps: obtaining a transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints based on a calibration object, and constructing a virtual image 3DGS model based on the map data and the transformation matrix; fitting the dynamic distortion of the HUD imaging model using a spherical harmonic function based on the virtual image 3DGS model to obtain a virtual image 3DGS model including the spherical harmonic function; and rendering based on the updated virtual image 3DGS model to obtain a target virtual image and project it for display.

[0007] In the above technical solution, a HUD virtual image 3DGS model can be constructed according to different viewing angles and eye positions, and the dynamic distortion of the HUD imaging model can be fitted using spherical harmonic functions. This more accurately fits the dynamic distortion, reduces image distortion caused by changes in eye position, effectively improves the accuracy of distortion correction, and provides users with a clearer and more realistic imaging effect. At the same time, it avoids exhaustive calculations of a large number of positions, simplifies the distortion correction process, reduces the amount of computation, and thus significantly improves processing efficiency.

[0008] Furthermore, the step of obtaining the transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints includes: placing a calibration object at a preset position of the target vehicle and obtaining the 3D coordinates of the calibration object in the world coordinate system; obtaining eye position images of the calibration object from several viewpoints based on a preset eye box range, so as to obtain the 2D coordinates of the calibration object in the eye position coordinate system based on the eye position images; and obtaining the transformation matrix corresponding to each viewpoint based on the 3D coordinates and each 2D coordinate.

[0009] The above technical solution can accurately establish the mapping relationship between the world coordinate system and the eye position coordinate system. Different perspectives mean that the driver may be in different eye positions. The transformation matrix can help the system transform the target information into the driver's eye position perspective, laying the foundation for the subsequent construction of an accurate virtual image model.

[0010] Furthermore, the construction of the virtual image 3DGS model includes: projecting the calibration chart and collecting chart data from several viewpoints based on the eye coordinate system; and extracting features from the calibration chart and the chart data respectively, and performing feature matching to obtain the projected virtual image.

[0011] In the above technical solution, the image data from different perspectives can reflect the features and changes of the image under various possible eye positions, providing comprehensive data support for the subsequent accurate construction of the virtual image model; through feature extraction and matching, the position and shape of the virtual image can be accurately determined, effectively improving the accuracy of the virtual image model construction.

[0012] Furthermore, the construction of the virtual image 3DGS model further includes: obtaining the projection coordinates of the projection virtual images in the world coordinate system based on each projection virtual image and the corresponding transformation matrix, so as to generate the virtual image 3D model based on the projection coordinates; and constructing the virtual image 3DGS model according to the preset model parameters and the virtual image 3D model.

[0013] In the above technical solution, coordinate transformation and parameter information are fully utilized, so that the virtual image 3DGS model can not only accurately reflect the actual projection situation, but also be customized according to preset requirements to meet different needs, making this application more flexible.

[0014] Furthermore, obtaining the spherical harmonic coefficients includes: performing perspective projection on the virtual image 3DGS model to several viewpoints to obtain image pixels as projection ground truth; establishing a dynamic distortion fitting equation based on the difference between the projection ground truth and the corresponding virtual image pixels, and the spherical harmonic functions corresponding to the several viewpoints, so as to obtain the spherical harmonic coefficients according to the dynamic distortion fitting equation.

[0015] In the above technical solution, the spherical harmonic function can effectively fit complex dynamic distortion situations. In this way, the imaging distortion caused by optical design and manufacturing assembly errors can be accurately corrected, so that the distortion of the image under different eye positions can be effectively controlled, thereby improving the imaging quality and visual effect.

[0016] Furthermore, obtaining the target virtual image and projecting it includes: obtaining multiple Gaussian spheres and corresponding eye poses based on the updated virtual image 3DGS model, and sorting each Gaussian sphere according to depth information, so as to project each Gaussian sphere onto a 2D image plane according to the sorting result and the eye pose; wherein, during the projection process, it also includes: performing distortion correction on the initial virtual image projected onto the 2D image plane to obtain the target virtual image; and projecting the target virtual image onto a target plane.

[0017] The above technical solution can reasonably process the depth information of the virtual image, so that during the projection process, virtual image elements of different depths can be processed in Gaussian order from far to near, thereby reducing the number of required mixing operations.

[0018] Furthermore, based on the same concept, this application also provides an ARHUD imaging system, comprising at least: a virtual image generation module, used to construct a virtual image 3DGS model based on a calibration object, and obtain a target virtual image based on the virtual image 3DGS model, so as to transmit the target virtual image to an optical projection device; the optical projection device is used to project the target virtual image onto a target plane.

[0019] Furthermore, the virtual image generation module includes at least a memory and a processor; the memory is used to store computer instructions for multiple functional layers, each functional layer including at least a construction module, a fitting module, and a generation module; the processor is used to execute the computer instructions for each of the multiple functional layers stored in the memory.

[0020] Furthermore, the construction module is used to obtain the transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints based on the calibration object, and to construct a virtual image 3DGS model based on the map data and the transformation matrix; the fitting module is used to fit the dynamic distortion of the HUD imaging model using spherical harmonic functions based on the virtual image 3DGS model to obtain a virtual image 3DGS model including spherical harmonic functions; the generation module is used to render based on the updated virtual image 3DGS model to obtain the target virtual image.

[0021] Furthermore, the system also includes a sensor for acquiring the eye pose of the driver's observation of the updated virtual image 3DGS model, so that the generation module renders the updated virtual image 3DGS model based on the eye pose.

[0022] In the above technical solution, the modules work together to achieve efficient and accurate ARHUD virtual image correction, thereby improving the user's experience with ARHUD.

[0023] Compared with the prior art, the beneficial effects of this application are as follows: This application constructs a 3DGS model of the HUD virtual image based on different viewpoints and eye positions, and uses spherical harmonic functions to fit the dynamic distortion of the HUD imaging model. This more accurately fits the dynamic distortion situation, effectively reducing image distortion caused by changes in eye position, greatly improving the accuracy of distortion correction, and thus presenting users with a clearer and more realistic imaging effect. In addition, this method avoids exhaustive calculations for a large number of positions, simplifying the distortion correction process, greatly reducing the amount of computation, and thus significantly improving processing efficiency. Attached Figure Description

[0024] Figure 1 This is a flowchart of the ARHUD imaging method described in this application.

[0025] Figure 2 This is a framework diagram of the ARHUD imaging system described in this application.

[0026] Figure 3 This is a framework diagram of the virtual image generation module described in this application.

[0027] Figure 4 This is a memory framework diagram as described in this application. Detailed Implementation

[0028] The following describes in further detail an ARHUD imaging method and imaging system of this application with reference to specific embodiments and accompanying drawings.

[0029] To address the issues that existing technologies suffer from significant dynamic distortion in the final image of ARHUD due to errors in optical design and manufacturing assembly, which negatively impacts visual effects and user experience; and to address the problems that existing imaging methods, which employ exhaustive methods, result in projection deviations and are cumbersome and inefficient in their implementation.

[0030] Please see Figure 1 This application provides an ARHUD imaging method, including the following steps S100-S300.

[0031] S100: Based on the calibration object, obtain the transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints, and construct a virtual image 3DGS model based on the map data and the transformation matrix.

[0032] S200: Based on the virtual image 3DGS model, the dynamic distortion of the HUD imaging model is fitted using spherical harmonic functions to obtain a virtual image 3DGS model that includes spherical harmonic functions.

[0033] S300: Renders based on the updated virtual image 3DGS model to obtain the target virtual image and projects it for display.

[0034] In some embodiments, a calibration object is placed at a fixed position in front of the vehicle. The 3D coordinates of the calibration object in the ADAS coordinate system (i.e., the world coordinate system) are obtained based on ADAS (Advanced Driver Assistance System). Then, the calibration object is photographed from several viewpoints using an eye-position camera to obtain the 2D coordinates of the calibration object in the eye-position coordinate system. Furthermore, the transformation matrix between the world coordinate system and the eye-position coordinate system from several viewpoints is obtained based on the 3D coordinates and 2D coordinates using the PnP algorithm. Further, a calibration chart is placed on the TFT optical engine of the ARHUD, and chart data from several viewpoints is photographed using an eye-position camera. A preliminary virtual image 3D model is constructed based on the chart data, the relevant feature points of the calibration chart, and the transformation matrix using the minimization 3D distance method. The model parameters are further set to obtain the virtual image 3DGS model.

[0035] Further, perspective projections from several viewpoints are obtained based on the virtual image 3DGS model. The difference between the true projection value and the corresponding virtual image pixel is calculated based on the perspective projection and the transformation matrix to establish a system of linear equations fitting the spherical harmonic function from several viewpoints. Then, the spherical harmonic coefficients are solved using the least squares method to update the virtual image 3DGS model. Furthermore, the eye position and pose of the driver observing the virtual image 3DGS model are obtained through a DMS camera. This eye position and pose are used to render the virtual image 3DGS model to generate a virtual image image. Finally, the virtual image image is transmitted to the TFT optical engine and projected onto the windshield to achieve a combination of virtual and real effects.

[0036] The following describes steps S100-S300 in detail.

[0037] The step S100, which involves obtaining the transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints, includes: placing a calibration object at a preset position of the target vehicle and obtaining the 3D coordinates of the calibration object in the world coordinate system; obtaining eye position images of the calibration object from several viewpoints based on a preset eye box range, so as to obtain the 2D coordinates of the calibration object in the eye position coordinate system based on the eye position images; and obtaining the transformation matrix corresponding to each viewpoint based on the 3D coordinates and each 2D coordinate.

[0038] In some embodiments, a calibration object is placed at a fixed position in front of the vehicle, such as placing multiple calibration objects along the lane lines on both sides in front of the vehicle. The calibration object is photographed by the forward-facing camera of the ADAS system to obtain the 3D coordinates of the calibration object in the world coordinate system (equivalent to the ADAS coordinate system). Then, the calibration object is photographed by an eye-position camera to obtain the 2D coordinates of the calibration object in the eye-position coordinate system. Within a preset eye box range, eye-position images from different perspectives are captured. Assuming there are N perspectives, N 2D coordinates are obtained.

[0039] Furthermore, the transformation matrix between the ADAS coordinate system and the eye position coordinate system is calculated by matching point pairs in 3D-2D; specifically, the transformation matrix is ​​obtained using PnP (Perspective-n-Point).

[0040] In the ADAS coordinate system, feature points in 3D coordinates are determined, and the set of 2D coordinate points corresponding to these feature points in the eye coordinate system is obtained. At the same time, the intrinsic parameter matrix of the camera corresponding to the eye coordinate system also needs to be obtained. The previously obtained 3D coordinate point set, 2D coordinate point set and intrinsic parameter matrix are processed by the PnP algorithm to obtain rotation vectors and translation vectors. The obtained rotation vectors are converted into rotation matrices, and then the rotation matrix and translation vectors are combined into a transformation matrix.

[0041] It should be noted that the available PnP algorithms include at least P3P (Perspective-3-Point), EPnP (Efficient Perspective-n-Point), and Iterative PnP. Those skilled in the art can choose any of the above algorithms to obtain the transformation matrix according to the actual application requirements.

[0042] The above technical solution can accurately establish the mapping relationship between the world coordinate system and the eye position coordinate system. Different perspectives mean that the driver may be in different eye positions. The transformation matrix can help the system transform the target information into the driver's eye position perspective, laying the foundation for the subsequent construction of an accurate virtual image model.

[0043] Furthermore, the construction of the virtual image 3DGS model in step S100 includes: projecting the calibration map card and collecting map card data from several viewpoints based on the eye position coordinate system; and extracting features from the calibration map card and the map card data respectively, and performing feature matching to obtain the projected virtual image.

[0044] In some embodiments, a calibration chart is set in the TFT (Thin Film Transistor) optical engine of the ARHUD and projected onto the windshield (i.e., the target plane below). Simultaneously, an eye-position camera is used to collect chart data from different viewing angles, which correspond one-to-one with the 2D coordinate viewing angles obtained above. Then, the Blob feature points in the calibration chart and the Blob feature points in the chart data are detected. By matching the two feature points, the projected virtual image of the TFT optical engine is obtained.

[0045] It should be noted that the function for detecting Blob feature points has a function interface in OpenCV that can be called directly.

[0046] Here, a Blob refers to a connected region in an image that has similar attributes (such as grayscale value, color, etc.). It can be understood as a set of interconnected pixels with similar characteristics in an image. Blob feature points are representative points of these connected regions, which can concisely represent the Blob region.

[0047] It should also be noted that the calibration chart in this embodiment is represented by a matrix of dots. Those skilled in the art can choose other calibration charts according to actual application needs, and are not limited to this one.

[0048] In the above technical solution, the image data from different perspectives can reflect the features and changes of the image under various possible eye positions, providing comprehensive data support for the subsequent accurate construction of the virtual image model; through feature extraction and matching, the position and shape of the virtual image can be accurately determined, effectively improving the accuracy of the virtual image model construction.

[0049] Furthermore, the construction of the virtual image 3DGS model in step S100 further includes: obtaining the projection coordinates of the projection virtual image in the world coordinate system based on each projection virtual image and the corresponding transformation matrix, so as to generate the virtual image 3D model based on the projection coordinates; and constructing the virtual image 3DGS model according to the preset model parameters and the virtual image 3D model.

[0050] In some embodiments, based on the projected virtual images and transformation matrices from different viewpoints, the 3D coordinates of each pixel of each projected virtual image in the ADAS coordinate system are solved by minimizing the 3D distance method; specifically: Obtain the unit observation vector in the eye position coordinate system { }j=1,...M: ,

[0051] in, These are the homogeneous coordinates observed in the normalize plane. It is the normalized unit direction vector. They are homogeneous coordinates.

[0052] This is a vector in the eye coordinate system. Transforming it to the world coordinate system yields the equation of the observation ray: The distance from the feature point to the ray is:

[0053] The objective function is defined as the feature points. Minimize the sum of squares of all observed distances (least squares):

[0054] The optimization problem is equivalent to finding the least-squares solution to a system of linear equations, and solving for the 3D coordinates of each projected virtual image pixel in the ADAS coordinate system. (i.e., the projected coordinates), based on 3D coordinates Generate virtual 3D models.

[0055] Furthermore, the radius, color, and center point value of each projection coordinate are set (i.e., the preset model parameters); based on the preset model parameters and the generated virtual 3D model, a virtual 3DGS model is obtained, and each projection coordinate point is transformed into a sphere representation.

[0056] In the above technical solution, coordinate transformation and parameter information are fully utilized, so that the virtual image 3DGS model can not only accurately reflect the actual projection situation, but also be customized according to preset requirements to meet different needs, making this application more flexible.

[0057] Furthermore, obtaining the spherical harmonic coefficients in step S200 includes: performing perspective projection on the virtual image 3DGS model to several viewpoints to obtain image pixels as projection ground truth; establishing a dynamic distortion fitting equation based on the difference between the projection ground truth and the corresponding virtual image pixels, and the spherical harmonic functions corresponding to several viewpoints, so as to obtain the spherical harmonic coefficients according to the dynamic distortion fitting equation.

[0058] In some embodiments, the vertices of the virtual 3DGS model are projected from the perspective of N eye-view cameras to obtain the image pixels of the eye-view cameras as the ground truth of dynamic distortion, i.e., the projection ground truth; the vertices are transformed into the eye-view coordinate system through the transformation matrix between the ADAS coordinate system and the eye-view coordinate system, and then transformed into the eye-view image plane and pixel plane through the intrinsic parameters of the eye-view cameras to obtain the virtual image pixels; then the difference between the projection ground truth and the corresponding virtual image pixels is calculated to establish the spherical harmonic function fitting linear equation under N viewpoints (i.e., the dynamic distortion fitting equation): Based on the linear equation fitted by the spherical harmonic function, the spherical harmonic coefficients, i.e., the dynamic distortion coefficients, are solved using the least squares method.

[0059] in,[ ] is a spherical harmonic function, [ ] represents the spherical harmonic coefficients. The dynamic distortion is fitted using a spherical harmonic function of order l=2. Without considering performance factors, the larger the value of l, the more viewing angles N has, the higher the fitting accuracy, and the closer it is to the real dynamic distortion.

[0060] Furthermore, the calculated spherical harmonic coefficients are applied to the virtual image 3DGS model, and the model parameters are also adjusted to obtain an updated implementation model.

[0061] In the above technical solution, the spherical harmonic function can effectively fit complex dynamic distortion situations. In this way, the imaging distortion caused by optical design and manufacturing assembly errors can be accurately corrected, so that the distortion of the image under different eye positions can be effectively controlled, thereby improving the imaging quality and visual effect.

[0062] Furthermore, the step S300 of obtaining the target virtual image and projecting it includes: obtaining multiple Gaussian spheres and corresponding eye poses based on the updated virtual image 3DGS model, and sorting each Gaussian sphere according to depth information, so as to project each Gaussian sphere onto a 2D image plane according to the sorting result and the eye pose; wherein, during the projection process, it also includes: performing distortion correction on the initial virtual image projected onto the 2D image plane to obtain the target virtual image; and projecting the target virtual image onto the target plane.

[0063] In some embodiments, the eye position and pose of the observed virtual image 3DGS model are obtained through a DMS camera; based on the eye position and pose, the virtual image 3DGS model is projected onto a 2D image plane. Each Gaussian sphere in the 3DGS is represented by its position (mean), covariance matrix (describing shape and orientation), color, and opacity (alpha); the projection onto the 2D image plane includes the position of the projection mean and the projection covariance matrix; the 3DGS uses a fast differentiable raster to divide the screen into 16x16 pixel tiles, and each 3D Gaussian sphere may overlap with multiple tiles, so it needs to be instantiated within these tiles.

[0064] Specifically, the Radix sorting algorithm on the GPU is used to sort all instantiated Gaussian spheres based on their depth information; the 3DGS Splatting rendering process is adopted to project the 3D Gaussian spheres in the 3DGS model onto the 2D image space. Dynamic distortion correction is added to the projection process to obtain the image coordinates after distortion correction (i.e., the target virtual image).

[0065] Furthermore, the rendered, distortion-corrected target virtual image is transmitted to the TFT optical engine, projected onto the windshield through three optical reflections, and then enters the human eye through specular reflection. The virtual image will undergo subtle changes according to the driver's posture, and the complete projection effect can be seen as long as it is within the dynamic range of the eye box.

[0066] The above technical solution can reasonably process the depth information of the virtual image, so that during the projection process, virtual image elements of different depths can be processed in Gaussian order from far to near, thereby reducing the number of required mixing operations.

[0067] In summary, this application can construct a HUD virtual image 3DGS model based on different viewpoints and eye positions, and use spherical harmonic functions to fit the dynamic distortion of the HUD imaging model, thereby more accurately fitting the dynamic distortion situation, reducing image distortion caused by changes in eye position, effectively improving the accuracy of distortion correction, and providing users with a clearer and more realistic imaging effect; at the same time, it avoids exhaustive calculation of a large number of positions, simplifies the distortion correction process, reduces the amount of computation, and thus significantly improves processing efficiency.

[0068] Furthermore, based on the same concept, please refer to Figure 2 This application also provides an ARHUD imaging system, comprising at least: a virtual image generation module, used to construct a virtual image 3DGS model based on a calibration object, and obtain a target virtual image based on the virtual image 3DGS model, so as to transmit the target virtual image to an optical projection device; the optical projection device is used to project the target virtual image onto a target plane.

[0069] For further details, please see Figure 3 The virtual image generation module includes at least a memory and a processor; the memory is used to store computer instructions for multiple functional layers, see [link to relevant documentation]. Figure 4 The functional layer includes at least a construction module, a fitting module, and a generation module; the processor is used to execute computer instructions for each of the multiple functional layers stored in the memory.

[0070] Furthermore, the construction module is used to obtain the transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints based on the calibration object, and to construct a virtual image 3DGS model based on the map data and the transformation matrix; the fitting module is used to fit the dynamic distortion of the HUD imaging model using spherical harmonic functions based on the virtual image 3DGS model to obtain a virtual image 3DGS model including spherical harmonic functions; the generation module is used to render based on the updated virtual image 3DGS model to obtain the target virtual image.

[0071] Furthermore, the system also includes a sensor for acquiring the eye pose of the driver's observation of the updated virtual image 3DGS model, so that the generation module renders the updated virtual image 3DGS model based on the eye pose.

[0072] It should be noted that the preferred sensor for obtaining eye position and pose is the DMS (Driver Monitoring System), so as to obtain the eye position and pose when the driver observes the updated virtual 3DGS model through the DMS system.

[0073] In some embodiments, the construction module first uses an ADAS system and an eye-position camera to obtain the 3D and 2D coordinates of a fixed-position calibration object in front of the vehicle in the world coordinate system and the eye-position coordinate system, respectively, and obtains a transformation matrix through the PnP algorithm. Following the above steps, within the eye box range, the eye-position camera captures eye-position images from different perspectives, thereby obtaining transformation matrices corresponding to several perspectives. A calibration chart is placed in the TFT optical engine, and the chart data from different perspectives is collected by the eye-position camera. Then, a virtual image 3D model is constructed based on the chart data and the pre-obtained transformation matrix. Furthermore, a virtual image 3DGS model is obtained based on the model parameters (such as the radius, color, and center point value of 3D points) and the virtual image 3D model.

[0074] The fitting module obtains the projection ground truth and virtual image pixels based on the virtual image 3DGS model and the transformation matrix, respectively. Then, it establishes a system of linear equations for fitting the spherical harmonic function under several viewpoints based on the difference between the projection ground truth and the virtual image pixels, and solves the spherical harmonic coefficients using the least squares method. Finally, it updates the virtual image 3DGS model based on the calculated spherical harmonic coefficients.

[0075] In the generation module, the eye position and pose of the observed virtual image 3DGS model are obtained through the DMS camera. The virtual image 3DGS model is rendered based on the eye position and pose using OpenGL to generate a virtual image. The rendering and projection process includes dynamic distortion correction. Finally, the virtual image is transmitted to the TFT optical engine and projected onto the windshield to achieve the effect of combining virtual and real images.

[0076] In the above technical solution, the modules work together to achieve efficient and accurate ARHUD virtual image correction, thereby improving the user's experience with ARHUD.

[0077] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0078] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0079] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0080] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0081] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0082] Although the description of this application has been made in conjunction with the specific embodiments described above, it will be apparent to those skilled in the art that many substitutions, modifications, and variations can be made based on the foregoing. Therefore, all such substitutions, modifications, and variations are included within the spirit and scope of the appended claims.

Claims

1. An ARHUD imaging method, characterized in that, Includes the following steps: Based on the calibration object, the transformation matrix between the world coordinate system and the eye position coordinate system under several viewpoints is obtained, and a virtual image 3DGS model is constructed based on the map data and the transformation matrix. Based on the virtual image 3DGS model, the dynamic distortion of the HUD imaging model is fitted using spherical harmonic functions to obtain a virtual image 3DGS model including spherical harmonic functions. Additionally, rendering is performed based on the updated virtual image 3DGS model to obtain the target virtual image and project it for display.

2. The ARHUD imaging method according to claim 1, characterized in that, The process of obtaining the transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints includes: Place a calibration object at a preset position on the target vehicle and obtain the 3D coordinates of the calibration object in the world coordinate system; Based on a preset eye box range, obtain eye position images of the calibration object from several viewpoints, and obtain the corresponding 2D coordinates of the calibration object in the eye position coordinate system based on the eye position images; Furthermore, based on the 3D coordinates and each 2D coordinate, the transformation matrix corresponding to each viewpoint is obtained.

3. The ARHUD imaging method according to claim 2, characterized in that, The construction of the virtual image 3DGS model includes: Project the calibration chart and collect chart data from several viewpoints based on the eye coordinate system; In addition, feature extraction and feature matching are performed on the calibration chart and the chart data respectively to obtain the projected virtual image.

4. The ARHUD imaging method according to claim 3, characterized in that, The construction of the virtual image 3DGS model also includes: Based on each projected virtual image and its corresponding transformation matrix, the projected coordinates of the projected virtual image in the world coordinate system are obtained, and a 3D model of the virtual image is generated based on the projected coordinates. Furthermore, a virtual image 3DGS model is constructed based on preset model parameters and the virtual image 3D model.

5. The ARHUD imaging method according to claim 4, characterized in that, The process of obtaining spherical harmonic coefficients includes: Based on the virtual image 3DGS model, perspective projection is performed on several viewpoints to obtain image pixels as projection truth values. Based on the difference between the true projection value and the corresponding virtual image pixel, and the spherical harmonic functions corresponding to several viewpoints, a dynamic distortion fitting equation is established to obtain the spherical harmonic coefficients according to the dynamic distortion fitting equation.

6. The ARHUD imaging method according to claim 5, characterized in that, The process of obtaining the target virtual image and projecting it includes: Multiple Gaussian spheres and their corresponding eye poses are obtained based on the updated virtual image 3DGS model. The Gaussian spheres are sorted according to depth information, and then projected onto a 2D image plane according to the sorting results and the eye poses. The projection process also includes: performing distortion correction on the initial virtual image projected onto the 2D image plane to obtain the target virtual image. And, the virtual image of the target is projected onto the target plane.

7. An ARHUD imaging system, characterized in that, At least including: The virtual image generation module is used to construct a virtual image 3DGS model based on the calibration object, and obtain a target virtual image based on the virtual image 3DGS model, so as to transmit the target virtual image to the optical projection device; The optical projection device is used to project the virtual image of the target onto the target plane.

8. The ARHUD imaging system according to claim 7, characterized in that, The virtual image generation module includes at least a memory and a processor; The memory is used to store computer instructions for multiple functional layers, each functional layer including at least a construction module, a fitting module, and a generation module. The processor is used to execute computer instructions for each of the multiple functional layers stored in memory.

9. The ARHUD imaging system according to claim 8, characterized in that, The construction module is used to obtain the transformation matrix between the world coordinate system and the eye position coordinate system from several viewpoints based on the calibration object, and to construct a virtual image 3DGS model based on the map data and the transformation matrix. The fitting module is used to fit the dynamic distortion of the HUD imaging model based on the virtual image 3DGS model using spherical harmonic functions, so as to obtain a virtual image 3DGS model including spherical harmonic functions. The generation module is used to render based on the updated virtual image 3DGS model to obtain the target virtual image.

10. The ARHUD imaging system according to claim 7, characterized in that, The system also includes a sensor for acquiring the eye position and pose of the driver's observation of the updated virtual image 3DGS model, so that the generation module renders the updated virtual image 3DGS model based on the eye position and pose.

Images