A CNN-based AR-HUD projection image distortion pre-correction method
By extracting features from AR-HUD projected images using a CNN-based method and generating a distortion stream for pixel resampling, the problem of optical distortion in AR-HUD systems is solved, simplifying optical design and improving processing efficiency and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- UNIV OF ELECTRONICS SCI & TECH OF CHINA
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-05
AI Technical Summary
In existing AR-HUD systems, virtual images suffer optical distortion due to projection from multiple mirrors and aspherical windshields. Traditional methods suffer from high complexity in optical design and manufacturing, large storage requirements, and abrupt changes in correction effects caused by perspective interpolation calculations.
A CNN-based approach is adopted to extract distorted image features through a model encoder, fit the distortion coefficients using a fully connected layer, generate a distortion stream, and perform pixel resampling to achieve pre-correction of the projected image.
It reduces the difficulty of optical design and manufacturing, improves processing efficiency, eliminates image jump problems when switching perspectives, enhances user experience, and ensures real-time performance and high efficiency.
Smart Images
Figure CN122156020A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically the field of AR-HUD projection image pre-distortion correction technology, and more specifically, a CNN-based AR-HUD projection image distortion pre-correction method. Background Technology
[0002] With the rapid development of the automotive industry, various unique solutions have emerged in the field of in-vehicle augmented reality, including augmented reality head-up displays (AR-HUDs), which can project road prompts, traffic signs, and warning information onto the windshield in real time. However, AR-HUD systems rely on multiple freeform mirrors to project virtual images into the viewer's eyes, resulting in a certain degree of distortion, which affects the driving experience and the accuracy of information. Therefore, how to pre-correct the distortion of the projected image to rectify optical distortion caused by projection and windshield reflection has become an important research direction for improving AR-HUD performance.
[0003] The mainstream methods for AR-HUD projection image distortion correction include traditional optical design methods and image processing methods. In the early stages of AR-HUD development, distortion correction was mainly achieved by improving the structure of two freeform mirrors in the projection system. Constructing a freeform reflection system adapted to the shape of the windshield can minimize image distortion at a specific viewing point. The literature "Z. An, X. Meng, X. Ji, X. Xu, Y. Liu, Design and performance of anoff-axis free-form mirror for a rear mounted augmented-reality head-updisplay system, IEEE Photonics J. 13 (1) (2021) 1–15" uses an optical design scheme based on an off-axis freeform mirror, adopting a step-by-step optimization strategy, and combining aspherical substrate and freeform surface modulation to balance aberration correction. However, this method requires optical optimization design for off-axis optical paths, resulting in high design and manufacturing complexity and lacking universality in practical applications.
[0004] With the advancement of image processing technology, projection image pre-correction methods based on image processing have gradually developed. The paper "X. Gao, J. Werner, M. Necker, W. Stork, A calibration method for automotive augmented reality head-up displays using a chessboard and warpingmaps, in: Twelfth International Conference on Machine Vision. Vol. 11433,ICMV 2019, International Society for Optics and Photonics, 2020, p. 114332W" proposes establishing and storing a pre-distortion mapping table based on several typical human eye positions, while using interpolation methods to generate correction results for atypical positions. However, constructing the mapping table requires a large amount of data, placing high demands on storage capacity. Furthermore, interpolating the distortion model under different viewpoints inevitably leads to abrupt changes in distortion correction results, negatively impacting the user experience.
[0005] Given the limitations of traditional methods, distortion correction methods based on neural networks are gradually emerging. Compared with traditional methods, neural network methods have stronger generalization ability and higher processing efficiency, while also helping to reduce problems caused by interpolation, and have good application prospects. Summary of the Invention
[0006] This invention provides a CNN-based AR-HUD projection image distortion pre-correction method to solve the technical problems in existing vehicle-mounted AR-HUD systems, where optical distortion occurs due to virtual images being projected through multiple mirrors and aspherical windshields. Furthermore, existing correction methods (such as optical design methods for optimizing freeform mirrors and image processing methods for establishing pre-distortion mapping tables) suffer from high complexity in optical design and manufacturing, large system storage requirements, and are prone to abrupt changes in correction results when interpolating at different viewpoints.
[0007] To solve the above problems, the technical solution adopted by the present invention is as follows:
[0008] A CNN-based AR-HUD projection image distortion pre-correction method includes:
[0009] Step S1: Obtain the original image of the AR-HUD projection and the distorted image output by the system, and use it as pre-correction data to input the CNN network;
[0010] Step S2: Extract feature information of the distorted image through the model encoder in the CNN network to obtain a three-dimensional feature map; wherein, the model encoder sequentially includes an extraction layer, a max pooling layer, a reduction layer and a CBAM attention module;
[0011] Step S3: Fit the distortion coefficients through a fully connected layer, and use the fully connected layer to convert the three-dimensional feature map into a one-dimensional distortion coefficient tensor;
[0012] Step S4: Use the one-dimensional distortion coefficient tensor and mapping transformation to generate a distortion stream, and perform pixel resampling on the projected image according to the distortion stream to obtain and output the AR-HUD projection correction image.
[0013] Furthermore, step S1 specifically includes the following sub-steps:
[0014] S1-1. Acquire typical AR-HUD projection images of driving scenarios, including lane markings, speed prompts, and road turn indicators.
[0015] S1-2. Import the original AR-HUD projection image into the AR-HUD system simulation model constructed by optical design software, and simulate the imaging distortion effect under the preset projection distance and specific field of view using a ray tracing algorithm to form a barrel distortion image as the distortion image, wherein the preset projection distance is 7.5m;
[0016] S1-3. The original AR-HUD projection image and the barrel distortion image are cropped to a size of 1024×1024 to ensure consistent image size, forming the pre-corrected data pair for input into the CNN network.
[0017] Further, in step S2: the extraction layer extracts feature information from the distorted image by using different convolution kernels on multiple branches to focus on image features of different scales; the max pooling layer converts the extracted image features into a fixed-size feature map so that the CNN network can be adapted to input images of various sizes; the reduction layer gradually shrinks the feature map and widens the number of feature map channels; in the reduction layer, except for the 1×1 convolution kernel, the stride of all other convolution kernels is adjusted to 2; while preserving texture information, 4×4 and 2×2 max pooling layers are used in the branches of symmetric convolution kernels to reduce the resolution of the feature map and reduce the information redundancy of the feature map.
[0018] Furthermore, the CBAM attention module includes a channel attention module;
[0019] The feature map obtained after the reduction layer is input as the input feature map F to the channel attention module. Average pooling and max pooling operations are used to aggregate the spatial information of the input feature map F to generate average pooling features. and max pooling features ;
[0020] The average pooling feature and the max pooling feature Forwarded to a multilayer perceptron shared network (MLP) to generate a channel attention map. The calculation formula is as follows:
[0021]
[0022] in, This represents the sigmoid function. This indicates the average pooling operation. This represents the max pooling operation;
[0023] The input feature map F is compared with the channel attention map. Perform convolution operations to obtain channel-refined feature maps. The calculation formula is as follows: .
[0024] Furthermore, the CBAM attention module also includes a spatial attention module, which refines the channel feature map. The processing includes:
[0025] The channel refined feature map Average pooling and max pooling operations are applied to obtain spatial average pooling features and spatial max pooling features, respectively. These two features are concatenated and then convolved through a standard convolutional layer to generate a two-dimensional spatial attention map. The calculation formula is as follows:
[0026]
[0027] in, This indicates a convolution operation with a kernel size of 7×7;
[0028] Refine the feature map of the channel With the aforementioned two-dimensional spatial attention map Perform convolution operations to obtain refined feature maps. The refined feature map That is, the 3D feature map is output to the fully connected layer, and its calculation formula is: .
[0029] Furthermore, step S3 specifically includes:
[0030] The barrel distortion degree is described using the standard radial distortion model, and its formula is as follows:
[0031]
[0032]
[0033] in, Let r be the center of the distortion model, and r be the Euclidean distance from a pixel to the center of the distortion model. The barrel distortion coefficient and ;
[0034] The fully connected layer converts the 3D feature map into a one-dimensional distortion coefficient tensor, and only the first distortion coefficient is processed. Perform a fitting output.
[0035] Further, in step S4, generating the distortion flow using the distortion coefficient tensor and mapping transformation specifically includes:
[0036] The image domain is mapped to the flow domain, transforming the correction problem into pixel-level prediction of the distortion flow; the distortion flow is a two-dimensional vector field mapped from the distorted image to the AR-HUD projected corrected image.
[0037] The fitted first distortion coefficient As input, a distorted stream is generated based on the mapping transformation. The transformation calculation formula is as follows:
[0038]
[0039]
[0040] in, and These are the x and y coordinates of a pixel in the undistorted image, respectively. and These are the x and y coordinates of the pixels in the distorted image, respectively.
[0041] Further, in step S4, resampling the projected image according to the distortion stream specifically includes: resampling pixels according to the calculated distortion stream. The AR-HUD projection correction image is generated by adjusting the position of each pixel in the original AR-HUD projection image through the pixel resampling module.
[0042] Compared with the prior art, the beneficial effects of the present invention are:
[0043] (1) This invention extracts distorted image features through a CNN network and directly fits and generates a distortion stream for pixel resampling. This enables distortion correction to be completed before projection, overcoming optical image distortion caused by aspherical windshields. It avoids the complexity of using freeform mirrors to eliminate image distortion in traditional AR-HUD projection systems (i.e., the system only needs to use a combination of projection-enlarged spherical mirrors), significantly reducing the optical design and manufacturing difficulty of the system. At the same time, compared with the traditional interpolation mapping table method, the neural network scheme of this invention has stronger generalization ability and higher processing efficiency, completely eliminating the image jump problem caused by perspective switching interpolation, and improving the user experience.
[0044] (2) In the reduction layer of the model encoder, by adjusting the stride of a specific convolution kernel and using a symmetrical max pooling layer, the present invention can effectively reduce the resolution of the feature map and reduce the information redundancy of the feature map while preserving the image texture information as much as possible, thereby significantly reducing the computational cost of the model.
[0045] (3) This invention innovatively introduces a CBAM attention module (including channel attention and spatial attention) into the model encoder. The channel attention mechanism enhances the network's attention to important feature channels and ensures the global consistency of information fusion; the spatial attention mechanism effectively captures local features in the image, enabling the network to focus on the key distorted parts of the image, which greatly optimizes the restoration effect of the corrected image details.
[0046] (4) When using the standard radial distortion model to describe the degree of barrel distortion, this invention only fits and outputs the first distortion coefficient through a fully connected layer, and generates a pixel-level predicted distortion flow by combining a mapping transformation. This design minimizes the computation of redundant parameters while fully meeting the basic correction accuracy requirements, ensuring the real-time performance and efficiency of the pre-correction algorithm in the vehicle environment.
[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description
[0048] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 This is a general flowchart of an embodiment of the present invention;
[0050] Figure 2 This is a diagram of the distortion pre-correction network structure according to an embodiment of the present invention;
[0051] Figure 3 This is the CBAM attention module in an embodiment of the present invention;
[0052] Figure 4 This is the CBAM channel attention module in an embodiment of the present invention;
[0053] Figure 5 This is the CBAM spatial attention module of an embodiment of the present invention;
[0054] Figure 6 Two sets of barrel distortion images directly projected by the AR-HUD system in the test set of this invention embodiment;
[0055] Figure 7 These are two sets of images (two groups) that underwent pre-correction processing in the test set according to an embodiment of the present invention.
[0056] Figure 8 These are two sets of images (two sets) showing the effect of pre-correcting images in the test set and then projecting them onto the AR-HUD system, according to an embodiment of the present invention. Detailed Implementation
[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.
[0058] See Figures 1 to 7 In the embodiments described below, the present invention provides a CNN-based AR-HUD projection image distortion pre-correction method, the overall flowchart of which is shown below. Figure 1 As shown, the CNN model framework is as follows: Figure 2 As shown. The method specifically includes the following steps:
[0059] Step S1: Obtain the original image of the AR-HUD projection and the distorted image output by the system, and use it as pre-correction data to input the CNN network.
[0060] Specifically, it includes the following sub-steps:
[0061] S1-1. Acquire typical AR-HUD projection images of driving scenarios, including lane markings, speed prompts, and road turning indicators.
[0062] S1-2. Import the original AR-HUD projection image into the AR-HUD system simulation model constructed by optical design software (such as ZEMAX), and simulate the imaging distortion effect under the preset projection distance and specific field of view (i.e. preset field of view conditions) by using a ray tracing algorithm to form a barrel distortion image as the distortion image, wherein the preset projection distance is 7.5m;
[0063] S1-3. The original AR-HUD projection image and the barrel distortion image are cropped to a size of 1024×1024 to ensure consistent image size, forming the pre-corrected data pair for input into the CNN network.
[0064] Step S2 involves extracting feature information from the distorted image using the model encoder in the CNN network to obtain a three-dimensional feature map. The model encoder sequentially includes an extraction layer, a max-pooling layer, a reduction layer, and a CBAM attention module. Specifically, it includes:
[0065] S2-1, The extraction layer extracts feature information from the distorted image by focusing on image features at different scales using different convolution kernels on multiple branches.
[0066] S2-2, The max pooling layer converts the extracted image features into a fixed-size feature map, so that the CNN network can be adapted to input images of various sizes.
[0067] S2-3. The reduction layer gradually shrinks the feature map and widens the number of feature map channels. In the reduction layer, except for the 1×1 convolution kernel, the stride of all other convolution kernels is adjusted to 2. While preserving texture information as much as possible, 4×4 and 2×2 max pooling layers are used in the branches of the symmetric convolution kernel to reduce the resolution of the feature map and reduce the information redundancy of the feature map.
[0068] S2-4. The feature map obtained after the reduction layer is input as the input feature map F and input to the CBAM attention module (e.g., Figure 3 As shown). The CBAM attention module includes a channel attention module (as shown). Figure 4 ) and spatial attention module ( Figure 5 ).
[0069] S2-5. Through the channel attention module, the representation of important channel features is enhanced, irrelevant feature channels are suppressed, and the role of different feature channels is strengthened to ensure global consistency during information fusion. For example... Figure 4As shown, the spatial information of the input feature map F is aggregated using average pooling and max pooling operations to generate two different spatial context descriptors: and , represent the average pooling feature and the max pooling feature, respectively. The average pooling feature... and the max pooling feature Forwarded to a multilayer perceptron shared network (MLP) to generate a channel attention map. The calculation formula is as follows:
[0070]
[0071] in, This represents the sigmoid activation function; This represents the average pooling operation performed on the feature map; This represents the max pooling operation performed on the feature map; MLP stands for Multilayer Perceptron Shared Network.
[0072] The input feature map F is compared with the channel attention map. Perform convolution operations to obtain channel-refined feature maps. The calculation formula is as follows: ,in, This represents a convolution operation that multiplies element by element.
[0073] S2-6. Refine the feature map of the channel using the spatial attention module. Processing is performed to capture local features in the image and optimize image detail restoration. For example... Figure 5 As shown, the refined feature map of the channel Average pooling and max pooling operations are applied to obtain spatial average pooling features and spatial max pooling features, respectively. These two features are concatenated and then convolved through a standard convolutional layer to generate a two-dimensional spatial attention map. The calculation formula is as follows:
[0074]
[0075] in, This represents the sigmoid activation function; This indicates a convolution operation with a kernel size of 7×7.
[0076] Refine the feature map of the channel With the aforementioned two-dimensional spatial attention map Perform convolution operations to obtain refined feature maps. The calculation formula is as follows:
[0077]
[0078] The refined feature map That is, the three-dimensional feature map is output to the subsequent fully connected layer.
[0079] Step S3 involves fitting distortion coefficients using a fully connected layer, and then converting the three-dimensional feature map into a one-dimensional distortion coefficient tensor using the fully connected layer. Specifically, this includes:
[0080] The barrel distortion degree is described using the standard radial distortion model, and its formula is as follows:
[0081]
[0082]
[0083] in, The center of the distortion model (usually the center coordinates of the image); x and y are the coordinates of the current pixel; r is the distance from pixel (x, y) to the center of the distortion model. The Euclidean distance; L(r) is the radial distortion factor; (where i = 1, 2, ..., n) are the barrel distortion coefficients, and satisfy... (The larger the absolute value, the more severe the barrel distortion.)
[0084] To reduce computational load, only the first distortion coefficient is considered. Fitting the data is sufficient to meet the basic correction requirements. Specifically, we convert the extracted 3D feature map into a one-dimensional distortion coefficient tensor using the fully connected layer, and only apply the first distortion coefficient... Perform a fitting output.
[0085] Step S4 involves generating a distortion stream using the one-dimensional distortion coefficient tensor and mapping transformation, and then resampling the projected image pixels based on the distortion stream to obtain and output the AR-HUD projection correction image. Specifically, this includes:
[0086] S4-1. Map the image domain to the flow domain, transforming the correction problem into pixel-level prediction of the distortion flow; the distortion flow is a two-dimensional vector field mapped from the distorted image to the AR-HUD projected corrected image, which describes how each pixel in the input distorted image should be displaced to obtain the corrected image.
[0087] S4-2, using the fitted first distortion coefficient As input, a distorted stream is generated based on the mapping transformation. The transformation calculation formula is as follows:
[0088]
[0089]
[0090] in, The calculated distortion flow vector; and These are the x and y coordinates of the pixels in the undistorted image (i.e., the original projected image); and These are the x and y coordinates of the pixels in the distorted image (i.e., the warped image); The center of the distortion model is r; r is the distance from the undistorted pixel to the distortion center. )and( ) represent the displacement components of the pixel in the horizontal and vertical directions, respectively.
[0091] S4-3, Based on the calculated distortion flow The AR-HUD projection correction image is generated by adjusting the position of each pixel in the original AR-HUD projection image through the pixel resampling module.
[0092] The images from the test set of this invention were used as input to the AR-HUD system in Zemax software for simulation. The projection effect after simulation is as follows: Figure 6 As shown, barrel distortion occurs. To solve the projection distortion problem of the AR-HUD system, the original image is first pre-corrected using the method of this invention, that is, after calculating the distortion flow, it is adjusted by the pixel resampling module to obtain the image shown. Figure 7 The pre-corrected image shown; then... Figure 7 The pre-corrected image is used as input to the AR-HUD, and the image formed after projection by the AR-HUD system is as follows: Figure 8 As shown in the figure. Experiments show that the present invention effectively overcomes the optical image distortion caused by aspherical windshields in vehicle-mounted AR-HUD systems by pre-correcting the distortion of the original projected image of the AR-HUD system using a CNN model, and avoids the complexity of using freeform mirrors to eliminate distortion in traditional solutions.
[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A CNN-based method for pre-correcting distortion of AR-HUD projection images, characterized in that, include: Step S1: Obtain the original image of the AR-HUD projection and the distorted image output by the system, and use it as pre-correction data to input the CNN network; Step S2: Extract feature information of the distorted image through the model encoder in the CNN network to obtain a three-dimensional feature map; wherein, the model encoder sequentially includes an extraction layer, a max pooling layer, a reduction layer and a CBAM attention module; Step S3: Fit the distortion coefficients through a fully connected layer, and use the fully connected layer to convert the three-dimensional feature map into a one-dimensional distortion coefficient tensor; Step S4: Use the one-dimensional distortion coefficient tensor and mapping transformation to generate a distortion stream, and perform pixel resampling on the projected image according to the distortion stream to obtain and output the AR-HUD projection correction image.
2. The method for pre-correcting distortion of AR-HUD projection images based on CNN according to claim 1, characterized in that, Step S1 specifically includes the following sub-steps: S1-1. Acquire typical AR-HUD projection images of driving scenarios, including lane markings, speed prompts, and road turn indicators. S1-2. Import the original AR-HUD projection image into the AR-HUD system simulation model constructed by optical design software, and simulate the imaging distortion effect under the preset projection distance and specific field of view using a ray tracing algorithm to form a barrel distortion image as the distortion image, wherein the preset projection distance is 7.5m; S1-3. The original AR-HUD projection image and the barrel distortion image are cropped to a size of 1024×1024 to ensure consistent image size, forming the pre-corrected data pair for input into the CNN network.
3. The method for pre-correcting distortion of AR-HUD projection images based on CNN according to claim 1, characterized in that, In step S2: the extraction layer extracts feature information from the distorted image by focusing on image features of different scales using different convolution kernels on multiple branches; the max pooling layer converts the extracted image features into a fixed-size feature map so that the CNN network can be adapted to input images of various sizes; the reduction layer gradually shrinks the feature map and widens the number of feature map channels; in the reduction layer, except for the 1×1 convolution kernel, the stride of all other convolution kernels is adjusted to 2; While preserving texture information, 4×4 and 2×2 max pooling layers are used in the branches of the symmetric convolution kernel to reduce the resolution of the feature map and reduce information redundancy in the feature map.
4. The CNN-based AR-HUD projection image distortion pre-correction method according to claim 3, characterized in that, The CBAM attention module includes a channel attention module; The feature map obtained after the reduction layer is input as the input feature map F to the channel attention module. Average pooling and max pooling operations are used to aggregate the spatial information of the input feature map F to generate average pooling features. and max pooling features ; The average pooling feature and the max pooling feature Forwarded to a multilayer perceptron shared network (MLP) to generate a channel attention map. The calculation formula is as follows: in, This represents the sigmoid function. This indicates the average pooling operation. This represents the max pooling operation; The input feature map F is compared with the channel attention map. Perform convolution operations to obtain channel-refined feature maps. The calculation formula is as follows: .
5. The CNN-based AR-HUD projection image distortion pre-correction method according to claim 4, characterized in that, The CBAM attention module also includes a spatial attention module, which refines the channel feature map. The processing includes: The channel refined feature map Average pooling and max pooling operations are applied to obtain spatial average pooling features and spatial max pooling features, respectively. These two features are concatenated and then convolved through a standard convolutional layer to generate a two-dimensional spatial attention map. The calculation formula is as follows: in, This indicates a convolution operation with a kernel size of 7×7; Refine the feature map of the channel With the aforementioned two-dimensional spatial attention map Perform convolution operations to obtain refined feature maps. The refined feature map That is, the 3D feature map is output to the fully connected layer, and its calculation formula is: .
6. The CNN-based AR-HUD projection image distortion pre-correction method according to claim 1, characterized in that, Step S3 specifically includes: The barrel distortion degree is described using the standard radial distortion model, and its formula is as follows: in, Let r be the center of the distortion model, and r be the Euclidean distance from a pixel to the center of the distortion model. The barrel distortion coefficient and ; The fully connected layer converts the 3D feature map into a one-dimensional distortion coefficient tensor, and only the first distortion coefficient is processed. Perform a fitting output.
7. The CNN-based AR-HUD projection image distortion pre-correction method according to claim 6, characterized in that, In step S4, the distortion coefficient tensor and mapping transformation are used to generate the distortion stream, specifically including: The image domain is mapped to the flow domain, transforming the correction problem into pixel-level prediction of the distortion flow; the distortion flow is a two-dimensional vector field mapped from the distorted image to the AR-HUD projected corrected image. The fitted first distortion coefficient As input, a distorted stream is generated based on the mapping transformation. The transformation calculation formula is as follows: in, and These are the x and y coordinates of a pixel in the undistorted image, respectively. and These are the x and y coordinates of the pixels in the distorted image, respectively.
8. The CNN-based AR-HUD projection image distortion pre-correction method according to claim 7, characterized in that, In step S4, pixel resampling of the projected image based on the distortion stream specifically includes: Based on the calculated distortion flow The AR-HUD projection correction image is generated by adjusting the position of each pixel in the original AR-HUD projection image through the pixel resampling module.