Electronic rearview mirror real-time image defogging method based on millimeter wave radar depth prior fusion

Abstract

The invention relates to the technical field of computer vision, image processing and intelligent automobile sensor fusion, in particular to an electronic rearview mirror real-time image defogging method based on millimeter-wave radar depth prior fusion, which comprises the following steps: synchronously acquiring point cloud data of a millimeter-wave radar and a foggy image of an electronic rearview mirror camera; processing the point cloud data to generate global radar depth priori information aligned with pixels of the foggy image; estimating a global atmospheric light value based on global radar depth prior information; and constructing an optimization cost function fused with global radar depth prior information, and solving the function through an optimization algorithm to obtain a globally optimized transmissivity graph. According to the method, the accurate depth prior information provided by the millimeter wave radar is fused, so that the problems of inaccurate depth estimation and poor defogging effect in extreme weather such as dense fog in the traditional method are effectively solved, the real color and details of the image can be recovered, and the contrast and definition of the image are improved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, image processing and intelligent vehicle sensor fusion technology, and in particular to a real-time image defogging method for electronic rearview mirrors based on millimeter-wave radar depth prior fusion. Background Technology

[0002] Electronic rearview mirrors (Camera Monitor Systems, CMS), as an alternative to traditional optical rearview mirrors, offer significant advantages such as reduced wind resistance, expanded field of vision, and protection from obstructions from interior clutter, and are gradually becoming standard equipment in intelligent connected vehicles. However, the performance of CMS is highly dependent on the image quality captured by the camera. In adverse weather conditions such as fog, haze, and sandstorms, suspended particles in the atmosphere absorb and scatter light, leading to degradation phenomena in the images captured by the camera, such as decreased contrast, color distortion, and loss of detail (i.e., "image fogging"), which seriously threatens driving safety.

[0003] Existing image dehazing techniques are mainly divided into two categories: 1. Image enhancement-based methods (such as histogram equalization and Retinex algorithm): These methods do not consider the physical causes of fog and directly improve image contrast. However, their effectiveness is limited in dense fog, they easily amplify noise, resulting in unnatural images, and they cannot restore the true depth and color of the scene.

[0004] 2. Physical Model-Based Methods: These methods utilize the Atmospheric Scattering Model as their core, with the formula I(x) = J(x)t(x) + A(1-t(x)) (where I(x) is the observed foggy image, J(x) is the fog-free image to be recovered, A is the global atmospheric light value, t(x) is the transmittance map, and the relationship between t(x) and the scene depth d(x) and the atmospheric scattering coefficient β is t(x) = exp(-β*d(x))). This type of method estimates the transmittance map t(x) and the atmospheric light value A to solve for the fog-free image J(x), but it has significant drawbacks: Prior-dependent types (such as dark channel prior DCP, color line prior, etc.): rely on statistical priors, which are prone to failure in areas such as the sky and white objects. The estimated depth map is coarse and inaccurate, resulting in problems such as halo and color cast after dehazing. Learning-assisted type: Uses networks such as CNN to help estimate model parameters, heavily relies on a large number of foggy / fog-free image pairs for training, has poor model generalization ability, and its performance may drop sharply in foggy conditions where it is not visible in the real world.

[0005] Furthermore, existing radar fusion-based target recognition solutions (such as CN114415173A) attempt to utilize radar information, but they are essentially a "post-remedial" strategy. After discovering a missed target during the recognition stage, they perform simple transmittance calculations and dehazing on local areas, which cannot comprehensively improve image quality. The process is cumbersome and cannot meet the real-time, global, and highly robust display requirements of electronic rearview mirrors. Moreover, their core function is still to serve target recognition rather than visual enhancement.

[0006] Millimeter-wave radar, as a core sensor in automobiles, boasts significant advantages such as accurate ranging and immunity to adverse weather conditions (fog, rain, snow, dust). However, current technologies do not deeply integrate its data with the image defogging task of the CMS (Continuous Image Management System). Therefore, there is an urgent need for a real-time image defogging method for electronic rearview mirrors based on millimeter-wave radar deep prior fusion to address the problems of poor image defogging performance, insufficient robustness, and difficulty in guaranteeing real-time performance in adverse weather conditions found in existing technologies. Summary of the Invention

[0007] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description and other accompanying drawings.

[0008] The purpose of this invention is to overcome the above-mentioned shortcomings and provide a real-time image defogging method for electronic rearview mirrors based on millimeter-wave radar depth prior fusion.

[0009] To achieve the above objectives, the technical solution of the present invention is: a real-time image dehazing method for electronic rearview mirrors based on millimeter-wave radar depth prior fusion, comprising the following steps: S1. Simultaneously acquire point cloud data from millimeter-wave radar and foggy images from electronic rearview mirror camera; S2. Process the point cloud data to generate global radar depth prior information aligned with the pixels of the foggy image; S3. Estimate global atmospheric light values ​​based on prior global radar depth information; S4. Construct an optimization cost function that integrates global radar depth prior information, and solve the function through an optimization algorithm to obtain a globally optimized transmittance map; S5. Based on the globally optimized transmittance map and global atmospheric light value, recover the global fog-free image according to the atmospheric scattering model and output it for display.

[0010] In some embodiments, in step S1, the point cloud data includes azimuth, pitch, distance, Doppler velocity, and reflection intensity information for each point.

[0011] In some embodiments, generating global radar depth prior information in step S2 specifically includes: S21. Based on the pre-defined coordinate transformation relationship between the millimeter-wave radar and the camera, transform the radar point cloud data from the radar coordinate system to the image pixel coordinate system; S22. Perform densification interpolation on the converted sparse depth point cloud to generate a dense radar depth map D_radar(x) with the same resolution as the foggy image. S23. Based on the atmospheric scattering model, convert the dense radar depth map D_radar(x) into a global radar transmission prior map T_prior(x).

[0012] In some embodiments, in step S21, the coordinate transformation relationship is obtained through joint calibration, specifically including: obtaining the rotation matrix R and translation vector T through the camera: radar hand-eye calibration method based on a specific calibration board, and obtaining the camera's intrinsic parameter matrix K and distortion coefficients through the Zhang Zhengyou calibration method.

[0013] In some embodiments, the densification interpolation process in step S22 employs an interpolation algorithm based on inverse distance weighting or a bilinear interpolation algorithm based on polar coordinate grids; during the interpolation process, a confidence weight map W(x) based on radar point reflection intensity and point cloud density is generated simultaneously.

[0014] In some embodiments, in step S23, the formula for calculating the global radar transmission prior image T_prior(x) is T_prior(x)=exp(-β*D_radar(x)), where β is the atmospheric scattering coefficient, which is determined by the algorithm developers based on experience or by debugging on a specific dataset, and is used to characterize the scattering ability of the fog medium.

[0015] In some embodiments, estimating the global atmospheric light value in step S3 specifically involves: S31. Based on the dense radar depth map D_radar(x), select the predetermined region with the largest depth value in the image. The predetermined region is the pixel region with the highest depth value, which is the top 0.1%-1% of the pixel regions. S32. Extract the brightness value of the corresponding pixel in the original foggy image I(x) of the predetermined region; S33. Use the maximum or average value of the brightness values ​​as an estimate of the global atmospheric light value A.

[0016] In some embodiments, in step S4, the optimized cost function is: ; Where I(x) is the foggy image, J(x) is the fog-free image to be recovered, t(x) is the transmittance to be solved, λ is the balance parameter, A is the global atmospheric light value, which is determined through experience or parameter tuning of a specific dataset and is used to control the importance weight of data fidelity terms and radar prior terms in the optimization process, and W(x) is the confidence weight map; the optimization algorithm is the gradient descent method, and the function is solved by the gradient descent method to obtain the refined transmittance map t(x).

[0017] In some embodiments, the process of generating the confidence weight map W(x) includes: W1. Calculate the initial reliability weight for each radar point based on the radar point cloud's reflection intensity (RCS) and distance information. W2. Using the same interpolation algorithm as the depth map densification, the sparse weights are interpolated to generate a preliminary weight map with the same resolution as the image. W3. Combining the local density distribution information of the radar point cloud, the preliminary weight map is corrected to finally obtain the confidence weight map W(x).

[0018] In some embodiments, in step S5, the formula for restoring the fog-free image is: J(x) = (I(x) - A) / max(t(x), t0) + A; where t0 is a preset minimum value, ranging from 0.01 to 0.1, used to avoid calculation abnormalities caused by the denominator t(x) being too small or 0.

[0019] By adopting the above technical solution, the beneficial effects of the present invention are: 1. Excellent defogging effect: By fusing precise depth prior information provided by millimeter-wave radar, it effectively solves the problems of inaccurate depth estimation and poor defogging effect of traditional methods in extreme weather conditions such as dense fog. It can restore the true color and details of the image and improve image contrast and clarity.

[0020] 2. Strong robustness: By introducing a confidence weight map and an adaptive fusion strategy, the depth prior constraint is strengthened in the reliable area of ​​radar data and weakened in the unreliable area. At the same time, the atmospheric light value is accurately estimated through radar depth prior, avoiding interference from foreground objects, so that the method can work stably under different fog conditions and different scenarios.

[0021] 3. High real-time performance: The highly parallelized algorithm pipeline is designed, and efficient interpolation and optimization algorithms are used to ensure that the delay of the entire defogging process is less than 33ms, which meets the real-time display requirements of electronic rearview mirrors.

[0022] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure.

[0023] Undoubtedly, such and other objects of the present invention will become more apparent after the following detailed description of the preferred embodiments, which are illustrated in various accompanying drawings and figures.

[0024] To make the above-mentioned beneficial effects and other objects, features and advantages of the present invention more apparent and understandable, one or more preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0025] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0026] In the accompanying drawings, the same parts use the same reference numerals, and the drawings are schematic and not necessarily drawn to actual scale.

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only one or more embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on such drawings without creative effort.

[0028] Explanation of key figure labels: Figure 1 This is a flowchart of a real-time image defogging method for electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to the present invention. Detailed Implementation

[0029] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and examples, so that the process of how the present invention uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly. It should be noted that, as long as there is no conflict, the various embodiments and features in the various embodiments of the present invention can be combined with each other, and the resulting technical solutions are all within the protection scope of the present invention.

[0030] Furthermore, numerous specific details are set forth in the following description for illustrative purposes to provide a thorough understanding of the embodiments of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without the specific details or particular methods described herein.

[0031] Please see Figure 1 This invention provides a real-time image dehazing method for electronic rearview mirrors based on millimeter-wave radar depth prior fusion, comprising the following steps: S1. Simultaneously acquire point cloud data from millimeter-wave radar and foggy images from electronic rearview mirror cameras; this ensures that the radar point cloud data and foggy images correspond to the same scene at the same time, providing a time consistency basis for subsequent fusion processing, and avoiding mismatch between depth priors and image information due to data asynchrony, which would affect the defogging effect.

[0032] S2. Process the point cloud data to generate global radar depth prior information aligned with the pixels of the foggy image. This transforms sparse radar point cloud data into global depth prior information aligned with the image pixels, providing accurate geometric constraints for transmittance estimation and solving the problem of inaccurate depth estimation caused by traditional methods relying on statistical priors or large amounts of training data.

[0033] S3. Based on global radar depth prior information, estimate global atmospheric light value; it uses radar depth prior to locate the far-end area of ​​the image, avoids interference from bright objects in the foreground, and improves the accuracy and robustness of atmospheric light value estimation. Atmospheric light value is a key parameter of the physical model for image dehazing, and its accuracy directly affects the restoration quality of fog-free images.

[0034] S4. Construct an optimization cost function that integrates global radar depth prior information, and solve the function through an optimization algorithm to obtain a globally optimized transmittance map. By constructing a cost function that integrates radar depth prior information, the dehazing problem is transformed into a convex optimization problem, which not only ensures data fidelity (consistency with the original image information) but also makes full use of the accuracy of radar depth prior information to obtain a globally optimized transmittance map, avoiding image discontinuity problems caused by local dehazing.

[0035] S5. Based on the globally optimized transmittance map and global atmospheric light value, a global fog-free image is recovered according to the atmospheric scattering model and displayed. It uses the optimized transmittance map and atmospheric light value to recover the fog-free image based on the atmospheric scattering model, and improves the image's visual effect through post-processing. The image is then output to the electronic rearview mirror display in real time, meeting the real-time requirements of in-vehicle scenarios and providing the driver with a clear field of vision.

[0036] According to some embodiments of this application, optionally, in step S1, the point cloud data includes the azimuth angle, pitch angle, distance, Doppler velocity and reflection intensity information of each point; synchronous acquisition is achieved through timestamp alignment to ensure that the point cloud data and the scene information corresponding to the foggy image are captured at the same time.

[0037] Specifically, the multi-dimensional information of radar point cloud data (azimuth, elevation, range, Doppler velocity, and reflection intensity) provides the foundational data for subsequent depth prior generation and confidence weight map construction. Azimuth and elevation are used to determine the position of a point in space, range information is the core of generating the depth map, Doppler velocity can help determine the target's motion state, and reflection intensity is used to evaluate the reliability of radar data, providing a basis for the generation of the confidence weight map. Timestamp alignment is a common and reliable method for achieving multi-sensor data synchronization. By adding timestamps to radar data and image data respectively, the set of data with the closest timestamps is selected as the synchronization data, ensuring that the radar data and image data correspond to the scene at the same moment. This avoids the mismatch between the depth prior and the target position in the image due to the difference in data acquisition time, ensuring the accuracy of fusion processing and thus improving the consistency of dehazing effects.

[0038] According to some embodiments of this application, optionally, generating global radar depth prior information in step S2 specifically includes: S21. Based on the pre-calibrated coordinate transformation relationship between the millimeter-wave radar and the camera, the radar point cloud data is transformed from the radar coordinate system to the image pixel coordinate system. This solves the problem of coordinate system differences caused by different installation positions of the radar and the camera, enabling the radar point cloud to be accurately projected onto the corresponding pixel position in the image, laying a spatial foundation for depth prior and pixel-level fusion of the image.

[0039] S22. Perform densification interpolation on the converted sparse depth point cloud to generate a dense radar depth map D_radar(x) with the same resolution as the foggy image. Millimeter-wave radar point clouds are naturally sparse, and direct use cannot cover all pixels of the image. A dense depth map is generated through an interpolation algorithm to ensure that each image pixel can obtain the corresponding depth information and achieve global depth prior coverage.

[0040] S23. Based on the atmospheric scattering model, the dense radar depth map D_radar(x) is converted into a global radar transmission prior map T_prior(x); based on the physical relationship between depth and transmittance in the atmospheric scattering model, the depth information is converted into a transmission prior, so that the radar depth prior can be directly applied to the parameter estimation of the image dehazing physical model, and the correlation between radar data and the dehazing model is established.

[0041] The generation of global radar depth prior information involves three stages: coordinate transformation, point cloud densification, and transmittance conversion. First, coordinate transformation maps the point cloud from the radar coordinate system to the image coordinate system, achieving spatial alignment between radar and image data. Then, interpolation algorithms transform the sparse point cloud into a dense depth map, meeting the depth requirements of each pixel in the image. Finally, based on the physical relationships of the atmospheric scattering model, the depth map is converted into a transmittance prior map, providing initial prior information for subsequent transmittance optimization.

[0042] According to some embodiments of this application, optionally, in step S21, the coordinate transformation relationship is obtained through joint calibration, specifically including: obtaining the rotation matrix R and translation vector T through the camera: radar hand-eye calibration method based on a specific calibration board, and obtaining the camera's intrinsic parameter matrix K and distortion coefficients through the Zhang Zhengyou calibration method.

[0043] Specifically, the radar hand-eye calibration method establishes the spatial geometric relationship between the radar and the camera through a specific calibration plate, and can accurately solve the rotation matrix R and translation vector T to describe the attitude and position offset of the radar coordinate system relative to the camera coordinate system; Zhang Zhengyou calibration method is a commonly used camera intrinsic parameter calibration method in industry. By taking chessboard images from different angles, it solves the camera's intrinsic parameter matrix K (including focal length, principal point coordinates, etc.) and distortion coefficients to correct the camera's optical distortion.

[0044] According to some embodiments of this application, optionally, the densification interpolation process in step S22 adopts an interpolation algorithm based on inverse distance weighting or a bilinear interpolation algorithm based on polar coordinate grid; during the interpolation process, a confidence weight map W(x) based on radar point reflection intensity and point cloud density is generated simultaneously.

[0045] Specifically, the interpolation algorithm based on inverse distance weighting (IDW) assigns weights according to the distance between the point to be interpolated and neighboring radar points. The closer the distance, the greater the weight. The weighted average is used to obtain the depth value of the point to be interpolated. This algorithm can ensure the dominant role of neighboring radar points in depth estimation and improve the local consistency of the depth map.

[0046] The bilinear interpolation algorithm based on polar coordinate grids: First, gridding and bilinear interpolation are performed on the original (range, azimuth) polar coordinate grid of the radar, and then projected onto the image coordinate system. The polar coordinate distribution characteristics of the radar data are utilized to improve the interpolation calculation efficiency.

[0047] Two interpolation algorithms with different characteristics are provided, which can be flexibly selected according to the computing power of the vehicle embedded platform and the defogging accuracy requirements. The IDW algorithm has higher accuracy, while the polar coordinate grid bilinear interpolation algorithm has higher efficiency, meeting the needs of different engineering application scenarios.

[0048] Confidence weight map generation is performed simultaneously: During depth interpolation, a reliability weight is assigned to each pixel based on the reflection intensity and point cloud density of the radar points, reflecting the credibility of the depth prior information corresponding to that pixel. This simultaneous generation of the confidence weight map provides a weighting basis for the radar prior term in the subsequent optimization cost function, enabling the fusion process to adapt to the reliability of the radar data. It strengthens the constraints of the depth prior in reliable radar data regions (high reflection intensity, dense point cloud) and weakens the constraints in unreliable regions, thus improving the robustness of the dehazing model.

[0049] According to some embodiments of this application, optionally, in step S23, the calculation formula for the global radar transmission prior map T_prior(x) is T_prior(x)=exp(-β*D_radar(x)), where β is the atmospheric scattering coefficient, which is determined by the algorithm developer based on experience or by debugging a specific dataset, and is used to characterize the scattering ability of the fog medium.

[0050] Specifically, according to the physical laws of the atmospheric scattering model, the transmittance t(x) decreases exponentially with the scene depth d(x), i.e., t(x) = exp(-β*d(x)), where β is the atmospheric scattering coefficient, characterizing the ability of fog, haze, and other media to scatter light. The larger the β value, the stronger the light scattering and the faster the transmittance decays. The β value in the formula is determined through experience or dataset tuning because the β value varies under different fog concentrations. Tuning allows for the selection of the optimal value that provides stable defogging effects under various fog conditions. The aim is to transform the dense radar depth map into a transmittance prior map, establishing a direct correlation between radar depth information and the physical model of image defogging. This allows the precise depth constraints provided by the radar to directly affect the transmittance estimation process, avoiding the subjectivity and inaccuracy of transmittance estimation in traditional methods, and providing a reliable initial prior for subsequent transmittance optimization.

[0051] According to some embodiments of this application, optionally, the estimation of the global atmospheric light value in step S3 specifically involves: S31. Based on the dense radar depth map D_radar(x), a predetermined region with the largest depth value in the image is selected. The predetermined region is the pixel region with the highest depth value, which is 0.1%-1%. It accurately identifies the far-end region of the image based on the radar depth map, avoiding the atmospheric light value estimation deviation caused by misjudgment of bright objects in the foreground (such as white vehicles and street lights) in traditional pure vision methods, and locks the candidate region of atmospheric light value from the perspective of physical distance.

[0052] S32. Extract the brightness values ​​of the corresponding pixels in the original foggy image I(x) of the predetermined region; provide raw data for atmospheric light value calculation.

[0053] S33. Use the maximum or average value of the brightness values ​​as the estimated value of the global atmospheric light value A. The maximum value can highlight the light intensity characteristics of the brightest area, while the average value is more stable and can be flexibly selected according to different fog conditions to ensure the accuracy and robustness of atmospheric light value estimation, thereby improving the restoration quality of fog-free images.

[0054] Specifically, atmospheric light value is the light intensity of the brightest area in a scene, typically located in the farthest region of the image (such as the sky or distant horizon). Utilizing radar depth priors, the region with the greatest depth in the image can be accurately located. This region is less affected by foreground objects, and its brightness value is closer to the true global atmospheric light value. By selecting a predetermined region with the greatest depth (0.1%-1% of pixels), the maximum or average brightness value of this region is extracted as the atmospheric light value, ensuring the reliability of the estimation results.

[0055] According to some embodiments of this application, optionally, in step S4, the optimized cost function is: ; Where I(x) is the foggy image, J(x) is the fog-free image to be recovered, t(x) is the transmittance to be solved, λ is the balance parameter, A is the global atmospheric light value, which is determined through experience or parameter tuning of a specific dataset and is used to control the importance weight of data fidelity terms and radar prior terms in the optimization process, and W(x) is the confidence weight map; the optimization algorithm is the gradient descent method, and the function is solved by the gradient descent method to obtain the refined transmittance map t(x).

[0056] Specifically, the cost function consists of two parts: a data fidelity term and a radar prior term. The data fidelity term [I(x)−J(x)t(x)−A(1−t(x))]² ensures that the solved transmittance map t(x) satisfies the atmospheric scattering model, maintaining data consistency between the recovered fog-free image J(x) and the original foggy image I(x). The radar prior term λW(x)[t(x)−T_prior(x)]² constrains the solved transmittance map t(x) to approximate the radar-provided transmittance prior map T_prior(x), leveraging the accuracy of the radar depth prior to improve the reliability of transmittance estimation. λ serves as a balancing parameter, controlling the importance of the two terms in the optimization process. When λ is large, the constraint of the radar prior term is stronger; when λ is small, the weight of the data fidelity term is higher. By constructing a cost function, radar depth priors are organically combined with image data fidelity, avoiding inaccurate transmittance estimation caused by relying solely on image information, while ensuring consistency between the dehazing results and the original image, thus balancing dehazing effect and image authenticity.

[0057] Gradient descent is a commonly used convex optimization algorithm that can efficiently find the minimum value of the cost function, obtaining a globally optimized transmittance map t(x). It also has moderate computational complexity, making it suitable for the real-time processing requirements of automotive embedded platforms. The optimization algorithm ensures that the cost function converges quickly to the optimal solution, resulting in a refined transmittance map that meets the latency requirements (<33ms) for real-time defogging of electronic rearview mirrors. Furthermore, the algorithm exhibits high stability and can reliably operate under various fog conditions.

[0058] According to some embodiments of this application, optionally, the process of generating the confidence weight map W(x) includes: W1. Calculate the initial reliability weight of each radar point based on the radar point cloud's reflection intensity (RCS) and distance information; assign initial weights to each radar point to establish the basis for reliability assessment of the radar data itself.

[0059] W2. Using the same interpolation algorithm as the depth map densification, the sparse weights are interpolated to generate a preliminary weight map with the same resolution as the image; the sparse radar point weights are extended to the entire image to ensure that each pixel has a corresponding confidence weight.

[0060] W3. Combining the local density distribution information of the radar point cloud, the preliminary weight map is corrected to obtain the final confidence weight map W(x). The weight is increased in the dense area of ​​the point cloud and decreased in the sparse area, so that the confidence weight map can more accurately reflect the local reliability of the radar data, provide accurate weight basis for optimizing the cost function, and improve the robustness of fusion dehazing.

[0061] Specifically, the confidence weight map W(x) is used to characterize the reliability of the radar depth prior at each pixel location. Its generation process fully utilizes the multi-dimensional information of the radar point cloud: the Reflection Cross Section (RCS) reflects the return strength of the radar signal; the higher the strength, the easier the target is to be detected by the radar, and the higher the data reliability; the distance information reflects the distance of the target; the radar detection accuracy varies at different distances and can be used to correct the weights; the local density of the point cloud reflects the richness of radar data in that area; the higher the density, the higher the data reliability. Through three steps—initial weight calculation, interpolation densification, and local density correction—a confidence weight map consistent with the image resolution is generated.

[0062] According to some embodiments of this application, optionally, in step S5, the obtained transmittance map t(x) and atmospheric light value A are substituted into the atmospheric scattering model to recover the fog-free image J(x), and after post-processing, it is output to the display screen in real time; the recovery formula of the fog-free image is: J(x) = (I(x) - A) / max(t(x), t0) + A; where t0 is a preset minimum value, with a value range of 0.01-0.1, used to avoid calculation abnormalities caused by the denominator t(x) being too small or 0; the post-processing includes color correction and contrast enhancement, and the delay of the real-time output is less than 33ms.

[0063] Specifically, the fog-free image restoration formula is derived based on the inverse operation of the atmospheric scattering model. By subtracting the atmospheric light value A from the foggy image I(x), dividing by the optimized transmittance map t(x) (taking the maximum value of t(x) and t0 to avoid an excessively small denominator), and then adding the atmospheric light value A, the fog-free image J(x) can be restored. t0 is a preset minimum value used to prevent the denominator from approaching 0 when t(x) is too small (e.g., in distant areas of dense fog), thus avoiding pixel overflow or distortion. Through accurate formula calculations, the effects of fog are removed from foggy images, restoring the true colors and details of the scene and providing drivers with a clear view.

[0064] Color correction in post-processing is used to correct color cast issues that may occur during defogging, while contrast enhancement is used to improve the visual clarity of fog-free images, optimize the visual effect of fog-free images, further enhance image recognizability, and make it easier for drivers to identify scene details.

[0065] The real-time output latency is less than 33ms because the electronic rearview mirror needs to reflect the vehicle's surrounding environment in real time. The 33ms latency corresponds to a refresh rate of about 30 frames per second, which can meet the driver's real-time requirements and avoid driving misjudgments caused by latency. Example

[0066] 1. Data synchronization acquisition (step S1): The electronic rearview mirror camera captures RGB foggy images (I) with a resolution of 1920×1080 and a frame rate of 30fps; the millimeter-wave radar captures scene point cloud data with a point cloud density of 100 points / frame, including azimuth, elevation, distance, Doppler velocity, and reflection intensity information for each point.

[0067] 2. Generation of global radar depth prior information (step S2): Step S21 (Coordinate Transformation): Using the camera calibration method based on a specific calibration board, obtain the rotation matrix R and translation vector T; using the Zhang Zhengyou calibration method, obtain the camera intrinsic parameter matrix K and distortion coefficients, and use these parameters to transform the radar point cloud data from the radar coordinate system to the image pixel coordinate system.

[0068] Step S22 (Point Cloud Denseening): The sparse point cloud after conversion is densed using an interpolation algorithm based on inverse distance weighting (IDW) to generate a dense radar depth map D_radar(x) with a resolution of 1920×1080; at the same time, a confidence weight map W(x) is generated simultaneously.

[0069] Step S23 (Transmittance Conversion): Set the atmospheric scattering coefficient β=1.0, and convert the dense radar depth map into a global radar transmission prior map T_prior(x) according to the formula T_prior(x)=exp(-1.0*D_radar(x)).

[0070] 3. Global atmospheric light value estimation (step S3): Step S31: Based on the dense radar depth map D_radar(x), select the pixel region with the top 0.5% depth value as the far-end region.

[0071] Step S32: Extract the brightness value of the corresponding pixel in the foggy image I(x) for the distant region.

[0072] Step S33: Use the maximum value among the brightness values ​​as the estimated value of the global atmospheric light value A.

[0073] 4. Optimization and solution of transmittance map (step S4): Constructing the optimization cost function , where λ=1.0.

[0074] The cost function was solved using the gradient descent method, with 50 iterations and a learning rate of 0.01, to obtain the globally optimized transmissivity map t(x).

[0075] 5. Haze-free image restoration and output (step S5): Set t0=0.05, and recover the haze-free image J(x) according to the formula J(x)=(I(x)-A) / max(t(x),0.05)+A.

[0076] The recovered fog-free image undergoes color correction and contrast enhancement post-processing, and the processed image is output to the electronic rearview mirror display in real time. The entire process has a delay of 28ms, which meets the real-time requirements.

[0077] It should be understood that the embodiments disclosed herein are not limited to the specific processing steps or materials disclosed herein, but should be extended to equivalent substitutions of such features as understood by those skilled in the art. It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0078] The term "embodiment" in this specification refers to a specific feature or characteristic described in connection with an embodiment that is included in at least one embodiment of the invention. Therefore, phrases or "embodiments" appearing in various places throughout the specification do not necessarily refer to the same embodiment.

[0079] Furthermore, the described features or characteristics can be incorporated into one or more embodiments in any other suitable manner. In the above description, specific details, such as thickness, quantity, etc., are provided to provide a comprehensive understanding of embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented without the aforementioned specific details or may be implemented using other methods, components, materials, etc.

Claims

1. A method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion, characterized in that, Includes the following steps: S1. Simultaneously acquire point cloud data from millimeter-wave radar and foggy images from electronic rearview mirror camera; S2. Process the point cloud data to generate global radar depth prior information aligned with the pixels of the foggy image; S3. Estimate global atmospheric light values ​​based on prior global radar depth information; S4. Construct an optimization cost function that integrates global radar depth prior information, and solve the function through an optimization algorithm to obtain a globally optimized transmittance map; S5. Based on the globally optimized transmittance map and global atmospheric light value, recover the global fog-free image according to the atmospheric scattering model and output it for display.

2. The method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 1, characterized in that, In step S1, the point cloud data includes the azimuth, pitch, distance, Doppler velocity, and reflection intensity information of each point.

3. The method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 1, characterized in that, The generation of global radar depth prior information in step S2 specifically includes: S21. Based on the pre-defined coordinate transformation relationship between the millimeter-wave radar and the camera, transform the radar point cloud data from the radar coordinate system to the image pixel coordinate system; S22. Perform densification interpolation on the converted sparse depth point cloud to generate a dense radar depth map D_radar(x) with the same resolution as the foggy image. S23. Based on the atmospheric scattering model, convert the dense radar depth map D_radar(x) into a global radar transmission prior map T_prior(x).

4. The method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 3, characterized in that, In step S21, the coordinate transformation relationship is obtained through joint calibration, specifically including: obtaining the rotation matrix R and translation vector T through the camera: radar hand-eye calibration method based on a specific calibration board, and obtaining the camera's intrinsic parameter matrix K and distortion coefficients through the Zhang Zhengyou calibration method.

5. The method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 3, characterized in that, The densification interpolation process in step S22 adopts an interpolation algorithm based on inverse distance weighting or a bilinear interpolation algorithm based on polar coordinate grids; during the interpolation process, a confidence weight map W(x) based on radar point reflection intensity and point cloud density is generated simultaneously.

6. The method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 3, characterized in that, In step S23, the formula for calculating the global radar transmission prior image T_prior(x) is T_prior(x)=exp(-β*D_radar(x)), where β is the atmospheric scattering coefficient, which is determined by the algorithm developers based on experience or by debugging on a specific dataset, and is used to characterize the scattering ability of the fog medium.

7. A method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 1 or 3, characterized in that, The estimation of global atmospheric light values ​​in step S3 is specifically as follows: S31. Based on the dense radar depth map D_radar(x), select the predetermined region with the largest depth value in the image. The predetermined region is the pixel region with the highest depth value, which is the top 0.1%-1% of the pixel regions. S32. Extract the brightness value of the corresponding pixel in the original foggy image I(x) of the predetermined region; S33. Use the maximum or average value of the brightness values ​​as an estimate of the global atmospheric light value A.

8. The method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 1, characterized in that, In step S4, the optimization cost function is: ； Where I(x) is the foggy image, J(x) is the fog-free image to be recovered, t(x) is the transmittance to be solved, λ is the balance parameter, A is the global atmospheric light value, which is determined through experience or parameter tuning of a specific dataset and is used to control the importance weight of data fidelity terms and radar prior terms in the optimization process, and W(x) is the confidence weight map; the optimization algorithm is the gradient descent method, and the function is solved by the gradient descent method to obtain the refined transmittance map t(x).

9. A method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 5, characterized in that, The process of generating the confidence weight map W(x) includes: W1. Calculate the initial reliability weight for each radar point based on the radar point cloud's reflection intensity (RCS) and distance information. W2. Using the same interpolation algorithm as the depth map densification, the sparse weights are interpolated to generate a preliminary weight map with the same resolution as the image. W3. Combining the local density distribution information of the radar point cloud, the preliminary weight map is corrected to finally obtain the confidence weight map W(x).

10. A method for real-time image dehazing of electronic rearview mirrors based on millimeter-wave radar depth prior fusion according to claim 8, characterized in that, In step S5, the formula for restoring the fog-free image is: J(x) = (I(x) - A) / max(t(x), t0) + A; where t0 is a preset minimum value, ranging from 0.01 to 0.1, used to avoid calculation abnormalities caused by the denominator t(x) being too small or 0.

Images