Night high-quality imaging method and system for electronic rearview mirror

Abstract

The invention belongs to the technical field of computer vision and image processing, discloses a nighttime high-quality imaging method for an electronic rearview mirror, and aims to solve the problem that daytime and nighttime imaging needs to be processed at the same time and respectively construct learning samples and solve light source masks for daytime and nighttime images according to a brightness threshold. And a two-dimensional U-net structure is used as a model basis, and attention and a multi-scale mechanism are added to construct a learning model, so that better CMS image high-quality imaging is realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision and image processing, and particularly relates to a high-quality night imaging method and system for electronic rearview mirror. BACKGROUND

[0002] Electronic rearview mirror (Camera Monitor System, CMS) composed of camera and monitor is an important alternative direction of traditional glass rearview mirror at present due to its advantages in terms of resistance reduction and field of view expansion. In addition, CMS technology upgrades the outside rearview mirror from a simple mechanical part to an Over-the-Air (OTA) visual electronic control unit (ECU), so that the rearview mirror providing the driver with the field of view around the vehicle becomes a growing and expandable intelligent sensing node, which can be directly connected to the Advanced Driver Assistance Systems (ADAS) and become the entrance of domain fusion, and can also become the source of road environment big data to realize the extension of vehicle data service. That is, the electronic rearview mirror is not only a "double necessity" of safety and energy efficiency, but also a key cut of the evolution of vehicles from mechanical terminals to intelligent sensing terminals in the era of "software-defined vehicles".

[0003] However, the problem of all-weather clear imaging of CMS has always been an important content hindering the further realization of industrial substitution of this technology, among which the low illumination at night driving and the glare problem caused by strong light and backlight are the most prominent. However, existing researches are mostly separate processing of dark image enhancement and glare elimination, specifically: existing glare elimination methods are mostly for daytime imaging and lack of discussion on low illumination conditions, while the research on dark image enhancement rarely considers the image glare problem. In actual scenes, glare and low illumination often coexist, and there is a contradiction between dark image enhancement and glare elimination, mainly manifested as that dark image enhancement will exacerbate the image glare problem, therefore, it is of great significance to develop a high-quality imaging method that can simultaneously address low illumination and glare problems for the further development of CMS technology.

[0004] Through the above analysis, the problems and defects of the prior art are: Existing researches are mostly separate processing of dark image enhancement and glare elimination, specifically: existing glare elimination methods are mostly for daytime imaging and lack of discussion on low illumination conditions, while the research on dark image enhancement rarely considers the image glare problem. In actual scenes, glare and low illumination often coexist, and there is a contradiction between dark image enhancement and glare elimination, mainly manifested as that dark image enhancement will exacerbate the image glare problem. SUMMARY

[0005] In view of the problems in the prior art, the present application provides a high-quality imaging method for electronic rearview mirror at night.

[0006] The present application is implemented in a high-quality imaging method for electronic rearview mirror at night, comprising: S1: establishment of learning samples for high-quality imaging of CMS images.

[0007] S2: building and training of an intelligent model for high-quality imaging of CMS images.

[0008] S3: high-quality imaging of unknown CMS images.

[0009] Further, the establishment of learning samples for high-quality imaging of CMS images in S1 specifically comprises the following steps: S11: collection of imaging images containing daytime and nighttime and glare images mainly containing reflected glare and scattered glare in the driving process.

[0010] S12: pre-processing of the images of the driving scene, such as deleting images with low resolution or insufficient imaging information, and then selecting non-glare images therefrom for subsequent sample making.

[0011] S13: random cropping of the non-glare daytime driving scene images with a size of (W, H), and downsampling of the glare images with a size of (W, H); then, random fusion of the cropped non-glare driving scene images and the downsampled glare images to obtain a synthesized daytime image dyed with glare, which together with the corresponding cropped non-glare daytime image constitutes a daytime image sample pair.

[0012] S14: cropping of the low-glare nighttime images as described in S13, and then dark image enhancement processing of the cropped nighttime images, followed by random fusion with the downsampled glare images to obtain a nighttime image dyed with glare, which together with the corresponding cropped non-glare nighttime image constitutes a nighttime image sample pair.

[0013] S15: merging of the daytime image samples and the nighttime image samples, and normalization of the samples.

[0014] It should be noted that during the random cropping of the non-glare driving scene images, images with insufficient imaging information may be obtained, so manual deletion is required before the synthesis of the glare images; in addition, since there is a light source in the nighttime image, it is difficult to completely avoid glare, so when selecting nighttime images to construct samples, the definition of glare is relaxed compared to daytime images, especially when the lighting is insufficient.

[0015] Further, the building and training of the intelligent model for high-quality imaging of CMS images in S2 specifically comprises the following steps: S21, using an encoder-decoder structure to construct a U-net, the size of the input of which is (W, H, 3), and determining the activation functions of each convolutional layer and the output layer.

[0016] S22, adding a channel attention module after the last pooling layer of the encoder to capture the inter-channel dependence, thereby realizing the difference weighting of the original feature map, and adding a multi-scale module after the attention module to realize multi-scale feature extraction, and finally fusing the obtained different scale features and passing them to the decoder.

[0017] S23, positioning the light source position in the image based on the input of the network to form a mask to modify the output of the model; due to the differences in brightness, contrast and the like between daytime and nighttime images, the original mask is subjected to differential filtering and morphological operation and the like to obtain a mask with good connectivity and in line with the characteristics of the light source.

[0018] S24, replacing the pixels of the light source position in the model output with the pixels of the light source position in the model input to form a modified model output.

[0019] S25, loading a pre-trained VGG16 model, selecting a feature extraction layer as the basis for calculating the perceptual loss, and then calculating the absolute error between the modified model output and the target image in the VGG feature space as the optimization direction of the network.

[0020] Further, the high-quality imaging of the unknown CMS image in S3 specifically comprises the following steps: S31, cropping the unknown driving scene image to a size of (W, H) and performing consistent normalization processing as the training sample.

[0021] S32, inputting the above image into the trained network to obtain the high-quality imaging result of the image.

[0022] Another object of the present application is to provide an electronic rearview mirror nighttime high-quality imaging system comprising: A sample establishment module for establishing learning samples for high-quality imaging of CMS images.

[0023] A training module for building and training an intelligent model for high-quality imaging of CMS images.

[0024] An imaging module for high-quality imaging of unknown CMS images.

[0025] Another object of the present application is to provide a computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to make the processor execute the steps of the electronic rearview mirror nighttime high-quality imaging method.

[0026] Another object of the present application is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the electronic rearview mirror night high-quality imaging method.

[0027] Another object of the present application is to provide an information data processing terminal for implementing the electronic rearview mirror night high-quality imaging system.

[0028] The technical scheme of the present application has the following advantages and positive effects: The present application proposes an electronic rearview mirror night high-quality imaging method. Considering the need to handle daytime and nighttime imaging simultaneously, learning samples and light source masks are constructed for daytime and nighttime images based on brightness thresholds, and a learning model is constructed based on a two-dimensional U-net structure as a model basis, while adding attention and multi-scale mechanisms to achieve better CMS image high-quality imaging.

[0029] (1) The expected revenue and commercial value of the technical scheme of the present application after transformation are: The technology proposed by the present application is of great significance to CMS (camera monitoring system). It not only significantly improves the imaging quality of CMS under complex lighting conditions, such as making night monitoring clearer, not overexposed or losing details under strong light, but also greatly broadens the application scenarios of CMS, including but not limited to intelligent driving assistance systems, security monitoring, medical imaging, etc., greatly enhancing user experience and system reliability. The commercial value of this technology lies in its ability to effectively improve the competitiveness of CMS equipment in the market, attract more high-end users and enterprise customers, and thus enhance brand value, bringing considerable economic benefits to manufacturers. At the same time, it promotes the entire CMS industry to develop towards higher quality and more extensive applications, with far-reaching technological progress and market expansion potential.

[0030] (2) The technical scheme of the present application fills the gap in the industry: The technology fills the gap in the field of low-illumination and glare processing by innovative algorithm design or deep learning models. It solves the limitations of existing technologies that can only handle low-illumination or glare problems separately, providing a comprehensive solution that can significantly improve imaging quality under complex lighting conditions such as night-time strong light exposure or reflection. The technology has achieved breakthroughs in dynamic scene adaptation, multi-modal fusion, lightweight and efficient computing, etc., not only improving the performance of CMS (camera monitoring system) in high-end application scenarios such as security monitoring, intelligent driving, medical imaging, but also expanding its market potential. It solves the key pain points in the industry by reducing image overexposure and detail loss, and promotes the development of related technologies towards higher quality and more extensive applications. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the flow chart of the electronic rearview mirror night high-quality imaging method provided by the embodiment of the present application.

[0032] Figure 2 is the feature analysis diagram of daytime and night-time images provided by the embodiment of the present application. In the processing of the light source mask, the threshold of brightness is set to 0.2. When the image brightness value is greater than 0.2, the image is identified as a daytime image, and the mask processing for daytime images is performed. Otherwise, the mask processing for night-time images is performed.

[0033] Figure 3 is the schematic diagram of the learning sample provided by the embodiment of the present application, wherein (a), (b), (c) and (d) are daytime scattered glare image, daytime reflected glare image, night-time scattered glare image and night-time reflected glare image, respectively.

[0034] Figure 4 is the process diagram of daytime image light source mask acquisition provided by the embodiment of the present application.

[0035] Figure 5 is the process diagram of night-time image light source mask acquisition provided by the embodiment of the present application.

[0036] Figure 6 is the comparison diagram of daytime image imaging effect provided by the embodiment of the present application.

[0037] Figure 7 is the comparison diagram of night-time image imaging effect provided by the embodiment of the present application.

[0038] Figure 8 is the structure block diagram of the electronic rearview mirror night high-quality imaging system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0040] As shown in Figure 1 The electronic rearview mirror night high-quality imaging method provided by the embodiments of the present application comprises the following steps: S1: Establishing learning samples for CMS image high-quality imaging.

[0041] S2: Building and training an intelligent model for CMS image high-quality imaging.

[0042] S3: High-quality imaging of unknown CMS images.

[0043] In specific implementation, the establishment of learning samples for CMS image high-quality imaging specifically comprises the following steps: S11: Collecting imaging images containing daytime and nighttime and glare images mainly containing reflected glare and scattered glare in driving process.

[0044] S12: Preprocessing the images of driving scenes, such as deleting images with low resolution or insufficient imaging information, and then selecting non-glare images from them for subsequent sample making.

[0045] S13: Randomly cropping the non-glare daytime driving scene images to a size of (512, 512), and down-sampling the glare images to a size of (512, 512). Then, randomly fusing the cropped non-glare daytime driving scene images and the down-sampled glare images to obtain synthesized daytime images invaded by glare, which together with the corresponding cropped non-glare daytime images constitute daytime image sample pairs.

[0046] S14: Cropping the low-glare nighttime images as described in S13, then gamma enhancing the cropped nighttime images to improve their brightness, and then randomly fusing them with the down-sampled glare images to obtain nighttime images invaded by glare, which together with the corresponding cropped non-glare nighttime images constitute nighttime image sample pairs.

[0047] It should be noted that the identification of daytime glare images and nighttime glare images by the subsequent model is based on the brightness value of the images, and in the present application, the brightness threshold is set to 0.2. As shown in Figure 2 is a feature comparison chart of daytime images and nighttime images.

[0048] S15: Merging daytime image samples and nighttime image samples, and normalizing the samples to [-1, 1].

[0049] AsFigure 3 The process of establishing daytime samples and nighttime samples of one embodiment of the application is shown, and the glare types include reflection glare and scattering glare.

[0050] In a specific implementation, the construction and training of the intelligent model for high-quality imaging of the CMS image specifically include the following steps: S21, a U-net is constructed using an encoder-decoder structure, the size of the input of the U-net is (512, 512, 3), the activation function of each convolutional layer is relu, and the activation function of the output layer is tanh.

[0051] S22, a channel attention module is added after the last pooling layer of the encoder, which captures the inter-channel dependency through global average pooling and global maximum pooling, and then generates attention weights through a fully connected layer, so as to realize the differential weighting of the original feature map, and then a multi-scale module is further added after the attention module, which includes four different scale feature extraction methods of original size, 3x3 convolution, two 3x3 convolutions (equivalent to 5x5 convolution) and average pooling combined with 1x1 convolution, to realize multi-scale feature extraction, and finally the obtained different scale features are fused and transmitted to the decoder.

[0052] S23, the light source position in the image is located based on the input of the network to form a mask to modify the output of the model. Among them, with the brightness threshold 0.2 as the boundary, the differential filtering and morphological operation of the original mask of the daytime and nighttime images are processed to obtain a mask with good connectivity and in line with the characteristics of the light source.

[0053] As shown in Figure 4 and Figure 5 The schematic diagram of the image mask in each processing stage is shown, from which it can be seen that as the processing goes deeper, the connectivity of the light source mask becomes better, the burr is reduced, and the smoothness of the mask boundary is also more in line with the characteristics of the light source.

[0054] S24, the pixel of the light source position in is replaced with the pixel of the light source position in the model output to form the modified model output, that is:

[0055] S25, the pre-trained VGG16 model is loaded, the features of the 'block1_conv2', 'block2_conv2', 'block3_conv4' and 'block4_conv4' layers are used as the calculation basis of the perceptual loss, and the weights are all 1, then the absolute error of the modified model output and the target image in the VGG feature space is calculated, which is used as the optimization direction of the network, and the formula is as follows:

[0056] In specific implementation, glare suppression processing for unknown CMS images includes the following steps: S31, crop the unknown driving scene image to a size of (512, 512) to match the size of the training sample, and then perform normalization processing consistent with the training sample.

[0057] S32, input the above image into the trained network to obtain a high-quality imaging result of the image.

[0058] like Figure 6 and Figure 7 The diagram shown illustrates the imaging effect. It can be seen that the CMS imaging method proposed in this invention has a better deglare removal effect on daytime images than U-net, and also performs better in image enhancement and glare elimination on nighttime images.

[0059] like Figure 6 and Figure 7 As shown, the electronic rearview mirror nighttime high-quality imaging method of the present invention exhibits significant advantages in removing strong daytime reflection glare and low-light noise at night. Traditional U-Net networks often exhibit over-smoothing and localized exposure residue under strong daytime light conditions, resulting in blurred edges and loss of texture details in reflected light areas. The present invention introduces a channel attention mechanism and multi-scale convolutional feature fusion at the encoding end. By assigning differentiated weights to high-energy regions at the feature level, it achieves precise separation of reflective and diffuse scattering components, thereby effectively eliminating glare trailing in bright areas.

[0060] Furthermore, in nighttime scenes, this method utilizes a dual mechanism based on dark light enhancement and light source mask correction to enhance the overall brightness of the image while suppressing local oversaturation caused by strong light sources such as streetlights and vehicle lights. The light source mask is spatially optimized through morphological connectivity, enabling dynamic suppression of halo regions and ensuring clear rendering of distant illumination, ground reflections, and target contours in nighttime scenes. Compared to U-Net results, the image output by the model of this invention is closer to the target image in terms of detail level, contrast, and structural fidelity, reflecting higher perceptual quality and imaging stability.

[0061] This method introduces physical constraints and a perception layer optimization mechanism into the network structure, enabling the CMS system to possess adaptive dynamic illumination compensation and glare removal capabilities during day-night imaging transitions. It maintains clear boundaries of reflective areas under strong daylight and accurately recovers dark textures under low-light conditions at night, achieving high dynamic range and realistic visual reproduction under all-weather conditions. This imaging strategy not only significantly improves the safety and reliability of vehicle electronic rearview mirrors in complex lighting environments but also provides an industrially deployable, highly robust image enhancement solution for intelligent driving visual perception systems.

[0062] like Figure 8 As shown, an embodiment of the present invention provides a high-quality nighttime imaging system for an electronic rearview mirror, comprising: The sample creation module is used to create learning samples for high-quality imaging of CMS images.

[0063] The training module is used to build and train intelligent models for high-quality imaging of CMS images.

[0064] Imaging module for high-quality imaging of unknown CMS images.

[0065] The main technical obstacles that electronic rearview mirrors have long faced in nighttime environments are uneven image brightness distribution under extreme lighting conditions, local overexposure caused by reflected and scattered glare, and loss of detail due to low-light noise. Although traditional CMOS image sensors have certain dynamic range extension methods, they still struggle to recover the true scene details under the interference of reflections from high-brightness areas such as vehicle headlights, taillights, and streetlights at night. The high-quality nighttime imaging method for electronic rearview mirrors proposed in this invention was developed to address the problem of vehicle CMS systems' inability to accurately reproduce details under complex nighttime lighting conditions, against this technical background.

[0066] In constructing the methodology, the researchers first started from the data level, simulating various lighting and glare distribution patterns through a systematic sample construction strategy. By cross-processing daytime and nighttime images, local perturbations in bright areas and detailed signals in low-brightness areas were incorporated into the model's learning space. This sample strategy overcomes the limitations of simple brightness adjustment in traditional data augmentation, enabling the model to learn the dynamic lighting distribution patterns in real-world scenes. In particular, by fusing glare-free samples with downsampled glare images, typical scattered light spot features on reflective surfaces and slippery roads were generated, thus providing a more physically consistent input distribution for model training.

[0067] In terms of algorithm structure, the core network adopts an improved U-Net architecture, embedding channel attention and multi-scale feature extraction mechanisms. The channel attention module strengthens high-energy channels through feature response weighting, enabling the system to adaptively focus on regions with significant reflective features in the image; the multi-scale convolutional units extract spatial frequency difference information layer by layer, achieving simultaneous representation of low-brightness backgrounds and high-brightness reflections. This layered fusion structure is physically equivalent to multi-layer deconstructing the reflected light path, separating and reconstructing the brightness and structural components of the original signal in the feature space.

[0068] The introduction of a light source mask is a significant engineering innovation in this method. During forward inference, the model generates a mask based on the spatial distribution of the light source to suppress erroneous recovery of oversaturated areas. The mask generation is not based on a simple threshold but utilizes differential filtering and morphological connectivity analysis to ensure spatial continuity of the light source region and natural transitions in edge features. The output after mask correction significantly reduces halo diffusion and reflection trailing, making the transition between bright and dark areas in nighttime vehicle rearview images more consistent with human visual perception.

[0069] During model optimization, perceptual loss is introduced as a constraint, calculating the feature differences between the output and target images based on the pre-trained VGG16 feature extraction layer. This is equivalent to measuring the structural similarity of images in the network's high-dimensional feature space, rather than relying solely on pixel-level errors. Through this mechanism, the model can gradually learn global brightness balance and detail preservation strategies for nighttime images during the training phase, thereby obtaining a robust model with stable imaging performance under different lighting conditions.

[0070] Ultimately, the system is implemented through a modular architecture in practical applications: the front-end image acquisition unit captures nighttime driving scenes in real time, the processing module performs the aforementioned multi-stage feature fusion and mask correction operations, and outputs enhanced high dynamic range (HMR) CMS images. Its imaging results maintain clear details and balanced brightness even in highly reflective road surfaces, slippery environments, and backlit scenes, providing greater visual safety redundancy for nighttime driving and possessing broad industrial application value in intelligent driving assistance systems, autonomous driving perception systems, and high dynamic range monitoring terminals.

[0071] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0072] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for high-quality nighttime imaging with an electronic rearview mirror, characterized in that, Includes the following steps: S1, Establish learning samples for high-quality imaging of CMS images; S2, build and train an intelligent model for high-quality imaging of CMS images; S3 utilizes a trained intelligent model to perform high-quality imaging processing on unknown CMS images.

2. The electronic rearview mirror nighttime high-quality imaging method as described in claim 1, characterized in that, Step S1 includes: S11 collects raw data including daytime, nighttime, and glare images during driving; S12 performs resolution and sharpness filtering on the original image to remove low-quality images; S13, randomly fuse the glare-free image with the downsampled glare image to form an image contaminated by glare, and form a sample pair with the corresponding glare-free image; S14, perform dark image enhancement on the low-brightness nighttime image and then fuse it with the glare image to generate a nighttime sample pair; S15, normalize the daytime and nighttime samples to form a learning sample set.

3. The electronic rearview mirror nighttime high-quality imaging method as described in claim 1, characterized in that, In step S2, the intelligent model adopts a U-Net network with an encoder-decoder structure, and sets a channel attention module and a multi-scale feature module at the end of the encoder to extract multi-level features.

4. The electronic rearview mirror nighttime high-quality imaging method as described in claim 3, characterized in that, The channel attention module is used to capture the dependencies between features of different channels and assign differentiated weights. The multi-scale feature module includes multiple convolutional layers with different kernel sizes to achieve multi-scale feature fusion.

5. The electronic rearview mirror nighttime high-quality imaging method as described in claim 1, characterized in that, Step S2 further includes: A light source mask is generated based on the light source region in the localization image input by the model. After differential filtering and morphological processing on images with different brightness, a light source mask with good connectivity is obtained. The pixels at the light source position in the model output are replaced with the corresponding pixels in the model input to obtain the corrected output result.

6. The electronic rearview mirror nighttime high-quality imaging method as described in claim 1, characterized in that, Step S2 further includes: A pre-trained VGG16 feature extraction network is introduced, and the absolute error between the model output image and the target image in the feature space is calculated. This error is used as the perceptual loss to optimize the network parameters.

7. The electronic rearview mirror nighttime high-quality imaging method as described in claim 1, characterized in that, Step S3 includes: The unknown driving scene image is cropped and normalized, then input into the trained model, and the corresponding high-quality imaging result is output.

8. A high-quality nighttime imaging system for an electronic rearview mirror, characterized in that, include: The sample creation module is used to create high-quality learning samples for CMS image imaging. The training module is used to build and train intelligent models for high-quality imaging of CMS images; The imaging module is used for high-quality imaging processing of unknown CMS images.

9. The electronic rearview mirror night-time high-quality imaging system as described in claim 8, characterized in that, The training module includes: The U-Net network with encoder-decoder structure, channel attention module, multi-scale feature module, and perceptual loss calculation unit are used to implement model training and optimization.

10. A high-quality nighttime imaging device for an electronic rearview mirror, characterized in that, include: The image acquisition unit is used to acquire images of nighttime driving scenes; An image processing unit is configured to perform the nighttime high-quality imaging method according to any one of claims 1 to 7; The display unit is used to output enhanced high-quality CMS images in real time for the driver to observe.

Images