Streaming media rearview mirror night vision enhancement method based on deep learning image processing

Abstract

The invention relates to a streaming media rearview mirror night vision enhancement method based on deep learning image processing, and belongs to the field of vehicle-mounted image processing.The method comprises the steps that after a low-light original image is obtained, multi-scale features are extracted through an improved residual network, and the multi-scale features comprise a self-adaptive convolution kernel adjustment layer and a channel attention module to optimize feature extraction; the features of different scales are processed in parallel through multi-branch cavity convolution of a feature enhancement module, and enhancement features are generated by combining pixel-by-pixel weighted fusion; gradually amplifying the feature map through an up-sampling network and fusing shallow details to recover the resolution; and after the output layer generates an enhanced image, the enhanced image is transmitted to the rearview mirror in real time by the vehicle-mounted system, and a cache preloading mechanism is started when the delay exceeds a preset value. By optimizing a network structure and a transmission mechanism, the night vision enhancement effect and the real-time performance are considered, and the picture definition and the dynamic response speed in a low-light scene are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicle-mounted image processing, in particular to a streaming media rearview mirror night vision enhancement method based on deep learning image processing. BACKGROUND

[0002] The streaming media rearview mirror collects the images of the surrounding environment of the vehicle through the vehicle-mounted camera and displays them in real time, solving the problem of the visual blind area of the traditional optical rearview mirror. However, in the night or low-light environment, the original images collected by the camera have obvious noise, loss of details, and low contrast, which leads to a blurred display picture and affects the driving safety.

[0003] In the existing night vision enhancement technology, the traditional image processing method improves the brightness through histogram equalization and gamma correction, but it is easy to cause local overexposure or noise amplification. Although the method based on deep learning can learn the feature mapping relationship through the network model, it generally has the problems of insufficient feature extraction and insufficient multi-scale information fusion, and the complex network structure leads to an increase in processing delay, which is difficult to meet the real-time requirements of the vehicle-mounted system. SUMMARY

[0004] The main purpose of the present application is to provide a streaming media rearview mirror night vision enhancement method based on deep learning image processing to solve the problems of large image noise, blurred details, and high processing delay of the streaming media rearview mirror in low-light environments.

[0005] To achieve the above purpose, the streaming media rearview mirror night vision enhancement method based on deep learning image processing provided by the present application comprises the following steps: Obtaining the original image of the vehicle-mounted camera in a low-light environment; Performing multi-scale feature extraction through an improved residual network, wherein the residual network comprises at least two cascaded feature extraction units, each feature extraction unit is composed of multiple convolution layers, and the residual structure is formed through jump connection between adjacent convolution layers; Inputting the features extracted by the multi-scale feature extraction into a feature enhancement module, wherein the feature enhancement module performs parallel processing on features of different scales through a multi-branch hollow convolution structure, and generates enhanced features through pixel-by-pixel fusion; Inputting the enhanced features into an up-sampling network for resolution recovery, wherein the up-sampling network comprises at least three deconvolution layers, gradually enlarges the size of the feature map and fuses the shallow detail information to obtain the recovered image; Inputting the recovered image into an output layer to generate an enhanced image, and outputting the enhanced image to the streaming media rearview mirror in real time through the vehicle-mounted display system.

[0006] Further, the improvement method of the residual network comprises: An adaptive convolution kernel adjustment layer is added before the first convolution layer of each feature extraction unit, which dynamically adjusts the size and step of the convolution kernel according to the brightness mean value of the input image, and increases the size of the convolution kernel and reduces the step when the brightness mean value is lower than the preset threshold; A channel attention module is embedded in the skip connection of the residual structure, which obtains the weight coefficient of the feature channel through global average pooling, and performs weighted screening on the input features in the channel dimension; The output features of the last convolution layer in each feature extraction unit are spliced with the input features of the unit in the channel dimension, and then the channel number is compressed through a 1x1 convolution layer, forming an improved residual structure.

[0007] Further, the step of parallel processing of features of different scales by the feature enhancement module through a multi-branch hollow convolution structure comprises: Divide the extracted features into at least three feature subgraphs in the channel dimension, and input them into three parallel hollow convolution branches respectively; The first branch uses a 3x3 convolution layer with a hollow rate of 2 to process the feature subgraph, the second branch uses a 3x3 convolution layer with a hollow rate of 4, and the third branch uses a 3x3 convolution layer with a hollow rate of 6; Each branch is sequentially subjected to batch normalization processing and ReLU activation function after convolution operation; The output features of the three branches are spliced in the channel dimension to form an intermediate feature map.

[0008] Further, the step of generating enhanced features by pixel-by-pixel fusion comprises: Local contrast calculation is performed on the intermediate feature map to obtain the local contrast value of each pixel point; The local contrast value is normalized by a Sigmoid function to obtain the weight coefficient corresponding to each pixel point; The feature values of the corresponding pixel points in the intermediate feature map are weighted and summed according to the weight coefficient to generate preliminary fusion features; A 3x3 convolution operation is performed on the preliminary fusion features to eliminate the splicing traces, and the enhanced features are obtained.

[0009] Further, the step of inputting the enhanced features into the up-sampling network for resolution recovery comprises: The enhanced features are input into the first deconvolution layer of the up-sampling network to generate a first scale feature map by 2 times magnification, and are spliced with the shallow layer features output by the first feature extraction unit in the residual network in the channel dimension; The spliced features are input into the second deconvolution layer to generate a second scale feature map by 2 times magnification, and are spliced with the original detail features output by the initial convolution layer in the residual network in the channel dimension; The stitched features are input into the third deconvolution layer, and a third-scale feature map is generated by magnification by 2 times. The resolution of the third-scale feature map is the same as that of the original image. An adjustment layer with a kernel size of 1×1 is set after each deconvolution layer to unify the number of channels in the spliced ​​features.

[0010] Furthermore, the steps to obtain the restored image by progressively enlarging the feature map size and fusing shallow detail information include: When the first deconvolutional layer amplifies the enhanced features by 2 times, the shallow features output by the first feature extraction unit of the residual network are weighted point by point according to the pixel position and the amplified feature map. The weights are dynamically allocated according to the gradient value of the shallow features, and the higher the gradient value, the greater the weight. When the second deconvolutional layer magnifies the weighted first-scale feature map by 2 times, the original detailed features output by the initial convolutional layer of the residual network are used to extract high-frequency components through the edge detection operator and then superimposed on the magnified feature map by the channel dimension. When the third deconvolution layer magnifies the stacked second-scale feature map by 2 times, it uses a combination of bilinear interpolation and deconvolution to maintain edge continuity. After magnification, the feature map is optimized as a whole through a 3×3 convolution layer to generate a restored image with the same resolution as the original image.

[0011] Furthermore, the step of generating an enhanced image from the restored image input / output layer and outputting it in real time to the streaming rearview mirror via the in-vehicle display system includes: The restored image is input into the output layer. The output layer integrates the image features through a 3×3 convolutional layer, and then adjusts the number of channels to 3 through a 2×2 convolutional layer to obtain an initial enhanced image with the same color space as the original image. The initial enhanced image is locally brightened by calculating the average brightness of each 8×8 pixel block in the image. The brightness values ​​of pixels below the preset brightness threshold are increased to the threshold level, while those above the threshold remain unchanged. The corrected and enhanced image is transmitted to the vehicle display system, which performs format conversion on the image data to match the display parameters of the streaming rearview mirror and monitors the transmission delay in real time. When the delay exceeds a preset value, the cache preloading process is initiated. The processed enhanced image is output to the streaming rearview mirror via the in-vehicle display system at a rate of no less than 30 frames per second, and the night vision enhanced image is displayed in real time.

[0012] Furthermore, the steps for initiating the cache preloading process when the delay exceeds a preset value include: When a transmission delay exceeding a preset value is detected, a buffer area is defined, the capacity of which is at least the amount of image data required for the streaming media rearview mirror to display for 5 seconds. The image processing module obtains the enhanced image data to be displayed, and loads the image data of the first 3 seconds into the cache area in the order of generation time. After the buffer area is full of 3 seconds of data, the image data in the buffer is output to the streaming rearview mirror in sequence, while the remaining 2 seconds of image data are loaded. The system monitors the current transmission latency in real time. When the latency drops below a preset value, it gradually reduces the amount of cached data loaded until it returns to direct transmission mode. Unused image data in the cache area is automatically cleared.

[0013] This invention proposes a streaming media rearview mirror night vision enhancement device based on deep learning image processing, comprising: The acquisition unit is used to acquire raw images from the vehicle-mounted camera in low-light environments. An extraction unit is used for multi-scale feature extraction through an improved residual network, wherein the residual network contains at least two cascaded feature extraction units, each feature extraction unit is composed of multiple convolutional layers, and adjacent convolutional layers are connected by skip connections to form a residual structure. An enhancement unit is used to input the features extracted from the multi-scale features into a feature enhancement module. The feature enhancement module processes features of different scales in parallel through a multi-branch dilated convolutional structure and generates enhanced features by pixel-wise fusion. The fusion unit is used to input the enhanced features into the upsampling network for resolution restoration. The upsampling network contains at least three deconvolutional layers, which progressively enlarge the feature map size and fuse shallow detail information to obtain the restored image. The display unit is used to generate an enhanced image from the restored image input / output layer and output it to the streaming media rearview mirror in real time through the vehicle display system.

[0014] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described streaming media rearview mirror night vision enhancement method based on deep learning image processing.

[0015] The night vision enhancement method for streaming rearview mirrors based on deep learning image processing provided by this invention has the following beneficial effects: it improves the feature extraction capability in low light environment through improved residual network, realizes multi-scale feature parallel enhancement through multi-branch dilated convolution, restores image resolution by progressive upsampling combined with shallow details, and ensures real-time display through cache preloading mechanism. It solves the problem of difficulty in balancing enhancement effect and real-time performance in the prior art, and significantly improves the image clarity and dynamic response speed of streaming rearview mirrors in night vision scenarios. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for enhancing night vision of a streaming media rearview mirror based on deep learning image processing, according to an embodiment of the present invention. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] Reference Figure 1 This is a flowchart illustrating the night vision enhancement method for streaming media rearview mirrors based on deep learning image processing proposed in this invention. The method includes: S1, acquire the raw image of the vehicle camera in low-light environment; S2, Multi-scale feature extraction is performed through an improved residual network, wherein the residual network contains at least two cascaded feature extraction units, each feature extraction unit is composed of multiple convolutional layers, and adjacent convolutional layers are connected by skip connections to form a residual structure; S3, the features extracted from the multi-scale features are input into the feature enhancement module. The feature enhancement module processes features of different scales in parallel through a multi-branch dilated convolutional structure and generates enhanced features by pixel-by-pixel fusion. S4, the enhanced features are input into an upsampling network for resolution restoration. The upsampling network contains at least three deconvolutional layers. The restored image is obtained by progressively enlarging the feature map size and fusing shallow detail information. S5 generates an enhanced image from the restored image input / output layer and outputs it to the streaming rearview mirror in real time through the in-vehicle display system.

[0019] In one embodiment, the method for improving the residual network includes: An adaptive convolution kernel adjustment layer is added before the first convolutional layer of each feature extraction unit. The adjustment layer dynamically adjusts the size and stride of the convolution kernel according to the average brightness of the input image. When the average brightness is lower than a preset threshold, the size of the convolution kernel is increased and the stride is decreased. A channel attention module is embedded in the skip connections of the residual structure. The attention module obtains the weight coefficients of the feature channels through global average pooling and performs channel-dimensional weighted filtering on the input features. The output features of the last convolutional layer in each feature extraction unit are concatenated with the input features of that unit channel by channel, and then the number of channels is compressed by a 1×1 convolutional layer to form the improved residual structure.

[0020] In this example, an adaptive convolution kernel adjustment layer is added before the first convolutional layer of each feature extraction unit. Its core function is to dynamically match the scale and stride of the convolution operation by sensing the image brightness in real time. This adjustment layer first iterates through each pixel of the input image, calculates the average brightness value of the three channels (R, G, B) (i.e., (R+G+B) / 3), and compares it with a preset threshold (e.g., 50 in practical applications, corresponding to darker scenes in the 0-255 grayscale range). When the average brightness of the nighttime road image is detected to be 30 (below the threshold), it indicates that the dark areas in the image are large and the details are scattered. At this point, the adjustment layer expands the 7×7 convolution kernel originally used for initial feature extraction to 9×9 to increase the receptive field of the convolution operation, thereby capturing a wider range of dark texture information. Simultaneously, the stride is reduced from 2 to 1 to avoid detail loss due to large stride compression of the feature map, ensuring that weak edges (such as road markings and pedestrian outlines) in low-light environments can be effectively captured. Secondly, the channel attention module embedded in the skip connections of the residual structure serves to filter out the more valuable information for night vision enhancement from multi-channel features and suppress noise-dominated invalid features. This module first performs global average pooling on the feature map. For example, for a 16×16 pixel feature sub-map, it calculates the pixel mean in a 2×2 pooling window for each region, compressing the two-dimensional feature of each channel into a 1×1 scalar value, forming a descriptive vector that can characterize the importance of the channel. Then, it performs a non-linear transformation on this vector through two fully connected layers, outputting weight coefficients (within the range of 0-1) corresponding to the number of channels. In low-light scenes, channels corresponding to edges and contours (such as channels capturing the edges of car headlights) are assigned higher weights (e.g., 0.8), while channels with concentrated noise (such as channels corresponding to random noise points on the road surface) have lower weights (e.g., 0.2). Through this weighted filtering, the features transmitted in the skip connections are more focused on effective information, reducing noise interference. Finally, the output fusion method of the residual structure is optimized by directly concatenating the output features of the last convolutional layer of each feature extraction unit with the input features of that unit on a channel-by-channel basis. For example, if the unit input is an initial feature with 64 channels (containing basic details of the original image) and the output is a deep feature with 128 channels (containing abstract semantic information), the concatenation results in a fused feature with 192 channels. A 1×1 convolutional layer then compresses the number of channels back to 128. This preserves the original details in the input features (such as subtle textures in dark areas) while fusing the semantic information of the deep features through convolution, avoiding information loss during feature transfer. For instance, when processing taillights at night, the taillight beam outline in the input features and the beam semantic features in the output features, after concatenation and compression, form a more complete feature representation, providing a more reliable foundation for subsequent enhancement processing. Through these three improvements, the feature extraction accuracy of the residual network in low-light environments is significantly improved, especially in balancing the preservation of dark details and noise suppression.

[0021] In one embodiment, the feature enhancement module performs parallel processing of features at different scales using a multi-branch dilated convolutional structure, including: The extracted features are divided into at least three feature sub-maps according to the channel dimension, and each sub-map is input into three parallel dilated convolution branches. The first branch uses a 3×3 convolutional layer with a dilation rate of 2 to process the feature sub-image, the second branch uses a 3×3 convolutional layer with a dilation rate of 4, and the third branch uses a 3×3 convolutional layer with a dilation rate of 6. Each branch undergoes batch normalization and ReLU activation function sequentially after the convolution operation; The output features of the three branches are concatenated along the channel dimension to form an intermediate feature map.

[0022] Specifically, the steps for generating enhanced features through pixel-by-pixel fusion include: Perform local contrast calculation on the intermediate feature map to obtain the local contrast value of each pixel; The local contrast values ​​are normalized using the Sigmoid function to obtain the weight coefficients corresponding to each pixel. Based on the weighting coefficients, the feature values ​​of corresponding pixels in the intermediate feature map are weighted and summed to generate preliminary fusion features; The preliminary fusion features are subjected to a 3×3 convolution operation to eliminate splicing traces, resulting in enhanced features.

[0023] In this implementation example, the feature enhancement module processes features at different scales in parallel using a multi-branch dilated convolutional structure. Its underlying logic addresses the differences in feature distribution among targets of different sizes in low-light environments (such as distant traffic signs, nearby road markings, and vehicles at medium distances). By simultaneously capturing various features through multi-scale receptive fields, it avoids feature omissions caused by single-scale processing. Specifically, the multi-scale features extracted by the residual network (e.g., a 128-channel feature map) are first divided into three sub-maps (each with 42-43 channels) according to the channel dimension. This equal division ensures that each sub-map contains relatively complete basic feature information while reducing the computational load of a single branch. The three sub-images are input into three parallel dilated convolution branches. The first branch uses a 3×3 convolutional layer with a dilution rate of 2, whose actual receptive field is equivalent to a 7×7 convolution (calculated as: receptive field = kernel size + (kernel size - 1) × (dilution rate - 1)), suitable for capturing medium-scale features (such as vehicle outlines). The second branch uses a 3×3 convolutional layer with a dilution rate of 4, expanding the receptive field to 11×11, used to extract large-scale features (such as the overall road trend). The third branch uses a 3×3 convolutional layer with a dilution rate of 6, with a receptive field of 15×15, focusing on capturing small-scale details (such as road surface cracks and pedestrian limbs). This design with different dilution rates allows each branch to focus on features of a specific scale without increasing the kernel parameters, solving the problem of feature dispersion and difficulty in covering features with a single scale in low-light environments. Each branch undergoes batch normalization and ReLU activation after convolution. Batch normalization standardizes the features of each batch to a mean of 0 and a variance of 1, reducing distribution shifts caused by fluctuations in low-light image features and stabilizing the feature distribution. The ReLU activation function suppresses negative features, enhances the non-linear expressive power of the features, highlights effective features (such as bright areas of headlights), and suppresses noise (such as random noise in dark areas). Finally, the output features of the three branches are concatenated along the channel dimension to form an intermediate feature map that integrates information at different scales (the number of channels is the sum of the outputs of each branch, such as 128 channels), providing a rich multi-scale feature foundation for subsequent fusion processing.

[0024] The core of the pixel-by-pixel fusion process for generating enhanced features lies in dynamically adjusting the contribution weights of features at different scales based on the local importance of pixels, allowing key regions (such as edges and details) to be more fully enhanced. Specifically, firstly, local contrast is calculated on the intermediate feature map. A 5×5 neighborhood window is taken centered on each pixel, and the difference between the maximum and minimum feature values ​​of the pixels within the window is used as the local contrast value of that pixel. In low-light images, the local contrast of edge or detail regions (such as vehicle edges and road markings) is usually higher than that of flat regions (such as dark road surfaces), so this value can effectively distinguish the importance of pixels. Subsequently, the local contrast value is input into the Sigmoid function (σ(x)=1 / (1+e^(-x))) for normalization, mapping the original contrast value to a weight coefficient in the range of 0-1. For example, high-contrast edge pixels may receive a weight of 0.8-0.9, while low-contrast flat region pixels may have a weight between 0.2-0.3. This normalization ensures the comparability and stability of the weights. Next, the feature values ​​of corresponding pixels in the intermediate feature map are weighted and summed according to these weight coefficients. That is, the final value of each pixel is the sum of the product of the value of that position in the three branch features and the corresponding weight. This process allows high-contrast key pixels to integrate more effective features from each branch (such as simultaneously preserving small-scale edge details and large-scale semantic information), while low-contrast regions weaken the superposition of features from different branches to avoid noise amplification. Finally, a 3×3 convolution operation (stride 1, padding 1) is performed on the preliminary fused features. The local weighted smoothing effect of the convolution kernel eliminates channel boundary traces that may be generated when splicing features from different branches, making the feature map more spatially coherent, and finally forming an enhanced feature that contains multi-scale information and highlights key details.

[0025] In one embodiment, the step of inputting the enhanced features into an upsampling network for resolution restoration includes: The enhanced features are input into the first deconvolution layer of the upsampling network, and the first-scale feature map is generated by magnification by 2 times. The feature map is then concatenated with the shallow features output by the first feature extraction unit in the residual network by channel. The concatenated features are input into the second deconvolutional layer, and a second-scale feature map is generated by magnification by 2 times. The feature map is then concatenated with the original detail features output by the initial convolutional layer in the residual network by channel. The stitched features are input into the third deconvolution layer, and a third-scale feature map is generated by magnification by 2 times. The resolution of the third-scale feature map is the same as that of the original image. An adjustment layer with a kernel size of 1×1 is set after each deconvolution layer to unify the number of channels in the spliced ​​features.

[0026] Specifically, the steps to obtain the restored image by progressively enlarging the feature map size and fusing shallow detail information include: When the first deconvolutional layer amplifies the enhanced features by 2 times, the shallow features output by the first feature extraction unit of the residual network are weighted point by point according to the pixel position and the amplified feature map. The weights are dynamically allocated according to the gradient value of the shallow features, and the higher the gradient value, the greater the weight. When the second deconvolutional layer magnifies the weighted first-scale feature map by 2 times, the original detailed features output by the initial convolutional layer of the residual network are used to extract high-frequency components through the edge detection operator and then superimposed on the magnified feature map by the channel dimension. When the third deconvolution layer magnifies the stacked second-scale feature map by 2 times, it uses a combination of bilinear interpolation and deconvolution to maintain edge continuity. After magnification, the feature map is optimized as a whole through a 3×3 convolution layer to generate a restored image with the same resolution as the original image.

[0027] In this example, the three deconvolutional layers of the upsampling network form a stepped magnification structure, each layer magnifying by a factor of 2, ultimately restoring the enhanced features (typically with lower resolution, such as 160×90 pixels) to the original image size (1280×720 pixels). The first deconvolutional layer first magnifies the enhanced features to 320×180 pixels, generating a first-scale feature map. At this point, to supplement mid-level details (such as the preliminary texture of the vehicle outline), this is concatenated with the shallow features (at the same scale, 320×180 pixels) output by the first feature extraction unit of the residual network, channel by channel. The shallow features retain more original details without depth compression, and the concatenation injects basic texture information into the magnified feature map. The number of feature channels after concatenation increases (e.g., 128 channels for enhanced features + 64 channels for shallow features = 192 channels). Therefore, a 1×1 convolutional layer is set as an adjustment layer after the first deconvolutional layer. The number of channels is unified to 128 through convolution calculation, which reduces redundant parameters and makes the fused features more consistent in the channel dimension. Next, the concatenated features are input into a second deconvolutional layer, magnified to 640×360 pixels to generate a second-scale feature map. Here, to supplement more original details (such as minor bumps in the road surface and the outlines of distant pedestrians), it is concatenated with the original detail features (at the same scale of 640×360 pixels) output from the initial convolutional layer of the residual network. These original detail features contain texture information closest to the input image, effectively mitigating detail loss during magnification. Similarly, a 1×1 convolutional layer unifies the number of channels to 64, balancing feature dimension and computational efficiency. Finally, a third deconvolutional layer magnifies the feature map to 1280×720 pixels, consistent with the original image resolution, and then a 1×1 convolutional layer adjusts it to 32 channels, providing a suitable feature dimension for subsequent optimization. This stepped magnification and staged fusion design avoids information loss caused by a one-time magnification, allowing details at different levels to be supplemented at an appropriate scale, laying the foundation for restoring a clear image.

[0028] The process of progressively enlarging the feature map size and fusing shallow detail information to obtain the restored image further addresses the problem of weak details and blurred edges in low-light environments by employing a differentiated fusion strategy to enhance the transmission of key information. When the first deconvolutional layer enlarges the enhanced features by a factor of 2, it performs point-by-point weighted fusion with the shallow features of the first feature extraction unit. First, the gradient value of each pixel in the shallow features is calculated using the Sobel operator (a higher gradient value indicates that the location may be an edge or texture region). Then, weights are dynamically assigned based on the gradient values: pixels with high gradient values ​​(such as road edges and vehicle outlines) are assigned a high weight of 0.6-0.7, allowing more details from the shallow features to be integrated into the enlarged feature map; flat areas with low gradient values ​​(such as dark road surfaces) are assigned a low weight of 0.3-0.4 to avoid introducing excessive noise. This weighting method effectively preserves key edge information from the initial enlargement stage. When the second deconvolutional layer magnifies the first-scale feature map by 2x, it extracts high-frequency components (mainly containing rapidly changing information such as edges and textures) from the original detailed features of the initial convolutional layer using the Canny operator. These components are then directly superimposed on the magnified feature map along the channel dimension. High-frequency components are often masked by noise in low-light images, and superposition can specifically enhance these weak detail signals, such as making the edges of road markings illuminated by headlights clearer. When the third deconvolutional layer magnifies the second-scale feature map by 2x, it uses a combination of bilinear interpolation and deconvolution: deconvolution is responsible for the initial magnification and maintaining the semantic coherence of the features, while bilinear interpolation refines the edge pixels, avoiding checkerboard artifacts that may be generated by deconvolution. This is especially effective in maintaining edge continuity when dealing with fine lines (such as lane lines) in low-light environments. After magnification, a 3×3 convolutional layer (stride 1, padding 1) is used to smooth and optimize the feature map as a whole, eliminating local inconsistencies caused by the fusion of each stage. Finally, a restored image with the same resolution as the original image, rich in detail, and with clear edges is generated. This progressive fusion strategy allows shallow details to be precisely injected at the most appropriate magnification stage, avoiding detail overload and ensuring that key information is not lost during resolution restoration, thus significantly improving the restoration quality of low-light images.

[0029] In one embodiment, the step of generating an enhanced image from the restored image input / output layer and outputting it in real time to a streaming media rearview mirror via an in-vehicle display system includes: The restored image is input into the output layer. The output layer integrates the image features through a 3×3 convolutional layer, and then adjusts the number of channels to 3 through a 2×2 convolutional layer to obtain an initial enhanced image with the same color space as the original image. The initial enhanced image is locally brightened by calculating the average brightness of each 8×8 pixel block in the image. The brightness values ​​of pixels below the preset brightness threshold are increased to the threshold level, while those above the threshold remain unchanged. The corrected and enhanced image is transmitted to the vehicle display system, which performs format conversion on the image data to match the display parameters of the streaming rearview mirror and monitors the transmission delay in real time. When the delay exceeds a preset value, the cache preloading process is initiated. The processed enhanced image is output to the streaming rearview mirror via the in-vehicle display system at a rate of no less than 30 frames per second, and the night vision enhanced image is displayed in real time.

[0030] Specifically, the steps to initiate the cache preloading process when the delay exceeds a preset value include: When a transmission delay exceeding a preset value is detected, a buffer area is defined, the capacity of which is at least the amount of image data required for the streaming media rearview mirror to display for 5 seconds. The image processing module obtains the enhanced image data to be displayed, and loads the image data of the first 3 seconds into the cache area in the order of generation time. After the buffer area is full of 3 seconds of data, the image data in the buffer is output to the streaming rearview mirror in sequence, while the remaining 2 seconds of image data are loaded. The system monitors the current transmission latency in real time. When the latency drops below a preset value, it gradually reduces the amount of cached data loaded until it returns to direct transmission mode. Unused image data in the cache area is automatically cleared.

[0031] In this example, the restored image first enters the output layer. The 3×3 convolutional layer of the output layer globally integrates image features through a large receptive field, smoothing feature differences between different regions. For example, it makes the feature transition between vehicle edges and the background road surface more natural. Subsequently, the 2×2 convolutional layer compresses the number of channels (from 32 channels to 3 channels) to convert the feature map into the same RGB color space as the original image, generating the initial enhanced image and ensuring that the output image meets the color standards of the display device. To address the problem of local over-darkness in low-light images, the local brightness correction stage adjusts the brightness of the region by dividing it into 8×8 pixel blocks: the average brightness of each pixel block is calculated (e.g., if the average brightness of a certain block is 60, the preset threshold is 100). The brightness of blocks below the threshold is increased to 100, while the brightness of blocks above the threshold remains unchanged. This local adjustment avoids overexposure of bright areas (such as the headlight area) caused by global brightening, while ensuring the visibility of details in dark areas (such as roadside obstacles). After the corrected image is transmitted to the in-vehicle display system, the system first converts the image data from RGB format to MIPI signal (matching the interface standard of the streaming rearview mirror). Simultaneously, a timer monitors the transmission latency (the time difference between image processing completion and display start) in real time. When the latency exceeds 50ms (a preset threshold), a cache preloading process is immediately initiated to maintain display smoothness. Finally, the processed enhanced image is output to the streaming rearview mirror at a rate of 30 frames per second, ensuring a smooth and uninterrupted view for the driver.

[0032] The cache preloading process, initiated when latency exceeds a preset value, works by pre-caching image data to buffer transmission delays and balance the conflict between image processing speed and real-time display. Specifically, once the system detects latency exceeding the threshold, it immediately allocates an independent cache area in the vehicle's memory. Its capacity is calculated based on the display requirements of a 5-second streaming rearview mirror (30 frames / second × 5 seconds = 150 frames), avoiding the consumption of other system resources. Subsequently, it prioritizes acquiring 90 frames of enhanced image data from the image processing module for the next 3 seconds and stores them in the cache in chronological order of generation. This 3-second data quickly establishes a buffer, ensuring sufficient data to maintain display even if subsequent latency continues. Once the cache is full (90 frames), the system begins sequentially reading data from the cache and outputting it to the rearview mirror, while simultaneously loading the remaining 60 frames for the next 2 seconds, forming a "display-as-you-go" mechanism to prevent cache exhaustion. Throughout the process, the system continuously monitors transmission latency. When the latency drops below 30ms (returning to normal levels), it gradually reduces the cache load (by 10 frames each time) until preloading is completely stopped, reverting to a mode where image processing is completed and transmission occurs directly. Unused cached data (such as the remaining 20 frames) is automatically cleared to free up memory. This dynamically adjusted caching strategy ensures smooth display when latency is too high and reduces resource consumption after system recovery, adapting to the characteristics of processor load fluctuations in the automotive environment.

[0033] This invention proposes a streaming media rearview mirror night vision enhancement device based on deep learning image processing, comprising: The acquisition unit is used to acquire raw images from the vehicle-mounted camera in low-light environments. An extraction unit is used for multi-scale feature extraction through an improved residual network, wherein the residual network contains at least two cascaded feature extraction units, each feature extraction unit is composed of multiple convolutional layers, and adjacent convolutional layers are connected by skip connections to form a residual structure. An enhancement unit is used to input the features extracted from the multi-scale features into a feature enhancement module. The feature enhancement module processes features of different scales in parallel through a multi-branch dilated convolutional structure and generates enhanced features by pixel-wise fusion. The fusion unit is used to input the enhanced features into the upsampling network for resolution restoration. The upsampling network contains at least three deconvolutional layers, which progressively enlarge the feature map size and fuse shallow detail information to obtain the restored image. The display unit is used to generate an enhanced image from the restored image input / output layer and output it to the streaming media rearview mirror in real time through the vehicle display system.

[0034] The present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described streaming media rearview mirror night vision enhancement method based on deep learning image processing.

[0035] In summary, the computer device proposed in this invention can be used as an in-vehicle display system to execute a night vision enhancement method for streaming rearview mirrors based on deep learning image processing. Specifically, it aims to solve the problems of high noise, blurred details, and high processing latency in streaming rearview mirror images under low-light conditions. The method includes: after acquiring the original low-light image, extracting multi-scale features through an improved residual network, which includes an adaptive convolutional kernel adjustment layer and a channel attention module to optimize feature extraction; parallel processing of features at different scales through multi-branch dilated convolutions of the feature enhancement module, combined with pixel-wise weighted fusion to generate enhanced features; progressively amplifying the feature map through an upsampling network and fusing shallow details to restore resolution; after the output layer generates the enhanced image, it is transmitted to the rearview mirror in real time by the in-vehicle system, and a cache preloading mechanism is activated when the latency exceeds a preset value. This invention, by optimizing the network structure and transmission mechanism, balances night vision enhancement effect and real-time performance, significantly improving image clarity and dynamic response speed in low-light scenes.

[0036] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A streaming media rearview mirror night vision enhancement method based on deep learning image processing, characterized in that, include: Acquire raw images from the vehicle-mounted camera in low-light environments; Multi-scale feature extraction is performed using an improved residual network, which contains at least two cascaded feature extraction units. Each feature extraction unit consists of multiple convolutional layers, and adjacent convolutional layers are connected by skip connections to form a residual structure. The features extracted from the multi-scale features are input into the feature enhancement module. The feature enhancement module processes features of different scales in parallel through a multi-branch dilated convolutional structure and generates enhanced features by pixel-by-pixel fusion. The enhanced features are input into an upsampling network for resolution restoration. The upsampling network contains at least three deconvolutional layers. By progressively enlarging the feature map size and fusing shallow detail information, the restored image is obtained. The restored image is used to generate an enhanced image from the input and output layers, and then output to the streaming rearview mirror in real time through the in-vehicle display system.

2. The streaming media rearview mirror night vision enhancement method based on deep learning image processing according to claim 1, characterized in that, The improved method for the residual network includes: An adaptive convolution kernel adjustment layer is added before the first convolutional layer of each feature extraction unit. The adjustment layer dynamically adjusts the size and stride of the convolution kernel according to the average brightness of the input image. When the average brightness is lower than a preset threshold, the size of the convolution kernel is increased and the stride is decreased. A channel attention module is embedded in the skip connections of the residual structure. The attention module obtains the weight coefficients of the feature channels through global average pooling and performs channel-dimensional weighted filtering on the input features. The output features of the last convolutional layer in each feature extraction unit are concatenated with the input features of that unit channel by channel, and then the number of channels is compressed by a 1×1 convolutional layer to form the improved residual structure.

3. The streaming media rearview mirror night vision enhancement method based on deep learning image processing according to claim 1, characterized in that, The feature enhancement module performs parallel processing of features at different scales using a multi-branch dilated convolutional structure, including: The extracted features are divided into at least three feature sub-maps according to the channel dimension, and each sub-map is input into three parallel dilated convolution branches. The first branch uses a 3×3 convolutional layer with a dilation rate of 2 to process the feature sub-image, the second branch uses a 3×3 convolutional layer with a dilation rate of 4, and the third branch uses a 3×3 convolutional layer with a dilation rate of 6. Each branch undergoes batch normalization and ReLU activation function sequentially after the convolution operation; The output features of the three branches are concatenated along the channel dimension to form an intermediate feature map.

4. The streaming media rearview mirror night vision enhancement method based on deep learning image processing according to claim 3, characterized in that, The steps for generating enhanced features through pixel-by-pixel fusion include: Perform local contrast calculation on the intermediate feature map to obtain the local contrast value of each pixel; The local contrast values ​​are normalized using the Sigmoid function to obtain the weight coefficients corresponding to each pixel. Based on the weighting coefficients, the feature values ​​of corresponding pixels in the intermediate feature map are weighted and summed to generate preliminary fusion features; The preliminary fusion features are subjected to a 3×3 convolution operation to eliminate splicing traces, resulting in enhanced features.

5. The streaming media rearview mirror night vision enhancement method based on deep learning image processing according to claim 1, characterized in that, The step of inputting the enhanced features into the upsampling network for resolution restoration includes: The enhanced features are input into the first deconvolution layer of the upsampling network, and the first-scale feature map is generated by magnification by 2 times. The feature map is then concatenated with the shallow features output by the first feature extraction unit in the residual network by channel. The concatenated features are input into the second deconvolutional layer, and a second-scale feature map is generated by magnification by 2 times. The feature map is then concatenated with the original detail features output by the initial convolutional layer in the residual network by channel. The stitched features are input into the third deconvolution layer, and a third-scale feature map is generated by magnification by 2 times. The resolution of the third-scale feature map is the same as that of the original image. An adjustment layer with a kernel size of 1×1 is set after each deconvolution layer to unify the number of channels in the spliced ​​features.

6. The streaming media rearview mirror night vision enhancement method based on deep learning image processing according to claim 5, characterized in that, The steps to obtain the restored image by progressively enlarging the feature map size and fusing shallow detail information include: When the first deconvolutional layer amplifies the enhanced features by 2 times, the shallow features output by the first feature extraction unit of the residual network are weighted point by point according to the pixel position and the amplified feature map. The weights are dynamically allocated according to the gradient value of the shallow features, and the higher the gradient value, the greater the weight. When the second deconvolutional layer magnifies the weighted first-scale feature map by 2 times, the original detailed features output by the initial convolutional layer of the residual network are used to extract high-frequency components through the edge detection operator and then superimposed on the magnified feature map by the channel dimension. When the third deconvolution layer magnifies the stacked second-scale feature map by 2 times, it uses a combination of bilinear interpolation and deconvolution to maintain edge continuity. After magnification, the feature map is optimized as a whole through a 3×3 convolution layer to generate a restored image with the same resolution as the original image.

7. The streaming media rearview mirror night vision enhancement method based on deep learning image processing according to claim 1, characterized in that, The steps of generating an enhanced image from the restored image input / output layer and outputting it in real time to the streaming rearview mirror via the in-vehicle display system include: The restored image is input into the output layer. The output layer integrates the image features through a 3×3 convolutional layer, and then adjusts the number of channels to 3 through a 2×2 convolutional layer to obtain an initial enhanced image with the same color space as the original image. The initial enhanced image is locally brightened by calculating the average brightness of each 8×8 pixel block in the image. The brightness values ​​of pixels below the preset brightness threshold are increased to the threshold level, while those above the threshold remain unchanged. The corrected and enhanced image is transmitted to the vehicle display system, which performs format conversion on the image data to match the display parameters of the streaming rearview mirror and monitors the transmission delay in real time. When the delay exceeds a preset value, the cache preloading process is initiated. The processed enhanced image is output to the streaming rearview mirror via the in-vehicle display system at a rate of no less than 30 frames per second, and the night vision enhanced image is displayed in real time.

8. The streaming media rearview mirror night vision enhancement method based on deep learning image processing according to claim 7, characterized in that, The steps to initiate the cache preloading process when the delay exceeds a preset value include: When a transmission delay exceeding a preset value is detected, a buffer area is defined, the capacity of which is at least the amount of image data required for the streaming media rearview mirror to display for 5 seconds. The image processing module obtains the enhanced image data to be displayed, and loads the image data of the first 3 seconds into the cache area in the order of generation time. After the buffer area is full of 3 seconds of data, the image data in the buffer is output to the streaming rearview mirror in sequence, while the remaining 2 seconds of image data are loaded. The system monitors the current transmission latency in real time. When the latency drops below a preset value, it gradually reduces the amount of cached data loaded until it returns to direct transmission mode. Unused image data in the cache area is automatically cleared.

9. A streaming media rearview mirror night vision enhancement device based on deep learning image processing, characterized in that, include: The acquisition unit is used to acquire raw images from the vehicle-mounted camera in low-light environments. An extraction unit is used for multi-scale feature extraction through an improved residual network, wherein the residual network contains at least two cascaded feature extraction units, each feature extraction unit is composed of multiple convolutional layers, and adjacent convolutional layers are connected by skip connections to form a residual structure. An enhancement unit is used to input the features extracted from the multi-scale features into a feature enhancement module. The feature enhancement module processes features of different scales in parallel through a multi-branch dilated convolutional structure and generates enhanced features by pixel-wise fusion. The fusion unit is used to input the enhanced features into the upsampling network for resolution restoration. The upsampling network contains at least three deconvolutional layers, which progressively enlarge the feature map size and fuse shallow detail information to obtain the restored image. The display unit is used to generate an enhanced image from the restored image input / output layer and output it to the streaming media rearview mirror in real time through the vehicle display system.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the streaming media rearview mirror night vision enhancement method based on deep learning image processing as described in any one of claims 1 to 8.

Images