An image processing method, apparatus and device

Abstract

The application provides an image processing method, device and equipment, and applies to the technical field of image processing. The method comprises the following steps: collecting a first resolution image of a vehicle-mounted environment during vehicle driving; performing degradation suppression processing on the first resolution image to obtain an intermediate image; performing iterative noise reduction on the intermediate image based on the environmental conditions of the vehicle-mounted environment to obtain a second resolution image of the vehicle-mounted environment. The resolution of the second resolution image is higher than that of the first resolution image. The second resolution image is displayed through a human-computer interaction interface of the vehicle, and the problems of serious imaging degradation, insufficient high-frequency detail recovery capability and poor adaptability to complex vehicle-mounted scenes existing in the traditional electronic rearview mirror system can be solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, apparatus and device. Background Technology

[0002] In the wave of automotive intelligence development, electronic rearview mirrors, as an important component of active safety systems, have evolved from simple optical reflective devices into comprehensive perception modules integrating image acquisition, processing, and display. Electronic rearview mirror systems capture environmental information through high dynamic range cameras and possess advanced functions such as digital zoom, multi-view switching, and real-time image enhancement. Their image quality directly affects the driver's accuracy in judging the surrounding environment. In related technologies, electronic rearview mirror systems generally use complementary metal-oxide-semiconductor (CMOS) sensors to acquire video streams. However, in low-light, rainy / foggy weather, or high-speed driving scenarios, the original image is prone to resolution degradation and loss of detail, posing a challenge to advanced driving assistance systems (ADAS) that rely on visual information for decision-making.

[0003] Super-resolution reconstruction technology, as a key means to improve image quality, has evolved from traditional bicubic interpolation and sample-based dictionary learning methods to the deep learning stage represented by convolutional neural networks (CNNs) and generative adversarial networks (GANs). CNN architecture achieves end-to-end mapping learning through multi-level feature extraction, but it is prone to over-smoothing effects in high-frequency detail reconstruction. While GAN frameworks can generate richer texture details, the adversarial training mechanism between the discriminator and generator leads to structural distortions and local artifacts in the output image.

[0004] Therefore, there is an urgent need for an image processing method, device, and equipment to solve the problems of severe image degradation, insufficient high-frequency detail recovery, and poor adaptability to complex vehicle scenarios that exist in traditional electronic rearview mirror systems. Summary of the Invention

[0005] The purpose of this application is to provide an image processing method, apparatus, and device to solve the problems of severe image degradation, insufficient high-frequency detail recovery, and poor adaptability to complex vehicle scenarios that exist in traditional electronic rearview mirror systems.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring a first-resolution image of the vehicle environment during vehicle operation; performing degradation suppression processing on the first-resolution image to obtain an intermediate image; performing iterative noise reduction on the intermediate image based on the environmental conditions of the vehicle environment to obtain a second-resolution image of the vehicle environment; the resolution of the second-resolution image being higher than that of the first-resolution image; and displaying the second-resolution image through the vehicle's human-machine interface.

[0007] The image processing method provided in this application acquires a first-resolution image of the in-vehicle environment during vehicle operation. The first-resolution image undergoes degradation suppression processing to obtain an intermediate image. Then, based on the environmental conditions of the in-vehicle environment, the intermediate image undergoes iterative noise reduction to obtain a second-resolution image of the in-vehicle environment. Finally, the second-resolution image is displayed through the vehicle's human-machine interface. This process effectively solves the problems of image texture smoothing distortion, loss of high-frequency details, poor environmental adaptability, and difficulty in balancing computational efficiency and image quality in traditional techniques by introducing a pre-suppression mechanism for in-vehicle imaging degradation and combining iterative fine reconstruction under environmental constraints.

[0008] One possible implementation involves performing degradation suppression processing on a first-resolution image to obtain an intermediate image, including: extracting shallow features related to basic visual structures in the vehicular environment from the first-resolution image; extracting deep features related to vehicular imaging degradation from the first-resolution image based on the shallow features; suppressing the vehicular imaging degradation-related feature components in the deep features; and fusing the suppressed deep features with the shallow features to obtain the intermediate image.

[0009] One possible implementation involves extracting deep features related to vehicle imaging degradation from a first-resolution image based on shallow features. This includes: identifying spatial blur features caused by vehicle motion and external interference features introduced by the vehicle environment from the first-resolution image based on the shallow features; and generating deep features based on the spatial blur features and external interference features.

[0010] One possible implementation involves fusing suppressed deep features with shallow features to obtain an intermediate image, including: adding the suppressed deep features and shallow features element-wise to obtain fused features; and performing resolution upscaling and reconstruction processing on the fused features to generate the intermediate image.

[0011] One possible implementation involves iteratively denoising an intermediate image based on the environmental conditions of the in-vehicle environment to obtain a second-resolution image of the in-vehicle environment. This includes: acquiring the environmental conditions of the in-vehicle environment, which include multiple environmental elements and their spatial distribution relationships; converting the intermediate image into a corresponding latent spatial feature representation, which is obtained by mapping the intermediate image to a low-dimensional continuous feature space; iteratively denoising the latent spatial feature representation based on the multiple environmental elements and their spatial distribution relationships to obtain a sharp latent spatial feature representation corresponding to the intermediate image; and reconstructing the sharp latent spatial feature representation to obtain the second-resolution image.

[0012] One possible implementation involves iteratively denoising the latent spatial feature representation based on multiple environmental elements and their spatial distribution relationships to obtain a clear latent spatial feature representation corresponding to an intermediate image. This includes: adding preset noise to the latent spatial feature representation to obtain an initial noise feature representation; generating constraints for iterative denoising based on the multiple environmental elements and their spatial distribution relationships; and performing multiple rounds of denoising operations on the initial noise feature representation based on the constraints to obtain a clear latent spatial feature representation. Each round of denoising involves adjusting the noise features input in the current round according to the constraints, predicting and eliminating the noise components of the noise features input in the current round.

[0013] One possible implementation, the image processing method provided in this application embodiment, further includes: calculating the gradient magnitude and local variance of a first-resolution image; determining a complexity index of the first-resolution image based on the gradient magnitude and local variance; and determining the number of iterations for iterative denoising of the latent spatial feature representation according to the complexity index and a preset complexity range.

[0014] Secondly, embodiments of this application provide an image processing apparatus, which includes: an acquisition module, a processing module, a noise reduction module, and a display module.

[0015] The acquisition module is used to acquire first-resolution images of the in-vehicle environment while the vehicle is in motion.

[0016] The processing module is used to perform degradation suppression processing on the first resolution image to obtain an intermediate image.

[0017] The noise reduction module iteratively reduces noise in the intermediate image based on the environmental conditions of the vehicle environment, obtaining a second-resolution image of the vehicle environment. The resolution of the second-resolution image is higher than that of the first-resolution image.

[0018] The display module is used to display a second-resolution image through the vehicle's human-machine interface.

[0019] One possible implementation involves a processing module that performs degradation suppression processing on a first-resolution image to obtain an intermediate image. Specifically, this module extracts shallow features related to the basic visual structures in the vehicular environment from the first-resolution image. Based on these shallow features, it extracts deep features related to vehicular imaging degradation from the first-resolution image. It then suppresses the vehicular imaging degradation-related feature components within the deep features. Finally, it fuses the suppressed deep features with the shallow features to obtain the intermediate image.

[0020] One possible implementation involves a processing module that, when extracting deep features related to vehicle imaging degradation from a first-resolution image based on shallow features, specifically: identifying spatial blur features caused by vehicle motion and external interference features introduced by the vehicle environment from the first-resolution image based on the shallow features; and generating deep features based on the spatial blur features and external interference features.

[0021] One possible implementation involves a processing module that fuses suppressed deep features with shallow features to obtain an intermediate image. Specifically, this process involves: adding the suppressed deep features and shallow features element-wise to obtain fused features; and then performing resolution upscaling and reconstruction on the fused features to generate the intermediate image.

[0022] One possible implementation involves a denoising module used to iteratively denoise the intermediate image based on the environmental conditions of the vehicle environment to obtain a second-resolution image of the vehicle environment. Specifically, this module is used to: acquire the environmental conditions of the vehicle environment, including multiple environmental elements and their spatial distribution relationships; convert the intermediate image into a corresponding latent spatial feature representation, which is obtained by mapping the intermediate image to a low-dimensional continuous feature space; iteratively denoise the latent spatial feature representation based on the multiple environmental elements and their spatial distribution relationships to obtain a clear latent spatial feature representation corresponding to the intermediate image; and reconstruct the clear latent spatial feature representation to obtain the second-resolution image.

[0023] One possible implementation involves a denoising module that iteratively denoises the latent spatial feature representation based on multiple environmental elements and their spatial distribution relationships to obtain a clear latent spatial feature representation corresponding to the intermediate image. Specifically, this module: adds preset noise to the latent spatial feature representation to obtain an initial noise feature representation; generates constraints for iterative denoising based on the multiple environmental elements and their spatial distribution relationships; and performs multiple rounds of denoising operations on the initial noise feature representation according to the constraints to obtain a clear latent spatial feature representation. Each round of denoising involves adjusting the noise features input in the current round based on the constraints, predicting and eliminating the noise components of the noise features input in the current round.

[0024] In one possible implementation, the image processing apparatus provided in this application embodiment is further configured to: calculate the gradient magnitude and local variance of a first-resolution image; determine a complexity index of the first-resolution image based on the gradient magnitude and local variance; and determine the number of iterations for iterative denoising of the latent spatial feature representation according to the complexity index and a preset complexity range.

[0025] Thirdly, embodiments of this application provide an image processing apparatus that has the function of implementing the image processing method described in the first aspect or any possible implementation of the first aspect. This function can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the image processing method described in the first aspect or any possible implementation thereof.

[0027] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, enable the computer to execute the image processing method described in the first aspect or any possible implementation thereof.

[0028] The technical effects of any of the design methods in aspects two through five can be found in aspect one or in different possible implementations of aspect one, and will not be repeated here. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 A system structure diagram of an image processing system provided in this application embodiment; Figure 2 A flowchart illustrating an image processing method provided in this application embodiment; Figure 3 A schematic diagram of an image processing apparatus provided in an embodiment of this application; Figure 4 Another system architecture diagram of an image processing system provided in this application embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0032] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] As automotive electronic rearview mirror systems evolve from traditional reflective optical devices to digital forms integrating smart cameras and displays, the requirements for image clarity, detail reproduction, and environmental adaptability in in-vehicle imaging systems have reached unprecedented levels. High-definition, realistic real-time images are not only a bonus for improving driving comfort but also a core element for ensuring driving safety in complex traffic scenarios. However, due to limitations in the physical size of in-vehicle cameras, sensor signal-to-noise ratio, and multiple degradation factors such as continuous vibration, rapid movement, drastic changes in lighting, and rain / fog obstruction during vehicle operation, the acquired raw images generally suffer from insufficient resolution, blurred edges, missing textures, and various complex mixed noises.

[0034] To address the aforementioned image degradation issues, related technologies mainly follow two technical routes: First, super-resolution methods based on convolutional neural networks reconstruct high-resolution images through end-to-end mapping learning. However, their outputs often exhibit overly smooth texture features, lacking the high-frequency details and sharp edges expected in real-world scenes, resulting in a noticeably plastic-like visual effect. Second, super-resolution methods based on generative adversarial networks, while improving texture realism to some extent, are prone to introducing structural artifacts, checkerboard noise, or local distortions into the generated results due to the inherent instability of adversarial training. Furthermore, the generator tends to output statistically safe, conservative textures, making it difficult to reproduce the complex and varied microscopic details of natural images. Crucially, both methods employ a single-pass, instantaneous generation paradigm, failing to simulate the human visual system's gradual refinement process from coarse to fine. This results in inherent limitations in reconstructing high-frequency information such as distant vehicle outlines, pedestrian boundaries, and traffic sign text. In scenarios where electronic rearview mirrors are frequently used for magnified observation, harsh and distorted images severely impair the driver's accurate judgment of road conditions.

[0035] Furthermore, the contradiction between the resource constraints of in-vehicle computing platforms and the stringent real-time requirements of electronic rearview mirrors further exacerbates the difficulty of implementing existing advanced algorithms. How to achieve high-quality reconstruction of complex degradation scenarios with limited computing power while avoiding unnecessary computational overhead in simple scenarios remains a technical challenge that existing solutions have not yet effectively addressed.

[0036] Based on this, embodiments of this application provide an image processing method, apparatus, and device. The method includes acquiring a first-resolution image of the in-vehicle environment during vehicle operation. Degradation suppression processing is applied to the first-resolution image to obtain an intermediate image. Iterative noise reduction is performed on the intermediate image based on the environmental conditions of the in-vehicle environment to obtain a second-resolution image of the in-vehicle environment. The resolution of the second-resolution image is higher than that of the first-resolution image. The second-resolution image is displayed through the vehicle's human-machine interface.

[0037] The image processing method provided in this application acquires a first-resolution image of the in-vehicle environment during vehicle operation. The first-resolution image undergoes degradation suppression processing to obtain an intermediate image. Then, based on the environmental conditions of the in-vehicle environment, the intermediate image undergoes iterative noise reduction to obtain a second-resolution image of the in-vehicle environment. Finally, the second-resolution image is displayed through the vehicle's human-machine interface. This process effectively solves the problems of image texture smoothing distortion, loss of high-frequency details, poor environmental adaptability, and difficulty in balancing computational efficiency and image quality in traditional techniques by introducing a pre-suppression mechanism for in-vehicle imaging degradation and combining iterative fine reconstruction under environmental constraints.

[0038] The methods provided in the embodiments of this application will now be described in conjunction with the specific accompanying drawings.

[0039] On the one hand, embodiments of this application provide an image processing system. For example... Figure 1 As shown, the image processing system 100 may include an image acquisition unit 101, an image processing unit 102, and an image display unit 103.

[0040] The image acquisition unit 101 is used to capture visual information of the vehicle's external environment in real time during vehicle operation and generate a raw first-resolution image. The image acquisition unit 101 can be deployed at the rearview mirror positions on both sides of the vehicle or other areas of the vehicle body requiring visual perception. The core component of the image acquisition unit 101 is a digital image sensor, such as a high dynamic range, low-light sensitive CMOS sensor like the Sony IMX series or OmniVision series, whose resolution can be configured to 640×480, 1280×720, or higher, depending on system requirements.

[0041] The image acquisition unit 101 may also integrate an optical lens assembly, an infrared filter, and a vibration-resistant fixing structure to cope with temperature changes, mechanical vibrations, and complex lighting conditions in the vehicle environment. Furthermore, the image acquisition unit 101 can reliably transmit the captured image data stream to the back-end image processing unit 102 via a high-bandwidth, low-latency serial interface such as MIPI CSI-2, GMSL, or FPD-Link III. For example, the image acquisition unit 101 can be a digital camera module arranged inside the electronic rearview mirror assembly, or it can be a surround-view camera or sensing camera distributed around the vehicle body.

[0042] The image processing unit 102 is used to receive and process a first-resolution image from the image acquisition unit 101, and to perform the core computing tasks of the image processing method provided in this application embodiment. The image processing unit 102 can be integrated into the intelligent driving domain controller or cockpit domain controller of the vehicle, and its hardware carrier includes, but is not limited to, high-performance system-on-chips (SoCs), such as NVIDIA Orin, Qualcomm SnapDragon Ride, Horizon Robotics Journey series, etc. These chips integrate a central processing unit (CPU) cluster, a graphics processing unit (GPU), and a neural network processing unit (NPU) designed for neural network acceleration.

[0043] The image processing unit 102 has multiple functional layers: at the bottom layer, it receives image data through a high-speed interface and performs necessary format parsing and cache management; at the algorithm layer, it loads and runs a pre-trained deep learning model, and sequentially performs core processing steps such as degradation suppression and iterative noise reduction on the input low-resolution image, reconstructing the low-quality, low-resolution input image into a high-resolution image with rich details and sharp edges; at the scheduling layer, its built-in dynamic computing scheduler analyzes the complexity of the input scene in real time, dynamically adjusts the iteration rounds and computing paths of the algorithm, and optimizes the overall power consumption and latency while ensuring the image quality of key scenes.

[0044] In addition, the image processing unit 102 can also interact with other domains of the vehicle (such as the body domain and chassis domain) to obtain auxiliary information such as vehicle speed and steering signals for motion compensation in image processing. For example, the image processing unit 102 can be a computing board integrating an AI acceleration chip, or it can be a set of dedicated logic circuits or a fixed software algorithm module on the domain controller motherboard.

[0045] The image display unit 103 receives the final high-resolution image output by the image processing unit 102 and converts it into a visible light signal for presentation to the driver. The image display unit 103 can be a high-definition display screen located on the A-pillars on the driver's and passenger's sides or on both sides of the dashboard. Its core parameters include: a resolution of at least 1920×1080 (1080p), a refresh rate of at least 60Hz, and brightness, contrast, and viewing angles that meet the stringent requirements of direct sunlight and multi-angle observation in a vehicle environment. The image display unit 103 can be connected to the image processing unit 102 via standardized digital video interfaces such as DisplayPort, HDMI, or eDP.

[0046] In addition to basic screen display functions, the image display unit 103 can also integrate a human-machine interface, which can overlay and display vehicle status information, warning prompts, auxiliary signs and other graphic elements, and provide users with interactive functions such as brightness adjustment, viewing angle switching and partial magnification. For example, the image display unit 103 can be a high-brightness touch screen dedicated to the electronic rearview mirror system, or it can be a central control screen or instrument panel shared with the in-vehicle infotainment system.

[0047] It should be noted that the above Figure 1 The image processing system 100 shown is merely an example illustrating the application scenario of the solution in this application, and is not intended to limit the application scenario of the solution in this application.

[0048] On the one hand, embodiments of this application provide an image processing method, which can be performed by... Figure 1 The image processing system 100 shown is executed. For example... Figure 2As shown, the method may include the following steps.

[0049] S201, during vehicle operation, acquires a first-resolution image of the in-vehicle environment.

[0050] The vehicular environment refers to the external physical space centered on the vehicle and covered by the field of view of the image acquisition unit. For example, the vehicular environment may include, but is not limited to, traffic participation elements and road structure elements such as the lane behind the vehicle, blind spots on the sides, vehicles in adjacent lanes, pedestrians, traffic signs, road edges, guardrails, and various background objects.

[0051] The first-resolution image refers to the original image directly captured and output by the image acquisition unit, without super-resolution reconstruction using the core processing method of this application. Its resolution is limited by the physical pixel size of the sensor and the performance of the optical system, and is usually in the form of standard video graphics array (VGA) or high-definition (HD) format, such as 640×480, 1280×720, or similar specifications. This image exists in the form of a continuous frame sequence or video stream during vehicle movement, and its image quality objectively reflects various complex degradation phenomena introduced by the vehicle imaging link, such as motion blur caused by vehicle movement and road bumps, Gaussian / Poisson mixed noise caused by insufficient exposure time and aperture configuration, edge blurring caused by physical aberrations and focus deviations of the vehicle lens, and atmospheric scattering and contrast reduction caused by adverse weather conditions such as rain, snow, fog, and haze.

[0052] Specifically, the optical signal is converted into an electrical signal by the photosensitive element in the vehicle's image acquisition unit. After basic processing such as black level correction, depigmentation, white balance, and gamma correction is completed by the analog-to-digital conversion and image signal processing (ISP) pipeline, a first-resolution digitized image is generated. This image is then transmitted in real time to the image processing unit via a high-bandwidth serial interface and stored in its DDR memory buffer, awaiting further processing.

[0053] For example, a vehicle is traveling at 60 km / h on an urban expressway. A digital camera (e.g., a module using a Sony IMX390 sensor) located within the left-side electronic rearview mirror assembly continuously captures images of the rear lane at a frame rate of 30 frames per second. The camera outputs image frames with a resolution of 1280×720 and a color depth of 24-bit RGB format. Because it is dusk, the ambient light intensity is approximately 200 lux, and the vehicle experiences slight high-frequency vibrations due to road surface seams. As a result, the edges of vehicles approaching from behind exhibit a trailing blur of approximately 3 to 5 pixels in the captured image, while the edges of the text on distant traffic signs show visible color misalignment and loss of detail.

[0054] Furthermore, after acquiring the first-resolution image, the first-resolution image is preprocessed.

[0055] Specifically, for any pixel in the first-resolution image, the original RGB values ​​of each pixel in the red, green, and blue channels are normalized from the integer range of [0, 255] output by the standard output of a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) sensor to the continuous range of [-1, 1] in floating-point format through a linear mapping function or equivalent transformation. This transformation makes the mean of the input data zero and compresses the dynamic range to the sensitive range of model weight initialization and activation function (such as ReLU, GELU) response. After this step, the original pixel value 128 (medium gray) will be mapped to 0, the pixel value 255 (white) will be mapped to 1, and the pixel value 0 (black) will be mapped to -1. The image information that has completed the normalization process is temporarily stored in the DDR (double data rate) memory buffer of the domain controller and marked as pending processing, waiting for subsequent degradation suppression processing modules to read and calculate.

[0056] S202, Degradation suppression processing is performed on the first resolution image to obtain the intermediate image.

[0057] One possible implementation involves performing degradation suppression processing on a first-resolution image to obtain an intermediate image, including: extracting shallow features related to basic visual structures in the vehicular environment from the first-resolution image; extracting deep features related to vehicular imaging degradation from the first-resolution image based on the shallow features; suppressing the vehicular imaging degradation-related feature components in the deep features; and fusing the suppressed deep features with the shallow features to obtain the intermediate image.

[0058] Shallow features refer to the most basic and lowest-level visual element representations in the first-resolution image, corresponding to the local pattern information perceived by the early convolutional layers of a convolutional neural network. These features do not involve high-level semantic understanding or have category discrimination capabilities, but they accurately characterize low-level attributes such as pixel-level brightness changes, color continuity, and local gradient direction and magnitude in vehicular scenes. For example, large low-texture areas such as roads and skies appear as uniform regions with stable responses in shallow features; while high-texture structures such as vehicle outlines, lane line edges, and pedestrian boundaries appear as edge responses with larger gradient magnitudes.

[0059] One possible implementation involves inputting a preprocessed first-resolution image into one or more cascaded 3×3 convolutional layers. These convolutional layers do not include downsampling operations to maintain the spatial dimensions of the feature maps consistent with the input image. A set of multi-channel feature maps, i.e., shallow feature maps, is generated by performing sliding window convolution with learnable convolutional kernels on the input image. .in H represents the number of shallow feature channels (usually set to 64 or 128), and H and W represent the height and width of the input image, respectively.

[0060] Deep features are feature components that are highly correlated with vehicle imaging degradation. They are not simple texture or edge representations, but high-level abstract features with clear physical correspondences.

[0061] One possible implementation involves extracting deep features related to vehicle imaging degradation from a first-resolution image based on shallow features. This includes: identifying spatial blur features caused by vehicle motion and external interference features introduced by the vehicle environment from the first-resolution image based on the shallow features; and generating deep features based on the spatial blur features and external interference features.

[0062] Among them, spatial blur is caused by the displacement of the projection of the vehicle scene on the sensor focal plane during the exposure time due to vehicle motion (such as uniform motion, acceleration and deceleration, and high-frequency vibration transmitted by road bumps), which is manifested in the image as edge trailing and loss of details along the direction of motion.

[0063] External interference characteristics may include, but are not limited to: Poisson-Gaussian mixed noise accompanying sensor gain amplification under low illumination conditions, atmospheric light scattering and contrast attenuation caused by rain, snow, fog, and haze, and localized halo and shading effects caused by dirt or condensation on the lens surface. These interferences manifest in the image as randomly scattered pixel-level noise, regional contrast collapse, or texture masking at specific frequencies.

[0064] Specifically, this can be accomplished using a deep network consisting of multiple cascaded attention enhancement modules. The internal computation of each attention enhancement module strictly follows the following mathematical form:

[0065]

[0066]

[0067] Where X is the feature map input to the current attention enhancement module. LN is the layer normalization operation. This is the feature map after layer normalization. WMSA is a windowed multi-head self-attention mechanism. CAB is a channel attention block. α is a constant weight factor, for example, α is 0.1. This is the feature map after joint modulation by spatial self-attention and channel attention. MLP stands for Multilayer Perceptron. To Perform layer normalization again. Y is the final output feature map of the current attention enhancement module.

[0068] By shallow features The input consists of a deep network composed of K cascaded attention enhancement modules (K=8 in the preferred embodiment of this application), and the system refines the feature representation step by step. In the first-level attention enhancement module, the network focuses on capturing local, low-order degradation patterns (such as fine-grained noise and short-tailed blur); in the attention enhancement module, as the receptive field expands and the level of feature abstraction increases, the network gradually models global, high-order degradation coupling effects (such as the combined interference of motion blur and low-light noise). The final output feature map is the deep feature, denoted as Fdp∈RCdp×H×W.

[0069] A further step, fusing the suppressed deep features with the shallow features to obtain an intermediate image, may include: adding the suppressed deep features and shallow features element-wise to obtain fused features; and performing resolution upscaling and reconstruction on the fused features to generate the intermediate image.

[0070] Specifically, the channel attention weights learned in the attention enhancement module are used to attenuate the feature map channels or spatial regions corresponding to spatially blurred features and external interference features. Since the channel weights λ∈[0,1] output by the CAB module have been optimized during training to: λ approaches 0 for feature channels representing degradation patterns; and λ approaches 1 for feature channels representing useful structural information such as edges and contours, the deep features after suppression are the output of the CAN module itself. Fdp has already intrinsically completed the selective suppression of degradation features, eliminating the need for additional post-processing gating modules. This design achieves a "extraction-is-suppression" soft attention mechanism. While modeling degradation features, the network simultaneously attenuates their amplitude, resulting in higher computational efficiency and end-to-end differentiability.

[0071] Finally, the suppressed deep features and shallow features are added element-wise to obtain the fused feature: Ffusion = Fsh + Fdp. This residual connection method allows gradients to flow smoothly during forward and backward propagation, while enabling the network to focus on predicting the residual between shallow features and the ideal output during learning, i.e., what information still needs to be supplemented or corrected, rather than learning the entire mapping relationship from scratch, which significantly reduces the optimization difficulty.

[0072] After obtaining the fused feature (Ffusion), resolution upsampling and reconstruction are performed. Since the spatial dimensions H×W of the fused feature map are still consistent with the input first-resolution image and have not yet reached the target output resolution, an upsampling operation is necessary. Resolution upsampling and reconstruction are typically implemented using sub-pixel convolution (Pixel Shuffle) layers. The core idea of ​​sub-pixel convolution is to first expand the number of channels of the fused feature map using a convolutional layer. , where r is the upsampling factor (e.g., 2x or 4x). The target output image has 3 channels (3 for RGB images). Then, a periodic shuffling operation rearranges the pixels in the channel dimension to the spatial dimension, transforming the feature map size from H×W to rH×rW. This operation is learnable and does not introduce additional checkerboard artifacts. After subpixel convolutional layer processing, the feature map is mapped to the target resolution size and finally reassembled into a three-channel RGB image format, i.e., the intermediate image, denoted as . .

[0073] S203, based on the environmental conditions of the vehicle environment, iteratively denoise the intermediate image to obtain a second-resolution image of the vehicle environment. The resolution of the second-resolution image is higher than that of the first-resolution image.

[0074] One possible implementation involves iteratively denoising an intermediate image based on the environmental conditions of the in-vehicle environment to obtain a second-resolution image of the in-vehicle environment. This includes: acquiring the environmental conditions of the in-vehicle environment, which include multiple environmental elements and their spatial distribution relationships; converting the intermediate image into a corresponding latent spatial feature representation, which is obtained by mapping the intermediate image to a low-dimensional continuous feature space; iteratively denoising the latent spatial feature representation based on the multiple environmental elements and their spatial distribution relationships to obtain a sharp latent spatial feature representation corresponding to the intermediate image; and reconstructing the sharp latent spatial feature representation to obtain the second-resolution image.

[0075] Among them, latent spatial feature representation refers to the vector or tensor obtained by mapping an image in pixel space to a low-dimensional, continuous implicit feature space through a pre-trained encoder.

[0076] Specifically, the intermediate image The input is fed into a lightweight joint perception module. This module consists of two parallel sub-networks. The first is an object detection sub-network, employing a YOLO-based or similar architecture to perform real-time detection of environmental elements such as vehicles, pedestrians, and traffic signs in the image, outputting the bounding box coordinates and category label for each detected object. The second is a semantic segmentation sub-network, using a lightweight segmentation model (such as Fast-SCNN) to perform pixel-level classification of road structure elements such as lane lines, drivable areas, and guardrails. Subsequently, the detection and segmentation results are post-processed and fused to generate a compact structured representation: a list of environmental elements (each element includes category, bounding box location, and confidence score) and a spatial distribution graph (encoding the semantic category of each pixel and its adjacency relationships in the form of feature maps). This structured representation constitutes the environmental conditions required for this step, denoted as […]. .

[0077] Then the intermediate image The input image is fed into an encoder module E, which is a pre-trained autoencoder. This encoder progressively compresses the input image through a series of convolutional layers and downsampling operations, ultimately outputting a low-resolution latent feature map. , where c is the number of potential channels (usually 4 or 8), and h and w are the spatial dimensions of the potential feature map (usually 1 / 8 or 1 / 16 of the input image size).

[0078] Finally, based on multiple environmental elements and the spatial distribution relationship between them, the latent spatial feature representation is iteratively denoised to obtain a clear latent spatial feature representation corresponding to the intermediate image.

[0079] Furthermore, based on multiple environmental elements and their spatial distribution relationships, iterative denoising is performed on the latent spatial feature representation to obtain a clear latent spatial feature representation corresponding to the intermediate image. This includes: adding preset noise to the latent spatial feature representation to obtain an initial noise feature representation; generating constraints for iterative denoising based on multiple environmental elements and their spatial distribution relationships; and performing multiple rounds of denoising operations on the initial noise feature representation according to the constraints to obtain a clear latent spatial feature representation. Each round of denoising involves adjusting the noise features input in the current round based on the constraints, predicting and eliminating the noise components of the noise features input in the current round.

[0080] Specifically, in the forward pass of the diffusion model, the distribution of real data... By gradually adding Gaussian noise, it is destroyed into a pure noise distribution. During the inference phase, the system performs this process in reverse, therefore a starting point is required. The initial noise feature representation is a noise tensor with the same dimensions (c×h×w) as the latent feature map z, randomly sampled from a standard Gaussian distribution N(0,I). This noise tensor does not contain any information from the input image and is the starting point for the entire iterative reconstruction process.

[0081] Then, to ensure the generated results are semantically reasonable within the context of vehicular traffic scenarios, environmental conditions need to be considered. This is transformed into constraints that the diffusion model can understand and follow. First, the detection and segmentation results are embedded and encoded: the bounding box coordinates and class label of each detected object are mapped to a feature vector; the semantic segmentation map is then encoded into a spatial feature map through a small convolutional network. These encoded information collectively constitute the conditional tensor. In each layer of the denoising U-Net, the latent feature map generates a query matrix through a linear transformation, while the conditional tensor... A key and value matrix is ​​generated through a linear transformation. Attention weights are calculated by taking the dot product of the query and the key, and then the attention weights are summed with the value matrix in a weighted manner to inject semantic constraints into each spatial location of the latent features.

[0082] The loop variable t is then decremented from the preset maximum time step T to 1, and iterative denoising is performed sequentially. In each iteration, the potential noise features of the current time step are... Time step encoding t and conditional tensor Input to the denoising U-Net network Predict the noise components contained in the current noise feature representation, i.e. Subsequently, a denoising sampler (a DDIM sampler is used in the preferred embodiment of this application) is employed to calculate according to the following update rules. :

[0083] in, For predefined noise scheduling parameters, For sampling randomness control parameters, Optional standard Gaussian noise. The output of step t=1 at the end of the loop. This is a clear representation of latent spatial features.

[0084] Finally, the noise characteristics of the current round input are adjusted by applying constraints, and the noise components of the noise characteristics of the current round input are predicted and eliminated.

[0085] Furthermore, the gradient magnitude and local variance of the first-resolution image are calculated. Based on the gradient magnitude and local variance, the complexity index of the first-resolution image is determined. According to the complexity index and a preset complexity range, the number of iterations for iterative denoising of the latent spatial feature representation is determined.

[0086] Specifically, for the first resolution image of the original input Perform Sobel convolution to calculate the gradient magnitude for each pixel. The global mean and standard deviation of the gradient magnitude are calculated. Simultaneously, the image is divided into several non-overlapping 16×16 pixel blocks, and the variance of the pixel values ​​in each block is calculated. The mean of these variances is then used as the representative value of the local variance.

[0087] Secondly, based on the gradient magnitude and local variance, the complexity index of the first-resolution image is determined. The complexity index C is calculated using the following empirical formula:

[0088] in, The mean of the gradient magnitude. The standard deviation of the gradient magnitude. This represents the mean of the local variances. , as well as These are normalized reference values, obtained from statistics of the calibration dataset; , as well as These are weighting coefficients (set to 0.4, 0.3, and 0.3 respectively in the preferred embodiments of this application).

[0089] Finally, based on the complexity index and the preset complexity range, the number of iterations for iterative denoising of the latent space feature representation is determined. The system presets two complexity thresholds. and Divide the complexity into three intervals: One possible implementation, if (For example, if the scene is a flat highway with little traffic and clear weather), it is determined to be a low-complexity scene, and the scheduler instructs the iterative denoising process to only execute [the following steps]. This setting can minimize inference latency while maintaining basic image quality.

[0090] Another possible implementation, if (Such as ordinary urban roads with moderate traffic density), classified as a medium-complexity scenario, and executed. This setting is the default mode, balancing image quality and efficiency.

[0091] Another possible implementation, if (Such as dense traffic intersections, busy urban areas with pedestrians, and low-visibility scenes in inclement weather), these are identified as high-complexity scenes and execution is performed. This setting enables a full iterative process to fully unleash the detailed reconstruction capabilities of the diffusion model.

[0092] Finally, the clear latent spatial feature representation is reconstructed to obtain a second-resolution image.

[0093] The specific implementation method of reconstruction is as follows: representing the clear latent spatial features of the iterative denoising output. The input is fed into the decoder module D of the pre-trained autoencoder. The decoder gradually restores the low-resolution, high-channel latent feature map into a high-resolution, three-channel RGB image, i.e., the second-resolution image, through a series of upsampling layers and convolutional layers. This image is the final high-definition output of this application, which can be directly pushed to the electronic rearview mirror display screen.

[0094] S204 displays a second-resolution image through the vehicle's human-machine interface.

[0095] Specifically, the image processing unit retrieves the generated second-resolution image from the computational memory buffer and transmits it to the display terminal via a standardized digital video interface. This transmission process must meet the stringent requirements of the automotive environment: low latency (end-to-end display latency is typically required to be less than 60 milliseconds), high bandwidth (supporting real-time transmission at 1080p and above resolutions and a refresh rate of 60 frames per second), and high reliability (resistance to electromagnetic interference and temperature changes). Common physical interfaces include DisplayPort, eDP (embedded DisplayPort), HDMI, or automotive-specific LVDS (low-voltage differential signaling) serial links. After receiving the image data, the display terminal uses its internal timing control chip (TCON) to drive the pixel array of the LCD or OLED panel, and through backlight modulation and liquid crystal deflection, restores the digital signal to a visible light image.

[0096] The above primarily describes the solutions provided in the embodiments of this application from the perspective of the working principle of the device. It is understood that, in order to achieve the above functions, the image processing device includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the algorithm steps of the examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] This application embodiment can divide the image processing device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module.

[0098] It should be noted that the module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. When dividing functional modules according to their respective functions, Figure 3 A schematic diagram illustrating a possible composition of the image processing apparatus involved in the above and embodiments is shown. For example... Figure 3 As shown, the image processing device 300 may include: an acquisition module 301, a processing module 302, a noise reduction module 303, and a display module 304.

[0099] The acquisition module 301 is used to support the image processing device 300 in performing its functions. Figure 2 S201 in the illustrated image processing method.

[0100] Processing module 302 is used to support the image processing device 300 in performing operations. Figure 2 S202 in the illustrated image processing method.

[0101] Noise reduction module 303 is used to support the image processing device 300 in performing [its functions]. Figure 2 S203 in the illustrated image processing method.

[0102] Display module 304 is used to support the execution of image processing device 300. Figure 2 S204 in the illustrated image processing method.

[0103] One possible implementation involves a processing module that performs degradation suppression processing on a first-resolution image to obtain an intermediate image. Specifically, this module extracts shallow features related to the basic visual structures in the vehicular environment from the first-resolution image. Based on these shallow features, it extracts deep features related to vehicular imaging degradation from the first-resolution image. It then suppresses the vehicular imaging degradation-related feature components within the deep features. Finally, it fuses the suppressed deep features with the shallow features to obtain the intermediate image.

[0104] One possible implementation involves a processing module that, when extracting deep features related to vehicle imaging degradation from a first-resolution image based on shallow features, specifically: identifying spatial blur features caused by vehicle motion and external interference features introduced by the vehicle environment from the first-resolution image based on the shallow features; and generating deep features based on the spatial blur features and external interference features.

[0105] One possible implementation involves a processing module that fuses suppressed deep features with shallow features to obtain an intermediate image. Specifically, this process involves: adding the suppressed deep features and shallow features element-wise to obtain fused features; and then performing resolution upscaling and reconstruction on the fused features to generate the intermediate image.

[0106] One possible implementation involves a denoising module used to iteratively denoise the intermediate image based on the environmental conditions of the vehicle environment to obtain a second-resolution image of the vehicle environment. Specifically, this module is used to: acquire the environmental conditions of the vehicle environment, including multiple environmental elements and their spatial distribution relationships; convert the intermediate image into a corresponding latent spatial feature representation, which is obtained by mapping the intermediate image to a low-dimensional continuous feature space; iteratively denoise the latent spatial feature representation based on the multiple environmental elements and their spatial distribution relationships to obtain a clear latent spatial feature representation corresponding to the intermediate image; and reconstruct the clear latent spatial feature representation to obtain the second-resolution image.

[0107] One possible implementation involves a denoising module that iteratively denoises the latent spatial feature representation based on multiple environmental elements and their spatial distribution relationships to obtain a clear latent spatial feature representation corresponding to the intermediate image. Specifically, this module: adds preset noise to the latent spatial feature representation to obtain an initial noise feature representation; generates constraints for iterative denoising based on the multiple environmental elements and their spatial distribution relationships; and performs multiple rounds of denoising operations on the initial noise feature representation according to the constraints to obtain a clear latent spatial feature representation. Each round of denoising involves adjusting the noise features input in the current round based on the constraints, predicting and eliminating the noise components of the noise features input in the current round.

[0108] In one possible implementation, the image processing apparatus provided in this application embodiment is further configured to: calculate the gradient magnitude and local variance of a first-resolution image; determine a complexity index of the first-resolution image based on the gradient magnitude and local variance; and determine the number of iterations for iterative denoising of the latent spatial feature representation according to the complexity index and a preset complexity range.

[0109] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0110] The image processing apparatus 300 provided in this application embodiment is used to perform the above-described... Figure 2 The image processing method shown can achieve the same effect as the image processing method described above.

[0111] This application also provides an image processing device that can execute the image processing methods and related steps described in the above method embodiments.

[0112] This application also provides a computer-readable storage medium storing instructions thereon, which, when executed, perform the image processing method and related steps described in the above method embodiments.

[0113] This application also provides a computer program product that, when run on a computer, causes the computer to execute the image processing method and related steps described in the above method embodiments.

[0114] In some embodiments, the methods shown in this application can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0115] This application also provides an image processing system 100, such as... Figure 4As shown, the image processing system 100 includes at least one processor 401 and at least one interface circuit 402.

[0116] As an example, when the image processing system 100 includes a processor and an interface circuit, the processor can be... Figure 4 The processor 401 shown in the solid box (or the processor 401 shown in the dashed box) can be an interface circuit. Figure 4 The interface circuit 402 is shown in the solid box (or the dashed box). When the image processing system 100 includes two processors and two interface circuits, the two processors include... Figure 4 The processor 401 shown in the solid box and the processor 401 shown in the dashed box, these two interface circuits include Figure 4 Interface circuit 402 is shown in both solid and dashed boxes. No limitations are imposed on this.

[0117] Processor 401 and interface circuit 402 can be interconnected via a line. For example, interface circuit 402 can be used to receive signals. Alternatively, interface circuit 402 can be used to send signals to other devices (e.g., processor 401). For instance, interface circuit 402 can read computer instructions stored in memory and send those instructions to processor 401. Processor 401 executes the instructions and, in conjunction with input / output devices, implements the various steps in the above embodiments, such as implementing... Figure 2 The illustrated method embodiments represent the various steps performed. Of course, this image processing system may also include other discrete components, and this application embodiment does not specifically limit this.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0119] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to it, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: During vehicle operation, first-resolution images of the in-vehicle environment are captured. The first resolution image is subjected to degradation suppression processing to obtain an intermediate image; Based on the environmental conditions of the vehicle environment, the intermediate image is iteratively denoised to obtain a second resolution image of the vehicle environment; the resolution of the second resolution image is higher than that of the first resolution image. The second resolution image is displayed through the vehicle's human-machine interface.

2. The method according to claim 1, characterized in that, The degradation suppression processing of the first resolution image to obtain the intermediate image includes: Extract shallow features from the first resolution image that are related to the basic visual structure in the vehicle environment; Based on the shallow features, deep features related to vehicle imaging degradation are extracted from the first resolution image; Suppress the vehicle imaging degradation-related feature components in the deep features; The suppressed deep features are fused with the shallow features to obtain the intermediate image.

3. The method according to claim 2, characterized in that, The step of extracting deep features related to vehicle imaging degradation from the first resolution image based on the shallow features includes: Based on the shallow features, spatial blur features caused by vehicle motion and external interference features introduced by the vehicle environment are identified from the first resolution image. The deep features are generated based on the spatial fuzziness features and the external interference features.

4. The method according to claim 2, characterized in that, The process of fusing the suppressed deep features with the shallow features to obtain the intermediate image includes: The suppressed deep features are added element-wise to the shallow features to obtain the fused features; The fused features are then subjected to resolution upscaling and reconstruction to generate the intermediate image.

5. The method according to claim 1, characterized in that, The iterative noise reduction of the intermediate image based on the environmental conditions of the vehicle environment to obtain a second resolution image of the vehicle environment includes: The environmental conditions of the vehicle environment are obtained; the environmental conditions include multiple environmental elements in the vehicle environment and the spatial distribution relationship between the various environmental elements. The intermediate image is converted into a corresponding latent spatial feature representation; the latent spatial feature representation is obtained by mapping the intermediate image to a low-dimensional continuous feature space. Based on the multiple environmental elements and the spatial distribution relationship between the environmental elements, the latent spatial feature representation is iteratively denoised to obtain a clear latent spatial feature representation corresponding to the intermediate image; The clear latent spatial feature representation is reconstructed to obtain the second resolution image.

6. The method according to claim 5, characterized in that, The step of iteratively denoising the latent spatial feature representation based on multiple environmental elements and the spatial distribution relationship between the environmental elements to obtain a clear latent spatial feature representation corresponding to the intermediate image includes: Add preset noise to the latent spatial feature representation to obtain an initial noise feature representation; The constraints for the iterative denoising are generated based on the multiple environmental elements and the spatial distribution relationship between them. According to the constraints, the initial noise feature representation is subjected to multiple rounds of denoising operations to obtain the clear latent space feature representation; wherein, each round of the denoising operation is to adjust the noise features input in the current round according to the constraints, and predict and eliminate the noise components of the noise features input in the current round.

7. The method according to claim 5 or 6, characterized in that, The method further includes: Calculate the gradient magnitude and local variance of the first resolution image; Based on the gradient magnitude and the local variance, the complexity index of the first resolution image is determined; Based on the complexity index and the preset complexity range, the number of iterations for iterative denoising of the latent spatial feature representation is determined.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire first-resolution images of the in-vehicle environment while the vehicle is in motion; The processing module is used to perform degradation suppression processing on the first resolution image to obtain an intermediate image; A noise reduction module is used to iteratively reduce the noise of the intermediate image based on the environmental conditions of the vehicle environment to obtain a second resolution image of the vehicle environment; the resolution of the second resolution image is higher than that of the first resolution image; The display module is used to display the second resolution image through the human-machine interface of the vehicle.

9. An image processing device, characterized in that, The image processing device includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image processing method according to any one of claims 1 to 7.

Images