Intelligent vehicle vision perception optimization method and system based on super-resolution reconstruction technology

Abstract

The present application belongs to the technical field of vehicle visual perception optimization, and particularly relates to an intelligent vehicle visual perception optimization method and system based on super-resolution reconstruction technology. The method comprises the following steps: obtaining an original image collected by a vehicle camera; inputting the original image into a super-resolution reconstruction neural network model, using a plurality of serially connected residual dense blocks to perform deep feature extraction, connecting a plurality of convolution layers using a dense connection mechanism in each residual dense block, and using a channel attention mechanism at the end of each residual dense block to generate a reconstructed image; and performing environment perception based on the reconstructed image to identify traffic participants, lane lines and traffic signs in the image. The present application restores key details in the image through super-resolution reconstruction technology, so that the target detection and segmentation model can more accurately identify small targets at a long distance and more clearly read the specific content of traffic signs, significantly reducing the false detection and missed detection rates and improving the overall perception accuracy.

Description

Technical Field

[0001] This invention belongs to the field of vehicle visual perception optimization technology, and particularly relates to a method and system for optimizing the visual perception of intelligent vehicles based on super-resolution reconstruction technology. Background Technology

[0002] Due to limitations in cost, power consumption, and data transmission bandwidth, some vehicles still widely use low-resolution cameras. These cameras suffer from severe loss of image detail in poor lighting, weather conditions, or long-distance scenarios, leading to problems such as unclear recognition of distant speed limit signs, false or missed detection of small obstacles, and blurred pedestrian outlines.

[0003] Traditional image interpolation and upscaling methods only perform interpolation calculations between existing pixels, failing to reconstruct high-frequency details lost during image degradation and thus offering limited benefits for improving perception accuracy. While general super-resolution algorithms can improve image resolution, their design goal is primarily to enhance visual perception. In pursuit of smoothness, they may suppress edge and texture features, even introducing artifacts, failing to achieve synergistic optimization with downstream perception tasks such as object detection and semantic segmentation. These factors result in low-resolution camera vision systems having short effective perception distances and low recognition accuracy, compressing the decision-making and planning window of intelligent vehicles. Summary of the Invention

[0004] To overcome the shortcomings of the existing technologies, this invention provides a method and system for optimizing visual perception of intelligent vehicles based on super-resolution reconstruction technology. Through the designed super-resolution reconstruction neural network model, lost high-frequency detail information can be recovered from low-resolution images, enabling subsequent target detection and semantic segmentation models to more accurately identify small targets at a distance and to read the specific content of traffic signs more clearly. This significantly reduces the false detection and false negative rates and improves the overall perception accuracy.

[0005] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a method for optimizing visual perception of intelligent vehicles based on super-resolution reconstruction technology.

[0006] The intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology includes the following steps: Acquire raw images captured by the vehicle's camera; The original image is input into a super-resolution reconstruction neural network model, and deep feature extraction is performed using multiple cascaded residual dense blocks. In each residual dense block, multiple convolutional layers are connected using a dense connection mechanism, and a channel attention mechanism is used at the end of each residual dense block to generate a reconstructed image. Based on the reconstructed image, environmental perception is performed to identify traffic participants, lane lines, and traffic signs in the image, thereby optimizing the vehicle's environmental perception.

[0007] As a preferred technical solution, before inputting the original image into the super-resolution reconstruction neural network model for deep feature extraction, a shallow feature extraction layer is first used to extract shallow features from the original image. The shallow feature extraction layer is a convolutional layer, and the shallow features include edge, corner, and texture features.

[0008] As a preferred technical solution, when performing deep feature extraction, the output of the previous residual dense block is used as the input of the next residual dense block to achieve multi-level feature extraction.

[0009] As a preferred technical solution, multiple convolutional layers are connected in each residual dense block using a dense connection mechanism, specifically: The output of each convolutional layer is not only passed to the next layer, but also passed to the input of all subsequent convolutional layers through cascading connections, thus enabling the reuse of features from each layer. Except for the last convolutional layer, a ReLU activation function is applied after each convolutional layer; The last convolutional layer does not use the ReLU activation function and directly outputs the feature map.

[0010] As a preferred technical solution, a channel attention mechanism is used at the end of each residual dense block, specifically: First, the feature map F_conv output by the last convolutional layer in the residual dense block is compressed into a channel statistics vector through a global average pooling layer. The channel statistical vectors are nonlinearly transformed through two fully connected layers to learn the importance weights of each channel and obtain the weight coefficients. The weight coefficients are multiplied channel by channel with the original feature map F_conv to complete the feature recalibration and obtain the weighted feature map. This also includes introducing local residual connections in each dense residual block, specifically: The weighted feature map output by the channel attention mechanism in the residual dense block is fused with the input features in the current residual dense block to obtain the output features of the current residual dense block; During weighted fusion, the weights corresponding to the weighted feature maps are adaptive scaling coefficients, which are learned from the input features of the residual dense blocks through a light quantum network. The local residual connection can be expressed by the following formula: F_out = F_in + α × F_weighted; Where F_out is the output feature of the current residual dense block; F_in is the input feature of the residual dense block; α is the adaptive scaling factor; and F_weighted is the weighted feature map output by the channel attention mechanism.

[0011] As a preferred technical solution, after performing deep feature extraction using multiple cascaded residual dense blocks, the method further includes: Deep features are processed at multiple scales to extract and fuse feature information at different scales to obtain a fused feature map. Using global residual connections, shallow features are weighted and fused with the fused feature map to obtain the final feature map; The final feature map is upsampled to reconstruct a high-resolution image from the low-resolution feature map, resulting in the reconstructed image. or, The reconstructed image is input into the object detection neural network model and the semantic segmentation neural network model. Forward propagation calculations are performed on the object detection neural network model and the semantic segmentation neural network model to identify traffic participants, lane lines and traffic signs in the image. or, After optimizing the vehicle's environmental perception, the following is also included: Based on the results of environmental perception, control signals are generated for vehicle control decisions, controlling the vehicle's throttle, braking, or steering mechanisms.

[0012] As a preferred technical solution, the loss function of the super-resolution reconstruction neural network model includes: Pixel-level L1 loss : ; in, SR To reconstruct the image, HR For true high-resolution images, n Total number of pixels; Represents pixel coordinates; Perceived loss : ; in, Indicates a feature extractor; This represents the features extracted by the feature extractor from the reconstructed image; This represents the features extracted by the feature extractor from the real high-resolution image; C, H, and W represent the number of channels, height, and width of the feature map, respectively. Task-driven loss : ; in, Indicates boundary loss, Indicates classification loss; Total loss function for: ; in, , and All of these represent weighting coefficients.

[0013] The second aspect of the present invention provides an intelligent vehicle visual perception optimization system based on super-resolution reconstruction technology.

[0014] The intelligent vehicle visual perception optimization system based on super-resolution reconstruction technology includes: The image acquisition module is configured to acquire raw images captured by the vehicle's camera. The image reconstruction module is configured to: input the original image into the super-resolution reconstruction neural network model, perform deep feature extraction using multiple cascaded residual dense blocks, connect multiple convolutional layers in each residual dense block using a dense connection mechanism, and use a channel attention mechanism at the end of each residual dense block to generate a reconstructed image; The environmental perception module is configured to: perform environmental perception based on the reconstructed image, identify traffic participants, lane lines and traffic signs in the image, and optimize the vehicle's environmental perception.

[0015] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in the first aspect of the present invention.

[0016] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in the first aspect of the present invention.

[0017] The above one or more technical solutions have the following beneficial effects: This invention provides a method and system for optimizing visual perception in intelligent vehicles based on super-resolution reconstruction technology. By employing a super-resolution reconstruction neural network model specifically optimized for intelligent vehicle perception tasks, lost high-frequency detail information is recovered from low-resolution images. This enables subsequent target detection and semantic segmentation models to more accurately identify small targets at a distance and to more clearly read the specific content of traffic signs, thereby significantly reducing false detection and false negative rates and improving overall perception accuracy. Multiple cascaded residual dense blocks in the deep feature extraction module fully reuse features from each layer using a dense connection mechanism, enhancing the network's feature learning ability. The channel attention mechanism adaptively adjusts the weight coefficients of each channel, strengthening the focus on key features of the perception task. The multi-scale feature fusion module processes feature maps of different scales, enhancing the model's adaptability to targets of different sizes. The synergistic effect of these mechanisms results in reconstructed images with richer detail information, improving their effective perception resolution. This effectively extends the visual range of the intelligent vehicle under the same hardware conditions, providing the decision-making system with longer reaction time and thus extending the effective perception distance.

[0018] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 This is a schematic diagram of the execution flow of the intelligent vehicle vision perception system of the present invention.

[0021] Figure 2 A schematic diagram of the improved EDSR overall network architecture.

[0022] Figure 3 A schematic diagram of the improved residual dense block architecture.

[0023] Figure 4 A detailed structural diagram of the channel attention mechanism.

[0024] Figure 5 This is a schematic diagram of the structural framework of the multi-scale feature fusion module. Detailed Implementation

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0027] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] Terminology Explanation To facilitate understanding of the technical solution of this invention, the core technical terms involved in this invention are first defined and explained: Low-resolution (LR) images refer to raw images captured by vehicle-mounted cameras. Due to hardware limitations, environmental factors, or transmission compression, these images are below ideal levels in terms of pixel count, detail information, and texture clarity. They are typically characterized by blurry images, unclear edges, and difficulty in identifying small targets.

[0029] High-resolution (HR) images refer to high-quality images used as ground truth during the training phase, or images captured by high-resolution cameras under ideal conditions. These images contain rich detail, clear texture features, and accurate edge contours, providing sufficient visual information for downstream perception tasks.

[0030] Super-resolution (SR) images refer to high-resolution images generated by processing low-resolution images through a super-resolution reconstruction neural network model. These images are significantly superior to the input low-resolution images in terms of pixel count, detail richness, and texture clarity, and are closer to real high-resolution images.

[0031] Super-resolution reconstruction (SR) refers to the technique of using deep learning neural network models to learn and recover lost high-frequency detail information from low-resolution images, generating high-resolution images. This process is not simple pixel interpolation, but rather reconstructs high-quality images that better reflect the real-world scene by learning the inherent patterns and feature representations of the image.

[0032] Enhanced Residual Dense Block (ERDB): This refers to the improved deep feature extraction unit of this invention. This unit contains multiple convolutional layers and adopts a dense connection mechanism so that the output of each convolutional layer is connected to the input of all subsequent convolutional layers. At the same time, local residual connections and channel attention mechanisms are introduced to fully reuse feature information and enhance the attention to key features.

[0033] Channel Attention (CA) is an adaptive feature weighting mechanism that uses global average pooling to obtain statistical information for each channel, learns the importance weights of each channel through a fully connected layer, generates attention weight coefficients between 0 and 1 through a sigmoid activation function, and finally multiplies the original feature map channel by channel to enhance attention to key feature channels of the perception task and suppress irrelevant feature channels.

[0034] Perceptual Loss: This refers to the loss function calculated based on high-level semantic features extracted by a pre-trained convolutional neural network (such as VGG-19). Instead of directly comparing pixel value differences, this loss function compares semantic differences in the feature space, which can guide the model to generate reconstruction results that are visually closer to the real image and avoid over-smoothing and loss of details.

[0035] Task-driven loss: refers to a loss function designed for specific perception tasks such as downstream object detection or semantic segmentation. This loss function synergistically optimizes super-resolution reconstruction and perception tasks, so that the reconstructed image not only has high visual quality, but also helps to improve the accuracy of subsequent perception tasks.

[0036] Sub-pixel convolution is an efficient upsampling technique that generates multi-channel features by performing convolution operations on low-resolution feature maps, and then rearranges these channels into a high-resolution feature map. Compared to traditional interpolation upsampling and transposed convolution, sub-pixel convolution can better preserve image details and reduce computation.

[0037] Dense Connection: This refers to a connection within a dense residual block where the output of each convolutional layer is not only passed to the next layer, but also to the input of all subsequent convolutional layers through cascading connections. This connection method can fully reuse the features of each layer, enhance the fluidity of features, and improve the feature learning ability of the network.

[0038] Adaptive Residual Scaling: This refers to a mechanism that dynamically adjusts the weights of residual connections based on the content of the input image. This mechanism learns a scaling factor α from the input features through a light quantum network. For images with rich details, the weights of residual connections are enhanced, while for smooth regions, the weights of residual connections are reduced, thereby improving the reconstruction quality and the network's adaptability.

[0039] Visual perception refers to the process by which intelligent vehicles collect environmental images through onboard cameras and use computer vision algorithms to identify key information such as traffic participants, lane lines, and traffic signs in the images. It is a core component of the intelligent vehicle's environmental perception system.

[0040] Semantic segmentation refers to the technique of classifying and labeling each pixel in an image to divide the image into regions with different semantic categories. In intelligent vehicle applications, it is mainly used to identify areas such as lane lines, drivable areas, sidewalks, and obstacles, providing a basis for path planning and decision control.

[0041] Technical background and problem definition of the present invention: In intelligent vehicle vision perception systems, existing technical solutions mainly include three approaches: First, directly replacing cameras with high-resolution ones to improve perception performance will significantly increase the hardware cost of the intelligent vehicle system and the subsequent data processing load.

[0042] Secondly, traditional image interpolation upscaling methods (such as bicubic interpolation), which are software alternatives, are limited to interpolation calculations between existing pixels and cannot reconstruct the high-frequency detail information lost during image degradation. Therefore, they have a very limited effect on improving the accuracy of environmental perception.

[0043] Furthermore, even if existing general super-resolution algorithms are used, their design purpose is mainly to improve the subjective visual perception of images, rather than to serve specific perception tasks such as object detection and semantic segmentation. In pursuit of visual smoothness, these algorithms may suppress edge and texture features that are crucial to perception, or even introduce unhelpful artifacts, resulting in their reconstruction results failing to achieve synergistic optimization with downstream intelligent vehicle perception tasks.

[0044] The above factors together result in a short effective perception distance and low recognition accuracy for vision systems based on low-resolution cameras, which compresses the decision-making and planning window of intelligent vehicles and brings potential safety risks.

[0045] Based on the above analysis of technical shortcomings, the logical starting point of this invention is to achieve a leap from low-resolution images to high-quality perception results through algorithmic innovation without significantly increasing hardware costs. Specifically, a super-resolution reconstruction neural network model optimized for intelligent vehicle perception tasks is designed. This model can not only recover lost high-frequency detail information in images, but more importantly, it can form a synergistic optimization with downstream perception tasks such as object detection and semantic segmentation, ensuring that the reconstructed image not only has high visual quality, but also contributes to improving the accuracy of subsequent perception tasks.

[0046] Overview of the technical implementation approach: The intelligent vehicle visual perception method based on super-resolution reconstruction provided by this invention can be summarized as follows: constructing a complete closed-loop system from image acquisition to vehicle control. This system consists of four core steps: image acquisition, super-resolution reconstruction, environmental perception, and decision control.

[0047] Image acquisition: Raw environmental images are captured by one or more low-resolution cameras deployed in front of, to the sides of, or behind the vehicle, and necessary preprocessing is performed, including automatic white balance, color space conversion (RGB to YUV, super-resolution processing only for the Y luminance channel), image normalization, etc., to improve the consistency of model input.

[0048] Super-resolution reconstruction: This is the core innovation of this invention, based on a specially optimized Perceptual-EDSR (Perceptual Enhanced Super-Resolution Reconstruction) neural network model. This model is based on an improved EDSR network architecture, and its core innovations include: First, an improved residual dense block (ERDB) is introduced, in which each block contains multiple convolutional layers and adopts a dense connection mechanism, so that the output of each layer is connected to the input of all subsequent layers, making full use of feature information; Second, a channel attention mechanism (CA) is added after each residual dense block. Attention weights are generated through global average pooling, fully connected layers and sigmoid activation function to adaptively adjust the importance of each channel and enhance the attention to key features of the perception task. Third, a multi-scale feature fusion module is adopted to downsample and upsample the feature map at different scales, extract multi-scale feature information and fuse it through 1×1 convolution, thereby enhancing the model's adaptability to targets of different sizes. Fourth, an adaptive residual scaling mechanism is introduced to dynamically adjust the weights of residual connections based on the content of the input image; Fifth, a multi-task loss function specifically designed for perception tasks is proposed. This loss function includes a weighted sum of pixel-level L1 loss, perception loss based on VGG-19 network features, and task-driven loss for downstream object detection or semantic segmentation tasks, enabling super-resolution reconstruction and downstream perception tasks to achieve synergistic optimization.

[0049] Environmental Awareness: Based on high-quality reconstructed images from the super-resolution reconstruction module, the data is input into a pre-trained object detection model (e.g., based on YOLOv3, YOLOv5, YOLOv8, Faster R-CNN, or Mask R-CNN architecture) and / or a semantic segmentation model (e.g., based on DeepLabv3+, PSPNet, or U-Net architecture). It accurately identifies and locates various traffic participants (vehicles, pedestrians, cyclists), lane lines, drivable areas, and traffic signs in the image, and outputs their bounding boxes, class labels, confidence scores, and pixel-level segmentation masks.

[0050] Decision Control: Based on the output of the environmental perception module and combined with the vehicle's own state information (such as vehicle speed, yaw rate, acceleration, etc.), real-time decision analysis is performed, and corresponding vehicle control signals are generated. For example, when a pedestrian or obstacle is detected ahead and a collision risk is determined, a pre-braking signal is generated; when the specific number of a speed limit sign in the distance is identified, a cruise control signal is generated. The final control signal is sent to the vehicle's drive-by-wire actuators (throttle actuator, brake actuator, or steering actuator) via the CAN bus or in-vehicle Ethernet, completing the closed loop from perception to control.

[0051] Example 1 This embodiment provides a visual perception method for intelligent vehicles based on super-resolution reconstruction, which includes the following steps S101 to S104.

[0052] S101, Image Acquisition Steps.

[0053] This step involves acquiring raw images of the surrounding environment using a low-resolution onboard camera and performing necessary preprocessing.

[0054] Camera Deployment Location: In this embodiment, the intelligent vehicle deploys a low-resolution camera in the forward position, mounted on the roof behind the windshield, with a 90° field of view, covering the road and traffic environment within a range of approximately 100 meters in front of the vehicle. In other embodiments, additional low-resolution cameras can be deployed in the side positions (below the left and right rearview mirrors) and the rear position (above the rear windshield) of the vehicle to form a 360° surround view system.

[0055] In real-world in-vehicle scenarios, raw images captured by low-resolution cameras often degrade due to the following factors: First, motion blur, caused by the bumps in the vehicle and the relative movement of the camera, resulting in blurred images, especially at high speeds or on uneven roads; Second, noise interference, including Gaussian noise (caused by sensor thermal noise) and Poisson noise (caused by the statistical characteristics of photon counting), which is particularly noticeable under low-light conditions; Third, loss of detail, due to camera resolution limitations and the diffraction effect of the optical system, detailed information of distant small targets is severely lost, such as the license plate number of a distant vehicle or the specific numbers of a speed limit sign, which are difficult to identify; Fourth, color cast and uneven exposure, caused by changes in ambient lighting (such as backlighting, tunnel entrances and exits, etc.) leading to color deviations and local overexposure or underexposure.

[0056] Preprocessing steps: To improve the consistency of model input and reduce computational load, this embodiment performs the following preprocessing operations on the original image: The first step, automatic white balance: The gray world hypothesis algorithm is used to correct the white balance of the image to eliminate the color cast under different lighting conditions.

[0057] The second step is color space conversion: converting the RGB image to the YUV color space.

[0058] The third step is image normalization: normalize the pixel values ​​of the Y luminance channel from the range of [0, 255] to the range of [0, 1].

[0059] After the above preprocessing, a normalized Y luminance channel image is obtained, which serves as the input to the super-resolution reconstruction module. The U and V chrominance channels retain their original resolution. After the super-resolution reconstruction is completed, they are upsampled to the same resolution as the reconstructed Y channel through bicubic interpolation. Finally, the Y, U, and V channels are merged and converted back to the RGB color space to obtain the final high-resolution reconstructed image.

[0060] The technical benefits achieved in this step are: the preprocessed image has better consistency and lower noise levels, which is beneficial for subsequent super-resolution reconstruction. By performing super-resolution processing only on the Y luminance channel, the computational load is reduced by approximately 60% while maintaining color fidelity.

[0061] S102 super-resolution reconstruction steps.

[0062] This step corresponds to Figure 2 , Figure 3 , Figure 4 , Figure 5 This is the core innovative step of the present invention. This step inputs the preprocessed low-resolution Y-luminance channel image into the Perceptual-EDSR super-resolution reconstruction neural network model to generate a high-resolution reconstructed image.

[0063] The complete architecture of the Perceptual-EDSR model is as follows: Figure 2 As shown, this model employs a hierarchical residual learning architecture. The network input is a low-resolution image (LR) with dimensions H×W×1, where H and W are the height and width of the image, respectively, and 1 represents a single-channel Y-luminance image. The network output is a super-resolution reconstructed image (SR) with dimensions 4H×4W×1, representing a 4-fold increase in resolution. The model mainly consists of the following components: The first part, the shallow feature extraction layer, uses a 3×3 convolutional layer to extract shallow features from the input low-resolution image. The convolutional layer has 64 kernels, a stride of 1, and padding of 1 (i.e., same padding, keeping the feature map size unchanged). The activation function is ReLU.

[0064] The shallow feature extraction layer extracts low-level features of the image, such as edges, corners, and textures. These features provide the foundation for subsequent deep feature extraction. The output feature map of the shallow feature extraction layer has a size of H×W×64.

[0065] The second part, the deep feature extraction module, is the core component of the Perceptual-EDSR model. It consists of 16 improved residual dense blocks (ERDBs) concatenated together. The internal structure of each ERDB is as follows: Figure 3 As shown, it contains eight 3×3 convolutional layers (each with 64 output channels) and employs a dense connection mechanism. The output of each convolutional layer is not only passed to the next layer, but also passed to the input of all subsequent convolutional layers through concatenation.

[0066] Specifically, the input to the i-th convolutional layer is a concatenation of the outputs of the previous i-1 convolutional layers, i.e., Input_i = Concat(F_in, F_1, F_2, ..., F_{i-1}), where F_in is the input feature of ERDB, and F_1 to F_{i-1} are the output features of the previous i-1 convolutional layers. This dense connection mechanism can fully reuse the features of each layer, enhance the fluidity of features, alleviate the gradient vanishing problem, and improve the feature learning ability of the network.

[0067] Each convolutional layer is followed by a ReLU activation function, introducing non-linear transformation capabilities. After the 8th convolutional layer of ERDB, ReLU activation is no longer applied; instead, the feature map F_conv is output directly.

[0068] At the end of the ERDB, a channel attention mechanism (CA) is set up, the detailed structure of which is as follows: Figure 4As shown. The input to the CA module is F_conv, which first compresses the feature map into a 1×1×64 channel statistical vector through a global average pooling layer (GAP).

[0069] The channel statistics vector is then subjected to a nonlinear transformation through two fully connected layers to learn the importance weights of each channel. The first fully connected layer has C / 16 output channels (where C=64, i.e., 4 output channels) and uses ReLU activation function; the second fully connected layer restores the output channel count to C (i.e., 64) and uses Sigmoid activation function. The Sigmoid activation function normalizes the importance weights to weight coefficients between 0 and 1, and the weight coefficient vector is denoted as W_ca (size 1×1×64).

[0070] Finally, the weight coefficients W_ca are multiplied channel-by-channel with the original feature map F_conv to complete feature recalibration, resulting in a weighted feature map F_weighted = F_conv ⊙ W_ca, where ⊙ represents the channel-by-channel multiplication operation. The channel attention mechanism adaptively enhances attention to key feature channels in the perception task while suppressing irrelevant feature channels, thereby improving the quality of the reconstructed image and the accuracy of the perception task.

[0071] ERDB also introduces local residual connections, which add the input feature F_in and the weighted feature F_weighted together to obtain the output feature F_out = F_in + α × F_weighted, where α is the adaptive scaling factor.

[0072] The value of α ranges from 0 to 1 and is learned from the input features F_in by a light quantum network. This sub-network consists of a global average pooling layer, a fully connected layer (with 1 output channel), and a sigmoid activation function.

[0073] The adaptive residual scaling mechanism can dynamically adjust the weights of residual connections according to the content of the input image. It enhances the weights of residual connections for images with rich details and reduces the weights of residual connections for smooth regions, thereby improving reconstruction quality and the network's adaptability.

[0074] Sixteen ERDBs are concatenated to form a deep feature extraction module, with the output of each ERDB serving as the input to the next. The overall function of the deep feature extraction module is to learn high-level semantic features and complex texture patterns of the image through multi-level feature extraction and transformation, providing rich feature representations for subsequent upsampling reconstruction.

[0075] Part Three, Multi-Scale Feature Fusion Module: The structural framework of this module is as follows... Figure 5As shown, its function is to perform multi-scale processing on the feature map output by the deep feature extraction module, extract feature information at different scales and fuse them, thereby enhancing the model's adaptability to targets of different sizes.

[0076] The multi-scale feature fusion module contains three parallel processing branches: The first branch is the original scale branch, which directly performs 3×3 convolution on the input feature F and outputs feature map F_1; The second branch is the 1 / 2 downsampling branch. First, the input feature F is downsampled to 1 / 2 of its original size through the average pooling layer AvgPool(2), then 3×3 convolution is performed, and finally the feature map is restored to its original size through bilinear interpolation upsampling UpSample(2), and the output feature map F_2 is output. The third branch is the 2x upsampling branch. First, the input feature F is upsampled to twice the original size by bilinear interpolation UpSample(2), and then 3×3 convolution is performed to output the feature map F_3.

[0077] It should be noted that, in order to keep the output feature maps of the three branches consistent in size, the output feature map F_3 of the third branch needs to be downsampled to H×W×64 by the average pooling layer AvgPool(2).

[0078] The output feature maps F_1, F_2, and F_3 from the three branches are concatenated along the channel dimension to obtain a stitched feature map. Then, a 1×1 convolutional layer is used to fuse features and reduce dimensionality, resulting in the final fused feature map F_fused. The role of the 1×1 convolutional layer is to linearly combine and fuse features from multiple channels while maintaining the spatial size of the feature map, learning the correlation between features at different scales.

[0079] The multi-scale feature fusion module processes feature maps of different scales, enabling it to simultaneously capture global contextual information of large targets and local detail information of small targets. This enhances the model's adaptability to targets of different sizes, and significantly improves the reconstruction effect for common long-distance small targets in intelligent vehicle scenarios, such as distant vehicles, pedestrians, and traffic signs.

[0080] Part Four, Global Residual Connections and Adaptive Residual Scaling: At the output of the multi-scale feature fusion module, a global residual connection is set up to weight and sum the output features of the shallow feature extraction layer (denoted as F_shallow) with the fused feature map F_fused to obtain the final feature map. F_final = F_shallow + β × F_fused Where β is the global adaptive scaling factor.

[0081] The learning method for β is the same as that for the local adaptive scaling factor α, which is learned from F_shallow by a light quantum network. Global residual connections can directly pass shallow features to the network output, alleviating the gradient vanishing problem in deep networks while preserving low-level details of the image.

[0082] Part 5, Upsampling Reconstruction Module: This module uses subpixel convolutional layers to perform a 4x upsampling process on the feature map F_final, reconstructing the low-resolution feature map into a high-resolution image.

[0083] Compared with traditional interpolation upsampling and transposed convolution, subpixel convolution has the following advantages: First, it can better preserve image details and avoid the blurring effect caused by interpolation upsampling; second, it has fewer parameters and less computation, and faster inference speed; third, it can learn more complex upsampling patterns and adapt to different types of image degradation.

[0084] The output of the upsampling reconstruction module is the super-resolution reconstructed Y luminance channel image SR_Y (size 4H×4W×1). Subsequently, the retained U and V chrominance channels are upsampled to 4H×4W resolution through bicubic interpolation, merged with SR_Y, and converted back to the RGB color space to obtain the final high-resolution reconstructed image SR.

[0085] Training method for the Perceptual-EDSR model: Training Data Construction: The training dataset uses large-scale public autonomous driving datasets, such as KITTI, Cityscapes, and BDD100K. High-resolution images (2560×1920 pixels or higher) are selected from these datasets as ground truth (HR) images.

[0086] To simulate the degradation process of a real vehicle camera, the HR image was processed as follows: First, a 4x downsampling was performed using bicubic interpolation to obtain a low-resolution image (640×480 pixels). Second, Gaussian noise was added, with the noise standard deviation σ randomly selected from 5 to 15. Third, Poisson noise was added to simulate the statistical characteristics of photon counting. Fourth, a motion blur kernel was applied, with a kernel size ranging from 5×5 to 15×15 and the blur direction randomly selected. After these processing steps, the low-resolution image LR was obtained. The HR and LR images were then used to construct training image pairs (LR, HR), resulting in approximately 50,000 training image pairs.

[0087] Multi-task loss function design: This invention designs a multi-task loss function specifically for perception tasks. This loss function consists of three parts: Part 1, Pixel-level L1 loss (L_pixel): This loss function directly compares the difference between the reconstructed image SR and the true high-resolution image HR at the pixel level, and the calculation formula is as follows: ; in SR To reconstruct the image, HR For true high-resolution images, n Total number of pixels Represents the coordinates of a pixel.

[0088] Compared to L2 loss (mean squared error), L1 loss is more robust to outliers and can better preserve the edge and detail information of the image.

[0089] Part Two, Perceptual Loss (L_perceptual): The loss function is calculated based on the feature extractor of the VGG-19 network pre-trained on the ImageNet dataset.

[0090] Specifically, the reconstructed image SR and the real image HR are input into the VGG-19 network, the convolutional features of the 5th layer (output of the conv5_4 layer) are extracted, and the difference between the two in the feature space is calculated.

[0091] The formula for calculating perceived loss is:

[0092] in This represents the feature extractor of the VGG-19 network pre-trained on the ImageNet dataset, which extracts the features from the 5th convolutional layer for computation; C, H, and W are the number of channels, height, and width of the feature map, respectively.

[0093] Perceptual loss can guide the model to generate reconstruction results that are visually closer to the real image, avoiding over-smoothing and loss of detail, especially for areas with complex textures such as trees and building surfaces, significantly improving the reconstruction effect.

[0094] Part Three, Task-Driven Loss (L_task): This loss function is designed for downstream object detection or semantic segmentation tasks, enabling super-resolution reconstruction and perception tasks to achieve synergistic optimization.

[0095] ; in, Indicates boundary loss, Represents classification loss. Indicates smoothing loss. This refers to an SR target detector pre-trained based on boundary loss. This refers to an HR target detector pre-trained based on boundary loss. This represents the cross-entropy loss function.

[0096] Detector() represents a pre-trained downstream task detector.

[0097] The total loss function is the weighted sum of the three losses mentioned above: , Where λ1, λ2, and λ3 are weighting coefficients, and their optimal values ​​are determined through cross-validation. In this embodiment, λ1 = 1.0, λ2 = 0.1, and λ3 = 0.05.

[0098] Training Strategy: The Adam optimizer was used for model training, with an initial learning rate of 1e-4 and a batch size of 16. Training lasted for a total of 500 epochs, employing a stepped learning rate decay strategy, halving the learning rate every 100 epochs (i.e., at epochs 100, 200, 300, and 400, the learning rate decayed to 5e-5, 2.5e-5, 1.25e-5, and 6.25e-6, respectively). During training, model performance was evaluated on a validation set every 10 epochs, and the weights of the best-performing model on the validation set were saved. Training was performed on an NVIDIA A100 GPU, with each training session taking approximately 48 hours.

[0099] After model training is complete, the trained model weights are deployed to the vehicle domain controller for real-time inference. During the inference phase, given a low-resolution image of 640×480 pixels as input, the model outputs a high-resolution reconstructed image of 2560×1920 pixels, with an inference time of approximately 30 milliseconds, meeting real-time requirements.

[0100] The technical effects achieved in this step are as follows: By introducing improved residual dense blocks, channel attention mechanisms, multi-scale feature fusion modules, and adaptive residual scaling mechanisms, the Perceptual-EDSR model can fully learn the high-level semantic features and complex texture patterns of images, generating high-resolution reconstructed images containing more detailed information. Through collaborative optimization of multi-task loss functions, the reconstructed images not only have high visual quality but are also more conducive to subsequent object detection and semantic segmentation.

[0101] Experimental data show that, compared with the model trained using only pixel-level L1 loss, the model trained with multi-task loss function improves the average accuracy (mAP) of the reconstructed image input to the YOLOv5 object detection model from 68.3% to 78.5%, an improvement of about 10.2 percentage points; and improves the average intersection-union ratio (mIoU) of the reconstructed image input to the DeepLabv3+ semantic segmentation model from 72.1% to 78.9%, an improvement of about 6.8 percentage points.

[0102] S103, Environmental Perception Steps.

[0103] This step inputs the high-resolution reconstructed image into the object detection neural network model and the semantic segmentation neural network model to identify traffic participants, lane lines, and traffic signs in the image.

[0104] Object Detection Neural Network Model: In this embodiment, YOLOv5 is used as the object detection model. YOLOv5 is a single-stage detection model based on convolutional neural networks, which has the advantages of fast inference speed and high detection accuracy, making it suitable for intelligent vehicle scenarios with high real-time requirements. The input of the YOLOv5 model is a high-resolution reconstructed image SR (size 2560×1920×3), and the output is the detection results, including the bounding box coordinates (x_min, y_min, x_max, y_max) of each detected object, the category label (such as vehicle, pedestrian, cyclist, traffic sign, etc.), and the confidence score.

[0105] Semantic Segmentation Neural Network Model: In this embodiment, DeepLabv3+ is used as the semantic segmentation model. DeepLabv3+ is a semantic segmentation model based on an encoder-decoder structure. The encoder extracts multi-scale features, and the decoder gradually restores the spatial resolution, generating fine pixel-level segmentation results. The input to the DeepLabv3+ model is a high-resolution reconstructed image SR (2560×1920×3 pixels), and the output is the segmentation result, i.e., the category label for each pixel (such as lane lines, drivable areas, sidewalks, obstacles, etc.).

[0106] In other embodiments, other semantic segmentation models, such as PSPNet or U-Net, can also be used. PSPNet extracts multi-scale contextual information through pyramid pooling modules, making it suitable for handling complex scenes; U-Net, through its symmetric encoder-decoder structure and skip connections, can generate high-precision segmentation results, making it suitable for medical image segmentation and small object segmentation.

[0107] Data structure of perception results: The output of the environment perception module includes object detection results and semantic segmentation results. The data structure of object detection results is: a list of detected objects, each containing bounding box coordinates, class label, and confidence score. The data structure of semantic segmentation results is: a pixel-level segmentation mask, where each pixel's value is an integer representing its class index, such as 0 for background, 1 for lane lines, 2 for drivable area, and 3 for pedestrian crossing.

[0108] The technical effect achieved by this step is that by inputting high-resolution reconstructed images into the perception model, the recognition accuracy of small targets at a distance can be significantly improved.

[0109] S104, Decision-making and control steps.

[0110] This step generates control signals for vehicle control decisions based on environmental perception results and sends the control signals to the vehicle's drive-by-wire actuators.

[0111] In low-resolution images, the target outline is blurred and the bounding box is inaccurate, resulting in low monocular ranging accuracy, which in turn affects the accuracy and safety of decision control. Because the target outline is clearer and the bounding box is more accurate in high-resolution reconstructed images, the measurement error of the target's pixel height in the image is smaller, thereby improving the accuracy of distance estimation.

[0112] Decision-making logic: The decision control module performs real-time decision analysis based on the output of the environmental perception module and vehicle status information. The decision-making logic includes the following typical scenarios: Scenario 1, Collision Risk Assessment and Pre-braking: When the target detection results identify a pedestrian or obstacle ahead, the decision module first estimates the relative distance between the vehicle and the target using a monocular ranging algorithm. Then, combining the vehicle's current speed and the target's relative speed, it calculates the Time to Collision (TTC). When the TTC is less than a preset safety threshold, a collision risk is determined, the decision module generates a pre-braking signal, and executes braking actions.

[0113] Scenario 2: Traffic Sign Recognition and Cruise Control: When a speed limit sign is detected in the target detection results, the decision module uses an OCR algorithm or a pre-trained digit recognition model to identify the specific digit on the speed limit sign. Because the digits of the speed limit sign are clearer in high-resolution reconstructed images, the recognition accuracy is significantly improved. After identifying the speed limit digit, the decision module generates a cruise control signal, setting the vehicle's cruise speed limit to the speed corresponding to the speed limit digit. The cruise control signal is sent to the powertrain system via the CAN bus to adjust the vehicle's acceleration or deceleration behavior, ensuring that the vehicle speed does not exceed the speed limit requirement.

[0114] Scenario 3, Lane Keeping Assist: When lane lines are identified in the semantic segmentation results, the decision module calculates the lateral offset of the vehicle relative to the lane centerline. The lateral offset is calculated as follows: First, the pixel coordinates of the left and right lane lines are extracted from the segmentation mask; then, the pixel coordinates of the lane centerline are calculated; finally, the pixel offset between the vehicle centerline and the lane centerline is calculated and converted into the actual lateral offset by combining camera calibration parameters. When the lateral offset exceeds a preset threshold, the decision module generates a steering control signal to adjust the steering wheel angle to correct the offset. The steering control signal is sent to the steering actuator via the CAN bus to execute the steering action.

[0115] Control Execution: The final control signals, including throttle control, braking control, and steering control signals, are transmitted to the vehicle's drive-by-wire actuators via the CAN bus or FlexRay bus. The throttle control signal is sent to the electronic throttle or motor controller to control the vehicle's acceleration; the braking control signal is sent to the electronic brake assist system or brake-by-wire system to control the vehicle's braking; and the steering control signal is sent to the electric power steering system or brake-by-wire system to control the vehicle's steering. Upon receiving the control signals, the drive-by-wire actuators execute the corresponding control actions, completing the closed loop from perception to control.

[0116] The technical effect achieved by this step is that by taking advantage of the clearer target outlines and more accurate pixel sizes in high-resolution reconstructed images, the accuracy of monocular ranging is significantly improved, thereby enhancing the accuracy and safety of decision control.

[0117] This invention restores key details in images through super-resolution reconstruction technology, enabling subsequent target detection and segmentation models to more accurately identify small targets at a distance (such as distant vehicles and pedestrians) and read the specific content of traffic signs more clearly, thereby significantly reducing false detection and false negative rates and improving overall perception accuracy.

[0118] Because the reconstructed image is richer in detail, its effective perception resolution is improved, which is equivalent to extending the visual range of the intelligent vehicle under the same hardware conditions, giving the decision-making system more reaction time and thus extending the effective perception distance.

[0119] Furthermore, the optimized super-resolution model exhibits stronger robustness against image degradation in complex environments such as low light and motion blur, effectively enhancing the perception stability of intelligent vehicles under various harsh conditions.

[0120] Ultimately, the aforementioned performance improvements are mainly achieved through algorithm upgrades, without the need for expensive hardware replacements. This provides a cost-effective solution for enhancing perception capabilities for existing platforms and mid-to-low-end models, solving the problems of weak long-distance perception, low accuracy in small target recognition, and poor perception reliability in harsh environments caused by the use of low-resolution in-vehicle cameras in existing smart cars. Thus, without significantly increasing hardware costs, the overall performance and safety of the smart car's visual perception system are improved.

[0121] Example 2 This embodiment discloses an intelligent vehicle visual perception optimization system based on super-resolution reconstruction technology.

[0122] The intelligent vehicle visual perception optimization system based on super-resolution reconstruction technology includes: The image acquisition module is configured to acquire raw images captured by the vehicle's camera. The image reconstruction module is configured to: input the original image into the super-resolution reconstruction neural network model, perform deep feature extraction using multiple cascaded residual dense blocks, connect multiple convolutional layers in each residual dense block using a dense connection mechanism, and use a channel attention mechanism at the end of each residual dense block to generate a reconstructed image; The environmental perception module is configured to: perform environmental perception based on the reconstructed image, identify traffic participants, lane lines and traffic signs in the image, and optimize the vehicle's environmental perception.

[0123] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same.

[0124] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.

[0125] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in Embodiment 1 of this disclosure.

[0126] Example 4 The purpose of this embodiment is to provide an electronic device.

[0127] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in Embodiment 1 of this disclosure.

[0128] The steps and methods involved in the apparatuses of Embodiments 2, 3, and 4 above correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0129] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0130] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for optimizing visual perception of intelligent vehicles based on super-resolution reconstruction technology, characterized in that, Includes the following steps: Acquire raw images captured by the vehicle's camera; The original image is input into a super-resolution reconstruction neural network model, and deep feature extraction is performed using multiple cascaded residual dense blocks. In each residual dense block, multiple convolutional layers are connected using a dense connection mechanism, and a channel attention mechanism is used at the end of each residual dense block to generate a reconstructed image. Based on the reconstructed image, environmental perception is performed to identify traffic participants, lane lines, and traffic signs in the image, thereby optimizing the vehicle's environmental perception.

2. The intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in claim 1, characterized in that, Before inputting the original image into the super-resolution reconstruction neural network model for deep feature extraction, a shallow feature extraction layer is first used to extract shallow features from the original image. The shallow feature extraction layer is a convolutional layer, and the shallow features include edge, corner, and texture features.

3. The intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in claim 1, characterized in that, When performing deep feature extraction, the output of the previous residual dense block is used as the input of the next residual dense block, thus realizing multi-level feature extraction.

4. The intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in claim 1, characterized in that, Within each residual dense block, multiple convolutional layers are connected using a dense connection mechanism, specifically: The output of each convolutional layer is not only passed to the next layer, but also passed to the input of all subsequent convolutional layers through cascading connections, thus enabling the reuse of features from each layer. Except for the last convolutional layer, a ReLU activation function is applied after each convolutional layer; The last convolutional layer does not use the ReLU activation function and directly outputs the feature map.

5. The intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in claim 1, characterized in that, A channel attention mechanism is used at the end of each residual dense block, specifically: First, the feature map F_conv output by the last convolutional layer in the residual dense block is compressed into a channel statistics vector through a global average pooling layer. The channel statistical vectors are nonlinearly transformed through two fully connected layers to learn the importance weights of each channel and obtain the weight coefficients. The weight coefficients are multiplied channel by channel with the original feature map F_conv to complete the feature recalibration and obtain the weighted feature map. This also includes introducing local residual connections in each dense residual block, specifically: The weighted feature map output by the channel attention mechanism in the residual dense block is fused with the input features in the current residual dense block to obtain the output features of the current residual dense block; During weighted fusion, the weights corresponding to the weighted feature maps are adaptive scaling coefficients, which are learned from the input features of the residual dense blocks through a light quantum network. The local residual connection can be expressed by the following formula: F_out = F_in + α × F_weighted; Where F_out is the output feature of the current residual dense block; F_in is the input feature of the residual dense block; α is the adaptive scaling factor; and F_weighted is the weighted feature map output by the channel attention mechanism.

6. The intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in claim 2, characterized in that, After performing deep feature extraction using multiple cascaded residual dense blocks, the process also includes: Deep features are processed at multiple scales to extract and fuse feature information at different scales to obtain a fused feature map. Using global residual connections, shallow features are weighted and fused with the fused feature map to obtain the final feature map; The final feature map is upsampled to reconstruct a high-resolution image from the low-resolution feature map, resulting in the reconstructed image. or, The reconstructed image is input into the object detection neural network model and the semantic segmentation neural network model. Forward propagation calculations are performed on the object detection neural network model and the semantic segmentation neural network model to identify traffic participants, lane lines and traffic signs in the image. or, After optimizing the vehicle's environmental perception, the following is also included: Based on the results of environmental perception, control signals are generated for vehicle control decisions, controlling the vehicle's throttle, braking, or steering mechanisms.

7. The intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in claim 1, characterized in that, The loss function for super-resolution reconstruction neural network models includes: Pixel-level L1 loss : ； in, SR To reconstruct the image, HR For true high-resolution images, n Total number of pixels; Represents pixel coordinates; Perceived loss : ； in, Indicates a feature extractor; This represents the features extracted by the feature extractor from the reconstructed image; This represents the features extracted by the feature extractor from the real high-resolution image; C, H, and W represent the number of channels, height, and width of the feature map, respectively. Task-driven loss : ； in, Indicates boundary loss, Indicates classification loss; Total loss function for: ； in, , and All of these represent weighting coefficients.

8. An intelligent vehicle visual perception optimization system based on super-resolution reconstruction technology, characterized in that, include: The image acquisition module is configured to acquire raw images captured by the vehicle's camera. The image reconstruction module is configured to: input the original image into the super-resolution reconstruction neural network model, perform deep feature extraction using multiple cascaded residual dense blocks, connect multiple convolutional layers in each residual dense block using a dense connection mechanism, and use a channel attention mechanism at the end of each residual dense block to generate a reconstructed image; The environmental perception module is configured to: perform environmental perception based on the reconstructed image, identify traffic participants, lane lines and traffic signs in the image, and optimize the vehicle's environmental perception.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the intelligent vehicle visual perception optimization method based on super-resolution reconstruction technology as described in any one of claims 1-7.

Images