A scene clarity restoration device and method for intelligent vehicle-mounted imaging camera

Abstract

The present invention discloses a method for scene clarity restoration of an intelligent vehicle-mounted imager, comprising: obtaining a blurry-clear data set pair; image enhancement; constructing a network structure; calculating a loss function; training and testing; and applying it to an intelligent vehicle. The vehicle-mounted device and image restoration technology designed by this method are dedicated to capturing images of non-uniformly moving rails. The original images have blurring artifacts and ringing effects, and the subsequent detection efficiency of abnormalities on the rail surface is extremely low. The image restored by the Strformer network has greatly improved clarity. By calculating the PSNR and SSIM in the test image, it was found that the image quality has reached a high level and is almost indistinguishable from a clear image. Experiments have shown that the restored image greatly improves the efficiency of defect detection, improves the efficiency of intelligent vehicles, and reduces a large amount of maintenance costs.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent vehicle-mounted image processing, and in particular to a device and method for scene clarity restoration of an intelligent vehicle-mounted imager. Background Art

[0002] In recent years, with increasing transportation volume, railways have continued to expand. Especially in economically developed and densely populated areas, railway speeds can reach as high as 350 km / h. This increase in speed and scale places high demands on railway inspection. Every year, the government invests heavily in maintenance costs to ensure the safe operation of trains.

[0003] Railway maintenance involves regular inspections of all components, with rails being a key component. Due to long-term wheel compression, collisions, wear, and aging, numerous defects appear on the rail surface. Common defects include scratches, distortion, corrosion, and weld loss. Manual visual inspection is one method, but it is inefficient, lacks real-time performance, and is subject to subjective factors. Recently, intelligent high-speed vehicles have been developed for rail fault detection. These vehicles use cameras to capture images of the rail surface for subsequent processing.

[0004] Due to the long length of the railway track, the intelligent vehicle needs to capture images at a certain speed to improve detection efficiency. At high speeds, the images of the rail surface captured by the camera exhibit motion blur, making subsequent defect detection difficult. Therefore, an efficient non-uniform motion image restoration technology is required. Summary of the Invention

[0005] The purpose of the present invention is to provide a scene clarity restoration device and method for an intelligent vehicle-mounted imager, which can achieve efficient non-uniform motion image restoration.

[0006] In order to solve the above technical problems, the technical solution of the present invention is:

[0007] A method for scene clarity restoration of an intelligent vehicle-mounted imager comprises the following steps:

[0008] Step 1: Obtain blurry-clear dataset;

[0009] Step 2: Perform the same image enhancement operations on the training images, including random horizontal inversion, random vertical inversion, and random 90-degree rotation.

[0010] Step 3: Build the Strformer network for image restoration;

[0011] Step 4: Calculate the loss and output the processing results

[0012] The loss function selects Charbonnier loss, edge loss and contrast loss as the joint loss function, and the calculation formula of the joint loss function is:

[0013] L loss =L char +λ1L edge +λ2L con

[0014] Among them L loss is the joint loss, L char For Charbonnier's loss, L edge is the edge loss, L con is the contrast loss, λ1, λ2 are coefficients;

[0015] Step 5: Network training and testing

[0016] During training, the blurred image in the dataset after image enhancement operation is input into the Strformer network, and the processed image is input into the network for training;

[0017] During testing, the network weights and model of the last training update are read, and SSIM and PSNR are used to calculate the difference between the restored image and the real clear image. These two values ​​are used as evaluation indicators to verify the effectiveness of the model.

[0018] Preferably, the method for obtaining the data set in step 1 is:

[0019] A continuous video is shot with a high-speed camera, and different numbers of consecutive delayed frames are averaged to produce different intensities of blur. When the high-speed camera's sensor receives light during the exposure process, a clear image signal is accumulated each time, resulting in a blurred image. The integrated signal is then converted into a pixel value through a nonlinear function. The blur accumulation formula is as follows:

[0020]

[0021] Where T represents the exposure time, S(t) represents the sensor signal of the clear image at time t, M represents the number of sampling frames, S[i] represents the i-th clear frame signal captured during the exposure time, and g is a nonlinear function that converts the potential signal into the observed image.

[0022] Preferably, the Strformer network includes a local fuzzy feature extraction module, a fuzzy information interaction network, and a global fuzzy feature restoration module. The local fuzzy feature extraction module includes 3 convolutional layers and 6 residual connection blocks; the fuzzy information interaction network includes 12 Transformer variant structures; the global fuzzy feature restoration module includes 2 convolutional layers, 2 transposed convolutional layers and 6 residual connection blocks. A LeakyReLU activation function is added to the output of each convolutional layer and residual connection. The original input image information is combined with the output of the entire network through a residual connection, and each layer of the local fuzzy feature extraction module is combined with each layer of the global fuzzy feature restoration module through a residual connection.

[0023] As a preferred method, the Strformer network restores the image:

[0024] First, the enhanced image is input into the local blur feature extraction module. The image passes through the first 3*3 convolutional layer with 3 input channels, 64 output channels, a stride of 1, a padding of 1, and 0 padding. After passing through the Leaky ReLU, the output is a 64-channel feature map.

[0025] Secondly, each residual connection structure consists of two 3*3 convolutional layers with 64 input channels, 64 output channels, a stride of 1, and a padding of 1. A Leaky ReLU activation is added to the output of each convolutional layer, and the input information is simply added to the output of the convolutional layer through the residual connection. This residual connection structure is stacked three times.

[0026] After that, a 3*3 convolutional layer with a stride of 2 processes the input 64-channel feature map into 128 channels, and downsamples the 256*256 size to 128*128. After the same residual connection structure, a convolutional layer with an output of 320 channels is added at the end with a stride of 2, downsampling the 128-size image to 64, and finally generating a feature map that retains local blur information;

[0027] The local fuzzy information is used as the input of the fuzzy information interaction network. MDWA is designed in the fuzzy information interaction network to replace self-attention. For a 256*256 image, the input is divided into three parts: query, key, and value. Each part goes through a linear layer, a deep convolution, and layer normalization. The feature map after the query and key operation is processed by Softmax to obtain the queryable attention. map, each position stores the weight of the pixel, and directly multiplies the value with the attentionmap to obtain the feature map after attention calculation. In the multi-head deep convolution attention, a convolution kernel of the deep convolution is only responsible for one channel, that is, the channel separation convolution is performed first, and then convolved together. The output of the layer normalization and multi-head deep convolution attention is passed to the block perceptron layer in the fuzzy information interaction network through the residual connection. In the forward propagation process, a simple multiplication is used as the activation function. The block perceptron structure input passes through a 1*1 convolution layer, and the feature map is directly divided into two parts in the channel dimension, and multiplied accordingly. After multiplication, it passes through a 1*1 convolution layer output, and the block perceptron layer is used to replace the feedforward structure of the basic structure. Each fuzzy information interaction network also contains two layers of normalization and residual connection, and 12 fuzzy information interaction networks are stacked to allow it to obtain more fuzzy information from the feature map.

[0028] The feature map output by the fuzzy information interaction network is used as the input of the global fuzzy feature restoration module. The global fuzzy feature restoration module is the inverse process of the local fuzzy feature extraction module. Since the size of the input image is 256*256, the size of the image after the local fuzzy feature extraction module is 64*64. To restore the image size to the same as the original, the global fuzzy feature restoration module is used. First, a 4*4 size, step-size 2 transposed convolution is used, the input channel is 320, and the output channel is 192. In the global fuzzy feature restoration module, the output of the first residual structure and the output of the transposed convolution are concat-operated in the channel dimension to obtain a 128+192 channel feature map. This feature map is input into the 1*1 convolution layer, and the output is 192 channels, which is equivalent to establishing a residual connection between the global fuzzy feature restoration module and the local fuzzy feature extraction module. The obtained feature map has both high-dimensional fuzzy information and low-dimensional fuzzy information. Finally, the feature map is restored to a 3-channel 256*256 image, which is the output image after the network.

[0029] Preferably, the calculation formula of the Charbonnier loss is:

[0030]

[0031] Among them, X srepresents the image restored by the neural network, Y represents the clear image in the training set, and ε is a constant term.

[0032] Preferably, the calculation formula of the edge loss is:

[0033]

[0034] Among them, Δ represents the Laplace operator, X s represents the image restored by the neural network, Y represents the clear image in the training set, and ε is a constant term.

[0035] Preferably, the calculation formula of the contrast loss is:

[0036]

[0037] Among them, L1 represents the L1 norm, X is the blurred image of the original input, S is the image restored by the neural network, R is the clear image in the training set, and ψ is the hidden feature extracted from the conv3-2 of the fixed pre-trained VGG19.

[0038] As a preference, when training the Strformer network, Adam is selected as the optimizer, and the initial learning rate is set to 10 -4 , gradually reduced to 10 through the cosine annealing strategy -7 , set the batch size to 2, the epoch to 3000, save the network weights every 300 epochs, and update the weights of the current best performing network.

[0039] The present invention also provides a scene clarity restoration device for an intelligent vehicle-mounted imager, comprising a high-speed camera, a camera sensor, and a central processing unit. The central processing unit is a Raspberry Pi, and the Raspberry Pi stores a computer program for implementing the above-mentioned scene clarity restoration method for an intelligent vehicle-mounted imager.

[0040] The present invention also provides an application of a scene clarity restoration device for an intelligent vehicle-mounted imaging camera. When the intelligent vehicle is driving fast, the camera continuously captures images, and the images of objects in the dynamic scene are blurred. The Raspberry Pi reads the blurred images captured by the camera, forms a clear image through the Strformer network, and saves it locally.

[0041] The present invention has the following characteristics and beneficial effects:

[0042] This patented vehicle-mounted device and image restoration technology are designed to capture images of railroad tracks with non-uniform motion. Original images exhibit blurring artifacts and ringing, making subsequent detection of rail surface anomalies extremely inefficient. The images restored using the Strformer network achieve significantly improved clarity. Calculating the PSNR and SSIM of the test images reveals that image quality has reached a high level, virtually indistinguishable from a clear image. Experiments have shown that the restored images significantly improve defect detection efficiency, enhance the efficiency of smart vehicles, and significantly reduce maintenance costs.

[0043] This paper addresses the deblurring problem of non-uniform motion images captured in intelligent vehicle scenarios by constructing a new dataset. The dataset consists of paired blurred and sharp images. Images captured on the intelligent vehicle serve as the sharp images in the dataset. A device for synthesizing blurred images is also developed. This device uses a high-speed camera and camera sensor to control the shutter time and, through integrated modeling, acquires a series of corresponding blurred images. After processing, a new dataset of paired sharp-blurred images is constructed.

[0044] A local blur feature extraction module is proposed, and a pyramid-like local blur feature extraction network is constructed. Convolutional blocks and residual blocks are stacked within the network to gradually extract non-uniform blur information from the image and reduce redundant information in the feature map. This module extracts local blur information from the image, which helps improve local clarity. Simultaneously, the input image is gradually downsampled, accelerating model training.

[0045] A novel fuzzy information interaction network, the Straight Transformer (Strformer), is proposed. Based on the attention mechanism and global modeling capabilities, the Strformer calculates attention scores between pixel blocks, assigns pixel weights based on the attention scores, and adaptively focuses on blurred regions in the image, thereby increasing the receptive field and achieving significant image restoration results.

[0046] Based on the Transformer network architecture, we designed a Multi-Head Deep Convolutional Attention (MDWA). Compared to conventional attention mechanisms, MDWA reduces model complexity. Secondly, we calculate interaction information in the channel dimension. After applying a softmax to the query and key, we generate an attention map that carries contextual information. We use the attention map as a dictionary to query the weight of the value, ultimately generating a feature map of each pixel block after information interaction.

[0047] Based on the Transformer network architecture, a block-based perceptron (BFF) architecture was designed. Nonlinear activation functions can be replaced with simple element-by-element multiplication of block feature maps, using blocks to simulate the multi-channel process of convolution. Based on the original Transformer architecture, the block-based perceptron architecture replaces the original feedforward network. This simplifies model complexity while enabling larger batch training and accelerating model convergence.

[0048] A global fuzzy feature restoration module is proposed, and a pyramid-like global fuzzy feature restoration network is constructed. Transposed convolution, residual blocks, and convolution blocks are stacked within the network, and information from the local fuzzy feature extraction module is fused. Deconvolution is performed based on the local fuzzy information, and this is continuously stacked to form global fuzzy information, which helps improve global clarity. The image size is also restored to its original size.

[0049] Unlike previous models that relied on single-scale input, the network supports multi-scale training. After data augmentation, it demonstrates excellent generalization across images of various sizes. Not only does it achieve excellent image restoration results on the original dataset, but it also significantly improves performance on other datasets. By maximizing complementarity, the model preserves information from the original image and minimizes loss during the restoration process. The restoration results demonstrate significant improvements in quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 This is an image synthesis blur device in an embodiment of the present invention.

[0052] Figure 2 Strformer structure diagram in an embodiment of the present invention.

[0053] Figure 3 It is a local fuzzy feature extraction module in an embodiment of the present invention.

[0054] Figure 4 2 is a structural diagram of the residual network in an embodiment of the present invention.

[0055] Figure 5 This is a fuzzy information interaction network in an embodiment of the present invention.

[0056] Figure 6 This is a diagram of the multi-head deep convolutional attention structure in an embodiment of the present invention.

[0057] Figure 7 This is a block perception machine layer structure diagram in an embodiment of the present invention.

[0058] Figure 8 This is the global fuzzy feature restoration module in the embodiment of the present invention. DETAILED DESCRIPTION

[0059] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0060] Example 1

[0061] This embodiment provides a method for restoring scene clarity using an intelligent vehicle-mounted imager, comprising the following steps:

[0062] Step 1: Obtain blurry-clear dataset pairs.

[0063] Specifically, it is used to obtain the data set device in the intelligent vehicle imaging scene, such as Figure 1 As shown, the system includes a high-speed camera, a camera sensor, and a central processing unit. A continuous video is captured using a high-speed camera, and different numbers of consecutive delayed frames are averaged to produce blurs of varying intensities. As the camera sensor receives light during exposure, it accumulates the sharp image signal each time, producing a blurred image. The integrated signal is then converted to a pixel value using a nonlinear function. The blur accumulation formula is as follows:

[0064]

[0065] Where T represents the exposure time, S(t) represents the sensor signal of the clear image at time t, M represents the number of sampled frames, and S[i] represents the signal of the i-th clear frame captured during the exposure time. g is a nonlinear function that converts the latent signal into the observed image. After processing by the central processing unit, 1000 blur-clear pairs were obtained for network training, and 300 unique blur-clear pairs were obtained for network testing.

[0066] Step 2: Image enhancement: Perform the same image enhancement operations on the training images, including random horizontal and vertical flips, and random 90-degree rotations. This series of image enhancements improves the model's generalization ability and the network's ability to restore new images.

[0067] Step 3: Network construction: The present invention uses the Strformer network to perform image restoration. The overall framework of the Strformer network is as follows Figure 2As shown in the figure, the network is a long straight structure consisting of three parts: a local fuzzy feature extraction module, a fuzzy information interaction network, and a global fuzzy feature restoration module. The core part is the fuzzy information interaction network. The local fuzzy feature extraction module includes three convolutional layers and six residual connection blocks; the fuzzy information interaction network includes 12 Transformer variant structures; and the global fuzzy feature restoration module includes two convolutional layers, two transposed convolutional layers, and six residual connection blocks. A leaky ReLU activation function is added to the output of each convolutional layer and residual connection. The original input image information is combined with the output of the entire network through a residual connection, and each layer of the local fuzzy feature extraction module is combined with each layer of the global fuzzy feature restoration module through a residual connection.

[0068] First, the enhanced image is input into the local fuzzy feature extraction module. The local fuzzy feature extraction module is as follows: Figure 3 As shown, the image passes through the first 3*3 convolutional layer, with 3 input channels, 64 output channels, a stride of 1, a padding of 1, and padding with 0. After passing through the Leaky ReLU, a 64-channel feature map is output. Compared with the original 3-channel image, each channel of the 64-channel feature map can handle different tasks, each channel focuses on different blur situations, and passes the blur information to the next layer. Multi-channels have stronger information capture capabilities. Secondly, a residual connection structure is designed. Specifically, each residual connection structure includes two 3*3 convolutional layers with 64 input channels, 64 output channels, a stride of 1, and a padding of 1. The output of each convolutional layer has a Leaky ReLU activation, and the input information is simply added to the output of the convolutional layer through the residual connection. Refer to ResNet, this residual structure makes it easier to learn a simple model to fit the data. In theory, the model can also learn simple things in the later layers without the residual connection, but in reality, it is difficult to learn this method without the guidance of the residual connection. In addition, the residual connection can reduce the complexity of the model, making it possible to train deep networks. Figure 4As shown in the figure, this residual connection structure is stacked three times. Next, a 3x3 convolutional layer with a stride of 2 is designed to process the input 64-channel feature map into 128-channels, while also downsampling the 256x256 image size to 128x128. Following the same residual connection structure, a final convolutional layer with a stride of 2 is added, outputting 320 channels, downsampling the 128-channel image size to 64. Thus, the local fuzzy feature extraction module consists of a 64-channel convolutional layer, a residual connection layer, a 128-channel convolutional layer, a residual connection layer, and a 320-channel convolutional layer. From a macroscopic perspective, it resembles a pyramidal structure, where the original 256-channel input image is downsampled to 128x128 and then to 64x64. The resulting feature map eliminates redundant information and maximizes the preservation of local fuzzy information. After passing through the local fuzzy feature extraction module, this local fuzzy information is passed to the next layer: the fuzzy information interaction network.

[0069] The Strformer structure is better than the ordinary CNN network. First, each parameter of the Strformer is dynamic, while the parameters of the CNN are static, which can easily reach a bottleneck. Second, CNN has prior information, while the Strformer obtains information from the image itself. Furthermore, the Strformer has a larger receptive field. The Strformer's receptive field is the entire image, while the CNN's receptive field is a convolution kernel, so the Strformer is better at establishing long-distance information dependencies. Fuzzy information interaction network such as Figure 5 As shown in Figure 2, self-attention is not used as the attention mechanism. Instead, multi-head deep convolutional attention (MDWA) is designed to replace self-attention. The MDWA structure is as follows: Figure 6As shown in the figure, because self-attention requires calculating an attention score between every two pixels, this is impractical for a 256x256 image. Therefore, a multi-head deep convolutional attention network was designed to reduce computational complexity. The input is divided into three parts: query, key, and value. Each part undergoes a linear layer, a deep convolution, and layer normalization. The feature map after the query and key operations is subjected to a softmax operation to generate a queryable attention map. Each pixel position stores the weight of the pixel. Generally speaking, highly correlated pixel blocks receive larger weights, resulting in more attention to those with large weights and less attention to those with small weights. This automatically focuses on heavily blurred areas of the image while ignoring clear areas. Simply multiplying the value with the attention map yields the feature map after the attention calculation. In the multi-head deep convolutional attention architecture, each convolution kernel is designed to focus on only one channel. This means that separate convolutions are performed before the channels are combined. This significantly reduces the number of parameters and computational complexity.

[0070] The output of layer normalization and multi-head deep convolutional attention is passed to the block perceptron layer in the fuzzy information interaction network via residual connections. During the forward propagation process, some nonlinear activation functions, such as GeLU, ReLU, Sigmoid, and Softmax, are unnecessary and can be replaced by simple multiplication. Therefore, a block perceptron architecture was designed. The input passes through a 1x1 convolutional layer, which directly divides the feature map into two parts along the channel dimension and multiplies them accordingly. After multiplication, the output passes through a 1x1 convolutional layer. The block perceptron (BFF) layer replaces the feedforward structure of the basic architecture. The BFF structure, shown in Figure 7, reduces complexity while achieving good performance. This completes the design of the fuzzy information interaction network, the core module of the overall network Strformer. It consists of multi-head deep convolutional attention and a block perceptron architecture. Each fuzzy information interaction network also includes two layers of normalization and residual connections. Twelve fuzzy information interaction networks are stacked to extract more fuzzy information from the feature map. In terms of effect, the designed fuzzy information interaction network significantly improves the effect of image restoration, eliminates image artifacts in the restoration process, and improves image clarity.

[0071] A global fuzzy feature restoration module is designed. Figure 8As shown in Figure 2. The global fuzzy feature restoration module is the inverse process of the local fuzzy feature extraction module. Since the input image size is 256*256, the image size after the local fuzzy feature extraction module is 64*64. The global fuzzy feature restoration module is used to restore the image size to the original size. First, a 4*4 transposed convolution with a stride of 2 is used, with 320 input channels and 192 output channels. In the global fuzzy feature restoration module, the output of the first residual structure is concatenated with the output of the transposed convolution in the channel dimension to obtain a feature map with 128+192 channels. This feature map is input into a 1*1 convolution layer, and the output is 192 channels. This is equivalent to establishing a residual connection between the global fuzzy feature restoration module and the local fuzzy feature extraction module. The advantage of this is that the information from the local fuzzy feature extraction module is passed to the global fuzzy feature restoration module. The resulting feature map contains both high-dimensional and low-dimensional fuzzy information. The residual connection part in the global fuzzy feature restoration module is basically the same as that in the local fuzzy feature extraction module. Finally, the feature map is restored to a 3-channel 256*256 image, which is the output image after passing through the network.

[0072] Step 4: Calculate the loss and output the processing results. The present invention selects Charbonnier loss, edge loss and contrast loss as the joint loss function. The calculation formula of the joint loss function is:

[0073] L loss =L char +λ1L edge +λ2L con

[0074] Among them L loss is the joint loss, L char For Charbonnier's loss, L edge is the edge loss, L con is the contrast loss, λ1 and λ2 are coefficients. Set λ1 to 0.05 and λ2 to 0.0005. The formula for Charbonnier loss is:

[0075]

[0076] Among them, X s represents the image restored by the neural network, Y represents the clear image in the training set, and ε is a constant term set to 0.001

[0077] The calculation formula of edge loss is:

[0078]

[0079] Among them, Δ represents the Laplace operator, Xs represents the image restored by the neural network, Y represents the clear image in the training set, and ε is a constant term set to 0.001

[0080] Contrastive learning is an effective self-supervised learning method. Specifically, in the absence of labels, the model classifies similar images as positive samples and different images as negative samples. During training, the loss function of positive samples is minimized and the loss function of negative samples is maximized. Using contrastive learning, the restored image and the ground truth clear image are recorded as positive samples, and the blurred input image is recorded as negative samples. The formula for calculating contrastive loss is:

[0081]

[0082] Where L1 represents the L1 norm, X is the original input blurred image, S is the image restored by the neural network, and R is the clear image in the training set. ψ is the hidden feature extracted from the conv3-2 of the fixed pre-trained VGG19.

[0083] Step 5: Network training and testing process. Before the blurred images in the dataset are fed into the network, image enhancement methods are applied, including random horizontal flip, random vertical flip, and random 90-degree flip. To speed up data convergence, data normalization is used. The processed images are fed into the network for training. Adam is selected as the optimizer, and the initial learning rate is set to 10. -4 . Gradually reduced to 10 through the cosine annealing strategy -7 . Set the batch size to 2 and the epoch to 3000. In order to ensure the stability of training, save the network weights every 300 epochs, and update the weights of the current best-performing network. In the testing phase, read the last updated network weights and model, use SSIM (structural similarity) and PSNR (peak signal-to-noise ratio) to calculate the difference between the restored image and the real clear image, and use these two values ​​as evaluation indicators to verify the effectiveness of the model. By comparing the image quality evaluation indicators of the test image and the clear image, the evaluation indicators of the images restored by the designed Strformer network are greatly improved compared with the evaluation indicators of the images restored by the traditional network.

[0084] Example 2

[0085] This embodiment also provides a scene clarity restoration device for an intelligent vehicle-mounted imager, comprising a high-speed camera, a camera sensor, and a central processing unit. The central processing unit is a Raspberry Pi, and the Raspberry Pi stores a computer program for implementing the scene clarity restoration method for an intelligent vehicle-mounted imager described in Example 1.

[0086] Furthermore, the computer program includes a data set and a Strformer network for image restoration.

[0087] Specifically, the overall framework of the Strformer network is as follows Figure 2 As shown in the figure, the network is a long straight structure consisting of three parts: a local fuzzy feature extraction module, a fuzzy information interaction network, and a global fuzzy feature restoration module. The core part is the fuzzy information interaction network. The local fuzzy feature extraction module includes three convolutional layers and six residual connection blocks; the fuzzy information interaction network includes 12 Transformer variant structures; and the global fuzzy feature restoration module includes two convolutional layers, two transposed convolutional layers, and six residual connection blocks. A leaky ReLU activation function is added to the output of each convolutional layer and residual connection. The original input image information is combined with the output of the entire network through a residual connection, and each layer of the local fuzzy feature extraction module is combined with each layer of the global fuzzy feature restoration module through a residual connection.

[0088] Example 3

[0089] This embodiment also provides an application of a scene sharpening and restoration device for an intelligent vehicle-mounted imaging camera. This device uses a Raspberry Pi as the central processor of the vehicle-mounted system, and stores the network model and training weights on the Pi. As the intelligent vehicle rapidly drives, its camera continuously captures images. Objects in dynamic scenes are imaged as blurred images. The Raspberry Pi reads these blurred images from the camera, transforms them into a clear image over the network, and stores them locally. This device effectively achieves image sharpening in intelligent vehicle-mounted imaging cameras.

[0090] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for scene clarity restoration of an intelligent vehicle-mounted imager, characterized in that: The following steps are involved: Step 1: Obtain blurry-clear dataset; Step 2: Perform the same image enhancement operations on the training images, including random horizontal inversion, random vertical inversion, and random 90-degree rotation. Step 3: Build the Strformer network for image restoration; The Strformer network includes a local fuzzy feature extraction module, a fuzzy information interaction network, and a global fuzzy feature restoration module. The local fuzzy feature extraction module includes 3 convolutional layers and 6 residual connection blocks; the fuzzy information interaction network includes a 12-block Transformer variant structure; the global fuzzy feature restoration module includes 2 convolutional layers, 2 transposed convolutional layers, and 6 residual connection blocks, wherein a Leaky ReLU activation function is added to the output part of each convolutional layer and residual connection, and the original input image information is combined with the output of the entire network through the residual connection, and each layer of the local fuzzy feature extraction module is combined with each layer of the global fuzzy feature restoration module through the residual connection; Step 4: Calculate the loss and output the processing results The loss function selects Charbonnier loss, edge loss and contrast loss as the joint loss function, and the calculation formula of the joint loss function is: L loss =L char +λ1L edge +λ2L con Among them L loss is the joint loss, L char For Charbonnier's loss, L edge is the edge loss, L con is the contrast loss, λ1, λ2 are coefficients; Step 5: Network training and testing During training, the blurred image in the dataset after image enhancement operation is input into the Strformer network, and the processed image is input into the network for training; During testing, the network weights and model of the last training update are read, and SSIM and PSNR are used to calculate the difference between the restored image and the real clear image. These two values ​​are used as evaluation indicators to verify the effectiveness of the model.

2. The scene clarity restoration method for an intelligent vehicle-mounted imager according to claim 1, characterized in that: Method for obtaining the data set in step 1: A continuous video is shot with a high-speed camera, and different numbers of consecutive delayed frames are averaged to produce different intensities of blur. When the high-speed camera's sensor receives light during the exposure process, a clear image signal is accumulated each time, resulting in a blurred image. The integrated signal is then converted into a pixel value through a nonlinear function. The blur accumulation formula is as follows: Where T represents the exposure time, S(t) represents the sensor signal of the clear image at time t, M represents the number of sampling frames, S[i] represents the i-th clear frame signal captured during the exposure time, and g is a nonlinear function that converts the potential signal into the observed image.

3. The scene clarity restoration method for an intelligent vehicle-mounted imager according to claim 2, characterized in that: Strformer network image restoration method: First, the enhanced image is input into the local blur feature extraction module. The image passes through the first 3*3 convolutional layer with 3 input channels, 64 output channels, a stride of 1, and a padding of 1, padded with 0, and passes through the LeakyReLU to output a 64-channel feature map. Secondly, each residual connection structure consists of two 3*3 convolutional layers with 64 input channels, 64 output channels, a stride of 1, and a padding of 1. A Leaky ReLU activation is added to the output of each convolutional layer, and the input information is simply added to the output of the convolutional layer through the residual connection. This residual connection structure is stacked three times. After that, a 3*3 convolutional layer with a stride of 2 processes the input 64-channel feature map into 128 channels, and downsamples the 256*256 size to 128*128. After the same residual connection structure, a convolutional layer with an output of 320 channels is added at the end with a stride of 2, downsampling the 128-size image to 64, and finally generating a feature map that retains local blur information; The local fuzzy information is used as the input of the fuzzy information interaction network. MDWA is designed in the fuzzy information interaction network to replace self-attention. For a 256*256 image, the input is divided into three parts: query, key, and value. Each part goes through a linear layer, a deep convolution, and layer normalization. The feature map after the query and key operations is processed by Softmax to obtain a queryable attention map. The weight of the pixel is saved at each position. The value and attention are combined. map is directly multiplied to obtain the feature map after attention calculation. In the multi-head deep convolution attention, a convolution kernel of the deep convolution is only responsible for one channel, that is, the channel separation convolution is performed first, and then convolution is performed together. The output of the layer normalization and multi-head deep convolution attention is passed to the block perceptron layer in the fuzzy information interaction network through the residual connection. In the forward propagation process, a simple multiplication is used as the activation function. The block perceptron structure input passes through a 1*1 convolution layer, and the feature map is directly divided into two parts in the channel dimension, and the corresponding parts are multiplied. After multiplication, it passes through a 1*1 convolution layer output, and the block perceptron layer is used to replace the feedforward structure of the basic structure. Each fuzzy information interaction network also contains two layers of normalization and residual connection, and 12 fuzzy information interaction networks are stacked to allow it to obtain more fuzzy information from the feature map. The feature map output by the fuzzy information interaction network is used as the input of the global fuzzy feature restoration module. The global fuzzy feature restoration module is the inverse process of the local fuzzy feature extraction module. Since the size of the input image is 256*256, the size of the image after the local fuzzy feature extraction module is 64*64. To restore the image size to the same as the original, the global fuzzy feature restoration module is used. First, a 4*4 size, step-size 2 transposed convolution is used, the input channel is 320, and the output channel is 192. In the global fuzzy feature restoration module, the output of the first residual structure and the output of the transposed convolution are concat-operated in the channel dimension to obtain a 128+192 channel feature map. This feature map is input into the 1*1 convolution layer, and the output is 192 channels, which is equivalent to establishing a residual connection between the global fuzzy feature restoration module and the local fuzzy feature extraction module. The obtained feature map has both high-dimensional fuzzy information and low-dimensional fuzzy information. Finally, the feature map is restored to a 3-channel 256*256 image, which is the output image after the network.

4. The method for scene clarity restoration of an intelligent vehicle-mounted imager according to claim 3, characterized in that: The calculation formula of the Charbonnier loss is: Among them, X s represents the image restored by the neural network, Y represents the clear image in the training set, and ε is a constant term.

5. The scene clarity restoration method for an intelligent vehicle-mounted imager according to claim 4, characterized in that: The calculation formula of the edge loss is: Among them, Δ represents the Laplace operator, X s represents the image restored by the neural network, Y represents the clear image in the training set, and ε is a constant term.

6. The method for scene clarity restoration of an intelligent vehicle-mounted imager according to claim 5, characterized in that: The calculation formula of the contrast loss is: Among them, L1 represents the L1 norm, X is the blurred image of the original input, S is the image restored by the neural network, R is the clear image in the training set, and ψ is the hidden feature extracted from the conv3-2 of the fixed pre-trained VGG19.

7. The method for scene clarity restoration of an intelligent vehicle-mounted imager according to claim 6, characterized in that: When the Strformer network is trained, Adam is selected as the optimizer and the initial learning rate is set to 10 -4 , gradually reduced to 10 through the cosine annealing strategy -7 , set the batch size to 2, the epoch to 3000, save the network weights every 300 epochs, and update the weights of the current best performing network.

8. A scene clarity restoration device for an intelligent vehicle-mounted imaging camera, characterized in that: The invention comprises a high-speed camera, a camera sensor and a central processing unit, wherein the central processing unit is a Raspberry Pi, and the Raspberry Pi stores a computer program for implementing the scene clarity restoration method of the intelligent vehicle-mounted imager according to any one of claims 1 to 7.

9. An application of the intelligent vehicle-mounted imager scene clarity restoration device according to claim 8, characterized in that: When the smart car is driving fast, the camera continuously captures images. Objects in the dynamic scene are imaged as blurred images. The Raspberry Pi reads the blurred images captured by the camera, forms a clear image through the Strformer network, and saves it locally.

Images