A vehicle-mounted image super-resolution reconstruction method, system, device and storage medium

Abstract

The application discloses a kind of vehicle-mounted image super-resolution reconstruction method, system, device and storage medium, wherein method includes: obtaining the low-resolution vehicle-mounted image to be reconstructed;Using the trained vehicle-mounted image super-resolution reconstruction model to the low-resolution vehicle-mounted image is reconstructed, obtains high-resolution vehicle-mounted image, completes image reconstruction task;Wherein, the vehicle-mounted image super-resolution reconstruction model is trained by using unsupervised degradation enhancement GAN network;In the training process, the feature information of vehicle-mounted image degradation is extracted, and the extracted feature information is fitted into the vehicle-mounted image super-resolution reconstruction model, to improve the reconstruction ability of model.The application fully extracts the feature information of noise, artifact, interference and the like leading to vehicle-mounted image degradation in the training process, and the extracted feature information is fitted into the model, thereby improving the reconstruction performance of the model.The application can be widely applied in the field of image processing technology.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, system, apparatus and storage medium for super-resolution reconstruction of vehicle-mounted images. Background Technology

[0002] With the continuous development of the times, technologies such as new energy vehicles, assisted driving, and autonomous driving are beginning to enter people's production and lives. In the field of assisted driving, clearly visible images are crucial for the recognition and judgment capabilities of in-vehicle AI. This relates to the maturity of assisted driving technology and whether it can be deployed in real-life scenarios. More importantly, it concerns people's lives and property safety. The video image quality of in-vehicle cameras plays an extremely important role in assisting drivers (such as reversing cameras), lane recognition by in-vehicle AI, and collision detection. However, due to the relative motion between the vehicle and roadside objects during driving, often accompanied by bumps and vibrations, the captured images contain varying degrees of motion blur, which is detrimental to the driver's judgment of road conditions. Therefore, de-blurring and super-resolution reconstruction of in-vehicle video images can help improve the accuracy of target classification in intelligent assisted driving and has certain practical significance.

[0003] Super-resolution reconstruction of in-vehicle images plays an increasingly important role in the development of new energy vehicles and autonomous driving technologies. Popular supervised learning methods use predefined degradation of high-resolution images to obtain low-resolution images, resulting in pairs of high- and low-resolution image data for model training. However, this approach suffers from several drawbacks. First, the predefined degradation method limits the model's reconstruction performance to a specific feature space, leading to overfitting in image feature prediction and data overfitting due to insufficient training data. Second, while various complex and sophisticated neural networks have been designed for super-resolution, achieving excellent reconstruction results, the excessive computational complexity and parameter space consumption result from the large number of network layers, leading to excessive memory and power consumption, making them unsuitable for porting to modern mobile devices. With the increasing prevalence of smart devices and the growing demand for high-resolution images, improving the model's generalization ability and reducing its computational complexity by designing more lightweight super-resolution models is the future direction of research in this field. Summary of the Invention

[0004] In order to at least partially solve one of the technical problems existing in the prior art, the present invention aims to provide a method, system, device and storage medium for super-resolution reconstruction of vehicle images.

[0005] The technical solution adopted in this invention is:

[0006] A method for super-resolution reconstruction of vehicle-mounted images includes the following steps:

[0007] Acquire low-resolution vehicle-mounted images to be reconstructed;

[0008] The trained vehicle image super-resolution reconstruction model is used to reconstruct the low-resolution vehicle image to obtain a high-resolution vehicle image, thus completing the image reconstruction task.

[0009] The vehicle image super-resolution reconstruction model is jointly trained using an unsupervised de-quality enhancement GAN network (Generative Adversarial Network). During training, de-quality features of the vehicle image are extracted and fitted into the vehicle image super-resolution reconstruction model to improve the model's reconstruction capability.

[0010] Furthermore, the network structure can be built in the following ways:

[0011] The degraded and enhanced GAN network and the vehicle image super-resolution reconstruction model are combined into a bidirectional, end-to-end vehicle image reconstruction system.

[0012] Among them, the unsupervised de-enhancing GAN network is used to extract noise from vehicle images in natural scenes to obtain low-resolution generated images;

[0013] The vehicle-mounted image super-resolution reconstruction model uses the obtained low-resolution generated image and the high-resolution image as label as training data for weight inference, thereby improving the overall model's ability to handle image degradation caused by various harsh scenes.

[0014] Furthermore, the degraded enhancement GAN network includes a degrade generator and a degrade discriminator;

[0015] The training steps specifically include:

[0016] Obtain high-resolution images of the underlying facts;

[0017] The base image and random noise are input into the degradation generator, which outputs a low-resolution degraded image containing random noise.

[0018] The degraded image and the unpaired noisy image containing natural random noise distribution are input into the degraded image detector for authenticity identification.

[0019] By generating and countering degraded discriminators and degraded generators, degraded enhancement GAN networks are made capable of spoofing noise types.

[0020] Furthermore, the loss function corresponding to the degradation discriminator is as follows:

[0021]

[0022] In the formula, A represents the ReLU activation function, D represents the degraded discriminator, BS represents the batch size, lrs represents the noisy image, and lr_gen represents the generated degraded image.

[0023] Furthermore, the loss function corresponding to the degradation generator is as follows:

[0024]

[0025] In the formula, α and β represent the mean square loss weight and the adversarial loss weight, respectively; downs represents the result of downsampling the base fact image; and W and H represent the width and height of the pixel resolution.

[0026] Furthermore, the degradation generator includes a first residual coding block, a lightweight second residual coding block, and a decoding block;

[0027] In the initial stage of encoding, the first residual coding block is used for operation;

[0028] As the encoding process deepens, the number of feature channels increases and the pixel blocks decrease; when the size of the degraded image is detected to be less than a preset threshold, the second residual coding block replaces the first residual coding block.

[0029] Furthermore, the noise image includes Gaussian noise image, artifact noise image, and distortion noise image.

[0030] Another technical solution adopted in this invention is:

[0031] A vehicle-mounted image super-resolution reconstruction system, comprising:

[0032] The image acquisition module is used to acquire low-resolution vehicle images to be reconstructed.

[0033] The image reconstruction module is used to reconstruct the low-resolution vehicle image using the trained vehicle image super-resolution reconstruction model to obtain a high-resolution vehicle image and complete the image reconstruction task.

[0034] The vehicle image super-resolution reconstruction model is jointly trained using an unsupervised de-quality enhancement GAN network. During training, de-quality features of the vehicle image are extracted and fitted into the vehicle image super-resolution reconstruction model to improve the model's reconstruction capability.

[0035] Another technical solution adopted in this invention is:

[0036] A vehicle-mounted image super-resolution reconstruction device, comprising:

[0037] At least one processor;

[0038] At least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0040] Another technical solution adopted in this invention is:

[0041] A computer-readable storage medium storing a processor-executable program, which, when executed by a processor, performs the method described above.

[0042] The beneficial effects of this invention are: This invention uses a degraded and enhanced GAN network to obtain low-resolution images to train a vehicle image super-resolution reconstruction model. During the training process, it fully extracts feature information such as noise, artifacts, and interference in natural scenes that cause vehicle image degradation, and fits the extracted feature information into the model, thereby improving the reconstruction performance of the model. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the vehicle image super-resolution reconstruction model based on an unsupervised degraded enhancement GAN network in an embodiment of the present invention;

[0045] Figure 2 This is a diagram of the encoding / decoding network structure of the degradation generator in this embodiment of the invention. Detailed Implementation

[0046] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0047] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0048] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0049] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0050] Traditional super-resolution reconstruction methods for blurred images often rely on prior assumptions about the sharp image to estimate the corresponding motion blur kernel. This kernel is then convolved with the sharp image, and noise is added to obtain the motion-blurred image. The kernel estimation process is complex, and the deblurring effect largely depends on prior assumptions about the image, making it difficult to generalize to other blur types. In recent years, the development of Convolutional Neural Networks (CNNs) has pointed to new directions in computer vision. In image super-resolution (SISR) tasks, the introduction of CNNs has not only overcome the limitations of traditional methods but also significantly improved image quality. Furthermore, reconstructing images by learning data distributions can yield more flexible models.

[0051] The first image super-resolution algorithm based on convolutional neural networks was proposed by Dong et al. in 2014. This method (SRCNN) introduced deep learning into the field of image super-resolution reconstruction, utilizing three convolutional layers to perform image feature extraction, feature mapping, and pixel reconstruction, ultimately resulting in an end-to-end vehicle image super-resolution reconstruction model—that is, a low-resolution input image can output a corresponding high-resolution image. Although this network has only three simple layers, it still produces a clearer high-resolution image than traditional interpolation methods. However, SRCNN requires preprocessing the low-resolution image by first performing bicubic interpolation to enlarge it to the target image size before it can be used as network input, resulting in high computational complexity. Therefore, the authors of this network subsequently proposed FSRCNN, replacing the last layer with a deconvolutional layer, allowing the network to learn directly at low resolution, greatly saving costs. In subsequent work, networks like ESPCN also avoid learning feature maps in high-dimensional space. This network proposes using sub-pixel convolutional layers, performing feature map rearrangement simultaneously in the last layer of the network. By using subpixel convolutional layers, the interpolation function for upscaling images from low to high resolution is implicitly included in the preceding convolutional layers. Since the preceding convolutional operations are performed on the low-resolution image, computational complexity is significantly reduced. Subsequent image super-resolution methods primarily employ subpixel convolutional operations in the image reconstruction stage. It has been found that the depth of deep neural networks is positively correlated with their ability to fit features, thus leading to a preference for training deeper networks. However, as the number of network layers increases, severe gradient overflow or gradient explosion problems arise, making the network extremely difficult to train. To address this issue, Kim et al. proposed the VDSR network. This network was the first to introduce residual learning into the field of image super-resolution reconstruction and used gradient clipping to solve the training difficulties caused by increasing network depth, increasing the network to 20 convolutional layers, ultimately further improving the network's performance. Zhang et al. subsequently proposed the RCAN network, which embeds an attention mechanism into the residual blocks. This attention mechanism captures the correlation information between channels and applies different processing to different channels. This approach, through grouping and utilizing long and short skip connections, allows the network to focus more on learning high-level features, while passing low-level feature information through sideways, ultimately increasing the network depth and achieving better super-resolution reconstruction results.

[0052] To address the aforementioned problems, this embodiment provides a method for super-resolution reconstruction of vehicle images based on an unsupervised de-quality enhancement GAN network, comprising the following steps:

[0053] S101. Acquire a wide variety of photo data from vehicle-mounted cameras under various road conditions, including autonomous driving and assisted driving, for use in model training and testing.

[0054] As an optional implementation, for step S101, the vehicle-mounted image acquisition process needs to ensure the following conditions: Real-world image data from autonomous driving and assisted driving scenarios, including urban areas, rural areas, markets, and highways, is acquired using the vehicle's front-mounted grayscale and color cameras. The acquired image range also needs to include adverse weather conditions such as rain, snow, and sandstorms to ensure sufficient data acquisition and to accommodate various extreme situations that may be encountered in autonomous driving scenarios. Images containing noise, artifacts, and pixel distortion are manually filtered to obtain low-quality LR (low-resolution) datasets (unsupervised, unpaired) for this invention, while the remaining high-quality images are used as high-quality labeled HR (high-resolution) datasets.

[0055] S102. Improve the existing supervised training method, propose a degraded enhancement GAN network, and fully provide the network's feature extraction capability by optimizing the residual dense blocks to extract various noise information of vehicle images in natural scenes.

[0056] As an optional implementation, for step S102, in order to prevent the vehicle image reconstruction network from negatively impacting the model's reconstruction capability due to predefined degradation methods, this embodiment of the invention uses a large number of paired high-definition and blurred vehicle images in natural scenes for training. This training method is the optimal solution from the perspective of data quality. From the perspective of image degradation, a branch network capable of fitting the noise distribution of naturally degraded images is considered to be embedded in the vehicle image super-resolution reconstruction network. This network is trained by inputting high-resolution vehicle image data and low-resolution images containing diverse natural noise distributions. The goal is to obtain a network capable of fitting the noise distribution in unpaired LR image data to the network's generation capability. Recent research and experiments have shown that GAN networks can generate visually realistic forged images through the method of mutual adversarial interaction between generators and adversaries. This embodiment designs a degradation-enhanced GAN network, and the overall result of the network is as follows. Figure 1 As shown.

[0057] S103. The de-enhancing GAN network from S102 is inversely embedded into the vehicle image super-resolution reconstruction network, combining the de-enhancing GAN network and the vehicle image super-resolution reconstruction network into a bidirectional, end-to-end vehicle image reconstruction system. The unsupervised de-enhancing GAN network is used to extract noise from vehicle images in natural scenes, while the reconstruction network uses the obtained LR-generated images and high-quality HR images as labels as training data for weighted inference, thereby improving the network's overall ability to handle image degradation caused by various harsh scenes.

[0058] S104. The trained network parameters are used in vehicle-mounted image sensors, such as grayscale cameras and color cameras, to assist in improving the ability to collect data and make judgments in driving scenarios.

[0059] The above method will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0060] See Figure 1 This embodiment provides a method for super-resolution reconstruction of vehicle images based on an unsupervised de-quality enhancement GAN network, including the following steps:

[0061] S201, Dataset Processing.

[0062] This invention eliminates the need for manual degradation methods (such as bicubic interpolation) to degrade HR images into low-resolution LR images. Instead, it automatically generates LR vehicle-mounted datasets using a degradation-enhanced generative adversarial network. In the degradation-enhanced discriminator, the image dataset serving as the baseline for low-resolution vehicle-mounted images containing natural scene noise is obtained as follows: images of driving scenes captured by vehicle-mounted grayscale and color cameras are selected as the dataset, and images with degradation types including but not limited to noise, artifacts, and distortion are filtered out as unpaired LR datasets. Because the HR data and natural scene LR data in this embodiment contain different numbers of images, this embodiment re-shuffles and reorders the LR dataset after each iteration to improve training robustness and allow the generative network to randomly extract natural noise during experiments.

[0063] S202, Network Setup.

[0064] from Figure 1 As can be seen from the generator section, our de-optimization generator employs an asymmetric encoder-decoder structure with unequal input and output sizes. Unlike symmetric SR encoder-decoder networks, the asymmetric encoder-decoder structure does not require bilinear downsampling of the discriminator's input. This reduces model complexity and avoids unnecessary impacts from redundant noise and artifacts on the discriminator's predictions.

[0065] See Figure 2 During the generator encoding process, the first residual encoding block (res-encoding block) is used to encode the upscaled image. While increasing the feature dimension to capture diverse features, the size of the feature maps is gradually reduced. This avoids performing convolution operations on a large number of large feature maps, allowing for the extraction of deeper feature maps with lower model computation. The structure of the res-encoding block is as follows: Figure 2 As shown in (a), by using local residual learning, the learning ability of the module is enhanced while reducing the training difficulty. The structure of the decoding block is as follows: Figure 2 As shown in (b), when generating natural noise LR images using a de-enhancing generator, the HR image is progressively encoded and its features are extracted. The number of feature channels is increased from 3 layers to 512 layers, while the pixel size is gradually reduced from the original pixel size to 1 / 4 of a pixel. During this encoding process, experiments comparing feature maps show that as the number of feature channels increases and the pixel size decreases, the features extracted by the network become increasingly detailed. However, the correlation strength between pixels weakens. As the depth of the encoding network increases, if... Figure 2 The 3×3 basic coding block in (a) would increase the number of parameters in the network. Therefore, we replace the 3×3 channel transform convolutional layer with... Figure 2 (c) shows the lightweight residual coding block (i.e., the second residual coding block). The final generator generates the original... Figure 1 For LR images of size 1 / 4 pixels, we replace the residual coding blocks in coding layers with the following: Figure 2 The structure shown in (c) remains unchanged for the rest of the coding layer structure.

[0066] S203, Model Training.

[0067] The dataset required for the entire generative adversarial network consists of a base-fact high-resolution in-vehicle image dataset, an unpaired natural scene noise dataset, and random noise distributed according to a Gaussian distribution. First, we need to mix the base-fact high-resolution (HR) image with the Gaussian random noise and then input it into the generator of the de-enhancement network to generate a low-resolution image with a size 1 / 4 of the original pixel values ​​w and h. This image will be affected by the random noise. Then, we input the de-enhancement low-resolution in-vehicle image with random noise and the unpaired low-resolution in-vehicle image with natural random noise distribution into our de-enhancement network discriminator, respectively. Since the generator's forgery ability is very weak at this point, the discriminator can easily distinguish between genuine and fake images. Finally, we input the predicted probabilities of the two types of images into the loss function shown in Equation 1:

[0068]

[0069] The adversarial loss of the discriminator is shown in Equation 1 above, where A represents the ReLU activation function, D represents the de-quality enhancement discriminator network, BS represents the batch size, lrs represents the base facts of the natural scene LR in-vehicle image, and lr_gen represents the generated LR in-vehicle image. Low-quality images in natural scenes contain various types of noise, including but not limited to Gaussian noise, artifacts, distortions, and other distortion types. This alternating generation and adversarial process between the discriminator and generator enables the generator network to forge these noise types. Our method uses a joint loss as the generator's loss function, where Equation 2 for de-quality enhancement of the generator network is:

[0070]

[0071] In the formula above, α and β represent the mean square loss weight and the adversarial loss weight, respectively, and downs represents the result of downsampling the base high-resolution image. The loss consists of two main parts: the first part is the pixel loss, which, through backpropagation, ensures that the pixel-level features of the reconstructed low-quality image remain undistorted. The second part of the formula incorporates the discriminator's judgment result as the adversarial loss. The purpose of introducing the adversarial loss is to guide the generator network to fit the distribution of vehicle noise in a natural scene, thus guiding the generator towards the desired outcome.

[0072] S204, Prediction Results.

[0073] For the low-resolution vehicle image to be enlarged, a trained image super-resolution model is used to reconstruct the image, and finally a high-resolution vehicle image enlarged by 4 times and the same size as the original image is obtained, thus completing the image reconstruction task.

[0074] This embodiment also provides an in-vehicle image super-resolution reconstruction system, including:

[0075] The image acquisition module is used to acquire low-resolution vehicle images to be reconstructed.

[0076] The image reconstruction module is used to reconstruct the low-resolution vehicle image using the trained vehicle image super-resolution reconstruction model to obtain a high-resolution vehicle image and complete the image reconstruction task.

[0077] The vehicle image super-resolution reconstruction model is jointly trained using an unsupervised de-quality enhancement GAN network. During training, de-quality features of the vehicle image are extracted and fitted into the vehicle image super-resolution reconstruction model to improve the model's reconstruction capability.

[0078] This embodiment of the vehicle-mounted image super-resolution reconstruction system can execute the vehicle-mounted image super-resolution reconstruction method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, possessing the corresponding functions and beneficial effects of the method.

[0079] This embodiment also provides an in-vehicle image super-resolution reconstruction device, including:

[0080] At least one processor;

[0081] At least one memory for storing at least one program;

[0082] When the at least one program is executed by the at least one processor, the at least one processor implements the method described above.

[0083] This embodiment of the vehicle-mounted image super-resolution reconstruction device can execute the vehicle-mounted image super-resolution reconstruction method provided in the method embodiment of the present invention, and can execute any combination of the implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.

[0084] This embodiment also provides a storage medium storing instructions or programs that can execute the vehicle image super-resolution reconstruction method provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0085] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0086] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0089] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0090] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0091] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0092] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0093] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

[0094] This invention patent was supported by the National Natural Science Foundation of China (NSFC) No. 62071183.

Claims

1. A method for super-resolution reconstruction of vehicle-mounted images, characterized in that, Includes the following steps: Acquire low-resolution vehicle-mounted images to be reconstructed; The trained vehicle image super-resolution reconstruction model is used to reconstruct the low-resolution vehicle image to obtain a high-resolution vehicle image, thus completing the image reconstruction task. The vehicle image super-resolution reconstruction model is jointly trained using an unsupervised de-quality enhancement GAN network. During the training process, the de-quality features of the vehicle image are extracted and fitted into the vehicle image super-resolution reconstruction model to improve the model's reconstruction capability. The network structure can be built using the following methods: The degraded and enhanced GAN network and the vehicle image super-resolution reconstruction model are combined into a bidirectional, end-to-end vehicle image reconstruction system. Among them, the unsupervised de-enhancing GAN network is used to extract noise from vehicle images in natural scenes to obtain low-resolution generated images; The vehicle-mounted image super-resolution reconstruction model uses the obtained low-resolution generated image and the high-resolution image as label as training data for weight inference, thereby improving the overall model's ability to handle image degradation caused by various harsh scenes. The degraded enhancement GAN network includes a degraded generator and a degraded discriminator; The training steps specifically include: Obtain high-resolution images of the underlying facts; The base image and random noise are input into the degradation generator, which outputs a low-resolution degraded image containing random noise. The degraded image and the unpaired noisy image containing natural random noise distribution are input into the degraded image detector for authenticity identification. By generating and countering degraded discriminators and degraded generators, degraded enhancement GAN networks are made capable of spoofing noise types.

2. The method for super-resolution reconstruction of vehicle-mounted images according to claim 1, characterized in that, The loss function corresponding to the degradation discriminator is as follows: In the formula, A represents the ReLU activation function, D represents the degraded discriminator, BS represents the batch size, lrs represents the noisy image, and lr_gen represents the generated degraded image.

3. The method for super-resolution reconstruction of vehicle-mounted images according to claim 2, characterized in that, The loss function corresponding to the degradation generator is as follows: In the formula, α and β represent the mean square loss weight and the adversarial loss weight, respectively; downs represents the result of downsampling the base fact image; and W and H represent the width and height of the pixel resolution.

4. The method for super-resolution reconstruction of vehicle-mounted images according to claim 1, characterized in that, The degradation generator includes a first residual coding block, a lightweight second residual coding block, and a decoding block; In the initial stage of encoding, the first residual coding block is used for operation; As the encoding process deepens, the number of feature channels increases and the pixel blocks decrease; when the size of the degraded image is detected to be less than a preset threshold, the second residual coding block replaces the first residual coding block.

5. The method for super-resolution reconstruction of vehicle-mounted images according to claim 1, characterized in that, The noise images include Gaussian noise images, artifact noise images, and distortion noise images.

6. A vehicle-mounted image super-resolution reconstruction system, applied to the method described in any one of claims 1-5, characterized in that, include: The image acquisition module is used to acquire low-resolution vehicle images to be reconstructed. The image reconstruction module is used to reconstruct the low-resolution vehicle image using the trained vehicle image super-resolution reconstruction model to obtain a high-resolution vehicle image and complete the image reconstruction task. The vehicle image super-resolution reconstruction model is jointly trained using an unsupervised de-quality enhancement GAN network. During training, de-quality features of the vehicle image are extracted and fitted into the vehicle image super-resolution reconstruction model to improve the model's reconstruction capability.

7. A vehicle-mounted image super-resolution reconstruction device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-5.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-5.

Images