Image enhancement device, method and storage medium

Abstract

The present invention discloses an image enhancement device, method and storage medium, and belongs to the field of image processing. In order to achieve image enhancement under various weather conditions, the present invention establishes a unified image enhancement scheme through a selectively fused neural network, which can simultaneously process source images under various weather conditions, achieve image clarity, and reduce the consumption of computing resources and storage resources; and for the first time, the selectively fused neural network and the adaptive filtering processing module are combined to further enhance the image, improve image details, alleviate local blur defects, and improve the quality of the enhanced source image at night. The present invention can be applied to image enhancement under various weather conditions at the same time, and the computing and storage resources are lower and the processing speed is faster.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image enhancement device and method, and a storage medium. Background Art

[0002] Image enhancement technology, as a key link in the field of machine vision, plays a vital role. It aims to restore the clarity of damaged images and highlight and strengthen key information. With the deepening of research, the performance of image enhancement solutions under different weather conditions has been significantly improved.

[0003] For foggy scenes, a series of mature image enhancement solutions have been developed, such as the traditional method based on dark channel prior (DCP), and the DehazeNet network and Generative Adversarial Networks (GANs) that integrate deep learning. The latter achieves efficient dehazing by learning complex image features.

[0004] Under rainy conditions, mainstream image enhancement solutions include rain spot removal solutions based on guided filtering, solutions that rely on prior knowledge, and strategies that use deep learning frameworks to automatically identify and correct the effects of rain.

[0005] Snow image enhancement can be achieved through dual-tree wavelet transform, which can effectively separate and process the snow cover layer. In addition, the preprocessing step combining semantic understanding and depth information further improves the recognizability of snow images.

[0006] As for night images, the enhancement technology involves a variety of classic schemes, such as histogram equalization, which is used to adjust the image contrast; the Retinex scheme, which simulates the adaptability of the human eye in low-light environments; and the dark channel prior scheme, which can improve image quality even under extreme lighting conditions.

[0007] The environmental characteristics of autonomous driving and railway stations are complex and changeable, which poses a challenge to the effective operation of the vision system. Although the above methods have shown satisfactory performance under specific meteorological conditions, their applicability is limited to a series of standard single weather conditions. However, in the process of deploying the station locomotive assisted driving system, for example, the system must have all-weather operation capabilities, that is, it must be able to properly cope with all possible meteorological conditions.

[0008] However, the existing technologies can only cover a single weather condition or only target a limited 2 to 3 weather conditions and are often ineffective when extended to more weather conditions. No technical solution that can simultaneously cover multiple weather conditions such as foggy days, rainy days, snowy days, and nights has been disclosed.

[0009] In order to ensure that the system can operate stably under different weather conditions, a simple and direct solution is to configure multiple sets of model parameters specifically for specific meteorological conditions, but this undoubtedly increases the computational complexity of the system and the consumption of storage resources, posing a technical challenge that needs to be solved urgently. In general, the basic technical problems that need to be solved in this field mainly include:

[0010] (1) Establish a unified image enhancement solution for various weather conditions such as fog, rain, snow, and night, and improve the solution's general application capabilities in real-world scenarios.

[0011] (2) Reduce the consumption of computing resources, storage resources, and / or the processing speed of image enhancement by the image enhancement scheme to avoid it becoming a delay bottleneck for post-processing. Summary of the invention

[0012] In order to alleviate or partially alleviate the above technical problems, the solution of the present invention is as follows:

[0013] In a first aspect, the present invention discloses an image enhancement device, comprising: an image acquisition device, configured to acquire a source image; a clarity processing module, configured to process the source image to obtain a cleared image; a filtering processing module, configured to process the cleared image to obtain a target enhanced image; wherein the clarity processing module comprises an N-layer neural network, the N-layer neural network comprises backbone neural network parameters, wherein the i-th layer neural network comprises the i-th layer backbone neural network parameters, N is a positive integer, and i is any positive integer not greater than N; wherein the backbone neural network parameters are obtained by jointly training the backbone neural network based on rainy day source images, foggy day source images, snowy day source images and night source images; and, at least one layer of the N-layer neural network only partially comprises the following characteristic network parameters about specific weather: characteristic network parameters about rainy days, characteristic network parameters about foggy days, characteristic network parameters about snowy days The invention relates to a method for preparing a filter according to the present invention, wherein the filter processing module comprises a hyperparameter estimation module and a hyperparameter application module; the filter processing module performs a downsampling operation on the cleared image and obtains the downsampled image, and uses the downsampled image as the input of the hyperparameter estimation module, and uses the cleared image as the input of the hyperparameter application module; the hyperparameter estimation module comprises a convolutional layer and a fully connected layer, and outputs the hyperparameters of the filter; the hyperparameter application module comprises a plurality of filters, and after applying the hyperparameters of the filter, the output of the hyperparameter application module is the target enhanced image.

[0014] Furthermore, when training the backbone neural network, the error of the backbone neural network with respect to the depth information is used as a part of the first loss function.

[0015] Furthermore, the characteristic network parameters about specific weather included in the N-layer neural network are obtained in the following manner: configuring the i-th layer neural network with a learnable scoring variable about specific weather, and adding the i-th layer characteristic network parameters about specific weather to the i-th layer neural network only when the learnable scoring variable about specific weather of the i-th layer neural network is greater than a threshold; and when training the N-layer neural network, adding the i-th layer characteristic network parameters about specific weather to the i-th layer neural network as part of the second loss function; wherein the characteristic network parameters about specific weather include: the i-th layer characteristic network parameters about specific weather, and i is any positive integer not greater than N.

[0016] Furthermore, the hyperparameter application module includes a white balance filter, a gamma filter, a hue filter, a contrast filter and a sharpening filter.

[0017] Furthermore, the tone mapping function of the tone filter is differentiable with respect to the input image and tone parameters of the tone filter; and the sharpening function of the sharpening filter is differentiable with respect to the position and the positive scale factor of the input image.

[0018] Furthermore, the hyperparameter estimation module includes 5 convolutional layers and 2 fully connected layers. The configuration parameters of the convolutional layers are: step size of 2, convolution kernel size of 3×3, activation function of ReLu function, and the number of output channels of the 5 convolutional layers are 16, 32, 32, 32, and 32 respectively.

[0019] In a second aspect, the present invention discloses an image enhancement method, comprising the following steps: calling an image acquisition device to acquire a source image; calling a clear processing module to process the source image to obtain a cleared image; calling a filtering processing module to process the cleared image to obtain a target enhanced image; wherein the clear processing module comprises an N-layer neural network, the N-layer neural network comprises backbone neural network parameters, wherein the i-th layer neural network comprises the i-th layer backbone neural network parameters, N is a positive integer, and i is any positive integer not greater than N; wherein the backbone neural network parameters are obtained by jointly training the backbone neural network based on rainy day source images, foggy day source images, snowy day source images and night source images; and, at least one layer of the N-layer neural network only partially comprises the following characteristic network parameters about specific weather: characteristic network parameters about rainy days, characteristic network parameters about foggy days, characteristic network parameters about snowy days Parameters and characteristic network parameters for night; wherein the characteristic network parameters for specific weather are trained according to source images of specific weather, which only include backbone neural network parameters and N-layer neural network composed of characteristic network parameters for specific weather, and only the characteristic network parameters for specific weather are modified during the training process to finally obtain the characteristic network parameters for specific weather; the filtering processing module includes a hyperparameter estimation module and a hyperparameter application module; the filtering processing module performs a downsampling operation after receiving the sharpened image and obtains the downsampled image, and uses the downsampled image as the input of the hyperparameter estimation module, and uses the sharpened image as the input of the hyperparameter application module; the hyperparameter estimation module includes a convolutional layer and a fully connected layer, and outputs the hyperparameters of the filter; the hyperparameter application module includes several filters, and after applying the hyperparameters of the filter, the output of the hyperparameter application module is the target enhanced image.

[0020] Furthermore, when training the backbone neural network, the error of the backbone neural network with respect to the depth information is used as a part of the first loss function.

[0021] Furthermore, the characteristic network parameters about specific weather included in the N-layer neural network are obtained in the following manner: configuring the i-th layer neural network with a learnable scoring variable about specific weather, and adding the i-th layer characteristic network parameters about specific weather to the i-th layer neural network only when the learnable scoring variable about specific weather of the i-th layer neural network is greater than a threshold; and when training the N-layer neural network, adding the i-th layer characteristic network parameters about specific weather to the i-th layer neural network as part of the second loss function; wherein the characteristic network parameters about specific weather include: the i-th layer characteristic network parameters about specific weather, and i is any positive integer not greater than N.

[0022] In a third aspect, the present invention discloses a storage medium storing source code or compiled code, and loading the compiled code or the compiled source code through a computing device to execute the image enhancement method as described in any of the preceding items.

[0023] The technical solution of the present invention has one or more of the following beneficial technical effects:

[0024] (1) Compared with traditional image enhancement schemes, the neural network-based scheme of the present invention can effectively improve the image enhancement processing speed.

[0025] (2) Compared with image enhancement schemes for a single type of weather conditions and enhancement schemes that can only be used for 2-3 types of weather conditions, the present invention can simultaneously provide image enhancement schemes for multiple types of severe weather conditions (including rainy days, foggy days, snowy days and nights, as well as combined rainy nights, foggy nights and snowy nights), with a wider scope of application and higher practical value.

[0026] (3) Compared with the image enhancement processing scheme that independently performs image enhancement through more than two neural networks, the unified image enhancement scheme constructed by the present invention and the selective fusion of neural networks avoids invalid computing power and invalid storage resource overhead, reduces the consumption of computing resources and storage resources, and further reduces processing delays.

[0027] (4) Through the adaptive filtering processing module, the defects of local blur and loss of image details caused by traditional dehazing and rain removal solutions can be avoided.

[0028] (5) For the first time, neural network-based image enhancement and adaptive filtering processing are combined together as a complete and coordinated image enhancement solution. The image enhancement effect is better than that of any single solution, it can effectively cope with a wider range of weather conditions, and it is more robust in practical applications.

[0029] In addition, other beneficial effects of the present invention will be mentioned in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of an image enhancement device of the present invention;

[0031] Figure 2 It is a schematic diagram of the conventional clarification scheme for different weather conditions;

[0032] Figure 3 It is a schematic diagram of the clarification scheme of the present invention for different weather conditions;

[0033] Figure 4 is a schematic diagram of the filtering processing module;

[0034] Figure 5 It is a schematic diagram of the hyperparameter application module;

[0035] Figure 6 Schematic diagram of the hyperparameter estimation module. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0038] Figure 1Schematic diagram of the image enhancement device of the present invention. Image acquisition devices, such as traditional frame image sensors and other photosensitive devices, collect image information in the environment to obtain source images. Affected by various weather conditions, such as snowy days, rainy days, foggy days, nights, etc., the quality of source images is poor. If these data are directly provided to post-processing solutions, such as Transformer, YOLO and other artificial neural networks, it is usually difficult to obtain ideal reasoning results, such as identifying pedestrians and vehicles in the field of view.

[0039] Therefore, the image enhancement device of the present invention can greatly reduce the impact of bad weather, especially after the source image collected in real time is enhanced, it is beneficial to various post-processing of the image and improve the overall processing effect. Unless otherwise specified, the various weather conditions and multiple weather conditions mentioned in the present invention refer to the four weather conditions of snowy days, rainy days, foggy days, and nights. The specific weather mentioned in the present invention refers to any one of snowy days, rainy days, foggy days, and nights.

[0040] The image enhancement device comprises a clearing processing module and a filtering processing module. The clearing processing module receives a source image and outputs a cleared image. The filtering processing module receives the cleared image and outputs a target enhanced image.

[0041] In the present invention, the clarity processing module includes N layers of neural networks, wherein the i-th layer of neural networks includes the i-th layer of backbone neural network parameters, N is a positive integer, and i is any positive integer not greater than N.

[0042] If neural networks are used to perform image enhancement, there are three main common approaches: (1) For different weather types, train a neural network for image enhancement under specific weather conditions separately, and use the matched trained neural network in the actual environment. (2) For different weather types, train a sub-neural network for image enhancement under specific weather conditions separately, and add a layer of trained neural network to the output layer. (3) For multiple weather types, train a neural network to handle multiple weather types. However, these solutions either consume a lot of computing and storage resources, or have poor versatility, or the final image enhancement effect is poor.

[0043] The present invention first uses a backbone neural network to learn the general characteristics of various weather conditions. In other words, the backbone neural network simultaneously learns the source images under various weather conditions to obtain the backbone neural network parameters under various weather conditions. For the backbone neural network, its forward propagation process can be described by the following function:

[0044] ;

[0045] in represents the backbone neural network, Represents the weather The source image collected at Represents the weather The source image collected at the time is restored through the backbone neural network, weather Specifically, in the present invention represents the rainy day source image, represents the foggy source image, represents the snow source image, represents the night source image, are the backbone neural network parameters, such as the weight values ​​representing synaptic weights.

[0046] Backbone neural network parameters Training or learning is required to obtain the optimal backbone neural network parameters, and then the backbone neural network can perform the expected reasoning work on the source image collected in real time based on the optimal backbone neural network parameters.

[0047] Furthermore, the backbone neural network parameters It is trained or learned based on source images under various weather conditions. The backbone neural network parameters obtained by learning source images under various weather conditions are shared network parameters in the overall solution, so they are recorded as Specifically, the backbone neural network parameters Based on rainy day source images , fog source image , Snow source images and night source images It is obtained by jointly training the backbone neural network.

[0048] The aforementioned N-layer neural network is denoted as L1, L2, ..., L i ,……,L N-1 , L N , they are respectively the 1st layer neural network, the 2nd layer neural network, ..., the i-th layer neural network, ..., the N-1th layer neural network, the N-th layer neural network, i and N are positive integers, and i is any positive integer not greater than N.

[0049] Backbone neural network parameters ={ },in The parameters of the backbone neural network of the i-th layer. In the present invention, each layer of the N-layer neural network includes a layer of corresponding backbone neural network parameters. In other words, the i-th layer neural network includes the i-th layer backbone neural network parameters, N is a positive integer, and i is any positive integer not greater than N, which reflects the backbone neural network parameters. shared features.

[0050] Furthermore, in order to alleviate the impact of bad weather conditions on the depth of the scene, the present invention also introduces depth information to train the backbone neural network. As an example, the first loss function used in the final backbone neural network training is for:

[0051] ;

[0052] in, represents the perceptual loss of the backbone neural network, Represents the weather The source image collected at the time is restored by the backbone neural network. Represents the weather The true value of the source image collected at the time; represents the balance weight, represents the 1-norm, which is equal to the sum of the absolute values ​​of the vector elements; represents a trained depth estimation network, which is another neural network different from the backbone neural network. In other words, when training the backbone neural network, the present invention uses the error of the backbone neural network with respect to the depth information as part of the first loss function.

[0053] So far, through the final backbone neural network and the first loss function , and mixed input data sets under various weather conditions, after training the backbone neural network, the backbone neural network parameters that make the backbone neural network run in the optimal state are obtained As for the specific training means, the techniques well known in the art can be used and will not be described in detail here.

[0054] Since in training the parameters of the backbone neural network When the source images under various weather conditions are input to the backbone neural network, the interference between different subtasks will reduce the performance of the backbone neural network in restoring the image. In this case, in order to learn the specific features of various weather conditions, the existing technology usually inserts the characteristic neural network parameters of various weather conditions into each layer of the neural network.

[0055] Figure 2 This is a schematic diagram of a conventional clearing solution for different weather conditions. If there is only a backbone neural network, the final image enhancement effect is usually not ideal when facing a certain type of weather condition.

[0056] An intuitive approach is to set the parameters of the backbone neural network in the i-th layer Insert information about specific weather The i-th layer feature network parameters ,Right now:

[0057] ;

[0058] in, is the i-th layer backbone neural network parameter, It's about specific weather The i-th layer feature network parameters; It's about specific weather The i-th layer neural network parameters.

[0059] The above solution is direct and concise, but it has great limitations due to the fixed parameters of the backbone neural network. Due to the existence of various weather conditions, a large number of network parameters will be introduced under a fixed expansion form, which will lead to a large overhead of computing resources and storage resources. Therefore, this solution still has shortcomings.

[0060] Figure 3 In order to realize the expansion of adaptive specific weather parameters under different weather conditions, in the first stage, the backbone neural network is used as the basic network. It is shared for different weather conditions.

[0061] In the present invention, at least one layer of the N-layer neural network only partially includes the following characteristic network parameters for specific weather: characteristic network parameters for rainy days, characteristic network parameters for foggy days, characteristic network parameters for snowy days, and characteristic network parameters for night. The characteristic network parameters for specific weather are obtained by training an N-layer neural network consisting of only backbone neural network parameters and characteristic network parameters for specific weather using source images of specific weather, and the backbone neural network parameters are not allowed to be modified during the training process, and only the characteristic network parameters for specific weather can be modified.

[0062] Specifically, in the second stage, the backbone neural network parameters is frozen, that is, it cannot be modified at will during the training phase, and on this basis, the introduction of specific weather The i-th layer feature network parameters and the i-th layer neural network about specific weather Learnable score variables , so that about a particular weather The parameters of the i-th layer neural network are ,in is the i-th layer of the neural network about a specific weather A learnable scoring variable used to evaluate the necessity of expanding the network parameters on the i-th layer of the neural network; It's about specific weather The i-th layer feature network parameters; is the parameter of the backbone neural network at the i-th layer; is an indicator function used to decide whether to add characteristic network parameters about specific weather, where τ is a threshold and also a hyperparameter. >τ, the characteristic network parameters related to specific weather are allowed to be added to the backbone neural network, and “*” represents multiplication.

[0063] When i traverses from 1 to N, about a specific weather The i-th layer feature network parameters The whole of the weather The characteristic network parameters Specifically, there are: characteristic network parameters for rainy days, characteristic network parameters for foggy days, characteristic network parameters for snowy days, and characteristic network parameters for night. The i-th layer feature network parameters About specific weather The characteristic network parameters .

[0064] In the second stage, the corresponding rainy day source images, foggy day source images, snowy day source images and night source images are used separately to train the parameters of the backbone neural network. and about specific weather The characteristic network parameters The N-layer neural network is composed of N layers, but during the training process, the backbone neural network parameters Cannot be modified, can only be modified or trained for specific weather The characteristic network parameters ,weather .

[0065] As an example, a convolution kernel is used as a characteristic network parameter about a specific weather and added to a backbone neural network. In one embodiment, the size of the convolution kernel can be 1×1 or 3×3.

[0066] Furthermore, when >τ, =1, otherwise it is equal to 0. Figure 3 About specific weather The i-th layer feature network parameters The dotted lines indicate that during the network reasoning process, these network parameters do not actually participate in the reasoning. The i-th layer feature network parameters If it does not involve reasoning, it can be obtained through training.

[0067] Accordingly, as an example, in the second stage, the second loss function for: ;

[0068] in, represents the loss of the backbone neural network, Represents the weather The source image collected at the time is restored by the backbone neural network. Represents the weather The true value of the source image collected at the time; is a hyperparameter of the regularization term; is the i-th layer of the neural network about a specific weather The learnable scoring variables are N, where N is the total number of layers of the neural network. In other words, the present invention will add the information about the specific weather in the i-th layer of the neural network. The i-th layer feature network parameters are used as part of the second loss function.

[0069] For example, It is a loss function based on the Mean Absolute Error (MAE).

[0070] The second loss function of the second stage above , which can guide the neural network to automatically discover and add certain weather-specific characteristic network parameters in certain layers. Experimental results show that the present invention can add relatively small weather-specific characteristic network parameters at the learned position, significantly improve the quality and effect of image enhancement, achieve effective and high-quality enhancement processing of source images under various weather conditions, and obtain a clear image.

[0071] Only using the above-mentioned clarity processing module including N layers of neural network can effectively improve the clarity of the image, and because the neural network adaptively learns to add characteristic network parameters of specific weather in different network layers, it has the characteristics of high efficiency and low resource consumption.

[0072] Although the above scheme has the advantages of fast processing speed and low resource consumption, the inventors used various weather condition source images (non-datasets) collected in practice to test the above N-layer neural network and found that the above scheme can achieve good image enhancement effects for rainy days, foggy days, and snowy days, and the peak signal-to-noise ratio can reach 20dB or even 30dB, but the image enhancement effect for night source images is not obvious. Based on the above N-layer neural network, the present invention introduces a filtering processing module to synergistically and significantly improve the image enhancement effect of night source images.

[0073] Figure 4Schematic diagram of a filtering processing module. For the cleared image, the present invention will use a filtering processing module including a plurality of adjustable filters to further perform filtering processing.

[0074] In the prior art, filters are usually adjusted manually based on visual inspection by engineers, which is obviously time-consuming, costly, and unreliable. To this end, the present invention uses a hyperparameter estimation module to automatically estimate the hyperparameters of the filter, and applies the estimated hyperparameters of the filter in the hyperparameter application module to obtain a template enhanced image.

[0075] In order to avoid the large amount of calculation required when estimating the hyperparameters of the filter by directly using the sharpened image, the present invention first downsamples the sharpened image to obtain a downsampled image. The resolution of the downsampled image is generally much lower than that of the sharpened image.

[0076] For example, the resolution of the sharpened image, such as 3000×4000, can be reduced to 256×256 by a bilinear interpolation method.

[0077] In the present invention, the hyper-parameter estimation of the filter is performed in the hyper-parameter estimation module through the down-sampled image.

[0078] Figure 5 Schematic diagram of the hyperparameter estimation module. As an example, the hyperparameter estimation module includes 5 convolutional layers and 2 fully connected layers, each convolutional layer includes the following configuration parameters: the stride is 2, the convolution kernel size is 3×3, the activation function is the ReLu function, and the number of output channels of the 5 convolutional layers is 16, 32, 32, 32, and 32 respectively.

[0079] Finally, the hyperparameters of the filter are output in the fully connected layer. In one example, when the number of hyperparameters of the filter is 15, the hyperparameter estimation module only includes 165,000 parameters, which often consumes less computing and storage resources to obtain the hyperparameters of the filter.

[0080] Figure 6 This is a schematic diagram of the hyperparameter application module. Through the hyperparameter application module, the potential information can be restored and the interference caused by weather conditions in the image can be suppressed.

[0081] For example, in the present invention, the super parameters of the filter involve white balance, gamma, hue, contrast and sharpening. In practical applications, only some or all of them may be adjusted, and the present invention is not limited to this.

[0082] The hyperparameter application module includes multiple filters. For example, the hyperparameter application module includes a white balance filter, a gamma filter, a hue filter, a contrast filter, and a sharpening filter. Among them, the first four filters can be implemented by pixel-level filters.

[0083] For example, the cleared image is sequentially passed through a white balance filter, a gamma filter, a hue filter, a contrast filter, and a sharpening filter to output a target enhanced image. The present invention is not limited to this sequence or combination and is not limited thereto.

[0084] For pixel-level filters, if the input pixel value , then the output pixel value after mapping ,in Represents the input pixel value The red, green and blue color channel values, Represents the output pixel value The red, green, and blue color channel values.

[0085] The hyperparameter application module receives the sharpened image, applies the hyperparameters of the filter, and outputs the target enhanced image.

[0086] Table 1: Parameters and mapping functions of pixel-level filters

[0087]

[0088] Regarding the contrast filter, for the function En(.):

[0089] ;

[0090] ;

[0091] ,in is the input pixel value.

[0092] For the hue filter, its mapping function is a monotone, piecewise linear function with L hue parameters: {t0, t1, …, t L-1}, where L is a positive integer. The points on the broken line in the mapping function of the hue filter can be expressed as (k / L, T k / T L ),in , k is any positive integer from 1 to L.

[0093] After the mapping of the above piecewise linear function, the pixel value is output ,in is the input pixel value The red channel value The mapped value, is the input pixel value The green channel value The mapped value, is the input pixel value The blue channel value Finally, like the mapping functions of other filters, the tone mapping function is related to the input image and tone parameters {t0, t1,..., t L-1 This ensures that the hyperparameter estimation module can perform gradient-based back-propagation optimization.

[0094] For the sharpening filter, the use of the sharpening filter can enhance the image details. In the present invention, the sharpening function The description is as follows:

[0095] ;

[0096] Where I(x) is the position An input image, such as the output from a contrast filter, is the function of the Gaussian filter, is a positive proportional factor. In the present invention, the sharpening function is about and are all differentiable, and the degree of sharpening can be adjusted by adjusting the proportional factor To achieve better object detection performance.

[0097] Through the combination of the clarity processing module and the filtering processing module, the present invention can handle a wider range of severe weather conditions. It is not only applicable to rainy days, foggy days, snowy days and nights, but also to combined rainy nights, foggy nights and snowy nights. The processed images will be clearer, and more importantly, it will occupy less storage resources and the image enhancement speed will be faster, which is of great value for fields that require high-speed image processing, such as autonomous driving.

[0098] In order to evaluate the technical effect of the present invention, the present invention selects two indicators, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), to measure. The former is mainly used to quantify the difference between the source image and the processed target enhanced image. The higher the PSNR value, the higher the image processing quality and the smaller the distortion. The unit is decibel (dB); the latter is an indicator for measuring the similarity of two images. The SSIM indicator pays more attention to the differences in image structure, brightness, contrast, etc., and can better reflect human subjective perception of image quality. The value range is between -1 and 1. The closer to 1, the more similar the two images are.

[0099] In addition to the public datasets PSCAL VOC, RTTS, and ExDark, the present invention receives images under normal lighting and weather conditions and images of foggy days and nights at the Tianjielou railway site, and divides them into training set, test set, and validation set in a ratio of 6:2:2.

[0100] Table 2: Image quality evaluation index table of the present invention

[0101]

[0102] It is not difficult to conclude from Table 2 that the present invention can effectively improve the image quality under various weather conditions, especially the quality of source images at night, which is of great value and significance for improving computer vision processing effects under various weather conditions, especially in the fields of assisted driving and autonomous driving.

[0103] In addition, the present invention also discloses a storage medium on which source code or compiled code is stored. The compiled code or the compiled source code is loaded through a computing device to execute the aforementioned image enhancement method.

[0104] The computing device may be an image enhancement device, such as a graphics processor, a field programmable gate array, an application specific integrated circuit, or the like.

[0105] Furthermore, in addition to loading the compiled code, network configuration parameters may be further loaded as necessary data for running the network.

[0106] In order to better illustrate the present invention, numerous specific details are provided in the above specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0107] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image enhancement device, characterized in that: include: An image acquisition device configured to acquire a source image; A sharpening processing module, configured to process the source image to obtain a sharpened image; The filtering processing module is configured to process the cleared image to obtain a target enhanced image; wherein, The clarification processing module includes an N-layer neural network, and the N-layer neural network includes backbone neural network parameters, wherein the I-th layer neural network includes the a-th layer backbone neural network parameters, N is a positive integer, and i is any positive integer not greater than N; The backbone neural network parameters are obtained by training the backbone neural network with rainy day source images, foggy day source images, snowy day source images and night source images; and At least one neural network layer in the N-layer neural network only partially includes the following characteristic network parameters about specific weather: characteristic network parameters about rainy days, characteristic network parameters about foggy days, characteristic network parameters about snowy days, and characteristic network parameters about night; The characteristic network parameters for the specific weather are obtained by training an N-layer neural network consisting of only backbone neural network parameters and characteristic network parameters for the specific weather according to a source image of the specific weather, and only modifying the characteristic network parameters for the specific weather during the training process; The filtering processing module includes a hyperparameter estimation module and a hyperparameter application module; the filtering processing module receives the sharpened image and performs a downsampling operation to obtain a downsampled image, and uses the downsampled image as an input to the hyperparameter estimation module, and uses the sharpened image as an input to the hyperparameter application module; The hyperparameter estimation module includes a convolutional layer and a fully connected layer, and outputs the hyperparameters of the filter; The hyperparameter application module includes a plurality of filters, and after applying the hyperparameters of the filters, the output of the hyperparameter application module is a target enhanced image.

2. The image enhancement device according to claim 1, characterized in that: When training the backbone neural network, the error of the backbone neural network with respect to the depth information is used as part of the first loss function.

3. The image enhancement device according to claim 1, characterized in that: The characteristic network parameters about specific weather included in the N-layer neural network are obtained in the following way: The i-th layer neural network is configured with a learnable scoring variable of the i-th layer neural network on specific weather, and only when the learnable scoring variable of the i-th layer neural network on specific weather is greater than a threshold value, the i-th layer feature network parameter on specific weather is added to the i-th layer neural network; And when training the N-layer neural network, adding the i-th layer feature network parameters related to the specific weather in the i-th layer neural network as part of the second loss function; The characteristic network parameters for specific weather include: characteristic network parameters of the i-th layer for specific weather, and i is any positive integer not greater than N.

4. The image enhancement device according to claim 3, characterized in that: The hyperparameter application module includes a white balance filter, a gamma filter, a hue filter, a contrast filter and a sharpening filter.

5. The image enhancement device according to claim 4, characterized in that: The tone mapping function of the tone filter is differentiable with respect to both the input image and the tone parameter of the tone filter; and, The sharpening function of the sharpening filter is differentiable with respect to both the position and the proportionality factor of the input image.

6. The image enhancement device according to claim 1, characterized in that: The hyperparameter estimation module includes 5 convolutional layers and 2 fully connected layers. The configuration parameters of the convolutional layers are: step size of 2, convolution kernel size of 3×3, activation function of ReLu function, and the number of output channels of the 5 convolutional layers are 16, 32, 32, 32, and 32 respectively.

7. An image enhancement method, characterized in that: The steps include: Calling an image acquisition device to acquire a source image; Calling a clearing processing module to process the source image to obtain a cleared image; The filtering processing module is called to process the cleared image to obtain a target enhanced image; wherein, The clarification processing module includes N layers of neural networks, and the N layers of neural networks include backbone neural network parameters, wherein the i-th layer of neural network includes the i-th layer of backbone neural network parameters, N is a positive integer, and i is any positive integer not greater than N; The backbone neural network parameters are obtained by training the backbone neural network with rainy day source images, foggy day source images, snowy day source images and night source images; and At least one neural network layer in the N-layer neural network only partially includes the following characteristic network parameters about specific weather: characteristic network parameters about rainy days, characteristic network parameters about foggy days, characteristic network parameters about snowy days, and characteristic network parameters about night; The characteristic network parameters for the specific weather are obtained by training an N-layer neural network consisting of only backbone neural network parameters and characteristic network parameters for the specific weather according to a source image of the specific weather, and only modifying the characteristic network parameters for the specific weather during the training process; The filtering processing module includes a hyperparameter estimation module and a hyperparameter application module; the filtering processing module receives the sharpened image and performs a downsampling operation to obtain a downsampled image, and uses the downsampled image as an input to the hyperparameter estimation module, and uses the sharpened image as an input to the hyperparameter application module; The hyperparameter estimation module includes a convolutional layer and a fully connected layer, and outputs the hyperparameters of the filter; The hyperparameter application module includes a plurality of filters, and after applying the hyperparameters of the filters, the output of the hyperparameter application module is a target enhanced image.

8. The image enhancement method according to claim 7, characterized in that: When training the backbone neural network, the error of the backbone neural network with respect to the depth information is used as part of the first loss function.

9. The image enhancement method according to claim 8, characterized in that: The characteristic network parameters about specific weather included in the N-layer neural network are obtained in the following way: The i-th layer neural network is configured with a learnable scoring variable of the i-th layer neural network on specific weather, and only when the learnable scoring variable of the i-th layer neural network on specific weather is greater than a threshold value, the i-th layer feature network parameter on specific weather is added to the i-th layer neural network; And when training the N-layer neural network, adding the i-th layer feature network parameters related to the specific weather in the i-th layer neural network as part of the second loss function; The characteristic network parameters for specific weather include: characteristic network parameters of the i-th layer for specific weather, and i is any positive integer not greater than N.

10. A storage medium having source code or compiled code stored thereon, characterized in that: The compiled code is loaded, or the compiled source code is loaded, through a computing device to execute the image enhancement method described in any one of claims 7 to 9.

Images