An image processing method and apparatus

Abstract

The present disclosure provides an image processing method and device. The method comprises: acquiring a to-be-processed image; determining a weather type corresponding to the to-be-processed image; acquiring an environmental noise feature corresponding to the weather type in the to-be-processed image according to the weather type; obtaining a non-noise image feature of the to-be-processed image according to the environmental noise feature; and generating a denoised image corresponding to the to-be-processed image according to the environmental noise feature, the non-noise image feature and the to-be-processed image. It can be seen that the present disclosure can solve the image quality degradation problem under more weather conditions using the same technical framework and improve the image denoising and enhancement effect.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to an image processing method and device. BACKGROUND

[0002] When an outdoor environment appears extreme weather such as heavy fog, heavy rain, heavy snow and the like, an image captured by an outdoor camera monitoring device has problems of unclear scene and lost image details, which limits the application of image recognition and video monitoring in the fields of traffic monitoring, target tracking, autonomous navigation and the like under extreme weather.

[0003] Existing image enhancement algorithms based on deep learning and physical modeling can only solve image quality degradation under a certain kind of extreme weather condition, such as can only realize image rain removal or can only realize image defogging, and cannot use the same technical framework to solve image quality degradation under more weather conditions; and in the process of denoising and restoring the image captured under extreme weather, there are often problems of unclear scene and lost image details, that is, it is difficult to completely remove noise such as rain lines, fog and snowflakes in the image, and the image restoration quality is poor. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide an image processing method, device, computer device and computer readable storage medium to solve the problem of poor image restoration quality in the prior art.

[0005] In a first aspect, an image processing method is provided, and the method comprises:

[0006] obtaining a to-be-processed image;

[0007] determining a weather type corresponding to the to-be-processed image;

[0008] obtaining an environmental noise feature corresponding to the weather type in the to-be-processed image according to the weather type;

[0009] obtaining a non-noise image feature of the to-be-processed image according to the environmental noise feature, and generating a denoised image corresponding to the to-be-processed image according to the environmental noise feature, the non-noise image feature and the to-be-processed image.

[0010] In a second aspect, an image processing device is provided, and the device comprises:

[0011] an image obtaining module configured to obtain a to-be-processed image;

[0012] a type determining module configured to determine a weather type corresponding to the to-be-processed image;

[0013] The feature acquisition module is configured to acquire, according to the weather type, an environmental noise feature corresponding to the weather type in the to-be-processed image.

[0014] The image generation module is configured to obtain a non-noise image feature of the to-be-processed image according to the environmental noise feature, and generate a denoised image corresponding to the to-be-processed image according to the environmental noise feature, the non-noise image feature and the to-be-processed image.

[0015] In a third aspect, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the above method when executing the computer program.

[0016] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0017] Compared with the prior art, the embodiments of the present disclosure have the beneficial effects that: the embodiments of the present disclosure can first acquire a to-be-processed image; then, a weather type corresponding to the to-be-processed image can be determined; then, according to the weather type, an environmental noise feature corresponding to the weather type in the to-be-processed image can be acquired; then, according to the environmental noise feature, a non-noise image feature of the to-be-processed image can be obtained, and according to the environmental noise feature, the non-noise image feature and the to-be-processed image, a denoised image corresponding to the to-be-processed image can be generated. Since the embodiments can acquire an environmental noise feature corresponding to different weather types, and can perform denoising processing on the to-be-processed image according to the environmental noise feature corresponding to the weather type, and can use a non-noise image feature to restore the image of the denoised region, the method provided by the embodiments can remove the noise caused by various weather types from the to-be-processed image, and can restore the image detail information while removing the noise caused by various weather types, thereby achieving the problem of solving more image quality degradation problems under different weather conditions using the same technical framework and improving the image denoising and enhancement effect. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1is a scene schematic diagram of an application scenario of an embodiment of the present disclosure;

[0020] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 3 is a network architecture schematic diagram of Resnet18 provided by an embodiment of the present disclosure;

[0022] Figure 4 is a network architecture schematic diagram of an image enhancement model provided by an embodiment of the present disclosure;

[0023] Figure 5 is a network architecture schematic diagram of a noise feature extraction module provided by an embodiment of the present disclosure;

[0024] Figure 6 is a network architecture schematic diagram of a feature aggregation dense convolution module provided by an embodiment of the present disclosure;

[0025] Figure 7 is a block diagram of an image processing apparatus provided by an embodiment of the present disclosure;

[0026] Figure 8 is a schematic diagram of a computer device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In the following description, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the present embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted in order to not obscure the description of the present disclosure with unnecessary detail.

[0028] An image processing method and apparatus according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] In the prior art, since the existing image enhancement algorithms based on deep learning and physical modeling can only solve image quality degradation under a certain extreme weather condition, such as only realizing image rain removal or only realizing image defogging, but cannot use the same technical framework to solve image quality degradation under more weather conditions; and in the process of denoising and restoring the image taken under extreme weather, the scene is often not clear and the image details are lost, that is, it is difficult to completely remove the noise such as rain lines, fog, snowflakes, etc. in the image, and the image restoration quality is poor.

[0030] To solve the above problems, the present application provides an image processing method. In the method, the embodiment can obtain the environmental noise characteristics corresponding to different weather types, and can perform denoising processing on the image to be processed according to the environmental noise characteristics corresponding to the weather type, and can use non-noise image features to restore the image of the denoised region. Therefore, the method provided by the embodiment can remove the noise caused by various weather types from the image to be processed, and can restore the image detail information while removing the noise caused by various weather types, thereby realizing the use of the same technical framework to solve the image quality degradation problem under more weather conditions and improving the image denoising and enhancement effect.

[0031] For example, the embodiment of the present application can be applied to the application scenario as shown in the figure. Figure 1 As shown in the figure, the application scenario can include a terminal device 1 and a server 2.

[0032] The terminal device 1 can be hardware or software. When the terminal device 1 is hardware, it can be various electronic devices with image acquisition and storage functions and support for communication with the server 2, including but not limited to smartphones, tablet computers, laptop computers, digital cameras, monitors, video recorders, and desktop computers, etc. When the terminal device 1 is software, it can be installed in the electronic devices as described above. The terminal device 1 can be implemented as multiple software or software modules, or as a single software or software module, and the present disclosure does not limit this. Further, the terminal device 1 can have various applications installed, such as image acquisition applications, image storage applications, instant messaging applications, etc.

[0033] The server 2 can be a server that provides various services, for example, a background server that receives requests sent by a terminal device that establishes a communication connection therewith. The background server can receive and analyze the request sent by the terminal device, etc., and generate a processing result. The server 2 can be a server, a server cluster composed of several servers, or a cloud computing service center, and the present disclosure does not limit this.

[0034] It should be noted that the server 2 can be hardware or software. When the server 2 is hardware, it can be various electronic devices that provide various services for the terminal device 1. When the server 2 is software, it can be multiple software or software modules that provide various services for the terminal device 1, or a single software or software module that provides various services for the terminal device 1, and the present disclosure does not limit this.

[0035] The terminal device 1 and the server 2 can be in communication connection through a network. The network can be a wired network using coaxial cable, twisted-pair wire and optical fiber connection, or a wireless network that can realize interconnection of various communication devices without wiring, for example, Bluetooth, Near Field Communication (NFC), Infrared, etc., and the present embodiment is not limited in this regard.

[0036] Specifically, the user can determine a to-be-processed image through the terminal device 1, and select to perform denoising processing on the noise in the to-be-processed image caused by weather. After the server 2 receives the to-be-processed image, the server 2 can first determine the weather type corresponding to the to-be-processed image. Then, the server 2 can obtain the environmental noise features corresponding to the weather type in the to-be-processed image according to the weather type. Next, the server 2 can obtain the non-noise image features of the to-be-processed image according to the environmental noise features, and generate a denoised image corresponding to the to-be-processed image according to the environmental noise features, the non-noise image features and the to-be-processed image. Finally, the server 2 can send the denoised image to the terminal device 1 so that the terminal device 1 shows the denoised image to the user. In this way, since the present embodiment can obtain the environmental noise features corresponding to different weather types, and can perform denoising processing on the to-be-processed image according to the environmental noise features corresponding to the weather type, and can use the non-noise image features to restore the image in the denoised region, the method provided by the present embodiment can remove the noise caused by various weather types from the to-be-processed image, and can restore the image detail information while removing the noise caused by various weather types, thereby realizing the use of the same technical framework to solve more image quality degradation problems under different weather conditions and improving the image denoising and enhancement effect.

[0037] It should be noted that the specific types, numbers and combinations of the terminal device 1, the server 2 and the network can be adjusted according to the actual needs of the application scenario, and the present embodiment is not limited in this regard.

[0038] It should be noted that the above application scenarios are only shown for the purpose of facilitating understanding of the present disclosure, and the embodiments of the present disclosure are not limited in this regard. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0039] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 2 The image processing method of the present embodiment can be executed by Figure 1 the terminal device or the server. As shown in Figure 2 , the image processing method comprises:

[0040] S201: Obtain an image to be processed.

[0041] In this embodiment, the image to be processed can be understood as an image or a video frame that needs to be denoised due to weather noise. For example, an image or a video frame of a video taken in extreme weather (such as heavy fog, heavy rain, heavy snow, etc.) can be used as the image to be processed. As an example, the terminal device can provide a page through which the user can upload an image and click a preset button to trigger denoising of the image due to weather noise. At this time, the image can be used as the image to be processed.

[0042] S202: Determine the weather type corresponding to the image to be processed.

[0043] Since the characteristics of noise caused by different weather are not the same, for example, the shape and distribution of noise caused by different weather types are not the same. For example, in an implementation, the weather type can include rain, snow, and fog; if the weather type is rain, the noise caused by the weather type is a rain line, and the shape of the rain line is a line, and the distribution of the rain line can include the direction of the rain line and the density of the rain line, wherein the density of the rain line can be divided into three categories of large density, medium density, and small density according to the density of the rain line, and the direction of the rain line can be understood as the direction of the rain corresponding to the rain line; if the weather type is snow, the noise caused by the weather type is snowflakes, and the shape of the snowflakes is a sheet, and the distribution of the snowflakes can include the direction of the snowflakes and the density of the snowflakes, wherein the density of the snowflakes can be divided into three categories of large density, medium density, and small density according to the density of the snowflakes, and the direction of the snowflakes can be understood as the direction of the snow corresponding to the snowflakes; if the weather type is fog, the noise caused by the weather type is fog, and the distribution of the fog can include the concentration of the fog, and the distribution of the fog is a random position. Therefore, after obtaining the image to be processed, the weather type corresponding to the image to be processed can be determined first, so that the environmental noise features can be extracted according to the characteristics of the noise corresponding to different weather types in the subsequent process.

[0044] In this embodiment, the image processing algorithm can be used to classify the weather type of the image to be processed to determine the weather type corresponding to the image to be processed. Of course, the neural network can also be used to classify the weather type of the image to be processed to determine the weather type corresponding to the image to be processed. It should be noted that other classification methods can also be used to classify the weather type of the image to be processed, which will not be described here.

[0045] S203: According to the weather type, obtain the environmental noise features corresponding to the weather type in the image to be processed.

[0046] In order to remove the noise caused by the weather in the to-be-processed image, the noise feature corresponding to the weather type of the to-be-processed image can be obtained according to the noise characteristics corresponding to the weather type, so that the noise corresponding to the weather type can be removed according to the noise feature corresponding to the weather type subsequently. It can be understood that the noise feature corresponding to a weather type can be understood as a feature that can reflect the distribution of the noise corresponding to the weather type in the to-be-processed image, for example, the noise feature corresponding to a weather type can include the shape, distribution density, distribution concentration, distribution area, and the like of the noise corresponding to the weather type in the to-be-processed image; it should be noted that the feature form of the noise feature can be a feature map, and of course, it can also be in the form of a feature matrix, which is not limited in the embodiment.

[0047] That is, in the embodiment, after determining the weather type of the to-be-processed image, the noise distribution information related to the noise characteristics corresponding to the weather type can be extracted in the to-be-processed image according to the noise characteristics corresponding to the weather type. Then, the noise feature corresponding to the weather type can be generated according to the noise distribution information.

[0048] For example, assuming that the weather type of the to-be-processed image is rain, the distribution information of the rain lines in the to-be-processed image, such as the shape, direction, density, and position of the rain lines, can be extracted, and the rain line noise feature can be generated according to the distribution information of the rain lines in the to-be-processed image. It can be understood that if the weather type is rain, the noise feature is the rain line noise feature.

[0049] For example, assuming that the weather type of the to-be-processed image is snow, the distribution information of the snowflakes in the to-be-processed image, such as the shape, direction, density, and position of the snowflakes, can be extracted, and the snowflake noise feature can be generated according to the distribution information of the snowflakes in the to-be-processed image. It can be understood that if the weather type is snow, the noise feature is the snowflake noise feature.

[0050] For example, assuming that the weather type of the to-be-processed image is fog, the distribution information of the fog in the to-be-processed image, such as the concentration and position of the fog, can be extracted, and the fog noise feature can be generated according to the distribution information of the fog in the to-be-processed image. It can be understood that if the weather type is fog, the noise feature is the fog noise feature.

[0051] S204: obtaining the non-noise image feature of the to-be-processed image according to the noise feature, and generating the denoised image corresponding to the to-be-processed image according to the noise feature, the non-noise image feature, and the to-be-processed image.

[0052] In this embodiment, after obtaining the environmental noise feature, the region of the to-be-processed image other than the region corresponding to the environmental noise feature can be taken as a non-noise image region, and feature extraction can be performed on the non-noise image region to obtain a non-noise image feature. It can be understood that the non-noise image feature is a feature capable of reflecting the details of the non-noise image region in the to-be-processed image, for example, the non-noise image feature can include texture features, edge features, color features, shape features, spatial features, and other feature information of the non-noise image region.

[0053] Then, the environmental noise feature can be used to remove the noise corresponding to the weather type in the to-be-processed image; for example, the environmental noise feature can be used to determine the position of the noise corresponding to the weather type in the to-be-processed image, and the pixel value of the pixel point corresponding to the position of the noise can be adjusted to a preset initial value. It should be noted that since the environmental noise feature is extracted based on the noise characteristics corresponding to the weather type, the environmental noise feature obtained can more clearly reflect the physical characteristics of the noise (such as rain lines, fog, and snowflakes) corresponding to the weather type in the to-be-processed image, and therefore, using the environmental noise feature to remove the noise corresponding to the weather type in the to-be-processed image can improve the removal effect of the noise corresponding to the weather type in the to-be-processed image. It should be further emphasized that since the environmental noise feature in the embodiment is extracted based on the noise characteristics corresponding to different weather types, the method provided in this embodiment can be compatible with removing noise caused by multiple weather types, that is, the method provided in this embodiment can be used to remove noise caused by each weather type, thereby realizing the use of the same technical framework to solve image quality degradation problems under more weather conditions.

[0054] Then, the non-noise image feature of the to-be-processed image can be used to restore the image detail information of the region after removing the noise. It should be noted that since the non-noise image feature includes the features of the details of the non-noise image region in the to-be-processed image, that is, the non-noise image feature can more clearly reflect the physical characteristics of the non-noise image region in the to-be-processed image; therefore, the feature relationship between the region after removing the noise and the non-noise image region can be calculated using the non-noise image feature, so that the region after removing the noise can be restored according to the non-noise image feature, to obtain a denoised image corresponding to the to-be-processed image, for example, the target pixel value of the pixel point in the region after removing the noise is calculated according to the non-noise image feature, and the pixel value of the pixel point in the region after removing the noise is adjusted to the target pixel value. In this way, the image detail restoration degree of the to-be-processed image after removing the noise can be improved, thereby reducing the distortion degree of the to-be-processed image after removing the noise, and further improving the clarity of the denoised image corresponding to the to-be-processed image.

[0055] It can be seen that the embodiment of the present disclosure has the beneficial effects compared with the prior art: the embodiment of the present disclosure can first acquire a to-be-processed image; then, a weather type corresponding to the to-be-processed image can be determined; then, according to the weather type, an environmental noise feature corresponding to the weather type in the to-be-processed image can be acquired; then, according to the environmental noise feature, a non-noise image feature of the to-be-processed image can be obtained, and according to the environmental noise feature, the non-noise image feature and the to-be-processed image, a denoised image corresponding to the to-be-processed image can be generated. Since the embodiment can acquire the environmental noise feature corresponding to different weather types, and can perform denoising processing on the to-be-processed image according to the environmental noise feature corresponding to the weather type, and can use the non-noise image feature to restore the image of the denoised region, the method provided by the embodiment can remove the noise caused by various weather types from the to-be-processed image, and can restore the image detail information while removing the noise caused by various weather types, thereby realizing the use of the same technical framework to solve the image quality degradation problem under more weather conditions and improving the image denoising and enhancement effect.

[0056] Next, an implementation of S202 will be introduced, that is, how to use a neural network to classify the weather type of the to-be-processed image. In the embodiment, S202“determining the weather type corresponding to the to-be-processed image” can include the following steps:

[0057] inputting the to-be-processed image into the trained weather type classification model to obtain the weather type corresponding to the to-be-processed image.

[0058] In the embodiment, the weather type classification model can be a residual network obtained by training based on sample images and weather type labels corresponding to the sample images. In an implementation, the residual network can be as follows Figure 3The Resent18 network is shown. In this embodiment, a plurality of sets of training samples can be pre-set, each set of training samples including a sample image and a weather type label corresponding to the sample image. The manner of training the residual network using the pre-set training samples can be: inputting the sample image in each set of training samples into the residual network to obtain a predicted weather type corresponding to the sample image, then determining a loss value according to the weather type label corresponding to the sample image and the predicted weather type, and then adjusting the network parameters of the residual network using the loss value, until the residual network meets a training condition, for example, the network parameters of the residual network are fitted, or the number of training times meets a pre-set training threshold, so that a trained weather type classification model can be obtained. For example, assuming that the pre-set weather types include rain, fog and snow, after inputting the image to be processed into the trained weather type classification model, the weather type of the image to be processed can be obtained as one of rain, fog and snow.

[0059] Next, an implementation of S203 will be introduced, that is, how to obtain the environmental noise feature corresponding to the weather type in the image to be processed. In this embodiment, S203“obtaining the environmental noise feature corresponding to the weather type in the image to be processed according to the weather type” can include the following steps:

[0060] S203a: determining an image enhancement model corresponding to the weather type according to the weather type; wherein the image enhancement model includes a noise feature extraction module.

[0061] In an implementation form of the present embodiment, different image enhancement models can be set for different weather types. It should be noted that the model network architectures of the image enhancement models corresponding to different weather types are all the same, and the difference is only that the training samples of the image enhancement models corresponding to different weather types are not the same. For example, it is assumed that the sample images in the training samples of the image enhancement model corresponding to the weather type "rain" are images with rain line noise, and the sample denoised images corresponding to the sample images are images with rain line noise removed and image details restored; it is assumed that the sample images in the training samples of the image enhancement model corresponding to the weather type "snow" are images with snowflake noise, and the sample denoised images corresponding to the sample images are images with snowflake noise removed and image details restored; it is assumed that the sample images in the training samples of the image enhancement model corresponding to the weather type "fog" are images with fog noise, and the sample denoised images corresponding to the sample images are images with fog noise removed and image details restored. It can be understood that the model network architectures of the image enhancement models corresponding to different weather types are all the same, but the model network parameters of the image enhancement models corresponding to different weather types can be different; it should be emphasized that since the subsequent implementation methods are introduced from the network architecture level, the subsequent implementation methods are applicable to the image enhancement models corresponding to all weather types. Of course, in another implementation form of the present embodiment, the same image enhancement model can be set for all weather types, that is, the same image enhancement model corresponds to all weather types, and it can be understood that in the present implementation form, the image enhancement model is trained based on the sample images corresponding to all weather types and the denoised images corresponding to the sample images. It should be noted that in the present embodiment, the case of setting different image enhancement models for different weather types will be exemplarily described.

[0062] In the case of setting different image enhancement models for different weather types, after obtaining the weather type of the to-be-processed image, the image enhancement model corresponding to the weather type can be determined from all the trained image enhancement models according to the weather type. As an example, the image enhancement model corresponding to each weather type is provided with a type identifier corresponding to the weather type, and therefore, after obtaining the weather type of the to-be-processed image, the image enhancement model with the same type identifier as the type identifier corresponding to the weather type can be determined from all the trained image enhancement models according to the type identifier corresponding to the weather type.

[0063] S203b: inputting the to-be-processed image into the noise feature extraction module to obtain the environmental noise feature corresponding to the weather type in the to-be-processed image.

[0064] In the present embodiment, as shown in FIG. 2, the image enhancement model corresponding to the weather type is determined from all the trained image enhancement models according to the weather type of the to-be-processed image, and then the to-be-processed image is input into the image enhancement model to obtain the denoised image corresponding to the to-be-processed image. Figure 4As shown, the image enhancement model can include a noise feature extraction module. After obtaining the to-be-processed image, the image enhancement model can be used to obtain the environmental noise feature corresponding to the weather type in the to-be-processed image, so as to subsequently remove the noise using the environmental noise feature.

[0065] In an implementation manner, as shown in Figure 4 The noise feature extraction module can be a multi-scale feature extraction network module (Feature Block). The process of extracting the environmental noise feature by the noise feature extraction module can be defined as F0=H0(I0), I0 represents the to-be-processed image, H0 represents a feature calculation function of the Feature Block, and F0 is the obtained environmental noise feature.

[0066] In order to be able to extract the features of noises of different shapes and different distribution modes (such as rain line noise features of different directions and different sizes, fog noise features of different concentrations, and snowflake noise features of different directions and different sizes), the noise feature extraction module can include N dilated convolution layers and an aggregation convolution layer, wherein the dilation sizes of the convolution kernels of each dilated convolution layer are different, and N is a positive integer greater than 1. It should be noted that, since the dilation sizes of the convolution kernels of each dilated convolution layer are different, the number of interval pixels between adjacent pixels collected in the convolution calculation process of each dilated convolution layer is different, so the size of the receptive field of each dilated convolution layer is also different, and thus the first noise features extracted by each dilated convolution layer are also different (such as different sizes, and different image feature information). In this way, first noise features of different scales, such as rain lines of different sizes and fog of different concentrations, can be extracted, more context spatial information can be aggregated in the training process, and fewer parameters are trained, so that the model is easier to train to fit.

[0067] Specifically, in the present implementation manner, the to-be-processed image can be input into the N dilated convolution layers respectively to obtain N first noise features, wherein the sizes of the first noise features are different. Then, the N first noise features can be aggregated by using the aggregation convolution layer to obtain the environmental noise feature corresponding to the weather type in the to-be-processed image. Since the image feature information contained in the first noise features of different scales is different, more context spatial information can be aggregated in the process of obtaining the environmental noise feature by aggregating the N first noise features, and the information in the aggregated environmental noise feature is more abundant, so that the extraction of the noise features in the to-be-processed image can be significantly improved, that is, the obtained environmental noise feature can more clearly reflect the noise corresponding to the weather type in the to-be-processed image.

[0068] Next, the implementation manner of the noise feature extraction module will be described in combination withFigure 5 The specific implementation of S203b is illustrated with an example. For example... Figure 5 As shown, the noise feature extraction module may include four dilated convolutional layers and three aggregated convolutional layers; specifically, the four dilated convolutional layers include a first dilated convolutional layer (Conv 3*3DF=1), a second dilated convolutional layer (Conv 3*3DF=2), a third dilated convolutional layer (Conv 3*3DF=3), and a fourth dilated convolutional layer (Conv 3*3DF=4), and the aggregated convolutional layers include a first aggregated convolutional layer (Conv 1*1DF=1), a second aggregated convolutional layer (Conv1*1DF=1), and a third aggregated convolutional layer (Conv 1*1DF=1). 1*1DF=1); where, the first dilated convolutional layer is a convolutional layer with a kernel of 3*3 and DF=1 (DF=1 means that there is no gap between adjacent pixels during the convolution calculation process), the second dilated convolutional layer is a convolutional layer with a kernel of 3*3 and DF=2 (DF=2 means that the number of gaps between adjacent pixels during the convolution calculation process is one pixel), the third dilated convolutional layer is a convolutional layer with a kernel of 3*3 and DF=3 (DF=3 means that the number of gaps between adjacent pixels during the convolution calculation process is two pixels), the fourth dilated convolutional layer is a convolutional layer with a kernel of 3*3 and DF=4 (DF=4 means that the number of gaps between adjacent pixels during the convolution calculation process is three pixels), and the first aggregated convolutional layer, the second aggregated convolutional layer, and the third aggregated convolutional layer are all convolutional layers with a kernel of 1*1 and DF=1. Specifically, the image to be processed is input into a first dilated convolutional layer, a second dilated convolutional layer, a third dilated convolutional layer, and a fourth dilated convolutional layer, respectively, to obtain four first noise features. The two first noise features output from the first and second dilated convolutional layers are input into a first aggregation convolutional layer to obtain a first sub-aggregated feature; the two first noise features output from the third and fourth dilated convolutional layers are input into a second aggregation convolutional layer to obtain a second aggregated sub-feature; the first and second sub-aggregated features are input into a third aggregation convolutional layer to obtain the environmental noise feature in the image to be processed corresponding to the weather type. It should be noted that in this example, a 1*1 convolutional layer is used to aggregate features at different scales. The feature information (i.e., environmental noise features) obtained after three feature aggregations is more comprehensive, significantly improving the extraction of environmental noise features from weather images.

[0069] Next, we will introduce one implementation of S204, namely, how to generate a denoised image corresponding to the image to be processed. In this embodiment, S204, "obtaining the non-noise image features of the image to be processed based on the environmental noise features, and generating a denoised image corresponding to the image to be processed based on the environmental noise features, the non-noise image features, and the image to be processed," may include the following steps:

[0070] S204a: input the ambient noise feature into the convolution layer to obtain a first feature map.

[0071] In this embodiment, as shown in Figure 4 The image enhancement model further includes a convolution layer. After obtaining the ambient noise feature, the ambient noise feature is input into the convolution layer for convolution processing to obtain a first feature map.

[0072] S204b: input the first feature map into the encoding network layer to obtain P down-sampling feature maps and P non-noise image features, wherein P is a positive integer greater than 1.

[0073] In this embodiment, as shown in Figure 4 The image enhancement model further includes an encoding network layer. The encoding network layer can include P cascaded encoding network modules, and each encoding network module includes a feature aggregation dense convolution module and a max-pooling layer; in an implementation manner, as shown in Figure 4 One encoding network module can include a feature aggregation dense convolution module and a max-pooling layer, for example, the feature aggregation dense convolution module can be FJDB (Feature Joint Dense Block) in Figure 4 , and the max-pooling layer can be Maxpooling in Figure 4 .

[0074] In this embodiment, in the encoding stage, the first feature map can be input into the encoding network layer, and the P cascaded encoding network modules of the encoding network layer are used to extract features from the first feature map to obtain P down-sampling feature maps and P non-noise image features; it should be noted that the P cascaded encoding network modules of the encoding network layer can be understood as down-sampling convolution layers, and the sizes of the feature maps output by the P cascaded encoding network modules are different, and the size of the feature map output by the encoding network module with a later order is smaller. The down-sampling feature map can be understood as the sum of the same scale features, and the down-sampling feature map can be used to remove noise caused by weather; the non-noise image feature is the image details lost in the pooling stage (i.e., the feature extraction stage of the encoding network module) recorded by the pooling index (Pooling Indices), and the non-noise image feature can be used to guide the recovery of the details features lost in the feature extraction stage of the encoding network module in the up-sampling stage (i.e., the subsequent decoding stage).

[0075] Specifically, the first feature map can be input into a feature aggregation dense convolution module in the first encoding network module to obtain a down-sampling feature map and a non-noise image feature output by a max-pooling layer in the first encoding network module. The down-sampling feature map output by the i-1th encoding network module can be input into a feature aggregation dense convolution module in the i th encoding network module to obtain a down-sampling feature map and a non-noise image feature output by a max-pooling layer in the i th encoding network module; i is a positive integer greater than 1 and less than or equal to P.

[0076] As an example, as shown in FIG. 1, the encoding network layer can include three cascaded encoding network modules, which are a first encoding network module, a second encoding network module and a third encoding network module respectively; and each encoding network module includes a feature aggregation dense convolution module (FIDB) and a max-pooling layer. The first feature map F1 is input into the first encoding network module, and the first encoding network module outputs a down-sampling feature map F2 and a non-noise image feature; the down-sampling feature map F2 output by the first encoding network module is input into the feature aggregation dense convolution module in the second encoding network module to obtain a down-sampling feature map F3 and a non-noise image feature output by the max-pooling layer in the second encoding network module; the down-sampling feature map F3 output by the second encoding network module is input into the feature aggregation dense convolution module in the third encoding network module to obtain a down-sampling feature map F4 and a non-noise image feature output by the max-pooling layer in the third encoding network module. Figure 4 S205c: inputting the P down-sampling feature maps and the P non-noise image features into a decoding network layer to obtain a denoising feature map;

[0077] In the embodiment, as shown in FIG. 1, the image enhancement model further includes a decoding network layer. The decoding network layer includes P cascaded decoding network modules, and each decoding network module includes a feature aggregation dense convolution module and an up-sampling max-pooling layer; in an implementation manner, as shown in FIG. 1, one decoding network module can include one feature aggregation dense convolution module and one up-sampling max-pooling layer, for example, the feature aggregation dense convolution module can be FIDB (Feature Joint Dense Block) in FIG. 1, and the up-sampling max-pooling layer can be UpMaxpooling in FIG. 1.

[0078] Figure 4 Figure 4 Figure 4 Figure 4

[0079] ​​​​​In the decoding stage, the image to be processed, the P down-sampling feature maps and the P non-noise image features can be input into the P cascaded decoding network modules of the decoding network layer, so that the P cascaded decoding network modules remove the weather type corresponding noise from the image to be processed by using the P down-sampling feature maps, and obtain the de-noised feature map by using the P non-noise image features and the recovery of the image detail information. It should be noted that the P cascaded decoding network modules of the decoding network layer can be understood as up-sampling convolutional layers, and the sizes of the feature maps output by the P cascaded decoding network modules are different, and the size of the feature map output by the decoding network module at the later stage is larger. It can be seen that, by integrating the encoding network layer and the decoding network layer together, the long-term spatial feature dependency relationship can be calculated. The pooling indices are used to record the loss and supplement of the image detail information in the down-sampling stage (i.e. the encoding stage) and the up-sampling stage (i.e. the decoding stage), each Maxpooling layer in the encoding stage corresponds to an UpMaxpooling layer, and the Maxpooling layer guides the UpMaxpooling layer to perform up-sampling through the pooling indices, so that more image details can be recovered in the up-sampling stage.

[0080] Specifically, the down-sampling feature map output by the Pth encoding network module can be input into the feature aggregation dense convolution module in the 1st decoding network module, and the non-noise image feature output by the Pth encoding network module can be input into the UpMaxpooling layer in the 1st decoding network module, to obtain an up-sampling feature map output by the UpMaxpooling layer. The down-sampling feature map output by the P-jth encoding network module and the up-sampling feature map output by the UpMaxpooling layer in the jth decoding network module can be input into the feature aggregation dense convolution module in the 1+jth decoding network module, and the non-noise image feature output by the P-jth encoding network module can be input into the UpMaxpooling layer in the 1+jth decoding network module, to obtain an up-sampling feature map output by the UpMaxpooling layer in the 1+jth decoding network module; j is a positive integer equal to or greater than 1 and less than P. The up-sampling feature map output by the UpMaxpooling layer in the Pth decoding network module can be used as the de-noised feature map.

[0081] As an example, as Figure 4As shown, the encoding network layer can include three cascaded encoding network modules, which are a first encoding network module, a second encoding network module and a third encoding network module respectively; the decoding network layer can include three cascaded decoding network modules, which are a first decoding network module, a second decoding network module and a third decoding network module respectively. The down-sampling feature map F4 output by the third encoding network module can be input into the feature aggregation dense convolution module in the first decoding network module, and the non-noise image feature output by the third encoding network module can be input into the up-sampling max-pooling layer in the first decoding network module, so as to obtain an up-sampling feature map F5 output by the up-sampling max-pooling layer. The down-sampling feature map F3 output by the second encoding network module and the up-sampling feature map F5 output by the up-sampling max-pooling layer in the first decoding network module can be input into the feature aggregation dense convolution module in the second decoding network module; and the non-noise image feature output by the second encoding network module can be input into the up-sampling max-pooling layer in the second decoding network module, so as to obtain an up-sampling feature map F6 output by the up-sampling max-pooling layer in the second decoding network module. The down-sampling feature map F2 output by the first encoding network module and the up-sampling feature map F6 output by the up-sampling max-pooling layer in the second decoding network module can be input into the feature aggregation dense convolution module in the third decoding network module; and the non-noise image feature output by the first encoding network module can be input into the up-sampling max-pooling layer in the third decoding network module, so as to obtain an up-sampling feature map F7 output by the up-sampling max-pooling layer in the third decoding network module, which can be taken as the denoising feature map.

[0082] S206d: inputting the denoising feature map, the first feature map, the environmental noise feature and the to-be-processed image into the output layer to obtain a denoised image corresponding to the to-be-processed image.

[0083] In this embodiment, as shown in Figure 6 The image enhancement model further includes an output layer. The denoising feature map, the first feature map, the environmental noise feature and the to-be-processed image are input into the output layer to obtain a denoised image corresponding to the to-be-processed image. For example, the output layer can fuse the denoising feature map, the first feature map, the environmental noise feature and the to-be-processed image to obtain a denoised image corresponding to the to-be-processed image.

[0084] In an implementation manner, as shown in Figure 6As shown, the output layer includes a first output layer and a second output layer, wherein the first output layer includes a convolution layer (1*1Conv) with a convolution kernel size of 1*1 and a convolution layer (Conv), and the second output layer includes a convolution layer (Conv) and an activation function layer (such as a Tanh function layer, i.e., Tanh). Wherein the input of the first output layer is the denoising feature map and the first feature map, and the output of the first output layer is the first output feature map; the input of the second output layer is the fusion feature of the first output feature map and the environmental noise feature F0, and the output of the second output layer is the second output feature map; then, the second output feature map can be fused with the to-be-processed image to obtain a denoised image corresponding to the to-be-processed image.

[0085] It should be noted that the feature aggregation dense convolution module mentioned in the above embodiments all include M dilated convolution layers, a dense connection network module and a full connection layer. Wherein the convolution kernel dilation size of each dilated convolution layer is different, and M is a positive integer greater than 1.

[0086] The M dilated convolution layers are used to generate M second noise features according to the feature map (such as the first feature map, the down-sampling feature map, the up-sampling feature map), wherein the size of each second noise feature is different; specifically, the feature map can be input into the M dilated convolution layers respectively to obtain the M second noise features. For example, as shown in FIG. 4, the first feature map is input into the M dilated convolution layers to obtain the M second noise features. Figure 6As shown, one feature aggregation dense convolution module can include 3 dilated convolution layers, and the three dilated convolution layers are respectively a first sub-dilated convolution layer (Conv 3*3 DF=1), a second sub-dilated convolution layer (Conv 3*3 DF=2), and a third sub-dilated convolution layer (Conv 3*3 DF=3). The first sub-dilated convolution layer is a dilated convolution layer with a convolution kernel of 3*3 and DF=1 (DF=1 represents that there is no interval between adjacent pixels collected in the convolution calculation process). The second sub-dilated convolution layer is a dilated convolution layer with a convolution kernel of 3*3 and DF=2 (DF=2 represents that the interval number between adjacent pixels collected in the convolution calculation process is one pixel). The third sub-dilated convolution layer is a dilated convolution layer with a convolution kernel of 3*3 and DF=3 (DF=3 represents that the interval number between adjacent pixels collected in the convolution calculation process is two pixels). In this embodiment, three dilated convolution layers with different convolution kernel sizes are used to aggregate feature information (i.e., feature maps) of different scales. The convolution kernel with a size of 3*3 can well extract noise features such as rain lines, fog, and snowflakes. In addition, the number of channels of the feature maps is kept consistent in the encoding stage and the decoding stage, that is, the number of channels of the feature maps output in the encoding network layer and the decoding network layer is the same. It can be seen that, in this embodiment, different dilated convolution layers with different convolution kernel sizes are used to aggregate features of different scales, which can improve the richness of the extracted features. It should be noted that, for noise features such as rain lines, snowflakes, and fog, which have irregular shapes and change with the wind direction, the use of multi-scale aggregation feature technology can improve the effect of extracting noise features. For example, the extracted features can be richer than those extracted by a single size convolution kernel, so that the obtained environmental noise features can more clearly reflect the physical characteristics of the noise (such as rain lines, fog, and snowflakes) corresponding to the weather type in the image to be processed, thereby improving the clarity of the denoised image corresponding to the image to be processed.

[0087] The dense connection network module is configured to perform convolution calculation and data filtering on the M second noise features to obtain a plurality of convolution feature maps. In one implementation manner, as shown in FIG. 6, the dense connection network module includes a plurality of dense connection modules, and each dense connection module includes a convolution layer (Conv) and an activation function layer (such as a ReLu function layer, i.e., ReLu). Figure 6

[0088] The full connection layer is configured to aggregate the M second noise features and the plurality of convolution feature maps to obtain aggregated features. In one implementation manner, as shown in FIG. 6, the full connection layer includes a plurality of full connection layers, and each full connection layer is configured to aggregate a plurality of convolution feature maps to obtain a plurality of aggregated features. Figure 6 ​As shown, the fully connected layer includes a concat function layer (i.e., Concat) and a convolutional layer with a 1*1 kernel (1*1Conv).

[0089] Combination Figure 7 For example, such as Figure 7 As shown, the densely connected network module includes three densely connected modules: the first densely connected module, the second densely connected module, and the third densely connected module. The fused feature obtained by fusing M second noise features can be used as the input to the first densely connected module. The fused feature obtained by fusing M second noise features and the convolutional feature map output by the first densely connected module can be used as the input to the second densely connected module. The fused feature obtained by fusing M second noise features, the convolutional feature map output by the first densely connected module, and the convolutional feature map output by the second densely connected module can be used as the input to the third densely connected module. The fused feature obtained by fusing M second noise features, the convolutional feature map output by the first densely connected module, the convolutional feature map output by the second densely connected module, and the convolutional feature map output by the third densely connected module can be used as the input to the fully connected layer. The feature map output by the fully connected layer can be fused with the feature maps (e.g., the first feature map, the downsampled feature map, and the upsampled feature map) of the input M dilated convolutional layers (i.e., aggregation processing) to obtain the aggregated feature.

[0090] As can be seen, in this embodiment, after fusing M second noise features of different scales, the fused feature obtained by fusing the M second noise features is input into a densely connected network module. Then, the features of all stages are aggregated through the Concat layer in the fully connected layer. This enables the feature information in the downsampled feature map and non-noise image features obtained in the encoding stage to be depicted in more detail. It also enables the removal of noise information (i.e., noise features) such as rain lines, fog, and snowflakes in the decoding stage to restore more image details, thereby reducing the distortion of the denoised image corresponding to the image to be processed.

[0091] It should be noted that the loss function of the image enhancement model can be a mean absolute error function. It should be noted that in this embodiment, since the noise information (i.e. noise features) such as rain lines, fog, snowflakes, etc. is relatively sparse, and this embodiment describes the difference between the predicted denoising image f(x) and the real noise-free image Y after removing the noise information such as rain lines, fog, snowflakes, etc., therefore this embodiment can select the MAE (Mean Absolute Error) function which is more sensitive to sparse features as the loss function for training. In this way, after determining the loss value corresponding to the input image of the model for model training by using the loss function, the model parameters of the image enhancement model can be adjusted by using the loss value until the training completion condition is met, for example, the training frequency reaches the preset frequency or the model parameters of the image enhancement model are fitted.

[0092] In an implementation manner, the MAE function is as shown in the following formula:

[0093]

[0094] In the above formula, represents the predicted denoising image output by the image enhancement model; H, W and C respectively represent the height, width and channel number of the input image of the image enhancement model; Y i，j，k represents the real noise-free image; i, j and k respectively represent the height, width and channel number of the image; and L represents the loss value.

[0095] All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0096] The following is an apparatus embodiment of the present disclosure, which can be used to execute the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0097] Figure 8 is a schematic diagram of an image processing apparatus provided by an embodiment of the present disclosure. As shown in Figure 8 , the image processing apparatus comprises:

[0098] An image acquisition module 701 is configured to acquire a to-be-processed image.

[0099] A type determination module 702 is configured to determine a weather type corresponding to the to-be-processed image.

[0100] A feature acquisition module 703 is configured to acquire, according to the weather type, an environmental noise feature corresponding to the weather type in the to-be-processed image.

[0101] The image generation module 704 is configured to obtain a non-noise image feature of the to-be-processed image according to the environmental noise feature, and generate a denoised image corresponding to the to-be-processed image according to the environmental noise feature, the non-noise image feature and the to-be-processed image.

[0102] In some embodiments, the type determination module 702 is configured to:

[0103] input the to-be-processed image into a trained weather type classification model to obtain a weather type corresponding to the to-be-processed image.

[0104] The weather type classification model is a residual network trained based on sample images and weather type labels corresponding to the sample images.

[0105] In some embodiments, the feature acquisition module 703 is configured to:

[0106] determine an image enhancement model corresponding to the weather type according to the weather type, wherein the image enhancement model comprises a noise feature extraction module.

[0107] input the to-be-processed image into the noise feature extraction module to obtain an environmental noise feature corresponding to the weather type in the to-be-processed image, wherein the environmental noise feature is a feature reflecting a distribution of environmental noise corresponding to the weather type in the to-be-processed image.

[0108] In some embodiments, the noise feature extraction module comprises N dilated convolution layers and an aggregation convolution layer, wherein a convolution kernel dilation size of each dilated convolution layer is different, and N is a positive integer greater than 1.

[0109] The feature acquisition module 703 is specifically configured to:

[0110] input the to-be-processed image into the N dilated convolution layers respectively to obtain N first noise features, wherein a size of each first noise feature is different.

[0111] perform aggregation processing on the N first noise features by using the aggregation convolution layer to obtain the environmental noise feature corresponding to the weather type in the to-be-processed image.

[0112] In some embodiments, the image enhancement model further comprises a convolution layer, an encoding network layer, a decoding network layer and an output layer.

[0113] The image generation module 704 is configured to:

[0114] input the environmental noise feature into the convolution layer to obtain a first feature map.

[0115] input the first feature map into the encoding network layer to obtain P down-sampling feature maps and P non-noise image features, wherein P is a positive integer greater than 1;

[0116] input the P down-sampling feature maps and the P non-noise image features into a decoding network layer to obtain a denoising feature map;

[0117] input the denoising feature map, the first feature map, the environmental noise feature and the to-be-processed image into the output layer to obtain a denoising image corresponding to the to-be-processed image.

[0118] In some embodiments, the encoding network layer includes P cascaded encoding network modules, and each encoding network module includes a feature aggregation dense convolution module and a max-pooling layer;

[0119] The image generation module 704 is specifically configured to:

[0120] input the first feature map into the feature aggregation dense convolution module in the first encoding network module to obtain a down-sampling feature map and a non-noise image feature output by the max-pooling layer in the first encoding network module;

[0121] input the down-sampling feature map output by the i-1th encoding network module into the feature aggregation dense convolution module in the i th encoding network module to obtain a down-sampling feature map and a non-noise image feature output by the max-pooling layer in the i th encoding network module; i is a positive integer greater than 1 and less than or equal to P.

[0122] In some embodiments, the decoding network layer includes P cascaded decoding network modules, and each decoding network module includes a feature aggregation dense convolution module and an up-sampling max-pooling layer;

[0123] The image generation module 704 is specifically configured to:

[0124] input a down-sampling feature map output by the P th encoding network module into the feature aggregation dense convolution module in the first decoding network module, and input a non-noise image feature output by the P th encoding network module into the up-sampling max-pooling layer in the first decoding network module to obtain an up-sampling feature map output by the up-sampling max-pooling layer;

[0125] inputting the down-sampling feature map output by the P-jth encoding network module and an up-sampling feature map output by an up-sampling max-pooling layer in the jth decoding network module into a feature aggregation dense convolution module in the 1+jth decoding network module; and inputting a non-noise image feature output by the P-jth encoding network module into the up-sampling max-pooling layer in the 1+jth decoding network module to obtain an up-sampling feature map output by the up-sampling max-pooling layer in the 1+jth decoding network module; j is a positive integer equal to or greater than 1 and less than P;

[0126] the up-sampling feature map output by the up-sampling max-pooling layer in the Pth decoding network module is taken as a de-noising feature map.

[0127] In some embodiments, the feature aggregation dense convolution module comprises M dilated convolution layers, a dense connection network module and a fully connected layer; wherein the dilation size of the convolution kernel of each dilated convolution layer is different, and M is a positive integer greater than 1.

[0128] The M dilated convolution layers are configured to generate M second noise features from the feature map, wherein the size of each second noise feature is different.

[0129] The dense connection network module is configured to perform convolution calculation processing and data screening processing on the M second noise features to obtain a plurality of convolution feature maps.

[0130] The fully connected layer is configured to perform aggregation processing on the M second noise features and the plurality of convolution feature maps to obtain an aggregated feature.

[0131] In some embodiments, the loss function of the image enhancement model is a mean absolute error function.

[0132] In some embodiments, the weather type comprises at least one of rain, snow and fog.

[0133] If the weather type is rain, the environmental noise feature is a rain line noise feature; if the weather type is snow, the environmental noise feature is a snowflake noise feature; and if the weather type is fog, the environmental noise feature is a fog noise feature.

[0134] According to the technical scheme provided by the embodiment of the present disclosure, the image processing device comprises: an image acquisition module, configured to acquire a to-be-processed image; a type determination module, configured to determine a weather type corresponding to the to-be-processed image; a feature acquisition module, configured to acquire an environmental noise feature corresponding to the weather type in the to-be-processed image according to the weather type; and an image generation module, configured to obtain a non-noise image feature of the to-be-processed image according to the environmental noise feature, and generate a denoised image corresponding to the to-be-processed image according to the environmental noise feature, the non-noise image feature and the to-be-processed image. Since the embodiment can acquire the environmental noise feature corresponding to different weather types, and can perform denoising processing on the to-be-processed image according to the environmental noise feature corresponding to the weather type, and can use the non-noise image feature to restore the image of the denoised region, the method provided by the embodiment can remove the noise caused by various weather types from the to-be-processed image, and can restore the image detail information while removing the noise caused by various weather types, so that the same technical framework can be used to solve the image quality degradation problem under more weather conditions, and the image denoising and enhancement effect is improved.

[0135] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present disclosure.

[0136] Figure 8 is a schematic diagram of the computer device 8 provided by the embodiment of the present disclosure. As shown in ​ the computer device 8 of the embodiment comprises a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. The processor 801 implements the steps in each of the above method embodiments when executing the computer program 803. Alternatively, the processor 801 implements the functions of each module / module in each of the above device embodiments when executing the computer program 803.

[0137] Exemplarily, the computer program 803 can be divided into one or more modules, which are stored in the memory 802 and executed by the processor 801 to complete the present disclosure. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 803 in the computer device 8.

[0138] The computer device 8 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device 8 can include, but is not limited to, a processor 801 and a memory 802. Those skilled in the art can understand that ​ The computer device 8 is only an example and does not constitute a limitation on the computer device 8, and can include more or fewer components than shown, or combine certain components, or different components, for example, the computer device can also include an input / output device, a network access device, a bus, and the like.

[0139] The processor 801 can be a central processing module (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0140] The memory 802 can be an internal storage module of the computer device 8, for example, a hard disk or a memory of the computer device 8. The memory 802 can also be an external storage device of the computer device 8, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the memory 802 can include both the internal storage module and the external storage device of the computer device 8. The memory 802 is used to store computer programs and other programs and data required by the computer device. The memory 802 can also be used to temporarily store data that has been output or will be output.

[0141] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned functional modules and module division are exemplified, and in actual application, the above-mentioned functions can be completed by different functional modules or modules according to needs, that is, the internal structure of the device is divided into different functional modules or modules to complete all or part of the functions described above. Each functional module or module in the embodiment can be integrated into one processing module, or each module can be physically present alone, or two or more modules can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or software function module. In addition, the specific name of each functional module or module is only for the convenience of mutual distinction, and does not limit the protection scope of the present disclosure. The specific working process of the module in the above-mentioned system can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

[0142] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0143] Those of ordinary skill in the art can realize that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0144] In the embodiments provided in the present disclosure, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the module or module division is only a logical function division, and actual implementation can have another division manner, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed ones can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0145] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, that is, they can be located in one place, or they can be distributed on multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] In addition, each functional module in each of the embodiments of the present disclosure can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0147] The integrated module / module, if realized in the form of a software functional module and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can be executed by a processor to implement the steps of each method embodiment. The computer program can include computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0148] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than limit them; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a to-be-processed image; determining a weather type corresponding to the to-be-processed image, the weather type comprising at least one of rain, snow, and fog; acquiring, according to the weather type, an environmental noise feature corresponding to the weather type in the to-be-processed image; obtaining a non-noise image feature of the to-be-processed image according to the environmental noise feature, and generating a denoised image corresponding to the to-be-processed image according to the environmental noise feature, the non-noise image feature, and the to-be-processed image; the acquiring, according to the weather type, of the environmental noise feature corresponding to the weather type in the to-be-processed image comprises: determining, according to the weather type, an image enhancement model corresponding to the weather type; wherein the image enhancement model comprises a noise feature extraction module; inputting the to-be-processed image into the noise feature extraction module to obtain the environmental noise feature corresponding to the weather type in the to-be-processed image; wherein the environmental noise feature is a feature reflecting the distribution of the environmental noise corresponding to the weather type in the to-be-processed image; the noise feature extraction module comprises N dilated convolution layers and an aggregation convolution layer, wherein the convolution kernel dilation sizes of each dilated convolution layer are different, and N is a positive integer greater than 1; the inputting of the to-be-processed image into the noise feature extraction module to obtain the environmental noise feature corresponding to the weather type in the to-be-processed image comprises: inputting the to-be-processed image into the N dilated convolution layers respectively to obtain N first noise features, wherein the sizes of each first noise feature are different; performing aggregation processing on the N first noise features by using the aggregation convolution layer to obtain the environmental noise feature corresponding to the weather type in the to-be-processed image; the image enhancement model further comprises a convolution layer, an encoding network layer, a decoding network layer, and an output layer; the obtaining of the non-noise image feature of the to-be-processed image according to the environmental noise feature, and the generating of the denoised image corresponding to the to-be-processed image according to the environmental noise feature, the non-noise image feature, and the to-be-processed image comprises: inputting the environmental noise feature into the convolution layer to obtain a first feature map; inputting the first feature map into the encoding network layer to obtain P down-sampling feature maps and P non-noise image features; wherein P is a positive integer greater than 1; inputting the P down-sampling feature maps and the P non-noise image features into the decoding network layer to obtain a denoised feature map; inputting the denoised feature map, the first feature map, the environmental noise feature, and the to-be-processed image into the output layer to obtain the denoised image corresponding to the to-be-processed image.

2. The image processing method of claim 1, wherein, the determining of the weather type corresponding to the to-be-processed image comprises: inputting the to-be-processed image into a trained weather type classification model to obtain the weather type corresponding to the to-be-processed image; wherein the weather type classification model is a residual network trained based on sample images and weather type labels corresponding to the sample images.

3. The image processing method of claim 1, wherein, The encoding network layer comprises P cascaded encoding network modules, and each encoding network module comprises a feature aggregation dense convolution module and a max-pooling layer; The inputting of the first feature map into the encoding network layer to obtain P down-sampling feature maps and P non-noise image features comprises: The inputting of the first feature map into the feature aggregation dense convolution module in the first encoding network module to obtain a down-sampling feature map and a non-noise image feature output by the max-pooling layer in the first encoding network module; The inputting of the down-sampling feature map output by the i-1th encoding network module into the feature aggregation dense convolution module in the i-th encoding network module to obtain a down-sampling feature map and a non-noise image feature output by the max-pooling layer in the i-th encoding network module; i is a positive integer greater than 1 and less than or equal to P.

4. The image processing method of claim 3, wherein, The decoding network layer comprises P cascaded decoding network modules, and each decoding network module comprises a feature aggregation dense convolution module and an up-sampling max-pooling layer; The inputting of the P down-sampling feature maps and the P non-noise image features into the decoding network layer to obtain a denoising feature map comprises: The inputting of a down-sampling feature map output by the Pth encoding network module into the feature aggregation dense convolution module in the first decoding network module, and the inputting of a non-noise image feature output by the Pth encoding network module into the up-sampling max-pooling layer in the first decoding network module to obtain an up-sampling feature map output by the up-sampling max-pooling layer; The inputting of a down-sampling feature map output by the P-jth encoding network module and an up-sampling feature map output by the up-sampling max-pooling layer in the jth decoding network module into the feature aggregation dense convolution module in the 1+jth decoding network module, and the inputting of a non-noise image feature output by the P-jth encoding network module into the up-sampling max-pooling layer in the 1+jth decoding network module to obtain an up-sampling feature map output by the up-sampling max-pooling layer in the 1+jth decoding network module; j is a positive integer equal to or greater than 1 and less than P; The up-sampling feature map output by the up-sampling max-pooling layer in the Pth decoding network module is taken as the denoising feature map.

5. The image processing method of claim 3 or 4, characterized in that, The feature aggregation dense convolution module comprises M dilated convolution layers, a dense connection network module and a full connection layer; wherein, the dilated convolution kernel sizes of each dilated convolution layer are different, and M is a positive integer greater than 1; The M dilated convolution layers are used for generating M second noise features according to the feature map, wherein the sizes of the second noise features are different; The dense connection network module is used for performing convolution calculation processing and data filtering processing on the M second noise features to obtain a plurality of convolution feature maps; The full connection layer is used for performing aggregation processing on the M second noise features and the plurality of convolution feature maps to obtain an aggregated feature.

6. The image processing method of any one of claims 3-5, wherein, The loss function of the image enhancement model is an average absolute error function.

7. The image processing method of any one of claims 1-6, wherein If the weather type is rain, the ambient noise feature is a rain line noise feature; if the weather type is snow, the ambient noise feature is a snowflake noise feature; if the weather type is fog, the ambient noise feature is a fog noise feature.

8. An image processing apparatus characterized by comprising: The device adopts the method of any one of claims 1-7, and the device comprises: an image acquisition module, configured to acquire a to-be-processed image; a type determination module, configured to determine a weather type corresponding to the to-be-processed image, the weather type comprising at least one of rain, snow and fog; a feature acquisition module, configured to acquire, according to the weather type, an ambient noise feature corresponding to the weather type in the to-be-processed image; an image generation module, configured to obtain a non-noise image feature of the to-be-processed image according to the ambient noise feature, and generate a denoised image corresponding to the to-be-processed image according to the ambient noise feature, the non-noise image feature and the to-be-processed image; the ambient noise feature corresponding to the weather type in the to-be-processed image according to the weather type, comprising: determining an image enhancement model corresponding to the weather type according to the weather type; wherein the image enhancement model comprises a noise feature extraction module; inputting the to-be-processed image into the noise feature extraction module to obtain the ambient noise feature corresponding to the weather type in the to-be-processed image; wherein the ambient noise feature is a feature reflecting the distribution of ambient noise corresponding to the weather type in the to-be-processed image; the noise feature extraction module comprises N dilated convolution layers and an aggregation convolution layer, wherein the convolution kernel dilation size of each dilated convolution layer is different, and N is a positive integer greater than 1; the inputting of the to-be-processed image into the noise feature extraction module to obtain the ambient noise feature corresponding to the weather type in the to-be-processed image, comprising: inputting the to-be-processed image into the N dilated convolution layers respectively to obtain N first noise features, wherein the size of each first noise feature is different; performing aggregation processing on the N first noise features by using the aggregation convolution layer to obtain the ambient noise feature corresponding to the weather type in the to-be-processed image; the image enhancement model further comprises a convolution layer, an encoding network layer, a decoding network layer and an output layer; the obtaining of the non-noise image feature of the to-be-processed image according to the ambient noise feature, and the generation of the denoised image corresponding to the to-be-processed image according to the ambient noise feature, the non-noise image feature and the to-be-processed image, comprising: inputting the ambient noise feature into the convolution layer to obtain a first feature map; inputting the first feature map into the encoding network layer to obtain P down-sampling feature maps and P non-noise image features; wherein P is a positive integer greater than 1; inputting the P down-sampling feature maps and the P non-noise image features into the decoding network layer to obtain a denoised feature map; inputting the denoised feature map, the first feature map, the ambient noise feature and the to-be-processed image into the output layer to obtain the denoised image corresponding to the to-be-processed image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1-9. The computer program is executed by the processor to implement the steps of the method according to any one of claims 1 to 7.

Images