Image processing method and device

Abstract

An image processing method and device includes the steps of: acquiring an image to be processed; identifying a weather type to which the image to be processed corresponds; acquiring environmental noise features of the image to be processed corresponding to the weather type based on the weather type; acquiring non-noise image features of the image to be processed based on the environmental noise features; and generating a denoised image to which the image to be processed corresponds based on the environmental noise features, the non-noise image features and the image to be processed. Thus, the same technology frame can be used to solve the image quality degradation problem under more weather conditions and improve the effect of image denoising and enhancement.

Description

[Technical field]

[0001] This disclosure relates to the field of computer technology, and more particularly to image processing methods and apparatus. [Background technology]

[0002] In outdoor environments, when extreme weather such as dense fog, heavy rain, and heavy snow occur, images captured by outdoor camera surveillance equipment have problems such as unclear scenes and loss of image details, which limits the applications of image recognition and video surveillance in extreme weather conditions, such as traffic monitoring, target tracking, and autonomous navigation.

[0003] Traditional image enhancement algorithms based on deep learning and physical modeling can only solve image quality degradation under certain extreme weather conditions, for example, they can only remove rain from images or only remove fog from images, and cannot solve image quality degradation under more weather conditions using the same technical framework. In addition, in the process of performing noise removal and restoration on images taken in extreme weather, problems such as unclear scenes and loss of image details often occur, that is, it is difficult to thoroughly remove noise such as rain lines, fog, and snowflakes in the images, resulting in low image restoration quality. Summary of the Invention

[0004] In view of this, embodiments of the present disclosure provide an image processing method, an apparatus, a computer device and a computer-readable storage medium to solve the problem of poor image restoration quality in the prior art.

[0005] In a first aspect of the disclosed embodiment, obtaining an image to be processed; Identifying a weather type to which the processing target image corresponds; obtaining environmental noise features corresponding to the weather type from the processing target image based on the weather type; An image processing method is provided, comprising: a step of acquiring non-noise image features of the image to be processed based on the environmental noise features; and a step of generating a noise-removed image corresponding to the image to be processed based on the environmental noise features, the non-noise image features, and the image to be processed.

[0006] In a second aspect of the disclosed embodiment, an image acquisition module for acquiring an image to be processed; a type identification module for identifying a weather type to which the processing target image corresponds; a feature acquisition module for acquiring environmental noise features corresponding to the weather type from the processing target image based on the weather type; An image processing device is provided, comprising: an image generation module for acquiring non-noise image features of the image to be processed based on the environmental noise features, and generating a noise-removed image corresponding to the image to be processed based on the environmental noise features, the non-noise image features, and the image to be processed.

[0007] In a third aspect of an embodiment of the present disclosure, there is provided a computing device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, the computing device implementing the steps of the method when the processor executes the computer program.

[0008] In a fourth aspect of an embodiment of the present disclosure, there is provided a computer readable storage medium having a computer program stored thereon, the computer readable storage medium implementing the steps of the above method when executed by a processor.

[0009] The advantageous effects of the embodiment of the present disclosure compared to the prior art are that the embodiment of the present disclosure can first obtain a processing target image, and then identify the weather type to which the processing target image corresponds, and then obtain the environmental noise feature of the processing target image corresponding to the weather type based on the weather type, and immediately thereafter obtain the non-noise image feature of the processing target image based on the environmental noise feature, and generate the noise-removed image to which the processing target image corresponds based on the environmental noise feature, the non-noise image feature and the processing target image. In the present embodiment, the environmental noise feature corresponding to the weather type can be obtained for different weather types, the noise removal process can be performed on the processing target image based on the environmental noise feature corresponding to the weather type, and the non-noise image feature can be used to perform image restoration and recovery on the area after noise removal, so that the method provided in the present embodiment can remove noise caused by various weather types from the processing target image, and can realize the removal of noise caused by various weather types and the recovery of image detail information at the same time, and can realize the use of the same technical frame to solve the image quality degradation problem under more weather conditions and improve the effect of image noise removal and enhancement. [Brief description of the drawings]

[0010] In order to more clearly describe the technical solutions in the embodiments of the present disclosure, the following briefly introduces drawings necessary for describing the embodiments or prior art. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without the need for creative work.

[0011] [Figure 1] FIG. 2 is a scenario schematic diagram of an application scenario of an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of an image processing method provided in an embodiment of the present disclosure. [Diagram 3]FIG. 1 is a schematic diagram of the network architecture of Resnet18 provided in an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a network architecture of an image enhancement model provided in an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram of the network architecture of a noise feature extraction module provided in an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of the network architecture of the feature aggregation dense convolution module provided in an embodiment of the present disclosure. [Figure 7] FIG. 2 is a block diagram of an image processing device provided in an embodiment of the present disclosure. [Figure 8] FIG. 2 is a schematic diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] In the following description, for the purposes of explanation, not limitation, specific details such as specific system structures, techniques, etc. are provided to thoroughly understand the embodiments of the present disclosure. However, those skilled in the art should understand that the present disclosure can be realized in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present disclosure with unnecessary details.

[0013] Hereinafter, an image processing method and device according to an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0014] In the prior art, traditional image enhancement algorithms based on deep learning and physical modeling can only solve image quality degradation under certain extreme weather conditions, for example, they can only remove rain from images, or only remove fog from images, and cannot solve image quality degradation under more weather conditions using the same technical framework. In addition, in the process of performing noise removal and restoration on images taken in extreme weather, problems such as unclear scenes and loss of image details often occur, that is, it is difficult to thoroughly remove noise such as rain lines, fog, and snowflakes in the images, causing problems of low image restoration quality.

[0015] In order to solve the above problems, the present invention provides an image processing method, in which in this embodiment, for different weather types, environmental noise features corresponding to the weather types can be obtained, noise removal processing can be performed on the image to be processed based on the environmental noise features corresponding to the weather types, and image restoration can be performed on the area after noise removal using non-noise image features. Therefore, the method provided in this embodiment can remove noise caused by various weather types from the image to be processed, and can realize the removal of noise caused by various weather types and the recovery of image detail information at the same time. Therefore, an image processing method is provided that can solve the problem of image quality degradation under more weather conditions using the same technical frame, and improve the effect of image noise removal and enhancement.

[0016] By way of example, an embodiment of the invention can be applied in an application scenario as shown in Figure 1. This scenario may comprise a terminal device 1 and a server 2.

[0017] The terminal device 1 may be hardware or software. When the terminal device 1 is hardware, it may be various electronic devices that have the function of collecting images and storing images and support communication with the server 2, including, but not limited to, smartphones, tablets, laptop computers, digital cameras, monitors, video recorders, and desktop computers, and when the terminal device 1 is software, it may be installed in, for example, the electronic devices described above. The terminal device 1 may be realized as multiple software or software modules, or may be realized as a single software or software module, and the embodiments of the present disclosure are not limited thereto. Furthermore, various apps, such as an image collection app, an image storage app, and a live chat app, may be installed in the terminal device 1.

[0018] The server 2 may be a server providing various services, such as a back-end server receiving a bill sent by a terminal device that establishes a communication connection with the server, and the back-end server may process the bill sent by the terminal device, such as receiving and analyzing, to generate a processing result. The server 2 may be a single server, a server cluster consisting of several servers, or a cloud computing service center, and the embodiments disclosed herein are not limited thereto.

[0019] The server 2 may be hardware or software. If the server 2 is hardware, it may be various electronic devices that provide various services to the terminal device 1. If the server 2 is software, it may be multiple software programs or software modules that provide various services to the terminal device 1, or it may be a single software program or software module that provides various services to the terminal device 1, and the embodiment of the present disclosure is not limited to this.

[0020] The terminal device 1 and the server 2 may be communicatively connected via a network. The network may be a wired network connected using a coaxial cable, a twisted pair, or an optical fiber, or may be a wireless network capable of realizing interconnection of various communication devices without the need for wiring, such as Bluetooth (registered trademark), Near Field Communication (NFC), or Infrared, and the embodiments disclosed herein are not limited thereto.

[0021] Specifically, the user may specify a processing target image by the terminal device 1, and may select noise due to weather factors from the processing target image to perform noise removal processing. After receiving the processing target image, the server 2 may first specify a weather type to which the processing target image corresponds. Then, the server 2 may acquire environmental noise features corresponding to the weather type from the processing target image based on the weather type. Then, the server 2 may acquire non-noise image features of the processing target image based on the environmental noise features, and generate a noise-removed image to which the processing target image corresponds based on the environmental noise features, the non-noise image features, and the processing target image. Finally, the server 2 may transmit the noise-removed image to the terminal device 1 so that the terminal device 1 displays the noise-removed image to the user. In this way, in this embodiment, environmental noise features corresponding to different weather types can be obtained for the different weather types, noise removal processing can be performed on the target image based on the environmental noise features corresponding to the weather type, and image restoration can be performed on the area after noise removal using non-noise image features. Therefore, the method provided in this embodiment can remove noise caused by various weather types from the target image, and can realize the removal of noise caused by various weather types and the recovery of image detail information at the same time. Therefore, it is possible to use the same technical frame to solve the problem of image quality degradation under more weather conditions, and to improve the effect of image noise removal and enhancement.

[0022] It should be noted that the specific types, numbers and combinations of the terminal device 1, the server 2 and the network can be adjusted according to the actual needs of the application scenario, and the embodiments disclosed herein are not limited thereto.

[0023] It should be noted that the above application scenarios are merely exemplified for easy understanding of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure may be used in any scenario of application.

[0024] Fig. 2 is a flow chart of an image processing method provided in an embodiment of the present disclosure. The image processing method of Fig. 2 may be performed by the terminal device or the server of Fig. 1. As shown in Fig. 2, the image processing method includes the following steps:

[0025] S201: The image to be processed is acquired.

[0026] In this embodiment, the target image may be understood as an image or video frame that needs to be denoised for weather-related noise. For example, an image or video frame of a video taken in extreme weather (such as heavy fog, heavy rain, heavy snow, etc.) may be the target image. As an example, the terminal device provides a page, from which a user can upload an image, and click a preset button to trigger denoising of weather-related noise in the image, at which time the image can be the target image.

[0027] S202: The weather type to which the image to be processed corresponds is identified.

[0028] Since the characteristics of noise caused by different weather are different, for example, the shapes and distribution manners of noise caused by different weather types are different. For example, in one embodiment, the weather types may include rain, snow, and fog, and when the weather type is rain, the noise caused by the weather type is rain lines, the shape of the rain lines is linear, and the distribution manner of the rain lines may include the direction of the rain lines and the density of the rain lines, and the density of the rain lines may be divided into three types, such as high density, medium density, and low density, according to the density of the rain lines, and the direction of the rain lines may be understood as the direction of the rain corresponding to the rain lines. When the weather type is snow, the noise caused by the weather type is snowflakes, and the snowflakes may be linear. The shape of the snowflake is flake-like, the distribution manner of the snowflakes may include the direction of the snowflakes and the density of the snowflakes, the density of the snowflakes may be divided into three types, such as high density, medium density, and low density, according to the density of the snowflakes, the direction of the snowflakes may be understood as the direction of the falling snowflakes corresponding to the snowflakes, when the weather type is fog, the noise caused by the weather type is fog, the distribution manner of the fog may include the density of the fog, and the distribution position of the fog is a random position. Therefore, after obtaining the image to be processed, the weather type corresponding to the image to be processed may be identified first, so that environmental noise features are extracted for the noise features corresponding to different weather types later.

[0029] In this embodiment, an image processing algorithm may be used to classify the weather type of the image to be processed, and the weather type to which the image to be processed corresponds may be identified. Of course, a neural network may be used to classify the weather type of the image to be processed, and the weather type to which the image to be processed corresponds may be identified. In this embodiment, other classification methods may be used to classify the weather type of the image to be processed, and will not be described one by one here.

[0030] S203: Based on the weather type, environmental noise features corresponding to the weather type are obtained from the processing target image.

[0031] In order to remove the noise caused by weather factors from the image to be processed, the environmental noise feature corresponding to the weather type of the image to be processed may be obtained based on the noise feature corresponding to the weather type of the image to be processed, so that the noise corresponding to the weather type can be removed based on the environmental noise feature corresponding to the weather type later. As can be understood, the environmental noise feature corresponding to one weather type may be understood as a feature that can reflect the distribution situation of the environmental noise corresponding to the weather type in the image to be processed, for example, the environmental noise feature corresponding to one weather type may include the shape, distribution density, distribution concentration, distribution area, etc. of the noise corresponding to the weather type in the image to be processed, and the feature format of the environmental noise feature may be a feature map, and of course, may be a feature matrix format, which is not limited in this embodiment.

[0032] That is, in this embodiment, after the weather type of the image to be processed is identified, noise distribution status information related to the noise features corresponding to the weather type may be extracted from the image to be processed based on the noise features corresponding to the weather type, and then environmental noise features corresponding to the weather type may be generated based on the noise distribution status information.

[0033] As an example, if the weather type of the image to be processed is rain, it is possible to extract information on the distribution of rain lines in the image to be processed, such as the shape, direction, density, and position of the rain lines, and it is possible to generate rain line noise features based on the distribution information of the rain lines in the image to be processed. As can be seen, when the weather type is rain, the environmental noise features are rain line noise features.

[0034] Next, as an example, if the weather type of the image to be processed is snow, snowflake distribution information such as snowflake shape, snowflake direction, snowflake density, and snowflake position in the image to be processed can be extracted, and snowflake noise features can be generated based on the snowflake distribution information in the image to be processed. As can be seen, when the weather type is snow, the environmental noise features are snowflake noise features.

[0035] Next, as an example, if the weather type of the image to be processed is fog, it is possible to extract information on the distribution state of fog in the image to be processed, such as the density of the fog and the position of the fog, and it is possible to generate fog noise features based on the information on the distribution state of fog in the image to be processed. As can be seen, when the weather type is fog, the environmental noise features are fog noise features.

[0036] S204: Obtain non-noise image features of the processing target image based on the environmental noise features, and generate a noise-removed image corresponding to the processing target image based on the environmental noise features, the non-noise image features and the processing target image.

[0037] In this embodiment, after obtaining the environmental noise features, the area of ​​the processing target image other than the area corresponding to the environmental noise features may be set as a non-noise image area, and feature extraction may be performed on the non-noise image area to obtain the non-noise image features. As can be understood, the non-noise image features are features that can reflect the details of the non-noise image area of ​​the processing target image, and for example, the non-noise image features may include feature information such as texture features, edge features, color features, shape features, and spatial features of the non-noise image area.

[0038] The environmental noise features may be used to remove noise corresponding to the weather type in the image to be processed. For example, the environmental noise features may be used to identify the location of the noise corresponding to the weather type in the image to be processed, and the pixel value of the pixel corresponding to the location of the noise may be adjusted to a predetermined initial value. Since the environmental noise features are extracted based on the noise features corresponding to the weather type, the obtained environmental noise features can more clearly reflect the physical features of the noise (e.g., rain line, fog, snowflake) corresponding to the weather type in the image to be processed. Therefore, when the environmental noise features are used to remove the noise corresponding to the weather type in the image to be processed, the effect of removing the noise corresponding to the weather type in the image to be processed can be improved. It should be further emphasized that since the environmental noise features are extracted for the noise features corresponding to different weather types in the embodiment, the method provided in the embodiment can support the removal of noise due to various weather types, that is, the method provided in the embodiment can be used to remove noise due to each weather type, thereby realizing the problem of image quality degradation under more weather conditions using the same technical framework.

[0039] Next, the non-noise image features of the processing target image may be used to restore and reduce image detail information to the area after noise removal. Note that the non-noise image features include the details of the non-noise image area in the processing target image, that is, the non-noise image features can more clearly reflect the physical features of the non-noise image area in the processing target image, so that the non-noise image features can calculate the feature relationship between the area after noise removal and the non-noise image area, and thus the image detail can be reduced to the area after noise removal based on the non-noise image features to obtain the noise-removed image corresponding to the processing target image, for example, the target pixel value of the pixel in the area after noise removal based on the non-noise image features, and the pixel value of the pixel in the area after noise removal is adjusted to the target pixel value. In this way, the image detail restoration degree of the processing target image after noise removal can be improved, the distortion level of the processing target image after noise removal can be reduced, and the clarity of the noise-removed image corresponding to the processing target image can be improved.

[0040] As can be seen, the beneficial effects of the embodiment of the present disclosure compared to the prior art include: first, acquiring a target image, identifying the weather type to which the target image corresponds, and then acquiring environmental noise features corresponding to the weather type from the target image based on the weather type; immediately after that, acquiring non-noise image features of the target image based on the environmental noise features, and generating a noise-removed image to which the target image corresponds based on the environmental noise features, the non-noise image features, and the target image. In this embodiment, environmental noise features corresponding to different weather types can be acquired for the target image, and noise removal processing can be performed on the target image based on the environmental noise features corresponding to the weather type, and image restoration and recovery can be performed on the area after noise removal using the non-noise image features. Therefore, the method provided in this embodiment can remove noises caused by various weather types from the target image, and can realize the removal of noises caused by various weather types and the recovery of image detail information at the same time, thereby realizing the use of the same technical frame to solve the image quality degradation problem under more weather conditions and improve the effect of image noise removal and enhancement.

[0041] Next, an embodiment of S202, that is, how to use a neural network to classify the weather type of the target image, will be described. In this embodiment, S202, which is "identify the weather type corresponding to the target image", may include the following steps:

[0042] The target image is input into a trained weather type classification model to obtain the weather type to which the target image corresponds.

[0043] In this embodiment, the weather type classification model may be a residual network obtained by training based on a sample image and a weather type flag corresponding to the sample image. In one embodiment, the residual network may be a Resent18 network as shown in FIG. 3. In this embodiment, several groups of training samples may be preset, and each group of training samples may include a sample image and a weather type flag corresponding to the sample image. The manner of training the residual network using a predetermined training sample is to input a sample image in each set of training samples into the residual network to obtain a predicted weather type corresponding to the sample image, and then, immediately after determining a loss value based on the weather type flag corresponding to the sample image and the predicted weather type, adjust the network parameters of the residual network according to the loss value until the residual network meets a training condition, for example, when the network parameters of the residual network are fitted or the number of trainings meets a predetermined training threshold, obtain a trained weather type classification model. For example, if the predetermined weather type includes rain, fog, and snow, after inputting the target image into the trained weather type classification model, it is obtained that the weather type of the target image is one of rain, fog, and snow.

[0044] Then, an embodiment of S203, that is, how to obtain environmental noise features corresponding to the weather type from the target image, will be described. In this embodiment, S203 for obtaining environmental noise features corresponding to the weather type from the target image based on the weather type may include the following steps:

[0045] S203a: According to the weather type, identify an image enhancement model corresponding to the weather type, where the image enhancement model comprises a noise feature extraction module.

[0046] In one embodiment of this embodiment, different image enhancement models may be set for different weather types. The model network architectures of the image enhancement models corresponding to different weather types are all the same, but the difference is that the training samples of the image enhancement models corresponding to different weather types are different. For example, if a sample image in the training sample of the image enhancement model corresponding to the weather type "rain" is an image with rain line noise, the corresponding sample noise-removed image of the sample image is an image in which the rain line noise has been removed and the image details have been restored; if a sample image in the training sample of the image enhancement model corresponding to the weather type "snow" is an image with snowflake noise, the corresponding sample noise-removed image of the sample image is an image in which the snowflake noise has been removed and the image details have been restored; if a sample image in the training sample of the image enhancement model corresponding to the weather type "fog" is an image with fog noise, the corresponding sample noise-removed image of the sample image is an image in which the fog noise has been removed and the image details have been restored. It can be understood that the model network architecture of the image enhancement model corresponding to different weather types is the same, but the model network parameters of the image enhancement model corresponding to different weather types may be different. It is necessary to emphasize that the implementation methods described below are all described for the network architecture level embodiment, so the embodiment described below applies to the image enhancement model corresponding to all weather types. Of course, in various embodiments of this embodiment, all weather types may set the same image enhancement model, that is, all weather types correspond to the same image enhancement model. It can be understood that in this embodiment, the image enhancement model is obtained by training based on the sample images corresponding to all weather types and the denoising images corresponding to the sample images. It should be noted that in this embodiment, the situation of setting different image enhancement models for different weather types is mainly taken as an example.

[0047] When different image enhancement models are set for different weather types, after obtaining the weather type of the image to be processed, the image enhancement model corresponding to the weather type can be identified in all trained image enhancement models based on the weather type. For example, the image enhancement models corresponding to various weather types are all provided with type identifiers corresponding to the weather type, so after obtaining the weather type of the image to be processed, the image enhancement model whose type identifier is the same as the type identifier corresponding to the weather type can be identified in all trained image enhancement models based on the type identifier corresponding to the weather type.

[0048] S203b: Input the target image to the noise feature extraction module to obtain environmental noise features corresponding to the weather type in the target image.

[0049] In this embodiment, the image enhancement model may include a noise feature extraction module, as shown in Figure 4. After obtaining the target image, the image enhancement model is first used to obtain the environmental noise feature corresponding to the weather type in the target image, and then the environmental noise feature is used to remove noise.

[0050] In one embodiment, the noise feature extraction module may be a multi-scale feature extraction network module (Feature Block), as shown in Figure 4. The process of the noise feature extraction module extracting environmental noise features may be defined as F0=H0(I0), where I0 represents the image to be processed, H0 represents the feature calculation function of the Feature Block, and F0 is the acquired environmental noise feature.

[0051] In order to extract noise features of different shapes and different distributions (for example, rain line noise features of different magnitudes in different directions, fog noise features of different concentrations, snowflake noise features of different magnitudes in different directions), the noise feature extraction module may include N dilated convolution layers and aggregated convolution layers, where the convolution kernel dilation size of each dilated convolution layer is different, and N is a positive integer greater than 1. Note that, since the convolution kernel dilation size of each dilated convolution layer is different, the number of interval pixels between adjacent pixels collected in the convolution operation process of each dilated convolution layer is different, so the receptive field size of each dilated convolution layer is also different, and the first noise features extracted from each dilated convolution layer are also different (for example, the size is different and the image feature information contained therein is different), so that the first noise features of different scales, such as different rain lines and fog of different concentrations, can be extracted, and more contextual spatial information can be aggregated in the training process, and the training parameters are smaller, making it easier to train the model to achieve fitting.

[0052] Specifically, in this embodiment, the target image may be first output to the N dilation convolution layers respectively to obtain N first noise features, and the sizes of the first noise features are different.Then, the aggregation convolution layer is used to perform aggregation processing on the N first noise features to obtain environmental noise features corresponding to the weather type in the target image, and since the image feature information contained in the first noise features of different scales is different, in the process of aggregating and obtaining environmental noise features using the N first noise features, more contextual spatial information can be aggregated, and the information of the environmental noise features obtained by aggregation can be made more detailed, so that the feature extraction of noise in the target image can be more significantly improved, that is, the acquired environmental noise features can more clearly reflect the noise in the target image corresponding to the weather type.

[0053] Next, a specific embodiment of S203b will be described with reference to FIG. As shown in FIG. 5, the noise feature extraction module may include four dilated convolution layers and three aggregated convolution layers. Specifically, the four dilated convolution layers include a first dilated convolution layer (Conv3*3DF=1), a second dilated convolution layer (Conv3*3DF=2), a third dilated convolution layer (Conv3*3DF=3) and a fourth dilated convolution layer (Conv3*3DF=4). The aggregated convolution layers include a first aggregated convolution layer (Conv1*1DF=1), a second aggregated convolution layer (Conv1*1DF=1) and a third aggregated convolution layer (Conv1*1DF=1). The first dilated convolution layer is a convolution layer with a convolution kernel of 3*3, DF=1 (DF=1 represents that there is no gap between adjacent pixels collected in the convolution calculation process), and the second dilated convolution layer is a convolution layer with a convolution kernel of 3*3, DF=1 (DF=1 represents that there is no gap between adjacent pixels collected in the convolution calculation process). The dilation convolution layer is a convolution layer with a convolution kernel of 3*3, DF=2 (DF=2 represents that the number of intervals between adjacent pixels collected in the convolution operation process is one pixel), the third dilation convolution layer is a convolution layer with a convolution kernel of 3*3, DF=3 (DF=3 represents that the number of intervals between adjacent pixels collected in the convolution operation process is two pixels), the fourth dilation convolution layer is a convolution layer with a convolution kernel of 3*3, DF=4 (DF=4 represents that the number of intervals between adjacent pixels collected in the convolution operation process is three pixels), and the first aggregation convolution layer, the second aggregation convolution layer and the third aggregation convolution layer are all convolution layers with a convolution kernel of 1*1, DF=1. Specifically, the processing target image is input to the first dilation convolution layer, the second dilation convolution layer, the third dilation convolution layer, and the fourth dilation convolution layer, respectively, to obtain four first noise features. The two first noise features output by the first dilation convolution layer and the second dilation convolution layer are input to the first aggregation convolution layer to obtain a first sub-aggregation feature, the two first noise features output by the third dilation convolution layer and the fourth dilation convolution layer are input to the second aggregation convolution layer to obtain a second sub-aggregation feature, and the first sub-aggregation feature and the second sub-aggregation feature are input to the third aggregation convolution layer to obtain an environmental noise feature corresponding to the weather type from the processing target image.In addition, in this example, a convolutional layer with a convolutional kernel of 1*1 is used to aggregate features at different scales, and the feature information (i.e., environmental noise features) obtained by performing three feature aggregations is more detailed, which significantly improves the extraction of environmental noise features in weather images.

[0054] Next, an embodiment of S204, that is, how to generate a noise-removed image corresponding to the processing target image, will be described. S204 of obtaining non-noise image features of the processing target image based on the environmental noise features, and generating a noise-removed image corresponding to the processing target image based on the environmental noise features, the non-noise image features, and the processing target image may include the following steps:

[0055] S204a: Input the environmental noise features into the convolution layer to obtain a first feature map.

[0056] In this embodiment, the image enhancement model further includes a convolution layer, as shown in Figure 4. After obtaining the environmental noise feature, the environmental noise feature is input into the convolution layer to perform convolution processing to obtain a first feature map.

[0057] S204b: Input the first feature map into the encoding network layer to obtain P down-sampled feature maps and P non-noise image features, where P is a positive integer greater than 1.

[0058] In this embodiment, the image enhancement model further includes an encoding network layer, as shown in Fig. 4. The encoding network layer may include P cascaded encoding network modules, and each encoding network module includes a feature aggregation dense convolution module and a max pooling layer, and in one embodiment, as shown in Fig. 4, an encoding network module may include a feature aggregation dense convolution module and a max pooling layer, for example, the feature aggregation dense convolution module may be the FJDB (Feature Joint Dense Block) in Fig. 4, and the max pooling layer may be the Maxpooling in Fig. 4.

[0059] In this embodiment, in the coding stage, the first feature map is input to the encoding network layer, and P cascaded encoding network modules of the encoding network layer are used to perform feature extraction on the first feature, to obtain P downsampling feature maps and P non-noise image features, where the P cascaded encoding network modules of the encoding network layer can be understood as downsampling convolution layers, the feature maps output by the P cascaded encoding network modules have different sizes, and the slower the sorting of the encoding network module, the smaller the size of the output feature map. The downsampling feature map can be understood as the sum of the sums of the same scale features, the downsampling feature map is for removing noise caused by weather factors, the non-noise image features are the image details lost in the pooling stage (i.e., the encoding network module extracts features) recorded by the pooling indexes, and the non-noise image features are for guiding the encoding network module to recover the detailed features lost in the feature extraction stage in the upsampling stage (i.e., the later decoding stage).

[0060] Specifically, the first feature map may be input to a feature aggregation dense convolution module in a first encoding network module to obtain one downsampled feature map and one non-noise image feature output by a max pooling layer in the first encoding network module. The downsampled feature map output by the (i-1)th encoding network module may be input to a feature aggregation dense convolution module in an i-th encoding network module to obtain one downsampled feature map and one non-noise image feature output by a max pooling layer in the i-th encoding network module, where i is a positive integer greater than 1 and less than or equal to P.

[0061] For example, as shown in FIG. 4, the encoding network layer may include three cascaded encoding network modules, namely a first encoding network module, a second encoding network module and a third encoding network module, and each encoding network module includes a feature aggregation dense convolution module (i.e., FIDB) and a maximum pooling layer (Maxpooling). A first feature map F1 is input to the first encoding network module, and the first encoding network module outputs a downsampled feature map F2 and a non-noise image feature. The first encoding network module outputs a downsampled feature map F3 and a non-noise image feature. The downsampled feature map F2 output by the network module is input to a feature aggregation dense convolution module of a second encoding network module to obtain one downsampled feature map F3 and one non-noise image feature output by a max pooling layer in the second encoding network module, and the downsampled feature map F3 output by the second encoding network module is input to a feature aggregation dense convolution module of a third encoding network module to obtain one downsampled feature map F4 and one non-noise image feature output by a max pooling layer in the third encoding network module.

[0062] S205c: Input the P downsampled feature maps and the P non-noise image features into a decoding network layer to obtain a denoised feature map.

[0063] In this embodiment, as shown in Fig. 4, the image enhancement model further includes a decoding network layer. The decoding network layer includes P cascade decoding network modules, and each decoding network module includes a feature aggregation dense convolution module and an upsampling max pooling layer. In one embodiment, as shown in Fig. 4, each decoding network module may include a feature aggregation dense convolution module and an upsampling max pooling layer, for example, the feature aggregation dense convolution module may be the FIDB (Feature Joint Dense Block) in Fig. 4, and the upsampling max pooling layer may be the UpMaxpooling in Fig. 4. Each decoding network module corresponds to an encoding network module, and the input of the decoding network module includes the downsampling feature map and one non-noise image feature output by the corresponding encoding network module.

[0064] In this embodiment, in the decoding stage, the target image, P downsampling feature maps and P non-noise image features are input to P cascade decoding network modules of the decoding network layer, so that the P cascade decoding network modules use the P downsampling feature maps to perform noise removal corresponding to weather type on the target image, and obtain the P non-noise image features and the denoising feature map by restoring and reducing the image detail information. Note that the P cascade decoding network modules of the decoding network layer can be understood as upsampling convolution layers, and the feature maps output by the P cascade decoding network modules are different in size, and the slower the sorting of the decoding network module, the larger the size of the output feature map. As can be seen, in this embodiment, the encoding network layer and the decoding network layer are integrated together to calculate the long-term spatial feature dependency. Pooling indices are used to record the loss and replenishment of image detail information in the downsampling step (i.e., the coding step) and the upsampling step (i.e., the decoding step), and each max pooling layer (Maxpooling) in the coding step corresponds to one upsampling max pooling layer (UpMaxpooling), and the max pooling layer (Maxpooling) guides the max pooling layer (UpMaxpooling) to upsample using the pooling indices, so that more image details can be restored in the upsampling step.

[0065] Specifically, one down-sampled feature map output by the P-th encoding network module may be input to a feature aggregation dense convolution module in a first decoding network module, and one non-noise image feature output by the P-th encoding network module may be input to an up-sampling max pooling layer in the first decoding network module to obtain one up-sampled feature map output by the up-sampling max pooling layer. One down-sampled feature map output by the Pj-th encoding network module and one up-sampled feature map output by the up-sampling max pooling layer in the j-th decoding network module may be input to a feature aggregation dense convolution module in a 1+j-th decoding network module, and one non-noise image feature output by the Pj-th encoding network module may be input to an up-sampling max pooling layer in the 1+j-th decoding network module to obtain one up-sampled feature map output by the up-sampling max pooling layer in the 1+j-th decoding network module, where j is a positive integer greater than or equal to 1 and less than P. The up-sampled feature map output by the up-sampling max pooling layer in the Pth decoding network module may be the denoising feature map.

[0066] As an example, as shown in Figure 4, the encoding network layer may include three cascaded encoding network modules, which are a first encoding network module, a second encoding network module, and a third encoding network module, and the decoding network layer may include three cascaded decoding network modules, which are a first decoding network module, a second decoding network module, and a third decoding network module. A down-sampled feature map F4 output by the third encoding network module may be input to a feature aggregation dense convolution module in the first decoding network module, and a non-noise image feature output by the third encoding network module may be input to an up-sampling max pooling layer in the first decoding network module to obtain an up-sampling feature map F5 output by the up-sampling max pooling layer. The one down-sampled feature map F3 output by the second encoding network module and the one up-sampled feature map F5 output by the up-sampling max pooling layer in the first decoding network module may be input into a feature aggregation dense convolution module in the second decoding network module, and the one non-noise image feature output by the second encoding network module may be input into the up-sampling max pooling layer in the second decoding network module to obtain the one up-sampled feature map F6 output by the up-sampling max pooling layer in the second decoding network module.A down-sampling feature map F2 output by the first encoding network module and an up-sampling feature map F6 output by the up-sampling max pooling layer in the second decoding network module may be input into a feature aggregation dense convolution module in a third decoding network module, and a non-noise image feature output by the first encoding network module may be input into the up-sampling max pooling layer in the third decoding network module to obtain an up-sampling feature map F7 output by the up-sampling max pooling layer in the third decoding network module, and the up-sampling feature map F7 output by the up-sampling max pooling layer in the third decoding network module may be used as a denoising feature map.

[0067] S206d: The noise-removed feature map, the first feature map, the environmental noise features, and the target image are input to the output layer to obtain a noise-removed image corresponding to the target image.

[0068] In this embodiment, as shown in Fig. 4, the image enhancement model further includes an output layer. The denoising feature map, the first feature map, the environmental noise feature and the target image are input to the output layer to obtain a denoising image corresponding to the target image, for example, the output layer fuses the denoising feature map, the first feature map, the environmental noise feature and the target image to obtain a denoising image corresponding to the target image.

[0069] In one embodiment, as shown in Fig. 4, the output layer includes a first output layer and a second output layer, the first output layer includes a convolution layer (1*1Conv) whose convolution kernel size is 1*1 and one convolution layer (Conv), and the second output layer includes one convolution layer (Conv) and one activation function layer (e.g. Tanh function layer, i.e. Tanh). The input of the first output layer is the denoising feature map and the first feature map, the output of the first output layer is the first output feature map, the input of the second output layer is the fusion feature of the first output feature map and the environmental noise feature F0, and the output of the second output layer is the second output feature map, and then the second output feature map and the processing target image may be fused to obtain a denoising image corresponding to the processing target image.

[0070] It should be noted that the feature aggregation dense convolution modules mentioned in the above embodiments each include M dilated convolution layers, a dense connection network module and a fully connected layer, and the convolution kernel dilation sizes of each dilated convolution layer are different, and M is a positive integer greater than 1.

[0071] The M dilated convolution layers are for generating M second noise features based on a feature map (e.g., a first feature map, a downsampled feature map, an upsampled feature map), and the sizes of the second noise features are different. Specifically, the feature maps may be input to the M dilated convolution layers respectively to obtain the M second noise features. For example, as shown in FIG. 6, one feature aggregation dense convolution module may include three dilated convolution layers, which are a first sub-dilated convolution layer (Conv3*3DF=1), a second sub-dilated convolution layer (Conv3*3DF=2) and a third sub-dilated convolution layer (Conv3*3DF=3). The first sub-dilated convolution layer is a dilated convolution layer with a convolution kernel of 3*3, DF=1 (DF=1 represents that there is no gap between adjacent pixels collected in the convolution operation process), and the second sub-dilated convolution layer is a dilated convolution layer with a convolution kernel of 3*3, DF=2 (DF=2 represents that there is no gap between adjacent pixels collected in the convolution operation process). The first sub-dilated convolution layer is a dilated convolution layer with a convolution kernel of 3*3, DF=3 (DF=3 represents that the number of intervals between adjacent pixels collected in the convolution operation process is 1 pixel), and the third sub-dilated convolution layer is a dilated convolution layer with a convolution kernel of 3*3, DF=3 (DF=3 represents that the number of intervals between adjacent pixels collected in the convolution operation process is 2 pixels). In this embodiment, the dilated convolution with three kinds of convolution kernel dilation sizes is used to aggregate feature information (i.e., feature map) of different scales, and the convolution kernel of size 3*3 can well extract noise features such as rain lines, fog, snowflakes, etc. In addition, the channel numbers of the feature maps in the coding stage and the decoding stage are kept consistent, that is, the channel numbers of the feature maps output in the encoding network layer and the decoding network layer are the same.As can be seen, this embodiment uses dilation convolution with different convolution kernel dilation sizes to aggregate features of different scales together, and improves the image information contained in the extracted features to be more detailed; for noise features such as rain lines, snowflakes, and fog, which have irregular shapes and sizes and change from time to time due to wind direction, the technical means of using multi-scale aggregated features can improve the effect of extracting noise features, and the extracted features can be more detailed than those of convolution using a single size convolution kernel, for example; thus, the acquired environmental noise features can clearly reflect the physical features of the noise (such as rain lines, fog, snowflakes) corresponding to the weather type in the target image, and further improve the clarity of the denoised image corresponding to the target image. In one embodiment, the feature aggregation dense convolution module may be input with feature blocks of different scales and output with feature blocks of invariant size.

[0072] The dense connection network module is for performing a convolution operation process and a data filtering process on the M second noise features to obtain a plurality of convolution feature maps. In one embodiment, as shown in Figure 6, the dense connection network module includes a plurality of dense connection modules, and each dense connection module includes a convolution layer (Conv) and an activation function layer (for example, a ReLu function layer, i.e., ReLu).

[0073] The fully connected layer is for performing an aggregation process on the M second noise features and the plurality of convolution feature maps to obtain an aggregate feature. In one embodiment, as shown in FIG. 6, the fully connected layer includes one concat function layer (Concat) and one convolution layer (1*1Conv) with a convolution kernel of 1*1.

[0074] Illustratively, referring to FIG. 6, suppose the densely connected network module includes three densely connected modules, namely a first densely connected module, a second densely connected module, and a third densely connected module. After fusing M second noise features, the obtained fusion feature may be input to the first densely connected module; after fusing M second noise features, the obtained fusion feature and the convolution feature map output by the first densely connected module may be input to the second densely connected module; after fusing M second noise features, the obtained fusion feature, the convolution feature map output by the first densely connected module, and the convolution feature map output by the first densely connected module may be input to the second densely connected module. The convolutional feature map output by the first dense connection module and the second dense connection module may be input to a third dense connection module. After fusing the M second noise features, the obtained fused feature, the convolutional feature map output by the first dense connection module, the convolutional feature map output by the second dense connection module, and the convolutional feature map output by the third dense connection module may be input to a fully connected layer. The feature map output by the fully connected layer may be fused (i.e., aggregated) with the feature maps input to the M dilated convolutional layers (e.g., the first feature map, the downsampled feature map, and the upsampled feature map).

[0075] As can be seen, after fusing M second noise features with different scales in this embodiment, the fusion feature obtained after fusing M second noise features is input into one densely connected network module, and then the features of all stages are aggregated by the Concat layer in the fully connected layer, so that the feature information in the down-sampling feature map and non-noise image features obtained in the coding stage can be more specifically depicted, and after the noise information (i.e., noise features) such as rain lines, fog, snowflakes, etc. are removed in the decoding stage, more image details can be restored, and the distortion level of the denoised image corresponding to the processed image can be reduced.

[0076] The loss function of the image enhancement model may be a mean absolute error function. In this embodiment, the noise information (i.e., noise features) such as rain lines, fog, and snowflakes are relatively sparse, and what is depicted in this embodiment is the degree of difference between the predicted noise-removed image f(x) and the true noise-free image Y after the noise information such as rain lines, fog, and snowflakes is removed. Therefore, in this embodiment, the MAE (Mean Absolute Error) function, which is sensitive to sparse features, may be selected as the loss function for training. In this way, after the loss value corresponding to the input image for model training is identified using the loss function, the model parameters of the image enhancement model can be adjusted using the loss value until the training completion condition is met, for example, until the number of training times reaches a predetermined number of times or the model parameters of the image enhancement model are fitted.

[0077] In one embodiment, the MAE function is shown in the following equation: JPEG0007658036000001.jpg30158

[0078] In the above formula, JPEG0007658036000002.jpg2028 represents the predicted denoised image output by the image enhancement model, H, W, and C represent the height, width, and number of channels of the input image of the image enhancement model, respectively, and Y i,j,k represents the true noise-free image, i, j, and k represent the height, width, and number of channels of the image, respectively, and L represents the loss value.

[0079] All the above-mentioned optional technical solutions may be combined in any combination to form optional embodiments of the present disclosure, and will not be described one by one here.

[0080] Below are apparatus embodiments of the present disclosure for carrying out the method embodiments of the present disclosure. For details not disclosed in the apparatus embodiments of the present disclosure, please refer to the method embodiments of the present disclosure.

[0081] 7 is a schematic diagram of an image processing device provided in an embodiment of the present disclosure. As shown in FIG. 7, the image processing device includes: an image acquisition module 701 for acquiring an image to be processed; a type identification module 702 for identifying a weather type to which the image to be processed corresponds; a feature acquisition module 703 for acquiring environmental noise features corresponding to the weather type from the processing target image based on the weather type; and an image generation module 704 for obtaining non-noise image features of the processing target image based on the environmental noise features, and for generating a noise-removed image corresponding to the processing target image based on the environmental noise features, the non-noise image features, and the processing target image.

[0082] In some embodiments, the type identification module 702 includes: inputting the target image into a trained weather type classification model to obtain a weather type corresponding to the target image; The weather type classification model is a residual network trained based on sample images and the weather type flags to which the sample images correspond.

[0083] In some embodiments, the feature acquisition module 703 comprises: and to identify an image enhancement model corresponding to the weather type based on the weather type, the image enhancement model comprising a noise feature extraction module.

[0084] The image to be processed is input into the noise feature extraction module, and environmental noise features corresponding to the weather type are obtained from the image to be processed, and the environmental noise features are features that reflect the distribution status of environmental noise corresponding to the weather type in the image to be processed.

[0085] In some embodiments, the noise feature extraction module includes N dilation convolution layers and aggregation convolution layers, and the convolution kernel dilation sizes of each dilation convolution layer are different, and N is a positive integer greater than 1; Specifically, the feature acquisition module 703: The target image is output to the N dilated convolution layers to obtain N first noise features, each of which has a different size; The aggregation convolution layer is used to perform an aggregation process on the N first noise features, and obtain environmental noise features corresponding to the weather type from the processing target image.

[0086] In some embodiments, the image enhancement model further comprises a convolution layer, an encoding network layer, a decoding network layer, and an output layer; The image generation module 704 includes: Input the environmental noise features into the convolutional layer to obtain a first feature map; Input the first feature map into the encoding network layer to obtain P down-sampled feature maps and P non-noise image features, where P is a positive integer greater than 1; Input the P downsampled feature maps and the P non-noise image features into a decoding network layer to obtain a denoised feature map; The noise removal feature map, the first feature map, the environmental noise features, and the processing target image are input to the output layer to obtain a noise removed image corresponding to the processing target image.

[0087] In some embodiments, the encoding network layer comprises P cascaded encoding network modules, each of which comprises a feature aggregation dense convolution module and a max pooling layer; Specifically, the image generation module 704: The first feature map is input to a feature aggregation dense convolution module in a first encoding network module to obtain one downsampled feature map and one non-noise image feature output by a max pooling layer in the first encoding network module; The downsampled feature map output by the (i-1)th encoding network module is input to a feature aggregation dense convolution module in the i-th encoding network module to obtain one downsampled feature map and one non-noise image feature output by the max pooling layer in the i-th encoding network module, where i is a positive integer greater than 1 and less than or equal to P.

[0088] In some embodiments, the decoding network layer comprises P cascaded decoding network modules, each of the decoding network modules comprising a feature aggregation dense convolution module and an upsampling max pooling layer; Specifically, the image generation module 704: Input one down-sampled feature map output by the Pth encoding network module into a feature aggregation dense convolution module in a first decoding network module, and input one non-noise image feature output by the Pth encoding network module into an up-sampling max pooling layer in the first decoding network module to obtain one up-sampled feature map output by the up-sampling max pooling layer; A down-sampled feature map output by the Pj-th encoding network module and an up-sampled feature map output by the up-sampling max pooling layer in the j-th decoding network module are input into a feature aggregation dense convolution module in the 1+j-th decoding network module; and a non-noise image feature output by the Pj-th encoding network module is input into the up-sampling max pooling layer in the 1+j-th decoding network module to obtain an up-sampled feature map output by the up-sampling max pooling layer in the 1+j-th decoding network module, where j is a positive integer greater than or equal to 1 and less than P; The upsampling feature map output by the upsampling max pooling layer in the Pth decoding network module is used as the denoising feature map.

[0089] In some embodiments, the feature aggregation dense convolution module includes M dilated convolution layers, a dense connection network module and a fully connected layer, and the convolution kernel dilation sizes of each dilated convolution layer are different, and M is a positive integer greater than 1; The M dilated convolution layers are for generating M second noise features based on the feature map, and the sizes of the second noise features are different in size; The densely connected network module is for performing a convolution operation and a data filtering operation on the M second noise features to obtain a plurality of convolution feature maps; The fully connected layer is for performing an aggregation process on the M second noise features and the plurality of convolutional feature maps to obtain an aggregate feature.

[0090] In some embodiments, the loss function of the image enhancement model is a mean absolute error function.

[0091] In some embodiments, the weather type includes at least one of rain, snow, and fog; When the weather type is rain, the environmental noise feature is a rain line noise feature, when the weather type is snow, the environmental noise feature is a snowflake noise feature, and when the weather type is fog, the environmental noise feature is a fog noise feature.

[0092] According to the technical solution provided in the embodiment of the present disclosure, the image processing device includes an image acquisition module for acquiring a target image, a type identification module for identifying a weather type corresponding to the target image, a feature acquisition module for acquiring environmental noise features corresponding to the weather type from the target image based on the weather type, and an image generation module for acquiring non-noise image features of the target image based on the environmental noise features, and generating a noise-removed image corresponding to the target image based on the environmental noise features, the non-noise image features, and the target image. In this embodiment, the environmental noise features corresponding to the weather types can be acquired for different weather types, the noise removal process can be performed on the target image based on the environmental noise features corresponding to the weather types, and the non-noise image features can be used to perform image restoration and recovery on the area after noise removal. Therefore, the method provided in this embodiment can remove noise caused by various weather types from the target image, and can realize the removal of noise caused by various weather types and the recovery of image detail information at the same time, thereby realizing the use of the same technical frame to solve the image quality degradation problem under more weather conditions and improve the effect of image noise removal and enhancement.

[0093] It should be understood that the size of the numbers of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and does not arbitrarily limit the implementation process of the embodiments disclosed herein.

[0094] Fig. 8 is a schematic diagram of a computer device 8 provided in an embodiment of the present disclosure. As shown in Fig. 8, the computer device 8 of the embodiment includes a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable by the processor 801. When the processor 801 executes the computer program 803, it realizes the steps in each of the above method embodiments. Or, when the processor 801 executes the computer program 803, it realizes each module / function of each module in each of the above device embodiments.

[0095] For illustrative purposes, the computer program 803 may be divided into one or more modules, and one or more modules may be stored in the memory 802 and executed by the processor 801 to accomplish the present disclosure. One or more modules may be a series of computer program command sections capable of performing a specific function, and the command sections are for explaining the process of the computer program 803 being executed in the computer device 8.

[0096] The computing device 8 may be a computing device such as a desktop computer, a laptop, a palmtop computer, and a cloud server. The computing device 8 may include, but is not limited to, a processor 801 and a memory 802. As will be appreciated by those skilled in the art, FIG. 8 is merely an example of the computing device 8, and is not intended to limit the computing device 8, and may include more or fewer components than shown, or may combine certain components or different components, for example, the computing device may include input / output devices, network access devices, buses, etc.

[0097] The processor 801 may be a Central Processing Unit (CPU) or other general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general purpose processor may be a microprocessor, any common processor, etc.

[0098] The memory 802 may be an internal storage module of the computer device 8, such as a hard disk or memory of the computer device 8. The memory 802 may be an external storage device of the computer device 8, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc., that is installed in the computer device 8. Furthermore, the memory 802 may include both an internal storage module and an external storage device of the computer device 8. The memory 802 is for storing computer programs and other programs and data required by the computer device. The memory 802 may also be used to temporarily store data that has been output or is to be output.

[0099] For convenience and simplicity of explanation, those skilled in the art can clearly understand that, although only the above-mentioned functional modules and division of modules are given as examples, in actual application, the above-mentioned functions can be assigned to different functional modules and completed as necessary, that is, by dividing the internal structure of the device into different functional modules or modules, all or part of the above-mentioned functions can be completed. Each functional module and module in the embodiment may be integrated into one processing module, each module may exist physically alone, or two or more modules may be integrated into one module, and the integrated module may be realized in the form of hardware or in the form of a software functional module. Note that the specific names of each functional module and module are provided for easy distinction from each other, and do not limit the scope of protection of the present disclosure. For the specific operation process of the modules and modules in the above system, reference may be made to the corresponding process in the embodiment of the above-mentioned method, and the description will be omitted here.

[0100] In the above embodiments, the description of each embodiment has its own emphasis, and for the details or parts not described in an embodiment, reference may be made to the relevant descriptions of other embodiments.

[0101] Those skilled in the art can be aware that the modules and algorithm steps of each example described in the embodiments disclosed herein can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software is determined by the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to realize the described functions for each specific application, but such realization should not be considered to go beyond the scope of this disclosure.

[0102] In the embodiments provided in this disclosure, it should be understood that the disclosed device / computer device and method can be realized in other ways. For example, the above-described device / computer device embodiments are merely exemplary, and the division of the modules or modules is merely a logical functional division, and other division methods may be applied in actual implementation, for example, multiple modules or components may be combined or integrated into other systems, or some features may be ignored or not implemented. In addition, the couplings or direct couplings or communication connections between the displayed or discussed interfaces may be indirect couplings or communication connections through some interfaces, devices or modules, and may be electrical, mechanical, or other forms.

[0103] Moreover, modules described as separate components may or may not be physically separate, and components described as modules may or may not be physical modules, i.e., located in one location or distributed across multiple network modules, some or all of which may be selected according to actual needs to achieve the objectives of the solutions of the present embodiment.

[0104] In addition, each functional module in each embodiment of the present disclosure may be integrated into one processing module, each module may exist physically alone, or two or more modules may be integrated into one module. The integrated module may be realized in the form of hardware or in the form of a software functional module.

[0105] The integrated module / module may be realized in the form of a software function module and stored in a computer-readable storage medium when sold or used as an independent product. With this understanding, the present disclosure can realize all or part of the processes in the method of the above embodiments by commanding relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of each of the above method embodiments can be realized. The computer program may include a computer program code, and the computer program code may be in source code form, object code form, executable file, or some intermediate form, etc. The computer-readable storage medium may include any entity or device capable of carrying a computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electric carrier wave signal, an electric communication signal, and a software distribution medium, etc. Furthermore, the content contained on a computer-readable storage medium may be increased or decreased as required by the legislation and patent practice of a jurisdiction. For example, in some jurisdictions, based on legislation and patent practice, a computer-readable storage medium does not include electrical carrier signals and telecommunications signals.

[0106] The above-mentioned embodiments are only for illustrating the technical solutions of the present disclosure, but are not intended to limit the same. Although the present disclosure has been described in detail with reference to the above embodiments, those skilled in the art may still amend the technical solutions described in the above embodiments, or equivalently replace some of the technical features therein, but it should be understood that such amendments or replacements do not deviate from the substance of the corresponding technical solutions from the idea and scope of the technical solutions of the embodiments of the present disclosure, and all of them should be included in the protection scope of the present disclosure.

Claims

1. 1. An image processing method, comprising: The method includes the steps of: acquiring an image to be processed; identifying a weather type to which the image to be processed corresponds; acquiring environmental noise features of the image to be processed that correspond to the weather type based on the weather type; acquiring non-noise image features of the image to be processed based on the environmental noise features; and generating a noise-removed image to which the image to be processed corresponds based on the environmental noise features, the non-noise image features, and the image to be processed; The step of acquiring environmental noise features corresponding to the weather type from the processing target image based on the weather type includes: Identifying an image enhancement model corresponding to the weather type according to the weather type, the image enhancement model comprising a noise feature extraction module; inputting the processing target image into the noise feature extraction module to obtain environmental noise features corresponding to the weather type from the processing target image, the environmental noise features being features reflecting a distribution state in the processing target image of environmental noise corresponding to the weather type; The noise feature extraction module includes N dilation convolution layers and aggregation convolution layers, each dilation convolution layer has a different convolution kernel dilation size, and N is a positive integer greater than 1; The step of inputting the processing target image to the noise feature extraction module to obtain environmental noise features corresponding to the weather type from the processing target image includes: outputting the target image to the N dilated convolution layers to obtain N first noise features, each of which has a different size; and performing an aggregation process on the N first noise features using the aggregation convolution layer to obtain environmental noise features corresponding to the weather type in the processing target image.

13. An image processing method comprising:

2. The step of identifying the weather type to which the image to be processed corresponds includes a step of inputting the image to be processed into a trained weather type classification model to obtain the weather type to which the image to be processed corresponds, and the weather type classification model is a residual network obtained by training based on a sample image and a weather type flag to which the sample image corresponds.

2. The image processing method according to claim 1.

3. 1. An image processing method, comprising: The method includes the steps of: acquiring an image to be processed; identifying a weather type to which the image to be processed corresponds; acquiring environmental noise features of the image to be processed that correspond to the weather type based on the weather type; acquiring non-noise image features of the image to be processed based on the environmental noise features; and generating a noise-removed image to which the image to be processed corresponds based on the environmental noise features, the non-noise image features, and the image to be processed; The step of acquiring environmental noise features corresponding to the weather type from the processing target image based on the weather type includes: Identifying an image enhancement model corresponding to the weather type according to the weather type, the image enhancement model comprising a noise feature extraction module; inputting the processing target image into the noise feature extraction module to obtain environmental noise features corresponding to the weather type from the processing target image, the environmental noise features being features reflecting a distribution state in the processing target image of environmental noise corresponding to the weather type; The image enhancement model further comprises a convolution layer, an encoding network layer, a decoding network layer, and an output layer; The steps of: acquiring non-noise image features of the processing target image based on the environmental noise features; and generating a noise-removed image corresponding to the processing target image based on the environmental noise features, the non-noise image features, and the processing target image, inputting the environmental noise features into the convolutional layer to obtain a first feature map; inputting the first feature map into the encoding network layer to obtain P down-sampled feature maps and P non-noise image features, where P is a positive integer greater than 1; inputting the P down-sampled feature maps and the P non-noise image features into a decoding network layer to obtain a denoised feature map; and inputting the denoised feature map, the first feature map, the environmental noise features and the target image into the output layer to obtain a denoised image corresponding to the target image.

13. An image processing method comprising:

4. The encoding network layer includes P cascaded encoding network modules, each of which includes a feature aggregation dense convolution module and a max pooling layer; The step of inputting the first feature map to the encoding network layer to obtain P downsampled feature maps and P non-noise image features includes: inputting the first feature map into a feature aggregation dense convolution module in a first encoding network module to obtain one downsampled feature map and one non-noise image feature output by a max pooling layer in the first encoding network module; inputting the downsampled feature map output by the (i-1)th encoding network module into a feature aggregation dense convolution module in the i-th encoding network module to obtain one downsampled feature map and one non-noise image feature output by the max pooling layer in the i-th encoding network module, where i is a positive integer greater than 1 and less than or equal to P; 4. The image processing method according to claim 3.

5. The decoding network layer includes P cascaded decoding network modules, each of which includes a feature aggregation dense convolution module and an upsampling max pooling layer; The step of inputting the P downsampled feature maps and the P non-noise image features into a decoding network layer to obtain a denoised feature map includes: Inputting a down-sampled feature map output by the Pth encoding network module into a feature aggregation dense convolution module in a first decoding network module, and inputting a non-noise image feature output by the Pth encoding network module into an up-sampling max pooling layer in the first decoding network module to obtain an up-sampling feature map output by the up-sampling max pooling layer; inputting one down-sampled feature map output by the P-j encoding network module and one up-sampled feature map output by the up-sampling max pooling layer in the j decoding network module into a feature aggregation dense convolution module in the 1+j decoding network module, and inputting one non-noise image feature output by the P-j encoding network module into the up-sampling max pooling layer in the 1+j decoding network module to obtain one up-sampled feature map output by the up-sampling max pooling layer in the 1+j decoding network module, where j is a positive integer greater than or equal to 1 and less than P; The upsampling feature map output by the upsampling max pooling layer in the Pth decoding network module is the denoising feature map.

5. The image processing method according to claim 4.

6. The feature aggregation dense convolution module includes M dilated convolution layers, a dense connection network module and a fully connected layer, and the convolution kernel dilation sizes of each dilated convolution layer are different, and M is a positive integer greater than 1; The M dilated convolution layers are for generating M second noise features based on the feature map, and the sizes of the second noise features are different in size; The densely connected network module is for performing a convolution operation and a data filtering operation on the M second noise features to obtain a plurality of convolution feature maps; The fully connected layer is for performing an aggregation process on the M second noise features and the plurality of convolutional feature maps to obtain an aggregate feature.

5. The image processing method according to claim 4.

7. The loss function of the image enhancement model is the mean absolute error function 2. The image processing method according to claim 1.

8. the weather type includes at least one of rain, snow, and fog; When the weather type is rain, the environmental noise feature is a rain line noise feature, when the weather type is snow, the environmental noise feature is a snowflake noise feature, when the weather type is fog, the environmental noise feature is a fog noise feature.

2. The image processing method according to claim 1.

9. An image processing device, an image acquisition module for acquiring an image to be processed; a type identification module for identifying a weather type to which the processing target image corresponds; a noise feature extraction module for acquiring environmental noise features corresponding to the weather type from the processing target image based on the weather type; an image generating module for acquiring non-noise image features of the processing target image based on the environmental noise features, and generating a noise-removed image corresponding to the processing target image based on the environmental noise features, the non-noise image features, and the processing target image; A noise feature extraction module for obtaining environmental noise features corresponding to the weather type from the target image according to the weather type includes N dilation convolution layers and aggregation convolution layers, each dilation convolution layer has a different convolution kernel dilation size, and N is a positive integer greater than 1; The step of inputting the processing target image to the noise feature extraction module to obtain environmental noise features corresponding to the weather type from the processing target image includes: outputting the target image to the N dilated convolution layers to obtain N first noise features, each of which has a different size; and performing an aggregation process on the N first noise features using the aggregation convolution layer to obtain environmental noise features corresponding to the weather type in the processing target image.

13. An image processing device comprising:

10. 1. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, comprising: The processor, when executing the computer program, performs the steps of the method of claim 1.

1. A computer device comprising:

11. A computer-readable storage medium on which a computer program is stored, The computer program, when executed by a processor, performs the steps of the method of claim 1. A computer-readable storage medium comprising:

Images