Image blind enhancement method and device, electronic device, and storage medium

Abstract

The present application provides a method and apparatus for blind enhancement of an image, an electronic device, and a computer-readable storage medium. The method comprises: inputting a target image into a trained residual feature extraction network to obtain residual features of the target image; inputting the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image; performing multi-scale filling processing on the texture features and structural features respectively through a multi-scale filling module of the image restoration network to obtain filled texture features and filled structural features; inputting the filled texture features, filled structural features, and residual features into a feature equalization module of the trained image restoration network to obtain balanced fusion features; and inputting the balanced fusion features, texture features, and structural features into a decoder of the trained image restoration network to obtain a blind enhanced image of the target image. The present application scheme can restore an image without determining the location of noise in the image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for blind enhancement of an image, an electronic device, and a computer-readable storage medium. Background Art

[0002] Most alloy materials will undergo a solidification process during the melting and casting process. During the solidification process, solid phase grains are gradually precipitated from the liquid alloy. As the temperature drops, the grains continue to nucleate, grow, and coarsen in different forms, gradually forming a dendritic microstructure with complex morphology and complex spatial structure. At the same time, complex structural defects such as intermetallic compounds, secondary precipitates, cracks, shrinkage cavities, etc. will be formed during the metal solidification process. The microstructure formed by the metal during the solidification process plays a decisive role in the final performance of the alloy material. At present, X-ray imaging technology can be used to collect synchrotron radiation images of alloy samples at various stages of the solidification process, so that the microstructure during the solidification process can be studied based on the images.

[0003] Synchrotron radiation images differ from natural images in several ways: 1. They have inevitable and non-negligible global Gaussian-like noise introduced by preprocessing steps (flat-field correction and histogram equalization); 2. Surface obstructions such as oxides can occur on alloy samples. These obstructions vary in shape and transparency, leading to the appearance of damaged areas containing valuable image information. The effects of obstructions on pixel intensity can vary, either enhancing or weakening it.

[0004] In the existing materials analysis process, traditional denoising algorithms are usually used to remove global noise from noisy alloy microstructure images. However, the non-generalizability of traditional algorithms makes the denoising process very time-consuming. In addition, damaged areas caused by occlusions are usually discarded directly. For some quantitative representation tasks in materials analysis (such as segmentation and counting), discarding images due to semantic noise caused by occlusions is undoubtedly a waste of data. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a blind enhancement method and device for an image, an electronic device, and a computer-readable storage medium, which are used to repair an image when the location of noise in the image is uncertain.

[0006] In one aspect, the present application provides a method for blind enhancement of an image, comprising:

[0007] Inputting the target image into the trained residual feature extraction network to obtain the residual features of the target image;

[0008] Inputting the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image;

[0009] Performing multi-scale filling processing on the texture features and the structural features respectively through the multi-scale filling module of the image restoration network to obtain filled texture features and filled structural features;

[0010] Inputting the filled texture feature, the filled structural feature and the residual feature into a feature equalization module of a trained image restoration network to obtain a balanced fusion feature;

[0011] The balanced fusion features, the texture features and the structural features are input into a decoder of a trained image restoration network to obtain a blind enhanced image of the target image.

[0012] In one embodiment, the encoder is an encoder of a trained U-Net-based image restoration network;

[0013] Inputting the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image includes:

[0014] Inputting the target image into the encoder so that the encoder performs encoding processing on the target image to obtain a plurality of shallow features and a plurality of deep features;

[0015] The shallow features are used as texture features, and the deep features are used as structural features.

[0016] In one embodiment, the multi-scale filling module includes a first multi-scale filling module and a second multi-scale filling module;

[0017] The multi-scale filling module of the image restoration network performs multi-scale filling processing on the texture features and the structural features respectively to obtain the filled texture features and the filled structural features, including:

[0018] Performing fusion processing on multiple texture features and multiple structural features respectively to obtain specified texture features and specified structural features;

[0019] Inputting the designated texture feature into the first multi-scale filling module, obtaining a plurality of filled sub-texture features output by the first multi-scale filling module, and fusing the plurality of filled sub-texture features into a filled texture feature;

[0020] The designated structural feature is input into the second multi-scale filling module to obtain a plurality of filled sub-structural features output by the second multi-scale filling module, and the plurality of filled sub-structural features are fused into a filled structural feature.

[0021] In one embodiment, inputting the filled texture feature, the filled structural feature, and the residual feature into a feature equalization module of a trained image restoration network to obtain a balanced fusion feature includes:

[0022] Splicing the filled texture feature and the filled structural feature to obtain the feature to be balanced;

[0023] Inputting the to-be-equalized features into the feature equalization module, so that the feature equalization module performs channel-dimensional equalization processing on the to-be-equalized features to obtain an intermediate feature map;

[0024] The feature equalization module performs spatial dimension equalization processing on the intermediate feature map and the residual feature to obtain the balanced fusion feature.

[0025] In one embodiment, the decoder is a decoder of a trained U-Net-based image restoration network;

[0026] The step of inputting the balanced fusion feature, the texture feature, and the structural feature into a decoder of a trained image restoration network to obtain a blind enhanced image of the target image comprises:

[0027] The balanced fusion features, the texture features, and the structural features are output to each network layer of the decoder through a jump connection, so that the decoder outputs a blind enhanced image of the target image.

[0028] In one embodiment, the residual feature extraction network and the image restoration network are trained in the following manner:

[0029] Inputting the sample images in the first sample data set into a pre-trained residual feature extraction network and an untrained image restoration network, respectively, to obtain sample residual features output by the pre-trained residual feature extraction network, sample texture features and sample structural features output by the encoder of the image restoration network, and predicted enhanced images output by the decoder of the image restoration network;

[0030] Determining a first difference between the sample residual feature and the residual label of the sample image, a second difference between the sample texture feature and the texture label of the sample image, a third difference between the sample structure feature and the structure label of the sample image, and a fourth difference between the predicted enhanced image and the target label of the sample image;

[0031] Under a generative adversarial mechanism, using the residual feature extraction network and the image restoration network as generators, and adjusting network parameters of the residual feature extraction network and the image restoration network based on the first difference, the second difference, the third difference, and the fourth difference;

[0032] The above process is repeated until the residual feature extraction network and the image restoration network converge.

[0033] In one embodiment, the pre-training process of the residual feature extraction network is as follows:

[0034] Inputting the sample images in the second sample data set into an untrained residual feature extraction network to obtain predicted residual features output by the residual feature extraction network;

[0035] Adjusting network parameters of the residual feature extraction network based on a difference between the predicted residual feature and the residual label of the sample image;

[0036] The above process is repeated until the residual feature extraction network converges.

[0037] On the other hand, the present application also provides a blind image enhancement device, comprising:

[0038] A first extraction module is used to input the target image into the trained residual feature extraction network to obtain the residual features of the target image;

[0039] A second extraction module is configured to input the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image;

[0040] A filling module, configured to perform multi-scale filling processing on the texture features and the structural features respectively through the multi-scale filling module of the image restoration network to obtain filled texture features and filled structural features;

[0041] An equalization module, configured to input the filled texture feature, the filled structural feature, and the residual feature into a feature equalization module of a trained image restoration network to obtain a balanced fusion feature;

[0042] A decoding module is used to input the balanced fusion feature, the texture feature and the structural feature into a decoder of a trained image restoration network to obtain a blind enhanced image of the target image.

[0043] Furthermore, the present application also provides an electronic device, comprising:

[0044] processor;

[0045] a memory for storing processor-executable instructions;

[0046] The processor is configured to execute the above-mentioned blind enhancement method for an image.

[0047] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to complete the above-mentioned blind enhancement method for an image.

[0048] The present application scheme proposes a multi-task blind strong enhancement algorithm for global noise and semantic noise for the target image to be repaired, which combines residual learning and repair learning to construct a residual feature extraction network and an image repair network; the residual features of the target image are extracted by the residual feature extraction network, and the texture features and structural features of the target image are extracted by the image repair network, and the multi-scale filling processing can be used to maximize the extraction ability of multi-scale texture features and multi-scale structural features, so as to better obtain the information of the damaged area in the target image and assist in repair; after the multi-scale filling processing, the inconsistency of texture features, structural features and residual features is eliminated in the channel dimension and spatial dimension, and then aggregation is performed, and encoding processing is performed based on the aggregated balanced fusion features, texture features and structural features to obtain a blind enhanced image with global noise and semantic noise removed; this scheme has strong generalization and can greatly reduce labor costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments of the present application.

[0050] Figure 1 A schematic diagram of an application scenario of the blind enhancement method for an image provided in one embodiment of the present application;

[0051] Figure 2 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0052] Figure 3 A flowchart of a blind enhancement method for an image provided in one embodiment of the present application;

[0053] Figure 4 A schematic diagram of a target image provided in one embodiment of the present application;

[0054] Figure 5 A schematic diagram of the architecture of a network model provided in one embodiment of the present application;

[0055] Figure 6 A schematic diagram of a multi-scale filling process according to an embodiment of the present application;

[0056] Figure 7A schematic structural diagram of a first multi-scale filling module provided in one embodiment of the present application;

[0057] Figure 8 A schematic structural diagram of a second multi-scale filling module provided in one embodiment of the present application;

[0058] Figure 9 A schematic diagram of a feature equalization process according to an embodiment of the present application;

[0059] Figure 10 A schematic diagram of a process for training a network model according to an embodiment of the present application;

[0060] Figures 11a-11b A schematic diagram of a sample image provided in an embodiment of the present application;

[0061] Figure 12 A schematic diagram of the process of pre-training a network model according to an embodiment of the present application;

[0062] Figure 13 A schematic diagram comparing the effects of image restoration provided in one embodiment of the present application;

[0063] Figure 14 A schematic diagram comparing the effects of image restoration provided by another embodiment of the present application;

[0064] Figure 15 This is a block diagram of a blind image enhancement device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0066] Similar reference numerals and letters denote similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0067] Figure 1 Schematic diagram of the application scenario of the blind enhancement method of the image provided in the embodiment of the present application. Figure 1 As shown, the application scenario includes a client 20 and a server 30; the client 20 can be a user terminal such as a host, mobile phone, tablet computer, etc., and is used to send an image that needs to be blind enhanced to eliminate noise to the server 30; the server 30 can be a server, a server cluster or a cloud computing center, and is used to obtain the image sent by the client 20 and perform blind enhancement processing on the image, thereby repairing the image.

[0068] like Figure 2As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 2 In the example, a processor 11 is used. Processor 11 and memory 12 are connected via bus 10. Memory 12 stores instructions executable by processor 11. Processor 11 executes these instructions, enabling electronic device 1 to perform all or part of the method described in the following embodiments. In one embodiment, electronic device 1 may be the aforementioned server 30, configured to perform the blind image enhancement method.

[0069] The memory 12 can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0070] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the processor 11 to complete the blind enhancement method of the image provided by the present application.

[0071] See also Figure 3 , which is a flow chart of a blind enhancement method for an image provided by an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps 310 to 350.

[0072] Step 310: Input the target image into the trained residual feature extraction network to obtain the residual features of the target image.

[0073] Among them, the residual feature extraction network is used to extract residual features of the noise level from the image, and the noise in the image can include global noise and semantic noise (local noise).

[0074] The target image refers to an image that needs to be subjected to blind enhancement processing. For example, the target image may refer to an image of an alloy sample acquired by X-ray imaging technology and subjected to preprocessing. Figure 4, is a schematic diagram of a target image provided in an embodiment of the present application. The target image is an image of an alloy sample acquired by X-ray imaging technology and processed by flat field correction, such as Figure 4 As shown in Figure 1, a layer of Gaussian-like noise is superimposed on the entire field of view of the target image.

[0075] In one embodiment, the residual feature extraction network may include an input layer, an intermediate layer, and an output layer; wherein the input layer may include a convolutional layer and a ReLU (Rectified Linear Unit), and the parameters of the convolutional layer may be expressed as (1, 64, 3) in the form of (number of input channels, number of output channels, convolution kernel size). The intermediate layer may include several convolutional layers, a BN (Batch Normalization) layer, and a ReLU layer, and the parameters of the convolutional layer may be expressed as (64, 64, 3). The output layer may include a convolutional layer and a Tanh layer, and the parameters of the convolutional layer may be expressed as (64, 1, 3).

[0076] In the process of processing the target image by the residual feature extraction network of this embodiment, the size of the image feature does not change. Here, the image feature can be a feature map.

[0077] Step 320: Input the target image into the encoder of the trained image restoration network to obtain the texture features and structural features of the target image.

[0078] The image restoration network may include an encoder, a multi-scale filling module, a feature equalization module and a decoder; the encoder may obtain multiple image features based on the target image encoding.

[0079] When executing step 310 , the server can input the target image into the encoder of the image restoration network in parallel, so that the encoder encodes the target image to obtain a plurality of texture features and a plurality of structural features.

[0080] Step 330: Perform multi-scale filling processing on the texture features and the structural features respectively through the multi-scale filling module of the image restoration network to obtain the filled texture features and the filled structural features.

[0081] Among them, the multi-scale filling module can perform multi-scale filling processing on texture features or structural features; for texture features, it can obtain texture details of contextual information under a larger receptive field in the damaged area of ​​the target image, thereby helping to reconstruct a more realistic texture effect; for structural features, it can construct finer structural content in the damaged area based on the image content characteristics of the damaged area of ​​the target image itself and the finer feature results in the damaged area.

[0082] The server can input several texture features into the multi-scale filling module, perform filling processing on the texture features through the multi-scale filling module to obtain several filled sub-texture features, and fuse the several filled sub-texture features to obtain the filled texture features.

[0083] The server can input several structural features into the multi-scale filling module, fill the structural features through the multi-scale filling module to obtain several filled sub-structural features, and fuse the several filled sub-structural features to obtain the filled structural features.

[0084] Step 340: Input the filled texture features, filled structural features and residual features into the feature equalization module of the trained image restoration network to obtain balanced fusion features.

[0085] The feature equalization module is used to perform equalization processing on the features, and the equalization processing may include equalization processing in the channel dimension and equalization processing in the spatial dimension.

[0086] After the server inputs the filled texture features, filled structural features, and residual features into the feature equalization module, the feature equalization module can perform equalization processing to obtain balanced fusion features.

[0087] Step 350: Input the balanced fusion features, texture features, and structural features into the decoder of the trained image restoration network to obtain a blind enhanced image of the target image.

[0088] After obtaining the balanced fusion features, the server can input the balanced fusion features and several texture features and several structural features output by the encoder into the decoder. After the decoder decodes the balanced fusion features, texture features and structural features, a blind enhanced image of the target image is obtained.

[0089] In one embodiment, the encoder is an encoder of a trained basic U-Net image restoration network. Figure 5 , is a schematic diagram of the architecture of a network model provided in an embodiment of the present application, such as Figure 5 As shown in the figure, the image restoration network includes an encoder, a multi-scale filling module, a feature equalization module and a decoder. The encoder and decoder are basically the same as U-Net. There can be multiple downsampling operations in the encoder and multiple upsampling operations in the decoder.

[0090] In the encoder, the intermediate layers may include LeakyReLU layers, convolutional layers, and BN layers. The shallowest layers contain only convolutional layers, while the deepest layers contain LeakyReLU layers and convolutional layers. In the decoder, the intermediate layers may include LeakyReLU layers, transposed convolutional layers, and BN layers. The shallowest layers contain only convolutional layers, while the deepest layers contain LeakyReLU layers and transposed convolutional layers.

[0091] For example, both the encoder and decoder include 6 convolutional layers. The parameters of each convolutional layer / transposed convolutional layer can be expressed in the form of (number of input channels, number of output channels, convolution kernel size, and stride) as shown in Table 1 below:

[0092]

[0093] Table 1

[0094] After the server inputs the target image into the encoder, the encoder can encode the target image to obtain multiple shallow features and multiple deep features. For example, the encoder encodes 6 features, the first 3 of which can be considered shallow features, and the last 3 can be considered deep features.

[0095] Shallow features contain relatively richer texture information, while deep features contain relatively richer structural information. The server can use shallow features as texture features and deep features as structural features.

[0096] In one embodiment, the multi-scale filling module of the image inpainting network can adopt an inception structure similar to that in Google LeNet, designing multi-scale tools and modules for different levels of features. The multi-scale filling module includes a first multi-scale filling module for processing texture features and a second multi-scale filling module for processing structural features.

[0097] See also Figure 6 , which is a schematic diagram of a multi-scale filling process according to an embodiment of the present application, such as Figure 6 As shown, the method may include the following steps 331-333.

[0098] Step 331: performing fusion processing on multiple texture features and multiple structural features respectively to obtain designated texture features and designated structural features.

[0099] The server can process multiple texture features of different scales through a convolution layer to obtain multiple texture features with the same width and height. The server splices the above multiple texture features in the channel direction and adjusts the number of channels of the spliced ​​texture features through a 1*1 convolution layer to obtain texture features with a specified number of channels as the specified texture features.

[0100] The server can process multiple structural features of different scales through a convolution layer to obtain multiple structural features with the same width and height. The server splices the above multiple structural features in the channel direction and adjusts the number of channels of the spliced ​​structural features through a 1*1 convolution layer to obtain a structural feature with a specified number of channels as the specified texture feature.

[0101] Step 332: Input the specified texture feature into the first multi-scale filling module of the image restoration network, obtain multiple filled sub-texture features output by the first multi-scale filling module, and fuse the multiple filled sub-texture features into a filled texture feature.

[0102] See also Figure 7 , which is a structural diagram of a first multi-scale filling module provided in an embodiment of the present application, such as Figure 7 As shown, the first multi-scale padding module includes three branches: the first branch is a single 3*3 convolution layer followed by a 1*1 convolution layer; the second branch is two consecutive 3*3 convolution layers followed by a 1*1 convolution layer. Under the setting of ordinary convolution kernel, the receptive field is 5*5, and under the setting of dilation rate 2, the receptive field is 9*9; the third branch is three consecutive 3*3 convolution layers followed by a 1*1 convolution layer. Under the setting of ordinary convolution kernel, the receptive field is 7*7, and under the setting of dilation rate 2, the receptive field is 13*13.

[0103] After the server inputs the specified texture feature into the first multi-scale filling module, it can perform filling processing on the specified texture feature through each branch of the first multi-scale filling module, thereby obtaining multiple filled sub-texture features. The server can splice the multiple filled sub-texture features in the channel direction and perform convolution calculation on the spliced ​​features through a 1*1 convolution layer to fuse them to obtain the filled texture feature.

[0104] Step 333: Input the specified structural feature into the second multi-scale filling module of the image restoration network, obtain multiple filled sub-structural features output by the second multi-scale filling module, and fuse the multiple filled sub-structural features into the filled structural feature.

[0105] See also Figure 8 , which is a structural diagram of a second multi-scale filling module provided in an embodiment of the present application, such as Figure 8 As shown, the second multi-scale padding module includes three branches: the first branch is a single 3*3 convolution layer followed by a 1*1 convolution layer; the second branch is two consecutive 3*3 convolution layers followed by a 1*1 convolution layer. Under the setting of ordinary convolution kernel, the receptive field is 5*5, and under the setting of dilation rate 2, the receptive field is 9*9; the third branch is three consecutive 3*3 convolution layers followed by a 1*1 convolution layer. Under the setting of ordinary convolution kernel, the receptive field is 7*7, and under the setting of dilation rate 2, the receptive field is 13*13.

[0106] After the server inputs the specified structural features into the first multi-scale filling module, the specified structural features can be filled in separately through each branch of the first multi-scale filling module, thereby obtaining multiple filled sub-structural features. The server can splice the multiple filled sub-structural features in the channel direction, and perform convolution calculation on the spliced ​​features through a 1*1 convolution layer, thereby fusing them to obtain the filled structural features. The above-mentioned multi-scale filling module containing multiple branches can perform channel dimensionality reduction through a 1*1 convolution layer to reduce the amount of subsequent calculations. The receptive field of the convolution kernel of each branch of the multi-scale filling module includes multiple scales from large to small. After aggregating the image features output by each branch, it can be used for the next stage to simultaneously learn and represent features of different scales. This multi-branch and multi-scale structure can maximize the network's ability to extract multi-scale features and reduce the multi-scale requirements for training samples, thereby better extracting information about damaged areas in the target image and assisting in repairing the target image.

[0107] In one embodiment, see Figure 9 , is a flow chart of feature equalization processing provided by an embodiment of the present application, such as Figure 9 As shown, the method may include the following steps 341-343.

[0108] Step 341: Concatenate the filled texture features and the filled structural features to obtain the features to be balanced.

[0109] The server can splice the filled texture features and the filled structural features in the channel direction to obtain the features to be balanced.

[0110] Step 342: Input the features to be equalized into the feature equalization module, so that the feature equalization module performs channel-dimensional equalization processing on the features to be equalized to obtain an intermediate feature map.

[0111] Among them, the feature equalization module can include SEblock (Sequeze and Excitationblock) and bilateral propagation activation function.

[0112] After the server inputs the above-mentioned features to be balanced into the feature equalization module, it can use the SE block to equalize the features to be balanced in the channel dimension to obtain an intermediate feature map. This intermediate feature map is the features to be balanced that have been equalized in the channel dimension.

[0113] The channel dimension equalization process can be expressed by the following formulas (1) to (3). Formula (1) is:

[0114]

[0115] Here, f c(i, j) represents the element value of the feature f to be balanced in the i-th row and j-th column of channel c; H represents the height of the feature f to be balanced; W represents the width of the feature f to be balanced; z c Indicates the intermediate weight value corresponding to the feature f to be equalized in the cth channel.

[0116] After obtaining the intermediate weight values ​​of each channel of the feature to be equalized, a multidimensional vector can be constructed based on the multiple intermediate weight values ​​as the intermediate weight values ​​corresponding to the feature to be equalized.

[0117] Formula (2) is:

[0118] s=σ(W2δ(W1z)) (2)

[0119] Here, z represents the intermediate weight value corresponding to the c channels of the feature to be balanced; δ() represents the sigmoid operation; σ() represents the ReLU operation; W1 and W2 represent two matrices containing learnable parameters; s represents the feature map weight corresponding to the c channels of the feature to be balanced.

[0120] Formula (3) is:

[0121]

[0122] Here, f c is the feature map of the feature f to be equalized in channel c; s c is f c The corresponding feature map weight; It is the feature map of the cth channel after equalization.

[0123] After the feature map of each channel in the feature to be balanced is equalized, an intermediate feature map that has been equalized in the spatial dimension can be obtained.

[0124] Step 343: Perform spatial dimension equalization processing on the intermediate feature map and the residual feature through the feature equalization module to obtain a balanced fusion feature.

[0125] The balancing operation in the spatial dimension is divided into three parts: weight redistribution based on spatial distance measurement, weight redistribution based on local similarity measurement, and spatial weight redistribution based on residual features. The first two can be implemented using the bilateral propagation activation function.

[0126] The weight redistribution operation based on spatial distance measurement is equivalent to the global continuity characterization of the feature map in the entire space. The unit it targets is the feature descriptor with a dimension of B*C*1*1, where C represents the number of channels in the intermediate feature map and B represents the batch size in the network training phase, which can be set to a value not exceeding 8 in this application. The feature descriptor represents the feature vectors corresponding to the positions in all channels in the intermediate feature map. By re-characterizing the feature descriptor, the real weight redistribution of the global spatial distance is achieved. This process can be expressed by the following formula (4):

[0127]

[0128] Here, x j represents the feature descriptor of the intermediate feature map at spatial position j; f i g represents the output feature descriptor at spatial position i; n represents the number of pixels in the spatial dimension; W represents the spatial domain equal to the height and width of the intermediate feature map; G σw (·) represents the Gaussian convolution kernel operation. Through the Gaussian convolution kernel, the spatial distance is sampled by the Gaussian kernel into a weight value inversely proportional to the distance, and then the feature descriptor is weighted.

[0129] Based on the weight redistribution of local similarity measurement, the re-characterization of feature descriptors is based on the similarity between several feature descriptors in the area around the location. This process can be expressed by the following formula (5):

[0130]

[0131] Here, x j represents the feature descriptor of the intermediate feature map at spatial position j; f i l represents the corresponding output feature descriptor at spatial position i; L represents the domain of spatial position i, the size of which can be preconfigured, for example, 3*3; l represents the number of all pixels in the domain; s(x i -x j ) represents the dot product operation of two feature descriptors. Through this operation, the similarity between the feature descriptors can be calculated, and the feature descriptors are weighted and summed using this similarity as the weight.

[0132] Based on the spatial weight redistribution of the residual features, after obtaining the residual features, a B*1*H*W weight distribution map can be obtained through the spatial attention mechanism based on the residual features, and the spatial dimension feature re-representation can be performed based on this. This process is expressed by the following formula (6):

[0133]

[0134] here, is the feature passed through a residual convolution block, x i is the feature descriptor of the residual feature map at spatial position i; f i r is the corresponding output feature descriptor of the residual feature at spatial position i; α i is the spatial attention weight calculated based on the residual feature map.

[0135] The spatial attention weights corresponding to all positions on the residual feature map can form an attention map. The attention map can be calculated using the following formula (7):

[0136] α=softmax(W i (tanh(W S X+b s ))+b i ) (7)

[0137] Here, α is the attention map; X is the residual feature; W i , W s , b i , b s is the learnable parameter matrix.

[0138] After completing the weight redistribution based on spatial distance measurement, the weight redistribution based on local similarity measurement, and the spatial weight redistribution based on residual features, the feature maps after the above three levels of redistribution can be obtained.

[0139] The server can splice the feature maps of the above three levels in the channel direction and pass them through a 1*1 convolution layer to obtain balanced fusion features.

[0140] In one embodiment, the decoder is a decoder of a trained basic U-Net image restoration network. During the decoding process, the server can output the balanced fusion features, multiple texture features, and multiple structural features to each network layer of the decoder via skip connections. After the decoder decodes the balanced fusion features, multiple texture features, and multiple structural features, it outputs a blindly enhanced image of the target image.

[0141] In one embodiment, before executing the blind enhancement method of the image of the present application, the residual feature extraction network and the image restoration network can be trained, see Figure 10 , is a flow chart of a training network model provided in an embodiment of the present application, such as Figure 10 As shown, the method may include the following steps 306 to 309.

[0142] Step 306: Input the sample images in the first sample data set into the pre-trained residual feature extraction network and the untrained image restoration network respectively to obtain the sample residual features output by the pre-trained residual feature extraction network, the sample texture features and sample structure features output by the encoder of the image restoration network, and the predicted enhanced image output by the decoder of the image restoration network.

[0143] The first sample dataset includes a large number of sample images, which may belong to the same category as the target image undergoing blind enhancement. For example, before performing enhancement on dendrite images of an alloy sample acquired and pre-processed using X-ray imaging technology, pre-collected dendrite images may be used as sample images to construct the first sample dataset.

[0144] The sample image can carry pre-labeled residual labels, texture labels, structure labels, and target labels. The residual label indicates the damage information in the sample image, including global noise and semantic noise; the texture label indicates the texture information in the sample image, which can be obtained by downsampling the sample image; the structure label indicates the contour information of objects (such as dendrites) in the sample image, which can be obtained by manual labeling; the target label is the expected effect of the sample image after blind enhancement, which can be the image after manually denoising the sample image.

[0145] See also Figures 11a-11b , is a schematic diagram of a sample image provided in an embodiment of the present application, such as Figure 11a As shown in , there is a pair of noisy and noise-free images, the noisy image is the sample image, and its corresponding noise-free image is the target label. Figure 11b As shown, there are images with oxide occlusion and images without oxide. The image with oxide is the sample image, and its corresponding image without oxide is the target label.

[0146] The server inputs the sample image into a pre-trained residual feature extraction network and an untrained image restoration network in parallel, and outputs the sample residual feature after calculating the sample image through the residual feature extraction network; the sample image is encoded by the encoder of the image restoration network to obtain a number of shallow features and a number of deep features, and the sample texture feature is obtained after the multi-scale filling module of the image restoration network fills the several shallow features, and the sample structure feature is obtained after the multi-scale filling module fills the several deep features; the sample texture feature, sample structure feature and sample residual feature are equalized by the feature equalization module of the image restoration network to obtain the sample balanced fusion feature, and the sample balanced fusion feature, the several shallow features and the several deep features can be decoded by the decoder of the image restoration network to obtain the predicted enhanced image corresponding to the sample image.

[0147] Step 307: Determine a first difference between the sample residual feature and the residual label of the sample image, a second difference between the sample texture feature and the texture label of the sample image, a third difference between the sample structure feature and the structure label of the sample image, and a fourth difference between the predicted enhanced image and the target label of the sample image.

[0148] The server can evaluate the first difference between the sample residual feature and the residual label, the second difference between the sample texture feature and the texture label, the third difference between the sample structure feature and the structure label, and the fourth difference between the predicted enhanced image and the target label based on a preset loss function.

[0149] The loss function for sample residual features and residual labels can be expressed by the following formula (8):

[0150]

[0151] Here, I res Represents the sample residual characteristics; I noise Represents the residual label. Both cannot be normalized to -1 to 1, because the noise information contains the suppression effect on brightness, and the residual feature should be able to reflect this part of the negative intensity information.

[0152] The server can calculate the first difference using the above formula (8).

[0153] The loss function for sample texture features and texture labels can be expressed by the following formula (9):

[0154] L txf =||I tx -I tx_gt ||1 (9)

[0155] Here, I tx Represents the sample texture feature; I tx_gt Represents a texture tag.

[0156] The server can calculate the second difference using the above formula (9).

[0157] The loss function for sample structural features and structural labels can be expressed by the following formula (10):

[0158] L stf =||I st -I st_gt ||1 (10)

[0159] Here, I st Represents the sample structure characteristics; I st_gt Represents a structure tag.

[0160] The server can calculate the third difference using the above formula (10).

[0161] The loss function for predicting enhanced images and target labels can be expressed as follows:

[0162] L cons =||I out -I gt ||1 (11)

[0163] Here, I out Represents the predicted enhanced image; I gt represents the target label, and the scale of both is consistent with the sample image.

[0164] The server can calculate the fourth difference using the above formula (11).

[0165] Step 308: Under the generative adversarial mechanism, the residual feature extraction network and the image restoration network are used as generators, and the network parameters of the residual feature extraction network and the image restoration network are adjusted in combination with the first difference, the second difference, the third difference, and the fourth difference.

[0166] Under the generative adversarial mechanism, the residual feature extraction network and the image restoration network serve as the generators of the generative adversarial network, and are trained through game theory with the discriminator of the generative adversarial network to achieve better training results.

[0167] In the generative adversarial network, the discriminator and the generator are trained alternately. During the discriminator training process, the loss function of the generative adversarial network can be expressed by the following formula (12):

[0168]

[0169] During the generator training process, the loss function of the generative adversarial network can be expressed by the following formula (13):

[0170]

[0171] in, I pre Represents the predicted enhanced image; I gt Represents the target label.

[0172] Having different loss function forms for the generator and discriminator can help the network benefit from the gradient of generated data and real data, thereby improving the stability of training.

[0173] The final total loss function can be expressed by the following formula (14):

[0174] L total =λ a Ladv +λ r L res +λ st L stf +λ tx L txf +λ c L cons (14)

[0175] Here, λ a ,λ r ,λ st ,λ tx ,λ c are pre-configured weights, for example, λ a =0.2;λ r =0.1;λ st =1;λ tx =1;λ c = 1. Here, Ladv is the loss function of the generative adversarial network. During the discriminator training phase, the loss function is the above formula (12), and during the generator training phase, the loss function is the above formula (13).

[0176] Step 309: Repeat the above process until the residual feature extraction network and the image restoration network converge.

[0177] After repeated iterative training, the network parameters of the residual feature extraction network and the image restoration network tend to be stable. At this time, it can be considered that the two have converged and can be used in the blind enhancement method of the image in this application.

[0178] In one embodiment, before training the residual feature extraction network and the image restoration network as a whole, the residual feature extraction network can be pre-trained. Figure 12 , is a flow chart of a pre-trained network model provided in an embodiment of the present application, such as Figure 12 As shown, the method may include the following steps 303 to 305.

[0179] Step 303: Input the sample images in the second sample data set into an untrained residual feature extraction network to obtain the predicted residual features output by the residual feature extraction network.

[0180] The second sample dataset includes a large number of sample images, and the sample images may belong to the same category as the target image undergoing blind enhancement processing. In one embodiment, the second sample dataset may belong to the first sample dataset, or the second sample dataset may be the first sample dataset.

[0181] The sample image can carry a pre-labeled residual label, which indicates the damage information in the sample image, including global noise and semantic noise.

[0182] The server inputs the sample image into the residual feature extraction network, calculates the sample image through the residual feature extraction network, and outputs the predicted residual features.

[0183] Step 304: Adjust the network parameters of the residual feature extraction network based on the difference between the predicted residual features and the residual labels of the sample images.

[0184] Step 305: Repeat the above process until the residual feature extraction network converges.

[0185] The server can evaluate the difference between the predicted residual features and the residual labels based on a preset loss function, and adjust the network parameters of the residual feature extraction network based on the function value of the loss function.

[0186] After repeated iterations, when the network parameters of the residual feature extraction network tend to be stable, it can be considered that the residual feature extraction network has initially converged after pre-training. This can be used in the subsequent second-stage training process combined with the image restoration network.

[0187] This application proposes a multi-task blind enhancement algorithm for global noise and semantic noise (for dendrite images, semantic noise can include oxides, etc.), which combines residual learning and repair learning to construct a two-stage network. The two-stage network includes a residual feature extraction network and an image repair network. The residual feature extraction network can be used to extract residual features, and the image repair network is used for fine denoising and image repair. For multiple features such as residual noise level, texture level, and structure level, the inconsistency of multi-level features is eliminated in the spatial dimension and channel dimension and aggregated, so that a blind enhanced image with higher repair quality can be obtained based on the balanced fusion features, texture features, and structural features after aggregation.

[0188] The application of this solution on the test set showed significant improvements in PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity) compared to related technologies. Table 2 below shows the comparison results of this solution with DIP (Deep Image Prior) and DBII (Deep Blind Image Inpainting).

[0189]

[0190] Table 2

[0191] See also Figure 13 , which is a schematic diagram comparing the effects of image restoration provided by an embodiment of the present application, such as Figure 13As shown, Figure a is the original image to be restored, Figure b is the image restored by the DBII solution, Figure c is the image restored by the DIP solution, and Figure d is the image restored by the solution of this application.

[0192] See also Figure 14 , is a schematic diagram comparing the effects of image restoration provided by another embodiment of the present application, such as Figure 13 As shown, Figure a is the original image to be restored, Figure b is the image restored by the DBII solution, Figure c is the image restored by the DIP solution, and Figure d is the image restored by the solution of this application.

[0193] As can be seen from the above comparison diagram, the image restored by the present application solution is closest to the distribution of the artificial denoised image. Not only is the image itself smooth and clean, effectively suppressing global noise, but the occluded area is also restored with high quality. For semi-transparent occluders, it can be perfectly restored, and the restoration result is basically close to the true value; for complete occluders, it can effectively fill the occluded area. Although dendrite shape restoration errors are inevitable in this process, the solid-liquid phase distribution is roughly close to the true value. Compared with the original very serious occluded area, the restoration effect can also meet expectations. For some strong noise images, the restoration results of this algorithm can even retain more solid dendrite details than the denoised images with manually adjusted parameters. For subsequent quantitative analysis subtasks such as solid-liquid phase segmentation and texture classification and recognition, this feature can help produce more accurate analysis results.

[0194] Figure 15 This is a blind enhancement device for an image according to an embodiment of the present invention. Figure 15 As shown, the device may include:

[0195] A first extraction module 410 is configured to input a target image into a trained residual feature extraction network to obtain residual features of the target image;

[0196] A second extraction module 420 is configured to input the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image;

[0197] A filling module 430 is configured to perform multi-scale filling processing on the texture features and the structural features respectively through the multi-scale filling module of the image restoration network to obtain filled texture features and filled structural features;

[0198] An equalization module 440 is configured to input the filled texture feature, the filled structural feature, and the residual feature into a feature equalization module of a trained image restoration network to obtain a balanced fusion feature;

[0199] The decoding module 450 is configured to input the balanced fusion feature, the texture feature, and the structural feature into a decoder of a trained image restoration network to obtain a blind enhanced image of the target image.

[0200] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned blind enhancement method for images, and will not be repeated here.

[0201] In several embodiments provided in this application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of the devices, methods, and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a portion of code, and the module, program segment, or a portion of code contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions marked in the boxes may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, may be implemented using a dedicated hardware-based system that performs the specified functions or actions, or may be implemented using a combination of dedicated hardware and computer instructions.

[0202] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0203] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A blind enhancement method for an image, characterized in that: include: Inputting the target image into the trained residual feature extraction network to obtain the residual features of the target image; Inputting the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image; Performing multi-scale filling processing on the texture features and the structural features respectively through the multi-scale filling module of the image restoration network to obtain filled texture features and filled structural features; Inputting the filled texture feature, the filled structural feature and the residual feature into a feature equalization module of a trained image restoration network to obtain a balanced fusion feature; Inputting the balanced fusion feature, the texture feature, and the structural feature into a decoder of a trained image restoration network to obtain a blind enhanced image of the target image; The residual feature extraction network and the image restoration network are trained in the following manner: Inputting the sample images in the first sample data set into a pre-trained residual feature extraction network and an untrained image restoration network, respectively, to obtain sample residual features output by the pre-trained residual feature extraction network, sample texture features and sample structural features output by the encoder of the image restoration network, and predicted enhanced images output by the decoder of the image restoration network; Determining a first difference between the sample residual feature and the residual label of the sample image, a second difference between the sample texture feature and the texture label of the sample image, a third difference between the sample structure feature and the structure label of the sample image, and a fourth difference between the predicted enhanced image and the target label of the sample image; Under a generative adversarial mechanism, using the residual feature extraction network and the image restoration network as generators, and adjusting network parameters of the residual feature extraction network and the image restoration network based on the first difference, the second difference, the third difference, and the fourth difference; The above process is repeated until the residual feature extraction network and the image restoration network converge.

2. The method according to claim 1, characterized in that The encoder is a trained encoder based on a U-Net image restoration network; Inputting the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image includes: Inputting the target image into the encoder so that the encoder performs encoding processing on the target image to obtain a plurality of shallow features and a plurality of deep features; The shallow features are used as texture features, and the deep features are used as structural features.

3. The method according to claim 1, characterized in that The multi-scale filling module includes a first multi-scale filling module and a second multi-scale filling module; The multi-scale filling module of the image restoration network performs multi-scale filling processing on the texture features and the structural features respectively to obtain the filled texture features and the filled structural features, including: Performing fusion processing on multiple texture features and multiple structural features respectively to obtain specified texture features and specified structural features; Inputting the designated texture feature into the first multi-scale filling module, obtaining a plurality of filled sub-texture features output by the first multi-scale filling module, and fusing the plurality of filled sub-texture features into a filled texture feature; The designated structural feature is input into the second multi-scale filling module to obtain a plurality of filled sub-structural features output by the second multi-scale filling module, and the plurality of filled sub-structural features are fused into a filled structural feature.

4. The method according to claim 1, wherein The step of inputting the filled texture feature, the filled structural feature, and the residual feature into a feature equalization module of a trained image restoration network to obtain a balanced fusion feature comprises: Splicing the filled texture feature and the filled structural feature to obtain the feature to be balanced; Inputting the to-be-equalized features into the feature equalization module, so that the feature equalization module performs channel-dimensional equalization processing on the to-be-equalized features to obtain an intermediate feature map; The feature equalization module performs spatial dimension equalization processing on the intermediate feature map and the residual feature to obtain the balanced fusion feature.

5. The method according to claim 1, characterized in that The decoder is a decoder of a trained U-Net-based image restoration network; The step of inputting the balanced fusion feature, the texture feature, and the structural feature into a decoder of a trained image restoration network to obtain a blind enhanced image of the target image comprises: The balanced fusion features, the texture features, and the structural features are output to each network layer of the decoder through a jump connection, so that the decoder outputs a blind enhanced image of the target image.

6. The method according to claim 1, characterized in that The pre-training process of the residual feature extraction network is as follows: Inputting the sample images in the second sample data set into an untrained residual feature extraction network to obtain predicted residual features output by the residual feature extraction network; Adjusting network parameters of the residual feature extraction network based on a difference between the predicted residual feature and the residual label of the sample image; The above process is repeated until the residual feature extraction network converges.

7. A blind enhancement device for an image, characterized in that: include: A first extraction module is used to input the target image into the trained residual feature extraction network to obtain the residual features of the target image; A second extraction module is configured to input the target image into an encoder of a trained image restoration network to obtain texture features and structural features of the target image; A filling module, configured to perform multi-scale filling processing on the texture features and the structural features respectively through the multi-scale filling module of the image restoration network to obtain filled texture features and filled structural features; An equalization module, configured to input the filled texture feature, the filled structural feature, and the residual feature into a feature equalization module of a trained image restoration network to obtain a balanced fusion feature; A decoding module, configured to input the balanced fusion feature, the texture feature, and the structural feature into a decoder of a trained image restoration network to obtain a blind enhanced image of the target image; The residual feature extraction network and the image restoration network are trained in the following manner: Inputting the sample images in the first sample data set into a pre-trained residual feature extraction network and an untrained image restoration network, respectively, to obtain sample residual features output by the pre-trained residual feature extraction network, sample texture features and sample structural features output by the encoder of the image restoration network, and predicted enhanced images output by the decoder of the image restoration network; Determining a first difference between the sample residual feature and the residual label of the sample image, a second difference between the sample texture feature and the texture label of the sample image, a third difference between the sample structure feature and the structure label of the sample image, and a fourth difference between the predicted enhanced image and the target label of the sample image; Under a generative adversarial mechanism, using the residual feature extraction network and the image restoration network as generators, and adjusting network parameters of the residual feature extraction network and the image restoration network based on the first difference, the second difference, the third difference, and the fourth difference; The above process is repeated until the residual feature extraction network and the image restoration network converge.

8. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the blind image enhancement method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program can be executed by a processor to complete the blind image enhancement method according to any one of claims 1 to 6.

Images