Image processing method and device, electronic equipment and computer storage medium

Abstract

The application provides a processing method and device of an image, an electronic device and a computer storage medium. The processing method of the image comprises the following steps: after receiving an image to be processed; performing first feature processing on the image to be processed to obtain global features of the image to be processed; performing second feature processing on the image to be processed to obtain local features of the image to be processed; finally, performing fusion processing on the local features, the global features and the image to be processed to obtain a normally exposed image. Thus, the dark image can be effectively processed to obtain a normally exposed image.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an image processing method, apparatus, electronic device, and computer storage medium. Background Technology

[0002] Images captured by cameras under insufficient or uneven lighting conditions often suffer from loss of detail, underexposure in certain areas, and low resolution. These types of images provide a very poor visual experience and also affect the accuracy of image information collected by computer vision systems. There are generally two solutions to this problem: upgrading the hardware or improving the algorithm.

[0003] With the development of industrial technology, camera equipment has been rapidly improved. However, when taking pictures at night, cameras often cannot capture high-brightness images. Therefore, researchers have attempted to solve the problem of insufficient lighting through software and have adopted image enhancement algorithms to improve the brightness and contrast of captured images.

[0004] In existing technologies, low-light enhancement methods typically include histogram equalization-based methods and Retinex model-based methods, with the latter receiving relatively more attention. Typical Retinex model-based methods decompose low-light images into reflection and illumination components through priors or regularization. This approach has several limitations: 1) The assumption that the reflection component is an ideal for the enhancement result does not always hold, especially considering various illumination characteristics, which can lead to unrealistic enhancements such as loss of detail and color distortion; 2) Noise is typically ignored in the Retinex model, and therefore, it is either preserved or amplified in the enhancement result; 3) Finding an effective prior or regularization is challenging, and inaccurate priors or regularization can lead to artifacts and color deviations in the enhancement result; 4) Due to its complex optimization process, the runtime is relatively long. Summary of the Invention

[0005] In view of this, this application provides an image processing method, apparatus, electronic device, and computer storage medium that can effectively process low-light images to obtain properly exposed images.

[0006] The first aspect of this application provides an image processing method, comprising:

[0007] Receive the image to be processed;

[0008] The image to be processed is subjected to a first feature processing to obtain the global features of the image to be processed;

[0009] The image to be processed is subjected to a second feature processing to obtain the local features of the image to be processed;

[0010] The local features, the global features, and the image to be processed are fused together to obtain a normally exposed image.

[0011] Optionally, the first feature processing of the image to be processed to obtain the global features of the image to be processed includes:

[0012] The size of the image to be processed is adjusted to a preset size to obtain the target image;

[0013] The target image is input to a first feature extraction component, and a first feature result is output; wherein, the first feature extraction component includes: a convolutional layer, a visual converter, and an average pooling layer;

[0014] The first feature result is input to the second feature extraction component, and the global features of the image to be processed are output; wherein, the second feature extraction component includes: a convolutional layer, a visual converter and an average pooling layer.

[0015] Optionally, the second feature processing of the image to be processed to obtain local features of the image to be processed includes:

[0016] The image to be processed is input into the first-level module of the downsampling network, and the first local feature is output; wherein, the first-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function;

[0017] The first local feature is input into the second-level module of the downsampling network, and the second local feature is output; wherein, the second-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function;

[0018] The second local feature is input into the third-level module of the downsampling network, and the third local feature is output; wherein, the third-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function;

[0019] The third local feature and the global feature are input into the fourth-level module of the downsampling network, and the fourth local feature is output. The fourth-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function.

[0020] The fourth local feature and the global feature are input into the fifth-level module of the downsampling network, and the fifth local feature is output. The fifth-level module of the downsampling network includes a max pooling layer, a convolutional layer and an activation function.

[0021] Optionally, the step size of the max pooling layer is 2.

[0022] Optionally, the step of fusing the local features, the global features, and the image to be processed to obtain a normally exposed image includes:

[0023] The fifth local feature is input into the fifth-level module of the upsampling network, and the fifth upsampled feature is output; wherein, the fifth-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0024] The global feature, the fifth local feature, and the fifth upsampled feature are concatenated to obtain the first fused feature;

[0025] The first fused feature is input into the fourth-level module of the upsampling network, and the output is the fourth upsampling feature; wherein, the fourth-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0026] The fourth local feature and the fourth upsampled feature are concatenated to obtain the second fused feature;

[0027] The second fused feature is input into the third-level module of the upsampling network, and the third upsampling feature is output; wherein, the third-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0028] The third local feature and the third upsampled feature are concatenated to obtain the third fused feature;

[0029] The third fused feature is input into the second-level module of the upsampling network, and the output is the second upsampling feature; wherein, the second-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0030] The second local feature and the second upsampled feature are concatenated to obtain the fourth fused feature;

[0031] The fourth fused feature is input into the first-level module of the upsampling network, and the first upsampling feature is output; wherein, the first-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0032] The first upsampled feature and the image to be processed are concatenated to obtain the fifth fused feature;

[0033] The fifth fusion feature is input into the image restoration module of the upsampling network to obtain a properly exposed image.

[0034] Optionally, the cascading operation of the global feature, the fifth local feature, and the fifth upsampled feature to obtain the first fused feature includes:

[0035] The global features are scaled in dimension using the fully connected layer of the injection module to obtain multiple global sub-features;

[0036] The multiple global sub-features are rearranged to obtain a target global sub-feature; wherein the shape of the target global sub-feature matches the fifth local feature;

[0037] For each of the target global sub-features, the product of the target global sub-feature and the fifth local feature is used as the preliminary fusion feature of the target global sub-feature;

[0038] The sum of the preliminary fusion features of all the target global sub-features is taken as the first fusion feature.

[0039] A second aspect of this application provides an image processing apparatus, comprising:

[0040] The receiving unit is used to receive the image to be processed;

[0041] The first feature processing unit is used to perform first feature processing on the image to be processed to obtain the global features of the image to be processed.

[0042] The second feature processing unit is used to perform second feature processing on the image to be processed to obtain local features of the image to be processed.

[0043] The fusion processing unit is used to fuse the local features, the global features, and the image to be processed to obtain a normally exposed image.

[0044] Optionally, the first feature processing unit includes:

[0045] A size adjustment unit is used to adjust the size of the image to be processed to a preset size to obtain the target image;

[0046] The first input unit is used to input the target image into the first feature extraction component and output a first feature result; wherein, the first feature extraction component includes: a convolutional layer, a visual converter, and an average pooling layer;

[0047] The second input unit is used to input the first feature result into the second feature extraction component and output the global features of the image to be processed; wherein, the second feature extraction component includes: a convolutional layer, a visual converter and an average pooling layer.

[0048] Optionally, the second feature processing unit includes:

[0049] The first downsampling subunit is used to input the image to be processed into the first-level module of the downsampling network and output the first local feature; wherein, the first-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function;

[0050] The second downsampling subunit is used to input the first local features into the second-level module of the downsampling network and output the second local features; wherein, the second-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function;

[0051] The third downsampling subunit is used to input the second local features into the third-level module of the downsampling network and output the third local features; wherein, the third-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function;

[0052] The fourth downsampling subunit is used to input the third local feature and the global feature into the fourth-level module of the downsampling network and output the fourth local feature; wherein, the fourth-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function;

[0053] The fifth downsampling subunit is used to input the fourth local feature and the global feature into the fifth-level module of the downsampling network and output the fifth local feature; wherein, the fifth-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function.

[0054] Optionally, the step size of the max pooling layer is 2.

[0055] Optionally, the fusion processing unit includes:

[0056] The first upsampling subunit is used to input the fifth local feature into the fifth-level module of the upsampling network and output the fifth upsampled feature; wherein, the fifth-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0057] The first fusion processing subunit is used to perform a cascade operation on the global feature, the fifth local feature and the fifth upsampled feature to obtain the first fused feature;

[0058] The second upsampling subunit is used to input the first fused feature into the fourth-level module of the upsampling network and output the fourth upsampling feature; wherein, the fourth-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0059] The second fusion processing subunit is used to perform a cascade operation on the fourth local feature and the fourth upsampled feature to obtain the second fused feature;

[0060] The third upsampling subunit is used to input the second fused feature into the third-level module of the upsampling network and output the third upsampling feature; wherein, the third-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0061] The third fusion processing subunit is used to perform a cascade operation on the third local feature and the third upsampled feature to obtain the third fused feature;

[0062] The fourth upsampling subunit is used to input the third fused feature into the second-level module of the upsampling network and output the second upsampling feature; wherein, the second-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0063] The fourth fusion processing subunit is used to cascade the second local feature and the second upsampled feature to obtain the fourth fused feature;

[0064] The fifth upsampling subunit is used to input the fourth fused feature into the first-level module of the upsampling network and output the first upsampling feature; wherein, the first-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer;

[0065] The fifth fusion processing subunit is used to cascade the first upsampled feature and the image to be processed to obtain the fifth fusion feature;

[0066] The third input unit is used to input the fifth fusion feature into the image restoration module of the upsampling network to obtain a normally exposed image.

[0067] Optionally, the first fusion processing subunit includes:

[0068] The scaling unit is used to scale the global features in dimensions using the fully connected layer of the injection module to obtain multiple global sub-features;

[0069] The rearrangement unit is used to rearrange the multiple global sub-features to obtain a target global sub-feature; wherein the shape of the target global sub-feature matches the fifth local feature;

[0070] The first calculation unit is used to take the product of the target global sub-feature and the fifth local feature as the preliminary fusion feature of the target global sub-feature for each target global sub-feature;

[0071] The second calculation unit is used to sum the preliminary fusion features of all the target global sub-features as the first fusion feature.

[0072] A third aspect of this application provides an electronic device, comprising:

[0073] One or more processors;

[0074] A storage device on which one or more programs are stored;

[0075] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of the first aspects.

[0076] The fourth aspect of this application provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the image processing method as described in any one of the first aspects.

[0077] As can be seen from the above solutions, this application provides an image processing method, apparatus, electronic device, and computer storage medium. The image processing method includes: after receiving an image to be processed; performing a first feature processing on the image to be processed to obtain local features of the image to be processed; then performing a second feature processing on the image to be processed to obtain global features of the image to be processed; finally, performing a fusion processing on the local features, the global features, and the image to be processed to obtain a normally exposed image. This achieves the goal of effectively processing low-light images to obtain normally exposed images. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0079] Figure 1 A detailed flowchart of an image processing method provided in an embodiment of this application;

[0080] Figure 2 A detailed flowchart of a method for obtaining global features of an image to be processed, provided in another embodiment of this application;

[0081] Figure 3 A detailed flowchart of a method for obtaining local features of an image to be processed, provided in another embodiment of this application;

[0082] Figure 4 A detailed flowchart of a method for obtaining a properly exposed image is provided in another embodiment of this application;

[0083] Figure 5 A detailed flowchart of a method for obtaining a first fusion feature is provided in another embodiment of this application;

[0084] Figure 6 A schematic diagram of a network structure for implementing an image processing method according to another embodiment of this application;

[0085] Figure 7 A schematic diagram of an image processing apparatus provided for another embodiment of this application;

[0086] Figure 8 This is a schematic diagram of an electronic device that implements an image processing method, according to another embodiment of this application. Detailed Implementation

[0087] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] It should be noted that the concepts of "first," "second," etc., mentioned in this application are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] This application provides an image processing method, such as... Figure 1 As shown, the specific steps include:

[0090] S101, Receive the image to be processed.

[0091] S102. Perform first feature processing on the image to be processed to obtain the global features of the image to be processed.

[0092] Optionally, in another embodiment of this application, one implementation of step S102 is as follows: Figure 2 As shown, it includes:

[0093] S201. Adjust the size of the image to be processed to the preset size to obtain the target image.

[0094] The preset size can be set or changed by technicians or authorized personnel, and there is no limitation here. For example, 128×128.

[0095] S202. Input the target image into the first feature extraction component and output the first feature result.

[0096] The first feature extraction component includes a convolutional layer (Conv), a vision transformer (ViT), and an average pooling layer (AvgPool).

[0097] In the specific implementation of this application, the structure of the first feature extraction component can be, but is not limited to: Conv+ViT+Conv+AvgPool.

[0098] S203. Input the first feature result into the second feature extraction component and output the global features of the image to be processed.

[0099] The second feature extraction component includes a convolutional layer, a visual converter, and an average pooling layer.

[0100] In the specific implementation of this application, the structure of the second feature extraction component can be, but is not limited to: Conv+ViT+Conv+AvgPool.

[0101] S103. Perform second feature processing on the image to be processed to obtain the local features of the image to be processed.

[0102] Optionally, in another embodiment of this application, one implementation of step S103 is as follows: Figure 3 As shown, it includes:

[0103] S301. Input the image to be processed into the first-level module of the downsampling network and output the first local feature.

[0104] The first-level module of the downsampling network includes a max-pooling layer, a convolutional layer, and an activation function (LReLU). The stride of the max-pooling layer is 2.

[0105] In the specific implementation of this application, the structure of the first-level module of the downsampling network can be, but is not limited to: Conv+LReLU+Conv+LReLU+MaxPool.

[0106] S302. Input the first local feature into the second-level module of the downsampling network and output the second local feature.

[0107] The second-level module of the downsampling network consists of a max-pooling layer, a convolutional layer, and an activation function. The stride of the max-pooling layer is 2.

[0108] In the specific implementation of this application, the structure of the second-level module of the downsampling network can be, but is not limited to: Conv+LReLU+MaxPool.

[0109] S303. Input the second local feature into the third-level module of the downsampling network and output the third local feature.

[0110] The third-level module of the downsampling network consists of a max-pooling layer, a convolutional layer, and an activation function. The stride of the max-pooling layer is 2.

[0111] In the specific implementation of this application, the structure of the third-level module of the downsampling network can be, but is not limited to: Conv+LReLU+MaxPool.

[0112] S304. Input the third local feature and the global feature into the fourth-level module of the downsampling network, and output the fourth local feature.

[0113] The fourth-level module of the downsampling network consists of a max-pooling layer, a convolutional layer, and an activation function. The stride of the max-pooling layer is 2.

[0114] In the specific implementation of this application, the structure of the fourth-level module of the downsampling network can be, but is not limited to, Conv+LReLU+MaxPool. The global features are fused into the third local features through the injection module (IM:Injection Module), and the fused features of the third local features and global features are then input into the fourth module of the downsampling network.

[0115] S305. Input the fourth local feature and the global feature into the fifth-level module of the downsampling network, and output the fifth local feature.

[0116] The fifth-level module of the downsampling network consists of a max-pooling layer, a convolutional layer, and an activation function. The stride of the max-pooling layer is 2.

[0117] In the specific implementation of this application, the structure of the fifth-level module of the downsampling network can be, but is not limited to, Conv+LReLU+ViT+MaxPool. The global features are fused into the fourth local features through the injection module (IM:Injection Module), and the fused features of the fourth local features and global features are then input into the fifth module of the downsampling network.

[0118] S104. The local features, global features, and the image to be processed are fused to obtain a properly exposed image.

[0119] Optionally, in another embodiment of this application, one implementation of step S104 is as follows: Figure 4 As shown, it includes:

[0120] S401. Input the fifth local feature into the fifth-level module of the upsampling network and output the fifth upsampling feature.

[0121] The fifth-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0122] In the specific implementation of this application, the structure of the fifth-level module of the upsampling network can be, but is not limited to: Conv+LReLU+ViT+Upsample.

[0123] S402. Concatenate the global feature, the fifth local feature, and the fifth upsampled feature to obtain the first fused feature.

[0124] Optionally, in another embodiment of this application, one implementation of step S402 is as follows: Figure 5 As shown, it includes:

[0125] S501. The fully connected layer of the injection module is used to scale the dimensions of the global features to obtain multiple global sub-features.

[0126] Specifically, in the implementation of this application, the global features are scaled in dimension through fully connected layers (FC) to obtain multiple global sub-features.

[0127] S502. Perform feature rearrangement on multiple global sub-features to obtain the target global sub-features.

[0128] Among them, the shape of the target global sub-feature matches the fifth local feature.

[0129] Specifically, in the implementation of this application, feature rearrangement is performed through the Reshape operation.

[0130] S503. For each target global sub-feature, the product of the target global sub-feature and the fifth local feature is used as the initial fusion feature of the target global sub-feature.

[0131] S504. The sum of the preliminary fusion features of all target global sub-features is taken as the first fusion feature.

[0132] S403. Input the first fused feature into the fourth-level module of the upsampling network and output the fourth upsampling feature.

[0133] The fourth-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0134] In the specific implementation of this application, the structure of the fourth secondary module of the upsampling network can be, but is not limited to: Conv+LReLU+ViT+Conv+LReLU+Upsample.

[0135] S404. Concatenate the fourth local feature and the fourth upsampled feature to obtain the second fused feature.

[0136] S405. Input the second fused feature into the third-level module of the upsampling network and output the third upsampling feature.

[0137] The third-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0138] In the specific implementation of this application, the structure of the third-level module of the upsampling network can be, but is not limited to: Conv+LReLU+Conv+LReLU+Upsample.

[0139] S406. Concatenate the third local feature and the third upsampled feature to obtain the third fused feature.

[0140] S407. Input the third fusion feature into the second-level module of the upsampling network and output the second upsampling feature.

[0141] The second-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0142] In the specific implementation of this application, the structure of the second-level module of the upsampling network can be, but is not limited to: Conv+LReLU+Conv+LReLU+Upsample.

[0143] S408. Concatenate the second local feature and the second upsampled feature to obtain the fourth fused feature.

[0144] S409. Input the fourth fusion feature into the first-level module of the upsampling network and output the first upsampling feature.

[0145] The first-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0146] In the specific implementation of this application, the structure of the first-level module of the upsampling network can be, but is not limited to: Conv+LReLU+Conv+LReLU+Upsample.

[0147] S410. The first upsampled feature and the image to be processed are concatenated to obtain the fifth fused feature.

[0148] S411. Input the fifth fusion feature into the image restoration module of the upsampling network to obtain a normally exposed image.

[0149] In the specific implementation of this application, the structure of the image restoration module of the upsampling network can be, but is not limited to: Conv+LReLU+Conv+LReLU+Conv.

[0150] It should be noted that the above embodiments of this application employ 5-time downsampling network processing and 5-time upsampling network processing. In the actual application of this application, 3 times, 7 times, 10 times, etc. can also be used. The above embodiments of this application are provided as examples. It can be understood that the more times, the better the effect of the finally restored normal exposure image.

[0151] Now combined with such Figure 6 The schematic diagram of the network structure for implementing the image processing method is shown below to illustrate this application: First, after receiving the image to be processed, the image to be processed (input) is input into the shrinking sub-network (the left half of the Transformer U Module) and the global information guidance sub-network (Global Guide Module), respectively.

[0152] The Global Guide Module first resizes the input image to obtain a 128x128 target image. Then, it uses the Conv+ViT+Conv+AvgPool component to extract features and downsample the target image. Finally, it passes through the Conv+ViT+Conv+AvgPool component again to obtain the global features of the image with global information.

[0153] The shrinking subnetwork (the left half of the Transformer U Module) performs downsampling operations with each shrinkage by a pooling layer with a stride of 2. First, the image to be processed is input into the first-level module of the downsampling network: Conv+LReLU+Conv+LReLU+MaxPool to obtain the first downsampling feature D1; then it goes through Conv+LReLU+MaxPool again to obtain the second downsampling feature D2; then through Conv+LReLU+MaxPool again to obtain the third downsampling feature D3; then through Conv+LReLU+MaxPool again to obtain the fourth downsampling feature D4; and then through Conv+LReLU+ViT+MaxPool again to obtain the fifth downsampling feature D5.

[0154] The global features of the image to be processed are introduced into the Transformer U Module through the Injection Module (IM) to aid in the final feature reconstruction.

[0155] The injection module scales the dimensions of the global features through a fully connected layer, and then rearranges the multiple global sub-features through a Reshape operation to obtain target global sub-features to match the shape of local features. Subsequently, the global sub-features are multiplied with the input local features to obtain preliminary fused features. The sum of the preliminary fused features of all target global sub-features is used as the fused feature.

[0156] The right half of the TransformerU Module is the upsampling subnetwork, which is upsampled twice by nearest neighbor interpolation (Upsample) for each expansion. First, the feature D5 of the shrinking sub-network is processed through Conv+LReLU+ViT+Upsample to obtain the upsampled feature U5. Then, D5 and U5 are cascaded to obtain the fused feature, and then processed through Conv+LReLU+ViT+Conv+LReLU+Upsample to obtain the upsampled feature U4. Then, D4 and U4 are cascaded to obtain the fused feature, and then processed through Conv+LReLU+Conv+LReLU+Upsample to obtain the upsampled feature U3. Then, D3 and U3 are cascaded to obtain the fused feature, and then processed through Conv+LReLU+Conv+LReLU+Upsample to obtain the upsampled feature U2. Then, D2 and U2 are cascaded to obtain the fused feature, and then processed through Conv+LReLU+Conv+LReLU+Upsample to obtain the upsampled feature U1. Finally, the input image to be processed and U1 are cascaded to obtain the fused feature, and then processed through Conv+LReLU+Conv+LReLU+Conv to obtain the restored normally exposed image.

[0157] The training method for the network that implements the above image processing method can be, but is not limited to, the following methods, and is not limited here:

[0158] 1. Data collection: The low-light dataset (LOL) collected by Chen et al. was used for training in this invention.

[0159] 2. Data Processing: The dark light image set and all bright light images are randomly cropped into image blocks of size 256*256*3. Data augmentation operations are performed on the cropped image blocks, specifically including: rotating 90 degrees clockwise, rotating 90 degrees counterclockwise, and rotating 180 degrees.

[0160] Because the image patches used during training only contain local information about the entire image, while the color of an image has a strong overall character, previous networks often resulted in color imbalances in the recovered images. Therefore, strong overall prior knowledge of color information is needed as guidance to better recover a color image. This invention therefore designs a global information guidance module. The input to this model is the entire image corresponding to the cropped image patch, and the output is a feature vector containing global information.

[0161] 3. Network Training: Using the PyTorch framework, the initial learning rate was 0.0001, and a total of 1 million training iterations were performed. Every 300,000 iterations, the learning rate was reduced to half of its previous value.

[0162] 4. Loss function: The mean absolute error (MAR) L1 loss function is used to optimize the network.

[0163] As can be seen from the above scheme, this application provides an image processing method: after receiving the image to be processed; performing a first feature processing on the image to be processed to obtain local features of the image to be processed; then performing a second feature processing on the image to be processed to obtain global features of the image to be processed; finally, fusing the local features, global features, and the image to be processed to obtain a normally exposed image. This achieves the goal of effectively processing low-light images to obtain normally exposed images.

[0164] Another embodiment of this application provides an image processing apparatus, such as... Figure 7 As shown, it specifically includes:

[0165] The receiving unit 701 is used to receive the image to be processed.

[0166] The first feature processing unit 702 is used to perform first feature processing on the image to be processed to obtain the global features of the image to be processed.

[0167] Optionally, in another embodiment of this application, one implementation of the first feature processing unit 702 includes:

[0168] The size adjustment unit is used to adjust the size of the image to be processed to a preset size to obtain the target image.

[0169] The first input unit is used to input the target image into the first feature extraction component and output the first feature result.

[0170] The first feature extraction component includes a convolutional layer, a visual converter, and an average pooling layer.

[0171] The second input unit is used to input the first feature result into the second feature extraction component and output the global features of the image to be processed.

[0172] The second feature extraction component includes a convolutional layer, a visual converter, and an average pooling layer.

[0173] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 2 As shown, it will not be elaborated further here.

[0174] The second feature processing unit 703 is used to perform second feature processing on the image to be processed to obtain local features of the image to be processed.

[0175] Optionally, in another embodiment of this application, one implementation of the second feature processing unit 703 includes:

[0176] The first downsampling subunit is used to input the image to be processed into the first-level module of the downsampling network and output the first local feature.

[0177] The first-level module of the downsampling network includes: a max pooling layer, a convolutional layer, and an activation function.

[0178] The second downsampling subunit is used to input the first local features into the second-level module of the downsampling network and output the second local features.

[0179] The second-level module of the downsampling network includes: a max pooling layer, a convolutional layer, and an activation function.

[0180] The third downsampling subunit is used to input the second local features into the third-level module of the downsampling network and output the third local features.

[0181] The third-level module of the downsampling network includes: a max pooling layer, a convolutional layer, and an activation function.

[0182] The fourth downsampling subunit is used to input the third local features and global features into the fourth-level module of the downsampling network and output the fourth local features.

[0183] The fourth-level module of the downsampling network includes: a max pooling layer, a convolutional layer, and an activation function.

[0184] The fifth downsampling subunit is used to input the fourth local feature and the global feature into the fifth-level module of the downsampling network, and output the fifth local feature.

[0185] The fifth-level module of the downsampling network includes: a max pooling layer, a convolutional layer, and an activation function.

[0186] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 3 As shown, it will not be elaborated further here.

[0187] Optionally, in another embodiment of this application, the step size of the max pooling layer is 2.

[0188] The fusion processing unit 704 is used to fuse local features, global features, and the image to be processed to obtain a properly exposed image.

[0189] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 1 As shown, it will not be elaborated further here.

[0190] Optionally, in another embodiment of the application, one implementation of the fusion processing unit 704 includes:

[0191] The first upsampling subunit is used to input the fifth local feature into the fifth-level module of the upsampling network and output the fifth upsampling feature.

[0192] The fifth-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0193] The first fusion processing subunit is used to cascade the global feature, the fifth local feature, and the fifth upsampled feature to obtain the first fused feature.

[0194] The second upsampling subunit is used to input the first fused feature into the fourth-level module of the upsampling network and output the fourth upsampling feature.

[0195] The fourth-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0196] The second fusion processing subunit is used to cascade the fourth local feature and the fourth upsampled feature to obtain the second fused feature.

[0197] The third upsampling subunit is used to input the second fused feature into the third-level module of the upsampling network and output the third upsampling feature.

[0198] The third-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0199] The third fusion processing subunit is used to cascade the third local feature and the third upsampled feature to obtain the third fused feature.

[0200] The fourth upsampling subunit is used to input the third fused feature into the second-level module of the upsampling network and output the second upsampling feature.

[0201] The second-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0202] The fourth fusion processing subunit is used to cascade the second local feature and the second upsampled feature to obtain the fourth fused feature.

[0203] The fifth upsampling subunit is used to input the fourth fused feature into the first-level module of the upsampling network and output the first upsampling feature.

[0204] The first-level module of the upsampling network includes: convolutional layers, activation functions, visual converters, and nearest neighbor interpolation upsampling layers.

[0205] The fifth fusion processing subunit is used to cascade the first upsampled feature and the image to be processed to obtain the fifth fused feature.

[0206] The third input unit is used to input the fifth fusion feature into the image restoration module of the upsampling network to obtain a properly exposed image.

[0207] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 4 As shown, it will not be elaborated further here.

[0208] Optionally, in another embodiment of this application, one implementation of the first fusion processing subunit includes:

[0209] The scaling unit is used to scale the dimensions of the global features using the fully connected layer of the injection module to obtain multiple global sub-features.

[0210] The rearrangement unit is used to rearrange multiple global sub-features to obtain the target global sub-feature.

[0211] Among them, the shape of the target global sub-feature matches the fifth local feature.

[0212] The first calculation unit is used to take the product of the target global sub-feature and the fifth local feature as the preliminary fusion feature of the target global sub-feature for each target global sub-feature.

[0213] The second calculation unit is used to sum the preliminary fusion features of all target global sub-features as the first fusion feature.

[0214] For details on the specific working process of the units disclosed in the above embodiments of this application, please refer to the corresponding method embodiments, such as... Figure 5 As shown, it will not be elaborated further here.

[0215] As can be seen from the above scheme, this application provides an image processing apparatus: after the receiving unit 701 receives the image to be processed; the first feature processing unit 702 performs first feature processing on the image to be processed to obtain local features of the image to be processed; the second feature processing unit 703 performs second feature processing on the image to be processed to obtain global features of the image to be processed; finally, the fusion processing unit 704 performs fusion processing on the local features, global features, and the image to be processed to obtain a normally exposed image. This achieves the goal of effectively processing low-light images to obtain normally exposed images.

[0216] Another embodiment of this application provides an electronic device, such as... Figure 8 As shown, it includes:

[0217] One or more processors 801.

[0218] Storage device 802, on which one or more programs are stored.

[0219] When the one or more programs are executed by the one or more processors 801, the one or more processors 801 implement the image processing method as described in any of the above embodiments.

[0220] Another embodiment of this application provides a computer storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the image processing method as described in any of the above embodiments.

[0221] In the embodiments disclosed in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0222] Furthermore, the functional modules in the various embodiments of this disclosure can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. If the functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a live streaming device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0223] Those skilled in the art will be able to implement or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, characterized in that, include: Receive the image to be processed; The image to be processed is subjected to a first feature processing to obtain the global features of the image to be processed; The image to be processed is subjected to a second feature processing to obtain local features of the image to be processed; the second feature processing includes multiple downsampling network processing, each downsampling network processing outputs a local feature; wherein, the input of the next downsampling processing includes at least the local features output by the previous downsampling network, and the first downsampling network processing takes the image to be processed as input; The local features, the global features, and the image to be processed are fused together to obtain a normally exposed image. The first feature processing of the image to be processed to obtain the global features of the image to be processed includes: The size of the image to be processed is adjusted to a preset size to obtain the target image; The target image is input to a first feature extraction component, and a first feature result is output; wherein, the first feature extraction component includes: a convolutional layer, a visual converter, and an average pooling layer; The first feature result is input to the second feature extraction component, and the global features of the image to be processed are output; wherein, the second feature extraction component includes: a convolutional layer, a visual converter and an average pooling layer.

2. The processing method according to claim 1, characterized in that, The second feature processing of the image to be processed to obtain local features of the image to be processed includes: The image to be processed is input into the first-level module of the downsampling network, and the first local feature is output; wherein, the first-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function; The first local feature is input into the second-level module of the downsampling network, and the second local feature is output; wherein, the second-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function; The second local feature is input into the third-level module of the downsampling network, and the third local feature is output; wherein, the third-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function; The third local feature and the global feature are input into the fourth-level module of the downsampling network, and the fourth local feature is output. The fourth-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function. The fourth local feature and the global feature are input into the fifth-level module of the downsampling network, and the fifth local feature is output. The fifth-level module of the downsampling network includes a max pooling layer, a convolutional layer and an activation function.

3. The processing method according to claim 2, characterized in that, The step size of the maximum pooling layer is 2.

4. The processing method according to claim 2, characterized in that, The process of fusing the local features, the global features, and the image to be processed to obtain a properly exposed image includes: The fifth local feature is input into the fifth-level module of the upsampling network, and the fifth upsampled feature is output; wherein, the fifth-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer; The global feature, the fifth local feature, and the fifth upsampled feature are concatenated to obtain the first fused feature; The first fused feature is input into the fourth-level module of the upsampling network, and the output is the fourth upsampling feature; wherein, the fourth-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer; The fourth local feature and the fourth upsampled feature are concatenated to obtain the second fused feature; The second fused feature is input into the third-level module of the upsampling network, and the third upsampling feature is output; wherein, the third-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer; The third local feature and the third upsampled feature are concatenated to obtain the third fused feature; The third fused feature is input into the second-level module of the upsampling network, and the output is the second upsampling feature; wherein, the second-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer; The second local feature and the second upsampled feature are concatenated to obtain the fourth fused feature; The fourth fused feature is input into the first-level module of the upsampling network, and the first upsampling feature is output; wherein, the first-level module of the upsampling network includes: a convolutional layer, an activation function, a visual converter, and a nearest neighbor interpolation upsampling layer; The first upsampled feature and the image to be processed are concatenated to obtain the fifth fused feature; The fifth fusion feature is input into the image restoration module of the upsampling network to obtain a properly exposed image.

5. The processing method according to claim 4, characterized in that, The step of concatenating the global feature, the fifth local feature, and the fifth upsampled feature to obtain the first fused feature includes: The global features are scaled in dimension using the fully connected layer of the injection module to obtain multiple global sub-features; The multiple global sub-features are rearranged to obtain a target global sub-feature; wherein the shape of the target global sub-feature matches the fifth local feature; For each of the target global sub-features, the product of the target global sub-feature and the fifth local feature is used as the preliminary fusion feature of the target global sub-feature; The sum of the preliminary fusion features of all the target global sub-features is taken as the first fusion feature.

6. An image processing apparatus, characterized in that, include: The receiving unit is used to receive the image to be processed; The first feature processing unit is used to perform first feature processing on the image to be processed to obtain the global features of the image to be processed. The second feature processing unit is used to perform second feature processing on the image to be processed to obtain local features of the image to be processed; the second feature processing includes multiple downsampling network processing, each downsampling network processing outputs a local feature; wherein, the input of the next downsampling processing includes at least the local features output by the previous downsampling network, and the first downsampling network processing takes the image to be processed as input; The fusion processing unit is used to fuse the local features, the global features, and the image to be processed to obtain a normally exposed image. The first feature processing unit includes: A size adjustment unit is used to adjust the size of the image to be processed to a preset size to obtain the target image; The first input unit is used to input the target image into the first feature extraction component and output a first feature result; wherein, the first feature extraction component includes: a convolutional layer, a visual converter, and an average pooling layer; The second input unit is used to input the first feature result into the second feature extraction component and output the global features of the image to be processed; wherein, the second feature extraction component includes: a convolutional layer, a visual converter and an average pooling layer.

7. The processing apparatus according to claim 6, characterized in that, The second feature processing unit includes: The first downsampling subunit is used to input the image to be processed into the first-level module of the downsampling network and output the first local feature; wherein, the first-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function; The second downsampling subunit is used to input the first local features into the second-level module of the downsampling network and output the second local features; wherein, the second-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function; The third downsampling subunit is used to input the second local features into the third-level module of the downsampling network and output the third local features; wherein, the third-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function; The fourth downsampling subunit is used to input the third local feature and the global feature into the fourth-level module of the downsampling network and output the fourth local feature; wherein, the fourth-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function; The fifth downsampling subunit is used to input the fourth local feature and the global feature into the fifth-level module of the downsampling network and output the fifth local feature; wherein, the fifth-level module of the downsampling network includes: a max pooling layer, a convolutional layer and an activation function.

8. An electronic device, characterized in that, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any one of claims 1 to 5.

9. A computer storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the image processing method as described in any one of claims 1 to 5.

Images