Image enhancement method and device, computer equipment and computer readable storage medium

Abstract

The invention relates to an image enhancement method and device, computer equipment and a computer readable storage medium. The method comprises the steps of obtaining an input image; performing feature extraction on the input image to obtain an initial local feature and an initial global feature; performing local information aggregation on the initial local features to obtain optimized local features, and performing convolution processing on the optimized local features to obtain local parameters; performing global information mining on the initial global features to obtain optimized global features, and performing convolution processing on the optimized global features to obtain global parameters; enhancing the pixel value of the input image based on the local parameter to obtain a local enhanced image; and processing the local enhanced image based on the global parameters to obtain an enhanced image. By adopting the method and the device, the image can be enhanced, and the quality of the enhanced image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image enhancement method, apparatus, computer equipment and computer-readable storage medium. Background Art

[0002] With the development of computer technology, people have higher and higher requirements for image processing. For example, in night scenes or dimly lit scenes, high-quality, high-contrast images are also required. However, current image enhancement methods cannot achieve local and content adaptation, which easily leads to hola edge noise, resulting in poor image enhancement effects. Summary of the invention

[0003] Based on this, it is necessary to provide an image enhancement method, apparatus, computer device, computer readable storage medium and computer program product to address the above technical problems, which can enhance the image and improve the quality of the enhanced image.

[0004] In a first aspect, the present application provides an image enhancement method, comprising: obtaining an input image;

[0005] Performing feature extraction on the input image to obtain initial local features and initial global features;

[0006] Performing local information aggregation on the initial local features to obtain optimized local features, and performing convolution processing on the optimized local features to obtain local parameters;

[0007] Performing global information mining on the initial global features to obtain optimized global features, and performing convolution processing on the optimized global features to obtain global parameters;

[0008] Performing enhancement processing on the pixel values ​​of the input image based on the local parameters to obtain a local enhanced image;

[0009] The local enhanced image is processed based on the global parameter to obtain an enhanced image.

[0010] In a second aspect, the present application provides an image enhancement device, comprising:

[0011] An acquisition module is used to acquire an input image; perform feature extraction on the input image to obtain initial local features and initial global features;

[0012] A local parameter acquisition module, used for performing local information aggregation on the initial local features to obtain optimized local features, and obtaining local parameters based on the optimized local features;

[0013] A global parameter acquisition module, used for performing global information mining on the initial global features to obtain optimized global features, and obtaining global parameters based on the optimized global features;

[0014] A local processing module, used for performing enhancement processing on the pixel values ​​of the input image based on the local parameters to obtain a local enhanced image;

[0015] The global processing module is used to process the local enhanced image based on the global parameters to obtain an enhanced image.

[0016] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0018] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps in the above method.

[0019] The above-mentioned image enhancement method, device, computer equipment, computer-readable storage medium and computer program product respectively process the features of the input image to obtain local parameters and global parameters, enhance the pixel values ​​of the input image based on the local parameters to obtain a local enhanced image, and then process the local enhanced image based on the global parameters to obtain the final enhanced image, wherein the local parameters are obtained based on the local information aggregation of the input image, and the global parameters are obtained based on the global information mining of the input image. Therefore, the present application performs local and global dual enhancement on the input image, and first considers the local enhancement, and then globally adjusts the image, which can improve the local and content adaptation and improve the coordination of the final generated image, and reduce edge noise. In addition, processing the input image based on local parameters and global parameters can improve processing efficiency, which can not only improve the quality of the enhanced image but also reduce the complexity of the enhancement algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An application environment diagram of an image enhancement method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application;

[0022] Figure 3A schematic diagram of the structure of the image enhancement model provided in the embodiment of the present application;

[0023] Figure 4 A schematic diagram of a process for obtaining optimized local features provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the process of local information aggregation step provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a process for obtaining a local enhanced image provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a formula for obtaining optimized global features provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of a process for obtaining an enhanced image provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram showing the performance comparison of the models provided in the embodiments of the present application;

[0029] Fig.10 A schematic diagram of the enhancement effect provided in the embodiment of the present application;

[0030] Fig.11 A structural block diagram of an image enhancement device provided in an embodiment of the present application;

[0031] Fig.12 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0032] Fig.13 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0033] Fig.14 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0035] The image enhancement method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The terminal 102 sends the image input by the user to the server 104. After acquiring the input image, the server 104 extracts features from the input image to obtain initial local features and initial global features. In order to obtain local parameters, the server can perform local information aggregation on the initial local features to obtain optimized local features, and perform convolution processing on the optimized local features to obtain local parameters. In order to obtain global parameters, the initial global features are subjected to global information mining to obtain optimized global features, and the optimized global features are subjected to convolution processing to obtain global parameters. Finally, the server can enhance the pixel values ​​of the input image based on the local parameters to obtain a local enhanced image. Then the local enhanced image is processed based on the global parameters to obtain the final enhanced image. The enhanced image is of higher quality than the input image, for example, it can be shown in higher height, higher contrast, higher clarity, higher degree of detail, etc. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 may be, but is not limited to, various personal computers, laptop computers, smart phones, and tablet computers. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0036] like Figure 2 As shown, the embodiment of the present application provides an image enhancement method, which is applied to Figure 1 The terminal 102 or the server 104 in the example is used for explanation. It is understandable that the computer device may include at least one of the terminal and the server. The method comprises the following steps:

[0037] S202: Obtain an input image.

[0038] The input image is the image that needs to be enhanced. Image enhancement refers to improving the visual effect of an image through a series of algorithms and technical means to make it more suitable for specific applications or meet specific visual needs. The goal of image enhancement is to make the image look clearer, with stronger contrast, richer details, or more suitable for the visual perception of human observers. The input image can be a dim image taken in a dim environment.

[0039] After obtaining the input image, the input image can be input into the image enhancement model to obtain an enhanced image using the image enhancement model. The structure of the image enhancement model can be referred to Figure 3The image enhancement model includes two parallel branches. One branch is a local image enhancement branch, and the other branch is a global image enhancement branch. The local image enhancement branch processes the input image to obtain local parameters. The global image enhancement branch processes the input image to obtain global parameters.

[0040] It is understandable that when the initial model is trained to obtain the image enhancement model, the data set used includes the image, the local parameters corresponding to the image, and the global parameters corresponding to the image. In this embodiment, after obtaining the local parameters and the global parameters using the trained image enhancement model, the local parameters, the global parameters, and the image pixel values ​​are fused according to the preset rules to obtain an enhanced image. Therefore, when the initial model is trained, the model parameters used, such as the gradient function and the loss function, are also related to the preset rules.

[0041] In some embodiments, the local image enhancement branch can extract local image information and obtain local parameters based on a convolution-adaptive local full-path enhancement module. The global image enhancement branch can extract global information of the image and obtain global parameters based on an attention branch model using a lightweight Q-former structure.

[0042] S204: Extract features from the input image to obtain initial local features and initial global features.

[0043] In order to enhance the input image, it is necessary to first understand the information of the input image itself. Therefore, the key features of the input image can be extracted by feature extraction, and then the input image can be processed based on these key features to achieve image enhancement. The key features can be color, texture, shape, edge, corner point, etc.

[0044] In this embodiment, feature extraction is first performed on the input image to obtain initial local features and initial global features. The initial local features are used in the local image enhancement branch, and the initial global features are used in the global image enhancement branch. In some embodiments, the initial local features and the initial global features may be the same. In other embodiments, the initial local features and the initial global features may be different. The initial local features are more conducive to subsequent local information aggregation. The initial global features are more conducive to subsequent global information mining.

[0045] In some embodiments, the input image may be processed based on a convolutional layer to obtain corresponding initial local features and initial global features.

[0046] S206, performing local information aggregation on the initial local features to obtain optimized local features, and performing convolution processing on the optimized local features to obtain local parameters.

[0047] Among them, Local Information Aggregation is a commonly used technology in computer vision and image processing, which aims to obtain a richer and more robust representation by merging the information of adjacent pixels or feature points in the image. Local parameters are parameters for local enhancement of the image based on local information.

[0048] This step is applied in the local image enhancement branch. By performing local information aggregation on the initial local features, optimized local features are obtained. The optimized local features thus obtained focus on the information aggregation of each local area in the image. Therefore, local parameters for local adjustment of the input image can be obtained based on the optimized local features.

[0049] In one embodiment, the initial local features may be divided into a plurality of image blocks. Local information aggregation may be reflected in processing information in the image block based on information of the image block itself.

[0050] In order to obtain enhanced information of the image using image information, the image information can be convolved. Through convolution processing, the key information in the image information can be analyzed and identified, and then the image can be enhanced or optimized in a targeted manner based on these features. Therefore, the local features can be optimized by convolution processing to obtain local parameters.

[0051] S208, performing global information mining on the initial global features to obtain optimized global features, and performing convolution processing on the optimized global features to obtain global parameters.

[0052] Among them, global information mining refers to the process of extracting overall patterns, trends and associations from image information. Global parameters are parameters for global enhancement of images based on the overall information of the image rather than the information of image blocks.

[0053] This step is applied in the global image enhancement branch, and the optimized global features are obtained by mining the global information of the initial global features. The optimized global features obtained in this way can take into account the correlation between other pixels in the input image and the current pixel. In turn, the global parameters obtained based on the optimized global features can adjust the input image based on the overall information of the image.

[0054] In one embodiment, the initial global features may be processed based on the self-attention model to obtain optimized global features, and then the optimized global features may be convolved to obtain global parameters that can enhance the image.

[0055] S210 , enhancing pixels of the input image based on local parameters to obtain a local enhanced image.

[0056] Exemplarily, after the local parameters are obtained, the pixels of the input image may be processed based on the local parameters to obtain a local enhanced image.

[0057] In one embodiment, the local parameter may be a two-dimensional feature map, and the local parameter stores the local adjustment value corresponding to each pixel of the input image. The pixel value is processed based on the local adjustment value to obtain the local enhancement value, thereby obtaining the local enhanced image. In one embodiment, the local parameter may also be a three-dimensional feature map. The data stored in each feature point in the three-dimensional feature map is used to process the R, G, and B values ​​of the corresponding pixel of the input image, respectively, thereby obtaining the value of the pixel in the local enhanced image. If the number of local parameters is more than one, the number of corresponding feature maps is also more than one.

[0058] S212: Process the local enhanced image based on the global parameters to obtain an enhanced image.

[0059] Exemplarily, after the input image is enhanced based on the local parameters, a local enhanced image is obtained. Then, the local enhanced image is processed based on the global parameters to obtain an enhanced image. In this way, the input image is first locally adjusted based on the situation of each small area of ​​the input image, and then the locally adjusted image is globally adjusted, so that the overall coordination of the final enhanced image is high and the enhancement effect is good.

[0060] In some embodiments, the global parameter may also be a feature map. The global adjustment value for adjusting the pixels of the local enhanced image is stored in the feature map. The number of global adjustment values ​​may be more than one. In some embodiments, if the global parameter corresponds to multiple global processes, for example, the parameters corresponding to the first global process are first fused with the pixels of the local enhanced image to obtain a global fusion value; and then the global fusion value is gamma-operated based on the parameters corresponding to the second global process to obtain a global enhancement value. In this case, the global parameter corresponds to two global processes. Then, at this time, the global adjustment value stored in the feature map includes both the parameters corresponding to the first global process, i.e., the first global parameters, and the parameters corresponding to the second global process, i.e., the second global parameters.

[0061] It can be seen that in the embodiment of the present application, the features of the input image are processed respectively to obtain local parameters and global parameters, the pixel values ​​of the input image are enhanced based on the local parameters to obtain a local enhanced image, and then the local enhanced image is processed based on the global parameters to obtain the final enhanced image. Since the local parameters are obtained based on the local information aggregation of the input image, and the global parameters are obtained based on the global information mining of the input image, the present application performs local and global dual enhancement on the input image, and first considers the local enhancement, which can be processed for the situation where the features of the input image are uneven. For example, when there is a significant difference in the brightness of each area of ​​the input image, the adjustment parameter of the pixel can be obtained based on the pixel information around the pixel, so that the brightness of the final enhanced image is coordinated, rather than some areas of the image after enhancement have normal brightness and some areas are overexposed. Then the image is adjusted globally, which can improve the coordination of the final generated image as a whole and reduce edge noise. In addition, processing the input image based on local parameters and global parameters can improve processing efficiency, which can not only improve the quality of the enhanced image but also reduce the complexity of the method.

[0062] It can be seen from the above content that the embodiment of the present application first obtains local parameters based on the local image enhancement branch, and then obtains global parameters based on the global image enhancement branch. Then, the input image is processed based on the local parameters and the global parameters to obtain an enhanced image after enhancement processing. The following first describes in detail how to obtain local parameters based on the local image enhancement branch. Based on the local image enhancement branch, the initial local features are first subjected to local information aggregation to obtain optimized local features. Then, the optimized local features are subjected to convolution processing to obtain local parameters.

[0063] In some embodiments, the initial local features may be divided to obtain multiple image blocks. Local information aggregation is then performed based on the image blocks to obtain optimized local features corresponding to the initial local features. Figure 4 As shown, the step of performing local information aggregation on the initial local features to obtain optimized local features includes steps 402 to 406. Among them:

[0064] Step 402: divide the initial local features to obtain multiple image blocks.

[0065] For example, Figure 3 As shown, the initial local features can be represented in the form of a feature map. It is understandable that the features described in this application can be represented in the form of a feature map. Each pixel value of the feature map can represent a feature response detected at a specific location. These responses may include edges, textures, shapes, colors, or other abstract features learned by the neural network. Therefore, dividing the initial local features can be dividing the feature map to obtain multiple image blocks of the same size. .

[0066] In some embodiments, the size of the image block may also be different. The image block may be divided based on the feature distribution of the initial local features to obtain a plurality of image blocks of different sizes. For example, the feature response of each feature point in the divided image block may be consistent.

[0067] Step 404: perform local information aggregation on the image block based on the convolution adaptive model to obtain an information aggregated image block.

[0068] For example, Figure 3 As shown, based on the W-CCA model, local information aggregation is performed on the information in the image block to obtain an information aggregated image block. Among them, the number of W-CCA models can be multiple. The W-CCA (weighted canonical correlation analysis) model is a statistical model for analyzing the correlation between two groups of variables. It is an extension based on CCA (canonical correlation analysis) and improves the model's ability to identify important variables by introducing weights. CCA is a method for finding the maximum correlation between two groups of variables. It aims to find a linear combination of two groups of variables so that the correlation between these combinations is maximized. In this embodiment, the above variables may be image features.

[0069] In one embodiment, for the i-th image block, three convolutional layers are used to extract the one-dimensional features of the three dimensions corresponding to the i-th image block. Then, the three-dimensional features are combined into a three-dimensional weight feature by using tensor combination. After obtaining the three-dimensional weight feature corresponding to the i-th image block, the pixel values ​​of the image block are adaptively aggregated based on the three-dimensional weight feature, thereby realizing the local information aggregation of the image block to obtain the information aggregated image block.

[0070] Step 406: Arrange the multiple information aggregation image blocks to obtain optimized local features.

[0071] Since local information aggregation is performed in the dimension of image blocks, in order to obtain optimized local features, after obtaining the information aggregation image blocks corresponding to the image blocks, it is necessary to arrange the information aggregation image blocks in a predetermined order. After being arranged in a predetermined order, the position of the information aggregation image blocks in the optimized local features can be consistent with the position of the image blocks in the initial local features.

[0072] It can be seen that in this embodiment, by dividing the initial local features, multiple image blocks are obtained, and then local information aggregation is performed on the image blocks based on the convolution adaptive model to obtain information aggregated image blocks. After that, the multiple information aggregated image blocks are arranged to obtain optimized local features. The optimized local features obtained in this way can characterize the information after local information aggregation, so that the local parameters obtained based on the optimized local features can be used to enhance the image locally. And by first dividing the initial local features and then performing local information aggregation based on the divided image blocks, the amount of data corresponding to the local information aggregation can be reduced, thereby reducing the computational complexity and improving the computational efficiency.

[0073] In some embodiments, Figure 5 As shown, for a certain image block, the steps of how to aggregate the information in the image block can refer to steps 502 to 506. Among them:

[0074] Step 502: extract features from the image blocks based on multiple convolutional layers to obtain corresponding multiple one-dimensional features.

[0075] Step 504 , performing tensor combination on multiple one-dimensional features to obtain a weight feature, where the dimension of the weight feature is the same as the dimension of the image block.

[0076] Step 506: Process the image block based on the weight feature to aggregate local information of the image block.

[0077] The dimension of the weight feature is the same as the dimension of the image block, which means that the width, height, and number of channels of the feature map corresponding to the weight feature are the same as the width, height, and number of channels corresponding to the image block. The dimensions of multiple one-dimensional features also correspond to the width, height, and number of channels corresponding to the image block.

[0078] Exemplarily, in order to perform local information aggregation on an image block, the information within the image block can be analyzed to obtain the correlation between other pixels in the image block and the current pixel, and then the current pixel can be processed based on the correlation so that the information corresponding to the pixel after processing can represent deeper information. Therefore, in order to perform local information aggregation on an image block, the information within the image block can be analyzed to obtain the weight feature corresponding to the image block, and then the image block can be processed based on the weight feature. To obtain the weight feature, the image block can be convolved based on three convolutional layers to obtain three one-dimensional features corresponding to the image block, and the three one-dimensional features can be tensor-combined to obtain the weight feature, which is a three-dimensional feature. The three-dimensional image block is then fused with the three-dimensional weight feature to obtain an information-aggregated image block. Optionally, refer to Figure 3, the pixel value of the image block can be multiplied by the feature value of the weight feature at the corresponding position, and the resulting product is determined as the value of the feature point of the optimized local feature at the corresponding position.

[0079] It can be seen that in this embodiment, by extracting features from image blocks based on multiple convolutional layers, multiple corresponding one-dimensional features are obtained, and then multiple one-dimensional features are combined as tensors to obtain weight features, and then the image blocks are processed based on the weight features to aggregate local information of the image blocks. This can improve the model's ability to aggregate local information, thereby making the quality of the obtained optimized local features higher and able to express deeper features.

[0080] After the above steps, the optimized local features have been obtained. After the optimized local features are obtained, feature extraction can be performed on the optimized local features, for example, the optimized local features are input into a convolution layer to obtain local parameters that can enhance the image.

[0081] In some embodiments, the local parameter includes a first local parameter and a second local parameter. This can improve the accuracy of local enhancement, and use one of the parameters to fine-tune the effect of local enhancement. Figure 6 As shown, the step of enhancing the pixel values ​​of the input image based on the local parameters to obtain the locally enhanced image includes steps 602 to 604. Among them:

[0082] Step 602: Process the pixel values ​​of the input image based on the first local parameter and the second local parameter to obtain a local enhancement value corresponding to the input image.

[0083] Step 604: Obtain a local enhanced image based on the local enhancement value.

[0084] For example, the local parameters can be represented in the form of a graph. The information stored in a pixel point in the graph includes a first local parameter and a second local parameter. After obtaining the local parameters, the first local parameter and the second local parameter corresponding to a certain pixel point are extracted from the local parameters, and the pixel value of the corresponding pixel point in the input image is processed based on the first local parameter and the second local parameter, so as to obtain the local enhancement value corresponding to the pixel point. The local enhancement value is the pixel value corresponding to the local enhanced image. Therefore, a local enhanced image can be obtained based on the local enhancement value.

[0085] In some embodiments, each pixel of the input image may be processed based on the first local parameter and the second local parameter to obtain a local enhancement value corresponding to each pixel, and then a local enhanced image may be obtained based on the local enhancement value.

[0086] In some embodiments, without affecting the enhancement effect, some pixels in the input image may be selectively processed to obtain local enhancement values ​​corresponding to some pixels. Then, the pixel values ​​of some pixels are adjusted to the local enhancement values ​​to obtain a local enhanced image. For example, one pixel out of every five pixels may not be locally enhanced, so that the processing efficiency can be improved while ensuring the enhancement effect.

[0087] It can be seen that in this embodiment, by processing the pixel points of the input image based on the first local parameter and the second local parameter, a local enhancement value corresponding to the input image is obtained, and then a local enhanced image is obtained based on the local enhancement value. In this way, the refinement of the local enhancement can be improved. For example, the second local parameter is used to fine-tune the enhancement effect, thereby improving the enhancement quality.

[0088] As mentioned above, the use of two local parameters can refine the local enhancement effect. In some embodiments, the first local parameter can be first fused with the pixel value of the input image to obtain a fused pixel value. The fused pixel value is then compensated based on the second local parameter to obtain a local enhancement value. For example, after obtaining the first local parameter and the second local parameter corresponding to a pixel point of the input image, the first local parameter and the pixel value of the current pixel point can be multiplied to obtain a product. The product is then added or subtracted from the second local parameter to obtain a local enhancement value.

[0089] Through the above steps, a local enhanced image has been obtained. In order to obtain the final enhanced image, the local enhanced image needs to be globally enhanced. It should be noted that the steps of obtaining local parameters and obtaining global parameters are not in a particular order. The following is a detailed description of the global image enhancement branch.

[0090] In the global image enhancement branch, the initial global features are firstly mined for global information to obtain optimized global features. Then the optimized global features are convolved to obtain global parameters. Finally, the local enhanced image is processed based on the global parameters to obtain the final enhanced image.

[0091] In some embodiments, global information mining can be performed on the initial global features based on the self-attention model. Exemplarily, based on the self-attention model, the initial global features are used to obtain corresponding query features, key features, and value features; attention scores are obtained based on the query features and key features; the attention scores are converted into probability distributions to obtain attention weights; and optimized global features are obtained based on the attention weights and value features.

[0092] Among them, based on the self-attention model, the query, key, and value of each element in the initial global feature can be obtained. The key feature is used to store the key value of each element in the initial global feature. The query feature is used to store the query value of each element in the initial global feature. The value feature is used to store the value corresponding to the initial global feature. In the self-attention model, K (Key) represents key information, which is used to compare and measure the importance of each element in the sequence. K is usually a representation of an element in the sequence, such as a vector obtained from the embedding layer. Q (Query): Q represents query information, which is used to inquire and compare the relationship between each element in the sequence and other elements. Q is also a representation of an element in the sequence, similar to K, which can be obtained from the embedding layer. V (Value): V represents value information, which provides the actual value of each element in the sequence. V is also a representation of an element in the sequence obtained from the embedding layer. In the self-attention mechanism, for each element in the sequence, we calculate the similarity (usually dot product) between Q and K to obtain the attention weights. Then, these attention weights are used to weighted sum V to obtain the final representation of each element.

[0093] Therefore, in this embodiment, for the initial global feature, an attention score is obtained based on the query feature corresponding to the initial global feature and the key feature corresponding to the initial global feature. The attention score is converted into a probability distribution to obtain an attention weight. Based on the attention weight and the value feature, a weighted sum is performed to obtain an optimized global feature. Figure 7 As shown in the figure, before obtaining the query features, key features, and value features, the initial global features can be reshaped to process the data more flexibly and facilitate subsequent calculations. Represents the initial global feature, and after the Reshape operation, the reshaped initial global feature is obtained. Then the reshaped initial global feature is processed. Figure 7 In , Represent the first global parameter and the second global parameter respectively. Represents optimization of global features. represents the attention weight.

[0094] It can be seen that in this embodiment, based on the self-attention model, global information mining can be performed on the information in the input image, so that global parameters can be obtained based on the information obtained by the global information mining. In this way, the global parameters can be used to perform global adjustments to the local enhanced image, which can further improve the image enhancement effect and ensure the consistency of the image enhancement.

[0095] In some embodiments, to improve the accuracy of global adjustment, the global parameters may also include a first global parameter and a second global parameter. In this way, the local enhanced image may be processed based on multiple parameters. Figure 8 As shown, the step of processing the local enhanced image based on the global parameters to obtain the enhanced image may include steps 802 to 804. Among them:

[0096] Step 802: Process the pixel values ​​of the local enhanced image based on the first global parameter and the second global parameter to obtain a global enhancement value corresponding to the local enhanced image.

[0097] Step 804: Obtain an enhanced image based on the global enhancement value.

[0098] For example, the global parameters can be represented in the form of a graph. The information stored in a pixel point in the graph may include a first global parameter and a second global parameter. After obtaining the global parameters, the first global parameter and the second global parameter corresponding to a certain pixel point are extracted from the global parameters, and the pixel value of the corresponding pixel point in the input image is processed based on the first global parameter and the second global parameter, so as to obtain the global enhancement value corresponding to the pixel point. The global enhancement value is the pixel value corresponding to the global enhanced image. Therefore, a global enhanced image can be obtained based on the global enhancement value.

[0099] In some embodiments, each pixel of the local enhanced image may be processed based on the first global parameter and the second global parameter to obtain a global enhancement value corresponding to each pixel, and then a global enhanced image may be obtained based on the global enhancement value.

[0100] It can be seen that in this embodiment, the local enhanced image is processed based on the first global parameter and the second global parameter, which can improve the accuracy of global enhancement and improve the quality of the enhanced image.

[0101] In some embodiments, when the pixel value of each pixel is processed based on the first global parameter and the second global parameter to obtain the global enhancement value corresponding to each pixel, the first global parameter may be firstly fused with the pixel of the local enhanced image to obtain the global fusion value, and then the global fusion value may be adjusted based on the second global parameter to obtain the global enhancement value.

[0102] For example, the first global parameter can be a value and the second global parameter can be a multidimensional array. Figure 8 , the second global parameter has 9 values. Figure 3, to obtain the global enhancement value, the pixel value of the local enhanced image is first multiplied by the first global parameter to obtain the product. Then the global fusion value is adjusted based on the second global parameter and the gamma operation to obtain the global enhancement value. Gamma Correction is a nonlinear transformation widely used in image processing and video processing to adjust the contrast of the image. In the gamma operation, the output pixel value is the γ power of the input pixel value. Among them, γ is the gamma value, which determines the relationship between the input pixel value and the output pixel value. When γ>1, the output pixel value will be brighter than the input pixel value; when γ<1, the output pixel value will be darker than the input pixel value. In image enhancement, the gamma operation can be used to adjust the contrast of the image so that the brightness distribution of the image is more in line with the perception characteristics of the human eye. A gamma value of 1 is equivalent to a linear transformation and does not change the contrast of the image. Increasing the gamma value will increase the contrast of the image and make the image brighter, but it may cause the image to be oversaturated or overexposed. Reducing the gamma value will reduce the contrast of the image and make the image darker, but may cause the image to be too dim or underexposed. Among them, the multiple values ​​of the second global parameter can be divided into multiple groups, for example, each group can be 3 values, so that the R, G, and B values ​​of the pixels of the local enhanced image can be processed by using these 3 values. After adjusting the pixels in the local enhanced image using each group of data, the three groups of adjusted values ​​can be processed, for example, averaged. This can improve the reliability of the global enhancement value.

[0103] refer to Fig. 9 and Fig.10 After the above steps, the present application can achieve the best PSNR performance and process 125 frames of image data per second. Compared with the mainstream self-attention models, such as SNR, Restormer, and Retinexformer, it has obvious advantages. Fig.10 From some visual comparisons shown, it can be seen that the method "Ours" of the present application achieves better enhancement effect and better noise removal.

[0104] In an exemplary embodiment, in order to enhance the input image to obtain a clearer, brighter, and more informative enhanced image, the input image may be first acquired. Feature extraction is performed on the input image to obtain initial local features and initial global features. For example, the input image may be input into a convolution layer for convolution processing to obtain initial local features and initial global features.

[0105] In order to obtain local parameters, it is necessary to first process the initial local features to obtain optimized local features. In order to obtain optimized local features, the initial local features are first divided to obtain multiple image blocks. Then, based on multiple convolutional layers, feature extraction is performed on the image blocks respectively to obtain corresponding multiple one-dimensional features. Multiple one-dimensional features are combined as tensors to obtain weighted features. Image blocks are processed based on weighted features to aggregate local information of image blocks to obtain information-aggregated image blocks. Finally, multiple information-aggregated image blocks are arranged to obtain optimized local features. First, the locally extracted features are evenly divided into N image blocks. For the nth image block , three convolutional layers can be used , extract one-dimensional features of three dimensions respectively Then we use the tensor combination calculation method to combine these three one-dimensional features into a three-dimensional weighted feature . Get the three-dimensional adaptive aggregation weight feature After that, we input the image block Adaptive aggregation is performed. Finally, we rearrange the N aggregated image blocks into new image features. The new image features are optimized local features.

[0106] After obtaining the optimized local features, the optimized local features are convolved to obtain local parameters. If the local parameters include a first local parameter and a second local parameter, the first local parameter is fused with the pixel value of the input image to obtain a fused pixel value. The fused pixel value is then compensated based on the second local parameter to obtain a local enhancement value. A local enhanced image is obtained based on the local enhancement value.

[0107] To obtain global parameters, we can use the initial global features based on the self-attention model to obtain the corresponding query features, key features, and value features; obtain the attention score based on the query features and key features; convert the attention score into a probability distribution to obtain the attention weight; and obtain the optimized global features based on the attention weight and value features.

[0108] After obtaining the optimized global features, the optimized global features are convolved to obtain global parameters. If the global parameters include a first global parameter and a second global parameter, the first global parameter is fused with the pixels of the local enhanced image to obtain a global fusion value; the global fusion value is adjusted based on the second global parameter to obtain a global enhancement value. An enhanced image is obtained based on the global enhancement value. Optionally, a gamma operation can be performed on the global fusion value based on the second global parameter to obtain a global enhancement value.

[0109] It can be seen that in this embodiment, through the above steps, the image can be enhanced and a good enhancement effect can be achieved, thereby improving the clarity, contrast, brightness and details of the image. In addition, this embodiment can also reduce the computational complexity and amount, which is beneficial to the efficiency of image enhancement.

[0110] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiment of the present application also provides an image enhancement device. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more image enhancement device embodiments provided below can refer to the limitations on the image enhancement method above, and will not be repeated here.

[0112] like Fig.11 As shown, an embodiment of the present application provides a device 1100, including:

[0113] The acquisition module 1101 is used to acquire an input image and perform feature extraction on the input image to obtain initial local features and initial global features.

[0114] The local parameter acquisition module 1102 is used to aggregate local information of the initial local features to obtain optimized local features, and obtain local parameters based on the optimized local features.

[0115] The global parameter acquisition module 1103 is used to perform global information mining on the initial global features to obtain optimized global features, and obtain global parameters based on the optimized global features.

[0116] The local processing module 1104 is used to enhance the pixel values ​​of the input image based on the local parameters to obtain a local enhanced image.

[0117] The global processing module 1105 is used to process the local enhanced image based on the global parameters to obtain an enhanced image.

[0118] In some embodiments, the local parameters include a first local parameter and a second local parameter, and the local processing module 1104 is used to process the pixels of the input image based on the first local parameter and the second local parameter to obtain a local enhancement value corresponding to the input image; and obtain a local enhanced image based on the local enhancement value.

[0119] In some embodiments, the local processing module 1104 is further configured to fuse the first local parameter with the pixel value of the input image to obtain a fused pixel value; and compensate the fused pixel value based on the second local parameter to obtain a local enhancement value.

[0120] In some embodiments, the global parameters include a first global parameter and a second global parameter. The global processing module 1105 is used to process the pixels of the local enhanced image based on the first global parameter and the second global parameter to obtain a global enhancement value corresponding to the local enhanced image; and obtain an enhanced image based on the global enhancement value.

[0121] In some embodiments, the local parameter acquisition module 1102 is also used to divide the initial local features to obtain multiple image blocks; perform local information aggregation on the image blocks based on the convolution adaptive model to obtain information aggregated image blocks; and arrange multiple information aggregated image blocks to obtain optimized local features.

[0122] In some embodiments, the local parameter acquisition module 1102 is also used to extract features from image blocks based on multiple convolutional layers to obtain corresponding multiple one-dimensional features; perform tensor combination on multiple one-dimensional features to obtain weight features, and the dimension of the weight features is the same as the dimension of the image blocks; and process the image blocks based on the weight features to aggregate local information on the image blocks.

[0123] In some embodiments, the global parameter acquisition module 1103 is also used to obtain corresponding query features, key features, and value features based on the self-attention model using the initial global features; obtain an attention score based on the query features and key features; convert the attention score into a probability distribution to obtain an attention weight; and obtain optimized global features based on the attention weight and value features.

[0124] Each module in the above-mentioned image enhancement device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0125] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Fig.12As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the operating parameters and input images of the image enhancement model. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above-mentioned image enhancement method are implemented.

[0126] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.13 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned image enhancement method are implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0127] Those skilled in the art will understand that Fig.12 or Fig.13The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0128] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0129] In some embodiments, Fig.14 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0130] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0133] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image enhancement method, It is characterized in that include: Get the input image; Performing feature extraction on the input image to obtain initial local features and initial global features; Performing local information aggregation on the initial local features to obtain optimized local features, and performing convolution processing on the optimized local features to obtain local parameters; Performing global information mining on the initial global features to obtain optimized global features, and performing convolution processing on the optimized global features to obtain global parameters; Performing enhancement processing on the pixel values ​​of the input image based on the local parameters to obtain a local enhanced image; The local enhanced image is processed based on the global parameter to obtain an enhanced image.

2. The method according to claim 1, It is characterized in that The local parameters include a first local parameter and a second local parameter, and the pixel values ​​of the input image are enhanced based on the local parameters to obtain a local enhanced image, including: Processing pixel values ​​of the input image based on the first local parameter and the second local parameter to obtain a local enhancement value corresponding to the input image; The local enhanced image is obtained based on the local enhancement value.

3. The method according to claim 2, It is characterized in that The processing of the pixel value of the input image based on the first local parameter and the second local parameter to obtain the local enhancement value corresponding to the input image includes: Fusing the first local parameter with the pixel value of the input image to obtain a fused pixel value; The fused pixel value is compensated based on the second local parameter to obtain the local enhancement value.

4. The method according to claim 1, It is characterized in that The global parameters include a first global parameter and a second global parameter, and the processing of the local enhanced image based on the global parameters to obtain the enhanced image includes: Processing pixel values ​​of the local enhanced image based on the first global parameter and the second global parameter to obtain a global enhancement value corresponding to the local enhanced image; The enhanced image is obtained based on the global enhancement value.

5. The method according to claim 4, It is characterized in that The step of processing the pixel value of each pixel based on the first global parameter and the second global parameter to obtain a global enhancement value corresponding to each pixel includes: Fusing the first global parameter with the pixel value of the local enhanced image to obtain a global fusion value; The global fusion value is adjusted based on the second global parameter to obtain the global enhancement value.

6. The method according to claim 1, It is characterized in that The performing local information aggregation on the initial local features to obtain optimized local features includes: Dividing the initial local features to obtain multiple image blocks; Performing local information aggregation on the image block based on a convolution adaptive model to obtain an information aggregated image block; Arrange a plurality of the information aggregation image blocks to obtain the optimized local features.

7. The method according to claim 6, It is characterized in that The performing local information aggregation on the image block based on the convolution adaptive model includes: Based on multiple convolutional layers, feature extraction is performed on the image blocks respectively to obtain corresponding multiple one-dimensional features; Performing tensor combination on the multiple one-dimensional features to obtain a weight feature, where the dimension of the weight feature is the same as the dimension of the image block; The image block is processed based on the weight feature to aggregate local information of the image block.

8. The method according to claim 1, It is characterized in that The performing global information mining on the initial global features to obtain optimized global features includes: Based on the self-attention model, the initial global features are used to obtain corresponding query features, key features, and value features; an attention score is obtained based on the query features and the key features; the attention score is converted into a probability distribution to obtain an attention weight; and the optimized global features are obtained based on the attention weight and the value features.

9. An image enhancement device, It is characterized in that include: An acquisition module is used to acquire an input image; perform feature extraction on the input image to obtain initial local features and initial global features; A local parameter acquisition module, used for performing local information aggregation on the initial local features to obtain optimized local features, and obtaining local parameters based on the optimized local features; A global parameter acquisition module, used for performing global information mining on the initial global features to obtain optimized global features, and obtaining global parameters based on the optimized global features; A local processing module, used for performing enhancement processing on the pixel values ​​of the input image based on the local parameters to obtain a local enhanced image; The global processing module is used to process the local enhanced image based on the global parameters to obtain an enhanced image.

10. A computer device, comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images