Aesthetically enhanced image correction method and electronic equipment

Abstract

The embodiment of the invention provides an aesthetically enhanced image correction method and electronic equipment. The method comprises the following steps: acquiring an abnormal exposure image and a corresponding initial feature image; performing feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain an initial fusion feature; obtaining first depth feature information, second depth feature information and a first fusion feature according to the initial fusion feature; according to the initial feature image, performing self-attention mechanism operation processing on the first depth feature information and the second depth feature information to obtain first depth associated information and second depth associated information; obtaining a target fusion feature image according to the first fusion feature, the second fusion feature and the initial fusion feature; and coding and decoding the target fusion feature image to obtain a normal exposure image. According to the method, the information of different depths is associated and fused, so that the data of the corrected image is more complete, and the aesthetic quality of the image is enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of computer image data processing, and in particular to an image correction method and electronic device for aesthetic enhancement. Background Art

[0002] In recent years, mobile devices such as cameras and mobile phones have been used to capture photos of various scenes at any time. Due to the different exposure conditions of different scenes, images taken with underexposure or overexposure will result in poor image aesthetic quality. Therefore, image exposure correction, as an important content in the field of aesthetic enhancement, has an increasingly strong demand. Application software or cameras that apply related technologies are likely to be favored by users. How to correct images to obtain high-quality images with aesthetic appeal has important research value.

[0003] The method of using image histogram to adjust pixel intensity values ​​or using tone curve adjustment to correct exposure errors can correct images, but this method focuses on underexposed or overexposed scenes, resulting in poor generalization for other exposures, making it impossible to deploy in practical applications. With the continuous development of artificial intelligence technology, in the existing technology, data under different exposure conditions taken by mobile devices such as cameras and mobile phones are usually used to train the network to improve its multiple exposure correction capabilities.

[0004] However, in the process of image exposure correction, the existing image correction network model has the problem that other image data information is missing when adjusting the exposure image data, resulting in insufficient integrity of the corrected image data. Summary of the invention

[0005] The embodiments of the present application provide an aesthetically enhanced image correction method and electronic device to solve the problem that when the existing image correction network performs exposure correction processing on an image, other image data information is missing when adjusting the exposure image data, resulting in insufficient integrity of the corrected image data.

[0006] In a first aspect, an embodiment of the present application provides an image correction method for aesthetic enhancement, comprising: Acquire an exposure abnormality image and an initial feature image of the exposure abnormality image; Performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain initial fused features; According to the initial fusion feature, first depth feature information, second depth feature information, and a first fusion feature are obtained, the first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information; Performing a self-attention mechanism operation on the first depth feature information and the initial feature image to obtain first depth association information; and performing a self-attention mechanism operation on the second depth feature information and the initial feature image to obtain second depth association information; A target fusion feature image is obtained according to the first fusion feature, the second fusion feature and the initial fusion feature, wherein the second fusion feature is obtained according to the first depth association information and the second depth association information; The target fusion feature image is encoded and decoded to obtain a normal exposure image.

[0007] In a possible implementation, feature fusion processing is performed on the residual features of the initial feature image and the normalized features of the initial feature image to obtain the initial fused features, including: Inputting the initial feature image into the third convolution unit of the normalization module in the target model for third convolution processing to obtain the residual features of the initial feature image; Inputting the initial feature image into the normalization unit of the normalization module in the target model to obtain the normalized features of the initial feature image, wherein the normalized features of the initial feature image are obtained by performing the third convolution process on the initial normalized features, and the initial normalized features are obtained by performing instance normalization process on the initial feature image; The residual features of the initial feature image and the normalized features of the initial feature image are subjected to feature fusion processing to obtain initial fused features.

[0008] In a possible implementation, obtaining the first depth feature information, the second depth feature information, and the first fusion feature according to the initial fusion feature includes: Obtaining first depth feature information and second depth feature information according to the initial fusion feature; Inputting the first depth feature information into a first self-attention mechanism processing unit of a compensation module in a target model to obtain a first K value and a first V value; Inputting the second depth feature information into the second self-attention mechanism processing unit of the compensation module in the target model to obtain a second K value and a second V value; Obtain first depth fusion information according to the first K value and the second K value; Obtain second depth fusion information according to the first V value and the second V value; The first depth fusion information and the first depth fusion information are input into the first fusion unit of the compensation module in the target model for first fusion processing to obtain a first fusion feature.

[0009] In a possible implementation, obtaining the first depth feature information and the second depth feature information according to the initial fusion feature includes: Inputting the initial fusion feature into the first convolution unit of the compensation module in the target model for a first convolution process to obtain first depth feature information; The initial fusion features are input into the second convolution unit of the compensation module in the target model for a second convolution process to obtain second depth feature information.

[0010] In a possible implementation, performing a self-attention mechanism operation on the first depth feature information and the initial feature image to obtain the first depth association information; and performing a self-attention mechanism operation on the second depth feature information and the initial feature image to obtain the second depth association information, including: Inputting the first depth feature information and the initial feature image into a first self-attention mechanism processing unit of a compensation module in a target model to perform a first self-attention mechanism operation process to obtain first depth association information, the first depth association information is determined according to the first depth feature information and a first association weight coefficient, and the first association weight coefficient is determined according to the first depth feature information and the initial feature image; The second depth feature information and the initial feature image are input into the second self-attention mechanism processing unit of the compensation module in the target model for second self-attention mechanism operation processing to obtain second depth association information. The second depth association information is determined according to the second depth feature information and the second association weight coefficient. The second association weight coefficient is determined according to the second depth feature information and the initial feature image.

[0011] In a possible implementation, obtaining a target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature includes: Inputting the first depth correlation information and the second depth correlation information into a second fusion unit of a compensation module in a target model for second fusion processing to obtain a target fusion feature; The first fusion feature, the second fusion feature and the initial fusion feature are input into the third fusion unit of the compensation module in the target model for third fusion processing to obtain a target fusion feature image.

[0012] In a possible implementation, encoding and decoding the target fusion feature image to obtain a normal exposure image includes: The target fusion feature image is input into the feature extraction and encoding module in the target model to perform downsampling processing in the feature space to obtain multi-level image data information; The multi-level image data information is input into the feature fusion decoding module in the target model for upsampling processing in the feature space to obtain a normally exposed image.

[0013] In a possible implementation, the method further includes: Acquire an image to be trained with abnormal exposure and an aligned image with normal exposure corresponding to the image to be trained with abnormal exposure; Input the training image with abnormal exposure into the training model to obtain the target correction image; The model to be trained is trained according to the target correction image and the exposure normal alignment image to obtain a target model, wherein the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

[0014] In a possible implementation, training a model to be trained according to a target correction image and an exposure-normal alignment image to obtain a target model includes: Determine whether there is a next convolutional neural network connected to the target convolutional neural network, where the target convolutional neural network is a convolutional neural network that obtains a target correction image in the model to be trained; If the target convolutional neural network has a next convolutional neural network connected to it, the target correction image is used as the exposure abnormality training image to be trained and input into the next convolutional neural network, and the step of inputting the exposure abnormality training image to the model to be trained is performed again to obtain the target correction image, until the target convolutional neural network does not have a next convolutional neural network connected to it, and the exposure correction image is obtained; Determine the loss function of the model to be trained according to the exposure-corrected image and the exposure-normally aligned image. The loss function of the model to be trained includes a Manhattan distance function, a cosine distance function, a structural similarity distance function, an aesthetic feature distance function, and an adversarial training loss function. According to the loss function of the model to be trained, the model to be trained is trained to obtain the target model.

[0015] In a second aspect, an embodiment of the present application provides an image correction device for aesthetic enhancement, comprising: An acquisition module, used for acquiring an exposure abnormality image and an initial feature image of the exposure abnormality image; A normalization module, used for performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain an initial fused feature; A first compensation module is used to obtain first depth feature information, second depth feature information, and a first fusion feature according to the initial fusion feature, the first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information; A second compensation module is used to perform a self-attention mechanism operation on the first depth feature information and the initial feature image to obtain first depth association information; and to perform a self-attention mechanism operation on the second depth feature information and the initial feature image to obtain second depth association information; a third compensation module, configured to obtain a target fusion feature image according to the first fusion feature, the second fusion feature and the initial fusion feature, wherein the second fusion feature is obtained according to the first depth association information and the second depth association information; The feature sampling module is used to encode and decode the target fusion feature image to obtain a normal exposure image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor; Memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.

[0019] The embodiment of the present application provides an aesthetically enhanced image correction method and electronic device, which obtains an abnormal exposure image and an initial feature image of the abnormal exposure image; performs feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain an initial fusion feature; obtains first depth feature information, second depth feature information, and a first fusion feature based on the initial fusion feature; uses the initial feature image to perform self-attention mechanism operation processing on the first depth feature information and the second depth feature information respectively to obtain first depth correlation information and second depth correlation information; obtains a target fusion feature image based on the first fusion feature, the second fusion feature and the initial fusion feature; performs encoding and decoding processing on the target fusion feature image to obtain a means for obtaining a normal exposure image, The initial feature image of exposure abnormality is mapped to the exposure invariant feature space. While using the normalized features to reduce the impact of exposure on the image, the residual features and the normalized features are fused to make the image information more complete and enhance the image's expressiveness. Then, the target fused features are obtained through deep feature extraction of convolution kernels of different sizes and self-attention mechanism operations. Among them, the features of convolution kernels of different sizes can more comprehensively capture the detailed data and global information of the image, while the self-attention mechanism enhances the model's ability to focus on key information data in the image and enhances the aesthetic quality of the image. Finally, through the encoding and decoding network structure, the fused features are effectively reconstructed to obtain the exposure-corrected image, which effectively corrects the exposure-abnormal image while retaining the important data information of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0021] Figure 1 A schematic diagram of a flow chart of an image correction method for aesthetic enhancement provided in the present application; Figure 2 A schematic flow chart of another method for image correction of aesthetic enhancement provided in the present application; Figure 3 A flowchart of an image correction model training method provided in this application; Figure 4 A schematic diagram of the structure of an exposure normalization module and a multi-scale compensation module in an image correction model provided in this application; Figure 5 A schematic diagram of the structure of a feature extraction encoding module and a feature fusion decoding module in an image correction model provided in this application; Figure 6 A schematic diagram of the structure of an aesthetic discriminator in an image correction model provided in this application; Figure 7 A schematic diagram of the structure of an image correction device for aesthetic enhancement provided by the present application; Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0022] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0024] In the prior art, data under different exposure conditions are usually used to train the network to improve its multiple exposure correction capability. For example, in the traditional low-light imaging algorithm, the exposure accuracy of the image is improved by performing Gamma correction on the preprocessed image, wherein the principle of Gamma correction is to draw a Gamma curve that meets the requirements based on the preset Gamma value, substitute the normalized pixel value in the image into the curve, obtain the corresponding output value, and then denormalize it to the integer value corresponding to the image. Although the method has simple steps and can directly process the data of the original sensor, the brightness of the low-light image is enhanced through the deep learning model. However, for multiple exposure images, the correction process of adjusting the underexposed image and the overexposed image to the image with normal exposure is the opposite process. In the image correction process, the network model cannot take into account the reverse correction of the underexposed and overexposed image data, resulting in the loss of some detail data of the image during the correction process, and the correction effect of the corrected image is not good. At the same time, this makes the performance of the network model poor and cannot be well used in multiple exposure scenes.

[0025] In order to solve the above problems, the embodiments of the present application provide an aesthetically enhanced image correction method and electronic device, which can map the initial feature image of exposure abnormality to the exposure invariant feature space, and fuse the residual features and the normalized features while using the normalized features to reduce the impact of exposure on the image, thereby enhancing the expression ability of the image, and then obtain the target fusion features through deep feature extraction of convolution kernels of different sizes and self-attention mechanism operation. Among them, combining the features of convolution kernels of different sizes can more comprehensively capture the detailed data and global information of the image, and the self-attention mechanism enhances the model's ability to focus on key information in the image, thereby enhancing the aesthetic quality of the image. Finally, through the encoding and decoding network structure, the fusion features are effectively reconstructed to obtain the exposure-corrected image, thereby achieving effective correction of the exposure-abnormal image and retaining the important data information of the image.

[0026] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0027] The embodiment of the present application provides an image correction method for aesthetic enhancement, and the execution subject of the method may be a server, wherein the server may be a camera, a mobile phone, a computer, a tablet, and other devices. The embodiments of the present application do not impose any particular restrictions on the implementation method of the execution subject, as long as the execution subject can obtain the abnormal exposure image and the initial feature image of the abnormal exposure image; perform feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain the initial fusion feature; obtain the first depth feature information, the second depth feature information, and the first fusion feature according to the initial fusion feature, the first depth feature information is obtained by performing a first convolution processing on the initial fusion feature, the second depth feature information is obtained by performing a second convolution processing on the initial fusion feature, the convolution kernel sizes in the first convolution processing and the second convolution processing are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information; perform a self-attention mechanism operation processing on the first depth feature information and the initial feature image to obtain the first depth association information; and perform a self-attention mechanism operation processing on the second depth feature information and the initial feature image to obtain the second depth association information; obtain the target fusion feature image according to the first fusion feature, the second fusion feature and the initial fusion feature, and the second fusion feature is obtained according to the first depth association information and the second depth association information; perform encoding and decoding processing on the target fusion feature image to obtain a normal exposure image.

[0028] Among them, the self-attention mechanism can be a deep learning technology that calculates the correlation between the first data information (corresponding to the query Q-Query) and the second data information (corresponding to the key K-Key and the value V-Value) in the model based on the attention score and the Softmax normalization function.

[0029] Figure 1 The flowchart of an aesthetically enhanced image correction method provided by the present application is shown in FIG. The execution subject of the method may be a server or other server storing the aesthetically enhanced image correction method, and this embodiment is not particularly limited here. Figure 1 As shown, the method may include: S101, acquiring an exposure abnormality image and an initial feature image of the exposure abnormality image.

[0030] Among them, abnormal exposure images can refer to underexposed images, overexposed images, and abnormal exposure images in high dynamic range scenes. It can also refer to abnormal exposure of certain specific areas in the image, such as abnormal images caused by different abilities of certain objects or areas in the scene to reflect light, or the camera's exposure control failing to properly adapt to changes in the scene. It can also refer to images in environments with improper flash use or long exposure. Abnormal exposure images can be obtained when images are captured by electronic devices such as cameras and mobile phones.

[0031] The initial feature image can be obtained by performing calculations on the exposure abnormality image according to the convolution and ReLU activation functions preset in the target model, thereby mapping the image to the encoding space and obtaining a feature image of the same size as the exposure abnormality image.

[0032] S102, performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain initial fused features.

[0033] Among them, the residual feature can refer to the feature after the initial feature image is convolved. The residual feature can be in the residual branch, which is a part of the network model, allowing the original input to bypass some layers and be directly added to the output. Through the residual feature, the network can more easily learn effective feature representation.

[0034] Normalized features can refer to features after normalizing the initial feature image. For example, instance normalization is used to calculate the mean and standard deviation of the features on each channel dimension of the initial feature image, and then the learning parameters of the model and these metric data are used to perform style normalization on the features in the feature space. Different degrees of exposure are regarded as a style, so instance normalization can align different exposures, thereby reducing their representation differences.

[0035] Furthermore, the normalized features can satisfy: ; in, is the normalized feature; is the initial feature image; Normalize the instance; is the average value of the feature; is the standard deviation of the feature; , are the learning parameters of the model.

[0036] The feature fusion processing method can be to splice and fuse the features of each channel dimension to reduce the number of channels and further integrate data information.

[0037] In the embodiment of the present application, the method of performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain the initial fused features may include: Inputting the initial feature image into the third convolution unit of the normalization module in the target model for third convolution processing to obtain the residual features of the initial feature image; Inputting the initial feature image into the normalization unit of the normalization module in the target model to obtain the normalized features of the initial feature image, wherein the normalized features of the initial feature image are obtained by performing the third convolution process on the initial normalized features, and the initial normalized features are obtained by performing instance normalization process on the initial feature image; The residual features of the initial feature image and the normalized features of the initial feature image are subjected to feature fusion processing to obtain initial fused features.

[0038] In some embodiments, the target model may refer to a network model that has been trained and is suitable for exposure image correction. The third convolution process may be performed using a 1×1 convolution.

[0039] S103. According to the initial fusion feature, first depth feature information, second depth feature information, and a first fusion feature are obtained, wherein the first depth feature information is obtained by performing a first convolution process on the initial fusion feature, and the second depth feature information is obtained by performing a second convolution process on the initial fusion feature. The convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined based on the first depth feature information and the second depth feature information.

[0040] In some embodiments, the first convolution process may be processed by 3×3 convolution (Conv), and the second convolution process may be processed by 5×5 convolution (Conv). For example: the first depth feature information satisfies , the second depth feature information satisfies ,in, is the first depth feature information; is the second depth feature information; is the initial fusion feature.

[0041] Since convolution kernels of different sizes capture different image information of the initial fusion features during convolution processing, smaller convolution kernels (such as 3×3 convolution) capture local details of the image, such as edges, textures, etc., while larger convolution kernels capture global features of the image, such as the overall shape of the object and the layout of the scene. Therefore, using multi-scale convolution to process features can enhance the local information of the image.

[0042] In the embodiment of the present application, the method of obtaining the first depth feature information, the second depth feature information, and the first fusion feature according to the initial fusion feature may include: Obtaining first depth feature information and second depth feature information according to the initial fusion feature; Inputting the first depth feature information into a first self-attention mechanism processing unit of a compensation module in a target model to obtain a first K value and a first V value; Inputting the second depth feature information into the second self-attention mechanism processing unit of the compensation module in the target model to obtain a second K value and a second V value; Obtain first depth fusion information according to the first K value and the second K value; Obtain second depth fusion information according to the first V value and the second V value; The first depth fusion information and the first depth fusion information are input into the first fusion unit of the compensation module in the target model for first fusion processing to obtain a first fusion feature.

[0043] Among them, in the self-attention mechanism processing unit, the model will calculate the query (Q), key (K) and value (V) corresponding to the input element respectively. These calculations are usually obtained through different linear transformations.

[0044] In the embodiment of the present application, the initial feature image is regarded as a query vector Q, and the depth feature information is regarded as a key-value pair vector KV. Since the convolution kernel sizes of the first depth feature information and the second depth feature information are different, when performing the first fusion process, in order to subsequently fuse with the initial fusion feature, the information is convolved during the fusion process so that the convolution scale of the first fusion feature is the same as the convolution scale of the initial fusion feature. For example: , ,in, is the first K value; is the first V value; is the second K value; is the second V value. Then, the first depth fusion information can satisfy ; The second deep fusion information can satisfy . The first fusion feature obtained can satisfy .

[0045] In the embodiment of the present application, the method of obtaining the first depth feature information and the second depth feature information according to the initial fusion feature may include: Inputting the initial fusion feature into the first convolution unit of the compensation module in the target model for a first convolution process to obtain first depth feature information; The initial fusion features are input into the second convolution unit of the compensation module in the target model for a second convolution process to obtain second depth feature information.

[0046] Among them, the first convolution processing and the second convolution processing here are only for distinguishing the description of convolutions of different sizes, and are not limited to convolutions of only two sizes. In some embodiments, the convolution unit in the compensation module and the corresponding self-attention mechanism processing unit can be added to improve the integrity of local information and image discrimination features, thereby avoiding the problem of data loss in the image during exposure correction.

[0047] S104, performing a self-attention mechanism operation on the first depth feature information and the initial feature image to obtain first depth association information; and performing a self-attention mechanism operation on the second depth feature information and the initial feature image to obtain second depth association information.

[0048] In the embodiment of the present application, the initial feature image is regarded as the query vector Q. Since the convolution kernel sizes of the first depth feature information and the second depth feature information are different, the initial feature image needs to be convolved with the same size before the self-attention mechanism is operated. For example, if the first convolution unit is a 3×3 convolution, the corresponding first Q value satisfies ; The second convolution unit is a 5×5 convolution, then the corresponding second Q value satisfies .

[0049] The self-attention mechanism operation processing can be to perform dot product processing on the Q value and K value in the self-attention mechanism processing unit to obtain the attention score, and then determine the attention weight through the Softmax function, and perform dot product processing on the attention weight and V value to obtain the deep correlation information. By associating and fusing the multi-scale feature information, the image integrity in the exposure correction process can be guaranteed, thereby ensuring that the image is aesthetically attractive both globally and locally.

[0050] For example, deep association information can satisfy: ; in, For deep correlation information; is a scaling factor to prevent the parameters from being unable to be updated due to the gradient being too small during back propagation; is the dimension of the K vector.

[0051] Among them, in the embodiment of the present application, the method of performing a self-attention mechanism operation processing on the first depth feature information and the initial feature image to obtain the first depth association information; and performing a self-attention mechanism operation processing on the second depth feature information and the initial feature image to obtain the second depth association information may include: Inputting the first depth feature information and the initial feature image into a first self-attention mechanism processing unit of a compensation module in a target model to perform a first self-attention mechanism operation process to obtain first depth association information, the first depth association information is determined according to the first depth feature information and a first association weight coefficient, and the first association weight coefficient is determined according to the first depth feature information and the initial feature image; The second depth feature information and the initial feature image are input into the second self-attention mechanism processing unit of the compensation module in the target model for second self-attention mechanism operation processing to obtain second depth association information. The second depth association information is determined according to the second depth feature information and the second association weight coefficient. The second association weight coefficient is determined according to the second depth feature information and the initial feature image.

[0052] In some embodiments, the depth feature information can be dot-producted with the initial feature image to obtain an attention score, and then the attention score can be processed by the Softmax function according to the vector dimension in the depth feature information to obtain an associated weight coefficient. The associated weight coefficient can characterize the data information in the initial feature image that is most relevant to the features in the current depth feature information. Therefore, dot-producting the associated weight coefficient with the depth feature information can make the pixel value corresponding to the important data in the depth associated information as close to 1 as possible, and omit irrelevant pixels (feature information whose associated weight coefficient is infinitely close to 0).

[0053] Among them, the first self-attention mechanism processing unit and the second self-attention mechanism processing unit here are designed to correspond to convolution units of different sizes so that they can perform calculations and processing respectively, and are not limited to dual-channel self-attention mechanism processing units. In some embodiments, convolution units and corresponding self-attention mechanism processing units in the compensation module can be added to improve the integrity of local information and image discrimination features, thereby avoiding the problem of data loss in the image during exposure correction.

[0054] S105. Obtain a target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature, wherein the second fusion feature is obtained according to the first depth association information and the second depth association information.

[0055] In some embodiments, the target fusion feature image may satisfy: ; in, Fusion feature image for the target.

[0056] Furthermore, in order to facilitate sampling processing of the feature extraction encoding module and the feature fusion decoding module in the model, the target fusion feature image can be first subjected to a fourth convolution process, wherein the convolution kernel size of the fourth convolution process is the same as the convolution kernel size in the feature extraction encoding module and the feature fusion decoding module. For example, if the feature extraction encoding module is composed of three convolution layers and the feature fusion decoding module is composed of three deconvolution layers, the target fusion feature image can be subjected to a 3×3 convolution process before executing subsequent steps.

[0057] Among them, in the embodiment of the present application, the method of obtaining the target fusion feature image according to the first fusion feature, the second fusion feature and the initial fusion feature may include: Inputting the first depth correlation information and the second depth correlation information into a second fusion unit of a compensation module in a target model for second fusion processing to obtain a target fusion feature; The first fusion feature, the second fusion feature and the initial fusion feature are input into the third fusion unit of the compensation module in the target model for third fusion processing to obtain a target fusion feature image.

[0058] In some embodiments, the second fusion feature may satisfy: ; in, It is the second fusion feature.

[0059] S106: Encode and decode the target fusion feature image to obtain a normally exposed image.

[0060] The encoding and decoding processing of the target fusion feature image may include encoding the target fusion feature image, using downsampling to extract important data information in the image layer by layer, such as texture, edge, shape, etc., and then decoding the important data information, using upsampling to fuse the information layer by layer to reconstruct the corrected image.

[0061] Among them, in the embodiment of the present application, the method of encoding and decoding the target fusion feature image to obtain the normally exposed image may include: The target fusion feature image is input into the feature extraction and encoding module in the target model to perform downsampling processing in the feature space to obtain multi-level image data information; The multi-level image data information is input into the feature fusion decoding module in the target model for upsampling processing in the feature space to obtain a normally exposed image.

[0062] In some embodiments, the feature extraction encoding module is composed of three convolutional layers, and the spatial resolution of the feature is downsampled; the feature fusion decoding module is composed of three deconvolutional layers, and the obtained image feature data information is upsampled. There are jump connections between the convolutional layers of the feature extraction encoding module and the feature fusion decoding module, and each jump connection is at scales S1=1 / 4, S2=1 / 2, and S3=1 and generates residual features, connecting features with the same spatial resolution from the shallow layer to the deep layer, so that the local information contained in the features generated by the shallow layer reaches the output.

[0063] In this embodiment of the present application, the method may further include: Acquire an image to be trained with abnormal exposure and an aligned image with normal exposure corresponding to the image to be trained with abnormal exposure; Input the training image with abnormal exposure into the training model to obtain the target correction image; The model to be trained is trained according to the target correction image and the exposure normal alignment image to obtain a target model, wherein the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

[0064] In some embodiments, the method of obtaining the exposure abnormality training image and the normal exposure alignment image corresponding to the exposure abnormality training image may be obtained through various data sets or specific photography techniques.

[0065] For example, a dataset can be created by taking photos of the same scene at different exposure settings, such as taking multiple photos of the same scene using bracketing techniques, and then selecting underexposed, normal, and overexposed images as the dataset. There are some public image datasets that contain images under different exposure conditions, such as some datasets specifically for image enhancement and correction tasks, which may contain the required pairs of images. Manually adjust the exposure of a set of images using image editing software (such as Adobe Photoshop) to create pairs of underexposed and normally exposed images. Use computer vision libraries (such as OpenCV) to write scripts to automatically adjust the exposure of a set of images to generate the required pairs of images.

[0066] Convolutional neural network can refer to the neural network in the U-Net structural block. The U-Net structural design includes a contraction (encoding) path and a symmetrical expansion (decoding) path. The overall shape is similar to the letter "U", hence the name. It can be trained end-to-end directly from the original image to the segmentation map without additional feature engineering, which speeds up the training.

[0067] In some embodiments, a target number of iterations may be set during training, and the current number of iterations may be recorded after each training to obtain a target model. If the current number of iterations meets the target number of iterations, the training is stopped to obtain a trained target model.

[0068] In the embodiment of the present application, the model to be trained is trained according to the target correction image and the exposure normal alignment image to obtain the target model, including: Determine whether there is a next convolutional neural network connected to the target convolutional neural network, where the target convolutional neural network is a convolutional neural network that obtains a target correction image in the model to be trained; If the target convolutional neural network has a next convolutional neural network connected to it, the target correction image is used as the exposure abnormality training image to be trained and input into the next convolutional neural network, and the step of inputting the exposure abnormality training image to the model to be trained is performed again to obtain the target correction image, until the target convolutional neural network does not have a next convolutional neural network connected to it, and the exposure correction image is obtained; Determine the loss function of the model to be trained according to the exposure-corrected image and the exposure-normally aligned image. The loss function of the model to be trained includes a Manhattan distance function, a cosine distance function, a structural similarity distance function, an aesthetic feature distance function, and an adversarial training loss function. According to the loss function of the model to be trained, the model to be trained is trained to obtain the target model.

[0069] Among them, in the embodiment of the present application, after determining whether the target convolutional neural network has a next convolutional neural network connected, the method further includes: If the target convolutional neural network does not have a next convolutional neural network connected, the target correction image is used as the exposure correction image; Determine the loss function of the model to be trained according to the exposure-corrected image and the exposure-normally aligned image; According to the loss function of the model to be trained, the model to be trained is trained to obtain the target model.

[0070] In some embodiments, the Manhattan distance function may refer to L1 distance (Manhattan Distance), which is the sum of the absolute values ​​of the differences in pixel values ​​corresponding to two images. The Manhattan distance function may satisfy: ; in, is the Manhattan distance function; Correct the image for exposure; Align the image for exposure normalization.

[0071] The cosine distance function can measure the similarity of two vectors in direction rather than their size. In image processing, the image can be converted into a vector and then the cosine distance can be calculated. The cosine distance function can satisfy: ; in, is the cosine distance function.

[0072] The structural similarity distance function can refer to the use of the structural similarity index (SSIM) to measure the structural similarity of two images, specifically considering brightness, contrast and structural information. The value of SSIM is between -1 and 1. The closer the value is to 1, the more similar the images are, indicating that the enhanced image effect is closer to the real image. The structural similarity distance function can satisfy: ; in, is the structural similarity distance function; is the average value of the exposure-corrected image; is the average value of the normally exposed aligned images; is the standard deviation of the exposure-corrected image; is the standard deviation of the exposure-normal aligned images; is the covariance of the exposure-corrected image and the exposure-normally aligned image; , is a constant to prevent the numerator and denominator from being too small.

[0073] The aesthetic feature distance function can refer to measuring the difference between images based on the aesthetic quality of the images, which can involve aesthetic-related features such as image tone, composition, texture, contrast, etc. The aesthetic feature distance function can satisfy: ; in, is the aesthetic feature distance function; is the aesthetic discriminator; Output the mean of each channel of the aesthetic features for the aesthetic discriminator; The variance of each channel of the aesthetic feature output by the aesthetic discriminator.

[0074] Furthermore, the aesthetic discriminator can be used to provide feedback to help the generative model optimize the generation process by comparing the aesthetic quality of generated images with real images.

[0075] The adversarial training loss function can refer to the introduction of an aesthetic discriminator component in the model, so that the model image generator and the aesthetic discriminator form an adversarial network. Using the adversarial training strategy, there is a minimum-maximum game between the generator and the discriminator. The generator tries to generate images that are as realistic as possible, while the discriminator tries to distinguish between real images and generated images, so that the training model generates more visually harmonious and realistic images. The adversarial training loss function can satisfy: ; in, is the adversarial training loss function.

[0076] The image correction method provided in the embodiment of the present application maps the initial feature image of exposure abnormality to the exposure invariant feature space, and fuses the residual features and the normalized features while reducing the influence of exposure on the image by using the normalized features to enhance the expression ability of the image. Then, through the deep feature extraction and self-attention mechanism operation of convolution kernels of different sizes, combined with the features of different convolution levels, the details and global information of the image can be captured more comprehensively, and the self-attention mechanism enhances the model's ability to focus on key information in the image and enhances the aesthetic quality of the image. Finally, through the encoding and decoding network structure, the fused features are effectively reconstructed to obtain the exposure-corrected image, and the exposure abnormality image is effectively corrected while retaining the important data information of the image. The method utilizes the powerful feature extraction and learning capabilities of the convolutional neural network in the deep information to improve the rationality and accuracy of the image data processing during the correction process, so that the corrected image effect is closer to the real image, and the correction accuracy is improved. Since the diversity of the exposure abnormality images to be trained is taken into account during model training, the model can be applied to different types of exposure abnormalities and has good generalization ability.

[0077] Figure 2 A schematic diagram of another method for image correction of aesthetic enhancement provided in an embodiment of the present application. Figure 2 As shown, the method may include: S201, obtaining overexposed or underexposed images of different degrees to be corrected.

[0078] S202: Input the obtained image with abnormal exposure into a trained multi-exposure correction model to obtain an image with good exposure.

[0079] Among them, the trained multi-exposure correction model can be a basic framework model composed of multiple continuous U-Net structure blocks. The U-Net structure specifically includes an exposure normalization and multi-scale compensation module, a feature extraction encoding module, and a feature fusion decoding module.

[0080] The image correction method provided in the embodiment of the present application performs image correction through a multi-exposure correction model, so that the corrected normal exposure image retains the data integrity of the image to be corrected, so as to achieve not only a normal exposure enhancement result, but also good reconstruction details and visually satisfactory contrast and color distribution.

[0081] Figure 3 A flowchart of an image correction model training method provided in an embodiment of the present application. Figure 3 As shown, the method may include: S301, based on each convolutional neural network structural block, input the poorly exposed image to be corrected into the exposure normalization module, and map the various exposure features to the exposure-independent feature space through the preset first convolution operation and normalization operation to obtain a first feature image with unchanged exposure.

[0082] Among them, before training, the model to be trained needs to be initialized, including the initialized feature extraction encoding module, exposure normalization and multi-scale compensation module, and feature fusion decoding module.

[0083] The poorly exposed images to be corrected may be images obtained by acquiring a pair of poorly exposed images in natural scenes and aligned properly exposed images as training samples.

[0084] Furthermore, the exposure normalization module consists of an instance normalization layer and multiple convolutional layers. Each time the image passes through a layer, a feature image of a specific size is obtained. The width and height dimensions are consistent with the input, but the channels are expanded, which contains the high-level features of the original image.

[0085] After the normalization operation is performed in the exposure normalization module to obtain the normalized features, the normalized features can also be subjected to normalized distillation loss, which is a technology used for knowledge distillation in machine learning. Knowledge distillation allows a lightweight model (student model) to learn and imitate the behavior of a more complex and better performing model (teacher model). Normalized distillation loss is one of the loss functions used in the distillation process, which aims to improve the performance of the student model.

[0086] S302: Input the first feature image into a multi-scale compensation module, and integrate and compensate to obtain a second feature image through a second preset convolution operation and a multi-branch fusion operation.

[0087] The multi-scale compensation module consists of two parallel convolutional layers of different sizes and a self-attention mechanism. Each time an image passes through a convolutional layer of a certain size, a feature image of the same size is obtained. The corresponding channel self-attention mechanism is performed on the feature images of each size to obtain a feature image enhanced by the self-attention mechanism. The output feature images of different sizes are kept consistent in size, and then the multiple output feature images are fused and spliced ​​by channel, and then the restored second feature image is output through a convolutional layer.

[0088] Furthermore, the parallel convolutional layers of the multi-scale compensation module mainly learn and restore the multi-scale texture information of the normalized features. Different scale branches correspond to convolution kernel sizes of different scales, thereby obtaining receptive fields of different sizes and restoring texture information of different scales.

[0089] The channel self-attention mechanism mainly learns the correlation weight relationship between the features of any two channels, treating the features of each channel as an element in the input sequence, and the number of channels is equal to the length of the input sequence. Through this self-attention mechanism associated with its own channels, the representation ability of the model method is enhanced.

[0090] Figure 4 A schematic diagram of the structure of an exposure normalization module and a multi-scale compensation module in an image correction model provided in an embodiment of the present application is shown in FIG. Figure 4 As shown: Input initial feature image into exposure normalization module , and then enter the double branches respectively, where the first branch first Perform instance normalization operation, and then obtain normalized features through 1x1 convolution operation , then the second branch is Perform convolution operation and use it as residual branch and normalized feature Fusion gets fusion features , and input to the multi-scale compensation module. Since the normalization operation will inevitably lead to the loss of discriminative features, resulting in insufficient information for image reconstruction, in order to solve this problem, we introduce a multi-scale compensation module to integrate the exposure invariant features and the unprocessed features of exposure normalization in the spatial and channel dimensions to ensure the integrity of the information, thereby obtaining satisfactory contrast and details.

[0091] The multi-scale compensation module first introduces multi-scale convolution to enhance local information, and also introduces dual-channel self-attention to focus on the initial input features. and fusion features Specifically, the fusion feature It is input into two parallel branches, and 3x3 and 5x5 deep convolutions are used to enhance the ability to extract multi-scale local information. After obtaining the features of two different receptive fields, they are cross-fused to obtain multi-scale feature information. Then, the dual-channel self-attention mechanism is used to convert the initial input features into Considered as query vector Query, fusion features It is regarded as a key-value pair vector Key-Value, and then the self-attention mechanism is operated, and feature fusion is performed to obtain the target fused feature image , and input into the feature extraction encoding module.

[0092] S303: input the second feature image into a feature extraction and encoding module, and perform a third preset multi-layer convolution operation to obtain multi-level image data information.

[0093] Among them, the feature extraction and encoding module consists of three convolutional layers. Each time the image passes through a convolutional layer, it will be downsampled to obtain a feature image of a specific scale, and finally a feature image containing high-level features of the original image is obtained, retaining the high-level image features of each layer in the above feature extraction and encoding module.

[0094] S304: Input the multi-level image data information into the feature fusion module, and process the multi-level image data information layer by layer through a preset multi-layer deconvolution operation to obtain an output image.

[0095] Among them, the feature fusion decoding module consists of three deconvolution layers. The high-level features of the image in the decoding module are first fused with the corresponding encoder output features of the same size. Each deconvolution layer will be upsampled to obtain a feature image of a specific size, and finally a U-Net structure block output result is obtained.

[0096] Figure 5 A structural diagram of a feature extraction encoding module and a feature fusion decoding module in an image correction model provided in an embodiment of the present application, such as Figure 5 As shown in the figure, the feature fusion decoding module consists of three convolutional layers and three deconvolutional layers. The spatial resolution of the features is first downsampled and then upsampled by deconvolution. There is a long jump connection between each convolutional layer. Each long jump connection generates residual features at scales S1=1 / 4, S2=1 / 2, and S3=1, connecting features with the same spatial resolution from the shallow layer to the deep layer, which helps the local information contained in the features generated by the shallow layer reach the output.

[0097] S305. Input the output image as the poorly exposed image to be corrected into the next convolutional neural network structure block, and repeat steps S301 to S304 until all the convolutional neural network structure block steps in the multi-exposure correction model to be trained are completed to obtain the target output image.

[0098] The output of each U-Net structural block is used as the input of the subsequent U-Net structural block network for iteration, and after several U-Net structural blocks, the final generated image with good exposure is output.

[0099] S306. Determine a model loss function according to the target output image and the normally exposed image corresponding to the poorly exposed image to be corrected, and train the multi-exposure correction model to be trained according to the model loss function to obtain a trained multi-exposure correction model.

[0100] The training model can be trained in an end-to-end manner, and the multi-exposure correction model is adjusted according to the L1 distance, cosine distance, structural similarity distance, and aesthetic feature distance between the target output image and the paired image with normal exposure (normal exposure image) in the training sample. Structural similarity is an indicator used to measure the similarity of the content of two images; and aesthetic feature distance is an indicator to measure the similarity of aesthetic feature statistics obtained after two images are input into the aesthetic discriminator.

[0101] Figure 6 A schematic diagram of the structure of an aesthetic discriminator in an image correction model provided in this application, such as Figure 6 As shown, E1, E2, and E3 represent encoders, each of which contains several convolutional layer operations. For example, it can be composed of four consecutive alternating pooling layers and convolutional layers, wherein each set of downsampling layers (pooling layers) and convolutional layers will reduce the output feature size to half of the original size and increase the number of channels to twice the original size, ultimately obtaining aesthetic features.

[0102] Among them, the number of model training iterations is preset, and when the number of iterations reaches the preset number, the training is stopped to obtain a trained multi-exposure correction model.

[0103] The image correction model training method provided in the embodiment of the present application designs an exposure normalization module and a multi-scale compensation module. The exposure normalization module maps different exposure features to an exposure invariant feature space. It roughly aligns different exposure features through instance normalization, and then performs normalized distillation loss on the normalized features to further reduce the impact of exposure. The multi-scale compensation module is used to integrate the unprocessed features of the exposure normalization part in the spatial and channel dimensions to compensate for the problem of loss of image discriminant features during image reconstruction due to normalization, thereby ensuring the integrity of information. Finally, an adversarial training strategy is introduced into the aesthetic discriminator to learn features that conform to human aesthetic perception from images with normal exposure, thereby generating more harmonious and realistic results, improving the aesthetic visual effect of the image, and achieving the effect of effectively correcting the abnormally exposed image while retaining important data information of the image.

[0104] Figure 7 A schematic diagram of the structure of an image correction device for aesthetic enhancement provided by the present application is shown in FIG. Figure 7 As shown, the present embodiment provides an aesthetically enhanced image correction device 70 comprising: An acquisition module 701 is used to acquire an exposure abnormality image and an initial feature image of the exposure abnormality image; A normalization module 702 is used to perform feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain an initial fused feature; A first compensation module 703 is used to obtain first depth feature information, second depth feature information, and a first fusion feature according to the initial fusion feature, wherein the first depth feature information is obtained by performing a first convolution process on the initial fusion feature, and the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, and the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information; The second compensation module 704 is used to perform a self-attention mechanism operation on the first depth feature information and the initial feature image to obtain first depth association information; and to perform a self-attention mechanism operation on the second depth feature information and the initial feature image to obtain second depth association information; The third compensation module 705 is used to obtain a target fusion feature image according to the first fusion feature, the second fusion feature and the initial fusion feature, wherein the second fusion feature is obtained according to the first depth association information and the second depth association information; The feature sampling module 706 is used to perform encoding and decoding processing on the target fused feature image to obtain a normally exposed image.

[0105] In a possible implementation, the normalization module 702 may also be used to: Inputting the initial feature image into the third convolution unit of the normalization module in the target model for third convolution processing to obtain the residual features of the initial feature image; Inputting the initial feature image into the normalization unit of the normalization module in the target model to obtain the normalized features of the initial feature image, wherein the normalized features of the initial feature image are obtained by performing the third convolution process on the initial normalized features, and the initial normalized features are obtained by performing instance normalization process on the initial feature image; The residual features of the initial feature image and the normalized features of the initial feature image are subjected to feature fusion processing to obtain initial fused features.

[0106] In a possible implementation, the first compensation module 703 may also be used to: Obtaining first depth feature information and second depth feature information according to the initial fusion feature; Inputting the first depth feature information into a first self-attention mechanism processing unit of a compensation module in a target model to obtain a first K value and a first V value; Inputting the second depth feature information into the second self-attention mechanism processing unit of the compensation module in the target model to obtain a second K value and a second V value; Obtain first depth fusion information according to the first K value and the second K value; Obtain second depth fusion information according to the first V value and the second V value; The first depth fusion information and the first depth fusion information are input into the first fusion unit of the compensation module in the target model for first fusion processing to obtain a first fusion feature.

[0107] In a possible implementation, the first compensation module 703 may also be used to: Inputting the initial fusion feature into the first convolution unit of the compensation module in the target model for a first convolution process to obtain first depth feature information; The initial fusion features are input into the second convolution unit of the compensation module in the target model for a second convolution process to obtain second depth feature information.

[0108] In a possible implementation, the second compensation module 704 may also be used to: Inputting the first depth feature information and the initial feature image into a first self-attention mechanism processing unit of a compensation module in a target model to perform a first self-attention mechanism operation process to obtain first depth association information, the first depth association information is determined according to the first depth feature information and a first association weight coefficient, and the first association weight coefficient is determined according to the first depth feature information and the initial feature image; The second depth feature information and the initial feature image are input into the second self-attention mechanism processing unit of the compensation module in the target model for second self-attention mechanism operation processing to obtain second depth association information. The second depth association information is determined according to the second depth feature information and the second association weight coefficient. The second association weight coefficient is determined according to the second depth feature information and the initial feature image.

[0109] In a possible implementation, the third compensation module 705 may also be used to: Inputting the first depth correlation information and the second depth correlation information into a second fusion unit of a compensation module in a target model for second fusion processing to obtain a target fusion feature; The first fusion feature, the second fusion feature and the initial fusion feature are input into the third fusion unit of the compensation module in the target model for third fusion processing to obtain a target fusion feature image.

[0110] In a possible implementation, the feature sampling module 706 may also be used to: The target fusion feature image is input into the feature extraction and encoding module in the target model to perform downsampling processing in the feature space to obtain multi-level image data information; The multi-level image data information is input into the feature fusion decoding module in the target model for upsampling processing in the feature space to obtain a normally exposed image.

[0111] In a possible implementation, the feature sampling module 706 may also be used to: Acquire an image to be trained with abnormal exposure and an aligned image with normal exposure corresponding to the image to be trained with abnormal exposure; Input the training image with abnormal exposure into the training model to obtain the target correction image; The model to be trained is trained according to the target correction image and the exposure normal alignment image to obtain a target model, wherein the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

[0112] In a possible implementation, the feature sampling module 706 may also be used to: Determine whether there is a next convolutional neural network connected to the target convolutional neural network, where the target convolutional neural network is a convolutional neural network that obtains a target correction image in the model to be trained; If the target convolutional neural network has a next convolutional neural network connected to it, the target correction image is used as the exposure abnormality training image to be trained and input into the next convolutional neural network, and the step of inputting the exposure abnormality training image to the model to be trained is performed again to obtain the target correction image, until the target convolutional neural network does not have a next convolutional neural network connected to it, and the exposure correction image is obtained; Determine the loss function of the model to be trained according to the exposure-corrected image and the exposure-normally aligned image. The loss function of the model to be trained includes a Manhattan distance function, a cosine distance function, a structural similarity distance function, an aesthetic feature distance function, and an adversarial training loss function. According to the loss function of the model to be trained, the model to be trained is trained to obtain the target model.

[0113] The present embodiment provides an aesthetically enhanced image correction device, which can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and are not described in detail in this embodiment.

[0114] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0115] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0116] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0117] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0118] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0119] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0120] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0121] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0122] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0123] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0124] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0127] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0128] Those skilled in the art can understand that all or part of the steps for implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes. Therefore, in addition to implementing the system and its various devices provided by the present invention in the form of computer-readable program code, it is entirely possible to implement the same functions by logically programming the method steps so that the system and its various devices provided by the present invention can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers.

[0129] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image correction method for aesthetic enhancement, characterized in that include: Acquire an exposure abnormality image and an initial characteristic image of the exposure abnormality image; Performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain initial fused features; According to the initial fusion feature, first depth feature information, second depth feature information, and a first fusion feature are obtained, wherein the first depth feature information is obtained by performing a first convolution process on the initial fusion feature, and the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, and the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information; Performing a self-attention mechanism operation on the first depth feature information and the initial feature image to obtain first depth association information; and performing a self-attention mechanism operation on the second depth feature information and the initial feature image to obtain second depth association information; Obtaining a target fusion feature image according to the first fusion feature, the second fusion feature and the initial fusion feature, wherein the second fusion feature is obtained according to the first depth association information and the second depth association information; The target fusion feature image is encoded and decoded to obtain a normally exposed image.

2. The method according to claim 1, characterized in that The performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain the initial fused features includes: Inputting the initial feature image into the third convolution unit of the normalization module in the target model for third convolution processing to obtain residual features of the initial feature image; Inputting the initial feature image into a normalization unit of a normalization module in the target model to obtain normalized features of the initial feature image, wherein the normalized features of the initial feature image are obtained by performing a third convolution process on the initial normalized features, and the initial normalized features are obtained by performing an instance normalization process on the initial feature image; The residual features of the initial feature image and the normalized features of the initial feature image are subjected to feature fusion processing to obtain initial fusion features.

3. The method according to claim 1, characterized in that The obtaining, according to the initial fusion feature, first depth feature information, second depth feature information, and first fusion feature, comprises: Obtaining first depth feature information and second depth feature information according to the initial fusion feature; Inputting the first depth feature information into a first self-attention mechanism processing unit of a compensation module in a target model to obtain a first K value and a first V value; Inputting the second depth feature information into a second self-attention mechanism processing unit of a compensation module in a target model to obtain a second K value and a second V value; Obtaining first depth fusion information according to the first K value and the second K value; Obtaining second depth fusion information according to the first V value and the second V value; The first depth fusion information and the first depth fusion information are input into a first fusion unit of a compensation module in a target model for a first fusion process to obtain a first fusion feature.

4. The method according to claim 3, characterized in that: The obtaining, according to the initial fusion feature, first depth feature information and second depth feature information, comprises: Inputting the initial fusion feature into the first convolution unit of the compensation module in the target model for a first convolution process to obtain first depth feature information; The initial fusion feature is input into the second convolution unit of the compensation module in the target model for a second convolution process to obtain second depth feature information.

5. The method according to claim 1, characterized in that The performing of a self-attention mechanism operation on the first depth feature information and the initial feature image to obtain first depth association information; and performing a self-attention mechanism operation on the second depth feature information and the initial feature image to obtain second depth association information, including: Inputting the first depth feature information and the initial feature image into a first self-attention mechanism processing unit of a compensation module in a target model to perform a first self-attention mechanism operation process to obtain first depth association information, where the first depth association information is determined according to the first depth feature information and a first association weight coefficient, and the first association weight coefficient is determined according to the first depth feature information and the initial feature image; The second depth feature information and the initial feature image are input into the second self-attention mechanism processing unit of the compensation module in the target model for second self-attention mechanism operation processing to obtain second depth association information, wherein the second depth association information is determined according to the second depth feature information and the second association weight coefficient, and the second association weight coefficient is determined according to the second depth feature information and the initial feature image.

6. The method according to claim 1, characterized in that The step of obtaining a target fusion feature image according to the first fusion feature, the second fusion feature and the initial fusion feature includes: Inputting the first depth association information and the second depth association information into a second fusion unit of a compensation module in a target model for second fusion processing to obtain a target fusion feature; The first fusion feature, the second fusion feature and the initial fusion feature are input into the third fusion unit of the compensation module in the target model for third fusion processing to obtain a target fusion feature image.

7. The method according to claim 1, characterized in that The encoding and decoding process is performed on the target fusion feature image to obtain a normal exposure image, including: Inputting the target fusion feature image into the feature extraction and encoding module in the target model to perform downsampling processing in the feature space to obtain multi-level image data information; The multi-level image data information is input into the feature fusion decoding module in the target model to perform upsampling processing in the feature space to obtain a normally exposed image.

8. The method according to any one of claims 2 to 7, characterized in that The method further comprises: Acquire an image to be trained with abnormal exposure and an aligned image with normal exposure corresponding to the image to be trained with abnormal exposure; Inputting the exposure abnormality training image into the training model to obtain a target correction image; The model to be trained is trained according to the target correction image and the normally exposed aligned image to obtain a target model, wherein the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

9. The method according to claim 8, characterized in that The step of training the model to be trained according to the target correction image and the normally exposed aligned image to obtain a target model comprises: Determine whether there is a next convolutional neural network connected to the target convolutional neural network, wherein the target convolutional neural network is a convolutional neural network that obtains the target corrected image in the model to be trained; If the target convolutional neural network has a next convolutional neural network connected to it, the target correction image is used as the exposure abnormality training image and input into the next convolutional neural network, and the step of inputting the exposure abnormality training image into the model to be trained to obtain the target correction image is performed again, until the target convolutional neural network does not have a next convolutional neural network connected to it, and an exposure correction image is obtained; Determine a loss function of a model to be trained according to the exposure-corrected image and the exposure-normally aligned image, wherein the loss function of the model to be trained includes a Manhattan distance function, a cosine distance function, a structural similarity distance function, an aesthetic feature distance function, and an adversarial training loss function; According to the loss function of the model to be trained, the model to be trained is trained to obtain a target model.

10. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 9.

Images