An aesthetic enhancement image correction method and an electronic device

Abstract

An embodiment of the present application provides an aesthetic enhancement image correction method and an electronic device. The method includes: obtaining an abnormally exposed image and a corresponding initial feature image; performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain initial fusion features; obtaining first depth feature information, second depth feature information, and first fusion features according to the initial fusion features; respectively performing self-attention mechanism operation processing on the first depth feature information and the second depth feature information according to the initial feature image to obtain first depth correlation information and second depth correlation information; obtaining a target fusion feature image according to the first fusion features, second fusion features, and initial fusion features; performing encoding and decoding processing on the target fusion feature image to obtain a normally exposed image. By associating and fusing different depth information, the present application makes the data of the corrected image more complete and enhances the aesthetic quality of the image.

Description

Technical Field

[0001] This application relates to the technical field of computer image data processing, and in particular, to an image correction method with aesthetic enhancement and an electronic device. Background Art

[0002] In recent years, mobile devices such as cameras and mobile phones have been used to capture photos of various scenes at any time. Due to different exposure conditions in different scenes, images taken with insufficient exposure or overexposure will result in poor aesthetic quality of the images. Therefore, image exposure correction, as an important part of the field of aesthetic enhancement, has an increasing strong demand. Application software or cameras using related technologies are easily favored by users. How to correct images to obtain high-quality images with aesthetic feeling has important research value.

[0003] Methods of using image histograms to adjust pixel intensity values or using tone curve adjustments to correct exposure errors can correct images. However, this method focuses on scenes with insufficient exposure or overexposure, resulting in poor generalization for other exposures, making it impossible to be deployed in practical applications. With the continuous development of artificial intelligence technology, in the prior art, data under different exposure conditions taken by mobile devices such as cameras and mobile phones are usually used to train a network to improve its multi-exposure correction ability.

[0004] However, in the process of image exposure correction by existing image correction network models, there is a problem that other image data information is missing when adjusting exposure image data, resulting in insufficient integrity of the corrected image data. Summary of the Invention

[0005] Embodiments of this application provide an image correction method with aesthetic enhancement and an electronic device, which are used to solve the problem that when existing image correction networks perform exposure correction processing on images, other image data information is missing when adjusting exposure image data, resulting in insufficient integrity of the corrected image data.

[0006] In a first aspect, embodiments of this application provide an image correction method with aesthetic enhancement, including:

[0007] Obtain an abnormally exposed image and an initial feature image of the abnormally exposed image;

[0008] Perform feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain an initial fusion feature;

[0009] Based on the initial fusion feature, first depth feature information, second depth feature information, and a first fusion feature are obtained. The first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information;

[0010] Perform a self-attention mechanism operation process on the first depth feature information and the initial feature image to obtain first depth correlation information; and perform a self-attention mechanism operation process on the second depth feature information and the initial feature image to obtain second depth correlation information;

[0011] Based on the first fusion feature, the second fusion feature, and the initial fusion feature, a target fusion feature image is obtained, and the second fusion feature is obtained according to the first depth correlation information and the second depth correlation information;

[0012] Perform an encoding and decoding process on the target fusion feature image to obtain a normally exposed image.

[0013] In a possible implementation manner, feature fusion processing is performed on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain the initial fusion feature, including:

[0014] Input the initial feature image into a third convolution unit of a normalization module in the target model to perform a third convolution process to obtain the residual feature of the initial feature image;

[0015] Input the initial feature image into the normalization unit of the normalization module in the target model to obtain the normalized feature of the initial feature image, where the normalized feature of the initial feature image is obtained by performing a third convolution process on the initial normalized feature, and the initial normalized feature is obtained by performing instance normalization processing on the initial feature image;

[0016] Perform feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain the initial fusion feature.

[0017] In a possible implementation manner, based on the initial fusion feature, first depth feature information, second depth feature information, and a first fusion feature are obtained, including:

[0018] Based on the initial fusion feature, first depth feature information and second depth feature information are obtained;

[0019] Input the first depth feature information into a first self-attention mechanism processing unit of a compensation module in the target model to obtain a first K value and a first V value;

[0020] Input the second depth feature information into the second self-attention mechanism processing unit of the compensation module in the target model to obtain the second K value and the second V value;

[0021] Obtain the first depth fusion information according to the first K value and the second K value;

[0022] Obtain the second depth fusion information according to the first V value and the second V value;

[0023] Input the first depth fusion information and the first depth fusion information into the first fusion unit of the compensation module in the target model for the first fusion processing to obtain the first fusion feature.

[0024] In a possible implementation manner, obtaining the first depth feature information and the second depth feature information according to the initial fusion feature includes:

[0025] Input the initial fusion feature into the first convolution unit of the compensation module in the target model for the first convolution processing to obtain the first depth feature information;

[0026] Input the initial fusion feature into the second convolution unit of the compensation module in the target model for the second convolution processing to obtain the second depth feature information.

[0027] In a possible implementation manner, performing self-attention mechanism operation processing on the first depth feature information and the initial feature image to obtain the first depth correlation information; and performing self-attention mechanism operation processing on the second depth feature information and the initial feature image to obtain the second depth correlation information, including:

[0028] Input the first depth feature information and the initial feature image into the first self-attention mechanism processing unit of the compensation module in the target model for the first self-attention mechanism operation processing to obtain the first depth correlation information, where the first depth correlation information is determined according to the first depth feature information and the first correlation weight coefficient, and the first correlation weight coefficient is determined according to the first depth feature information and the initial feature image;

[0029] Input the second depth feature information and the initial feature image into the second self-attention mechanism processing unit of the compensation module in the target model for the second self-attention mechanism operation processing to obtain the second depth correlation information, where the second depth correlation information is determined according to the second depth feature information and the second correlation weight coefficient, and the second correlation weight coefficient is determined according to the second depth feature information and the initial feature image.

[0030] In a possible implementation manner, obtaining the target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature includes:

[0031] Input the first depth correlation information and the second depth correlation information into the second fusion unit of the compensation module in the target model for second fusion processing to obtain the target fusion feature;

[0032] Input the first fusion feature, the second fusion feature, and the initial fusion feature into the third fusion unit of the compensation module in the target model for third fusion processing to obtain the target fusion feature image.

[0033] In a possible implementation manner, performing encoding and decoding processing on the target fusion feature image to obtain a normally exposed image, including:

[0034] Input the target fusion feature image into the feature extraction and encoding module in the target model for downsampling processing in the feature space to obtain multi-level image data information;

[0035] Input the multi-level image data information into the feature fusion and decoding module in the target model for upsampling processing in the feature space to obtain a normally exposed image.

[0036] In a possible implementation manner, the method further includes:

[0037] Obtain an image with abnormal exposure to be trained and a normally exposed aligned image corresponding to the image with abnormal exposure to be trained;

[0038] Input the image with abnormal exposure to be trained into the model to be trained to obtain a target corrected image;

[0039] Train the model to be trained according to the target corrected image and the normally exposed aligned image to obtain the target model, where the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

[0040] In a possible implementation manner, training the model to be trained according to the target corrected image and the normally exposed aligned image to obtain the target model includes:

[0041] Determine whether there is a connected next convolutional neural network in the target convolutional neural network, where the target convolutional network is the convolutional neural network in the model to be trained that obtains the target corrected image;

[0042] If there is a connected next convolutional neural network in the target convolutional neural network, then use the target corrected image as the image with abnormal exposure to be trained and input it into the next convolutional neural network, and execute again the step of inputting the image with abnormal exposure to be trained into the model to be trained to obtain the target corrected image until there is no connected next convolutional neural network in the target convolutional neural network to obtain the exposure corrected image;

[0043] Determine the loss function of the model to be trained based on the exposure-corrected image and the normally-exposed aligned image. The loss function of the model to be trained includes a Manhattan distance function, a cosine distance function, a structural similarity distance function, an aesthetic feature distance function, and an adversarial training loss function;

[0044] Train the model to be trained according to the loss function of the model to be trained to obtain the target model.

[0045] In a second aspect, an image correction device for aesthetic enhancement provided by an embodiment of the present application includes:

[0046] An acquisition module, configured to acquire an abnormally-exposed image and an initial feature image of the abnormally-exposed image;

[0047] A normalization module, configured to perform feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain an initial fusion feature;

[0048] A first compensation module, configured to obtain first depth feature information, second depth feature information, and a first fusion feature according to the initial fusion feature. The first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information;

[0049] A second compensation module, configured to perform a self-attention mechanism operation process on the first depth feature information and the initial feature image to obtain first depth correlation information; and perform a self-attention mechanism operation process on the second depth feature information and the initial feature image to obtain second depth correlation information;

[0050] A third compensation module, configured to obtain a target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature. The second fusion feature is obtained according to the first depth correlation information and the second depth correlation information;

[0051] A feature sampling module, configured to perform an encoding and decoding process on the target fusion feature image to obtain a normally-exposed image.

[0052] In a third aspect, an electronic device provided by an embodiment of the present application includes: a memory, a processor;

[0053] The memory stores computer-executable instructions;

[0054] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementation manners of the first aspect.

[0055] Fourthly, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect and / or various possible implementation manners of the first aspect.

[0056] Fifthly, an embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the above first aspect and / or various possible implementation manners of the first aspect.

[0057] An aesthetic-enhanced image correction method and an electronic device provided by an embodiment of the present application obtain an overexposed or underexposed image and an initial feature image of the overexposed or underexposed image; perform feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain initial fusion features; obtain first depth feature information, second depth feature information, and a first fusion feature according to the initial fusion features; use the initial feature image to perform self-attention mechanism operation processing on the first depth feature information and the second depth feature information respectively to obtain first depth correlation information and second depth correlation information; obtain a target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature; perform encoding and decoding processing on the target fusion feature image to obtain an image with normal exposure. By this means, the initial feature image with abnormal exposure is mapped to an exposure-invariant feature space. While reducing the influence of exposure on the image by using the normalized features, the residual features and the normalized features are fused, making the image information more complete and enhancing the expression ability of the image. Then, through depth feature extraction with different-sized convolutional kernels and self-attention mechanism operations, target fusion features are obtained. Among them, the features of different-sized convolutional kernels can capture the detailed data and global information of the image more comprehensively, and the self-attention mechanism enhances the focusing ability of the model on the key information data in the image, enhancing the aesthetic quality of the image. Finally, through the encoding and decoding network structure, the effective reconstruction of the fusion features is realized to obtain an image after exposure correction, effectively correcting the overexposed or underexposed image while retaining the important data information of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application and used together with the specification to explain the principles of the present application.

[0059] Figure 1 It is a schematic flowchart of an aesthetic-enhanced image correction method provided by the present application;

[0060] Figure 2 It is a schematic flowchart of another aesthetic-enhanced image correction method provided by the present application;

[0061] Figure 3 Schematic flowchart of a method for training an image correction model provided by this application;

[0062] Figure 4 Schematic diagram of the structures of an exposure normalization module and a multi-scale compensation module in an image correction model provided by this application;

[0063] Figure 5 Schematic diagram of the structures of a feature extraction and encoding module and a feature fusion and decoding module in an image correction model provided by this application;

[0064] Figure 6 Schematic diagram of the structure of an aesthetic discriminator in an image correction model provided by this application;

[0065] Figure 7 Schematic diagram of the structure of an image correction device for aesthetic enhancement provided by this application;

[0066] Figure 8 Schematic diagram of the structure of an electronic device provided by this application.

[0067] Through the above-mentioned drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0068] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0069] In the prior art, data under different exposure conditions are usually used to train a network to improve its multi-exposure correction ability. For example, in a traditional low-light imaging algorithm, the preprocessed image is subjected to Gamma correction to improve the exposure accuracy of the image. Among them, the principle of Gamma correction is to draw a Gamma curve that meets the requirements based on a preset Gamma value, substitute the normalized pixel values in the image into the curve to obtain the corresponding output values, and then inverse-normalize them into the integer values corresponding to the image. Although this method has simple steps and can directly process the data of the original sensor, and enhance the brightness of low-light images through a deep learning model. However, for multi-exposure images, the correction process of adjusting underexposed images and overexposed images to normally exposed images is a reverse process. Therefore, in the image correction process, there will be a situation where the network model cannot take into account the underexposed and overexposed image data for reverse correction, resulting in the loss of some detailed data of the image during the correction process, and there is a problem of poor correction effect in the corrected image. At the same time, this makes the performance of the network model poor and cannot be well used in multi-exposure scenarios.

[0070] To solve the above problems, the embodiments of the present application provide an aesthetic enhancement image correction method and an electronic device, which can map an initial feature image with abnormal exposure to an exposure-invariant feature space, and while using the normalized features to reduce the impact of exposure on the image, fuse the residual features and the normalized features to enhance the expression ability of the image. Then, through the depth feature extraction with different-sized convolutional kernels and the self-attention mechanism operation, the target fusion feature is obtained. Among them, by combining the features of different-sized convolutional kernels, the detailed data and global information of the image can be captured more comprehensively, and the self-attention mechanism enhances the focusing ability of the model on the key information in the image, enhancing the aesthetic quality of the image. Finally, through the encoder-decoder network structure, the effective reconstruction of the fusion feature is realized to obtain the image after exposure correction, realizing the effective correction of the image with abnormal exposure while retaining the important data information of the image.

[0071] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. These specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0072] An embodiment of the present application provides an aesthetic enhancement image correction method, and the execution subject of this method can be a server. Among them, the server can be devices such as cameras, mobile phones, computers, tablets, etc. The embodiment of the present application does not particularly limit the implementation manner of the execution subject, as long as the execution subject can obtain an overexposed or underexposed image and the initial feature image of the overexposed or underexposed image; perform feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain an initial fusion feature; according to the initial fusion feature, obtain first depth feature information, second depth feature information, and a first fusion feature, where the first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information; perform a self-attention mechanism operation process on the first depth feature information and the initial feature image to obtain first depth correlation information; and perform a self-attention mechanism operation process on the second depth feature information and the initial feature image to obtain second depth correlation information; according to the first fusion feature, the second fusion feature, and the initial fusion feature, obtain a target fusion feature image, where the second fusion feature is obtained according to the first depth correlation information and the second depth correlation information; perform an encoding and decoding process on the target fusion feature image to obtain a normally exposed image.

[0073] Among them, the self-attention mechanism can be a deep learning technology that calculates the correlation degree of the first data information (corresponding to query Q-Query) and the second data information (corresponding to key K-Key and value V-Value) in the model according to the attention score and the Softmax normalization function.

[0074] Figure 1 It is a schematic flowchart of an aesthetic enhancement image correction method provided by the present application. The execution subject of this method can be a server storing the aesthetic enhancement image correction method or other servers, and this embodiment does not make special restrictions here, such as Figure 1 As shown, this method may include:

[0075] S101. Obtain an overexposed or underexposed image and the initial feature image of the overexposed or underexposed image.

[0076] Among them, the overexposed or underexposed image may refer to an underexposed image, an overexposed image, and an overexposed or underexposed image in a high dynamic range scene. It may also refer to an image with overexposure or underexposure in certain specific areas, for example, an abnormal image caused by different light reflection capabilities of certain objects or areas in the scene, or the failure of the camera's exposure control to correctly adapt to the changes in the scene. It may also refer to an image with improper use of a flash or in a long-exposure environment. The overexposed or underexposed image can be obtained when taking images through electronic devices such as cameras and mobile phones.

[0077] The initial feature image can perform arithmetic processing on the overexposed abnormal image according to the preset convolution and ReLU activation functions in the target model, so as to map the image to the encoding space and obtain a feature image with the same size as the overexposed abnormal image.

[0078] S102. Perform feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain initial fusion features.

[0079] Among them, the residual features can refer to the features after performing convolution operations on the initial feature image. The residual features can be in the residual branch, which is a part of the network model and allows the original input to bypass some layers and be directly added to the output. Through the residual features, the network can more easily learn effective feature representations.

[0080] The normalized features can refer to the features after normalizing the initial feature image. For example, use instance normalization to calculate the mean and standard deviation of the features on each channel dimension of the initial feature image, and then use the learning parameters of the model and these metric data to perform style normalization on the features in the feature space. Regarding different degrees of exposure as a kind of style, so using instance normalization can align different exposures, thereby reducing their representational differences.

[0081] Furthermore, the normalized features can satisfy:

[0082] ;

[0083] Among them, is the normalized feature; is the initial feature image; is the instance normalization; is the mean of the features; is the standard deviation of the features; , are the learning parameters of the model.

[0084] The method of feature fusion processing can be to splice and fuse the features on each channel dimension to reduce the number of channels and further integrate data information.

[0085] Among them, in the embodiments of the present application, the method of performing feature fusion processing on the residual features of the initial feature image and the normalized features of the initial feature image to obtain initial fusion features may include:

[0086] Input the initial feature image into the third convolution unit of the normalization module in the target model to perform third convolution processing to obtain the residual features of the initial feature image;

[0087] Input the initial feature image into the normalization unit of the normalization module in the target model to obtain the normalized feature of the initial feature image. Among them, the normalized feature of the initial feature image is obtained by performing a third convolution process on the initial normalized feature, and the initial normalized feature is obtained by performing instance normalization on the initial feature image;

[0088] Perform feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain the initial fusion feature.

[0089] In some embodiments, the target model may refer to a network model that has been trained and is applicable to exposure image correction. The third convolution process can be performed using a 1×1 convolution.

[0090] S103. According to the initial fusion feature, obtain the first depth feature information, the second depth feature information, and the first fusion feature. The first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information.

[0091] In some embodiments, the first convolution process can be performed using a 3×3 convolution (Conv), and the second convolution process can be performed using a 5×5 convolution (Conv). For example: the first depth feature information satisfies and the second depth feature information satisfies , where is the first depth feature information; is the second depth feature information; is the initial fusion feature.

[0092] Since convolution kernels of different sizes capture different image information of the initial fusion feature during convolution processing, a smaller-sized convolution kernel (such as a 3×3 convolution) captures local details of the image, such as edges and textures, while a larger-sized convolution kernel captures global features of the image, such as the overall shape of an object and the layout of a scene. Therefore, using multi-scale convolution to process features can enhance the local information of the image.

[0093] Among them, in the embodiments of the present application, the method for obtaining the first depth feature information, the second depth feature information, and the first fusion feature according to the initial fusion feature may include:

[0094] According to the initial fusion feature, obtain the first depth feature information and the second depth feature information;

[0095] Input the first depth feature information into the first self-attention mechanism processing unit of the compensation module in the target model to obtain the first K value and the first V value;

[0096] Input the second depth feature information into the second self-attention mechanism processing unit of the compensation module in the target model to obtain the second K value and the second V value;

[0097] Obtain the first depth fusion information according to the first K value and the second K value;

[0098] Obtain the second depth fusion information according to the first V value and the second V value;

[0099] Input the first depth fusion information and the first depth fusion information into the first fusion unit of the compensation module in the target model for the first fusion process to obtain the first fusion feature.

[0100] Among them, in the self-attention mechanism processing unit, the model will calculate the query (Q), key (K), and value (V) corresponding to the input elements respectively, and these calculations are usually obtained through different linear transformations.

[0101] In the embodiments of the present application, the initial feature image is regarded as the query vector Q, and the depth feature information is regarded as the key-value pair vector K-V. Among them, since the convolution kernel sizes of the first depth feature information and the second depth feature information are different, when performing the first fusion process, in order to fuse with the initial fusion feature later, convolution processing is performed on the information during fusion so that the convolution scale of the first fusion feature is the same as that of the initial fusion feature. For example: 、 ,where is the first K value; is the first V value; is the second K value; is the second V value. Then, the first depth fusion information can satisfy ; the second depth fusion information can satisfy . Thus, the obtained first fusion feature can satisfy .

[0102] Among them, in the embodiments of the present application, the method for obtaining the first depth feature information and the second depth feature information according to the initial fusion feature may include:

[0103] Input the initial fusion feature into the first convolution unit of the compensation module in the target model for the first convolution process to obtain the first depth feature information;

[0104] Input the initial fusion feature into the second convolution unit of the compensation module in the target model for the second convolution process to obtain the second depth feature information.

[0105] Among them, the first convolution process and the second convolution process here are only for the purpose of differentially describing convolutions of different sizes, rather than being limited to only two sizes of convolutions. In some embodiments, convolutional units and corresponding self-attention mechanism processing units in the compensation module can be added to improve the integrity of local information and image discrimination features, thereby avoiding the problem of data loss during the exposure correction process of the image.

[0106] S104. Perform self-attention mechanism operation processing on the first depth feature information and the initial feature image to obtain the first depth correlation information; and perform self-attention mechanism operation processing on the second depth feature information and the initial feature image to obtain the second depth correlation information.

[0107] In the embodiment of the present application, the initial feature image is regarded as the query vector Q. Since the convolutional kernel sizes of the first depth feature information and the second depth feature information are different, before performing the self-attention mechanism operation processing, it is necessary to perform convolutional processing on the initial feature image with the same size. For example: if the first convolutional unit is a 3×3 convolution, then the corresponding first Q value satisfies ; if the second convolutional unit is a 5×5 convolution, then the corresponding second Q value satisfies .

[0108] The self-attention mechanism operation processing can be to perform a dot product processing on the Q value and the K value in the self-attention mechanism processing unit to obtain the attention score, then determine the attention weight through the Softmax function, and perform a dot product processing on the attention weight and the V value to obtain the depth correlation information. By performing correlation fusion on multi-scale feature information, the integrity of the image during the exposure correction process can be ensured, thereby ensuring that the image is aesthetically attractive both globally and locally.

[0109] For example: the depth correlation information can satisfy:

[0110] ;

[0111] Among them, is the depth correlation information; is the scaling factor to prevent the gradient from being too small during backpropagation, resulting in the inability to update the parameters; is the dimension of the K vector.

[0112] Among them, in the embodiment of the present application, the method of performing self-attention mechanism operation processing on the first depth feature information and the initial feature image to obtain the first depth correlation information; and performing self-attention mechanism operation processing on the second depth feature information and the initial feature image to obtain the second depth correlation information may include:

[0113] Input the first depth feature information and the initial feature image into the first self-attention mechanism processing unit of the compensation module in the target model to perform the first self-attention mechanism operation and processing, obtaining the first depth correlation information. The first depth correlation information is determined according to the first depth feature information and the first correlation weight coefficient, and the first correlation weight coefficient is determined according to the first depth feature information and the initial feature image.

[0114] Input the second depth feature information and the initial feature image into the second self-attention mechanism processing unit of the compensation module in the target model to perform the second self-attention mechanism operation and processing, obtaining the second depth correlation information. The second depth correlation information is determined according to the second depth feature information and the second correlation weight coefficient, and the second correlation weight coefficient is determined according to the second depth feature information and the initial feature image.

[0115] In some embodiments, the depth feature information and the initial feature image can be subjected to dot product processing to obtain attention scores, and then, according to the vector dimension in the depth feature information, the attention scores are processed through the Softmax function to obtain the correlation weight coefficient. The correlation weight coefficient can represent the data information in the initial feature image that is most relevant to the features in the current depth feature information. Therefore, performing dot product processing on the correlation weight coefficient and the depth feature information can make the pixel values corresponding to the important data in the depth correlation information as close as possible to 1, and eliminate the irrelevant pixel points (feature information with the correlation weight coefficient approaching 0 infinitely).

[0116] Among them, the first self-attention mechanism processing unit and the second self-attention mechanism processing unit here are to correspond to convolution units of different sizes so that each can perform operation and processing, rather than being limited to dual-channel self-attention mechanism processing units. In some implementation manners, the convolution units and the corresponding self-attention mechanism processing units in the compensation module can be increased to improve the integrity of local information and image discrimination features, thereby avoiding the problem of data loss during the exposure correction process of the image.

[0117] S105. Obtain the target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature, where the second fusion feature is obtained according to the first depth correlation information and the second depth correlation information.

[0118] In some embodiments, the target fusion feature image can satisfy:

[0119] ;

[0120] Among them, is the target fusion feature image.

[0121] Further, in order to facilitate sampling processing for the feature extraction and encoding module and the feature fusion and decoding module in the model, the target fusion feature image can be first subjected to a fourth convolution process, where the convolution kernel size of the fourth convolution process is the same as that of the convolution kernels in the feature extraction and encoding module and the feature fusion and decoding module. For example, if the feature extraction and encoding module consists of three convolutional layers and the feature fusion and decoding module consists of three transposed convolutional layers, then the target fusion feature image can be subjected to a 3×3 convolution process before performing subsequent steps.

[0122] Among them, in the embodiments of the present application, the method for obtaining the target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature may include:

[0123] Input the first depth correlation information and the second depth correlation information into the second fusion unit of the compensation module in the target model for second fusion processing to obtain a target fusion feature;

[0124] Input the first fusion feature, the second fusion feature, and the initial fusion feature into the third fusion unit of the compensation module in the target model for third fusion processing to obtain a target fusion feature image.

[0125] In some embodiments, the second fusion feature may satisfy:

[0126] ;

[0127] Among them, is the second fusion feature.

[0128] S106. Perform encoding and decoding processing on the target fusion feature image to obtain a normally exposed image.

[0129] Performing encoding and decoding processing on the target fusion feature image may include encoding the target fusion feature image, using downsampling to gradually extract important data information in the image, such as texture, edges, shapes, etc., and then decoding the important data information, using upsampling to gradually fuse the information to reconstruct the corrected image.

[0130] Among them, in the embodiments of the present application, the method for performing encoding and decoding processing on the target fusion feature image to obtain a normally exposed image may include:

[0131] Input the target fusion feature image into the feature extraction and encoding module in the target model for downsampling processing in the feature space to obtain multi-level image data information;

[0132] Input the multi-level image data information into the feature fusion and decoding module in the target model for upsampling processing in the feature space to obtain a normally exposed image.

[0133] In some embodiments, the feature extraction and encoding module consists of three convolutional layers, which perform downsampling on the spatial resolution of the features; the feature fusion and decoding module consists of three transposed convolutional layers, which perform upsampling on the obtained image feature data information. There are skip connections between the convolutional layers of the feature extraction and encoding module and the feature fusion and decoding module. Each skip connection generates residual features at scales S1 = 1 / 4, S2 = 1 / 2, and S3 = 1 respectively, connecting features with the same spatial resolution from the shallow layer to the deep layer, so that the local information contained in the features generated by the shallow layer reaches the output.

[0134] Wherein, in the embodiments of the present application, the method may further include:

[0135] Obtain an image to be trained with abnormal exposure and an exposure-normal aligned image corresponding to the image to be trained with abnormal exposure;

[0136] Input the image to be trained with abnormal exposure into the model to be trained to obtain a target corrected image;

[0137] Train the model to be trained according to the target corrected image and the exposure-normal aligned image to obtain a target model, wherein the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

[0138] In some embodiments, the method for obtaining an image to be trained with abnormal exposure and an exposure-normal aligned image corresponding to the image to be trained with abnormal exposure may be obtained through various data sets or specific photography techniques.

[0139] For example, a data set can be created by taking photos of the same scene under different exposure settings, such as taking multiple photos of the same scene using the bracketing exposure technique, and then selecting underexposed, normal, and overexposed images as the data set. There are some publicly available image data sets that contain images under different exposure conditions. For example, some data sets are specifically for image enhancement and correction tasks and may contain the required paired images. Use image editing software (such as Adobe Photoshop) to manually adjust the exposure of a set of images to create paired images with poor exposure and normal exposure. Write scripts using a computer vision library (such as OpenCV) to automatically adjust the exposure of a set of images to generate the required paired images.

[0140] The convolutional neural network may refer to the neural network in the U-Net structural block. The U-Net structure design includes a contracting (encoding) path and a symmetric expanding (decoding) path. The overall shape is similar to the letter "U", so it gets its name. It can perform end-to-end training directly from the original image to the segmentation map without additional feature engineering, which speeds up the training process.

[0141] In some embodiments, the target number of iterations can be set during training. After each training to obtain the target model, the current number of iterations is recorded. When the current number of iterations meets the target number of iterations, the training is stopped, and the trained target model is obtained.

[0142] Among them, in the embodiments of the present application, the model to be trained is trained according to the target correction image and the exposure-normal aligned image to obtain the target model, including:

[0143] Determine whether there is a connected next convolutional neural network in the target convolutional neural network. The target convolutional network is the convolutional neural network that obtains the target correction image in the model to be trained;

[0144] If there is a connected next convolutional neural network in the target convolutional neural network, then use the target correction image as the exposure-abnormal image to be trained and input it into the next convolutional neural network, and then execute the step of inputting the exposure-abnormal image to be trained into the model to be trained to obtain the target correction image again, until there is no connected next convolutional neural network in the target convolutional neural network, and the exposure correction image is obtained;

[0145] According to the exposure correction image and the exposure-normal aligned image, determine the loss function of the model to be trained. The loss function of the model to be trained includes Manhattan distance function, cosine distance function, structural similarity distance function, aesthetic feature distance function, and adversarial training loss function;

[0146] Train the model to be trained according to the loss function of the model to be trained to obtain the target model.

[0147] Among them, in the embodiments of the present application, after determining whether there is a connected next convolutional neural network in the target convolutional neural network, the method further includes:

[0148] If there is no connected next convolutional neural network in the target convolutional neural network, then use the target correction image as the exposure correction image;

[0149] According to the exposure correction image and the exposure-normal aligned image, determine the loss function of the model to be trained;

[0150] Train the model to be trained according to the loss function of the model to be trained to obtain the target model.

[0151] In some embodiments, the Manhattan distance function can refer to the L1 distance (Manhattan Distance), which is the sum of the absolute values of the differences in pixel values corresponding to two images. The Manhattan distance function can satisfy:

[0152] ;

[0153] Among them, is the Manhattan distance function; is the exposure-corrected image; is the normally-exposed aligned image.

[0154] The cosine distance function can measure the similarity in direction between two vectors rather than their magnitudes. In image processing, an image can be converted into a vector, and then the cosine distance can be calculated. The cosine distance function can satisfy:

[0155] ;

[0156] where, is the cosine distance function.

[0157] The structural similarity distance function can refer to using the structural similarity distance (Structural Similarity Index, SSIM) to measure the structural similarity between two images, specifically considering luminance, contrast, and structural information. The value of SSIM ranges from -1 to 1, and the closer the value is to 1, the more similar the images are, indicating that the enhanced image effect is closer to the real image. The structural similarity distance function can satisfy:

[0158] ;

[0159] where, is the structural similarity distance function; is the average value of the exposure-corrected image; is the average value of the normally-exposed aligned image; is the standard deviation of the exposure-corrected image; is the standard deviation of the normally-exposed aligned image; is the covariance between the exposure-corrected image and the normally-exposed aligned image; 、 are constants to prevent the situation where the numerator and denominator are too small.

[0160] The aesthetic feature distance function can refer to measuring the difference between images based on the aesthetic quality of the images, which can involve aesthetic-related features such as the tone, composition, texture, and contrast of the images. The aesthetic feature distance function can satisfy:

[0161] ;

[0162] where, is the aesthetic feature distance function; is the aesthetic discriminator; is the mean value of each channel of the aesthetic features output by the aesthetic discriminator; is the variance of each channel of the aesthetic features output by the aesthetic discriminator.

[0163] Furthermore, the aesthetic discriminator can be used to provide feedback by comparing the aesthetic quality of the generated image with that of the real image, helping the generation model optimize the generation process.

[0164] The adversarial training loss function can refer to introducing an aesthetic discriminator component into the model, enabling the model's image generator and the aesthetic discriminator to form an adversarial network. Using the adversarial training strategy, there is a min-max game between the generator and the discriminator. The generator attempts to generate as realistic an image as possible, while the discriminator tries to distinguish between real images and generated images, making the training model generate visually more harmonious and realistic images. The adversarial training loss function can satisfy:

[0165] ;

[0166] where is the adversarial training loss function.

[0167] The image correction method provided by the embodiments of the present application maps the initial feature image with abnormal exposure to the exposure-invariant feature space, fuses the residual feature and the normalized feature while using the normalized feature to reduce the impact of exposure on the image, enhances the expressive ability of the image, and then through the depth feature extraction with different-sized convolutional kernels and the self-attention mechanism operation, combines the features of different convolutional levels, can capture the details and global information of the image more comprehensively. The self-attention mechanism enhances the model's focusing ability on the key information in the image, enhances the aesthetic quality of the image. Finally, through the encoder-decoder network structure, the effective reconstruction of the fused feature is realized to obtain the image after exposure correction, achieving the effective correction of the image with abnormal exposure, while retaining the important data information of the image. This method utilizes the powerful feature extraction and learning ability of the convolutional neural network in the depth information to improve the rationality and accuracy of image data processing during the correction process, making the corrected image effect closer to the real image, improving the accuracy of the correction. Since the diversity of the training images with abnormal exposure is considered during model training, the model can be applied to different types of abnormal exposure and has good generalization ability.

[0168] Figure 2 is a schematic flowchart of another aesthetic enhancement image correction method provided by the embodiments of the present application. As Figure 2 shown, the method may include:

[0169] S201. Obtain overexposed or underexposed images with different degrees to be corrected.

[0170] S202. Input the obtained images with abnormal exposure into the trained multi-exposure correction model to obtain images with good exposure.

[0171] Among them, the trained multi-exposure correction model can be a basic framework model composed of multiple consecutive U-Net structural blocks. The U-Net structure specifically includes an exposure normalization and multi-scale compensation module, a feature extraction and encoding module, and a feature fusion and decoding module.

[0172] The image correction method provided by the embodiments of this application can perform image correction through a multi-exposure correction model, enabling the corrected normally exposed image to retain the data integrity of the image to be corrected, not only achieving an enhanced result with normal exposure, but also having good reconstructed details and visually satisfactory contrast and color distribution.

[0173] Figure 3 It is a schematic flowchart of a method for training an image correction model provided by the embodiments of this application. As Figure 3 shown, the method may include:

[0174] S301. Based on each convolutional neural network structural block, input the poorly exposed image to be corrected into the exposure normalization module. Through a preset first convolutional operation and normalization operation, map diverse exposure features to an exposure-independent feature space to obtain a first feature image with invariant exposure.

[0175] Among them, before training, it is also necessary to initialize the model to be trained, including initializing the feature extraction and encoding module, the exposure normalization and multi-scale compensation module, and the feature fusion and decoding module.

[0176] The poorly exposed image to be corrected can be an image obtained by acquiring paired poorly exposed images and aligned normally exposed images in a natural scene as training sample images.

[0177] Furthermore, the exposure normalization module is composed of an instance normalization layer and multiple convolutional layers. Each time the image passes through a layer, a feature image of a specific size will be obtained, with the same width and height dimensions as the input but channel expansion, containing high-level features of the original image.

[0178] After obtaining the normalized features through the normalization operation in the exposure normalization module, the normalized features can also be subjected to a Normalized Distillation Loss. It is a technique used for Knowledge Distillation in machine learning. Knowledge Distillation allows a lightweight model (student model) to learn and imitate the behavior of a more complex and better-performing model (teacher model). The Normalized Distillation Loss is one of the loss functions used in the distillation process, aiming to improve the performance of the student model.

[0179] S302. Input the first feature image into the multi-scale compensation module, and through the second preset convolution operation and multi-branch fusion operation, integrate and compensate to obtain the second feature image.

[0180] Among them, the multi-scale compensation module consists of two parallel convolutional layers with different sizes and a self-attention mechanism. Each time the image passes through a convolutional layer of a certain size, a feature image of the same size will be obtained. Perform the corresponding channel self-attention mechanism on the feature images of their respective sizes to obtain the feature images enhanced by the self-attention mechanism. The output feature image sizes under different sizes remain the same. Then, the multi-output feature images are fused and stitched by channels, and then a convolutional layer is used to output the restored second feature image.

[0181] Furthermore, the parallel convolutional layers of the multi-scale compensation module mainly learn and restore the multi-scale texture information of the normalized features. Different scale branches correspond to different sizes of convolutional kernel sizes, so as to obtain different receptive fields and restore the texture information of different scales.

[0182] The channel self-attention mechanism mainly learns the correlation weight relationship between the features of any two channels, treats the features of each channel as an element in the input sequence, and the number of channels is equal to the length of the input sequence. Through this self- and self-channel related attention mechanism, the representation ability of the model method is enhanced.

[0183] Figure 4 The structural schematic diagrams of the exposure normalization module and the multi-scale compensation module in an image correction model provided by an embodiment of the present application are as Figure 4 shown:

[0184] In the exposure normalization module, the initial feature image is input , and then it enters the two branches respectively. Among them, the first branch first performs instance normalization operation on , and then obtains the normalized feature through a 1x1 convolution operation. Then, the second branch performs a convolution operation on , and is used as the residual branch to fuse with the normalized feature to obtain the fused feature , and input it into the multi-scale compensation module. Since the normalization operation will inevitably lead to the loss of discriminative features, resulting in insufficient information for image reconstruction. To solve this problem, therefore, we introduce the multi-scale compensation module to integrate the exposure-invariant features that are not processed by the exposure normalization in the spatial and channel dimensions, ensuring the integrity of information, so as to obtain satisfactory contrast and details.

[0185] The multi-scale compensation module first introduces multi-scale convolution to enhance local information. In addition, it introduces dual-channel self-attention, by focusing on the initial input features and the channel information relationship with the fusion features to enhance the discriminative learning ability of the network. Specifically, the fusion features are input into two parallel branches, and 3x3 and 5x5 depth convolutions are used to enhance the multi-scale local information extraction ability. After obtaining the features with two different receptive fields, cross-fusion is performed to obtain multi-scale feature information. Then, the dual-channel self-attention mechanism is used to regard the initial input features as the query vector Query, and the fusion features as the key-value pair vector Key-Value, and then the self-attention mechanism operation is carried out, and after feature fusion, the target fusion feature image is obtained and input into the feature extraction and encoding module.

[0186] S303. Input the second feature image into the feature extraction and encoding module, and perform the third preset multi-layer convolution operation to obtain multi-level image data information.

[0187] Among them, the feature extraction and encoding module consists of three convolutional layers. Each time the image passes through a convolutional layer, it is downsampled to obtain a feature image with a specific scale. Finally, a feature image containing the high-level features of the original image is obtained, and the high-level image features of each layer in the above feature extraction and encoding module are retained.

[0188] S304. Input the multi-level image data information into the feature fusion module, and perform layer-by-layer processing on the multi-level image data information through the preset multi-layer deconvolution operation to obtain the output image.

[0189] Among them, the feature fusion and decoding module consists of three deconvolutional layers. The high-level features of the image in the decoding module are first fused with the output features of the corresponding encoder with the same size. Each time it passes through a deconvolutional layer, it is upsampled to obtain a feature image with a specific size. Finally, the output result of a certain U-Net structure block is obtained.

[0190] Figure 5 is a schematic diagram of the structures of the feature extraction and encoding module and the feature fusion and decoding module in an image correction model provided by an embodiment of the present application. As Figure 5 shown, the feature fusion and decoding module consists of three convolutional layers and three deconvolutional layers. First, the spatial resolution of the features is downsampled, and then upsampling is performed through deconvolution. There are skip connections between each convolutional layer. Each skip connection generates residual features at scales S1 = 1 / 4, S2 = 1 / 2, and S3 = 1 respectively, connecting the features with the same spatial resolution from the shallow layer to the deep layer, which helps the local information contained in the features generated by the shallow layer reach the output.

[0191] S305. Input the output image as the underexposed image to be corrected into the next convolutional neural network building block, and repeat the steps of S301 - S304 until all the convolutional neural network building block steps in the multi-exposure correction model to be trained are completed, obtaining the target output image.

[0192] Among them, the output of each U-Net building block is used as the input of the subsequent U-Net building block network for iteration, and after several U-Net building blocks, the final well-exposed generated image is output.

[0193] S306. Determine the model loss function based on the target output image and the normally exposed image corresponding to the underexposed image to be corrected, and train the multi-exposure correction model to be trained according to the model loss function, obtaining the trained multi-exposure correction model.

[0194] Among them, the way to train the model can be an end-to-end training method. By calculating the L1 distance, cosine distance, structural similarity distance, and aesthetic feature distance between the target output image and the normally exposed paired image (normally exposed image) in the training sample, the multi-exposure correction model is adjusted. Among them, structural similarity is an index used to measure the similarity degree of the content of two images; the aesthetic feature distance is an index used to measure the similarity of the aesthetic feature statistics obtained after two images are input into the aesthetic discriminator.

[0195] Figure 6 This is a schematic diagram of the structure of the aesthetic discriminator in an image correction model provided by this application. As Figure 6 shown, E1, E2, and E3 represent encoders. Each encoder contains several convolutional layer operations. For example, it can be composed of four consecutive and alternating pooling layers and convolutional layers. Among them, after each group of downsampling layers (pooling layers) and convolutional layers, the output feature size is reduced to half of the original, and the number of channels is increased to twice the original, finally obtaining the aesthetic features.

[0196] Among them, a preset number of model training iterations is set. When the number of iterations reaches the preset number, the training is stopped, and the trained multi-exposure correction model is obtained.

[0197] The image correction model training method provided by the embodiments of the present application designs an exposure normalization module and a multi-scale compensation module. The exposure normalization module maps different exposure features to an exposure-invariant feature space. It roughly aligns different exposure features through instance normalization, and then performs normalized distillation loss on the normalized features to further reduce the exposure impact. The multi-scale compensation module is used to integrate the features not processed by the exposure normalization part in the spatial and channel dimensions, compensating for the problem of loss of image discrimination features during image reconstruction caused by normalization and ensuring the integrity of information. Finally, an adversarial training strategy is introduced into the aesthetic discriminator to learn features that conform to human aesthetic perception from images with normal exposure, thereby generating more harmonious and realistic results, improving the aesthetic visual effect of the images, and achieving the effect of effectively correcting images with abnormal exposure while retaining important data information of the images.

[0198] Figure 7 The following is a schematic structural diagram of an image correction device with aesthetic enhancement provided by the present application, as Figure 7 shown, an image correction device 70 with aesthetic enhancement provided by this embodiment includes:

[0199] An acquisition module 701, configured to acquire an image with abnormal exposure and an initial feature image of the image with abnormal exposure;

[0200] A normalization module 702, configured to perform feature fusion processing on the residual features and the normalized features of the initial feature image to obtain an initial fusion feature;

[0201] A first compensation module 703, configured to obtain first depth feature information, second depth feature information, and a first fusion feature according to the initial fusion feature. The first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information;

[0202] A second compensation module 704, configured to perform self-attention mechanism operation processing on the first depth feature information and the initial feature image to obtain first depth correlation information; and perform self-attention mechanism operation processing on the second depth feature information and the initial feature image to obtain second depth correlation information;

[0203] A third compensation module 705, configured to obtain a target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature, and the second fusion feature is obtained according to the first depth correlation information and the second depth correlation information;

[0204] The feature sampling module 706 is used to perform encoding and decoding processing on the target fusion feature image to obtain a normally exposed image.

[0205] In a possible implementation, the normalization module 702 can also be used to:

[0206] Input the initial feature image into the third convolution unit of the normalization module in the target model for third convolution processing to obtain the residual feature of the initial feature image;

[0207] Input the initial feature image into the normalization unit of the normalization module in the target model to obtain the normalized feature of the initial feature image, where the normalized feature of the initial feature image is obtained by performing third convolution processing on the initial normalized feature, and the initial normalized feature is obtained by performing instance normalization processing on the initial feature image;

[0208] Perform feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain the initial fusion feature.

[0209] In a possible implementation, the first compensation module 703 can also be used to:

[0210] Obtain the first depth feature information and the second depth feature information according to the initial fusion feature;

[0211] Input the first depth feature information into the first self-attention mechanism processing unit of the compensation module in the target model to obtain the first K value and the first V value;

[0212] Input the second depth feature information into the second self-attention mechanism processing unit of the compensation module in the target model to obtain the second K value and the second V value;

[0213] Obtain the first depth fusion information according to the first K value and the second K value;

[0214] Obtain the second depth fusion information according to the first V value and the second V value;

[0215] Input the first depth fusion information and the first depth fusion information into the first fusion unit of the compensation module in the target model for first fusion processing to obtain the first fusion feature.

[0216] In a possible implementation, the first compensation module 703 can also be used to:

[0217] Input the initial fusion feature into the first convolution unit of the compensation module in the target model for first convolution processing to obtain the first depth feature information;

[0218] Input the initial fusion feature into the second convolution unit of the compensation module in the target model for second convolution processing to obtain the second depth feature information.

[0219] In a possible implementation, the second compensation module 704 can also be used to:

[0220] Input the first depth feature information and the initial feature image into the first self-attention mechanism processing unit of the compensation module in the target model for the first self-attention mechanism operation processing to obtain the first depth correlation information. The first depth correlation information is determined according to the first depth feature information and the first correlation weight coefficient, and the first correlation weight coefficient is determined according to the first depth feature information and the initial feature image;

[0221] Input the second depth feature information and the initial feature image into the second self-attention mechanism processing unit of the compensation module in the target model for the second self-attention mechanism operation processing to obtain the second depth correlation information. The second depth correlation information is determined according to the second depth feature information and the second correlation weight coefficient, and the second correlation weight coefficient is determined according to the second depth feature information and the initial feature image.

[0222] In a possible implementation, the third compensation module 705 can also be used to:

[0223] Input the first depth correlation information and the second depth correlation information into the second fusion unit of the compensation module in the target model for the second fusion processing to obtain the target fusion feature;

[0224] Input the first fusion feature, the second fusion feature, and the initial fusion feature into the third fusion unit of the compensation module in the target model for the third fusion processing to obtain the target fusion feature image.

[0225] In a possible implementation, the feature sampling module 706 can also be used to:

[0226] Input the target fusion feature image into the feature extraction and encoding module in the target model for downsampling processing in the feature space to obtain multi-level image data information;

[0227] Input the multi-level image data information into the feature fusion and decoding module in the target model for upsampling processing in the feature space to obtain a normally exposed image.

[0228] In a possible implementation, the feature sampling module 706 can also be used to:

[0229] Obtain an image with abnormal exposure to be trained and a normally exposed alignment image corresponding to the image with abnormal exposure to be trained;

[0230] Input the image with abnormal exposure to be trained into the model to be trained to obtain a target corrected image;

[0231] Train the model to be trained based on the target corrected image and the normally exposed aligned image to obtain the target model, where the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

[0232] In a possible implementation, the feature sampling module 706 can also be used to:

[0233] Determine whether there is a next connected convolutional neural network in the target convolutional neural network, where the target convolutional network is the convolutional neural network in the model to be trained that obtains the target corrected image;

[0234] If there is a next connected convolutional neural network in the target convolutional neural network, then use the target corrected image as the abnormally exposed image to be trained and input it into the next convolutional neural network, and execute the step of inputting the abnormally exposed image to be trained into the model to be trained again to obtain the target corrected image until there is no next connected convolutional neural network in the target convolutional neural network to obtain the exposure corrected image;

[0235] Determine the loss function of the model to be trained based on the exposure corrected image and the normally exposed aligned image, where the loss function of the model to be trained includes Manhattan distance function, cosine distance function, structural similarity distance function, aesthetic feature distance function, and adversarial training loss function;

[0236] Train the model to be trained according to the loss function of the model to be trained to obtain the target model.

[0237] An aesthetic enhancement image correction device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0238] Figure 8 It is a schematic structural diagram of an electronic device provided in this application. As Figure 8 shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. Among them, the processor 801, the memory 802, and the communication component 803 are connected through a bus 804.

[0239] In the specific implementation process, at least one processor 801 executes the computer execution instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0240] The specific implementation process of the processor 801 can refer to the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here in this embodiment.

[0241] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU for short), or other general-purpose processors, digital signal processors (DSP for short), application specific integrated circuits (ASIC for short), etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by the execution of the hardware processor, or implemented by the combination of hardware and software modules in the processor.

[0242] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0243] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0244] This application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0245] This application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the processor executes the computer-executable instructions, the above method is implemented.

[0246] The above-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, a magnetic disk or an optical disc. The readable storage medium can be any available medium accessible by a general-purpose or special-purpose computer.

[0247] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an Application Specific Integrated Circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in a device.

[0248] The division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the couplings or direct couplings or communication connections shown or discussed between each other can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0249] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0250] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0251] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in each embodiment of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0252] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments; and the foregoing storage medium includes: various media such as ROM, RAM, magnetic disk, or optical disk that can store program codes. Therefore, in addition to implementing the system and its various devices provided by the present invention in the form of computer-readable program codes, it is entirely possible to make the system and its various devices provided by the present invention be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions.

[0253] Finally, it should be noted that: after considering the specification and practicing the invention disclosed herein, those skilled in the art will readily think of other embodiments of the present invention. The present invention is intended to cover any variations, uses, or adaptations of the present invention, which follow the general principles of the present invention and include the common general knowledge or conventional technical means in the technical field not disclosed by the present invention. It is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present invention is only limited by the appended claims.

Claims

1. An aesthetic enhancement image correction method, characterized in that, Including: Obtaining an exposure abnormal image and an initial feature image of the exposure abnormal image; Performing feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain an initial fusion feature; According to the initial fusion feature, obtaining first depth feature information, second depth feature information, and a first fusion feature, where the first depth feature information is obtained by performing a first convolution process on the initial fusion feature, the second depth feature information is obtained by performing a second convolution process on the initial fusion feature, the convolution kernel sizes in the first convolution process and the second convolution process are different, and the first fusion feature is determined according to the first depth feature information and the second depth feature information; Performing a self-attention mechanism operation process on the first depth feature information and the initial feature image to obtain first depth correlation information; and performing a self-attention mechanism operation process on the second depth feature information and the initial feature image to obtain second depth correlation information; According to the first fusion feature, a second fusion feature, and the initial fusion feature, obtaining a target fusion feature image, where the second fusion feature is obtained according to the first depth correlation information and the second depth correlation information; Performing an encoding and decoding process on the target fusion feature image to obtain an exposure normal image; The performing feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain an initial fusion feature includes: Inputting the initial feature image into a third convolution unit of a normalization module in a target model to perform a third convolution process to obtain the residual feature of the initial feature image; Inputting the initial feature image into a normalization unit of the normalization module in the target model to obtain the normalized feature of the initial feature image, where the normalized feature of the initial feature image is obtained by performing a third convolution process on an initial normalized feature, and the initial normalized feature is obtained by performing instance normalization processing on the initial feature image; Performing feature fusion processing on the residual feature of the initial feature image and the normalized feature of the initial feature image to obtain an initial fusion feature.

2. The method according to claim 1, characterized in that, The obtaining first depth feature information, second depth feature information, and a first fusion feature according to the initial fusion feature includes: Obtaining first depth feature information and second depth feature information according to the initial fusion feature; Inputting the first depth feature information into a first self-attention mechanism processing unit of a compensation module in a target model to obtain a first K value and a first V value; Inputting the second depth feature information into a second self-attention mechanism processing unit of the compensation module in the target model to obtain a second K value and a second V value; Obtaining first depth fusion information according to the first K value and the second K value; Obtaining second depth fusion information according to the first V value and the second V value; Inputting the first depth fusion information and the first depth fusion information into a first fusion unit of the compensation module in the target model to perform a first fusion process to obtain a first fusion feature.

3. The method according to claim 2, characterized in that, Obtaining the first depth feature information and the second depth feature information according to the initial fusion feature includes: Inputting the initial fusion feature into the first convolutional unit of the compensation module in the target model for the first convolutional process to obtain the first depth feature information; Inputting the initial fusion feature into the second convolutional unit of the compensation module in the target model for the second convolutional process to obtain the second depth feature information.

4. The method according to claim 1, wherein Performing self-attention mechanism operation processing on the first depth feature information and the initial feature image to obtain the first depth correlation information; and performing self-attention mechanism operation processing on the second depth feature information and the initial feature image to obtain the second depth correlation information, including: Inputting the first depth feature information and the initial feature image into the first self-attention mechanism processing unit of the compensation module in the target model for the first self-attention mechanism operation processing to obtain the first depth correlation information, where the first depth correlation information is determined according to the first depth feature information and the first correlation weight coefficient, and the first correlation weight coefficient is determined according to the first depth feature information and the initial feature image; Inputting the second depth feature information and the initial feature image into the second self-attention mechanism processing unit of the compensation module in the target model for the second self-attention mechanism operation processing to obtain the second depth correlation information, where the second depth correlation information is determined according to the second depth feature information and the second correlation weight coefficient, and the second correlation weight coefficient is determined according to the second depth feature information and the initial feature image.

5. The method according to claim 1, wherein Obtaining the target fusion feature image according to the first fusion feature, the second fusion feature, and the initial fusion feature includes: Inputting the first depth correlation information and the second depth correlation information into the second fusion unit of the compensation module in the target model for the second fusion process to obtain the target fusion feature; Inputting the first fusion feature, the second fusion feature, and the initial fusion feature into the third fusion unit of the compensation module in the target model for the third fusion process to obtain the target fusion feature image.

6. The method according to claim 1, characterized in that, Performing encoding and decoding processing on the target fusion feature image to obtain the normally exposed image, including: Inputting the target fusion feature image into the feature extraction and encoding module in the target model for downsampling processing in the feature space to obtain multi-level image data information; Inputting the multi-level image data information into the feature fusion and decoding module in the target model for upsampling processing in the feature space to obtain the normally exposed image.

7. The method according to any one of claims 1 to 6, characterized in that The method further includes: Obtaining the abnormally exposed image to be trained and the normally exposed aligned image corresponding to the abnormally exposed image to be trained; Inputting the abnormally exposed image to be trained into the model to be trained to obtain the target corrected image; Training the model to be trained according to the target corrected image and the normally exposed aligned image to obtain the target model, where the model to be trained is a model obtained by sequentially connecting multiple convolutional neural networks.

8. The method according to claim 7, wherein Training the to-be-trained model by aligning the target corrected image and the normally exposed image as described above includes: Determining whether there is a connected next convolutional neural network in the target convolutional neural network, where the target convolutional neural network is the convolutional neural network in the to-be-trained model that obtains the target corrected image; If there is a connected next convolutional neural network in the target convolutional neural network, using the target corrected image as an image to be trained with abnormal exposure and inputting it into the next convolutional neural network, and then repeating the step of inputting the image to be trained with abnormal exposure into the to-be-trained model to obtain the target corrected image until there is no connected next convolutional neural network in the target convolutional neural network, thereby obtaining an exposure corrected image; Determining a loss function of the to-be-trained model according to the exposure corrected image and the normally exposed image, where the loss function of the to-be-trained model includes a Manhattan distance function, a cosine distance function, a structural similarity distance function, an aesthetic feature distance function, and an adversarial training loss function; Training the to-be-trained model according to the loss function of the to-be-trained model to obtain a target model.

9. An electronic device, characterized in that, Including: A memory and a processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, such that the processor executes the method according to any one of claims 1-8.

Images