Image enhancement method, device, equipment and storage medium

Abstract

The present application discloses an image enhancement method, device, equipment and storage medium, belonging to the field of image processing technology. The method includes: obtaining multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data includes two images with the same image content but different image quality, each group of unpaired sample data includes two images with different image content and image quality, and the multiple groups of unpaired sample data include images in multiple fields; based on the multiple groups of paired sample data and the multiple groups of unpaired sample data, the image enhancement model is trained to obtain the trained image enhancement model; when the image to be enhanced is obtained, the image to be enhanced is enhanced by the trained image enhancement model to obtain the enhanced image. This scheme enables the image enhancement model to generate images with higher fidelity and improves the generalization ability of the image enhancement model.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image enhancement method, device, equipment and storage medium. Background Art

[0002] Due to the limitations of photography technology and photography equipment, the image quality of the images taken by users is poor, and it is necessary to process the captured images through image enhancement technology to improve the image quality of the images. Among them, image enhancement refers to improving the visual effect of the image to make the improved image more in line with people's aesthetics.

[0003] At present, an image with poor image quality can be input into an image enhancement model, which processes the image and outputs an image with higher image quality. The image enhancement model is trained by paired sample data, which includes any low-quality image obtained from the Internet and a high-quality image obtained by professionals through technical means such as PS (Photoshop, a photo editing tool) to process the low-quality image.

[0004] Since the PS process is very time-consuming, the number of paired sample data is small, and the generalization ability of the trained image enhancement model is poor. Summary of the invention

[0005] The embodiments of the present application provide an image enhancement method, apparatus, device and storage medium, which improve the generalization ability of the image enhancement model. The technical solution is as follows:

[0006] In one aspect, an image enhancement method is provided, the method comprising:

[0007] Acquire multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data includes two images with the same image content but different image quality, each group of unpaired sample data includes two images with different image content and image quality, and the multiple groups of unpaired sample data include images in multiple fields; based on the multiple groups of paired sample data and the multiple groups of unpaired sample data, train an image enhancement model to obtain the trained image enhancement model, the paired sample data are used to enable the trained image enhancement model to learn the ability to enhance the image quality of input images, and the unpaired sample data are used to enable the trained image enhancement model to learn the ability to adapt to input images in multiple fields; when an image to be enhanced is acquired, perform image enhancement on the image to be enhanced by using the trained image enhancement model to obtain an enhanced image.

[0008] In one possible implementation, determining the first loss value based on the fifth image and the second image includes: processing the fifth image and the second image to obtain a fourth loss value; processing image features of the fifth image and image features of the second image to obtain a fifth loss value; and weighting the fourth loss value and the fifth loss value to obtain the first loss value.

[0009] In one possible implementation, the processing of the fifth image and the second image to obtain a fourth loss value includes: obtaining the difference in absolute values ​​between the fifth image and the second image; obtaining the difference in Euclidean distances between the fifth image and the second image; determining a first weight for the difference in absolute values ​​and a second weight for the difference in Euclidean distances based on the number of training times of the image enhancement model; and weighted processing of the difference in absolute values ​​and the difference in Euclidean distances based on the first weight and the second weight to obtain the fourth loss value.

[0010] In one aspect, an image enhancement device is provided, the device comprising:

[0011] A sample acquisition module, used to acquire multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data includes two images with the same image content but different image quality, each group of unpaired sample data includes two images with different image content and image quality, and the multiple groups of unpaired sample data include images of multiple fields;

[0012] A training module, used for training the image enhancement model based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model, wherein the paired sample data are used to enable the trained image enhancement model to learn the ability to enhance the image quality of the input image, and the unpaired sample data are used to enable the trained image enhancement model to learn the ability to adapt to input images in multiple fields;

[0013] The processing module is used to perform image enhancement on the image to be enhanced by using the trained image enhancement model to obtain an enhanced image when the image to be enhanced is acquired.

[0014] In a possible implementation, each group of paired sample data includes a first image and a second image, and the image quality of the second image is higher than that of the first image; each group of unpaired sample data includes a third image and a fourth image, and the image quality of the fourth image is higher than that of the third image;

[0015] The training module comprises:

[0016] For any set of paired sample data and any set of unpaired sample data,

[0017] A first processing unit, configured to perform image enhancement on the first image in the paired sample data by using the image enhancement model to obtain a fifth image;

[0018] A second processing unit is used to perform image enhancement on the third image in the unpaired sample data by using the image enhancement model to obtain a sixth image;

[0019] A training unit is used to train the image enhancement model based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image, so as to obtain the trained image enhancement model.

[0020] In a possible implementation, the first processing unit includes:

[0021] an adjusting subunit, configured to adjust the exposure of the first image to obtain a seventh image having an exposure of a first value and an eighth image having an exposure of a second value;

[0022] A first processing subunit is used to perform image enhancement on the seventh image by using the image enhancement model to obtain a ninth image;

[0023] A second processing subunit is used to perform image enhancement on the eighth image by using the image enhancement model to obtain a tenth image;

[0024] The fusion subunit is used to fuse the ninth image and the tenth image through the image enhancement model to obtain the fifth image.

[0025] In one possible implementation, the first processing subunit is used to perform feature extraction on the seventh image through the image enhancement model to obtain a first feature map of the seventh image; perform global information and local information enhancement on the first feature map through the image enhancement model to obtain a second feature map; and decode the second feature map through the image enhancement model to obtain the ninth image.

[0026] In one possible implementation, the fusion subunit is used to perform feature extraction on the second feature map used to generate the ninth image and the third feature map used to generate the tenth image through the image enhancement model to obtain a fourth feature map, where the fourth feature map is used to represent the relative quality of the ninth image and the tenth image in the same area; and based on the fourth feature map, the ninth image and the tenth image are fused through the image enhancement model to obtain the fifth image.

[0027] In a possible implementation, the second processing unit includes:

[0028] a feature extraction subunit, configured to perform feature extraction on the third image by using the image enhancement model to obtain a fifth feature map of the third image;

[0029] an enhancement subunit, configured to enhance the global information and the local information of the fifth feature map by using the image enhancement model to obtain a sixth feature map;

[0030] A decoding subunit is used to decode the sixth feature map through the image enhancement model to obtain the sixth image.

[0031] In one possible implementation, the enhancement subunit is used to perform feature extraction on each row of the fifth feature map through the image enhancement model to obtain a seventh feature map; perform feature extraction on each column of the fifth feature map through the image enhancement model to obtain an eighth feature map; fuse the seventh feature map and the eighth feature map through the image enhancement model to obtain a ninth feature map; perform global information enhancement on the fifth feature map based on the ninth feature map through the image enhancement model to obtain a tenth feature map; perform feature extraction on the tenth feature map through the image enhancement model to obtain an eleventh feature map; perform local information enhancement on the tenth feature map based on the eleventh feature map through the image enhancement model to obtain a twelfth feature map; and fuse the tenth feature map and the twelfth feature map through the image enhancement model to obtain the sixth feature map.

[0032] In one possible implementation, the enhancement subunit is used to perform feature extraction on the ninth feature map through the image enhancement model to obtain a thirteenth feature map, wherein the thirteenth feature map represents global information of the third image; normalize the thirteenth feature map through the image enhancement model to obtain a fourteenth feature map; and perform a dot product of the fourteenth feature map and the fifth feature map through the image enhancement model to obtain the tenth feature map.

[0033] In a possible implementation, the training unit includes:

[0034] A first determining subunit, configured to determine a first loss value based on the fifth image and the second image, wherein the first loss value is used to represent a difference between the fifth image and the second image;

[0035] A second determining subunit, configured to determine a second loss value based on the sixth image and the fourth image, wherein the second loss value is used to represent a quality difference between the sixth image and the fourth image;

[0036] a third determining subunit, configured to determine a third loss value based on the first loss value and the second loss value;

[0037] A training subunit is used to train the image enhancement model based on the third loss value to obtain the trained image enhancement model.

[0038] In one possible implementation, the first determination subunit is used to process the fifth image and the second image to obtain a fourth loss value; process image features of the fifth image and image features of the second image to obtain a fifth loss value; and weight the fourth loss value and the fifth loss value to obtain the first loss value.

[0039] In one possible implementation, the first determination subunit is used to obtain the difference in absolute values ​​between the fifth image and the second image; obtain the difference in Euclidean distances between the fifth image and the second image; determine a first weight for the difference in absolute values ​​and a second weight for the difference in Euclidean distances based on the number of training times of the image enhancement model; and perform weighted processing on the difference in absolute values ​​and the difference in Euclidean distances based on the first weight and the second weight to obtain the fourth loss value.

[0040] In one possible implementation, the second determination subunit is used to determine a sixth loss value based on the image quality information of the sixth image and the image quality information of the fourth image; to capture a target number of partial images from the sixth image and the fourth image, respectively, according to the target size; for each partial image of the sixth image, to determine a seventh loss value based on the image quality information of the partial image and the image quality information of the corresponding partial image in the fourth image; and to weight the sixth loss value and each seventh loss value to obtain the second loss value.

[0041] In one possible implementation, the second determination subunit is used to process the image quality information of the sixth image and the image quality information of the fourth image according to the first loss function of the image enhancement model to determine an eighth loss value; process the image quality information of the sixth image according to the second loss function of the adversarial model to obtain a ninth loss value; and weight the eighth loss value and the ninth loss value to obtain the sixth loss value.

[0042] In a possible implementation, the third determination subunit is used to determine a tenth loss value based on the sixth image and the third image; and weight the first loss value, the second loss value and the tenth loss value to obtain the third loss value.

[0043] On the one hand, a computer device is provided, which includes one or more processors and one or more memories, wherein the one or more memories store at least one program code, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the image enhancement method in any possible implementation manner as described above.

[0044] On the one hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed by the image enhancement method in any possible implementation manner as described above.

[0045] On the one hand, a computer program or a computer program product is provided, wherein the computer program or the computer program product comprises: a computer program code, wherein when the computer program code is executed by a computer, the computer implements the operations performed by the image enhancement method in any possible implementation manner as described above.

[0046] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0047] The image enhancement method, apparatus, device and storage medium provided in the embodiments of the present application train an image enhancement model by using paired sample data and unpaired sample data together. Since paired sample data can provide mapping information between low-quality images and high-quality images, using paired sample data to train the image enhancement model can enable the image enhancement model to learn the ability to enhance the image quality of the input image. Since the low-quality images and high-quality images in the unpaired sample data are not paired, a large amount of unpaired sample data in different fields can be obtained from the Internet. Therefore, using unpaired sample data can enable the image enhancement model to learn the ability to adapt to input images in multiple fields. Thus, by training the image enhancement model with paired sample data and unpaired sample data together, the image enhancement model can generate images with higher fidelity and improve the generalization ability of the image enhancement model. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0049] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0050] Figure 2is a schematic diagram of an image enhancement model provided in an embodiment of the present application;

[0051] Figure 3 is a schematic diagram of another image enhancement model provided in an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of another image enhancement model provided in an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of another image enhancement model provided in an embodiment of the present application;

[0054] Figure 6 is a flow chart of an image enhancement method provided in an embodiment of the present application;

[0055] Figure 7 is a flow chart of an image enhancement method provided in an embodiment of the present application;

[0056] Figure 8 is a processing flow chart of an image fusion layer provided in an embodiment of the present application;

[0057] Fig. 9 is a processing flow chart of global information and local information enhancement provided by an embodiment of the present application;

[0058] Fig.10 is a flow chart of a training image enhancement method provided in an embodiment of the present application;

[0059] Fig.11 is a schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application;

[0060] Fig.12 is a schematic diagram of the structure of another image enhancement device provided in an embodiment of the present application;

[0061] Fig.13 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0062] Fig.14 It is a schematic diagram of the structure of the server provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0064] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image without departing from the scope of this application.

[0065] The terms "at least one", "multiple", "each" and "any" used in this application, at least one includes one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, the multiple images include 3 images, and each refers to each of the 3 images, and any refers to any one of the 3 images, which can be the first, the second or the third.

[0066] The image enhancement method provided in the embodiment of the present application is performed by a computer device. In one possible implementation, the computer device is a terminal, for example, the terminal is any type of terminal such as a desktop computer, a tablet computer or a mobile phone. In another possible implementation, the computer device is a server. For example, the server can be a server, or a server cluster composed of several servers, or a cloud computing service center. In another possible implementation, the computer device includes a terminal and a server.

[0067] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network.

[0068] Optionally, the terminal 101 is any type of terminal such as a desktop computer, a tablet computer or a mobile phone, etc. The server 102 is a server, or a server cluster composed of several servers, or a cloud computing service center.

[0069] The target application provided by the server 102 is installed on the terminal 101, and the terminal 101 can realize functions such as data transmission and message interaction through the target application. Optionally, the target application is an application in the operating system of the terminal 101, or an application provided by a third party. For example, the target application is an image processing application, which has the function of processing images. Of course, the image processing application can also have other functions, such as commenting function, sharing function, etc.

[0070] Optionally, the server 102 obtains multiple groups of paired sample data and multiple groups of unpaired sample data, and trains the image enhancement model based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain a trained image enhancement model. The server 102 sends the trained image enhancement model to the terminal 101. When the terminal 101 obtains the image to be enhanced, it performs image enhancement on the image to be enhanced by using the trained image enhancement model to obtain an enhanced image.

[0071] The image enhancement method provided in the embodiments of the present application can be applied to any image processing scenario.

[0072] For example, it is applied to beauty scenes.

[0073] Users can take photos with their mobile phones. If the image enhancement method provided in the embodiment of the present application is used, the images taken by the users can be enhanced to make the images more in line with people's aesthetic tastes. There is no need for users to manually beautify the images. On the basis of ensuring image quality, the user's operations are reduced and the image processing efficiency is improved.

[0074] It should be noted that the embodiments of the present application only exemplify the image processing scenario by using the beautification scenario, and do not limit the image processing scenario. Optionally, the image processing scenario may also be an image special effects generation scenario, etc.

[0075] It should be noted that the embodiment of the present application provides an image enhancement model, which is used to enhance the input image to improve the image quality of the input image. In a possible implementation, Figure 2 As shown, the image enhancement model includes a feature extraction layer and an image generation layer. The feature extraction layer is used to extract features from the input image to obtain a feature map of the image, and the image generation layer is used to decode the feature map obtained by the feature extraction layer to obtain an enhanced image. Optionally, the feature extraction layer is any convolutional neural network, such as a U-net network. Optionally, the image generation layer is any decoder. The embodiment of the present application does not limit the feature extraction layer and the image generation layer.

[0076] In another possible implementation, Figure 3 As shown, the image enhancement model also includes an attention layer, which is connected to the feature extraction layer and the image generation layer respectively. The attention layer is used to enhance the global information and local information of the feature map obtained by the feature extraction layer to obtain a new feature map. The image generation layer is used to decode the feature map obtained by the attention layer to obtain an enhanced image.

[0077] In another possible implementation, Figure 4As shown, the image enhancement model includes a first feature extraction layer, a second feature extraction layer, a first image generation layer, a second image generation layer and an image fusion layer, wherein the first feature extraction layer is connected to the first image generation layer, the second feature extraction layer is connected to the second image generation layer, and the first image generation layer and the second image generation layer are both connected to the image fusion layer. Optionally, the model parameters of the first feature extraction layer and the second feature extraction layer are consistent, and the parameters of the first image generation layer and the second image generation layer are consistent. That is, the second feature extraction layer can be regarded as the first feature extraction layer obtained by copying, and the second image generation layer can be regarded as the first image generation layer obtained by copying. Optionally, for any low-quality image A, the exposure of the low-quality image A1 is adjusted to obtain an image A2 with a first exposure value and an image A3 with a second exposure value, image A2 is input into the first feature extraction layer, image A3 is input into the second feature extraction layer, the first feature extraction layer is used to extract a feature map B1 of image A2, the feature map B1 is input into the first image generation layer, the first image generation layer is used to decode the feature map B1, and the enhanced image A4 is obtained. The second feature extraction layer is used to extract the feature map B2 of image A3, and the feature map B2 is input to the second image generation layer, which is used to decode the feature map B2 to obtain the enhanced image A5. Image A4 and image A5 are input to the image fusion layer, which is used to fuse image A4 and image A5 to obtain image A6. Image A6 is the final image enhanced by the image enhancement model.

[0078] Optionally, there is an attention layer between the feature extraction layer and the image generation layer. Figure 5 As shown, the image enhancement model also includes a first feature extraction layer, a second feature extraction layer, a first attention layer, a second attention layer, a first image generation layer, a second image generation layer and an image fusion layer. The first attention layer is located between the first feature extraction layer and the first image generation layer, the second attention layer is located between the second feature extraction layer and the second image generation layer, and the first image generation layer and the second image generation layer are both connected to the image fusion layer. The model parameters of the first attention layer are the same as the model parameters of the second attention layer. That is, the second attention layer can be obtained by copying the first attention layer.

[0079] Optionally, for any low-quality image A, adjust the exposure of the low-quality image A1 to obtain an image A2 with a first exposure value and an image A3 with a second exposure value, input image A2 into the first feature extraction layer, input image A3 into the second feature extraction layer, the first feature extraction layer is used to extract a feature map B1 of image A2, input the feature map B1 into the first attention layer, the first attention layer is used to perform global and local information enhancement on the feature map B1, obtain feature map B3, input the feature map B3 into the first image generation layer, the first image generation layer is used to decode the feature map B3, and obtain an enhanced image A7. The second feature extraction layer is used to extract a feature map B2 of image A3, input the feature map B2 into the second attention layer, the second attention layer is used to perform global and local information enhancement on the feature map B2, obtain feature map B4, input the feature map B4 into the second image generation layer, the second image generation layer is used to decode the feature map B4, and obtain an enhanced image A8. Image A7 and image A8 are input to the image fusion layer, and the image fusion layer is used to fuse image A7 and image A8 to obtain image A9. Image A9 is the final image enhanced by the image enhancement model.

[0080] Figure 6 This is a flow chart of an image enhancement method provided in an embodiment of the present application. This embodiment of the present application is illustrated by taking a computer device as an example of an execution subject. The embodiment includes:

[0081] 601. A computer device obtains multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data includes two images with the same image content but different image quality, each group of unpaired sample data includes two images with different image content and image quality, and the multiple groups of unpaired sample data include images of multiple fields.

[0082] In an embodiment of the present application, the image quality of the two images included in the paired sample data is different, and one image is an image with poor image quality that does not conform to people's aesthetics. For example, an overexposed image, an image taken in dim sunlight, etc. The other image is an image with high image quality that conforms to people's aesthetics. For example, an image with moderate exposure, a clear image, an image with moderate color, etc. The image content of the two images in the paired sample data is the same, and it is difficult to find two images with the same image content but different image quality on the Internet. Therefore, in some cases, some low-quality images can be obtained, and professionals can process the low-quality images through technical means such as PS to obtain corresponding high-quality images, and use the low-quality images and the obtained high-quality images as a set of paired sample data. In some cases, some high-quality images can also be obtained, and professionals can process the high-quality images through technical means such as PS to obtain corresponding low-quality images, and use the high-quality images and the obtained low-quality images as a set of paired sample data.

[0083] The image quality and image content of the two images included in the unpaired sample data are different. Therefore, the two images included in the unpaired sample data can be any two images with different image quality. That is, two images with different image quality can be obtained from the Internet at will to form the unpaired sample data. It should be noted that in order to ensure the training effect, the image with relatively higher image quality in the unpaired sample data needs to meet people's aesthetic taste and reach a certain quality level.

[0084] 602. The computer device trains the image enhancement model based on multiple groups of paired sample data and multiple groups of unpaired sample data to obtain a trained image enhancement model, wherein the paired sample data is used to enable the trained image enhancement model to learn the ability to enhance the image quality of the input image, and the unpaired sample data is used to enable the trained image enhancement model to learn the ability to adapt to input images in multiple fields.

[0085] Since the image contents of the low-quality image and the high-quality image in the paired sample data are the same, there is mapping information between the pixels of the low-quality image and the high-quality image. Therefore, using paired sample data to train the image enhancement model enables the image enhancement model to learn the ability to enhance the image quality of the input image.

[0086] Since the contents of low-quality images and high-quality images in unpaired sample data are different, any low-quality image and any high-quality image can be obtained on the Internet to form unpaired sample data. Therefore, a large amount of unpaired sample data can be obtained from the Internet, which reduces the cost of obtaining sample data, increases the number of sample data for training image enhancement models, and thereby improves the accuracy of image enhancement models.

[0087] It should be noted that the multiple groups of unpaired sample data include images from multiple fields. Optionally, images from multiple fields refer to: the image contents of the multiple images belong to multiple fields. For example, the multiple images include person images, landscape images, building images, etc. Optionally, images from multiple fields refer to: images from different fields. For example, images from the field of beauty pictures; images from the field of hotel recommendations; images from the field of scenic spot recommendations, etc.

[0088] Since the multiple groups of unpaired sample data include images from multiple fields, using the multiple groups of unpaired sample data to train the image enhancement model can enable the image enhancement model to learn the ability to adapt to input images from multiple fields.

[0089] In an embodiment of the present application, by training an image enhancement model with paired sample data and unpaired sample data together, the image enhancement model can generate images with higher fidelity and improve the generalization ability of the image enhancement model.

[0090] 603. When the computer device obtains the image to be enhanced, it performs image enhancement on the image to be enhanced by using the trained image enhancement model to obtain an enhanced image.

[0091] In the embodiment of the present application, after the image enhancement model is trained, the image enhancement model can also be used to enhance the image to be enhanced. In the embodiment of the present application, the image to be enhanced can be obtained by the image enhancement application, or by the user inputting the image to be enhanced. The embodiment of the present application does not limit this.

[0092] The image enhancement method provided in the embodiment of the present application trains an image enhancement model by using paired sample data and unpaired sample data together. Since paired sample data can provide mapping information between low-quality images and high-quality images, using paired sample data to train the image enhancement model can enable the image enhancement model to learn the ability to enhance the image quality of the input image. Since the low-quality images and high-quality images in the unpaired sample data are not paired, a large amount of unpaired sample data in different fields can be obtained from the Internet. Therefore, using unpaired sample data can enable the image enhancement model to learn the ability to adapt to input images in multiple fields. Thus, by training the image enhancement model with paired sample data and unpaired sample data together, the image enhancement model can generate images with higher fidelity and improve the generalization ability of the image enhancement model.

[0093] Figure 7 This is a flow chart of an image enhancement method provided in an embodiment of the present application. This embodiment of the present application is illustrated by taking a computer device as an example of an execution subject. The embodiment includes:

[0094] 701. A computer device obtains multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data includes a first image and a second image with the same image content, and the image quality of the second image is higher than that of the first image, and each group of unpaired sample data includes a third image and a fourth image with different image content, and the image quality of the fourth image is higher than that of the third image.

[0095] It should be noted that step 701 is the same as step 601 and will not be described in detail here.

[0096] It should be noted that in the embodiment of the present application, after the computer device obtains multiple groups of paired sample data and multiple groups of unpaired sample data, it can train the image enhancement model based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model. In the embodiment of the present application, only steps 702 to 704 are taken as an example to illustrate the process of "training the image enhancement model based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model", and the process is not limited. In another embodiment, the image enhancement model is trained based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model, including: first training the image enhancement model based on the multiple groups of paired sample data to obtain the trained first image enhancement model; then continuing to train the first image enhancement model based on the multiple groups of unpaired sample data to obtain the trained second image enhancement model. The second image enhancement model is an image enhancement model that has been trained and put into use. In another embodiment, the image enhancement model is trained based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain a trained image enhancement model, including: firstly training the image enhancement model based on the multiple groups of unpaired sample data to obtain a trained third image enhancement model; and then continuing to train the third image enhancement model based on the multiple groups of unpaired sample data to obtain a trained fourth image enhancement model. The fourth image enhancement model is an image enhancement model that has been trained and put into use.

[0097] 702. For any group of paired sample data and any group of unpaired sample data, the computer device performs image enhancement on a first image in the paired sample data by using an image enhancement model to obtain a fifth image.

[0098] In the embodiment of the present application, the image enhancement model processes the paired sample data and the unpaired sample data respectively. The embodiment of the present application takes step 702 as an example to exemplarily illustrate the process of processing the paired sample data by the image enhancement model.

[0099] By using the image enhancement model, the first image in the paired sample data is enhanced. It is necessary to extract features from the first image to obtain a feature map of the first image; that is, the first image is encoded to obtain a feature map of the first image. The obtained feature map is then decoded to obtain the first image after image enhancement. In one possible implementation, the image enhancement model includes a feature extraction layer and an image generation layer. The computer device performs image enhancement on the first image in the paired sample data through the image enhancement model to obtain a fifth image, including: the computer device extracts features from the first image in the paired sample data through the feature extraction layer to obtain a feature map of the first image; the obtained feature map is decoded through the image generation layer to obtain a fifth image, and the fifth image is the first image after image enhancement.

[0100] It should be noted that the images generated by the image enhancement methods provided in the related art are often overexposed or underexposed, and the lost details of the overexposed or underexposed areas cannot be effectively restored. In order to solve this problem, the embodiment of the present application provides a multi-exposure fusion strategy, which adaptively fuses multiple images with different exposure levels into a globally good image to generate a satisfactory visual effect.

[0101] It should be noted that the multiple images with different exposure levels can be any number of images with different exposure levels. For example, 2 images with different exposure levels; another example, 3 images with different exposure levels; another example, 5 images with different exposure levels, etc. The embodiment of the present application does not limit the number of the multiple images with different exposure levels.

[0102] The embodiment of the present application only takes two images with different exposure levels as an example to illustrate the strategy of multi-exposure fusion. In one possible implementation, the first image in the paired sample data is enhanced by the image enhancement model to obtain a fifth image, including: adjusting the exposure of the first image to obtain a seventh image with an exposure of the first value and an eighth image with an exposure of the second value; performing image enhancement on the seventh image by the image enhancement model to obtain a ninth image; performing image enhancement on the eighth image by the image enhancement model to obtain a tenth image; and fusing the ninth image and the tenth image by the image enhancement model to obtain the fifth image.

[0103] The first value and the second value may be any two different values ​​between 0 and 1. For example, the first value is 0.5, and the second value is 0.8. Alternatively, the first value and the second value may also be values ​​set by a technician. For example, the first value is 0.5, and the second value is 1. Alternatively, the first value is the optimal exposure of the first image, and the second value is the initial exposure of the first image. The first value and the second value are not limited in the embodiments of the present application.

[0104] The embodiment of the present application can use any method to adjust the exposure of the first image. The embodiment of the present application only takes the Retinex-based method to adjust the exposure of the first image as an example to exemplify the method of adjusting the exposure. In one possible implementation, the computer device adjusts the exposure of the first image to obtain a seventh image with an exposure of a first value and an eighth image with an exposure of a second value, including: for each pixel in the first image, obtaining the maximum value of the three channels RGB (Red, Green, Blue) of the pixel as the illumination component, and adjusting the illumination component to obtain the seventh image with an exposure of the first value and the eighth image with an exposure of the second value.

[0105] For example, given an image I scr ∈R H*W*3 , the maximum value of the three channels of RGB for each pixel is taken as the illumination component L. L=max(I pc ), Then, according to the following formula, the obtained illumination component is used to generate images I with exposure 1 and exposure 0.5 scr0 and I scr1 .in, I scr0 =I scr .

[0106] In one possible implementation, the image enhancement model includes two identical feature extraction layers and two identical image generation layers, and the seventh image with an exposure of a first value and the eighth image with an exposure of a second value are processed respectively by the two identical feature extraction layers and the two identical image generation layers. Optionally, the image enhancement model includes a first feature extraction layer, a second feature extraction layer, a first image generation layer, a second image generation layer and an image fusion layer. The model parameters of the first feature extraction layer are the same as the model parameters of the second feature extraction layer, and the second feature extraction layer can be obtained by copying the first feature extraction layer. The model parameters of the first image generation layer are the same as the model parameters of the second image generation layer. The second image generation layer can be obtained by copying the first image generation layer. The computer device adjusts the exposure of the first image to obtain a seventh image with an exposure of the first value and an eighth image with an exposure of the second value, and then inputs the seventh image into the first feature extraction layer, performs feature extraction on the seventh image through the first feature extraction layer, and obtains a first feature map of the seventh image; inputs the first feature map into the first image generation layer, decodes the first feature map through the first image generation layer, and obtains a ninth image; inputs the eighth image into the second feature extraction layer, performs feature extraction on the eighth image through the second feature extraction layer, and obtains a fifteenth feature map of the eighth image; inputs the fifteenth feature map into the second image generation layer, decodes the fifteenth feature map through the second image generation layer, and obtains a tenth image; inputs the ninth image and the tenth image into the image fusion layer, and fuses the ninth image and the tenth image through the image fusion layer to obtain a fifth image.

[0107] It should be noted that the embodiment of the present application only takes the image enhancement model including the first feature extraction layer, the second feature extraction layer, the first image generation layer and the second image generation layer as an example to exemplify the process of the image enhancement model processing multiple images with different exposures, and does not limit the processing process of the image enhancement model. In another embodiment, the image enhancement model includes a feature extraction layer, an image generation layer and an image fusion layer. The computer device sequentially enhances multiple images with different exposures through the feature extraction layer and the image generation layer to obtain multiple corrected images, and fuses the multiple corrected images through the image fusion layer.

[0108] In addition, the purpose of the multi-exposure fusion strategy provided in the embodiment of the present application is to learn the optimal exposure. Therefore, when the image enhancement model fuses the ninth image and the tenth image, it needs to learn to retain the content with better exposure and discard the content with poorer exposure. In one possible implementation, when the computer device fuses the ninth image and the tenth image through the image enhancement model, it first determines the relative quality of each area in the ninth image and the tenth image, and fuses them according to the relative quality of each area. The computer device fuses the ninth image and the tenth image through the image enhancement model to obtain the fifth image, including: extracting features from the second feature map used to generate the ninth image and the third feature map used to generate the tenth image through the image enhancement model to obtain a fourth feature map, which is used to represent the relative quality of the ninth image and the tenth image in the same area; fusing the ninth image and the tenth image based on the fourth feature map through the image enhancement model to obtain the fifth image.

[0109] The ninth image is obtained by decoding the second feature map by the image enhancement model, and the tenth feature map is obtained by decoding the third feature map by the image enhancement model. The embodiment of the present application does not limit the method for obtaining the second feature map and the third feature map.

[0110] Optionally, the image enhancement model includes an image fusion layer, and the computer device extracts features from the second feature map used to generate the ninth image and the third feature map used to generate the tenth image through the image fusion layer to obtain a fourth feature map, and the fourth feature map is used to represent the relative quality of the ninth image and the tenth image in the same area; through the image fusion layer, based on the fourth feature map, the ninth image and the tenth image are fused to obtain the fifth image.

[0111] Optionally, feature extraction is performed on the second feature map used to generate the ninth image and the third feature map used to generate the tenth image to obtain a fourth feature map, including: splicing the second feature map and the third feature map to obtain a sixteenth feature map, and performing feature extraction on the sixteenth feature map to obtain a fourth feature map.

[0112] Figure 8 An image fusion layer under a multi-exposure fusion strategy is exemplarily provided, such as Figure 8 As shown, F scr0 is the second feature map, F scr1 is the third characteristic graph, I gen0 For the ninth image, I gen1 is the tenth image. First, the feature map F scr0 and F scr1 After concatenation, the concatenated feature map undergoes a 3×3 convolution, a 1×1 convolution, and a sigmoid (activation function) activation to obtain the fourth feature map. in, Represents any real number, H represents the height of the feature map, W represents the width of the feature map, and 3 represents that the feature map is a 3-channel feature map. fuse This is the fifth image.

[0113] Among them, I fuse =I gen0 *α+I gen1 *(1-α).

[0114] Among them, the feature map F scr0 and F scr1 is a three-channel feature map, and F scr0 and F scr1 After concatenation, a six-channel feature map can be obtained. Each value in the fourth feature map is any value between 0 and 1. The 3×3 convolution and 1×1 convolution are used to determine the relative quality of the ninth image and the tenth image in the same area.

[0115] It should be noted that the images generated by the image enhancement methods provided in the related art often have artifacts and poor visual consistency. In view of the fact that the images generated in the existing scheme have artifacts and unclear details, the embodiment of the present application also introduces a global-local context attention module to eliminate the artifacts and unclear details in the generated images, so that the generated images have better visual consistency. In one possible implementation, the computer device extracts features from the seventh image through an image enhancement model to obtain a first feature map of the seventh image; enhances the first feature map with global information and local information through the image enhancement model to obtain a second feature map; and decodes the second feature map through the image enhancement model to obtain a ninth image. The computer device extracts features from the eighth image through the image enhancement model to obtain the fifteenth feature map of the eighth image; enhances the sixteenth feature map with global information and local information through the image enhancement model to obtain a third feature map; and decodes the third feature map through the image enhancement model to obtain a tenth image.

[0116] Among them, the global information and local information enhancement of the first feature map or the sixteenth feature map can be based on the first feature map and the sixteenth feature map, superimposing global features and local features; or the global features and local features in the first feature map and the sixteenth feature map are sequentially enhanced. The embodiment of the present application does not limit the method of enhancing global information and local information.

[0117] In one possible implementation, vertical and horizontal strip pooling is used on the computer device to expand the receptive field and obtain global information from the feature map. Since only considering global information and ignoring local information will cause the feature map to be over-smoothed, the color of the generated image is not realistic enough and the details are not prominent, the computer device will pay attention to local information after considering global information and can automatically select enhanced pixels to supplement the over-smoothed features, so that the generated human image has clear details, realistic colors, and better visual effects.

[0118] The computer device first enhances the global information in the fifth feature map, and then enhances the local information. The computer device enhances the global information and local information of the first feature map through the image enhancement model to obtain the second feature map, including: extracting features from each row of the first feature map through the image enhancement model to obtain the seventeenth feature map; extracting features from each column of the first feature map through the image enhancement model to obtain the eighteenth feature map; fusing the seventeenth feature map and the eighteenth feature map through the image enhancement model to obtain the nineteenth feature map; enhancing the global information of the first feature map based on the nineteenth feature map through the image enhancement model to obtain the twentieth feature map; extracting features from the twentieth feature map through the image enhancement model to obtain the twenty-first feature map; enhancing the local information of the twentieth feature map based on the twenty-first feature map through the image enhancement model to obtain the twenty-second feature map; fusing the twentieth feature map and the twenty-second feature map through the image enhancement model to obtain the second feature map.

[0119] It should be noted that after feature extraction is performed on each column and each row of the first feature map, the dimension of the feature map will be changed. When the feature maps after feature extraction are fused, the feature maps obtained after feature extraction can be expanded into feature maps of the same dimension and then superimposed.

[0120] For example, Fig. 9 As shown, the first feature map is F x , Among them, C represents the number of channels of the feature map. The first feature map F x Input the horizontal and vertical strip pools to obtain the seventeenth feature map and the eighteenth characteristic graph The dimensionality of the seventeenth and eighteenth feature maps is expanded to obtain and Will and Add together to get the nineteenth feature map Based on the nineteenth characteristic map For the first feature map F x Perform global information enhancement to obtain the twentieth feature map F z .

[0121] Among them, based on the nineteenth feature map, the first feature map is enhanced for global information to obtain the twentieth feature map: the nineteenth feature map is superimposed on the first feature map to obtain the twentieth feature map; it can also be feature extraction of the nineteenth feature map to obtain the twenty-third feature map, superimposing the twenty-third feature map with the first feature map to obtain the twentieth feature map; it can also be feature extraction of the nineteenth feature map to obtain the twenty-third feature map, the twenty-third feature map represents the global information of the seventh image, the twenty-third feature map is normalized to obtain the twenty-fourth feature map, the twenty-fourth feature map is dot-multiplied with the first feature map to obtain the twentieth feature map. In the embodiment of the present application, superimposing two feature maps means: the values ​​of the same dimension in the two feature maps are multiplied.

[0122] For example, Fig. 9 As shown, for the nineteenth feature graph F y Perform a 1×1 convolution and a sigmoid (activation function) activation to obtain the 24th feature map. Among them, the 1×1 convolution is performed on the one hand to obtain global information, and on the other hand to limit the dimension of the feature map. The Sigmoid activation function can limit the value of the feature map to the range of 0 to 1, so that the 24th feature map can be used as a weight, and the dot multiplication of the 24th feature map and the first feature map can enhance the global information in the first feature map.

[0123] In an embodiment of the present application, any method can be used to expand the dimension of the feature map. Optionally, the feature map is expanded in dimension by copying. For example, the width of the first feature map is 10, the length is 10, and it is a feature map of 3 channels. After the first feature map is input into the horizontal and vertical bar pools, the seventeenth feature map and the eighteenth feature map are obtained respectively. Among them, the width of the seventeenth feature map is 10, the length is 1, and the number of channels is 3. The width of the eighteenth feature map is 1, the length is 10, and the number of passes is 3. Expanding the dimension of the seventeenth feature map means copying the values ​​of each column and using the copied values ​​as the values ​​of the 2nd to 10th columns in the feature map. Expanding the dimension of the eighteenth feature map means copying the values ​​of each row and using the copied values ​​as the values ​​of the 2nd to 10th rows in the feature map.

[0124] After global information enhancement, local information enhancement is also performed. The embodiment of the present application exemplifies the method of local information enhancement. In one possible implementation, after obtaining the twentieth feature map after global enhancement, local features can also be obtained from the twentieth feature map to enhance the local features in the twentieth feature map. Optionally, the computer device performs feature extraction on the twentieth feature map through an image enhancement model to obtain a twenty-first feature map; through the image enhancement model, based on the twenty-first feature map, the twentieth feature map is enhanced locally to obtain a twenty-second feature map; through the image enhancement model, the twentieth feature map and the twenty-second feature map are fused to obtain a second feature map.

[0125] Among them, the computer device can perform feature extraction on the twentieth feature map through a 3×3 convolution, thereby extracting local features in the twentieth feature map; the process of performing local information enhancement on the twentieth feature map based on the local features in the twentieth feature map is the same as the process of performing global information enhancement on the first feature map based on the nineteenth feature map.

[0126] For example, a 1×1 convolution and a sigmoid (activation function) activation are performed on the 21st feature map to obtain the spatial attention map Perform dot multiplication of the 21st feature map and the spatial attention map to obtain the 22nd feature map; fuse the 20th feature map and the 22nd feature map to obtain the second feature map.

[0127] Optionally, fusing the twentieth feature map obtained after global information enhancement with the twenty-second feature map after local information enhancement means: superimposing the twentieth feature map with the twenty-second feature map. It should be noted that the embodiment of the present application only takes superposition as an example to exemplify the fusion of feature maps, and does not limit the fusion process. In another embodiment, the twentieth feature map is fused with the twenty-second feature map, and feature extraction can also be performed on the twentieth feature map and the twenty-second feature map to obtain a second feature map.

[0128] It should be noted that the embodiment of the present application is only to take the local information enhancement based on the twentieth feature map as an example to exemplify the local information enhancement process. In another embodiment, the computer device performs local information enhancement based on the first feature map. In one possible implementation, the computer device performs global information and local information enhancement on the first feature map through an image enhancement model to obtain a second feature map, including: extracting features from each row of the first feature map through the image enhancement model to obtain a seventeenth feature map; extracting features from each column of the first feature map through the image enhancement model to obtain an eighteenth feature map; fusing the seventeenth feature map and the eighteenth feature map through the image enhancement model to obtain a nineteenth feature map; performing global information enhancement on the first feature map based on the nineteenth feature map through the image enhancement model to obtain a twentieth feature map; extracting features from the first feature map through the image enhancement model to obtain a twenty-fifth feature map; performing local information enhancement on the first feature map based on the twenty-fifth feature map through the image enhancement model to obtain a twenty-sixth feature map; fusing the twentieth feature map and the twenty-sixth feature map through the image enhancement model to obtain a second feature map.

[0129] It should be noted that, in the embodiment of the present application, the processing method for the seventh image is the same as the processing method for the eighth image. Therefore, the processing method for the eighth image will no longer be described one by one in the embodiment of the present application.

[0130] 703. The computer device performs image enhancement on the third image in the unpaired sample data through an image enhancement model to obtain a sixth image.

[0131] The unpaired sample data includes a third image and a fourth image, the third image is a low-quality image in the unpaired sample data, and the fourth image is a high-quality image in the unpaired sample data.

[0132] In a possible implementation, the computer device generates the sixth image by encoding and decoding the third image. Optionally, the computer device performs image enhancement on the third image through an image enhancement model to obtain the sixth image, including: extracting features from the third image through the image enhancement model to obtain a fifth feature map of the third image; and decoding the fifth feature map through the image enhancement model to obtain the sixth image.

[0133] It should be noted that the images generated by the image enhancement methods provided in the related art often have artifacts and poor visual consistency. In view of the fact that the images generated in the existing solutions have artifacts and unclear details, the embodiments of the present application also introduce a global-local context attention module to eliminate the artifacts and unclear details in the generated images, so that the generated images have better visual consistency. In one possible implementation, a computer device performs image enhancement on a third image in unpaired sample data through an image enhancement model to obtain a sixth image, including: extracting features from the third image through the image enhancement model to obtain a fifth feature map of the third image; enhancing the fifth feature map with global information and local information through the image enhancement model to obtain a sixth feature map; decoding the sixth feature map through the image enhancement model to obtain a sixth image.

[0134] In the embodiment of the present application, the fifth feature map is enhanced with global information and local information by first performing global information enhancement on the fifth feature map and then performing local information enhancement on the result of global information enhancement; or the fifth feature map is enhanced with global information and local information respectively, and then the result of global information enhancement and the result of local information enhancement are fused. It should be noted that the embodiment of the present application is only an exemplary description of the method of enhancing global information and local information, and does not limit it.

[0135] In one possible implementation, the computer device first performs global information enhancement on the fifth feature map, and then performs local information enhancement on the result of the global information enhancement. Optionally, the computer device performs global information and local information enhancement on the fifth feature map through an image enhancement model to obtain a sixth feature map, including: extracting features from each row of the fifth feature map through the image enhancement model to obtain a seventh feature map; extracting features from each column of the fifth feature map through the image enhancement model to obtain an eighth feature map; fusing the seventh feature map and the eighth feature map through the image enhancement model to obtain a ninth feature map; performing global information enhancement on the fifth feature map based on the ninth feature map through the image enhancement model to obtain a tenth feature map; performing feature extraction on the tenth feature map through the image enhancement model to obtain an eleventh feature map; performing local information enhancement on the tenth feature map based on the eleventh feature map through the image enhancement model to obtain a twelfth feature map; fusing the tenth feature map and the twelfth feature map through the image enhancement model to obtain a sixth feature map.

[0136] Optionally, the computer device performs global information enhancement on the fifth feature map based on the ninth feature map through an image enhancement model to obtain a tenth feature map, including: performing feature extraction on the ninth feature map through the image enhancement model to obtain a thirteenth feature map, wherein the thirteenth feature map represents global information of the third image; normalizing the thirteenth feature map through the image enhancement model to obtain a fourteenth feature map; and performing a dot product of the fourteenth feature map and the fifth feature map through the image enhancement model to obtain a tenth feature map.

[0137] It should be noted that the manner of enhancing the global information and local information of the feature map in step 703 is the same as the manner of enhancing the global information and local information of the feature map in step 702, and will not be described in detail here.

[0138] 704. The computer device trains the image enhancement model based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image to obtain a trained image enhancement model.

[0139] Since the image content of the first image and the second image is the same, the second image can be used as the supervision result of the fifth image, so that the image enhancement model can be trained based on the difference between the fifth image and the second image. The image content of the third image and the fourth image is different, so only the quality difference between the sixth image and the fourth image needs to be considered, and the image enhancement model is trained based on the quality difference between the sixth image and the fourth image, so that the quality of the image processed by the image enhancement model approaches the image quality of the fourth image, thereby generating a higher quality image.

[0140] Among them, training the image enhancement model based on the difference between the fifth image and the second image is a supervised training, and training the image enhancement model based on the quality difference between the sixth image and the fourth image is an unsupervised training. That is to say, the embodiment of the present application actually provides a semi-supervised training strategy.

[0141] It should be noted that the computer device can first use paired sample data to train the image enhancement model, and then use unpaired sample data for training; it can also first use unpaired sample data to train the image enhancement model, and then use paired sample data for training; it can also use paired sample data and unpaired sample data for training at the same time.

[0142] In a possible implementation, the computer device first trains the image enhancement model using paired sample data, and then trains using unpaired sample data. The computer device trains the image enhancement model based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image, to obtain the trained image enhancement, including: determining a first loss value based on the fifth image and the second image, the first loss value is used to represent the difference between the fifth image and the second image; training the image enhancement model based on the first loss value to obtain the trained image enhancement model; determining a second loss value based on the sixth image and the fourth image, the second loss value is used to represent the quality difference between the sixth image and the fourth image; and continuing to train the trained image enhancement model based on the second loss value to obtain the trained image enhancement model.

[0143] In another possible implementation, the computer device first trains the image enhancement model using unpaired sample data, and then trains using paired sample data. The computer device trains the image enhancement model based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image, to obtain the trained image enhancement, including: determining a second loss value based on the sixth image and the fourth image, the second loss value being used to represent the quality difference between the sixth image and the fourth image; training the image enhancement model based on the second loss value to obtain the trained image enhancement model; determining a first loss value based on the fifth image and the second image, the first loss value being used to represent the difference between the fifth image and the second image; and continuing to train the trained image enhancement model based on the first loss value to obtain the trained image enhancement model.

[0144] In another possible implementation, the computer device uses paired sample data and unpaired sample data for training at the same time. The computer device trains the image enhancement model based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image, to obtain the trained image enhancement, including: determining a first loss value based on the fifth image and the second image, the first loss value is used to represent the difference between the fifth image and the second image; determining a second loss value based on the sixth image and the fourth image, the second loss value is used to represent the quality difference between the sixth image and the fourth image; determining a third loss value based on the first loss value and the second loss value; and training the image enhancement model based on the third loss value to obtain the trained image enhancement model.

[0145] Among them, when the computer device determines the first loss value based on the fifth image and the second image, it can be determined based on the image difference between the two images, or based on the difference between the image features of the two images, or based on the image difference between the two images and the difference between the image features of the two images. The embodiment of the present application does not limit this.

[0146] In one possible implementation, the computer device determines the first loss value based on the image difference between the fifth image and the second image and the difference between the image features of the two images. Optionally, the computer device determines the first loss value based on the fifth image and the second image, including: processing the fifth image and the second image to obtain a fourth loss value; processing the image feature of the fifth image and the image feature of the second image to obtain a fifth loss value; and weighting the fourth loss value and the fifth loss value to obtain the first loss value.

[0147] When obtaining the fourth loss value and the fifth loss value, the computer device may use any loss function for calculation, which is not limited in the present embodiment. Moreover, the loss functions used to obtain the fourth loss value and the fifth loss value may be the same or different.

[0148] The embodiment of the present application provides a specific way to obtain the fourth loss value. In order to train the image enhancement model faster and better, considering that the absolute value gradient of the fifth image and the second image is large, it can be faster to train the image enhancement model by the absolute value difference between the fifth image and the second image in the early stage. However, the absolute value difference between the fifth image and the second image is a symmetrical straight line. It is difficult to find the optimal solution by training the image enhancement model by the absolute value difference between the fifth image and the second image. Therefore, the present application also considers the difference in the Euclidean distance between the fifth image and the second image. In this way, when the computer device trains the image enhancement model in the early stage, it mainly focuses on the absolute value difference between the fifth image and the second image, and when training the image enhancement model in the later stage, it pays more attention to the difference in the Euclidean distance between the fifth image and the second image.

[0149] In one possible implementation, a computer device processes a fifth image and a second image to obtain a fourth loss value, including: obtaining the difference in absolute values ​​between the fifth image and the second image; obtaining the difference in Euclidean distances between the fifth image and the second image; determining a first weight for the difference in absolute values ​​and a second weight for the difference in Euclidean distances based on the number of training times of the image enhancement model; and weighted processing the difference in absolute values ​​and the difference in Euclidean distances based on the first weight and the second weight to obtain a fourth loss value.

[0150] The first weight of the absolute value difference decreases as the number of training times increases, and the second weight of the Euclidean distance difference increases as the number of training times increases.

[0151] For example, the fourth loss value is The fifth loss value is

[0152]

[0153]

[0154] Among them, epoch is the number of training times of the image enhancement model; I fuse is the fifth image, I gt is the second image; ||I fuse -I gt ||1 represents the absolute value difference between the fifth image and the second image; ||I fuse -I gt ||2 represents the difference between the Euclidean distance of the fifth image and the second image. fuse ) represents the image feature of the fifth image; φ(I gt ) represents the image feature of the second image; ||φ(I fuse )-φ(I gt )||1 represents the difference in absolute value between the image feature of the fifth image and the image feature of the second image.

[0155] In the embodiment of the present application, in the unsupervised learning part, an adversarial network can be used for training. In one possible implementation, the computer device determines the second loss value based on the image quality information of the sixth image and the image quality information of the fourth image, including: processing the image quality information of the sixth image and the image quality information of the fourth image according to the first loss function of the image enhancement model to obtain an eighth loss value; processing the image quality information of the sixth image according to the second loss function of the adversarial model to obtain a ninth loss value; and weighting the eighth loss value and the ninth loss value to obtain a sixth loss value.

[0156] In an embodiment of the present application, the sixth image is input into an adversarial network and the image enhancement model is trained using the adversarial network so that the image processed by the image enhancement model can "fool" the adversarial network, causing the adversarial network to believe that the image is a high-quality image.

[0157] In the embodiment of the present application, in the unsupervised learning part, in order to ensure the accuracy of training, the loss calculation can be performed on the global and local sixth image respectively. In one possible implementation, the embodiment of the present application constructs a global discriminator and a local discriminator to ensure that the generated image is of high quality in whole and in part. Optionally, the computer device determines the second loss value based on the sixth image and the fourth image, including: determining the sixth loss value according to the image quality information of the sixth image and the image quality information of the fourth image; intercepting the target number of local images from the sixth image and the fourth image respectively according to the target size; for each local image of the sixth image, determining the seventh loss value according to the image quality information of the local image and the image quality information of the corresponding local image in the fourth image; weighting the sixth loss value and the seventh loss value to obtain the second loss value. Among them, the target number can be any positive integer, and the embodiment of the present application does not limit the target number.

[0158] It should be noted that the adversarial model training method and the global-local discriminator training method provided in the embodiments of the present application can be selected either alone or in combination. In a possible implementation, the computer device determines the second loss value based on the image quality information of the sixth image and the image quality information of the fourth image, including: processing the image quality information of the sixth image and the image quality information of the fourth image according to the first loss function of the image enhancement model to obtain an eighth loss value; processing the image quality information of the sixth image according to the second loss function of the adversarial model to obtain a ninth loss value; weighting the eighth loss value and the ninth loss value to obtain a sixth loss value; intercepting a target number of local images from the sixth image and the fourth image respectively according to the target size; for each local image of the sixth image, processing the image quality information of the local image and the image quality of the corresponding local image in the fourth image according to the first loss function of the image enhancement model to obtain an eleventh loss value; for each local image of the sixth image, processing the image quality information of the local image according to the second loss function of the adversarial model to obtain a twelfth loss value; weighting the eleventh loss value and the twelfth loss value corresponding to each local image to obtain a seventh loss value corresponding to each local image; weighting the sixth loss value and each seventh loss value to obtain a second loss value.

[0159] For example, the eighth loss value is The ninth loss value is

[0160] The first loss function is:

[0161] The second loss function is:

[0162] Where D represents the discriminator, represents the sample distribution information of the fourth image, represents the sample distribution information of the sixth image, x r represents the fourth image, x f Indicates the sixth image. It should be noted that in the embodiment of the present application, images are distributed according to the image quality. If two images are in the same distribution, the image quality of the two images is similar; if the two images are in different distributions, the image quality of the two images is quite different. The image quality information of the image can be indicated by the sample distribution information of the image.

[0163] The computer device determines the loss value based on the local image in the same manner as the computer device determines the loss value based on the global image. The twelfth loss value is

[0164] in,

[0165]

[0166] Where D represents the discriminator, Representing sample distribution information of a local image in the fourth image; represents the sample distribution information of the local image in the sixth image, x r represents a local image in the fourth image; x f represents a partial image in the sixth image.

[0167] Since the third image and the fourth image are not paired, in order to further retain the content of the image, the embodiment of the present application also adds a self-perception loss. In one possible implementation, the input image and the output image are compared, and the model is trained by the difference between the input image and the output image so that the model retains the image content in the input image. Optionally, the computer device determines a third loss value based on the first loss value and the second loss value, including: determining a tenth loss value based on the sixth image and the third image; and weighting the first loss value, the second loss value, and the tenth loss value to obtain the third loss value.

[0168] The tenth loss value may be determined by any loss function, which is not limited in the present embodiment, and is only illustrated by the following example: For example, the tenth loss value is determined by the difference between the image features of the input image and the image features of the output image.

[0169] It should be noted that the embodiment of the present application also provides a method for enhancing local information and global information, so the self-perception loss can be added to the global and local training in sequence. For example, based on the image features of the sixth image and the image features of the third image, the thirteenth loss value is determined; for each local image in the sixth image, based on the image features of the local image and the image features of the corresponding local image in the third image, the fourteenth loss value is determined; the thirteenth loss value and each fourteenth loss value are weighted to obtain the tenth loss value.

[0170] For example, the thirteenth loss value is:

[0171] The fourteenth loss value is:

[0172] Among them, φ(I′ gen ) is the image feature of the sixth image; φ(I scr ) is the image feature of the third image. ||φ(I′ gen )-φ(I scr )||1 represents the absolute value difference between the image feature of the sixth image and the image feature of the third image. φ(I′ gen-p ) is the image feature of the local image in the sixth image; φ(I scr-p ) is the image feature of the local image in the third image. ||φ(I′ gen-p )-φ(I scr-p )||1 represents the difference in absolute value between the image feature of the partial image in the sixth image and the image feature of the partial image in the third image.

[0173] It should be noted that in the embodiment of the present application, when weighted processing is performed on the loss value, the weight of the loss value is not limited, and the weight of each loss value can be any value greater than 0 and less than or equal to 1. For example, the third loss value is L.

[0174] Among them, λ1=λ3=λ4=λ5=λ6=λ7=λ8=1, λ2=0.1.

[0175] It should be noted that in the embodiment of the present application, the feature extraction layer, attention layer, and image generation layer in each branch that processes the image in the image enhancement model are the same. Therefore, when training the image enhancement model, the feature extraction layer, attention layer, and image generation layer in any branch can be adjusted according to the obtained loss value, and then the feature extraction layer, attention layer, and image generation layer in each branch can be synchronized.

[0176] It should be noted that, in the embodiment of the present application, training the image enhancement model through the third loss value means: adjusting the model parameters of the image enhancement model through the third loss value so that the third loss value converges.

[0177] 705. When the computer device obtains the image to be enhanced, it performs image enhancement on the image to be enhanced by using the trained image enhancement model to obtain an enhanced image.

[0178] It should be noted that, in the embodiment of the present application, the process of the image enhancement model performing image enhancement on the image to be enhanced is the same as the process of the image enhancement model processing the sample data, which will not be described one by one here.

[0179] The image enhancement method provided in the embodiment of the present application trains an image enhancement model by using paired sample data and unpaired sample data together. Since paired sample data can provide mapping information between low-quality images and high-quality images, using paired sample data to train the image enhancement model can enable the image enhancement model to learn the ability to enhance the image quality of the input image. Since the low-quality images and high-quality images in the unpaired sample data are not paired, a large amount of unpaired sample data in different fields can be obtained from the Internet. Therefore, using unpaired sample data can enable the image enhancement model to learn the ability to adapt to input images in multiple fields. Thus, by training the image enhancement model with paired sample data and unpaired sample data together, the image enhancement model can generate images with higher fidelity and improve the generalization ability of the image enhancement model.

[0180] The embodiment of the present application also provides a multi-exposure fusion strategy, which can adaptively fuse multiple images with different exposure levels into a globally better image, that is, enable the image enhancement model to learn a better exposure, thereby improving the accuracy of the image enhancement model.

[0181] The embodiment of the present application also provides a method for enhancing global information and local information, which can eliminate artifacts existing in the image generated by the image enhancement model, so that the generated image has better visual consistency. In addition, it can also make the details of the generated image clearer.

[0182] It should be noted that the related art provides five methods: GLADNet (global illumination perception and detail preservation network), DeepLPF (deep low-pass filter), HDRNet (high dynamic range neural network), EnlightenGAN (efficient unsupervised generative adversarial network) and Lime. We use the implementation method provided in the embodiment of the present application (using Fig.10 The image enhancement model obtained by the training method shown in the figure) and the above five methods were respectively subjected to qualitative and quantitative experiments on the MIT-Adobe5K dataset. As shown in Table 1, the PSNR (Peak Signal to Noise Ratio) index of the method provided in the embodiment of the present application achieved the best effect.

[0183] Table 1

[0184] method PSNR Lime 15.95 GLADNet 16.04 EnlightenGAN 13.98 HDRNet 19.69 DeepLPF 22.17 The image enhancement method provided by the present application 22.75

[0185] In addition, this application also conducted a user subjective survey to evaluate the effectiveness of the method provided in this application and other methods. The relevant personnel randomly selected 20 pictures from the test set and recruited 25 volunteers to evaluate the results of each method. The images were displayed in random order. The 25 volunteers were asked to consider the following situations: (1) whether the image contains visible noise; (2) whether the image is underexposed or overexposed; (3) whether the image has artifacts; (4) whether the image has unrealistic colors or distortion. For each result, the volunteers were asked to give a score of 1 (worst) to 6 (best). The method provided in this application received 208 best scores and 162 second best scores, accounting for 92.5% of all scores. Compared with the other 5 methods, the method provided in this application was the highest rated among all volunteers.

[0186] Fig.11 is a schematic diagram of the structure of an image enhancement device provided in an embodiment of the present application, see Fig.11 , the device comprises:

[0187] The sample acquisition module 1101 is used to acquire multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data includes two images with the same image content but different image quality, each group of unpaired sample data includes two images with different image content and image quality, and the multiple groups of unpaired sample data include images in multiple fields;

[0188] A training module 1102 is used to train the image enhancement model based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model, wherein the paired sample data is used to enable the trained image enhancement model to learn the ability to enhance the image quality of the input image, and the unpaired sample data is used to enable the trained image enhancement model to learn the ability to adapt to input images in multiple fields;

[0189] The processing module 1103 is used to perform image enhancement on the image to be enhanced by using the trained image enhancement model to obtain an enhanced image when the image to be enhanced is acquired.

[0190] like Fig.12 As shown, in a possible implementation, each group of paired sample data includes a first image and a second image, and the image quality of the second image is higher than that of the first image, and each group of unpaired sample data includes a third image and a fourth image, and the image quality of the fourth image is higher than that of the third image;

[0191] The training module 1102 includes:

[0192] For any set of paired sample data and any set of unpaired sample data,

[0193] A first processing unit 1112 is configured to perform image enhancement on the first image in the paired sample data by using the image enhancement model to obtain a fifth image;

[0194] A second processing unit 1122 is used to perform image enhancement on the third image in the unpaired sample data by using the image enhancement model to obtain a sixth image;

[0195] The training unit 1132 is used to train the image enhancement model based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image, to obtain the trained image enhancement model.

[0196] In a possible implementation, the first processing unit 1112 includes:

[0197] an adjusting subunit, configured to adjust the exposure of the first image to obtain a seventh image having an exposure of a first value and an eighth image having an exposure of a second value;

[0198] A first processing subunit is used to perform image enhancement on the seventh image by using the image enhancement model to obtain a ninth image;

[0199] A second processing subunit is used to perform image enhancement on the eighth image by using the image enhancement model to obtain a tenth image;

[0200] The fusion subunit is used to fuse the ninth image and the tenth image through the image enhancement model to obtain the fifth image.

[0201] In one possible implementation, the first processing subunit is used to perform feature extraction on the seventh image through the image enhancement model to obtain a first feature map of the seventh image; perform global information and local information enhancement on the first feature map through the image enhancement model to obtain a second feature map; and decode the second feature map through the image enhancement model to obtain the ninth image.

[0202] In one possible implementation, the fusion subunit is used to perform feature extraction on the second feature map used to generate the ninth image and the third feature map used to generate the tenth image through the image enhancement model to obtain a fourth feature map, where the fourth feature map is used to represent the relative quality of the ninth image and the tenth image in the same area; and based on the fourth feature map, the ninth image and the tenth image are fused through the image enhancement model to obtain the fifth image.

[0203] In a possible implementation, the second processing unit 1122 includes:

[0204] a feature extraction subunit, configured to perform feature extraction on the third image by using the image enhancement model to obtain a fifth feature map of the third image;

[0205] an enhancement subunit, configured to enhance the global information and the local information of the fifth feature map by using the image enhancement model to obtain a sixth feature map;

[0206] A decoding subunit is used to decode the sixth feature map through the image enhancement model to obtain the sixth image.

[0207] In one possible implementation, the enhancement subunit is used to perform feature extraction on each row of the fifth feature map through the image enhancement model to obtain a seventh feature map; perform feature extraction on each column of the fifth feature map through the image enhancement model to obtain an eighth feature map; fuse the seventh feature map and the eighth feature map through the image enhancement model to obtain a ninth feature map; perform global information enhancement on the fifth feature map based on the ninth feature map through the image enhancement model to obtain a tenth feature map; perform feature extraction on the tenth feature map through the image enhancement model to obtain an eleventh feature map; perform local information enhancement on the tenth feature map based on the eleventh feature map through the image enhancement model to obtain a twelfth feature map; and fuse the tenth feature map and the twelfth feature map through the image enhancement model to obtain the sixth feature map.

[0208] In one possible implementation, the enhancement subunit is used to perform feature extraction on the ninth feature map through the image enhancement model to obtain a thirteenth feature map, wherein the thirteenth feature map represents global information of the third image; normalize the thirteenth feature map through the image enhancement model to obtain a fourteenth feature map; and perform a dot product of the fourteenth feature map and the fifth feature map through the image enhancement model to obtain the tenth feature map.

[0209] In a possible implementation, the training unit 1132 includes:

[0210] A first determining subunit, configured to determine a first loss value based on the fifth image and the second image, wherein the first loss value is used to represent a difference between the fifth image and the second image;

[0211] A second determining subunit, configured to determine a second loss value based on the sixth image and the fourth image, wherein the second loss value is used to represent a quality difference between the sixth image and the fourth image;

[0212] a third determining subunit, configured to determine a third loss value based on the first loss value and the second loss value;

[0213] A training subunit is used to train the image enhancement model based on the third loss value to obtain the trained image enhancement model.

[0214] In one possible implementation, the first determination subunit is used to process the fifth image and the second image to obtain a fourth loss value; process image features of the fifth image and image features of the second image to obtain a fifth loss value; and weight the fourth loss value and the fifth loss value to obtain the first loss value.

[0215] In one possible implementation, the first determination subunit is used to obtain the difference in absolute values ​​between the fifth image and the second image; obtain the difference in Euclidean distances between the fifth image and the second image; determine a first weight for the difference in absolute values ​​and a second weight for the difference in Euclidean distances based on the number of training times of the image enhancement model; and perform weighted processing on the difference in absolute values ​​and the difference in Euclidean distances based on the first weight and the second weight to obtain the fourth loss value.

[0216] In one possible implementation, the second determination subunit is used to determine a sixth loss value based on the image quality information of the sixth image and the image quality information of the fourth image; to capture a target number of partial images from the sixth image and the fourth image, respectively, according to the target size; for each partial image of the sixth image, to determine a seventh loss value based on the image quality information of the partial image and the image quality information of the corresponding partial image in the fourth image; and to weight the sixth loss value and each seventh loss value to obtain the second loss value.

[0217] In one possible implementation, the second determination subunit is used to process the image quality information of the sixth image and the image quality information of the fourth image according to the first loss function of the image enhancement model to determine an eighth loss value; process the image quality information of the sixth image according to the second loss function of the adversarial model to obtain a ninth loss value; and weight the eighth loss value and the ninth loss value to obtain the sixth loss value.

[0218] In a possible implementation, the third determination subunit is used to determine a tenth loss value based on the sixth image and the third image; and weight the first loss value, the second loss value and the tenth loss value to obtain the third loss value.

[0219] It should be noted that: the image enhancement device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when enhancing an image. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the image enhancement device provided in the above embodiment and the image enhancement method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0220] In an exemplary embodiment, a computer device is provided, which includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the image enhancement method as described in the above embodiment.

[0221] Optionally, the computer device is provided as a terminal. Fig.13 The structure block diagram of a terminal 1300 provided by an exemplary embodiment of the present application is shown. The terminal 1300 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a laptop computer or a desktop computer. The terminal 1300 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0222] The terminal 1300 includes a processor 1301 and a memory 1302 .

[0223] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0224] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one program code, which is used to be executed by the processor 1301 to implement the image enhancement method provided in the method embodiment of the present application.

[0225] In some embodiments, the terminal 1300 may also optionally include: a peripheral device interface 1303 and at least one peripheral device. The processor 1301, the memory 1302 and the peripheral device interface 1303 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1304, a display screen 1305, a camera 1306, an audio circuit 1307, a positioning component 1308 and a power supply 1309.

[0226] The peripheral device interface 1303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0227] The radio frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1304 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0228] The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1305 is a touch display screen, the display screen 1305 also has the ability to collect touch signals on the surface or above the surface of the display screen 1305. The touch signal can be input to the processor 1301 as a control signal for processing. At this time, the display screen 1305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1305 can be one, and the front panel of the terminal 1300 is set; in other embodiments, the display screen 1305 can be at least two, which are respectively set on different surfaces of the terminal 1300 or are folded; in some other embodiments, the display screen 1305 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1300. Even, the display screen 1305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0229] The camera assembly 1306 is used to capture images or videos. Optionally, the camera assembly 1306 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1306 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0230] The audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1301 for processing, or input them into the radio frequency circuit 1304 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1300. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1307 may also include a headphone jack.

[0231] The positioning component 1308 is used to locate the current geographical location of the terminal 1300 to implement navigation or LBS (Location Based Service). The positioning component 1308 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0232] The power supply 1309 is used to power various components in the terminal 1300. The power supply 1309 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1309 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0233] In some embodiments, the terminal 1300 further includes one or more sensors 1310 , including but not limited to: an acceleration sensor 1311 , a gyroscope sensor 1312 , a pressure sensor 1313 , a fingerprint sensor 1314 , an optical sensor 1315 , and a proximity sensor 1316 .

[0234] The acceleration sensor 1311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1300. For example, the acceleration sensor 1311 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1301 can control the display screen 1305 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1311. The acceleration sensor 1311 can also be used to collect motion data of games or users.

[0235] The gyro sensor 1312 can detect the body direction and rotation angle of the terminal 1300, and the gyro sensor 1312 can cooperate with the acceleration sensor 1311 to collect the user's 3D actions on the terminal 1300. The processor 1301 can implement the following functions based on the data collected by the gyro sensor 1312: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0236] The pressure sensor 1313 can be set on the side frame of the terminal 1300 and / or the lower layer of the display screen 1305. When the pressure sensor 1313 is set on the side frame of the terminal 1300, it can detect the user's holding signal of the terminal 1300, and the processor 1301 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1313. When the pressure sensor 1313 is set on the lower layer of the display screen 1305, the processor 1301 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1305. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0237] The fingerprint sensor 1314 is used to collect the user's fingerprint, and the processor 1301 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 1314, or the fingerprint sensor 1314 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 1301 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 1314 can be set on the front, back, or side of the terminal 1300. When a physical button or a manufacturer logo is set on the terminal 1300, the fingerprint sensor 1314 can be integrated with the physical button or the manufacturer logo.

[0238] The optical sensor 1315 is used to collect the ambient light intensity. In one embodiment, the processor 1301 can control the display brightness of the display screen 1305 according to the ambient light intensity collected by the optical sensor 1315. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1305 is increased; when the ambient light intensity is low, the display brightness of the display screen 1305 is reduced. In another embodiment, the processor 1301 can also dynamically adjust the shooting parameters of the camera assembly 1306 according to the ambient light intensity collected by the optical sensor 1315.

[0239] The proximity sensor 1316, also called a distance sensor, is disposed on the front panel of the terminal 1300. The proximity sensor 1316 is used to collect the distance between the user and the front of the terminal 1300. In one embodiment, when the proximity sensor 1316 detects that the distance between the user and the front of the terminal 1300 is gradually decreasing, the processor 1301 controls the display screen 1305 to switch from the screen-on state to the screen-off state; when the proximity sensor 1316 detects that the distance between the user and the front of the terminal 1300 is gradually increasing, the processor 1301 controls the display screen 1305 to switch from the screen-off state to the screen-on state.

[0240] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the terminal 1300, and the terminal 1300 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0241] Optionally, the computer device is provided as a server. Fig.14 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1401 and one or more memories 1402, wherein the memory 1402 stores at least one program code, and the at least one program code is loaded and executed by the processor 1401 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0242] The server 1400 is used to execute the steps executed by the server in the above method embodiment.

[0243] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a program code, and the program code can be executed by a processor in a computer device to perform the image enhancement method in the above embodiment. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0244] In an exemplary embodiment, a computer program or a computer program product is further provided. The computer program or the computer program product includes a computer program code. When the computer program code is executed by a computer, the computer implements the image enhancement method in the above embodiment.

[0245] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0246] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image enhancement method, characterized in that: The method comprises: obtaining multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data comprises two images with the same image content but different image quality, each group of unpaired sample data comprises two images with different image content and image quality, and the multiple groups of unpaired sample data comprises images in multiple fields; based on the multiple groups of paired sample data and the multiple groups of unpaired sample data, an image enhancement model is trained to obtain the trained image enhancement model, the paired sample data are used to enable the trained image enhancement model to learn the ability to enhance the image quality of input images, and the unpaired sample data are used to enable the trained image enhancement model to learn the ability to adapt to input images in multiple fields; when an image to be enhanced is obtained, the image to be enhanced is enhanced by using the trained image enhancement model to obtain an enhanced image; Each group of paired sample data includes a first image and a second image, and the image quality of the second image is higher than that of the first image; each group of unpaired sample data includes a third image and a fourth image, and the image quality of the fourth image is higher than that of the third image; the image enhancement model is trained based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model, including: for any group of paired sample data and any group of unpaired sample data, the first image in the paired sample data is enhanced by the image enhancement model to obtain a fifth image; the third image in the unpaired sample data is enhanced by the image enhancement model to obtain a sixth image; based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image, the image enhancement model is trained to obtain the trained image enhancement model; The step of performing image enhancement on the third image in the unpaired sample data by the image enhancement model to obtain a sixth image includes: performing feature extraction on the third image by the image enhancement model to obtain a fifth feature map of the third image; performing global information and local information enhancement on the fifth feature map by the image enhancement model to obtain a sixth feature map; and decoding the sixth feature map by the image enhancement model to obtain the sixth image. The method of performing global information and local information enhancement on the fifth feature map through the image enhancement model to obtain a sixth feature map includes: performing feature extraction on each row of the fifth feature map through the image enhancement model to obtain a seventh feature map; performing feature extraction on each column of the fifth feature map through the image enhancement model to obtain an eighth feature map; fusing the seventh feature map with the eighth feature map through the image enhancement model to obtain a ninth feature map; performing global information enhancement on the fifth feature map based on the ninth feature map through the image enhancement model to obtain a tenth feature map; performing feature extraction on the tenth feature map through the image enhancement model to obtain an eleventh feature map; performing local information enhancement on the tenth feature map based on the eleventh feature map through the image enhancement model to obtain a twelfth feature map; and fusing the tenth feature map with the twelfth feature map through the image enhancement model to obtain the sixth feature map.

2. The method according to claim 1, characterized in that The image enhancement model is used to perform image enhancement on the first image in the paired sample data to obtain a fifth image, including: adjusting the exposure of the first image to obtain a seventh image with an exposure of a first value and an eighth image with an exposure of a second value; performing image enhancement on the seventh image to obtain a ninth image by the image enhancement model; performing image enhancement on the eighth image to obtain a tenth image by the image enhancement model; and fusing the ninth image and the tenth image by the image enhancement model to obtain the fifth image.

3. The method according to claim 2, characterized in that The method of performing image enhancement on the seventh image through the image enhancement model to obtain a ninth image includes: performing feature extraction on the seventh image through the image enhancement model to obtain a first feature map of the seventh image; performing global information and local information enhancement on the first feature map through the image enhancement model to obtain a second feature map; and decoding the second feature map through the image enhancement model to obtain the ninth image.

4. The method according to claim 2, characterized in that: The method of fusing the ninth image and the tenth image through the image enhancement model to obtain the fifth image includes: extracting features from a second feature map used to generate the ninth image and a third feature map used to generate the tenth image through the image enhancement model to obtain a fourth feature map, wherein the fourth feature map is used to represent the relative quality of the ninth image and the tenth image in the same area; and fusing the ninth image and the tenth image based on the fourth feature map through the image enhancement model to obtain the fifth image.

5. The method according to claim 1, characterized in that The method of performing global information enhancement on the fifth feature map based on the ninth feature map through the image enhancement model to obtain a tenth feature map includes: performing feature extraction on the ninth feature map through the image enhancement model to obtain a thirteenth feature map, wherein the thirteenth feature map represents global information of the third image; normalizing the thirteenth feature map through the image enhancement model to obtain a fourteenth feature map; and performing dot multiplication of the fourteenth feature map and the fifth feature map through the image enhancement model to obtain the tenth feature map.

6. The method according to claim 1, characterized in that The method of training the image enhancement model based on the difference between the fifth image and the second image and the quality difference between the sixth image and the fourth image to obtain the trained image enhancement model includes: determining a first loss value based on the fifth image and the second image, wherein the first loss value is used to represent the difference between the fifth image and the second image; determining a second loss value based on the sixth image and the fourth image, wherein the second loss value is used to represent the quality difference between the sixth image and the fourth image; determining a third loss value based on the first loss value and the second loss value; and training the image enhancement model based on the third loss value to obtain the trained image enhancement model.

7. The method according to claim 6, characterized in that The second loss value is determined based on the sixth image and the fourth image, comprising: determining the sixth loss value according to image quality information of the sixth image and image quality information of the fourth image; intercepting a target number of partial images from the sixth image and the fourth image, respectively, according to a target size; for each partial image of the sixth image, determining a seventh loss value according to image quality information of the partial image and image quality information of the corresponding partial image in the fourth image; and weighting the sixth loss value and each seventh loss value to obtain the second loss value.

8. The method according to claim 7, characterized in that The determining of the sixth loss value based on the image quality information of the sixth image and the image quality information of the fourth image comprises: processing the image quality information of the sixth image and the image quality information of the fourth image according to the first loss function of the image enhancement model to determine the eighth loss value; processing the image quality information of the sixth image according to the second loss function of the adversarial model to obtain the ninth loss value; and weighting the eighth loss value and the ninth loss value to obtain the sixth loss value.

9. The method according to claim 8, characterized in that Determining the third loss value based on the first loss value and the second loss value includes: determining the tenth loss value based on the sixth image and the third image; and weighting the first loss value, the second loss value and the tenth loss value to obtain the third loss value.

10. An image enhancement device, characterized in that: The device comprises: a sample acquisition module, used for acquiring multiple groups of paired sample data and multiple groups of unpaired sample data, each group of paired sample data comprises two images with the same image content but different image quality, each group of unpaired sample data comprises two images with different image content and image quality, and the multiple groups of unpaired sample data comprise images from multiple fields; a training module, used for training an image enhancement model based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model, the paired sample data are used to enable the trained image enhancement model to learn the ability to enhance the image quality of an input image, and the unpaired sample data are used to enable the trained image enhancement model to learn the ability to adapt to input images from multiple fields; a processing module, used for, when an image to be enhanced is acquired, performing image enhancement on the image to be enhanced by using the trained image enhancement model to obtain an enhanced image; Each group of paired sample data includes a first image and a second image, and the image quality of the second image is higher than that of the first image; each group of unpaired sample data includes a third image and a fourth image, and the image quality of the fourth image is higher than that of the third image; the image enhancement model is trained based on the multiple groups of paired sample data and the multiple groups of unpaired sample data to obtain the trained image enhancement model, including: for any group of paired sample data and any group of unpaired sample data, the first image in the paired sample data is enhanced by the image enhancement model to obtain a fifth image; the third image in the unpaired sample data is enhanced by the image enhancement model to obtain a sixth image; based on the difference between the fifth image and the second image, and the quality difference between the sixth image and the fourth image, the image enhancement model is trained to obtain the trained image enhancement model; The step of performing image enhancement on the third image in the unpaired sample data by the image enhancement model to obtain a sixth image includes: performing feature extraction on the third image by the image enhancement model to obtain a fifth feature map of the third image; performing global information and local information enhancement on the fifth feature map by the image enhancement model to obtain a sixth feature map; and decoding the sixth feature map by the image enhancement model to obtain the sixth image. The method of performing global information and local information enhancement on the fifth feature map through the image enhancement model to obtain a sixth feature map includes: performing feature extraction on each row of the fifth feature map through the image enhancement model to obtain a seventh feature map; performing feature extraction on each column of the fifth feature map through the image enhancement model to obtain an eighth feature map; fusing the seventh feature map with the eighth feature map through the image enhancement model to obtain a ninth feature map; performing global information enhancement on the fifth feature map based on the ninth feature map through the image enhancement model to obtain a tenth feature map; performing feature extraction on the tenth feature map through the image enhancement model to obtain an eleventh feature map; performing local information enhancement on the tenth feature map based on the eleventh feature map through the image enhancement model to obtain a twelfth feature map; and fusing the tenth feature map with the twelfth feature map through the image enhancement model to obtain the sixth feature map.

11. A computer device, characterized in that: The computer device includes one or more processors and one or more memories, wherein the one or more memories store at least one program code, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the image enhancement method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed by the image enhancement method according to any one of claims 1 to 9.

Images