Model training method, image enhancement method and electronic device

Abstract

The embodiments of the present application provide a model training method, an image enhancement method and an electronic device, and relate to the field of terminal technology. The method includes: the image enhancement model includes a first codebook module and a second codebook module, wherein the first codebook module depends on external prior information. During training, the first codebook module is omitted with a certain probability, and only the second codebook module is used for processing, and the error is calculated and the model parameters are adjusted. When the model is used for image enhancement, if the external prior information is not available, only the second codebook module is used for corresponding processing to obtain an enhanced target image. In this way, on the one hand, when the prior information is available, the model has a relatively better image enhancement effect, and on the other hand, when the prior information is not available, the model can still perform effective image enhancement processing, that is, the model has good versatility and good model performance.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a model training method, an image enhancement method, an electronic device, a chip system and a storage medium. Background Art

[0002] In face photography, various factors may cause the faces in the image to appear blurry, noisy, or missing. Similar problems may also exist for old photos. People hope to use some image augmentation methods to deal with the above problems.

[0003] A current method of image enhancement is to use prior information as additional input in the model, where the prior information can be, for example, facial key point information. Based on the use of prior information, the model can restore the details of the face, so that the face in the enhanced image can maintain similarity while also having high clarity.

[0004] However, the above image enhancement methods have the following problem: the normal reasoning of the model depends on external prior information, and when the prior information is not available, image enhancement cannot be performed. Summary of the invention

[0005] The embodiment of the present application provides a model training method, an image enhancement method and an electronic device, which are applied to the field of terminal technology. Through the model training method and the image enhancement method, the image enhancement model can perform image enhancement and have good model performance without external prior information.

[0006] In a first aspect, an embodiment of the present application proposes a model training method, comprising:

[0007] Acquire at least one set of training data, wherein the set of training data includes a training image, training prior information corresponding to the training image, and an actual enhanced image corresponding to the training image;

[0008] For any one of the at least one set of training data, input a training image in the training data into a first model, wherein the first model includes a first codebook module and a second codebook module, wherein the input data of the first codebook module includes prior information, and the input data of the second codebook module does not include prior information;

[0009] Performing model processing on the training image according to the first model to obtain at least one predicted enhanced image corresponding to the training image, wherein the second codebook module is used to participate in the model processing process, and the first codebook module is used to participate in the model processing process with a first probability;

[0010] The model parameters of the first model are adjusted according to at least one predicted enhanced image corresponding to the training image and the actual enhanced image corresponding to the training image.

[0011] In this implementation, a method for training an image enhancement model is described. In addition to using training images and actual enhanced images, the image enhancement model also uses training prior information during training. The image enhancement model includes two important codebook modules, and the first codebook module relies on prior information as input. During the training process, the first codebook module will not participate in the model processing process according to the first probability, and only the second codebook module will be used for processing. The model obtained by such a model training method can not only ensure that the model has basic image enhancement functions without prior information, but also further improve the image enhancement effect with prior information, combining versatility and model performance.

[0012] In a possible implementation, performing model processing on the training image according to the first model to obtain at least one predicted enhanced image corresponding to the training image includes:

[0013] Performing a first training process on the training image according to the first model to obtain a first predicted enhanced image, wherein in the first training process, the first codebook module performs data processing according to the training image and training prior information corresponding to the training image, and the second codebook module performs data processing according to the training image;

[0014] Performing a second training process on the training image according to the first model and the first probability to obtain a second predicted enhanced image, wherein in the second training process, the second codebook module performs data processing according to the training image, and the first codebook module does not participate in the model processing process;

[0015] The at least one predicted enhanced image includes a first predicted enhanced image and a second predicted enhanced image.

[0016] In this implementation, it is explained in detail that the model training method can be divided into two training processes. In the first training process, both codebook modules participate in the processing of the training image. The second training process is performed with the first probability, and only the second codebook module is used to participate in data processing in the second training process. If the second training process is not performed in the manner of the first probability, but the first training process is always performed, the model cannot learn how to perform image enhancement processing when it is difficult to obtain prior information. On the contrary, if the second training process is always performed, the first codebook module that depends on the prior information cannot be learned, so even if the prior information is available, the model cannot use it and the model performance cannot be further improved.

[0017] In a possible implementation, adjusting the model parameters of the first model according to at least one predicted enhanced image corresponding to the training image and the actual enhanced image corresponding to the training image includes:

[0018] Determine a first error according to the first predicted enhanced image and the actual enhanced image, and adjust a model parameter of the first model according to the first error;

[0019] When the first model performs the second training process, a second error is determined according to the second predicted enhanced image and the actual enhanced image, and the model parameters of the first model are adjusted according to the second error.

[0020] In this implementation, it is explained how to adjust the model parameters during the training process. Corresponding to the two training processes, the errors can also be calculated separately when calculating the errors, and then the corresponding parameters can be adjusted according to the errors. For example, in the first training process, the first codebook module and the second codebook module are both involved in the processing of the data. Then, when the parameters are adjusted by the first error, the parameters in the two codebook modules will also be adjusted. In the second training process, only the second codebook module is involved in the processing of the data. At this time, the relevant parameters in the first codebook module will not be adjusted according to the second error. Through such error calculation and parameter adjustment, it can be ensured that the model can learn how to perform image enhancement and can operate correctly in both modes of obtaining prior information or not.

[0021] In a possible implementation, the first model further includes an encoder and a decoder, and the encoder is used to process the training image to obtain first feature information corresponding to the training image;

[0022] The first codebook module and the second codebook module are in a series relationship;

[0023] In the case where the first codebook module participates in the model processing, the first codebook module is used to output the second feature information according to the first feature information and the training prior information corresponding to the training image; the second codebook module is used to output the third feature information according to the second feature information;

[0024] When the first codebook module does not participate in the model processing, the second codebook module is used to output fourth feature information according to the first feature information;

[0025] The decoder is used to process the third feature information or the fourth feature information to obtain a predicted enhanced image corresponding to the training image.

[0026] In this implementation, the specific structure of the trained model and the connection relationship therein are described in detail. In particular, the first codebook module and the second codebook module in this implementation are in series, but there is no restriction on which one comes first or after. The encoder in the model can be understood as preprocessing the input training image and converting it into feature information. The first codebook module and the second codebook module further process the feature information, and integrate the effective information in each codebook or prior information into the feature information to obtain new feature information. Based on the feature information processed by the two codebook modules, the decoder processes the final predicted enhanced image. The complete model structure proposed in this implementation can realize image enhancement processing after training, but the specific structure of the modules therein is not restricted here.

[0027] In a possible implementation, the first model further includes an encoder and a decoder, and the encoder is used to process the training image to obtain fifth feature information corresponding to the training image;

[0028] The first codebook module and the second codebook module are in a parallel relationship;

[0029] In the case where the first codebook module participates in the model processing, the first codebook module is used to output the sixth feature information according to the fifth feature information and the training prior information corresponding to the training image; the second codebook module is used to output the seventh feature information according to the fifth feature information; the first model is also used to fuse the sixth feature information and the seventh feature information to obtain the eighth feature information;

[0030] When the first codebook module does not participate in the model processing, the second codebook module is used to output ninth feature information according to the fifth feature information;

[0031] The decoder is used to process the eighth feature information or the ninth feature information to obtain a predicted enhanced image corresponding to the training image.

[0032] In this implementation, another model training method is described, in which the first codebook module and the second codebook module of the model are in a parallel relationship. That is to say, the two codebook modules will receive the fifth feature information processed by the encoder in parallel, and then output the sixth feature information and the seventh feature information respectively after processing. Then the sixth feature information and the seventh feature information are fused, and subsequent related processing is performed. By processing two codebook modules in parallel, the feature information can be processed faster, the time required for model processing can be shortened, and faster and more efficient training can be performed.

[0033] In a second aspect, an embodiment of the present application provides an image enhancement method, comprising:

[0034] Acquire a first image and target prior information corresponding to the first image;

[0035] Inputting the first image and the target prior information into a first model, wherein the first model includes a first codebook module and a second codebook module, wherein the input data of the first codebook module includes the prior information, and the input data of the second codebook module does not include the prior information;

[0036] Performing model processing on the first image according to the first model to obtain a target enhanced image corresponding to the first image, wherein the first codebook module participates in the model processing process when target prior information is available, and the second codebook module does not participate in the model processing process when the target prior information is not available;

[0037] The first model is trained according to the method described in the first aspect.

[0038] In this implementation, corresponding to the model training method of the first aspect, an image enhancement method is proposed, in which two important codebook modules are provided. When the image enhancement method is actually used, different reasoning processes are performed according to whether the prior information is available, that is, when the prior information is available, both the first codebook module and the second codebook module participate in the model reasoning; and when the prior information is not available, only the second codebook module that does not depend on the prior information is used to participate in the model reasoning. In this way, on the one hand, the lower limit of the model performance can be ensured based on the second codebook module, and the basic image enhancement function can be provided in various situations, and it will not be unable to process due to the unavailability of prior information. On the other hand, when there is available prior information, the performance of the model can be further improved based on the first codebook module, so that it has a better image enhancement effect. In this way, the image enhancement method can have both versatility and high model performance.

[0039] In a possible implementation, the method further includes:

[0040] Processing the first image according to the first algorithm unit to obtain target prior information corresponding to the first image;

[0041] The first algorithm unit is further used to output the confidence of the target prior information. When the confidence is greater than or equal to a preset threshold, the target prior information is available. When the confidence is less than the preset threshold, the target prior information is unavailable.

[0042] In this implementation, a method for using target prior information in an image enhancement method is described. In this implementation, it is explained that the target prior information corresponding to the first image obtained by the first algorithm unit also includes confidence. The confidence can be used to determine whether the prior information is available, that is, it can be used by setting a preset threshold. When the confidence is greater than or equal to the preset threshold, the surface target prior information is available, otherwise it is not available. When the present implementation is actually used for image enhancement, the preset threshold can be appropriately adjusted. For example, if the preset threshold is set relatively large, the validity of the prior information can be further limited, thereby avoiding interference from some invalid prior information; or if the preset threshold is set relatively small, available prior information can be obtained in more cases. In this way, the present implementation has a certain flexibility, and the user can adjust the preset threshold according to the actual scene and needs to achieve a better image enhancement effect.

[0043] In a possible implementation, the first model further includes an encoder and a decoder, and the encoder is used to process the first image to obtain first feature information corresponding to the first image;

[0044] The first codebook module and the second codebook module are in a series relationship;

[0045] When the target prior information is available, the first codebook module is used to output the second feature information according to the first feature information and the target prior information corresponding to the first image, and the second codebook module is used to output the third feature information according to the second feature information;

[0046] When the target prior information is unavailable, the second codebook module is used to output fourth feature information according to the first feature information;

[0047] The decoder is used to process the third feature information or the fourth feature information to obtain a target enhanced image corresponding to the first image.

[0048] In this implementation, corresponding to the relevant content in the model training method of the first aspect, this implementation explains that the first codebook module and the second codebook module are in a series relationship, and also explains the process of image enhancement processing of each module in the entire model. Among them, this implementation does not make specific restrictions on which of the first codebook module and the second codebook module comes first or last. In the process of model reasoning, when the target prior information is not available, the first codebook module is omitted, and only the second codebook module is used to process the corresponding data. In this way, the image enhancement method based on the model can perform high-quality enhancement processing on the details of the image in some cases, and can also be versatile and can be used in various scenarios.

[0049] In a possible implementation, the first model further includes an encoder and a decoder, and the encoder is used to process the first image to obtain fifth feature information corresponding to the first image;

[0050] The first codebook module and the second codebook module are in a parallel relationship;

[0051] When the target prior information is available, the first codebook module is used to output the sixth feature information according to the fifth feature information and the target prior information corresponding to the first image; the second codebook module is used to output the seventh feature information according to the fifth feature information; the first model is also used to fuse the sixth feature information and the seventh feature information to obtain the eighth feature information;

[0052] When the target prior information is unavailable, the second codebook module is used to output ninth feature information according to the fifth feature information;

[0053] The decoder is used to process the eighth feature information or the ninth feature information to obtain a target enhanced image corresponding to the first image.

[0054] In this implementation, the processing flow of the first image in the image enhancement model is explained, and it is emphasized that the first codebook module and the second codebook module are in a parallel relationship. Since the two codebook modules are in a parallel relationship, the fifth feature information from the encoder can be received in parallel and each performs corresponding processing. Therefore, the image enhancement module under this implementation can have a faster reasoning speed. In this way, considering that when users actually use the image enhancement function, they always hope to shorten the processing time so as to see the new image obtained after processing more quickly, the improvements made in this implementation can improve the user experience to a certain extent.

[0055] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, the memory being used to store code instructions, and the processor being used to run the code instructions to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is run on a computer, the computer executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0057] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computer program. When the computer program runs on a computer, the computer executes the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0058] In a sixth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect. The communication interface in the chip can be an input / output interface, a pin, or a circuit, etc.

[0059] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0060] It should be understood that the third to sixth aspects of the present application correspond to the technical solutions of the first or second aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of the face image enhancement effect provided in an embodiment of the present application;

[0062] Figure 2 Schematic diagram of the implementation of the image enhancement model provided in the embodiment of the present application Figure 1 ;

[0063] Figure 3 Schematic diagram of the implementation of the image enhancement model provided in the embodiment of the present application Figure 2 ;

[0064] Figure 4 A schematic diagram of the prior information of facial parts provided in an embodiment of the present application;

[0065] Figure 5 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0066] Figure 6 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application;

[0067] Figure 7A schematic diagram of the implementation of the model training process selection provided in the embodiment of the present application;

[0068] Figure 8 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 1 ;

[0069] Fig. 9 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 2 ;

[0070] Fig.10 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 3 ;

[0071] Fig.11 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 4 ;

[0072] Fig.12 A flowchart of a model training method provided in an embodiment of the present application;

[0073] Fig.13 A schematic diagram of implementing image enhancement in a terminal provided in an embodiment of the present application;

[0074] Fig.14 Schematic diagram of the implementation of the image enhancement model provided in the embodiment of the present application Figure 3 ;

[0075] Fig.15 Schematic diagram of the implementation of the image enhancement model provided in the embodiment of the present application Figure 4 ;

[0076] Fig.16 A schematic diagram of an implementation of obtaining prior information of facial parts provided in an embodiment of the present application;

[0077] Fig.17 A schematic diagram of an implementation of a location codebook module provided in an embodiment of the present application;

[0078] Fig.18 A schematic diagram of an implementation of a vector codebook module provided in an embodiment of the present application;

[0079] Fig.19 Implementation diagram of the codebook information module provided in the embodiment of the present application Figure 1 ;

[0080] Fig. 20 Implementation diagram of the codebook information module provided in the embodiment of the present application Figure 2 ;

[0081] Fig.21 A schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application;

[0082] Fig. 22 A schematic diagram of the implementation of obtaining external image prior information provided in an embodiment of the present application;

[0083] Fig.23 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0085] 1. Image enhancement

[0086] In the embodiments of the present application, image enhancement refers to a method of converting a low-quality image into a high-quality image. A model method is generally used to achieve image enhancement. A low-quality image can be understood as an image with problems such as blur, partial image loss, noise or low resolution, while a high-quality image refers to an image that is substantially the same as a low-quality image in terms of image content, but is clear, without loss, noise or high resolution.

[0087] Image enhancement may be called by other names in different scenarios, such as image restoration, super-resolution restoration, etc. Image enhancement technology is also commonly used on facial images, including enhancing faces that are not clearly photographed or blurred faces in old photos, so that high-quality facial appearance can be seen.

[0088] 2. Prior information / part prior information

[0089] In a broad sense, prior information generally refers to effective information that can be used without other restrictions. Taking the face image enhancement method as an example, the prior information used may include: face parsing code, face key points, face images of the same person, face images of different people, and some physiological attributes of the person himself. By combining prior information for image enhancement, better results can generally be obtained compared to without prior information.

[0090] Part prior information is a type of prior information, which refers to the related information between different parts of an object. Taking the face as an example, the face part prior information can be understood as the information used to represent the eyes, nose, mouth and other parts of the face, such as the position, size or shape of the above parts.

[0091] 3. Codebook

[0092] In the embodiment of the present application, the codebook can be understood as a data structure similar to a dictionary in programming, for example, it can contain a key (Key) and a corresponding value (Value). In particular, in face image enhancement, the codebook can have different specific forms. For example, in one form, the Key can be a natural number sequence from 0 to N, and the Value is a vector, each of which contains specific face information.

[0093] The role of the codebook is to obtain the corresponding value from the codebook through some matching mechanism, and then further process the obtained value. Using face image enhancement as an example, after matching the corresponding value from the codebook through the information of the low-quality image, since each value in the codebook in this scenario may represent high-quality information of a certain part of the face, these values ​​can be used to restore a high-quality face image.

[0094] 4. Encoder / Decoder

[0095] In a broad sense, the role of an encoder is to encode the original information in a certain way to obtain another form of information, and in some cases the encoded information is compressed for easy storage and transmission. The decoder processes the encoded information in reverse order to restore the original information before encoding as much as possible, or obtain information after additional adjustments.

[0096] In the field of deep learning models for image enhancement, the encoder can be understood as the front-end network in the model, which processes the original image through a series of processes and can be regarded as a form of encoding. The decoder can be understood as the back-end network in the model, which receives the data after the front-end network encoding process, and then decodes it to output a new enhanced image. Specifically, the encoder or decoder in the model can be composed of a convolutional neural network layer and a pooling layer.

[0097] 5. Intermediate data / feature information / feature graph

[0098] In the field of deep learning, the model contains structural information and specific model parameter information. When data is input into the model, intermediate data, also called feature information, will be obtained during the model reasoning process. Each piece of intermediate data can be regarded as the result of the calculation of the previous piece of intermediate data and the corresponding network layer, just like the data flows and changes in the model.

[0099] In particular, in the field of image processing related to deep learning, the input data is image data, for example, data composed of image pixels, with width, height, and RGB channels. Then the corresponding intermediate data can also reflect certain aspects of the input image data. Such intermediate data can also be called feature maps.

[0100] 6. Loss Function

[0101] In the field of optimization or machine learning, the parameters of the model are generally random at the beginning and need to be adjusted. The role of the loss function is to evaluate the performance of the current model, such as the error between the target and the accuracy of the output results.

[0102] For example, in binary classification problems, the cross entropy loss function is often used to optimize the model. After the loss function is obtained, the model parameters can be adjusted in combination with the loss function so that the error value of the loss function corresponding to the adjusted model is lower. Specific methods for adjusting model parameters based on the loss function include, for example, the gradient back propagation algorithm, the simulated annealing algorithm, or the genetic algorithm.

[0103] 7. Region of Interest / Region of Interest Alignment

[0104] In the embodiment of the present application, a region of interest (RoI) refers to an interesting or key area in an image, including the location, size or shape information of the area, etc. For example, in the field of face image enhancement, the RoI may be relevant information of areas such as eyes, nose or mouth in a face.

[0105] Region of interest alignment (RoIAlign) is generally used as a module in deep learning models. It is understandable that since RoIs may have different aspect ratios and sizes, when processing RoI-related information, sometimes we want the processed data to have a unified format and dimension, so we need to convert the original RoI information to meet the expectations. For example, the original 4×3 or 5×6 matrix can be converted to a 3×3 matrix through bilinear interpolation.

[0106] 8. Nearest Neighbor Matching

[0107] In the embodiment of the present application, the nearest neighbor matching refers to finding the vector that is closest to a given vector, or the nearest neighbor, among many different vectors. This involves, on the one hand, the problem of how to evaluate the similarity of two vectors, for example, the vector inner product or Euclidean distance can be used for evaluation. On the other hand, it involves how to match efficiently, for example, the vector pool can be clustered first to reduce the complexity of the retrieval. The corresponding content can refer to the prior art and will not be repeated here.

[0108] 9. Attention Mechanism

[0109] In the field of deep learning, the attention mechanism, in a broad sense, focuses on the parts of the input data that the model considers important or related to each other, in order to achieve better model performance. Generally speaking, the attention mechanism also includes the process of similarity calculation or matching.

[0110] A common attention mechanism is that after the input data is processed by the corresponding module in the model, three types of data are obtained, namely, query, key and value. The weights are calculated using query and key, and then the value is weighted averaged based on the weight to obtain the output data.

[0111] 10. Other terms

[0112] In the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0113] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0114] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0115] 11. Electronic devices

[0116] The electronic devices of the embodiments of the present application may include handheld devices with camera functions, vehicle-mounted devices, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MID), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, personal digital assistants (PDA), handheld devices with camera functions, computing devices, terminal devices in 5G networks, or terminal devices in future evolved public land mobile networks (PLMN), etc., and the embodiments of the present application are not limited to this.

[0117] In addition, in the embodiments of the present application, the electronic device may also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network that interconnects people and machines and things.

[0118] The electronic device in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0119] In an embodiment of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an IOS operating system, or a Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0120] In combination with the above-mentioned related contents, the related technologies involved in this application are further introduced in detail below.

[0121] In photography, especially in face photography, there are many factors that may cause the quality of the photos taken to be poor, including: 1. Lighting conditions. Too strong or too weak light will affect the quality of the photos. For example, direct sunlight may cause overexposure, while insufficient light will increase noise and blur facial features; 2. Shooting angle. Inappropriate shooting angle may cause distortion, especially when shooting at close range with a wide-angle lens, facial features may be distorted; 3. Motion blur. When the subject or the camera moves during the shooting process, the photo may appear blurry; 4. Incorrect focus. If the camera does not focus on the face correctly, the facial details will appear blurred; 5. Insufficient resolution. When shooting with a low-pixel device, the photos may lack details, especially when zoomed in; 6. Occlusion. Some other objects block part of the face, such as blocking an eye or a corner of the forehead.

[0122] In some cases, the above problems can be solved by reshooting. However, due to objective conditions, the same problem will still exist even if the photos are taken again, and high-quality facial images cannot be directly obtained. Also, for some old photos, the faces in the photos may be blurred or even partially missing due to aging or the shooting conditions at the time. As a medium for recording the past, people certainly hope that the contents, including faces, can be clearly presented and preserved. Therefore, image enhancement technology is particularly important and may become more and more valuable in the future.

[0123] like Figure 1 As shown, Figure 1 This is a schematic diagram of the face image enhancement effect. Figure 1 The left eye area of ​​the face in (a) is partially missing. Figure 1 The face in (b) is blurred. Figure 1 The image in (c) contains noise in the form of black dots. Through image enhancement technology, Figure 1 (a), (b) or (c) is transformed into Figure 1 The image shown in (d) is a high-quality face image that is complete, clear and noise-free.

[0124] For the image enhancement method that transforms low-quality images into high-quality images, the following will combine Figures 2 to 3 To explain, Figure 2 Schematic diagram of the implementation of the image enhancement model Figure 1 , Figure 3 Schematic diagram of the implementation of the image enhancement model Figure 2 .

[0125] A model for implementing image enhancement methods can be referenced Figure 2 The content in Figure 2 In the figure, image 205 is a blurred low-quality face image, which is used as the input of the image enhancement model 200. In the image enhancement model 200, there are three main modules, namely encoder 202, vector codebook module 201 and decoder 203, and there is a residual network connection 204 between the encoder 202 and the decoder 203.

[0126] Among them, the vector codebook module 201 can convert or replace the feature map obtained by the encoder 202, so that it changes from the feature information representing the low-quality input image 205 to the feature information representing the high-quality image. The residual network connection 204 can directly transfer the feature information in the encoder 202 to the decoder 203, so that the output image 206 generated by the decoder can have higher fidelity, that is, more like the original input image 205. After the input image 205 is processed by the encoder 202, the vector codebook module 201 and the decoder 203, a high-quality output image 206 is obtained, which retains the facial features in the input image 205, improves clarity, eliminates noise, etc.

[0127] It should be noted that, during the reasoning process of the image enhancement model 200 , in addition to the input image 205 , no additional external information needs to be added to complete the reasoning and obtain the output image 206 .

[0128] Another model that implements image enhancement methods can refer to Figure 3 The content in Figure 3 In, with Figure 2 Similarly, image 306 is a blurred, low-quality face image, which serves as the input of the image enhancement model 300. In the image enhancement model 300, the two ends are the encoder 302 and the decoder 303, respectively, and the middle part is the intermediate network layer 304, wherein the intermediate network layer 304 can be understood as further processing the intermediate data passing through the encoder 302. In some implementations, the intermediate network layer 304 can be constructed by more pairs of encoders and decoders. There is an adaptive dictionary module 301 above the intermediate network layer 304, which forms a structure similar to the residual network with the intermediate network layer 304, or can also be regarded as a parallel structure.

[0129] Among them, the adaptive dictionary module 301 needs to rely on the prior information 305 as one of the inputs, and then combined with the feature information output by the encoder 302, the adaptive dictionary module 301 can process the two and integrate the prior information 305 into the subsequent feature information to improve the image enhancement effect.

[0130] However, there are some problems with the above-mentioned implementation methods of image enhancement, which may result in the image enhancement effect being not good enough, or even in some cases, image enhancement processing cannot be performed.

[0131] Scenario 1: Prior information is not utilized, and the effectiveness of image enhancement methods is limited.

[0132] Refer to the previous article Figure 2The image enhancement model introduced here does not rely on prior information in the use of some models in the prior art. It is understandable that the benefit brought by this is that the model itself has good versatility, that is, it can perform effective image enhancement processing in various situations.

[0133] Although the above models usually have a higher lower limit and can have better model performance in most cases, or such models are robust, compared with some models that can use prior information for image enhancement, the models that do not rely on prior information themselves generally have a lower upper limit in performance.

[0134] It is understandable that the prior information contains additional information that helps to achieve better image enhancement effects. In other words, when effective prior information can be obtained, the model that does not rely on prior information cannot use the prior information and cannot achieve better image enhancement effects. In theory, such a model still has room for improvement in performance and has not yet achieved the best ideal effect.

[0135] Scenario 2: When prior information is not available, image enhancement methods cannot be used or the effect is poor.

[0136] It can be understood that in the image enhancement method, the image enhancement effect can be better by adding the use of prior information. Figure 4 Introducing a specific form of prior information, Figure 4 It is a schematic diagram of the prior information of face parts.

[0137] like Figure 4 As shown in the figure, taking the image enhancement method of the face as an example, assuming that what you want to obtain is the prior information of the face parts, such as the position and size of the eyes, nose and mouth in the face, corresponding to Figure 4 In (a), a left eye 401, a right eye 402, a nose 403, and a mouth 404 are outlined with dotted lines.

[0138] Furthermore, some data format is needed to represent the prior information of face parts, which can be referred to Figure 4 (b). Figure 4 The left side of (b) is a table 405, in which each row is numbered corresponding to Figure 4 The part number in (a); Figure 4In (b), there is a table 406 on the right side. Each row of data in table 406 represents the prior information of a part, and corresponds to the number in table 405 on the left side. In table 406, it is assumed that the upper left corner of the face image is the origin of the coordinate system, the positive direction of the X axis is to the right, and the positive direction of the Y axis is to the bottom. The unit can be the number of pixels. Among them, the center coordinates X and Y are used to represent the pixel coordinate position of the face part in the image, corresponding to the center of the dotted rectangular box. The width and height represent the size of the corresponding part. It is also necessary to indicate which specific part the corresponding part is. In addition, similar to Figure 4 The part prior information such as (a) in the figure can be obtained using an algorithm such as target detection. In this case, a confidence score will be output. The confidence score is a probability value. The closer the confidence score is to 1, the more reliable the target detection model believes the corresponding part result is. On the contrary, the closer the confidence score is to 0, the more unreliable the target detection model believes the corresponding part result is.

[0139] For example Figure 4 In the first row of data in table 406 in (b), the center coordinate X is 100, the center coordinate Y is 100, the width is 30, the height is 20, the part category is left eye, and the confidence is 0.90. Since the confidence is high, it can be considered that there is a key part of the corresponding size at the coordinate (100, 100) in the image, and its category is left eye.

[0140] In addition to the form of prior information described above, there can be other types of prior information. For example, as users' daily lives become richer and smartphones become more powerful, more and more people are using their phones to record the beautiful moments in their lives. In the user's mobile phone album, there are usually face photos taken in the past. These face photos of the same person can also be added as a prior information to face image enhancement. Moreover, if the number of these face photos is relatively large and they are high-quality face photos, the final face image enhancement effect will be better.

[0141] However, in image enhancement methods, prior information is not always available. Figure 4 Such part prior information in the image may cause the algorithm for obtaining part prior information to fail if the original image to be used for image enhancement is too blurred, has partial missing parts, is too dark, or the angle of the face in the image is too biased. For example, if the light in the image is too dark or too blurred, the confidence of each part is very low, or some parts are difficult to detect and identify due to occlusion.

[0142] Here you can refer to Figure 3Image enhancement models in , models like this can utilize prior information, but problems may arise when prior information cannot be obtained or when incomplete prior information is not obtained.

[0143] For example, assuming that for some reason, no prior information is obtained, then the image enhancement model that relies on the prior information cannot perform reasoning at this time, which is equivalent to being completely ineffective.

[0144] For example, suppose that in some cases, the prior information is obtained, but the prior information is incomplete or inaccurate. For example, the part prior information that is originally intended to be obtained includes eyes, nose and mouth, but the prior information actually obtained only includes eyes and nose. Or the prior information obtained is inaccurate, for example, the information indicating eyes contained in the part prior information is actually the information of a decoration on a person's face. In these cases, Figure 3 The corresponding image enhancement model may not be able to reason properly when relying on complete prior information, and the quality of the final output image may be worse than that of Figure 2 The image output by the corresponding image enhancement model.

[0145] All of the above scenarios involve problems and corresponding requirements caused by prior information in image enhancement methods. For image enhancement models that do not use prior information, if prior information is available, the effect will generally be slightly worse than that of models that can use prior information. For image enhancement models that rely on prior information, if the required prior information cannot be obtained in some scenarios, or only part of the prior information can be obtained, then the image enhancement model at this time may not be able to be used, or even the effect may be worse than that of models that do not rely on prior information. The existing image enhancement solutions cannot solve the problems in these scenarios, so it is difficult to maintain relatively good image enhancement effects in various situations, and it is impossible to further improve the user experience. Therefore, a new technical solution is needed that can solve the problems in some of the above scenarios and meet the needs of users.

[0146] Based on this, the embodiment of the present application proposes the following technical concept: in an image enhancement model, two codebook modules are set, namely a part codebook module and a vector codebook module. The characteristics of these two codebook modules are that the part codebook module relies on prior information and can integrate existing information into the feature information of the model; while the vector codebook module does not rely on additional prior information, or it can be understood that some relevant information used for image enhancement processing has been encoded into the vector codebook module. When the model performs image enhancement on a low-quality image, if the corresponding prior information can be obtained, then both the part codebook module and the vector codebook module will participate in the model reasoning; conversely, if the corresponding prior information cannot be obtained, then only the vector codebook module will participate in the model reasoning, and the part codebook module that relies on the prior information will be ignored.

[0147] In order to train an image enhancement model that meets the above expectations, the embodiment of the present application also proposes a corresponding model training method. During the training process, the part codebook module is discarded with a certain probability and only the vector codebook module is used for reasoning, so that the scenario where no prior information is available during use can be simulated, so that the model can learn that image enhancement processing is also possible when only the vector codebook module is used.

[0148] The image enhancement model obtained by the above model training method can effectively enhance the model performance and improve the effect of image enhancement when prior information can be obtained, and obtain an image with both fidelity and high quality. It can also perform model reasoning when prior information cannot be obtained to obtain a high-quality image after image enhancement. In other words, the image enhancement method proposed in the embodiment of the present application is not only versatile, highly available, and robust, but also has the ability to approach the theoretical upper limit of model performance.

[0149] The technical solution provided in this application can be applied to terminal devices. The following first briefly introduces the terminal devices.

[0150] For example, Figure 5 A schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application is shown.

[0151] Figure 5A schematic diagram of the structure of a terminal device provided in an embodiment of the present application. The terminal device 500 may include a processor 510, an external memory interface 520, an internal memory 521, a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, an earphone interface 570D, a sensor module 580, a button 590, a motor 591, an indicator 592, a camera 593, a display screen 594, and a subscriber identification module (SIM) card interface 595, etc. The sensor module 580 may include a pressure sensor 580A, a gyroscope sensor 580B, an air pressure sensor 580C, a magnetic sensor 580D, an acceleration sensor 580E, a distance sensor 580F, a proximity light sensor 580G, a fingerprint sensor 580H, a temperature sensor 580J, a touch sensor 580K, an ambient light sensor 580L, a bone conduction sensor 580M, and the like.

[0152] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 500. In other embodiments of the present application, the terminal device 500 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0153] The processor 510 may include one or more processing units, for example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0154] The processor 510 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory may store instructions or data that the processor 510 has just used or cyclically used. If the processor 510 needs to use the instruction or data again, it may be called from the memory. This avoids repeated access, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0155] The terminal device 500 implements the display function through a GPU, a display screen 594, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 594 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 510 may include one or more GPUs, which execute program instructions to generate or change display information.

[0156] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 500 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0157] It is understandable that for image enhancement methods based on deep learning models, the model can be run in the GPU or in the NPU according to actual conditions and needs. Alternatively, if the process of the image enhancement method is relatively complex, for example, the input image needs to be preprocessed and the model needs to be used for reasoning, then the GPU and NPU can also be considered for use together. For example, the input image is preprocessed in the GPU, and then the preprocessed image is passed to the NPU model for reasoning to obtain an enhanced predicted image.

[0158] The software system of the terminal device 500 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 500.

[0159] For example, Figure 6 A software structure diagram of a terminal device provided in an embodiment of the present application. Figure 6As shown, the layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through interfaces. In some embodiments, the system may include an application layer, an application framework layer, an Android runtime (Android runtime) and a system library, a hardware abstraction layer (HAL) and a kernel layer. It should be noted that the embodiment of the present application is illustrated by taking the Android system as an example. In other operating systems (such as Hongmeng system, IOS system, etc.), as long as the functions implemented by each functional module are similar to those of the embodiment of the present application, the solution of the present application can also be implemented.

[0160] Among them, the application layer can include a series of application packages.

[0161] like Figure 6 As shown, the application layer may include applications such as camera, gallery, calendar, call, map, navigation, wireless local area network (WLAN), Bluetooth, music, video, short message, lock screen application, setting application, etc. Of course, the application layer may also include other application packages, such as payment application, shopping application, banking application, social application and other third-party applications, which are not limited in this application.

[0162] Among them, third-party applications can have functions such as face recognition, video calls, scanning, photo taking and video recording.

[0163] In some implementations, the image enhancement function in the embodiments of the present application can be deployed in the application layer. For example, the image enhancement function can be added to the gallery or album application. When the user browses the photos taken in the album, if he thinks that a certain picture is blurry or noisy, he can click the image enhancement function in the function bar to process it based on the model to obtain high-quality photos.

[0164] The application framework layer provides an application programming interface (API) and a programming framework for the application programs in the application layer. The application framework layer includes some predefined functions. For example, it may include an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, etc., and the embodiments of the present application do not impose any restrictions on this.

[0165] In some implementations, the image enhancement function in the embodiments of the present application may also be deployed in the framework layer. For example, after a user uses a camera application to take a photo and obtains the photo, if the user believes that the photo does not meet expectations, but it may be difficult to retake it for some reason, then the image enhancement service provided by the framework layer can be called in the camera to adjust and optimize the originally taken photo. Among them, the image enhancement service can be understood as being implemented by relying on the corresponding model deployed in the framework layer.

[0166] The system library can include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0167] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0168] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0169] OpenGL ES is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0170] SGL is a graphics engine for 2D graphics.

[0171] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0172] The HAL layer is an encapsulation of the Linux kernel driver, providing an interface to the upper layer and shielding the implementation details of the low-level hardware.

[0173] The HAL layer may include Wi Fi HAL, audio HAL, HAL layer's camera service (Camera HALServer) unit, and software code library, etc.

[0174] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, charging driver, etc.

[0175] Among them, the camera driver is the driver layer of the camera device or camera device, which is mainly responsible for the interaction with the hardware module.

[0176] The following, in conjunction with the accompanying drawings, describes in detail the technical solutions of the embodiments of the present application and how the technical solutions of the embodiments of the present application solve the above-mentioned technical problems with specific embodiments. The following specific embodiments can be implemented independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0177] First combine Figure 7 and Figure 8 A model training method provided in an embodiment of the present application is described. Figure 7 A schematic diagram of the implementation of the model training process selection provided in the embodiment of the present application, Figure 8 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 1 , Fig. 9 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 2 .

[0178] It is understandable that general machine learning models or deep learning models need to be trained before normal use to learn relevant patterns or knowledge from the prepared data. Therefore, a model training method corresponding to the image enhancement method will be introduced in detail below.

[0179] Furthermore, the model training process needs to match the actual usage scenario as much as possible. Combined with the brief description of the concept of the image enhancement method in the previous article, it can be known that two codebook modules are set in the image enhancement model, one of which depends on prior information, and when the prior information cannot be obtained, only the codebook module that does not depend on prior information is used for reasoning. Therefore, when training the model, it is necessary to find a way to make the model adapt to these two situations. The following is a detailed description of one implementation method.

[0180] like Figure 7 As shown, in the model training method provided in the embodiment of the present application, different training processes can be performed by presetting a probability threshold:

[0181] S701: Generate a random probability value.

[0182] There is a random number generator here. Each time the random number generator is called, it will output a random number between 0 and 1 according to a uniform distribution as the generated probability value.

[0183] S702: Determine whether the generated random value is less than a set probability threshold.

[0184] After the generated random value is obtained, it is compared with the pre-set probability threshold. For example, assuming that the set probability threshold is 30%, then when the random value is less than 30%, jump to S703, and vice versa, when the random value is greater than or equal to 30%, jump to S704. In the embodiment of the present application, such a preset probability threshold can also be called the first probability.

[0185] In some implementations, the judgment rule may be reversed, for example, when the random value is greater than 30%, the process jumps to S703, and vice versa. The embodiment of the present application does not impose any specific limitation on this.

[0186] S703: Only use the vector codebook module for processing.

[0187] For the specific process, please refer to Fig. 9 The content shown will be described in detail below and will not be repeated here.

[0188] S704: Use two codebook modules for processing.

[0189] For the specific process, please refer to Figure 8 The content shown will be described in detail below and will not be repeated here.

[0190] After combining the above Figure 7 From the introduction, it can be known that the training method provided in the embodiment of the present application includes two different training processes. The training process corresponding to S704 can be called the first training process, and the training process corresponding to S703 can be called the second training process.

[0191] It should be noted that Figure 7 The method of performing different training processes separately based on probability thresholds is only one of many possible methods. Theoretically, any method can be adopted as long as it can achieve similar effects. The embodiments of the present application do not impose specific restrictions on this.

[0192] It is understandable that model training requires training data. Here we first introduce the training data that needs to be prepared for training. As a supervised learning algorithm or model, in a set of training data, at least the input data of the model and the target data for comparing with the results predicted by the model and calculating the error are required. In the training method of the image enhancement model provided in the embodiment of the present application, a set of training data includes training data as the input of the model, the actual enhanced image as the target data involved in calculating the error, and training prior information as the input of the part codebook module.

[0193] In the entire training process of the model, multiple sets of training data are generally required. Assume that there are 1,000 sets of training data in total. The model adopts an iterative training method. Assume that the model needs 3 rounds (epochs), each round of training has 100 steps (steps), and each step of training will use 10 sets of training data. Among them, each training round means that the model has learned all 1,000 sets of training data.

[0194] Thus, for the above Figure 7 The method of dividing the two training processes shown can also be implemented in different ways.

[0195] In some implementations, in each training step, all training data of the training step may be trained according to the first training process or the second training process. Here, the above example of training data is still used. In each round of training of 100 steps, assuming that the probability threshold is 30%, 70 steps are executed according to the first training process, and the remaining 30 steps are executed according to the second training process.

[0196] In some implementations, in each step of training, a corresponding random probability value may be generated for each group of training data and judged. That is, in the same step of training, the first training process may be performed on some groups of training data, and the second training process may be performed on other groups of training data. For example, 10 groups of training data may be used in each step of training, and assuming that the probability threshold is 30%, 7 of the 10 groups of training data are used to perform the first training process, and the remaining 3 groups are used to perform the second training process.

[0197] In some implementations, in each training step, the first training process can be performed for multiple sets of training data in the training step, and then the second training process can be performed for a portion of the training data according to a preset probability. For example, if there are 10 sets of training data in one training step, and the probability threshold is preset to 30%, then 3 sets of training data from the 10 sets are randomly selected and the second training process is additionally performed. This ensures that the prior information of all training data can be used and learned, while also ensuring that a portion of the training data is used to perform the second training process, so that a mode that can operate normally without prior information can be learned.

[0198] After the above introduction, it is clear that the training method of the embodiment of the present application includes two training processes. Figure 8 The first training process is described in detail.

[0199] like Figure 8As shown, the image enhancement model 800 first receives the training image 806 as input, and obtains the first feature information after being processed by the encoder 804. Then, the first feature information is used as the input of the cascade codebook module 801.

[0200] In the cascade codebook module 801, two codebook modules are included, namely the part codebook module 802 and the vector codebook module 803, which are in series. In the embodiment of the present application, the part codebook module 802 can also be referred to as the first codebook module, and the vector codebook module 803 can be referred to as the second codebook module. While the part codebook module 802 receives the first feature information, it also needs to receive the training prior information 807 from the training data, wherein the training prior information 807 can be understood as having been acquired and saved when preparing the training data, and generally does not need to be temporarily acquired again. After the first feature information is processed by the part codebook module 802, the second feature information is obtained. In the second feature information, it can be understood that it contains both information from the original training image 806 and information from the training prior information 807. Then the vector codebook module processes the second feature information again and outputs the third feature information, which can also be regarded as the output of the cascade codebook module 801.

[0201] Then, the third feature information is processed by the decoder 805 to finally obtain the output of the image enhancement model 800, that is, the predicted enhanced image 809.

[0202] After completing the above-mentioned model reasoning process or processing process, it is necessary to adjust the model parameters. First, the error is calculated in combination with the predicted enhanced image 809 and the actual enhanced image 808. The error is also referred to as the first error 810 in the embodiment of the present application. As an example, a specific error calculation method will be shown below. It can be understood that the error calculation method in the embodiment of the present application can be changed according to the actual situation and is not limited to a fixed form.

[0203] Here is an explanation with the formula:

[0204] Formula 1

[0205] Formula 2

[0206] Formula 3

[0207] Formula 4

[0208]

[0209] Formula 5

[0210] Among them, in formula 1, Corresponding to the training image 806, its subscript indicates low quality. Corresponding to encoder 804, its subscript represents encoder, Corresponding to the first feature information. In formula 2, Corresponding to the cascaded codebook module 801, Corresponding to the training prior information 807, The parameters corresponding to the cascaded codebook module 801 itself can be understood as the parameters in the codebook. Corresponding to the third feature information. In formula 3, Corresponding to decoder 805, its subscript represents decoder, Corresponding to the predicted enhanced image 809, its subscript represents recovery. In formula 4, it is equivalent to integrating formulas 1 to 3, where It can be regarded as the entire image generation model. The above formulas can be understood as describing the reasoning process of the model. The following introduces the corresponding loss function, or the calculation method of the first error 810.

[0211] In formula 5 corresponding to the loss function, represents the final error, and this total error can be decomposed into multiple sub-errors. Formula 5 contains 4 sub-errors, which are explained below. , both represent the weight or importance of the sub-error in the total error, and can be adjusted according to the actual situation to make the final trained model more in line with expectations. Corresponding to the actual enhanced image 808, its subscript represents high quality.

[0212] For the first sub-error , its calculation form can be expressed as , which can be understood as and As vectors, we can directly compare the difference between the two vectors, for example, we can calculate the first-order Manhattan distance. In other words, the role of this error term is to make and The closer the better, which is a relatively strict restriction.

[0213] For the second sub-error , the subscript indicates perceptual, and its calculation form can be expressed as , among which For feature extraction networks, such as pre-trained image recognition models. Compared with the first sub-error direct comparison and ,The second sub-error compares the feature information obtained after processing by the feature extraction network.

[0214] For the third sub-error , the subscript indicates adversarial, and its calculation form can be expressed as This error comes from the generative adversarial network, which treats the image enhancement model 800 as a generator and then sets up an additional discriminator. , and then use the above cross entropy loss function to calculate the error. Getting closer to the actual enhanced image 808 or , which can achieve the effect of indistinguishable from real and fake, thus finally making the discriminator Hard to tell Is it The purpose is achieved.

[0215] For the fourth sub-error Different from the previous three sub-errors, which can be used to adjust the parameters of the entire model, Focus on the parameters of the codebook module. It can be understood that the codebook module needs to be combined with the output of the encoder 804 and its own codebook To process, the codebook is required With output One way to adjust is to make the output With codebook The closest vectors in the codebook tend to be more consistent. The specific structure of the codebook will be described in detail in the following embodiments and will not be repeated here.

[0216] After the calculation of the first error 810 is completed, the parameters of the image enhancement model 800 can be adjusted according to the corresponding optimization algorithm. For example, the gradient back propagation algorithm can be used for adjustment, that is, the partial derivative of the first error 810 is calculated by the chain rule, and the gradient of the model parameters is calculated step by step from back to front. The gradient is a vector, including size and direction. If the parameter is adjusted in the direction of its corresponding gradient, the first error 810 will increase, otherwise the first error 810 will decrease.

[0217] Through repeated training, the first error 810 can be gradually reduced. It can be understood that the smaller the first error 810 is, the better the performance or effect of the image enhancement model 800 is. When the conditions for stopping training are met, such as the number of training rounds is reached, or the first error is difficult to continue to decrease, the model training can be regarded as completed. It should be noted that the condition for stopping training mentioned here that the first error is difficult to continue to decrease generally does not refer to the first error on the training data set, but the first error on the verification data set.

[0218] Combined with the above Figure 8 The first training process has been explained. Fig. 9 It should be noted that different figures in the embodiments of the present application may contain modules with the same number, which are not the same modules, but can be understood as modules with similar properties or attributes.

[0219] Fig. 9 Part of the content and Figure 8 The contents in are the same, so we will not repeat them here, and only explain the differences. First, Fig. 9 In the cascade codebook module 801, the part codebook module 802 does not participate in the processing of the first feature information, but directly uses the first feature information as the input of the vector codebook module 803. After being processed by the vector codebook module 803, the fourth feature information is obtained as the output of the cascade codebook module 801. Similarly, the predicted enhanced image 809 and the actual enhanced image 808 output by the image enhancement model 800 are used to calculate the second error 811, and the parameters of the image enhancement model 800 are adjusted. In other words, in the second training process, since the part codebook module 802 is omitted, there is no need to train the prior information 807, and only the training image 806 is required as input.

[0220] On the basis of the model training method introduced above, the method is further summarized and described below. The model training method provided in the embodiment of the present application can perform two training processes according to a preset probability. For the first training process, the part codebook module and the vector codebook module in the cascade codebook module are involved in the processing of feature information, while in the second training process, only the vector codebook module is involved in the processing of feature information. If this method is not used for training, then when the image enhancement model is used for reasoning in a real scene, if the corresponding prior information is not obtained, the part codebook module cannot be used normally and needs to be omitted, and the feature information output by the encoder is processed separately by the vector codebook module. If the vector codebook module has never learned a similar pattern during the training process, it is highly likely that effective image enhancement cannot be performed, and the output predicted enhanced image may even be seriously separated from the input image due to the mismatch between the feature information and the codebook parameters.

[0221] That is to say, the model training method proposed in the embodiment of the present application is designed based on the scenarios and problems that may be encountered when the model is actually used. Through this model training method, the separate vector codebook module can effectively process feature information normally, so that when it is difficult to obtain prior information, the model can still be used for image enhancement. Such an image enhancement model has good versatility.

[0222] Based on the above embodiments, Fig.10 and Fig.11 Another model training method provided in the embodiment of the present application is described in detail. Fig.10 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 3 , Fig.11 Schematic diagram of the implementation of the model training method provided in the embodiment of the present application Figure 4 .

[0223] In the previous article, a model training method was introduced. In the corresponding image enhancement model, an important module is the cascade codebook module that includes the part codebook module and the vector codebook module. Figure 8 In the corresponding model training method, the part codebook module and the vector codebook module are in a series relationship. Then another form will be introduced in the embodiment of the present application. Fig.10 To explain, Fig.10 Part of the content and Figure 8 The same, no further elaboration is given here.

[0224] like Fig.10As shown, in the image enhancement model 1000, the training image 806 is processed by the encoder 804 and the fifth feature information is output. In the subsequent parallel codebook module 1001, the vector codebook module 803 and the part codebook module 802 are in a parallel relationship, and both will receive the fifth feature information in parallel and then process it separately. After processing, the vector codebook module 803 outputs the sixth feature information to the fusion processing module, and correspondingly, the part codebook module 802 outputs the seventh feature information to the fusion processing module after combining the training prior information 807. The fusion processing module 1002 can fuse the input sixth feature information and the seventh feature information and output the eighth feature information. The eighth feature information can also be regarded as the output of the parallel codebook module 1001.

[0225] The embodiment of the present application does not impose any specific restrictions on the implementation of the fusion processing module 1002, and some possible implementations are listed below.

[0226] In some implementations, assuming that the dimensions of the sixth feature information and the seventh feature information are inconsistent, the fusion processing module 1002 may first process the sixth feature information and the seventh feature information through a neural network to make their dimensions consistent. Then, the two feature information with consistent dimensions after processing may be weighted and merged to obtain the eighth feature information. The weight used for weighting may generally be a parameter that can be learned during training.

[0227] In some implementations, the sixth feature information and the seventh feature information may be concatenated in the fusion processing module 1002, and then processed by a neural network to obtain the eighth feature information. The neural network may be, for example, a multi-layer convolutional neural network and a pooling layer.

[0228] In some implementations, the dedicated fusion processing module 1002 may be omitted, that is, the sixth feature information and the seventh feature information are directly output to the decoder 805, and the decoder 805 performs part of the fusion function.

[0229] After processing the eighth feature information, the decoder 805 in the image enhancement model 1000 outputs a predicted enhanced image 809, and then calculates the first error 810 in the same manner, and adjusts the model parameters according to the first error 810. For related content, please refer to the previous Figure 8 The corresponding content will not be repeated here.

[0230] Similarly, the part codebook module 802 in the parallel codebook module 1001 may not participate in the processing of feature information during the training process. Fig.11 The content shown in Fig.11In the parallel codebook module 1001, only the vector codebook module 803 participates in the processing of feature information. After processing the training image 806, the encoder 804 outputs the fifth feature information to the parallel codebook module. After receiving and processing the fifth feature information, the vector codebook module outputs the ninth feature information to the decoder 805. It can be understood that the ninth feature information here can be equivalent to Fig.10 The sixth characteristic information in .

[0231] Next, the second error 811 is calculated based on the predicted enhanced image 809 obtained after being processed by the vector codebook module 803 and the actual enhanced image 808, and then the parameters of the image enhancement model 1000 are adjusted through a similar back propagation algorithm.

[0232] The scheme of the embodiment of the present application is summarized below. On the basis of the previous embodiment, the embodiment of the present application replaces the cascade codebook module in which the part codebook module and the vector codebook module belong to a series relationship with a parallel codebook module in which the part codebook module and the vector codebook module belong to a parallel relationship, and there are no excessive changes in other aspects. In other words, the connection relationship between the part codebook module and the vector codebook module is not fixed, and it can be determined according to actual conditions, for example, which specific connection relationship to use can be determined by the image enhancement performance in a specific scene. In addition, it can be understood that since the part codebook module and the vector codebook module can process the received feature information in parallel, when the computing resources are sufficient, a part of the reasoning time can be saved, the processing speed of the image enhancement can be accelerated, so that the predicted enhanced image can be output faster, which can also improve the user experience to a certain extent.

[0233] Based on the above embodiments, Fig.12 Another model training method provided in the embodiment of the present application is described. Fig.12 A flowchart of the model training method provided in an embodiment of the present application.

[0234] S1201. Obtain at least one set of training data, where the set of training data includes a training image, training priori information corresponding to the training image, and an actual enhanced image corresponding to the training image.

[0235] It can be understood that the embodiment of the present application adopts a model approach to perform image enhancement, and in the field of deep learning, the quantity and quality of training data are crucial. In the training data prepared in advance, the training image can be understood as a low-quality image, such as a blurred or noisy face image. The actual enhanced image is a high-quality image corresponding to the training image, for example, the actual enhanced image is a high-definition and complete face image. It should be noted that the above-mentioned training image corresponds to the actual enhanced image, which means that the content in the two images is substantially the same. For example, both are facial images of the same person with the same expression.

[0236] In terms of the specific method of preparing training data, the actual enhanced image can be prepared first, and then the actual enhanced image can be processed in various ways to transform it into a low-quality image. It is understandable that it is difficult to transform a low-quality image into a high-quality image, but it is relatively easy to transform a high-quality image into a low-quality image. For example, some image processing methods can be used, such as using the average of neighboring pixels to blur the image, or randomly adding black dots to simulate a certain noise. In addition, when using a method similar to the above to prepare training data, it is necessary to ensure that the low-quality image obtained by transforming the high-quality image has a similar pattern to the low-quality image that needs to be enhanced in the real scene, otherwise the performance of the model obtained using such training data in the real scene may be reduced.

[0237] The training prior information can be obtained through a method similar to the target detection model. In addition, considering that it is difficult for the model to directly obtain the prior information of some training images, manual annotation can also be used to verify and improve the prior information.

[0238] S1202. For any one of at least one set of training data, input a training image in the training data into a first model, wherein the first model includes a first codebook module and a second codebook module, wherein input data of the first codebook module includes prior information, and input data of the second codebook module does not include prior information.

[0239] In combination with the above embodiments, the image enhancement model includes two codebook modules, and the connection relationship between the two modules can be in many different forms, for example, Figure 8 The series connection shown can also be Fig.10 The parallel form shown.

[0240] And, similar to Figure 8The two codebook modules in the series connection form can be that the first codebook module that needs to input the prior information is in front, that is, closer to the encoder, and the second codebook module is after the first codebook module. Alternatively, the second codebook module is closer to the encoder and is after the first codebook module.

[0241] S1203. Perform model processing on the training image according to the first model to obtain at least one predicted enhanced image corresponding to the training image, wherein the second codebook module is used to participate in the model processing process, and the first codebook module is used to participate in the model processing process with a first probability.

[0242] With reference to the embodiment in the foregoing text, it is introduced that during the training process, judgment can be made by the first probability, and two training processes are performed, namely the first training process and the second training process. In the first training process, both the first codebook module and the second codebook module participate in the processing of the feature information, while in the second training process, only the second codebook module participates in the processing of the feature information.

[0243] S1204: Adjust model parameters of the first model according to at least one predicted enhanced image corresponding to the training image and the actual enhanced image corresponding to the training image.

[0244] This step can refer to the previous article, for example Figure 8 The content of adjusting model parameters according to model errors described in will not be repeated here.

[0245] Here we summarize the technical solutions proposed in this application. In an embodiment of the present application, a method for training a model in the field of image enhancement technology is described. The model includes two important codebook modules, both of which can enhance the image enhancement effect. The first codebook module depends on prior information, and the second codebook module does not depend on prior information. During training, the first codebook module is not involved in the processing of feature information according to the first probability, and only the second codebook module is used. That is to say, such a training process does not depend on prior information, and can be used to simulate situations where it is difficult to obtain prior information when the model is used in real scenes in the future.

[0246] In this way, on the one hand, during the training process, the two codebook modules are jointly involved in the processing of feature information, so as to carry out corresponding learning. Due to the incorporation of additional prior information, the final image enhancement effect can be better. On the other hand, during the training process, the second codebook module is also allowed to independently participate in the processing of feature information and carry out learning. In this case, since it does not rely on prior information, the model can have a wider range of usage scenarios in actual use, and has good versatility and robustness.

[0247] Based on the above embodiments, Fig.13 An image enhancement method provided in an embodiment of the present application is introduced. Fig.13 A schematic diagram of implementing image enhancement in a terminal provided in an embodiment of the present application.

[0248] In the embodiment of the present application, it is assumed that an image enhancement model has been trained by implementing the model training method provided in the foregoing text. Then, the image enhancement model can be deployed in the terminal device, and the specific deployment method is not specifically limited in the embodiment of the present application. For example, the model can be bound to an application, and the application can run, manage and update the image enhancement model; the model can also be used as a system function service. When other applications want to implement related functions, they can run the image enhancement model by calling the corresponding interface of the service.

[0249] The following will be combined Fig.13 Here, we will describe an implementation of an image enhancement method in a terminal device. Here, we take the photo album application in the terminal device as an example, and assume that the photo album application can call the reasoning function of the image enhancement model. It is understandable that other applications, such as camera, video or social applications, can also perform image enhancement through similar methods.

[0250] like Fig.13 As shown, in Fig.13 In (a), the user clicks on a reduced image 1301 in the photo album application and jumps to Fig.13 The viewing interface shown in (b).

[0251] exist Fig.13 In the viewing interface of (b), it can be seen that a relatively blurred face image 1302 is displayed. Now the user hopes to enhance the face image to make it clear. Therefore, the user can click the edit control 1303 at the bottom of the interface, and then the face image 1302 can be enhanced. Fig.13 It should be noted that the interface and the control layout in the interface provided in the embodiment of the present application are only used as an example, and may have various designs and forms, and are not specifically limited thereto.

[0252] After the user clicks the edit control 1303, the following Fig.13 The user can then click on the enhancement control 1304 to perform various editing operations shown in (c). After receiving the signal, the album application can send the image data of the face image 1302 to the deployed image enhancement model. After the image enhancement model completes the inference, it returns the predicted enhanced image to the album application, and you can see Fig.13 The picture shown in (d).

[0253] exist Fig.13 In (d), it can be seen that after being processed by the image enhancement model, the new face image 1305 displayed in the interface is consistent with the original face image 1302 in morphology, and has clearer details, thereby effectively achieving enhancement of the face image.

[0254] The embodiment of the present application illustrates a method of using image enhancement from the perspective of an application program of a terminal device. Generally speaking, users do not need to care about how the specific technical details behind image enhancement are implemented, which can be developed and packaged by the R&D personnel of the terminal device or the application program. When the user thinks that the photographed photo or an existing image is blurry, has missing or noisy problems, the image enhancement function can be used to process it, so that the image is transformed from low quality to high quality. Moreover, this transformation is different from general photo beautification processing. If general image processing is likened to modification based on the original image, then the image enhancement method proposed in the embodiment of the present application is a kind of improvement and optimization of information, which can bring a unique user experience to users.

[0255] Based on the above embodiments, Figure 14~Figure 20 An image enhancement method provided in an embodiment of the present application is described in detail. Fig.14 Schematic diagram of the implementation of the image enhancement model Figure 3 , Fig.15 Schematic diagram of the implementation of the image enhancement model Figure 4 , Fig.16 Schematic diagram for obtaining prior information of face parts. Fig.17 Schematic diagram of the implementation of the part codebook module. Fig.18 It is a schematic diagram of the implementation of the vector codebook module. Fig.19 Schematic diagram of the implementation of the codebook information module Figure 1 , Fig. 20 Schematic diagram of the implementation of the codebook information module Figure 2 .

[0256] It can be understood that the image enhancement method proposed in the embodiment of the present application is implemented based on a model, and the image enhancement model can be obtained using the model training method described above. Fig.14 and Fig.15 In this paper, the implementation process of the model-based image enhancement method is demonstrated.

[0257] like Fig.14As shown, the input image 1406 can be understood as a low-quality image, which is intended to be processed by the image enhancement model 800. In the image enhancement model 800, after the encoder 804 receives the input image 1406, it performs corresponding processing and then outputs the first feature information. Next, the cascaded codebook module 801 receives the first feature information, which can also be understood as the part codebook module 802 in the cascaded codebook module 801 receiving the first feature information, and after processing in combination with the input target prior information 1407, outputs the second feature information. Then the vector codebook module receives the second feature information, processes it accordingly and outputs the third feature information. Finally, after the decoder 805 processes the third feature information, the target enhanced image 1409 is obtained.

[0258] As you can see, the overall processing above is basically the same as the model training method in the previous article. For more relevant content, please refer to Figure 8 , Fig. 9 The corresponding embodiments will not be described in detail here. There are several differences, which will be explained in detail below.

[0259] First, for example, Figure 8 In the corresponding embodiment, the prior information that the part codebook module 802 relies on, that is, the training prior information 807 is not temporarily acquired during the training process, but is acquired when the training data is prepared before the training process, and may have gone through other models and manual labeling processes. Fig.14 In the method shown, the prior information that the part codebook module 802 relies on, namely the target prior information 1407, is temporarily obtained during the model inference process. A method for obtaining the prior information can refer to Fig.16 The content in.

[0260] like Fig.16 As shown, the input image 1406 is a blurred low-quality face image. Fig.14 The input image 1406 in can be understood as the same image. Fig.16 It is assumed that a target detection model 1601 is used to process the input image 1406 and output the corresponding prior information. The output prior information can be in different forms, such as the position, size or shape of each part of the face, or some key point information of the face. Fig.16The prior information 1603 in the image detection model 1601 is displayed in a table to show the results obtained by the target detection model 1601. In addition, the process of the target detection model 1601 obtaining the prior information can be before the image enhancement model 800 performs reasoning; it can also be synchronized with the image enhancement model 800 in parallel, and when the image enhancement model 800 is reasoning to the part codebook module 802, if the target detection model 1601 has not completed reasoning, then it is necessary to pause the reasoning and wait. Figure 4 There is relevant content in the corresponding introduction, which will not be repeated here.

[0261] It should be noted that in Fig.16 The method for obtaining prior information in the image enhancement model inference process shown in is different from obtaining training prior information in the model training method. The target prior information 1407 may not be obtained. For example, due to the fact that some areas in the input image 1406 are too blurred or the light is not good, the target detection model 1601 may not be able to effectively identify the face part to be detected, and the confidence in the output result will be correspondingly low.

[0262] Generally speaking, when using the image enhancement model for reasoning, some corresponding hyperparameters are set to ensure that the model can run well, which may include hyperparameter information on whether the target prior information 1407 is available. For example, a confidence threshold of the prior information can be set. Fig.16 In the result obtained by the method shown, if the confidence is lower than the preset confidence threshold, then the information in the result can be considered unreliable and cannot be used as target prior information 1407, that is, the prior information acquisition fails in this case.

[0263] When the prior information cannot be obtained normally, or is judged to have failed to be obtained, the image enhancement model 800 can be changed to Fig.15 Inference is performed in the form shown. This is also different from the previous model training embodiment. In the previous model training method, a probability is preset and the first training process or the second training process is performed on the model using a generated random number or other method. That is to say, in the model training method, the switching of the image enhancement model reasoning method can be actively controlled by humans, while in the image enhancement method provided in the embodiment of the present application, the switching of the image enhancement model reasoning method is determined by objective factors. For example, when the target prior information 1407 can be obtained normally, use the following method: Fig.14The method shown in the figure performs model inference, that is, both the part codebook module 802 and the vector codebook module 803 are involved in the processing of feature information. When the target prior information 1407 is judged to be failed to be obtained, an indication signal can be transmitted to the image enhancement model 800, and the indication signal is used to make the image enhancement model 800 use Fig.15 The method described performs model inference, that is, only the vector codebook module 803 is involved in the processing of feature information.

[0264] Based on the above introduction on how to use the image enhancement model 800 as a whole, some key modules of the image enhancement model 800 itself are described in detail.

[0265] For an implementation of the part codebook module 802 that relies on prior information, please refer to Fig.17 .exist Fig.17 In , the input image 1406 is processed by the encoder 804 to obtain a feature map 1706, where the feature map 1706 can also be understood as the feature information mentioned above. Fig.17 The part contained in the dotted box 1712 can be regarded as the internal structure of the part codebook module. The feature map 1706 will be input into the alignment (RoIAlign) module 1701, and the alignment module 1701 also receives the target prior information 1407 as input. The alignment module 1701 will obtain the feature map of the corresponding area or part in the feature map 1706 through the target prior information 1407, and then perform dimension conversion on the obtained part feature map to obtain the aligned part feature map, such as Fig.17 The left eye feature map 1702, the right eye feature map 1703, the nose feature map 1704 and the mouth feature map 1705. These part feature maps are matrices or tensors, but they can also be regarded as vectors, similar to a matrix 1707 composed of vectors, which can more conveniently participate in the subsequent reasoning process.

[0266] Next, the part feature map will be used as the input of the codebook information module 1708. In the codebook information module 1708, it can be understood that the codebook information module 1708 will select the information in the codebook and integrate it into the part feature map. Among them, the specific implementation method of the codebook information module 1708 will be described in detail later. Then, after being processed by the codebook information module 1708, it is necessary to use the reverse alignment (ReverseRoIAlign) module 1709 to restore the unified dimension of the part feature map to the original dimension.

[0267] Finally, through a structure similar to a residual network, the intermediate feature map 1706 and the part feature map output by the reverse alignment module 1709 are fused to output a new feature map 1711 as feature information for subsequent model module processing. One fusion method, for example, is to replace or overwrite the corresponding data in the intermediate feature map 1706 with the part feature map.

[0268] A specific internal structure of the part codebook module 802 is introduced above, and an internal structure of the vector codebook module 803 is described below. Fig.18 As shown, the content in the dotted box 1801 can be regarded as the internal structure of the vector codebook module 803. Since the vector codebook module itself does not rely on external prior information, the internal structure can be simpler than the part codebook module 802. In the dotted box 1801, the feature map 1706 is first processed by the codebook information module 1802, and after being integrated into the relevant feature information in the codebook information module 1802, it is fused with the feature map 1706 in the fusion module 1803. Among them, the fusion method can refer to the prior technology, and the fusion module 1803 is also optional. Fig.18 The rest of the content is the same as Fig.17 The corresponding contents in are basically the same and will not be repeated here.

[0269] After the above combination Fig.17 and Fig.18 The internal structure of the part codebook module and the vector codebook module is explained below. It can be seen that there is a codebook information module with similar functions inside the two types of modules. The function of the codebook information module mentioned above is to integrate the information in the codebook into the feature information. In order to better understand the implementation and function of the codebook information module, Fig.19 and Fig. 20 Provide explanation.

[0270] It can be understood that there are many ways to implement the information codebook module. Two ways are provided in the embodiment of the present application. Other implementations with corresponding functions can also be used to construct the information codebook module. Fig.19 As shown, in Fig.19 There is a nearest neighbor matching module 1900, which has two inputs, one is feature information 1901, and the other is codebook 1902. Among them, feature information 1901 can be understood as feature information output by a module in the previous image enhancement model of the information codebook module, and codebook 1902 is part of the parameters of the information codebook module itself.

[0271] Furthermore, the feature information 1901 can be regarded as a tensor composed of multiple vectors, for example Fig.19The feature information 1901 in can be a 4×4×M tensor. The codebook 1902 can be regarded as a matrix composed of multiple vectors, such as Fig.19 The codebook 1902 in is an N×M matrix, and the numbers on the left are used to identify the number of each vector. It can be seen that the dimensions of the vectors in the feature information 1901 and the codebook 1902 are consistent, both M, which is for the subsequent corresponding processing in the nearest neighbor matching module 1900.

[0272] In the nearest neighbor matching module 1900, the similarity between the vector in the feature information 1901 and the vector in the codebook 1902 can be evaluated by some similarity calculation methods. For example, the inner product, Euclidean distance, Manhattan distance or cosine similarity of two vectors can be calculated. After determining a similarity calculation method, for each vector in the feature information 1901, it is necessary to find the corresponding most similar vector in the codebook 1902 to replace the corresponding vector in the feature information 1901. This process can be understood in conjunction with the matrix 1904. For example, for the vector in the upper left corner of the feature information 1901, after calculating the similarity with all the vectors in the codebook 1902, it is found that it is most similar to the vector numbered 10, so this position records a number 10. By analogy, the most similar vector corresponding to the first row and second column in the feature information 1901 is numbered 42, the most similar vector corresponding to the first row and second column is numbered 1, and so on. After obtaining the number matrix 1904 of the most similar vectors, a new feature information 1903 can be constructed in combination with the code book 1902. The dimension of the feature information 1903 is consistent with the feature information 1901. The vectors at each position therein correspond to the number matrix 1904 and are copied from the code book 1902.

[0273] The above process can be expressed using the formula, as follows:

[0274] Formula 6

[0275] in, represents one of the vectors in the feature information 1901, represents a vector in codebook 1902, The vector representing the corresponding position in the feature information 1903 is obtained by The nearest neighbor matching calculation is performed in codebook 1902.

[0276] It can be understood that the feature information 1901 contains the information of the low-quality image originally input, and the codebook 1902 contains the information of the corresponding high-quality image. Assuming that a certain vector of the feature information 1901 represents the blurred eyes in the original face image, after similarity comparison, the information of the eyes that are similar but high-definition can be obtained from the codebook 1902. In other words, through such a calculation process, the image enhancement function can be effectively realized.

[0277] Another implementation of the information codebook module can be found in Fig. 20 The content shown. Fig. 20 What is shown is, in general, a method based on the attention mechanism, in which the parts contained in the dashed boxes 2007 and 2008 can be considered to be equivalent to Fig.19 In codebook 1902. First, after the matrix calculation of the feature information vector 2001 after the Q (Query) matrix 2002, the Q vector 2003 is obtained. Then, the Q vector 2003 and multiple K (Key) vectors are calculated together in the similarity calculation and normalization module 2004. For example, one calculation method is to calculate the inner product of Q vector 2003 and K vector 1, K vector 2 and K vector 3 respectively, and then normalize, and limit the calculated value to between 0 and 1, so that weight 1 to weight 3 are obtained, and the sum of the three weights is 1. Then, the three weights and the three V (Value) vectors are calculated by the weight merging module 2005, and the feature information vector 2006 is output. Among them, one calculation method of the weight merging module 2005 can be to first multiply weight 1 to weight 3 by V vector 1 to V vector 3 respectively, and then add and merge the three V vectors to obtain a new vector of the same dimension. Fig. 20 Compared with the implementation of the information codebook module Fig.19 The method shown is relatively complex and can be selected based on specific scenarios, experimental results, or resource configuration conditions.

[0278] The above is a detailed description of the image enhancement method provided by the embodiment of the present application, including how to use the model for reasoning. During the reasoning process, an additional sub-model can be used to obtain the target prior information. And the method of reasoning can be selected based on whether the target prior information is effectively obtained. When the target prior information can be obtained, both the part codebook module and the vector codebook module are used, otherwise only the vector codebook module is used. In addition, the internal structure and implementation of the part codebook module and the vector codebook module are also described in detail, and the effectiveness of the image enhancement is explained accordingly in combination with the structure of the codebook.

[0279] The image enhancement model provided in the embodiment of the present application does not limit the specific implementation methods of the two important codebook modules. In principle, only one of the modules needs to be able to make full use of prior information. On this basis, different reasoning processes can be flexibly selected according to the situation in the real scene to ensure the versatility of the image enhancement model. And when prior information can be obtained, the details in the image can be better restored and enhanced to achieve the best possible image enhancement effect.

[0280] Based on the above embodiments, Fig.21 and Fig. 22 An image enhancement method provided in an embodiment of the present application is described. Fig.21 A schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application, Fig. 22 A schematic diagram of the implementation of obtaining external image prior information provided in an embodiment of the present application.

[0281] S2101. Obtain a first image and target prior information corresponding to the first image.

[0282] For this step, reference may be made to the description of the relevant contents in the above embodiment, which will not be repeated here.

[0283] For target prior information, in addition to similar Fig.16 The method of obtaining the prior information of the face part from the first image through the target detection model is shown, and other types of target prior information may also be used. The first image corresponds to the input image in the above embodiment, that is, the image that is desired to be enhanced or restored.

[0284] It is understandable that, still taking the image enhancement of human face as an example, in theory, the same person may look slightly different at different times or in different states, but the features of the parts of his face are consistent. That is to say, in addition to obtaining the target prior information from the first image, the target prior information can also be obtained from the image that belongs to the same person as the first image. Fig. 22 A corresponding method is described.

[0285] like Fig. 22As shown, the person in the face image 2202 is assumed to be the same person as the person in the first image, and after being processed by a feature information extraction model 2201, a feature information matrix 2203 can be obtained. The feature information matrix 2203 can be regarded as composed of multiple vectors, each of which contains specific face information. In addition, the feature information matrix 2203 can be adapted to the codebook 1902 introduced in the above embodiment, and the feature information matrix 2203 can be an extension of the codebook 1902. In this way, after the codebook 1902 is expanded using the face image 2202, the codebook module can integrate more accurate prior information in the process of processing the feature information of the first image, so as to achieve a better enhancement effect.

[0286] It should be noted that the face image 2202 can be one or more, and high-quality face images are used as much as possible, such as clear, unobstructed, and noise-free images. In addition, for the feature information extraction model 2201, its structure can be composed of a multi-layer convolutional neural network and a fully connected network, etc., and the embodiment of the present application does not impose specific restrictions on this. In order to make the feature information matrix 2203 processed by the feature information extraction model 2201 match the code book 1902, the feature information extraction model 2201 can be added to the training during the training of the image enhancement model. It is understandable that the training data prepared at this time should also be supplemented with a portion of images such as the face image 2202 that belong to the same person as the training image.

[0287] S2102, inputting the first image and target prior information into a first model, wherein the first model includes a first codebook module and a second codebook module, wherein input data of the first codebook module includes prior information, and input data of the second codebook module does not include prior information.

[0288] Corresponding to the embodiments related to the model training method in the previous text, the first codebook module and the second codebook module in the embodiments of the present application can be either in series or in parallel. The specific model architecture can be designed according to actual conditions, test results or related needs, and no specific restrictions are made here.

[0289] For this step, reference may be made to the description of the relevant contents in the above embodiment, which will not be repeated here.

[0290] S2103. Perform model processing on the first image according to the first model to obtain a target enhanced image corresponding to the first image, wherein when target prior information is available, the first codebook module participates in the model processing process, and when the target prior information is not available, the second codebook module does not participate in the model processing process.

[0291] In this step, since the second codebook module depends on the target prior information, when the target prior information is not available, the second codebook module does not participate in the model processing process, and only the first codebook module is used to participate in the processing. It should be noted that in some existing image enhancement technologies that use prior information, the judgment that the target prior information is not available is relatively strict. Taking the facial part prior information introduced in the previous article as an example, assuming that the target detection model is used to detect the position, size, etc. of the facial part, then the confidence item is generally included in the output result. It can be understood that assuming that the expected output result includes four sets of data such as two eyes, a nose and a mouth, in some cases, the target detection model may only output three sets of data, that is, the output data is incomplete; in other cases, in the result output by the target detection model, the confidence of some data is low, but the confidence of some data is relatively high.

[0292] In the existing technology that relies on prior information, for the above-mentioned situation where the prior information is incomplete, it may be directly regarded as that the prior information has not been successfully obtained, and the image enhancement processing cannot be performed. The possible reason is that the effect of image enhancement is strongly dependent on the prior information, and when the prior information is incomplete, it is likely to cause the effect of image enhancement to decrease. For example, based on the prior information of the human face, if the prior information about the nose is missing, then the nose part of the final output image may not match the expected details.

[0293] In the image enhancement method proposed in the embodiment of the present application, since the first codebook module and the second codebook module are introduced at the same time, and in some cases, the first codebook module can be regarded as a bottom-up module of the second codebook module to ensure the lower limit of the model, and the second codebook module can also be regarded as a performance improvement module of the first codebook module, so that the model can have a higher upper limit. That is to say, the image enhancement method proposed in the embodiment of the present application can allow only the first codebook module to be used for reasoning when the prior information is completely unavailable, and can also use the first codebook module and the second codebook module together for reasoning when the prior information is partially available. At this time, it can be understood that the second codebook module incorporates more prior information into the feature information as much as possible, and more prior information may result in better image enhancement effects, and the final output image will not have obvious problems that do not meet expectations due to the lack of some prior information.

[0294] For more relevant contents of this step, please refer to the above embodiment, which will not be described again here.

[0295] The image enhancement method proposed in the embodiment of the present application is summarized and described below. First, an image enhancement model with two important codebook modules is adopted, in which the second codebook module relies on prior information. In terms of the acquisition of prior information, it is not limited to being based only on the first image, but can also be acquired through other images associated with the first image, which broadens the way to obtain prior information, and more effective prior information can also bring better image enhancement effects. In terms of processing existing information, the image enhancement method proposed in the embodiment of the present application also makes full use of it, that is, when the prior information is not fully acquired, the second codebook module can also be used to participate in the processing of feature information. This is because the first codebook module has the ability to ensure the effect of the image enhancement model alone, and the second codebook module can be regarded as adding to the first codebook module. The more effective prior information is integrated, the better the image enhancement effect. In other words, the image enhancement method proposed in the embodiment of the present application can improve the image enhancement effect as much as possible according to the actual situation while ensuring the basic image enhancement function, and has both versatility and high performance.

[0296] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0297] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0298] The model training method and the image enhancement method of the embodiment of the present application have been described above, and the device for executing the above method is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device can execute the steps in the above model training method and the image enhancement method.

[0299] The above model training method and image enhancement method can be applied to electronic devices with image processing functions. The electronic devices include terminal devices, and the specific device form of the terminal device can refer to the above related descriptions, which will not be repeated here.

[0300] In one implementation, an embodiment of the present application provides an electronic device, Fig.23 Schematic diagram of the hardware structure of the electronic device.

[0301] like Fig.23As shown, the electronic device 230 includes: a processor 2301 and a memory 2302; the memory 2302 stores computer-executable instructions; the processor 2301 executes the computer-executable instructions stored in the memory 2302, so that the electronic device 230 executes the above method.

[0302] When the memory 2302 is independently provided, the electronic device further includes a bus 2303 for connecting the memory 2302 and the processor 2301 .

[0303] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0304] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0305] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage device, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of medium. Disks and optical disks as used herein include optical disks, laser disks, optical disks, digital versatile discs (DVD), floppy disks and Blu-ray discs, where disks usually reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0306] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0307] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0308] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A model training method, characterized in that: include: Acquire at least one set of training data, wherein the set of training data includes a training image, training priori information corresponding to the training image, and an actual enhanced image corresponding to the training image; For any one of the at least one set of training data, input a training image in the training data into a first model, wherein the first model includes a first codebook module and a second codebook module, wherein input data of the first codebook module includes prior information, and input data of the second codebook module does not include prior information; Performing model processing on the training image according to the first model to obtain at least one predicted enhanced image corresponding to the training image, wherein the second codebook module always participates in the process of the model processing, and the first codebook module does not participate in the process of the model processing under a first probability, and participates in the process of the model processing under a non-first probability; The model parameters of the first model are adjusted according to at least one predicted enhanced image corresponding to the training image and the actual enhanced image corresponding to the training image.

2. The method according to claim 1, characterized in that The performing model processing on the training image according to the first model to obtain at least one predicted enhanced image corresponding to the training image includes: performing a first training process on the training image according to the first model to obtain a first predicted enhanced image, wherein in the first training process, the first codebook module performs data processing according to the training image and training prior information corresponding to the training image, and the second codebook module performs data processing according to the training image; Performing a second training process on the training image according to the first model and the first probability to obtain a second predicted enhanced image, wherein in the second training process, the second codebook module performs data processing according to the training image, and the first codebook module does not participate in the model processing process; The at least one predicted enhanced image includes the first predicted enhanced image and the second predicted enhanced image.

3. The method according to claim 2, characterized in that The adjusting the model parameters of the first model according to at least one predicted enhanced image corresponding to the training image and the actual enhanced image corresponding to the training image includes: determining a first error according to the first predicted enhanced image and the actual enhanced image, and adjusting a model parameter of the first model according to the first error; When the first model performs the second training process, a second error is determined according to the second predicted enhanced image and the actual enhanced image, and a model parameter of the first model is adjusted according to the second error.

4. The method according to any one of claims 1 to 3, characterized in that: The first model also includes an encoder and a decoder, wherein the encoder is used to process the training image to obtain first feature information corresponding to the training image; The first codebook module and the second codebook module are in a series relationship; In the case where the first codebook module participates in model processing, the first codebook module is used to output second feature information according to the first feature information and training prior information corresponding to the training image; the second codebook module is used to output third feature information according to the second feature information; When the first codebook module does not participate in model processing, the second codebook module is used to output fourth feature information according to the first feature information; The decoder is used to process the third feature information or the fourth feature information to obtain a predicted enhanced image corresponding to the training image.

5. The method according to any one of claims 1 to 3, characterized in that: The first model also includes an encoder and a decoder, wherein the encoder is used to process the training image to obtain fifth feature information corresponding to the training image; The first codebook module and the second codebook module are in a parallel relationship; In the case where the first codebook module participates in model processing, the first codebook module is used to output sixth feature information according to the fifth feature information and the training prior information corresponding to the training image; the second codebook module is used to output seventh feature information according to the fifth feature information; The first model is further used to fuse the sixth feature information and the seventh feature information to obtain eighth feature information; When the first codebook module does not participate in model processing, the second codebook module is used to output ninth feature information according to the fifth feature information; The decoder is used to process the eighth feature information or the ninth feature information to obtain a predicted enhanced image corresponding to the training image.

6. An image enhancement method, characterized in that: include: Acquire a first image and target prior information corresponding to the first image; Inputting the first image and the target prior information into a first model, wherein the first model includes a first codebook module and a second codebook module, wherein input data of the first codebook module includes the prior information, and input data of the second codebook module does not include the prior information; Performing model processing on the first image according to the first model to obtain a target enhanced image corresponding to the first image, wherein when the target prior information is available, both the first codebook module and the second codebook module participate in the model processing process, and when the target prior information is not available, the first codebook module does not participate in the model processing process; Wherein, the first model is trained according to the method according to any one of claims 1-5.

7. The method according to claim 6, characterized in that The method further comprises: Processing the first image according to the first algorithm unit to obtain target prior information corresponding to the first image; Wherein, the first algorithm unit is also used to output the confidence of the target prior information. When the confidence is greater than or equal to a preset threshold, the target prior information is available. When the confidence is less than the preset threshold, the target prior information is unavailable.

8. The method according to claim 6 or 7, characterized in that: The first model also includes an encoder and a decoder, wherein the encoder is used to process the first image to obtain first feature information corresponding to the first image; The first codebook module and the second codebook module are in a series relationship; When the target prior information is available, the first codebook module is used to output second feature information according to the first feature information and the target prior information corresponding to the first image, and the second codebook module is used to output third feature information according to the second feature information; When the target priori information is unavailable, the second codebook module is used to output fourth feature information according to the first feature information; The decoder is used to process the third feature information or the fourth feature information to obtain a target enhanced image corresponding to the first image.

9. The method according to claim 6 or 7, characterized in that: The first model further includes an encoder and a decoder, the encoder is used to process the first image to obtain fifth feature information corresponding to the first image; the first codebook module and the second codebook module are in parallel relationship; When the target prior information is available, the first codebook module is used to output sixth feature information according to the fifth feature information and the target prior information corresponding to the first image; the second codebook module is used to output seventh feature information according to the fifth feature information; The first model is further used to fuse the sixth feature information and the seventh feature information to obtain eighth feature information; When the target priori information is unavailable, the second codebook module is used to output ninth feature information according to the fifth feature information; The decoder is used to process the eighth feature information or the ninth feature information to obtain a target enhanced image corresponding to the first image.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to perform the method as claimed in any one of claims 1 to 9.

11. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device is caused to perform the method as claimed in any one of claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to perform the method according to any one of claims 1 to 9.

Images