Image enhancement model training method, image enhancement method and related equipment

Abstract

The invention provides an image enhancement model training method, an image enhancement method and related equipment, and relates to the technical field of image processing. The method comprises the following steps: acquiring a low-resolution image sample and a corresponding high-resolution image sample; based on the low-resolution image sample and the high-resolution image sample, training a re-parameterized convolution filter to obtain a trained re-parameterized convolution filter; performing re-parameterization operation on the trained re-parameterization convolution filter to obtain a target convolution filter; and obtaining an image enhancement model by using the target convolution filter. The image enhancement model obtained by the method can accelerate the image enhancement processing speed on the premise of keeping the image enhancement effect, and the image processing performance of the equipment can be improved by applying the image enhancement model to the mobile equipment.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image enhancement model training method, an image enhancement method, an image enhancement model training device, an image enhancement device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In recent years, with the rapid development of short video applications, users' demand for video quality improvement technology has become more and more obvious, especially in the use scenarios of mobile devices. Image enhancement technology aims to improve the clarity, contrast and detail of the image and enhance the user's visual experience. In related technologies, image enhancement methods include interpolation methods, deep learning methods based on convolutional neural networks, and methods based on machine learning.

[0003] However, when processing complex or rapidly changing image data, the interpolation algorithm may not provide accurate interpolation results, resulting in image distortion or quality degradation. Also, given the requirements of mobile devices in terms of computing resources, energy consumption control, and real-time performance, it is challenging to directly apply deep learning methods based on convolutional neural networks or machine learning methods to mobile devices. Summary of the invention

[0004] The present disclosure provides an image enhancement model training method, an image enhancement method, an image enhancement model training device, an image enhancement device, an electronic device, a computer-readable storage medium and a computer program product to overcome the above-mentioned problems or at least partially solve the above-mentioned problems.

[0005] According to one aspect of an embodiment of the present disclosure, there is provided an image enhancement model training method, the method comprising: obtaining a low-resolution image sample and a high-resolution image sample corresponding to the low-resolution image sample; training a reparameterized convolution filter based on the low-resolution image sample and the high-resolution image sample to obtain a trained reparameterized convolution filter; performing a reparameterization operation on the trained reparameterized convolution filter to obtain a target convolution filter; and obtaining an image enhancement model using the target convolution filter.

[0006] In some embodiments of the present disclosure, the reparameterized convolution filter is trained based on the low-resolution image samples and the high-resolution image samples to obtain the trained reparameterized convolution filter, including: upsampling the low-resolution image samples to obtain upsampled image samples; classifying the pixels in the upsampled image samples to obtain a pixel set; inputting the pixel set into the reparameterized convolution filter to obtain a feature map; training the reparameterized convolution filter according to the feature map and the image area in the high-resolution image samples corresponding to the pixel set to obtain the trained reparameterized convolution filter.

[0007] In some embodiments of the present disclosure, the reparameterized convolution filter is trained according to the feature map and the image area corresponding to the set of pixel points in the high-resolution image sample to obtain the trained reparameterized convolution filter, including: calculating the loss value between the feature map and the image area corresponding to the set of pixel points in the high-resolution image sample based on a preset loss function; adjusting the parameters of the reparameterized convolution filter according to the loss value until the calculated loss value is less than a preset value, thereby obtaining the trained reparameterized convolution filter.

[0008] In some embodiments of the present disclosure, the re-parameterized convolution filter includes: a first branch and multiple second branches arranged in parallel; the first branch is provided with a first convolution filter; the second branch is provided with a second convolution filter and a target filter, and the target filter includes one of a third convolution filter, a Sobel filter and a Laplace filter.

[0009] In some embodiments of the present disclosure, the method further includes: setting initial weights for the re-parameterized convolution filter according to the weights of a trained reference convolution filter.

[0010] In some embodiments of the present disclosure, the initial weight is set for the re-parameterized convolution filter based on the weight of the trained reference convolution filter, including: in response to the convolution kernel size of the trained reference convolution filter being the same as the convolution kernel size of the first convolution filter and the convolution kernel size of the third convolution filter, the weight of the trained reference convolution filter is set to the initial weight of the first convolution filter and the initial weight of the third convolution filter; in response to the convolution kernel size of the trained reference convolution filter being different from the convolution kernel size of the second convolution filter, the weight center of the trained reference convolution filter is set to the initial weight of the second convolution filter.

[0011] In some embodiments of the present disclosure, the method further includes: acquiring a reference convolution filter; training the reference convolution filter based on the low-resolution image samples and the high-resolution image samples to obtain the trained reference convolution filter.

[0012] According to another aspect of an embodiment of the present disclosure, there is provided an image enhancement method, the method comprising: acquiring a target image; performing upsampling processing on the target image to obtain an upsampled image; classifying pixels in the upsampled image to obtain a target pixel set; inputting the target pixel set into a target convolution filter in an image enhancement model to obtain a target feature map of the target pixel set; the image enhancement model is generated according to the above-mentioned image enhancement model training method; and according to the target feature map of the target pixel set, obtaining an enhanced image corresponding to the target image.

[0013] According to another aspect of an embodiment of the present disclosure, there is provided an image enhancement model training device, the device comprising: a sample acquisition module, configured to acquire low-resolution image samples and high-resolution image samples corresponding to the low-resolution image samples; a training module, configured to train a re-parameterized convolution filter based on the low-resolution image samples and the high-resolution image samples to obtain a trained re-parameterized convolution filter; a model generation module, configured to perform a re-parameterization operation on the trained re-parameterized convolution filter to obtain a target convolution filter; and using the target convolution filter to obtain an image enhancement model.

[0014] In some embodiments of the present disclosure, the training module is further configured to: upsample the low-resolution image samples to obtain upsampled image samples; classify the pixels in the upsampled image samples to obtain a pixel set; input the pixel set into the reparameterized convolution filter to obtain a feature map; train the reparameterized convolution filter according to the feature map and the image area in the high-resolution image samples corresponding to the pixel set to obtain the trained reparameterized convolution filter.

[0015] In some embodiments of the present disclosure, the training module is further configured to: calculate the loss value between the feature map and the image area corresponding to the set of pixel points in the high-resolution image sample based on a preset loss function; adjust the parameters of the reparameterized convolution filter according to the loss value until the calculated loss value is less than a preset value, thereby obtaining the trained reparameterized convolution filter.

[0016] In some embodiments of the present disclosure, the re-parameterized convolution filter includes: a first branch and multiple second branches arranged in parallel; the first branch is provided with a first convolution filter; the second branch is provided with a second convolution filter and a target filter, and the target filter includes one of a third convolution filter, a Sobel filter and a Laplace filter.

[0017] In some embodiments of the present disclosure, the training module is further configured to: set initial weights for the re-parameterized convolution filter according to the weights of the trained reference convolution filter.

[0018] In some embodiments of the present disclosure, the training module is also configured to: in response to the convolution kernel size of the trained reference convolution filter being the same as the convolution kernel size of the first convolution filter and the convolution kernel size of the third convolution filter, set the weight of the trained reference convolution filter to the initial weight of the first convolution filter and the initial weight of the third convolution filter; in response to the convolution kernel size of the trained reference convolution filter being different from the convolution kernel size of the second convolution filter, set the weight center of the trained reference convolution filter to the initial weight of the second convolution filter.

[0019] In some embodiments of the present disclosure, the training module is further configured to: obtain a reference convolution filter; train the reference convolution filter based on the low-resolution image samples and the high-resolution image samples to obtain the trained reference convolution filter.

[0020] According to another aspect of an embodiment of the present disclosure, an image enhancement device is provided, the device comprising: an image acquisition module, configured to acquire a target image; an image processing module, configured to upsample the target image to obtain an upsampled image; classify the pixels in the upsampled image to obtain a target pixel set; an image enhancement module, configured to input the target pixel set into a target convolution filter in an image enhancement model to obtain a target feature map of the target pixel set; the image enhancement model is generated according to the above-mentioned image enhancement model training method; and according to the target feature map of the target pixel set, an enhanced image corresponding to the target image is obtained.

[0021] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the above-mentioned image enhancement model training method, or to implement the above-mentioned image enhancement method.

[0022] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned image enhancement model training method, or execute the above-mentioned image enhancement method.

[0023] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the computer program implements the above-mentioned image enhancement model training method, or implements the above-mentioned image enhancement method.

[0024] The image enhancement model training method provided by the embodiment of the present disclosure performs targeted training on the re-parameterized convolution filter by acquiring low-resolution image samples and corresponding high-resolution samples. This process not only ensures that the model can learn effective mapping from low resolution to high resolution, but also uses re-parameterization technology to enable a more complex model structure to be used in the training stage to capture more details and features in the image; after the training is completed, the re-parameterization operation is performed on the trained re-parameterized convolution filter to simplify the complex model structure into a lighter and more efficient form. Therefore, the image enhancement model obtained by the training method can accelerate the speed of image enhancement processing while maintaining the image enhancement effect, and applying the image enhancement model to a mobile device can improve the image processing performance of the device. In addition, compared with the difference algorithm, the image enhancement model provided by the embodiment of the present disclosure achieves higher precision processing of image data through a refined training process, and image enhancement processing based on the image enhancement model can effectively avoid image distortion or quality degradation.

[0025] The image enhancement method provided by the embodiment of the present disclosure performs upsampling processing on the target image, and then performs fine classification processing on the pixels in the upsampled image to form a target pixel set, and then inputs the target pixel set into the target convolution filter of the image enhancement model, and finally obtains the enhanced image corresponding to the target image. Since the target convolution filter is optimized from the trained reparameterized convolution filter using the reparameterization technology, this process not only ensures the complexity and accuracy of the model in the training stage, but also simplifies the model structure through the reparameterization operation, reduces the computational complexity of the reasoning stage, and enables the image enhancement model to run more efficiently during reasoning while maintaining the high quality of image enhancement. Therefore, the image enhancement method is suitable for scenarios such as short video applications and social media platforms that require rapid processing and sharing of high-quality images, and can achieve real-time image enhancement with limited computing resources and energy consumption.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0028] Figure 1 A schematic diagram showing an exemplary system architecture to which the image enhancement model training method or image enhancement method according to an embodiment of the present disclosure can be applied;

[0029] Figure 2 A flowchart of an image enhancement model training method according to an embodiment of the present disclosure is shown;

[0030] Figure 3 A schematic diagram of the structure of a re-parameterized convolution filter according to an embodiment of the present disclosure is shown;

[0031] Figure 4 A schematic diagram of the training process of the reference convolution filter of an embodiment of the present disclosure is shown;

[0032] Figure 5 A schematic diagram of the training process of the re-parameterized convolution filter according to an embodiment of the present disclosure is shown;

[0033] Figure 6 A flow chart of an image enhancement method according to an embodiment of the present disclosure is shown;

[0034] Figure 7 A block diagram of an image enhancement model training device according to an embodiment of the present disclosure is shown;

[0035] Figure 8 A block diagram of an image enhancement device according to an embodiment of the present disclosure is shown;

[0036] Fig. 9 A schematic structural diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0038] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0039] The collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of the user personal information involved in this disclosure are in compliance with the relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0040] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in at least one hardware module or integrated circuit, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0042] In this specification, the terms "a", "an", "the", "said" and "at least one" are used to indicate the presence of at least one element / component / etc.; the term "plurality" refers to two or more; the terms "comprising", "including" and "having" are used to express an open-ended inclusion and mean that additional elements / components / etc. may exist in addition to the listed elements / components / etc.; the terms "first", "second" and "third" etc. are used only as labels and are not intended to limit the quantity of their objects.

[0043] Figure 1 A schematic diagram of an exemplary system architecture to which the image enhancement model training method or image enhancement method according to an embodiment of the present disclosure can be applied is shown.

[0044] like Figure 1As shown, the system architecture may include a server 101, a network 102 and a terminal device 103. The network 102 is used to provide a medium for a communication link between the terminal device 103 and the server 101. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0045] In an exemplary embodiment, the terminal device 103 for data transmission with the server 101 may include but is not limited to mobile devices such as smart phones, tablet computers, and laptop computers, as well as terminal devices with specific functions or forms such as smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, and smart wearable devices. Alternatively, the terminal device 103 may also be a personal computer, such as a laptop computer and a desktop computer. Optionally, the operating system running on the electronic device may include but is not limited to Android, IOS, Linux, Windows, etc.

[0046] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some practical applications, server 101 can also be a server of a network platform, which can be, for example, a trading platform, a live broadcast platform, a social platform, or an audio platform, etc., which is not limited in the embodiments of the present disclosure. Among them, the server can be a single server or a cluster formed by multiple servers, and the present disclosure does not limit the specific architecture of the server.

[0047] In some embodiments of the present disclosure, the process of image enhancement model training by server 101 may be: obtaining low-resolution image samples and high-resolution image samples corresponding to the low-resolution image samples; upsampling the low-resolution image samples to obtain upsampled image samples; classifying the pixels in the upsampled image samples to obtain a pixel set; training a reparameterized convolution filter according to the pixel set and the high-resolution image samples to obtain a trained reparameterized convolution filter; performing a reparameterization operation on the trained reparameterized convolution filter to obtain a target convolution filter; and obtaining an image enhancement model using the target convolution filter.

[0048] In some embodiments of the present disclosure, the process of image enhancement by the terminal device 103 may be: acquiring a target image; upsampling the target image to obtain an upsampled image; classifying the pixels in the upsampled image to obtain a target pixel set; inputting the target pixel set into a target convolution filter in an image enhancement model to obtain a target feature map of the target pixel set; the image enhancement model is generated according to the above-mentioned image enhancement model training method; and according to the target feature map of the target pixel set, an enhanced image corresponding to the target image is obtained.

[0049] In addition, it should be noted that Figure 1 What is shown is merely one application environment of the image enhancement model training method or image enhancement method provided by the present disclosure. Figure 1 The number of terminal devices 103, networks 102 and servers 101 is merely illustrative, and any number of terminal devices, networks and servers may be provided according to actual needs.

[0050] Figure 2 A flowchart of an image enhancement model training method according to an embodiment of the present disclosure is shown. Figure 2 The execution subject of the method provided in the embodiment may be any electronic device, such as Figure 1 The server 101 in the embodiment is also as follows Figure 1 The server 101 and the terminal device 103 in the embodiment jointly implement the image enhancement model training method, but the present disclosure is not limited thereto. Figure 2 The image enhancement model training method provided in the embodiment of the present disclosure includes the following steps.

[0051] Step S210: obtaining low-resolution image samples and high-resolution image samples corresponding to the low-resolution image samples.

[0052] In the disclosed embodiment, a training sample set is obtained, the training sample set including a plurality of paired training samples, each pair of training samples including a low-resolution image sample and a high-resolution sample corresponding to the low-resolution image sample.

[0053] Among them, the low-resolution image sample can be an image with a resolution lower than a preset resolution threshold, and the high-resolution image sample corresponding to the low-resolution image sample can be an image with the same content as the low-resolution image sample but a higher resolution than the low-resolution image sample.

[0054] Step S220: training the re-parameterized convolution filter based on the low-resolution image samples and the high-resolution image samples to obtain a trained re-parameterized convolution filter.

[0055] In the disclosed embodiment, the reparameterized convolution filter is a special convolution filter comprising multiple branches arranged in parallel. These branches work together during the training process, but the parameters of these branches are merged into a single convolution filter during the inference stage.

[0056] Among them, the reparameterized filter is called "reparameterized" because it uses different parameter representation methods in different stages (training and reasoning). In the training stage, the model uses a multi-branch structure to capture richer feature information, and each branch has its own independent parameters. These branches can be regarded as different feature extractors that work together on the input data to learn more complex feature representations. In the reasoning stage, in order to increase the running speed of the model and reduce the computational complexity, these multi-branch structures are merged into a simpler structure. This is the process of reparameterization: the multi-branch parameters of the training stage are merged into the parameters of a single convolution filter in some way (such as weight addition, bias addition, etc.). In this way, during reasoning, this single convolution filter can be used to replace the original multi-branch structure, thereby achieving efficient calculation. In the disclosed embodiment, during the training process, a low-resolution sample is input into the reparameterized convolution filter, and a high-resolution image sample is used as the desired output. The reparameterized convolution filter is trained to obtain a trained reparameterized convolution filter.

[0057] Step S230, performing a re-parameterization operation on the trained re-parameterized convolution filter to obtain a target convolution filter.

[0058] Step S240, using the target convolution filter to obtain an image enhancement model.

[0059] Among them, the reparameterization operation is a method of re-representing the weights of the model parameters. The weights of the model are reparameterized by performing a new linear transformation on the weights, and the outputs of the two models before and after the reparameterization process are kept unchanged.

[0060] In the disclosed embodiment, the reparameterized convolution filter is a special convolution filter comprising multiple branches arranged in parallel. The multiple branches arranged in parallel refer to multiple computational paths or subnetworks running simultaneously inside the reparameterized convolution filter, which process data independently during the training phase, but are ultimately integrated into a single convolution filter through the reparameterization operation to represent the collective effect of these parallel branches. In short, the branches arranged in parallel are multiple parallel processing paths designed inside the reparameterized convolution filter to achieve more complex feature extraction and learning capabilities.

[0061] After obtaining the trained reparameterized convolution filter, the corresponding target convolution filter is obtained by reparameterizing the trained reparameterized convolution filter, thereby realizing the use of a single convolution filter to represent multiple parallel branches. Then, the target convolution filter is used to obtain an image enhancement model, and the image enhancement task is performed using the image enhancement model. Under the premise of ensuring the same processing effect as the trained reparameterized convolution filter, the computational complexity of the target convolution filter is reduced, thereby improving the image enhancement processing performance of the image enhancement model on mobile devices.

[0062] The image enhancement model training method provided by the embodiment of the present disclosure performs targeted training on the re-parameterized convolution filter by acquiring low-resolution image samples and corresponding high-resolution samples. This process not only ensures that the model can learn effective mapping from low resolution to high resolution, but also uses re-parameterization technology to enable the training stage to adopt a more complex model structure to capture more details and features in the image; after the training is completed, the re-parameterization operation is performed on the trained re-parameterized convolution filter to simplify the complex model structure into a lighter and more efficient form. Therefore, the image enhancement model obtained by the training method can accelerate the speed of image enhancement processing while maintaining the image enhancement effect, and applying the image enhancement model to a mobile device can improve the image processing performance of the device. In addition, compared with the interpolation algorithm, the image enhancement model provided by the embodiment of the present disclosure achieves higher precision processing of image data through a refined training process, and image enhancement processing based on the image enhancement model can effectively avoid image distortion or quality degradation.

[0063] In some embodiments of the present disclosure, the re-parameterized convolution filter includes: a first branch and multiple second branches arranged in parallel; the first branch is provided with a first convolution filter; the second branch is provided with a second convolution filter and a target filter, and the target filter includes a third convolution filter, a Sobel filter and a Laplace filter.

[0064] In the disclosed embodiment, the first branch can be understood as a branch provided with a convolution filter. The second branch can be understood as a branch provided with two filters, the two filters including a convolution filter and a target filter. The target filter can be one of a convolution filter, a Sobel filter and a Laplace filter.

[0065] A convolution filter is a filter constructed using the convolution operation principle and is used to extract features from images or signals. In image processing, a convolution filter is usually a convolution kernel (also called a filter kernel), which slides on the image according to certain rules and performs product-sum operations with each pixel in the image to obtain a new image or feature map. A Sobel filter is a linear filter used for edge detection. It approximates the gradient of the image by calculating the difference in the image pixel values, thereby realizing edge detection. A Laplacian filter is an image enhancement filter based on second-order differentials. It uses the Laplacian operator to calculate the gradient change of each pixel in the image.

[0066] Figure 3 FIG. 4 shows a schematic diagram of the structure of the re-parameterized convolution filter of an embodiment of the present disclosure. Figure 3 As shown, the re-parameterized convolution filter 300 includes a branch 310, a branch 320, a branch 330, a branch 340, and a branch 350. Moreover, these five branches are in parallel, the branch 310 can be regarded as the first branch, and the branches 320, 330, 340, and 350 can be regarded as the second branch.

[0067] Figure 3 In the figure, a convolution filter 311 is arranged on branch 310, a convolution filter 321 and a convolution filter 322 are arranged on branch 320, a convolution filter 331 and a Sobel filter 332 are arranged on branch 330, a convolution filter 341 and a Sobel filter 342 are arranged on branch 340, and a convolution filter 351 and a Laplace filter 352 are arranged on branch 350.

[0068] In an exemplary embodiment, the convolution kernel size of convolution filter 311 and convolution filter 322 is the same, such as N1x N1. The convolution kernel size of convolution filters 321, 331, 341 and 351 is the same, such as 1x1. Sobel filter 332 and Sobel filter 342 are different, such as the convolution kernel of Sobel filter 332 is The convolution kernel of the Sobel filter 342 is The two Sobel filters are used to detect edges in the horizontal direction and the vertical direction respectively. The convolution kernel of the Laplace filter 352 is It can enhance the edge and detail information in the image to make the image clearer.

[0069] In the image enhancement model training method of the disclosed embodiment, the re-parameterized convolution filter can capture multiple features in the image by setting multiple branches in parallel, each branch containing a different combination of filters; the output results of multiple branches can be fused to combine the advantages of different filters to form a more comprehensive and accurate feature representation, and this fusion helps to improve the performance of the model; and the setting of the Sobel filter and the Laplace filter enhances the sensitivity to image edges and details.

[0070] In some embodiments of the present disclosure, the image enhancement model training method further includes: setting initial weights for the re-parameterized convolution filter based on the weights of the trained reference convolution filter.

[0071] In the disclosed embodiment, the trained reference convolution filter refers to a preliminarily trained convolution filter, which is used to provide a reference before or during the training of the reparameterized convolution filter. Specifically, the initial weights of each convolution filter in the reparameterized convolution filter can be set according to the weights of the trained reference convolution filter. In addition, the trained reference convolution filter can also be a pre-designed convolution filter. For example, based on actual experience or image enhancement application requirements, a convolution filter with specific weights is pre-designed and used as the trained reference convolution filter.

[0072] In the image enhancement model training method of the disclosed embodiment, the weights of the trained reference convolution filter are used as the initial weights of the re-parameterized convolution filter, so that the re-parameterized convolution filter already has a certain feature extraction capability at the beginning of training, thereby accelerating the training process.

[0073] In some embodiments of the present disclosure, the image enhancement model training method further includes: obtaining a reference convolution filter; training the reference convolution filter based on low-resolution image samples and high-resolution image samples to obtain a trained reference convolution filter.

[0074] In the disclosed embodiments, the reference convolution filter may be regarded as an initial or reference model to provide guidance for subsequent re-parameterized convolution filter training.

[0075] Exemplarily, based on low-resolution image samples and high-resolution image samples, a reference convolution filter is trained to obtain a trained reference convolution filter, including: upsampling the low-resolution image samples to obtain upsampled image samples; classifying the pixels in the upsampled image samples to obtain a pixel set; inputting the pixel set into the reference convolution filter to obtain a feature map; training the reference convolution filter according to the feature map and the image area corresponding to the pixel set in the high-resolution image samples to obtain a trained reference convolution filter.

[0076] Figure 4 FIG. 2 shows a schematic diagram of the training process of the reference convolution filter of an embodiment of the present disclosure. Figure 4 As shown, the reference convolution filter can be trained as follows:

[0077] (1) Upsampling the low-resolution image samples to obtain upsampled image samples.

[0078] After obtaining the low-resolution image samples, an upsampling algorithm may be used to enlarge the low-resolution image samples to the same size as the corresponding high-resolution image samples, thereby obtaining upsampled image samples corresponding to the low-resolution image samples.

[0079] In the embodiments of the present disclosure, upsampling refers to enlarging an image, and the main purpose is to improve the resolution of the image. After obtaining a low-resolution image sample, an upsampling algorithm can be used to enlarge the low-resolution image sample to the same size as the corresponding high-resolution image sample to obtain an upsampled image sample corresponding to the low-resolution image sample. Exemplarily, the upsampling algorithm is bilinear interpolation, and of course it can also be other upsampling algorithms, which are not limited in the embodiments of the present disclosure.

[0080] (2) Calculate the basic features of the patch area around each pixel in the upsampled image sample, and classify the pixels into different buckets based on the basic features to obtain multiple pixel sets.

[0081] In the disclosed embodiment, for each pixel in the upsampled image sample, the basic features of the patch area around the pixel are calculated, wherein the basic features may include but are not limited to grayscale values, gradient information, texture features, etc. After obtaining the basic features of the surrounding patches of each pixel in the upsampled image sample, the pixels are classified into different buckets (clusters) according to the obtained basic features. During the classification process, the similarity between the pixels can be judged based on the distance between the basic features of the surrounding patches of the pixels, and then similar pixels can be classified into a bucket. The pixels in each bucket are regarded as a pixel set, and the pixel sets in different buckets represent groups of pixels with different features in the image.

[0082] After classifying the pixels in the upsampled image samples into different buckets, each bucket corresponds to a reference convolution filter and a re-parameterized convolution filter. The reference convolution filter corresponding to the bucket is trained using the set of pixels in each bucket, and then the reference convolution filter corresponding to the bucket is used to initialize the re-parameterized convolution filter corresponding to the bucket.

[0083] (3) For each set of pixels in each bucket, optimization is performed based on the least squares method to obtain a trained reference convolution filter. In the disclosed embodiment, for each set of pixels in each bucket, a reference convolution filter is initialized. Next, an objective function is constructed to measure the processing effect of the reference convolution filter on the pixel set. This objective function is based on the least squares method, that is, minimizing the sum of squared errors between the filter output and the expected output. Among them, the features of the image area corresponding to the pixel set in the high-resolution image sample can be used as the expected output, and the processing result of the reference convolution filter on the pixel set can be used as the filter output, and then the sum of squared errors between the two is calculated as the objective function. Then, an optimization algorithm is used to solve the optimal weight of the reference convolution filter. During the optimization process, the weight of the reference convolution filter is continuously iterated and updated until the value of the objective function reaches a value less than a preset threshold, at which time the reference convolution filter is trained.

[0084] After the above step (3), each bucket will get a trained reference convolution filter. And, after classifying the pixels in the upsampled image samples into different buckets, each bucket corresponds to a reference convolution filter and a reparameterized convolution filter. The trained reference convolution filter of each bucket can be used to initialize the reparameterized convolution filter corresponding to each bucket. In this way, the same set of pixels can be used to train the reference convolution filter and the reparameterized convolution filter, which can ensure that the two have consistent data distribution and feature representation when processing the same input data, which helps to reduce the model performance differences caused by data inconsistency, so as to further accelerate and optimize the training process of the reparameterized convolution filter.

[0085] In the image enhancement model training method provided by the embodiment of the present disclosure, a reference convolution filter is trained based on a low-resolution image sample and a low-resolution image sample to obtain a trained reference convolution filter, and the trained reference convolution filter is subsequently used to initialize a re-parameterized convolution filter. By using the weights of the trained reference convolution filter to initialize the re-parameterized convolution filter, the need to learn the re-parameterized convolution filter from scratch can be avoided, thereby accelerating the training process.

[0086] In some embodiments of the present disclosure, initial weights are set for re-parameterized convolution filters based on the weights of the trained reference convolution filters, including: in response to the convolution kernel size of the trained reference convolution filter being the same as the convolution kernel size of the first convolution filter and the convolution kernel size of the third convolution filter, the weights of the trained reference convolution filter are set as the initial weights of the first convolution filter and the initial weights of the third convolution filter; in response to the convolution kernel size of the trained reference convolution filter being different from the convolution kernel size of the second convolution filter, the weight center of the trained reference convolution filter is set as the initial weight of the second convolution filter.

[0087] Among them, the re-parameterized convolution filter includes a first branch and multiple second branches arranged in parallel; the first branch is provided with a first convolution filter; the second branch is provided with a second convolution filter and a target filter, and the target filter includes one of a third convolution filter, a Sobel filter and a Laplace filter.

[0088] In an exemplary embodiment, the convolution kernel of the reference convolution filter is the same size as the convolution kernel of the first convolution filter and the third convolution filter, for example, the convolution kernels are all N1x N1, and the weight of the trained reference convolution filter is used as the initial weight of the first convolution filter and the third convolution filter. The convolution kernel of the reference convolution filter is different in size from the convolution kernel of the second convolution filter, for example, the convolution kernel of the reference convolution filter is N1x N1, and the convolution kernel of the second convolution filter is 1x 1, and the weight center of the trained reference convolution filter is used as the initial weight of the second convolution filter.

[0089] The image enhancement model training method provided by the embodiment of the present disclosure, during the initialization process of the re-parameterized convolution filter, for filters with the same convolution kernel size, directly uses the weights of the reference convolution filter for initialization, which can retain the learned feature extraction ability to the greatest extent; for filters with mismatched convolution kernel sizes, the weight center of the reference convolution filter is taken for initialization, which can retain its feature extraction ability to a certain extent and can adapt to the requirements of different convolution kernel sizes.

[0090] In some embodiments of the present disclosure, a reparameterized convolution filter is trained based on low-resolution image samples and high-resolution image samples to obtain a trained reparameterized convolution filter, including: upsampling the low-resolution image samples to obtain upsampled image samples; classifying the pixels in the upsampled image samples to obtain a pixel set; inputting the pixel set into the reparameterized convolution filter to obtain a feature map; training the reparameterized convolution filter according to the feature map and the image area corresponding to the pixel set in the high-resolution image samples to obtain a trained reparameterized convolution filter.

[0091] In some embodiments of the present disclosure, a reparameterized convolution filter is trained according to an image region corresponding to a set of pixel points in a feature map and a high-resolution image sample to obtain a trained reparameterized convolution filter, including: calculating a loss value between an image region corresponding to a set of pixel points in a feature map and a high-resolution image sample based on a preset loss function; adjusting parameters of the reparameterized convolution filter according to the loss value until the calculated loss value is less than a preset value, thereby obtaining a trained reparameterized convolution filter.

[0092] In the embodiments of the present disclosure, upsampling refers to enlarging an image, and the main purpose is to improve the resolution of the image. After obtaining a low-resolution image sample, an upsampling algorithm can be used to enlarge the low-resolution image sample to the same size as the corresponding high-resolution image sample to obtain an upsampled image sample corresponding to the low-resolution image sample. Exemplarily, the upsampling algorithm is bilinear interpolation, and of course it can also be other upsampling algorithms, which are not limited in the embodiments of the present disclosure.

[0093] For each pixel in the upsampled image sample, the basic features of the patch around the pixel are calculated, wherein the basic features may include but are not limited to grayscale values, gradient information, texture features, etc. After obtaining the basic features of the patch around each pixel in the upsampled image sample, the pixels are classified into different buckets (clusters) according to the obtained basic features. During the classification process, the similarity between the pixels can be judged based on the distance between the basic features of the patch around the pixel, and then similar pixels can be classified into a bucket. The pixels in each bucket are regarded as a pixel set, and the pixel sets in different buckets represent groups of pixels with different features in the image.

[0094] In the disclosed embodiment, there are multiple pixel point sets. Each pixel point set corresponds to a reparameterized convolution filter, and the corresponding reparameterized convolution filter is trained using the pixel point set. The specific training process is: input the pixel point set into the corresponding reparameterized convolution filter, output the feature map corresponding to the pixel point set, and use the features of the image area corresponding to the pixel point set in the high-resolution image sample as the expected output; perform training based on the feature map corresponding to the pixel point set and the expected output to obtain a trained reparameterized convolution filter.

[0095] The training process based on the feature map output by the reparameterized convolution filter corresponding to the pixel set and the expected output (i.e., the features of the image area corresponding to the pixel set in the high-resolution image sample) is as follows:

[0096] (1) using a preset loss function to calculate the loss value between the feature map and the desired output (i.e., the features of the image area corresponding to the pixel point set in the high-resolution image sample), wherein the preset loss function includes mean square error, cross entropy loss, etc.;

[0097] (2) Use the back-propagation algorithm to pass the loss value to the various parameters of the re-parameterized convolution filter. During the back-propagation process, the loss value will be converted into the gradient of each parameter;

[0098] (3) Based on the gradient obtained by back propagation, an optimization algorithm is used to update the parameters of the reparameterized convolution filter so that the difference between the feature map and (i.e., the features of the image area corresponding to the set of pixels in the high-resolution image sample) gradually decreases;

[0099] (4) After the parameters are updated, the pixel point set is input into the re-parameterized convolution filter again, and the above-mentioned process of calculating the loss value, back propagation and updating the parameters is repeated until the loss value is less than a preset threshold or the preset number of iterations is reached, and the training is stopped to obtain the trained re-parameterized convolution filter corresponding to the pixel point set.

[0100] In the image enhancement model training method provided by the embodiment of the present disclosure, the reparameterized convolution filter can learn a more refined feature representation than the traditional convolution filter through training, and because the set of pixel points in each bucket is classified based on its features, the corresponding reparameterized convolution filter can learn more specific feature patterns.

[0101] In the disclosed embodiment, the low-resolution image samples are upsampled to obtain upsampled image samples, and then the basic features of the block area (patch) around each pixel in the upsampled image samples are calculated, and the pixels are classified into different buckets by the basic features to obtain multiple pixel sets, each bucket corresponds to a reference convolution filter and a reparameterized convolution filter, and two stages of training can be performed. In the first stage of training, for each set of pixels in a bucket, a corresponding reference convolution filter is trained; in the second stage of training, the weights of the reference convolution filter obtained in the first stage of training are used to set the initial weights of the reparameterized convolution filter corresponding to each bucket, and then the corresponding reparameterized convolution filter is trained using the pixel set in each bucket.

[0102] Figure 5 FIG. 2 shows a schematic diagram of the training process of the re-parameterized convolution filter of an embodiment of the present disclosure. Figure 5 As shown, the re-parameterized convolution filter can be trained as follows:

[0103] (1) Upsampling the low-resolution image samples to obtain upsampled image samples.

[0104] After obtaining the low-resolution image samples, an upsampling algorithm may be used to enlarge the low-resolution image samples to the same size as the corresponding high-resolution image samples, thereby obtaining upsampled image samples corresponding to the low-resolution image samples.

[0105] (2) Calculate the basic features of the patch area around each pixel in the upsampled image sample, and classify the pixels into different buckets based on the basic features to obtain multiple pixel sets.

[0106] Specifically, for each pixel in the upsampled image sample, the basic features of the patch around the pixel are calculated, wherein the basic features may include but are not limited to grayscale values, gradient information, texture features, etc. After obtaining the basic features of the patch around each pixel in the upsampled image sample, the pixels are classified into different buckets (clusters) according to the obtained basic features. During the classification process, the similarity between the pixels can be judged based on the distance between the basic features of the patch around the pixel, and then similar pixels can be classified into a bucket. The pixels in each bucket are regarded as a pixel set, and the pixel sets in different buckets represent groups of pixels with different features in the image.

[0107] After classifying the pixels in the upsampled image samples into different buckets, each bucket corresponds to a re-parameterized convolution filter, and the re-parameterized convolution filter corresponding to the bucket is trained using the set of pixels in each bucket.

[0108] (3) For the set of pixel points in each bucket, the re-parameterized convolution filter is trained based on gradient descent to obtain a trained re-parameterized convolution filter.

[0109] The specific implementation is that for each set of pixels in each bucket, a reparameterized convolution filter is initialized, which can be initialized using the trained reference convolution filter corresponding to the bucket. Next, the features of the image area corresponding to the pixel set in the high-resolution image sample are taken as the desired output, and the processing result of the reparameterized convolution filter on the pixel set is taken as the filter output, and the loss value between the two is calculated based on the preset loss function. Then, the parameters of the reparameterized convolution filter are updated using the gradient descent method, and the pixel set is input into the reparameterized convolution filter again, and the above-mentioned process of calculating the loss value, back propagation, and parameter updating is repeated until the loss value is less than the preset threshold or the preset number of iterations is reached. The training is stopped to obtain the trained reparameterized convolution filter. Among them, the structure of the reparameterized convolution filter is as follows: Figure 3As shown, no further description is given here.

[0110] After the above step (3), each bucket will get a trained reparameterized convolution filter. The trained reparameterized convolution filter is reparameterized to obtain the target convolution filter corresponding to each bucket. The target convolution filters corresponding to all buckets are used to obtain the image enhancement model. It can be seen that by training the reparameterized convolution filter in the training stage, a more complex model structure can be used to capture more details and features in the image; after the training is completed, the target convolution filter is obtained by reparameterizing the trained reparameterized convolution filter, which simplifies the complex model structure into a lighter and more efficient form, reduces the computational complexity of the inference stage, and enables the final image enhancement model to accelerate the image enhancement processing speed while maintaining the image enhancement effect. Applying the image enhancement model to mobile devices can improve the image processing performance of the device.

[0111] Figure 6 A flowchart of an image enhancement method according to an embodiment of the present disclosure is shown. Figure 6 The execution subject of the method provided in the embodiment may be any electronic device, such as Figure 1 The terminal device 103 in the embodiment is also as follows Figure 1 The server 101 and the terminal device 103 in the embodiment jointly implement the image enhancement method, but the present disclosure is not limited thereto. Figure 6 The image enhancement method provided by the embodiment of the present disclosure includes the following steps.

[0112] Step S610, acquiring a target image.

[0113] The target image refers to the image that needs to be enhanced.

[0114] Step S620: up-sample the target image to obtain an up-sampled image.

[0115] In the disclosed embodiment, upsampling refers to enlarging an image, and the main purpose is to improve the resolution of the image. After obtaining a target image, the target image is upsampled to obtain an upsampled image corresponding to the target image. Exemplarily, the upsampling algorithm is bilinear interpolation, and of course other upsampling algorithms can also be used, which is not limited in the disclosed embodiment.

[0116] Step S630: classify the pixels in the upsampled image to obtain a target pixel set.

[0117] For each pixel in the upsampled image, the basic features of the patch around the pixel are calculated, where the basic features may include but are not limited to grayscale value, gradient information, texture features, etc. After obtaining the basic features of the patch around each pixel in the upsampled image, the pixels are classified into different buckets (clusters) according to the obtained basic features. The pixels in each bucket are regarded as a target pixel set, and the target pixel sets in different buckets represent groups of pixels with different features in the upsampled image.

[0118] Step S640, inputting the target pixel point set into the target convolution filter in the image enhancement model to obtain a target feature map of the target pixel point set; wherein the image enhancement model is generated according to the image enhancement model training method described in the above embodiment.

[0119] In the disclosed embodiment, there are multiple target pixel point sets. For each target pixel point set, it is input into the target convolution filter corresponding to the bucket to which it belongs, and the target feature map corresponding to the target pixel point set is obtained.

[0120] Step S650, obtaining an enhanced image corresponding to the target image according to the target feature map of the target pixel point set.

[0121] In the disclosed embodiment, according to step S640, a target feature map corresponding to each target pixel point set is obtained, and all target feature maps are fused to obtain an enhanced image corresponding to the target image.

[0122] The image enhancement method provided by the embodiment of the present disclosure performs upsampling processing on the target image, and then performs fine classification processing on the pixels in the upsampled image to form a target pixel set, and then inputs the target pixel set into the target convolution filter of the image enhancement model, and finally obtains the enhanced image corresponding to the target image. Since the target convolution filter is optimized from the trained reparameterized convolution filter using the reparameterization technology, this process not only ensures the complexity and accuracy of the model in the training stage, but also simplifies the model structure through the reparameterization operation, reduces the computational complexity of the reasoning stage, and enables the image enhancement model to run more efficiently during reasoning while maintaining the high quality of image enhancement. Therefore, the image enhancement method is suitable for scenarios such as short video applications and social media platforms that require rapid processing and sharing of high-quality images, and can achieve real-time image enhancement with limited computing resources and energy consumption.

[0123] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0124] Figure 7 FIG. 4 is a block diagram of an image enhancement model training device according to an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a sample acquisition module 710 , a training module 720 and a model generation module 730 .

[0125] The sample acquisition module 710 is configured to: acquire low-resolution image samples and high-resolution image samples corresponding to the low-resolution image samples. The training module 720 is configured to: train the re-parameterized convolution filter based on the low-resolution image samples and the high-resolution image samples to obtain the trained re-parameterized convolution filter. The model generation module 730 is configured to: perform a re-parameterization operation on the trained re-parameterized convolution filter to obtain a target convolution filter; and obtain an image enhancement model using the target convolution filter.

[0126] In some embodiments of the present disclosure, the training module 720 is further configured to: upsample the low-resolution image samples to obtain upsampled image samples; classify the pixels in the upsampled image samples to obtain a pixel set; input the pixel set to a reparameterized convolution filter to obtain a feature map; train the reparameterized convolution filter according to the feature map and the image area corresponding to the pixel set in the high-resolution image samples to obtain a trained reparameterized convolution filter.

[0127] In some embodiments of the present disclosure, the training module 720 is further configured to: calculate the loss value between the first feature map and the image area corresponding to the set of pixel points in the high-resolution image sample based on a preset loss function; adjust the parameters of the reparameterized convolution filter according to the loss value until the calculated loss value is less than the preset value, thereby obtaining a trained reparameterized convolution filter.

[0128] In some embodiments of the present disclosure, the re-parameterized convolution filter includes: a first branch and multiple second branches arranged in parallel; the first branch is provided with a first convolution filter; the second branch is provided with a second convolution filter and a target filter, and the target filter includes a third convolution filter, a Sobel filter and a Laplace filter.

[0129] In some embodiments of the present disclosure, the training module 720 is further configured to set initial weights for the re-parameterized convolution filter according to the weights of the trained reference convolution filter.

[0130] In some embodiments of the present disclosure, the training module 720 is also configured to: in response to the convolution kernel size of the trained reference convolution filter being the same as the convolution kernel size of the first convolution filter and the convolution kernel size of the third convolution filter, set the weight of the trained reference convolution filter to the initial weight of the first convolution filter and the initial weight of the third convolution filter; in response to the convolution kernel size of the trained reference convolution filter being different from the convolution kernel size of the second convolution filter, set the weight center of the trained reference convolution filter to the initial weight of the second convolution filter.

[0131] In some embodiments of the present disclosure, the training module 720 is further configured to: obtain a reference convolution filter; and train the reference convolution filter based on low-resolution image samples and high-resolution image samples to obtain a trained reference convolution filter.

[0132] Figure 8 FIG. 1 is a block diagram of an image enhancement device according to an embodiment of the present disclosure. Figure 8 As shown, the device 800 includes an image acquisition module 810 , an image processing module 820 and an image enhancement module 830 .

[0133] Among them, the image acquisition module 810 is configured to: acquire a target image. The image processing module 820 is configured to: upsample the target image to obtain an upsampled image; classify the pixels in the upsampled image to obtain a target pixel set. The image enhancement module 830 is configured to: input the target pixel set into the target convolution filter in the image enhancement model to obtain a target feature map of the target pixel set; the image enhancement model is generated according to the image enhancement model training method of the above embodiment; according to the target feature map of the target pixel set, an enhanced image corresponding to the target image is obtained.

[0134] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0135] Fig. 9 FIG. 1 is a schematic diagram showing the structure of an electronic device according to an embodiment of the present disclosure. It should be noted that: Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0136] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0137] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 2 Follow the steps shown in .

[0138] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0139] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0140] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0141] The electronic device 900 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 940. As shown, the network adapter 940 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0142] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0143] The program product for implementing the above method according to the embodiment of the present invention may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0144] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0145] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0146] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0147] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0148] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0149] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0150] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0151] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0152] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an image enhancement model, characterized in that: The method comprises: Acquire low-resolution image samples and high-resolution image samples corresponding to the low-resolution image samples; Based on the low-resolution image samples and the high-resolution image samples, training a re-parameterized convolution filter to obtain a trained re-parameterized convolution filter; Performing a reparameterization operation on the trained reparameterized convolution filter to obtain a target convolution filter; Using the target convolution filter, an image enhancement model is obtained.

2. The method according to claim 1, characterized in that The step of training the re-parameterized convolution filter based on the low-resolution image sample and the high-resolution image sample to obtain a trained re-parameterized convolution filter comprises: Performing upsampling processing on the low-resolution image samples to obtain upsampled image samples; classifying the pixels in the upsampled image samples to obtain a pixel set; inputting the pixel set into the reparameterized convolution filter to obtain a feature map; The re-parameterized convolution filter is trained according to the feature map and the image area corresponding to the pixel point set in the high-resolution image sample to obtain the trained re-parameterized convolution filter.

3. The method according to claim 2, characterized in that The step of training the re-parameterized convolution filter according to the feature map and the image area corresponding to the pixel point set in the high-resolution image sample to obtain the trained re-parameterized convolution filter comprises: Based on a preset loss function, calculating a loss value between the feature map and an image region corresponding to the set of pixel points in the high-resolution image sample; The parameters of the re-parameterized convolution filter are adjusted according to the loss value until the calculated loss value is less than a preset value, thereby obtaining the trained re-parameterized convolution filter.

4. The method according to any one of claims 1 to 3, characterized in that: The re-parameterized convolution filter includes: a first branch and multiple second branches arranged in parallel; the first branch is provided with a first convolution filter; the second branch is provided with a second convolution filter and a target filter, and the target filter includes one of a third convolution filter, a Sobel filter and a Laplace filter.

5. The method according to claim 4, characterized in that The method further comprises: According to the weight of the trained reference convolution filter, an initial weight is set for the re-parameterized convolution filter.

6. The method according to claim 5, characterized in that The step of setting an initial weight for the re-parameterized convolution filter according to the weight of the trained reference convolution filter comprises: In response to the convolution kernel size of the trained reference convolution filter being the same as the convolution kernel size of the first convolution filter and the convolution kernel size of the third convolution filter, setting the weight of the trained reference convolution filter as the initial weight of the first convolution filter and the initial weight of the third convolution filter; In response to the convolution kernel size of the trained reference convolution filter being different from the convolution kernel size of the second convolution filter, the weight center of the trained reference convolution filter is set as the initial weight of the second convolution filter.

7. The method according to claim 5, characterized in that The method further comprises: Get the reference convolution filter; Based on the low-resolution image samples and the high-resolution image samples, the reference convolution filter is trained to obtain the trained reference convolution filter.

8. An image enhancement method, characterized in that: The method comprises: Get the target image; Performing upsampling processing on the target image to obtain an upsampled image; classifying the pixels in the upsampled image to obtain a target pixel set; Inputting the target pixel point set into a target convolution filter in an image enhancement model to obtain a target feature map of the target pixel point set; the image enhancement model is generated by the image enhancement model training method according to any one of claims 1 to 7; According to the target feature map of the target pixel point set, an enhanced image corresponding to the target image is obtained.

9. An image enhancement model training device, characterized in that: The device comprises: A sample acquisition module, configured to acquire low-resolution image samples and high-resolution image samples corresponding to the low-resolution image samples; A training module is configured to train the re-parameterized convolution filter based on the low-resolution image samples and the high-resolution image samples to obtain a trained re-parameterized convolution filter; The model generation module is configured to perform a reparameterization operation on the trained reparameterized convolution filter to obtain a target convolution filter; and use the target convolution filter to obtain an image enhancement model.

10. An image enhancement device, characterized in that: The device comprises: An image acquisition module is configured to acquire a target image; An image processing module is configured to perform upsampling processing on the target image to obtain an upsampled image; classify the pixels in the upsampled image to obtain a target pixel set; An image enhancement module is configured to input the target pixel point set into a target convolution filter in an image enhancement model to obtain a target feature map of the target pixel point set; the image enhancement model is generated according to the image enhancement model training method described in any one of claims 1 to 7; and based on the target feature map of the target pixel point set, an enhanced image corresponding to the target image is obtained.

11. An electronic device, characterized in that: include: processor; A memory for storing the processor executable instructions; wherein the processor is configured to execute the executable instructions to implement the image enhancement model training method as described in any one of claims 1 to 7, or to implement the image enhancement method as described in claim 8.

12. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the image enhancement model training method as described in any one of claims 1 to 7, or execute the image enhancement method as described in claim 8.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the image enhancement model training method as described in any one of claims 1 to 7 is implemented, or the image enhancement method as described in claim 8 is implemented.

Images