Image quality enhancement model distillation method and device, electronic equipment and storage medium

Abstract

The invention provides an image quality enhancement model distillation method and device, electronic equipment and a storage medium. The distillation method comprises the following steps: acquiring a training image set; obtaining a first enhanced residual image set of the training image set through a pre-trained first image quality enhancement model; inputting the training image set into a to-be-trained second image quality enhancement model to obtain a second enhanced residual image set; and performing knowledge distillation training on the second image quality enhancement model according to the loss of the second enhanced residual image set and the first enhanced residual image set. Compared with the prior art, the first image quality enhancement model obtains the first enhanced residual image set corresponding to the training image set as the guidance information, so that the second image quality enhancement model can simulate the performance of the first image quality enhancement model under limited resources, learn enhanced residual details, and improve the image quality enhancement accuracy. Therefore, the image quality can be enhanced on equipment with limited computing power.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and more specifically, to a method, device, electronic device and storage medium for distilling an image quality enhancement model. Background Art

[0002] Nowadays, image quality enhancement methods based on deep learning have gradually become one of the core technologies for improving the quality of live broadcasts due to their excellent effects. These methods process images through complex neural network models, which can effectively improve the details and clarity of live broadcast images. However, deep learning models usually require a lot of computing resources, especially when processing high-resolution video images in real time, which often relies on powerful GPU (graphics processing unit) computing power. For some devices with low computing power resources, conventional deep learning models find it difficult to process high-resolution video images. Summary of the invention

[0003] The present invention aims to overcome at least one defect of the above-mentioned prior art and provide a method, device, electronic device and storage medium for distilling an image quality enhancement model, which are used to obtain an image quality enhancement model based on knowledge distillation, and effectively implement image quality enhancement operations on devices with low computing power resources.

[0004] According to a first aspect of the present application, a method for distilling an image quality enhancement model is provided, the distillation method comprising:

[0005] Get a training image set;

[0006] Acquire a first enhanced residual image set of the training image set through a pre-trained first image quality enhancement model;

[0007] Inputting the training image set into a second image quality enhancement model to be trained to obtain a second enhanced residual image set;

[0008] The second image quality enhancement model is trained by knowledge distillation according to the losses of the second enhanced residual image set and the first enhanced residual image set.

[0009] Optionally, obtaining a first enhanced residual image set of the training image set by using a pre-trained first image quality enhancement model specifically includes:

[0010] The training image set is input into the first image quality enhancement model to obtain the first enhanced residual image set corresponding to the training image set.

[0011] Optionally, obtaining a first enhanced residual image set of the training image set by using a pre-trained first image quality enhancement model specifically includes:

[0012] The training image set is input into the first image quality enhancement model to obtain an enhanced residual image set corresponding to the training image set.

[0013] Superimposing the images in the training image set and the images in the enhanced residual image set correspondingly to obtain an enhanced prediction image set;

[0014] Repeat the enhancement at least once, each time inputting the enhanced predicted image set obtained in the previous enhancement into the first image quality enhancement model for enhancement, obtaining the corresponding enhanced residual image set and enhanced predicted image set, and using the enhanced residual image set obtained in the last prediction as the first enhanced residual image set.

[0015] Optionally, the number of enhancements does not exceed a preset number threshold.

[0016] Optionally, the acquiring of a training image set specifically includes:

[0017] Acquire first video data;

[0018] Reducing the bit rate of the first video data to obtain corresponding second video data;

[0019] According to a number of time points, corresponding image frames are acquired in the first video data and the second video data respectively, and a first image set and a second image set are constructed respectively;

[0020] The second image set is used as the training image set.

[0021] Optionally, the first image quality enhancement model and the second image quality enhancement model both include a plurality of residual blocks;

[0022] The processing flow of the first image quality enhancement model and the second image quality enhancement model includes:

[0023] Performing convolution processing on the input image to obtain convolution features of the input image;

[0024] Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0025] Passing the nonlinear features through each of the residual blocks in sequence to obtain residual extraction features of each of the residual blocks;

[0026] The residual extraction feature output by the last residual block is used as the residual image of the input image.

[0027] Optionally, the training of the first image quality enhancement model includes:

[0028] Inputting the second image set into the first image quality enhancement model to obtain a predicted enhanced residual image set;

[0029] Superimposing the images in the second image set and the images in the prediction enhanced residual image set correspondingly to obtain a prediction enhanced image set;

[0030] Calculate a first model loss of the predicted enhanced image set and the first image set, and update the first image quality enhancement model according to the first model loss.

[0031] According to a second aspect of the present application, a picture quality enhancement model distillation device is provided, the distillation device comprising:

[0032] A training set acquisition module, used to acquire a training image set;

[0033] A first image quality residual acquisition module, used for acquiring a first enhanced residual image set of the training image set through a pre-trained first image quality enhancement model;

[0034] A second image quality residual acquisition module, used for inputting the training image set into a second image quality enhancement model to be trained to obtain a second enhanced residual image set;

[0035] A distillation training module is used to perform knowledge distillation training on the second image quality enhancement model according to the losses of the second enhanced residual image set and the first enhanced residual image set.

[0036] According to a third aspect of the present application, an electronic device is provided, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement a method for distilling an image quality enhancement model as described in the first aspect above.

[0037] According to a fourth aspect of the present application, a computer storage medium is provided, on which a computer-readable program is stored. When the computer-readable program is executed, the image quality enhancement model distillation method described in the first aspect is implemented.

[0038] Based on any one of the above aspects, a picture quality enhancement model distillation method, device, electronic device and storage medium provided by the present application uses a fully trained first picture quality enhancement model as a teacher model for knowledge distillation training, and uses a second picture quality enhancement model to be trained as a student model. The first picture quality enhancement model obtains the first enhanced residual image set corresponding to the training image set as guidance information, so that the second picture quality enhancement model can imitate the performance of the first picture quality enhancement model under limited resources, learn to enhance the residual details, and then obtain the image quality enhancement on a device with limited computing power.

[0039] At the same time, the present application provides a method, device, electronic device and storage medium for distilling an image quality enhancement model, which obtains a low-definition training image set and performs multiple enhancement processing on the training image set through the first image quality enhancement model, thereby obtaining a corresponding high-definition enhanced residual image set. These high-definition enhanced residual image sets contain key and rich enhancement detail information, which can well guide the second image quality enhancement model as a student model on how to improve image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of an application scenario of the enhancement method provided in this embodiment.

[0042] Figure 2 A flow chart of the steps of the distillation method provided in this embodiment.

[0043] Figure 3 A flow chart of the steps for obtaining the training image set provided in this embodiment.

[0044] Figure 4 A flow chart of the training steps of the first image quality enhancement model provided in this embodiment.

[0045] Figure 5 This is a structural diagram of the distillation apparatus provided in this embodiment.

[0046] Figure 6 This is a device structure diagram of the electronic device provided in this embodiment.

[0047] Figures are labeled: server 100, terminal 200, training set acquisition module 11, first image quality residual acquisition module 12, second image quality residual acquisition module 13, distillation training module 14, first model training module 15, memory 21, processor 22, bus 23, communication interface 24. DETAILED DESCRIPTION

[0048] The drawings of the present invention are only for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the following embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it is understandable to those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] Nowadays, image quality enhancement technology is widely used in many fields. For example, in photography and videography, image quality enhancement is used to enhance the details of pictures and videos to make the works more attractive; in medical imaging, image quality enhancement technology helps doctors analyze X-rays and MRI images more accurately and improves the accuracy of diagnosis; in the field of security monitoring, image quality enhancement technology is also used to improve the image quality and clarity of surveillance videos, especially for some low-light environments. Improving the image quality and clarity of surveillance videos can help identify and track targets. In addition, in the field of live video broadcasting, image quality enhancement technology is also particularly important. With the popularity of live broadcast platforms, the audience's demand for high-quality images continues to increase, and with image quality enhancement technology, live broadcast content can improve details and clarity while reducing noise and other interference, which not only improves the audience's viewing experience, but also helps the anchor to better display the content.

[0052] Among the numerous image quality enhancement technologies, image quality enhancement technology based on deep learning has gradually become one of the core technologies for improving the image quality of live video due to its excellent effect. This type of technical method processes images through complex neural network models, which can effectively improve the details and clarity of live broadcast images. However, deep learning models usually require a lot of computing resources, especially when processing high-resolution videos in real time, they often rely on powerful GPU computing power. With the advancement of technology, the development speed of various device chips, especially mobile phone chips, is amazing. Although these chips have been improved in performance, their power consumption remains at a low level. Therefore, in order to run deep learning models on mobile chips with limited computing power, the model must be pruned. This means that the size and number of parameters of the model need to be reduced in order to achieve the required performance with smaller computing resources. Through pruning, the computing requirements of the model can be greatly reduced, thereby adapting to the computing power limitations of mobile devices. However, after the model is significantly pruned, directly using the training data and methods of the original large model may result in a significant decrease in the effect. The small model may not be able to fully capture the complex characteristics of the large model, resulting in unsatisfactory enhancement effects. In addition, some side effects may be introduced, such as increased image noise or other image quality issues.

[0053] This embodiment provides a technical solution that can solve the above-mentioned problem. The specific implementation methods of this application are described in detail below in conjunction with the accompanying drawings.

[0054] For example, a schematic diagram of an application scenario of a method for image quality enhancement model distillation provided in an embodiment of the present application is provided. Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100. The server 100 has an image processing function and can also have a data transmission function for video streams and audio streams; the terminal 200 has a streaming media playback function and can also have an image processing function.

[0055] It is understandable that the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smart phone terminal, a personal computer, a tablet computer, a car terminal, etc., but is not limited thereto.

[0056] In an operative manner, the server 100 and the terminal 200 may respectively execute a method for distilling an image quality enhancement model provided in an embodiment of the present application, or, optionally, the method for distilling an image quality enhancement model provided in an embodiment of the present application may be partially executed in the server 100 and partially executed in the terminal 200.

[0057] like Figure 2 As shown, this embodiment provides a method for distilling an image quality enhancement model;

[0058] Among them, knowledge distillation refers to guiding the training of a smaller student model through a larger teacher model, and by transferring the intrinsic knowledge of the teacher model, the student model can retain high performance while simplifying its structure. Therefore, a fully trained large image quality enhancement model can be used as a teacher model to perform knowledge distillation training on the small image quality enhancement model, so as to achieve effective image quality enhancement on devices with low computing power while reducing computing power consumption.

[0059] It is understandable that in order to achieve knowledge distillation, it is necessary to pre-train the first image quality enhancement model as a teacher model. After obtaining the trained first image quality enhancement model, the second image quality enhancement model as a student model is trained through the first image quality enhancement model.

[0060] The distillation method of this embodiment may specifically include:

[0061] S1: Get the training image set;

[0062] As described above, in this embodiment, in order to realize knowledge distillation, it is necessary to obtain a trained teacher model in advance, that is, the first image quality enhancement model. In fact, since the training of the student model is the learning of the teacher model, the training set of the second image quality enhancement model as the student model corresponds to the training set of the first image quality enhancement model as the teacher model. Therefore, in step S1, the actual acquisition process of the training image set also includes the acquisition of the training set of the first image quality enhancement model.

[0063] Specifically, Figure 3 As shown, in this embodiment, the acquisition of the training image set may specifically include:

[0064] A1: Acquire first video data;

[0065] The first video data is preferably high-definition video data with a high resolution and bit rate. High-definition video data with a high resolution and a high bit rate has richer details so as to provide more sufficient information for model training. The first video data can be acquired through a video acquisition device, such as a high-definition camera, or by cropping an existing high-definition video. In actual acquisition, a high-definition resolution threshold and a high-definition bit rate threshold can be set. Only when the video meets the high-definition resolution threshold and the high-definition bit rate threshold, the video is acquired to obtain the first video data.

[0066] A2: reducing the bit rate of the first video data to obtain corresponding second video data;

[0067] In order to simulate the change of video quality in a real environment, it is necessary to perform coding degradation processing on the collected high-definition first video data, that is, to reduce the bit rate of the first video data. Specifically, different video clips in the first video data can be encoded at different bit rates, and a series of low-definition videos with different clarity can be generated as the second video data. Different bit rates represent the quality performance of the corresponding video under specific network conditions or storage limitations. Through this coding degradation, low-definition videos can be effectively created for training the performance of the model under different quality conditions.

[0068] A3: acquiring corresponding image frames in the first video data and the second video data according to a number of time points, and constructing a first image set and a second image set respectively;

[0069] Specifically, since the second video data is obtained by reducing the bit rate of the first video data, the main difference between the first video data and the second video data is the picture quality, and the video content is the same. Therefore, in the first video data and the second video data, the image frames at the same time point are the same. Then, by acquiring the corresponding image frames at several time points, the image contents of the first image set and the second image set constructed are the same, and the image quality of the images in the second image set is lower than that in the first image set.

[0070] A4: Use the second image set as the training image set.

[0071] It is understandable that in order to train the image quality enhancement model, low-quality images are required as input of the model, so in this embodiment, the second image set is used as the training image set, and the first image set can be used as the training target.

[0072] In a preferred implementation of this embodiment, since the first video data and the second video data have the same content but different bit rates, the same video clip can be captured at different bit rates to obtain the first video data and the second video data.

[0073] S2: Obtaining a first enhanced residual image set of the training image set through a pre-trained first image quality enhancement model;

[0074] Specifically, in this embodiment, the first image quality enhancement model can be constructed based on a super-resolution network. Preferably, the first image quality enhancement model can be constructed based on an existing EDRN (Enhanced Deep Residual Networks for Single Image Super-Resolution) network, which has excellent image enhancement capabilities and good generalization performance.

[0075] Furthermore, this embodiment makes certain improvements to the existing EDRN network, and the improvements mainly include making the network finally output a residual image with enhanced image quality, rather than directly outputting an enhanced image. In this way, the image enhancement information can be retained in the residual image, and then the residual image containing the enhanced information is used as guidance information to guide the learning of the student model, so that the student model can better learn the image enhancement information during the model knowledge distillation training process, thereby improving the performance of the student model, as well as the stability and applicability of the model.

[0076] In this embodiment, the improved first image quality enhancement model retains the residual block in the super-resolution network to extract the residual image. The processing flow of the first image quality enhancement model may specifically include the following:

[0077] Performing convolution processing on the input image to obtain convolution features of the input image;

[0078] Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0079] Passing the nonlinear features through each of the residual blocks in sequence to obtain residual extraction features of each of the residual blocks;

[0080] The residual extraction feature output by the last residual block is used as the residual image of the input image.

[0081] Understandable, such as Figure 4 As shown, the training steps of the first image quality enhancement model may specifically include:

[0082] B1: inputting the second image set into the first image quality enhancement model to obtain a predicted enhanced residual image set;

[0083] B2: superimposing the images in the second image set and the images in the prediction enhanced residual image set correspondingly to obtain a prediction enhanced image set;

[0084] Among them, in order to further increase the flexibility of the first image quality enhancement model and the second image quality enhancement model, a residual adjustment coefficient is introduced in the superposition process, which is specifically expressed as: enhanced result = input before enhancement + image quality enhancement residual * residual adjustment coefficient. The setting range of the residual adjustment coefficient is 0 to 1. When the residual adjustment coefficient is set to 1.0, it means that the default full enhancement effect is applied. When the residual adjustment coefficient is set to 0.2, the enhancement effect is weaker and is suitable for scenes that require slight enhancement. By setting the residual adjustment coefficient, the superposition enhancement situation can be flexibly adjusted.

[0085] B3: Calculate the first model loss of the predicted enhanced image set and the first image set, and update the first image quality enhancement model according to the first model loss.

[0086] The first image quality enhancement model is trained by using a large amount of image data under different image quality conditions and application scenarios, so that it can learn rich image restoration and enhancement capabilities, ensuring that the first image quality enhancement model can perform well in different situations, and then can effectively serve as a teacher model to perform knowledge distillation training on the second image quality enhancement model as a student model.

[0087] Corresponding to the acquisition of the first enhanced residual image set in step S2, in a preferred implementation of the present embodiment, specifically, the training image set can be input into the first image quality enhancement model to obtain the first enhanced residual image set corresponding to the training image set.

[0088] In this implementation, the training image set is only enhanced once after being input into the first image quality enhancement model, and a residual image set corresponding to the one-time image quality enhancement is extracted as the first enhanced residual image set. In this implementation, the processing time of the first image quality enhancement model can be reduced by one-time image quality enhancement, thereby improving the efficiency of knowledge distillation training, effectively reducing the computing power consumption of the device, and ensuring the normal operation of the device.

[0089] Considering that the scale of the second image quality enhancement model is small and its learning ability is limited, and the first image quality enhancement model only performs image quality enhancement once and contains limited information, thereby reducing the learning effect of the second image quality enhancement model, so in another implementation manner of this implementation, the acquisition of the first enhanced residual image set may specifically include:

[0090] The training image set is input into the first image quality enhancement model to obtain an enhanced residual image set corresponding to the training image set.

[0091] Superimposing the images in the training image set and the images in the enhanced residual image set correspondingly to obtain an enhanced prediction image set;

[0092] Repeat the enhancement at least once, each time inputting the enhanced predicted image set obtained in the previous enhancement into the first image quality enhancement model for enhancement, obtaining the corresponding enhanced residual image set and enhanced predicted image set, and using the enhanced residual image set obtained in the last prediction as the first enhanced residual image set.

[0093] It can be understood that after the training image is input into the first image quality enhancement model for one enhancement, the corresponding enhanced residual image set is obtained, and after the enhanced residual image set is superimposed with the training image set to obtain the enhanced predicted image set, the obtained enhanced predicted image set is input into the first image quality enhancement model for a second enhancement to obtain the enhanced residual image set after the second enhancement, the enhanced residual image set after the second enhancement is superimposed with the enhanced predicted image set obtained by the first enhancement to obtain the enhanced predicted image set of the second enhancement, and then the enhanced predicted image set of the second enhancement is input into the first image quality enhancement model for a third enhancement, and so on.

[0094] The first enhanced residual image set obtained through such multiple transition enhancements continuously has more enhancement information superimposed thereon, which enables the student model to learn appropriate enhancement information during the learning process and then apply it to itself, thereby effectively improving the performance of the model.

[0095] It should be noted that although multiple transition enhancements can improve the performance of the student model, too many enhancements will increase the computing power consumption of the device and affect the learning of the student model. Because the learning ability of the student model is limited, inputting too much information during the learning process will reduce the learning efficiency and affect the stability of the learning model. Therefore, in the actual setting process, a number threshold can be set, which is set according to the scale of the student model, and the number of enhancements cannot exceed the number threshold.

[0096] S3: inputting the training image set into a second image quality enhancement model to be trained to obtain a second enhanced residual image set;

[0097] In this embodiment, the first enhanced residual image set obtained by the first image quality enhancement model is used as guidance information to guide the learning of the second image quality enhancement model. In order to better learn this guidance information, the second image quality enhancement model also needs to extract residual images from the training image set to obtain the second enhanced residual image set.

[0098] As described above, the second image quality enhancement model, as a student model, needs to learn and imitate the first image quality enhancement model as a teacher model to realize the extraction of the residual image, so in the basic structure of the model, the second image quality enhancement model needs to be the same or similar to the first image quality enhancement model, so the second image quality enhancement model of this embodiment can be constructed based on the existing EDRN network, then the second image quality enhancement model also retains a number of residual blocks for extracting the residual image, it can be understood that the processing flow of the second image quality enhancement model is the same as that of the first image quality enhancement model, and specifically may include:

[0099] Performing convolution processing on the input image to obtain convolution features of the input image;

[0100] Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature;

[0101] Passing the nonlinear features through each of the residual blocks in sequence to obtain residual extraction features of each of the residual blocks;

[0102] The residual extraction feature output by the last residual block is used as the residual image of the input image.

[0103] In addition, in order to make the second image quality enhancement model adaptable to devices with lower computing power, this embodiment performs cropping based on the first image quality enhancement model, and reduces the complexity and parameter quantity of the second image quality enhancement model through cropping. Specific cropping may include:

[0104] Reduction of the number of residual blocks: The existing EDRN network includes at least 32 residual blocks. In order to give full play to the image quality enhancement capability of the EDRN network, the first image quality enhancement model retains all the residual blocks in the existing EDRN network, while the second image quality enhancement model needs to reduce the processing amount of the model, so the 32 residual blocks are reduced to 6 residual blocks by cropping;

[0105] Reducing the number of channels: Each layer of the existing EDRN network includes at least 256 channels. The first image quality enhancement model retains all channels for learning different representations. The more channels there are, the greater the amount of data processing. In order to reduce the size of the model, the second image quality enhancement model needs to reduce the number of channels to one-sixth, that is, only retain at least 16 channels.

[0106] S4: performing knowledge distillation training on the second image quality enhancement model according to the losses of the second enhanced residual image set and the first enhanced residual image set;

[0107] In this embodiment, the first enhanced residual image set is the guidance information extracted by the first image quality enhancement model that has been fully trained. The second image quality enhancement model needs to use this guidance information as a training target to perform knowledge distillation training, so that the second enhanced residual image set predicted by itself is continuously close to the first enhanced residual image set. Specifically, by calculating the loss of the second enhanced residual image set and the first enhanced residual image set, and continuously updating and optimizing the second image quality enhancement model according to the loss, the second enhanced residual image set predicted by the second image quality enhancement model is continuously converged to the first enhanced residual image set. Through continuous learning and iterative optimization, the trained second image quality enhancement model is finally obtained, which can effectively restore the details of the high-definition image from the low-definition image.

[0108] Preferably, the loss function of the loss calculation can be set to mean square error (MSE) or the like to measure the difference between the second enhanced residual image set output by the second image quality enhancement model and the expected first enhanced residual image set.

[0109] It should be noted that, in this embodiment, although in the description, the second image set used for training the first image quality enhancement model and the second image set (training image set) used for training the second image quality enhancement model are obtained through the same step process, but in actual use, preferably, in order to ensure the stability and reliability of the model, the training sets for training the two models are different. The above description is for the purpose of expressing that since the two models have the same structure, the corresponding training sets can be obtained using the same steps.

[0110] The distillation method provided in this embodiment uses a fully trained first image quality enhancement model as a teacher model for knowledge distillation training, and uses a second image quality enhancement model to be trained as a student model. The first image quality enhancement model obtains a first enhanced residual image set corresponding to the training image set as guidance information, so that the second image quality enhancement model can imitate the performance of the first image quality enhancement model under limited resources, learn to enhance residual details, and then achieve image quality enhancement on devices with limited computing power.

[0111] At the same time, by performing multiple enhancement processing on the training image set, the corresponding high-definition enhanced residual image set is obtained. These high-definition enhanced residual image sets contain key and rich enhancement detail information, which can well guide the second image quality enhancement model as a student model on how to improve image quality.

[0112] Based on the same technical concept, such as Figure 5As shown, this embodiment also provides a picture quality enhancement model distillation device, and the distillation device may specifically include:

[0113] A training set acquisition module 11 is used to acquire a training image set;

[0114] In this embodiment, the training set acquisition module 11 can be used to perform Figure 2 The step S1 shown, more specifically, can be used to perform Figure 3 For the detailed description of the training set acquisition module 11 , reference may be made to the description of the step S1 and the steps A1 - A4 .

[0115] A first image quality residual acquisition module 12, used for acquiring a first enhanced residual image set of the training image set by using a pre-trained first image quality enhancement model;

[0116] In this embodiment, the first image quality residual acquisition module 12 can be used to perform Figure 2 As shown in step S2, for the specific description of the first image quality residual acquisition module 12, reference can be made to the description of step S2.

[0117] Specifically, the distillation device further includes a first model training module 15, which is used to train the first image quality enhancement model;

[0118] In this embodiment, the first model training module 15 can be used to perform Figure 4 For the detailed description of the first model training module 15 , please refer to the description of the steps B1-B3.

[0119] A second image quality residual acquisition module 13, used for inputting the training image set into a second image quality enhancement model to be trained to obtain a second enhanced residual image set;

[0120] In this embodiment, the second image quality residual acquisition module 13 can be used to perform Figure 2 As shown in step S3, for the specific description of the second image quality residual acquisition module 13, reference may be made to the description of step S3.

[0121] A distillation training module 14, configured to perform knowledge distillation training on the second image quality enhancement model according to the losses of the second enhanced residual image set and the first enhanced residual image set;

[0122] In this embodiment, the distillation training module 14 can be used to perform Figure 2 As shown in step S4, for the specific description of the distillation training module 14, reference may be made to the description of step S4.

[0123] Based on the same inventive concept, this embodiment also provides an electronic device, Figure 6 The structural diagram of the electronic device of this embodiment is shown, including a memory 21 and a processor 22, the memory 21 stores computer-readable instructions, and the processor 22 executes the computer-readable instructions to implement the image quality enhancement model distillation method of this embodiment.

[0124] Preferably, the electronic device further includes a bus 23 and a communication interface 24 , and the processor 22 , the communication interface 24 and the memory 21 are connected via the bus 23 .

[0125] The memory 21 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 24 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 23 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus 23 may be divided into an address bus, a data bus, a control bus, etc. (not fully drawn in the figure).

[0126] The processor 22 may be an integrated circuit chip with signal processing capability. In the specific implementation process, each step in the above method embodiment may be completed by the hardware integrated logic circuit or software instructions in the processor 22. The above processor 22 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, which may implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or the processor 22 may also be any conventional processor 22, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 21, and the processor 22 reads the information in the memory 21 and completes the steps of the method of the above embodiment in combination with its hardware.

[0127] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor 22, the computer-executable instructions prompt the processor 22 to implement the above-mentioned image quality enhancement model distillation method. The specific implementation can be found in the embodiment and will not be repeated here.

[0128] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0129] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A method for distilling an image quality enhancement model, characterized in that: The distillation method comprises: Get a training image set; Acquire a first enhanced residual image set of the training image set through a pre-trained first image quality enhancement model; Inputting the training image set into a second image quality enhancement model to be trained to obtain a second enhanced residual image set; The second image quality enhancement model is trained by knowledge distillation according to the losses of the second enhanced residual image set and the first enhanced residual image set.

2. The image quality enhancement model distillation method according to claim 1, characterized in that: The step of obtaining a first enhanced residual image set of the training image set by using a pre-trained first image quality enhancement model specifically includes: The training image set is input into the first image quality enhancement model to obtain the first enhanced residual image set corresponding to the training image set.

3. The image quality enhancement model distillation method according to claim 1, characterized in that: The step of obtaining a first enhanced residual image set of the training image set by using a pre-trained first image quality enhancement model specifically includes: The training image set is input into the first image quality enhancement model to obtain an enhanced residual image set corresponding to the training image set. Superimposing the images in the training image set and the images in the enhanced residual image set correspondingly to obtain an enhanced prediction image set; Repeat the enhancement at least once, each time inputting the enhanced predicted image set obtained in the previous enhancement into the first image quality enhancement model for enhancement, obtaining the corresponding enhanced residual image set and enhanced predicted image set, and using the enhanced residual image set obtained in the last prediction as the first enhanced residual image set.

4. The image quality enhancement model distillation method according to claim 3, characterized in that: The number of enhancements does not exceed a preset number threshold.

5. The image quality enhancement model distillation method according to any one of claims 1 to 3, characterized in that: The obtaining of the training image set specifically includes: Acquire first video data; Reducing the bit rate of the first video data to obtain corresponding second video data; According to a number of time points, corresponding image frames are acquired in the first video data and the second video data respectively, and a first image set and a second image set are constructed respectively; The second image set is used as the training image set.

6. The image quality enhancement model distillation method according to claim 5, characterized in that: The first image quality enhancement model and the second image quality enhancement model both include a plurality of residual blocks; The processing flow of the first image quality enhancement model and the second image quality enhancement model includes: Performing convolution processing on the input image to obtain convolution features of the input image; Performing a nonlinear transformation on the convolution feature to obtain a nonlinear feature; Passing the nonlinear features through each of the residual blocks in sequence to obtain residual extraction features of each of the residual blocks; The residual extraction feature output by the last residual block is used as the residual image of the input image.

7. The image quality enhancement model distillation method according to claim 5, characterized in that: The training of the first image quality enhancement model includes: Inputting the second image set into the first image quality enhancement model to obtain a predicted enhanced residual image set; Superimposing the images in the second image set and the images in the prediction enhanced residual image set correspondingly to obtain a prediction enhanced image set; Calculate a first model loss of the predicted enhanced image set and the first image set, and update the first image quality enhancement model according to the first model loss.

8. A picture quality enhancement model distillation device, characterized in that: The distillation apparatus comprises: A training set acquisition module, used to acquire a training image set; A first image quality residual acquisition module, used for acquiring a first enhanced residual image set of the training image set through a pre-trained first image quality enhancement model; A second image quality residual acquisition module, used for inputting the training image set into a second image quality enhancement model to be trained to obtain a second enhanced residual image set; A distillation training module is used to perform knowledge distillation training on the second image quality enhancement model according to the losses of the second enhanced residual image set and the first enhanced residual image set.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the image quality enhancement model distillation method described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer-readable program is stored thereon, and when the computer-readable program is executed, a method for distilling an image quality enhancement model as described in any one of claims 1 to 7 is implemented.

Images