Model training method, image data enhancement method, device, equipment and product

Abstract

The invention provides a model training method, an image data enhancement method and device, electronic equipment and a computer program product, and relates to the technical field of audio and video processing. The method comprises the following steps: acquiring real degraded image data and real reference image data; performing data enhancement processing on the real degraded image data to obtain fitting reference image data; performing data degradation processing on the real reference image data to obtain fitting degraded image data; and on the basis of the real degraded image data, the fitting reference image data, the fitting degraded image data and the real reference image data, constructing a model training data pair, the model training data pair being used for training a pre-constructed initial model to obtain an image data enhancement model. According to the method, the model can adapt to a multi-source training data fitting strategy, the advantages of the multi-source data are integrated, and the model performance is stably improved under the condition that the operation resources and the model size are limited.

Description

Technical Field

[0001] The present disclosure relates to the field of audio and video processing technology, and in particular to a model training method, an image data enhancement method, a model training device, an image data enhancement device, an electronic device and a computer program product. Background Art

[0002] With the popularization of mobile Internet, video platforms have provided people with a new way of socializing, and video has become one of the most important information media today. However, due to factors such as limitations of shooting equipment, compression encoding process, multiple uploads and downloads, online video sources are complex and the quality varies. In order to bring users a better visual experience, enhancing the quality of videos has become an important task, and video quality enhancement algorithms based on deep learning have emerged.

[0003] However, processing massive amounts of online videos will incur a lot of computing costs and resource loss, and directly using lightweight small models usually cannot achieve the desired results. How to design a lightweight video quality enhancement model that is both resource-friendly and has excellent performance is still an unresolved problem. In addition, as a data-driven method, the selection and processing of training data determines the final performance of the deep learning model to a certain extent. How to properly construct the input and output data for training visual enhancement models is also a key technical problem. Summary of the invention

[0004] The present disclosure provides a model training method, an image data enhancement method, a model training device, an image data enhancement device, an electronic device, a computer-readable storage medium, and a computer program product, to at least solve the problem in the related art that a lightweight model cannot achieve the expected effect due to limited computing resources and model size, and that the constructed model training data has poor model generalization and lacks the ability to adapt to specific inputs due to domain differences. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a model training method is provided, comprising: acquiring real degraded image data and real reference image data; performing data enhancement processing on the real degraded image data to obtain fitted reference image data; performing data degradation processing on the real reference image data to obtain fitted degraded image data; and constructing a model training data pair based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data, wherein the model training data pair is used to train a pre-constructed initial model to obtain an image data enhancement model.

[0006] In an exemplary embodiment of the present disclosure, the performing data enhancement processing on the real degraded image data to obtain the fitted reference image data includes: obtaining a pre-built image enhancement teacher model; and using the image enhancement teacher model to perform data enhancement processing on the real degraded image data to obtain the fitted reference image data.

[0007] In an exemplary embodiment of the present disclosure, the performing data degradation processing on the real reference image data to obtain fitted degraded image data includes: acquiring a pre-constructed image data degradation model, the image data degradation model including an image data degradation order; and performing data degradation processing on the real reference image data based on the image data degradation order by the image data degradation model to obtain the fitted degraded image data.

[0008] In an exemplary embodiment of the present disclosure, constructing a model training data pair based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data includes: constructing a first training data pair based on the real degraded image data and the fitted reference image data; constructing a second training data pair based on the fitted degraded image data and the real reference image data; determining a data mixing ratio based on expected performance requirements; and mixing the first training data pair and the second training data pair based on the data mixing ratio to obtain the model training data pair.

[0009] In an exemplary embodiment of the present disclosure, constructing a first training data pair based on the real degraded image data and the fitted reference image data includes: using multiple data enhancement schemes to perform data enhancement processing on the fitted reference image data respectively to obtain multiple first enhanced reference image data; determining a first image data quality index value corresponding to each of the first enhanced reference image data; determining first target enhanced reference image data from the multiple first enhanced reference image data according to the first image data quality index value; and constructing the first training data pair based on the real degraded image data and the first target enhanced reference image data.

[0010] In an exemplary embodiment of the present disclosure, constructing a second training data pair based on the fitted degraded image data and the real reference image data includes: using multiple data enhancement schemes to perform data enhancement processing on the real reference image data respectively to obtain multiple second enhanced reference image data; determining a second image data quality index value corresponding to each second enhanced reference image data; determining second target enhanced reference image data from the multiple second enhanced reference image data according to the second image data quality index value; and constructing the second training data pair based on the fitted degraded image data and the second target enhanced reference image data.

[0011] In an exemplary embodiment of the present disclosure, the method further includes: acquiring the model training data pair, the model training data pair including initial degraded image data and reference enhanced image data; inputting the initial degraded image data into the initial model to obtain predicted enhanced image data; determining an image data loss value based on the predicted enhanced image data and the reference enhanced image data; and training the initial model based on the image data loss value to obtain the image data enhancement model.

[0012] According to a second aspect of an embodiment of the present disclosure, there is provided an image data enhancement method, comprising: obtaining image data to be enhanced, the image data to be enhanced comprising at least one of an image to be enhanced or a video to be enhanced; obtaining a preselected trained image data enhancement model, the image data enhancement model being trained based on any one of the above-mentioned model training methods; inputting the image data to be enhanced into the image data enhancement model, and having the image data enhancement model perform data enhancement processing on the image data to be enhanced to obtain enhanced image data.

[0013] According to a third aspect of an embodiment of the present disclosure, a model training device is provided, comprising: a real data acquisition module, used to acquire real degraded image data and real reference image data; a data enhancement module, used to perform data enhancement processing on the real degraded image data to obtain fitted reference image data; a data degradation module, used to perform data degradation processing on the real reference image data to obtain fitted degraded image data; a data pair construction module, used to construct a model training data pair based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data, wherein the model training data pair is used to train a pre-constructed initial model to obtain an image data enhancement model.

[0014] In an exemplary embodiment of the present disclosure, the data enhancement module includes a data enhancement unit, which is used to: obtain a pre-built image enhancement teacher model; use the image enhancement teacher model to perform data enhancement processing on the real degraded image data to obtain the fitted reference image data.

[0015] In an exemplary embodiment of the present disclosure, the data degradation module includes a data degradation unit, which is used to: obtain a pre-constructed image data degradation model, the image data degradation model includes an image data degradation order; and perform data degradation processing on the real reference image data based on the image data degradation order by the image data degradation model to obtain the fitted degraded image data.

[0016] In an exemplary embodiment of the present disclosure, the data pair construction module includes a data pair construction unit, which is used to: construct a first training data pair according to the real degraded image data and the fitted reference image data; construct a second training data pair according to the fitted degraded image data and the real reference image data; determine a data mixing ratio according to expected performance requirements; and based on the data mixing ratio, mix the first training data pair and the second training data pair to obtain the model training data pair.

[0017] In an exemplary embodiment of the present disclosure, the data pair construction unit includes a first construction subunit, which is used to: adopt multiple data enhancement schemes to perform data enhancement processing on the fitting reference image data respectively to obtain multiple first enhanced reference image data; determine the first image data quality index value corresponding to each of the first enhanced reference image data; determine the first target enhanced reference image data from the multiple first enhanced reference image data according to the first image data quality index value; and construct the first training data pair according to the real degraded image data and the first target enhanced reference image data.

[0018] In an exemplary embodiment of the present disclosure, the data pair construction unit includes a second construction subunit, which is used to: adopt multiple data enhancement schemes to perform data enhancement processing on the real reference image data respectively to obtain multiple second enhanced reference image data; determine the second image data quality index value corresponding to each second enhanced reference image data; determine the second target enhanced reference image data from the multiple second enhanced reference image data according to the second image data quality index value; and construct the second training data pair according to the fitted degraded image data and the second target enhanced reference image data.

[0019] In an exemplary embodiment of the present disclosure, the model training device also includes a model training module, which is used to: obtain the model training data pair, the model training data pair including initial degraded image data and reference enhanced image data; input the initial degraded image data into the initial model to obtain predicted enhanced image data; determine an image data loss value based on the predicted enhanced image data and the reference enhanced image data; and train the initial model based on the image data loss value to obtain the image data enhancement model.

[0020] According to a fourth aspect of an embodiment of the present disclosure, an image data enhancement device is provided, comprising: a data acquisition module for acquiring image data to be enhanced, wherein the image data to be enhanced includes at least one of an image to be enhanced or a video to be enhanced; a model acquisition module, wherein the image data enhancement model is acquired through preselected training, wherein the image data enhancement model is trained based on any one of the above-mentioned model training methods; and an image data enhancement module, wherein the image data to be enhanced is input into the image data enhancement model, and the image data enhancement model performs data enhancement processing on the image data to be enhanced to obtain enhanced image data.

[0021] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute instructions to implement any one of the above-described model training methods, or to implement the above-described image data enhancement method.

[0022] According to the sixth aspect of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any one of the above-mentioned model training methods, or implement the above-mentioned image data enhancement method.

[0023] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements any one of the above-mentioned model training methods, or implements the above-mentioned image data enhancement method.

[0024] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0025] On the one hand, using real degraded image data as model training data will eliminate the domain difference problem between training data and test data, and can improve the generalization of the model in real scenes. On the other hand, using real reference image data as model training data can greatly improve the model enhancement results obtained through training. On the other hand, training data pairs are constructed based on real degraded image data and fitted degraded image data, and are used in the model training process, so that the model can adapt to the multi-source training data fitting strategy, and integrate the advantages of each multi-source data, so that the model can steadily improve the model performance under the condition of limited computing resources and model size.

[0026] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0028] Figure 1 The figure is a flowchart of a model training method according to an exemplary embodiment.

[0029] Figure 2 It is an overall flow chart of an image data enhancement technical solution according to an exemplary embodiment.

[0030] Figure 3 The figure is a flow chart showing a method of performing data degradation processing on real reference image data according to an exemplary embodiment.

[0031] Figure 4 The figure is a flow chart of an image data enhancement method according to an exemplary embodiment.

[0032] Figure 5 It is a schematic diagram showing that the disclosure has the effect of enhancing the clarity of results according to an exemplary embodiment.

[0033] Figure 6 The figure is a schematic diagram showing the effect of the disclosure having an enhanced result without noise artifacts according to an exemplary embodiment.

[0034] Figure 7 It is a block diagram of a model training device according to an exemplary embodiment.

[0035] Figure 8 The figure is a block diagram of an image data enhancement device according to an exemplary embodiment.

[0036] Fig. 9A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0037] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0039] In a related technical solution, a commonly used model lightweighting method is model distillation. Specifically, model distillation usually selects a pre-trained strong model with a large number of parameters as the teacher model, and provides the intermediate features or final output of the model operation as an additional supervision signal; the lightweight model to be trained is used as the student model and is trained under the additional supervision signal provided by the teacher model. For visual enhancement models, the teacher model can be directly used as a fitting model for low-quality and high-quality training data pairs.

[0040] However, the above scheme has two main problems: (1) Although the teacher model has excellent performance, the student model will also learn some unfavorable characteristics of the teacher model itself, such as color cast and texture repair loss, while distilling and learning the pre-training knowledge of the teacher model. (2) Directly using the output of the teacher model as the supervision signal for the training of the student model, the teacher model restricts the performance upper limit of the student model, resulting in insufficient clarity of the video enhancement results of the student model.

[0041] In another related scheme, a high-order degradation model is also a method for constructing training data pairs for video enhancement models. The high-order degradation model can design and introduce multiple degradation operations, and simulate the complex degradation process in the real world by combining and randomizing parameters, fitting low-quality data from real high-quality data, and obtaining data pairs for training video enhancement models.

[0042] Although the implementation scheme of the high-order degradation model is simple and easy to implement, it also has two problems: (1) There is a domain difference between the low-quality data constructed by the high-order degradation model and the real data. Therefore, the enhanced model obtained by this scheme will have poor generalization in real scenarios due to the domain offset between the training data and the test data. (2) Although combination and randomization can expand the domain range of fitting data, they still lack the ability to adapt to specific inputs.

[0043] Based on this, according to the embodiments of the present disclosure, a model training method, an image data enhancement method, a model training device, an image data enhancement device, an electronic device, a computer-readable storage medium, and a computer program product are proposed.

[0044] Figure 1 is a flow chart of a model training method according to an exemplary embodiment. Figure 1 As shown, the model training method can be used in a computer device, wherein the computer device described in the present disclosure may include mobile terminal devices such as mobile phones, tablet computers, laptops, PDAs, and fixed terminal devices such as desktop computers. This exemplary embodiment uses the method applied to a computer device as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a computer device and a server, and is implemented through the interaction between the computer device and the server. Specifically, the following steps are included.

[0045] Step S110, acquiring real degraded image data and real reference image data;

[0046] Step S120, performing data enhancement processing on the real degraded image data to obtain fitting reference image data;

[0047] Step S130, performing data degradation processing on the real reference image data to obtain fitted degraded image data;

[0048] Step S140, constructing a model training data pair based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data, the model training data pair is used to train the pre-built initial model to obtain an image data enhancement model.

[0049] According to the model training method in this example embodiment, on the one hand, by using real degraded image data as model training data, there will be no domain difference problem between training data and test data, which can improve the generalization of the model in real scenes. On the other hand, by using real reference image data as model training data, the model enhancement results obtained by training can be greatly improved. On the other hand, a training data pair is constructed based on real degraded image data and fitted degraded image data, and is used in the model training process, so that the model can adapt to the multi-source training data fitting strategy, and integrate the advantages of each multi-source data, so that the model can steadily improve the model performance under the condition of limited computing resources and model size.

[0050] The model training method in this example embodiment will be further described below.

[0051] In step S110 , real degraded image data and real reference image data are acquired.

[0052] In an exemplary embodiment of the present disclosure, the real degraded image data may be degraded image data with low picture quality obtained from a real application scene, and the real degraded image data may be image data generated due to factors such as image degradation. For example, the real degraded image data may be an image or video in which one or more of the quality indicator parameters are less than a predefined quality parameter threshold. For example, the image quality indicator may include but is not limited to the resolution, color depth, signal-to-noise ratio (SNR), contrast, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), etc. of the image. The real reference image data may be image data with high picture quality obtained from a real application scene, for example, the real reference image data may be an image or video in which one or more of the image data quality indicator parameter values ​​are greater than or equal to a predefined quality parameter threshold.

[0053] Image degradation refers to the phenomenon that the image quality decreases during the formation, recording, processing and transmission of the image due to the imperfections of the imaging system, recording equipment, transmission medium and processing method. In actual application scenarios, due to factors such as shooting equipment limitations, compression encoding process, multiple uploads and downloads, the image quality of the real image is reduced, that is, real degraded image data is generated. Before constructing the model training data pair, low-quality real degraded image data and high-quality real reference image data can be obtained respectively. Both types of image data are real image data, and there is a difference in picture quality between the two. The image content of the two types of image data can be related or unrelated.

[0054] In step S120, data enhancement processing is performed on the real degraded image data to obtain fitting reference image data.

[0055] In an exemplary embodiment of the present disclosure, data enhancement processing may be a process of enhancing low-quality real degraded image data to obtain high-quality image output results. Fitting reference image data may be high-quality output data obtained after data enhancement processing is performed on real degraded image data.

[0056] For the low-quality real degraded image data obtained, data enhancement processing can be performed on the real degraded image data. There are various methods for data enhancement, including traditional image processing-based methods such as flipping, rotation, scaling, translation, cropping, color transformation, noise addition, blurring and affine transformation, etc.; there are also model generation-based methods such as Generative Adversarial Network (GAN), Variational Auto-Encoders (VAE) and enhanced adversarial samples, etc. In addition, there are active learning enhancement methods such as active learning, domain adaptation, hybrid strategy, etc. Using the above data enhancement methods, high-quality fitting reference image data corresponding to low-quality real degraded image data can be generated.

[0057] In an exemplary embodiment of the present disclosure, for step S120, data enhancement processing is performed on the real degraded image data to obtain fitting reference image data, including: obtaining a pre-built image enhancement teacher model; using the image enhancement teacher model, performing data enhancement processing on the real degraded image data to obtain fitting reference image data.

[0058] Model distillation is a commonly used model lightweight method. In order to overcome the shortcomings of using both teacher model and student model for model training in related technical solutions, this embodiment proposes to use a pre-trained strong model (i.e., teacher model) to process low-quality real degraded image data to obtain high-quality image data output corresponding to data enhancement, i.e., fitting reference image data.

[0059] Specifically, refer to Figure 2 , Figure 2 FIG. 1 is an overall flow chart of an image data enhancement technical solution according to an exemplary embodiment. Figure 2In the data fitting link 1, low-quality real degraded image data is input into the pre-trained strong model, and the real low-quality image data is processed with the help of the pre-trained strong model to obtain enhanced high-quality image data output as fitting reference image data. The pre-trained strong model, also known as the image enhancement teacher model, can be a pre-trained strong model with a large number of parameters used in the model distillation technology. The fitting reference image data obtained by fitting the pre-trained strong model can be combined with the real degraded image data to construct a model training data pair, providing richer model training data.

[0060] In step S130, data degradation processing is performed on the real reference image data to obtain fitted degraded image data.

[0061] In an exemplary embodiment of the present disclosure, data degradation processing may be a process of obtaining low-quality image data corresponding to real reference image data through multiple degradation operations. Fitted degraded image data may be low-quality fitted degraded image data obtained by performing data degradation processing on real reference image data. Fitted degraded image data may be image data in which one or more of the quality indicator parameter values ​​are less than a predefined quality parameter threshold.

[0062] For the acquired high-quality real reference image data, data degradation processing can be performed on the real reference image data, such as data degradation processing can include image downsampling, adding noise, image compression and other processing methods; data degradation processing can also include using an image data degradation model. Through data degradation processing, corresponding low-quality fitting degraded image data is obtained. Using the above data degradation method, low-quality fitting degraded image data corresponding to high-quality real reference image data can be generated, so as to construct a model training data pair based on the fitting degraded image data.

[0063] In an exemplary embodiment of the present disclosure, for step S130, data degradation processing is performed on real reference image data to obtain fitted degraded image data, including: obtaining a pre-constructed image data degradation model, the image data degradation model including an image data degradation order; and performing data degradation processing on the real reference image data based on the image data degradation order by the image data degradation model to obtain fitted degraded image data.

[0064] The image data degradation order may be a specific order of degradation processing performed by the image data degradation model on the real reference image data. The n-order model involves n repeated degradation processes, wherein each degradation process adopts a classical degradation model with the same process but different hyperparameters.

[0065] Continue to refer Figure 2 , Figure 2The data fitting link 2 in the method processes the high-quality real reference image data through a pre-built high-order degradation model (i.e., the image data degradation model) to obtain a degraded low-quality image output, i.e., fitting the degraded image data.

[0066] refer to Figure 3 , Figure 3 FIG. 1 is a flow chart showing a method of performing data degradation processing on real reference image data according to an exemplary embodiment. Figure 3 It can be seen that the high-order degradation model in this embodiment adopts a second-order degradation process. In the first-order degradation process, the real reference image data input to the high-order degradation model is blurred, scaled, denoised, and compressed in the Joint Photographic Experts Group (JPEG) storage format to obtain the intermediate image data after the first-order degradation process.

[0067] Then the intermediate image data is subjected to second-order degradation processing, which also includes blurring, scale change, and JPEG compression processing; in addition, compared with the first-order degradation processing, the second-order degradation processing also adds sine filtering processing, and finally obtains the fitted degraded image data. The subsequent fitted degraded image data can be combined with the real reference image data to construct a model training data pair, making the model training data pair richer.

[0068] In step S140, based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data, a model training data pair is constructed, and the model training data pair is used to train the pre-constructed initial model to obtain an image data enhancement model.

[0069] In an exemplary embodiment of the present disclosure, the model training data pair may be a set of data pairs for training an image data enhancement model, and the model training data pair may include an input-output supervision pair consisting of a plurality of low-quality image data and corresponding high-quality reference image data. The image data enhancement model may be a network model for performing data enhancement processing on low-quality image data to output high-quality image data. The initial model may be a pre-trained lightweight model. In the model distillation technology, the lightweight model has fewer model parameters than the strong model.

[0070] After the fitting reference image data and the fitting degraded image data are constructed through the above steps, they can be paired with the corresponding real degraded image data and real reference image data respectively to construct multiple input-output supervision pairs for model training as model training data pairs. The constructed model training data pairs are used in the training process of the initial model to finally obtain the image data enhancement model.

[0071] In an exemplary embodiment of the present disclosure, for step S140, a model training data pair is constructed based on real degraded image data, fitted reference image data, fitted degraded image data and real reference image data, including: constructing a first training data pair according to the real degraded image data and the fitted reference image data; constructing a second training data pair according to the fitted degraded image data and the real reference image data; determining a data mixing ratio according to expected performance requirements; and mixing the first training data pair and the second training data pair based on the data mixing ratio to obtain a model training data pair.

[0072] Among them, the first training data pair can be an input-output supervision pair obtained by combining real degraded image data and fitted reference image data. The second training data pair can be an input-output supervision pair obtained by combining fitted degraded image data and real reference image data. The expected performance requirement can be the requirement of the corresponding training goal that the user expects the image data enhancement model to ultimately achieve when training the model, such as the user's expected performance requirement can be the expectation of obtaining a more natural visual enhancement effect, or a sharper visual enhancement result. The data mixing ratio can be the mixing ratio of the first training data pair and the second training data pair.

[0073] When a certain data mixing ratio is used to mix the data constructed by the two data fitting links, the existence of data fitting link one ensures the generalization of the overall model, while the existence of data fitting link two makes up for the limited sharpness of link one and improves the overall effectiveness of the model. The expected model characteristics can also be controlled by adjusting the mixing ratio of the two link data. If you want the model enhancement result to be more natural, you can increase the proportion of data fitting link one; if you want the model enhancement result to be sharper, you can increase the proportion of data fitting link two. The specific value of the data mixing ratio can be adjusted according to the expected performance requirements, so that the model training data pair constructed after the adjustment can meet the expected performance requirements.

[0074] In an exemplary embodiment of the present disclosure, a first training data pair is constructed based on real degraded image data and fitted reference image data, including: using multiple data enhancement schemes to perform data enhancement processing on the fitted reference image data respectively to obtain multiple first enhanced reference image data; determining a first image data quality index value corresponding to each first enhanced reference image data; determining first target enhanced reference image data from the multiple first enhanced reference image data according to the first image data quality index value; and constructing a first training data pair based on the real degraded image data and the first target enhanced reference image data.

[0075] Among them, the data enhancement scheme may include a traditional data enhancement scheme, and may also include a data enhancement scheme implemented by a data enhancement model based on deep learning. The first enhanced reference image data may be enhanced high-quality image data obtained after data enhancement processing is performed on the fitting reference image data using multiple data enhancement schemes. The first image data quality index value may be a specific index value for quality assessment of multiple first enhanced reference image data. The first target enhanced reference image data may be enhanced high-quality image data selected from multiple first enhanced reference image data based on the specific value of the first image data quality index value. The first target enhanced reference image data may be used to construct a model training data pair.

[0076] Continue to refer Figure 2 ,In order to further improve the model performance and enhance the model's adaptability to different inputs, Figure 2 In both data fitting links, a high-quality data enhancement strategy based on objective indicator screening is added. For the fitting reference image data of data fitting link 1 and the real reference image data of data fitting link 2, multiple data enhancement schemes are used to perform data enhancement processing on both.

[0077] For the data fitting link 1, multiple data enhancement schemes are used to perform data enhancement processing on the fitting reference image data respectively, and multiple first enhanced reference image data can be obtained. For example, the data enhancement scheme may include traditional enhancement methods, such as the Unsharp Masking (USM) method. The USM method is a technology that increases the clarity by enhancing the edge details of the image. Its basic principle is to enhance the details and edges of the image by identifying the difference between the original image and its blurred version. The data enhancement scheme may also include a data enhancement process implemented by a pre-trained enhancement network based on deep learning.

[0078] After obtaining a plurality of first enhanced reference image data, objective evaluation is performed on each first enhanced reference image data, for example, the quality of different enhancement results is evaluated using objective evaluation indicators to obtain the first image data quality indicator value corresponding to each first enhanced reference image data. The objective evaluation indicator may include a video / picture quality evaluation (Kuaishou Visual Quality, KVQ) indicator; the objective evaluation indicator may also include an image quality evaluation indicator, such as PSNR, SSIM and other indicators.

[0079] Then, according to the first image data quality index value, the first target enhanced reference image data with the highest score is selected from multiple first enhanced reference image data to serve as a supervisory signal for training the lightweight image data enhancement model. The real degraded image data and the selected first target enhanced reference image data are processed into data pairs to obtain a first training data pair. The real degraded image data (i.e., real low-quality data) and the first target enhanced reference image data (i.e., enhanced image data output by the strong model) can constitute the input and output supervision pair for training the lightweight image enhancement model. Since the input of this link is real low-quality data, there is no domain difference problem between the training data and the test data, which can improve the generalization of the lightweight image data enhancement model in real scenes.

[0080] In an exemplary embodiment of the present disclosure, a second training data pair is constructed based on the fitted degraded image data and the real reference image data, including: using multiple data enhancement schemes to perform data enhancement processing on the real reference image data respectively to obtain multiple second enhanced reference image data; determining a second image data quality index value corresponding to each second enhanced reference image data; determining second target enhanced reference image data from the multiple second enhanced reference image data according to the second image data quality index value; and constructing a second training data pair based on the fitted degraded image data and the second target enhanced reference image data.

[0081] The second enhanced reference image data may be enhanced high-quality image data obtained by performing data enhancement processing on the real reference image data using multiple data enhancement schemes. The second image data quality index value may be a specific index value used to perform quality assessment on multiple second enhanced reference image data. The second target enhanced reference image data may be enhanced high-quality image data selected from multiple second enhanced reference image data according to the specific value of the second image data quality index value. The second target enhanced reference image data may be used to construct a model training data pair.

[0082] Continue to refer Figure 2 Similarly, for the data fitting link 2, multiple data enhancement schemes are used to perform data enhancement processing on the real reference image data respectively, and multiple second enhanced reference image data can be obtained. After obtaining multiple second enhanced reference image data, each second enhanced reference image data is objectively evaluated to obtain the second image data quality index value corresponding to each second enhanced reference image data.

[0083] Then, according to the second image data quality index value, the second target enhanced reference image data with the highest score is selected from the plurality of second enhanced reference image data to be used as a supervisory signal for training the lightweight image data enhancement model. The data enhancement scheme and objective evaluation index adopted in the data fitting link 2 are the same as those in the data fitting link 1, and will not be described in detail in this disclosure.

[0084] The second training data pair can be obtained by constructing the fitted degraded image data and the selected second target enhanced reference image data. The fitted degraded image data (i.e., the low-quality image data degraded by the model) and the second target enhanced reference image data (i.e., the enhanced real reference image data) can also constitute the input and output supervision pair for training the lightweight image enhancement model. This data fitting link uses the real high-quality data as the reference target for model training, and the clarity of the model enhancement results obtained through this link training can be greatly improved.

[0085] In an exemplary embodiment of the present disclosure, a model training data pair is obtained, the model training data pair including initial degraded image data and reference enhanced image data; the initial degraded image data is input into an initial model to obtain predicted enhanced image data; an image data loss value is determined based on the predicted enhanced image data and the reference enhanced image data; and the initial model is trained based on the image data loss value to obtain an image data enhancement model.

[0086] Among them, the initial degraded image data, i.e., the low-quality training data used for model training, can be the low-quality image data in the model training data pair, and the initial degraded image data can be the image data whose data quality index value is less than the preset quality parameter threshold. The initial degraded image data includes real degraded image data and fitted degraded image data. The reference enhanced image data, i.e., the high-quality training data used for model training, can be the high-quality image data corresponding to the initial degraded image data, and the reference enhanced image data can include real reference image data and fitted reference image data, as well as enhanced high-quality data obtained after data enhancement processing of both. The predicted enhanced image data can be the enhanced image data obtained after the initial model enhancement processing, and the predicted enhanced image data can be considered as the enhanced result output by the model. The image data loss value can be an indicator used to measure the difference between the predicted enhanced image data (the enhanced result predicted by the model) and the reference enhanced image data (the target reference result), and the high or low loss value reflects the degree of fit of the model to the training data under the current parameters.

[0087] Continue to refer Figure 2After constructing the model training data pair, the initial degraded image data in the model training data pair is used as the model input of the initial model. The initial model can be a pre-trained lightweight image data enhancement model, such as the initial model can be a student model in the model distillation technology. The initial degraded image data is enhanced by the pre-constructed initial model to obtain the predicted enhanced image data; in addition, the reference enhanced image data in the model training data pair is used as the supervision data of the model output, and the image data loss value between the predicted enhanced image data and the reference enhanced image data is determined.

[0088] Subsequently, the initial model can be trained according to the image data loss value until the loss function converges or the specified training round has been reached, or other model training termination conditions are met, and the image data enhancement model is obtained. For example, when the image data loss value fluctuates within a certain range and no longer decreases significantly, it can be considered that the model has converged and the training can be terminated. Through the above steps, a network model for image data enhancement processing can be trained. The trained image data enhancement model can integrate the advantages of each multi-source data processing link, maximize the potential of the lightweight model, and achieve the dual advantages of computing power and performance.

[0089] In summary, the model training method disclosed in the present invention obtains real degraded image data and real reference image data; performs data enhancement processing on the real degraded image data to obtain fitting reference image data; performs data degradation processing on the real reference image data to obtain fitting degraded image data; constructs a model training data pair based on the real degraded image data, fitting reference image data, fitting degraded image data and real reference image data, and the model training data pair is used to train the pre-built initial model to obtain an image data enhancement model. On the one hand, using the real degraded image data as the model training data will not have the domain difference problem between the training data and the test data, and can improve the generalization of the model in the real scene. On the other hand, using the real reference image data as the model training data can greatly improve the model enhancement result obtained by training. On the other hand, constructing a training data pair based on the real degraded image data and the fitting degraded image data, and using it in the model training process, the model can adapt to the multi-source training data fitting strategy, and integrate the advantages of each of the multi-source data, so that the model can steadily improve the model performance under the condition of limited computing resources and model size. On the other hand, the lightweight image data enhancement solution based on knowledge distillation and multi-source data hybrid training can maximize the potential of lightweight models and achieve the dual advantages of computing power and performance through strong model knowledge distillation in the pre-deployment training stage and adaptive multi-source training data fitting strategy, when computing resources and model size are limited.

[0090] Secondly, the present disclosure also proposes an image data enhancement method, referring to Figure 4 , Figure 4 is a flowchart of an image data enhancement method according to an exemplary embodiment. Figure 4 As shown, the image data enhancement method can be used in a computer device, the method can also be applied to a server, and can also be applied to a system including a computer device and a server, and is implemented through the interaction between the computer device and the server. Specifically, the following steps are included.

[0091] In step S410, image data to be enhanced is acquired, where the image data to be enhanced includes at least one of an image to be enhanced or a video to be enhanced.

[0092] In an exemplary embodiment of the present disclosure, the image data to be enhanced may be image data to be subjected to data enhancement processing, and the image data to be enhanced may include but is not limited to image data or video data.

[0093] Obtain the image data to be enhanced. In real application scenarios, the image data to be enhanced may be low-quality video data or image data uploaded by users to the video platform. For example, the quality of the video data or image data may be low due to limitations of the shooting equipment.

[0094] In step S420, a pre-selected trained image data enhancement model is obtained, where the image data enhancement model is trained based on a model training method.

[0095] In an exemplary embodiment of the present disclosure, a pre-trained image data enhancement model is obtained. The image data enhancement model is a lightweight image data enhancement model obtained based on knowledge distillation and multi-source data mixed training. Under the condition of limited computing resources and model size, through strong model knowledge distillation in the pre-deployment training stage and adaptive multi-source training data fitting strategy, the respective advantages of multi-source data processing links are integrated, so that the final model training result releases the potential of the lightweight model to the greatest extent and achieves the dual advantages of computing power and performance.

[0096] In step S430, the image data to be enhanced is input into the image data enhancement model, and the image data enhancement model performs data enhancement processing on the image data to be enhanced to obtain enhanced image data.

[0097] In an exemplary embodiment of the present disclosure, the enhanced image data may be high-quality image data that has been enhanced by an image data enhancement model.

[0098] The acquired image data to be enhanced is input into the pre-trained image data enhancement model, which can perform data enhancement processing on it and output the corresponding enhanced image data. Figure 5 , Figure 5FIG. 1 is a schematic diagram showing that the disclosure has a high clarity effect of enhancing the result according to an exemplary embodiment. Figure 5 It can be seen that for the input video frame, if only the data fitting link 1 is used to construct the data pair training model, the enhanced video frame output by the model has insufficient clarity and noise amplification due to insufficient generalization. By adopting the solution disclosed in the present invention, the above shortcomings can be avoided and an enhanced output with higher clarity can be obtained.

[0099] Further, refer to Figure 6 , Figure 6 FIG. 1 is a schematic diagram showing the effect of the disclosure having an enhanced result without noise artifacts according to an exemplary embodiment. Figure 6 It can be seen that for the input video frame, if only the data fitting link 2 is used to construct the data pair training model, the enhanced video frame output by the model will have noise artifacts; the enhanced visual image output by the image data enhancement model trained by the present disclosure does not have noise artifacts, and a clear and natural enhanced output is obtained.

[0100] In summary, the image data enhancement method disclosed in the present invention can, on the one hand, avoid the noise amplification phenomenon caused by insufficient clarity of the enhanced output obtained by training a lightweight model using a model distillation scheme, so that the enhanced output of the model has higher clarity. On the other hand, the use of a multi-source data hybrid training method that has adaptive capabilities for different data can enhance the generalization ability of the model, that is, the model's adaptive capabilities for specific inputs, to obtain an enhanced output that is both clear and natural.

[0101] Figure 7 is a block diagram of a model training device according to an exemplary embodiment. Figure 7 The model training device 700 includes: a real data acquisition module 710, a data enhancement module 720, a data degradation module 730 and a data pair construction module 740.

[0102] Specifically, the real data acquisition module 710 is used to acquire real degraded image data and real reference image data; the data enhancement module 720 is used to perform data enhancement processing on the real degraded image data to obtain fitted reference image data; the data degradation module 730 is used to perform data degradation processing on the real reference image data to obtain fitted degraded image data; the data pair construction module 740 is used to construct a model training data pair based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data, and the model training data pair is used to train the pre-constructed initial model to obtain an image data enhancement model.

[0103] In an exemplary embodiment of the present disclosure, the data enhancement module 720 includes a data enhancement unit, which is used to: obtain a pre-built image enhancement teacher model; use the image enhancement teacher model to perform data enhancement processing on real degraded image data to obtain fitting reference image data.

[0104] In an exemplary embodiment of the present disclosure, the data degradation module 730 includes a data degradation unit, which is used to: obtain a pre-constructed image data degradation model, the image data degradation model includes an image data degradation order; and perform data degradation processing on real reference image data based on the image data degradation order by the image data degradation model to obtain fitted degraded image data.

[0105] In an exemplary embodiment of the present disclosure, the data pair construction module 740 includes a data pair construction unit, which is used to: construct a first training data pair based on real degraded image data and fitted reference image data; construct a second training data pair based on fitted degraded image data and real reference image data; determine the data mixing ratio according to expected performance requirements; and based on the data mixing ratio, mix the first training data pair and the second training data pair to obtain a model training data pair.

[0106] In an exemplary embodiment of the present disclosure, the data pair construction unit includes a first construction subunit, which is used to: adopt multiple data enhancement schemes to perform data enhancement processing on the fitting reference image data respectively to obtain multiple first enhanced reference image data; determine the first image data quality index value corresponding to each first enhanced reference image data; determine the first target enhanced reference image data from the multiple first enhanced reference image data according to the first image data quality index value; and construct a first training data pair according to the real degraded image data and the first target enhanced reference image data.

[0107] In an exemplary embodiment of the present disclosure, the data pair construction unit includes a second construction subunit, which is used to: adopt multiple data enhancement schemes to perform data enhancement processing on real reference image data respectively to obtain multiple second enhanced reference image data; determine the second image data quality index value corresponding to each second enhanced reference image data; determine the second target enhanced reference image data from the multiple second enhanced reference image data according to the second image data quality index value; and construct a second training data pair according to the fitted degraded image data and the second target enhanced reference image data.

[0108] In an exemplary embodiment of the present disclosure, the model training device 700 also includes a model training module, which is used to: obtain a model training data pair, the model training data pair including initial degraded image data and reference enhanced image data; input the initial degraded image data into the initial model to obtain predicted enhanced image data; determine the image data loss value based on the predicted enhanced image data and the reference enhanced image data; train the initial model based on the image data loss value to obtain an image data enhancement model.

[0109] Figure 8 is a block diagram of an image data enhancement device according to an exemplary embodiment. Figure 8 The image data enhancement device 800 includes: a data acquisition module 810 for enhancement, a model acquisition module 820 and an image data enhancement module 830.

[0110] Specifically, the data acquisition module 810 for enhancement is used to acquire the image data to be enhanced, which includes at least one of the images to be enhanced or the videos to be enhanced; the model acquisition module 820 is used to acquire a pre-selected trained image data enhancement model, which is trained based on any of the above-mentioned model training methods; the image data enhancement module 830 is used to input the image data to be enhanced into the image data enhancement model, and the image data enhancement model performs data enhancement processing on the image data to be enhanced to obtain enhanced image data.

[0111] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0112] Reference below Fig. 9 hereinafter describes an electronic device 900 according to such an embodiment of the present disclosure. Fig. 9 The electronic device 900 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0113] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include, but are not limited to: the at least one processing unit 910, the at least one storage unit 920, a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910), and a display unit 940.

[0114] The storage unit stores program codes, which can be executed by the processing unit 910, so that the processing unit 910 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0115] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 921 and / or a cache memory unit 922 , and may further include a read-only memory unit (ROM) 923 .

[0116] The storage unit 920 may include a program / utility 924 having a set (at least one) of program modules 925, such program modules 925 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0117] Bus 930 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0118] The electronic device 900 may also communicate with one or more external devices 970 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0119] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, and the above instructions can be executed by a processor of the device to complete the above model training method or image data enhancement method. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0120] In an exemplary embodiment, a computer program product is also provided, including a computer program, which implements any one of the above-mentioned model training methods or image data enhancement methods when executed by a processor.

[0121] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0122] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that: include: Acquire real degraded image data and real reference image data; Performing data enhancement processing on the real degraded image data to obtain fitting reference image data; Performing data degradation processing on the real reference image data to obtain fitted degraded image data; Based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data, a model training data pair is constructed, and the model training data pair is used to train a pre-constructed initial model to obtain an image data enhancement model.

2. The method according to claim 1, characterized in that The performing data enhancement processing on the real degraded image data to obtain fitting reference image data includes: Get pre-built image enhancement teacher models; The image enhancement teacher model is used to perform data enhancement processing on the real degraded image data to obtain the fitted reference image data.

3. The method according to claim 1, characterized in that The performing data degradation processing on the real reference image data to obtain fitted degraded image data includes: Acquire a pre-built image data degradation model, wherein the image data degradation model includes an image data degradation order; The image data degradation model performs data degradation processing on the real reference image data based on the image data degradation order to obtain the fitted degraded image data.

4. The method according to claim 1, characterized in that The constructing of a model training data pair based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data comprises: Constructing a first training data pair according to the real degraded image data and the fitted reference image data; constructing a second training data pair according to the fitted degraded image data and the real reference image data; Determine the data mix ratio based on expected performance requirements; Based on the data mixing ratio, the first training data pair and the second training data pair are mixed to obtain the model training data pair.

5. The method according to claim 4, characterized in that The step of constructing a first training data pair according to the real degraded image data and the fitted reference image data comprises: Using a plurality of data enhancement schemes, respectively performing data enhancement processing on the fitting reference image data to obtain a plurality of first enhanced reference image data; Determining a first image data quality index value corresponding to each of the first enhanced reference image data; Determining first target enhanced reference image data from the plurality of first enhanced reference image data according to the first image data quality indicator value; The first training data pair is constructed according to the real degraded image data and the first target enhanced reference image data.

6. The method according to claim 4, characterized in that The step of constructing a second training data pair according to the fitted degraded image data and the real reference image data comprises: Using multiple data enhancement schemes, respectively performing data enhancement processing on the real reference image data to obtain multiple second enhanced reference image data; Determining a second image data quality index value corresponding to each of the second enhanced reference image data; determining second target enhanced reference image data from the plurality of second enhanced reference image data according to the second image data quality indicator value; The second training data pair is constructed according to the fitted degraded image data and the second target enhanced reference image data.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire the model training data pair, wherein the model training data pair includes initial degraded image data and reference enhanced image data; Inputting the initial degraded image data into the initial model to obtain predicted enhanced image data; Determining an image data loss value according to the predicted enhanced image data and the reference enhanced image data; The initial model is trained based on the image data loss value to obtain the image data enhancement model.

8. An image data enhancement method, characterized in that: include: Acquire image data to be enhanced, where the image data to be enhanced includes at least one of an image to be enhanced or a video to be enhanced; Obtaining a pre-selected trained image data enhancement model, wherein the image data enhancement model is trained based on the model training method according to any one of claims 1 to 7; The image data to be enhanced is input into the image data enhancement model, and the image data enhancement model performs data enhancement processing on the image data to be enhanced to obtain enhanced image data.

9. A model training device, characterized in that: include: A real data acquisition module, used to acquire real degraded image data and real reference image data; A data enhancement module, used for performing data enhancement processing on the real degraded image data to obtain fitting reference image data; A data degradation module, used for performing data degradation processing on the real reference image data to obtain fitted degraded image data; A data pair construction module is used to construct a model training data pair based on the real degraded image data, the fitted reference image data, the fitted degraded image data and the real reference image data, and the model training data pair is used to train a pre-constructed initial model to obtain an image data enhancement model.

10. An image data enhancement device, characterized in that: include: A module for acquiring data to be enhanced, used for acquiring image data to be enhanced, wherein the image data to be enhanced includes at least one of an image to be enhanced or a video to be enhanced; A model acquisition module, used to acquire a pre-selected trained image data enhancement model, wherein the image data enhancement model is trained based on the model training method according to any one of claims 1 to 7; The image data enhancement module is used to input the image data to be enhanced into the image data enhancement model, and the image data enhancement model performs data enhancement processing on the image data to be enhanced to obtain enhanced image data.

11. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the model training method as described in any one of claims 1 to 7, or to implement the image data enhancement method as described in claim 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the model training method as described in any one of claims 1 to 7, or implements the image data enhancement method as described in claim 8.

Images