Image processing model training method and device, electronic device

Abstract

The present disclosure relates to a training method for an image processing model, comprising: inputting a sample image into a target model for prediction to obtain a first predicted image, wherein the target model is obtained by training a first preset machine learning model; obtaining a first target image having an image frequency higher than a preset frequency from the first predicted image, wherein the image frequency is used to characterize the variation of the pixel grayscale value of the image in space; training a second preset machine learning model based on the sample image and the first target image to obtain an image processing model, wherein the parameter amount and the calculation amount of the image processing model are less than the parameter amount and the calculation amount of the target model, and the image processing model is used for predictive processing of the image.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning, and in particular to a method and device for compressing a neural network model, and an electronic device. Background Art

[0002] In recent years, generative adversarial neural networks (GANs) have made significant breakthroughs in many fields such as image generation, and have been widely used in tasks such as image stylization and image editing, and have achieved good results. However, existing GAN models often have a large number of parameters and computational complexity. Therefore, such models are difficult to run on devices with limited computing and storage capabilities, which seriously limits the application of GAN models.

[0003] Image stylization is one of the most commonly used applications of the GAN model. It can perform some style conversion on the input image, such as converting ordinary natural pictures into hand-painted style pictures. However, the standard image stylization model (for example, CycleGAN) requires about 56.8 billion multiplication and addition operations and X million parameters to process a single picture. Generally speaking, the computing and storage capabilities of common mobile devices are relatively limited. Therefore, if the standard image stylization model is directly deployed on a mobile device, the model will run extremely slowly.

[0004] Currently, due to the large number of parameters and computational complexity, the generative adversarial network model is difficult to run on devices with limited computing and storage capabilities, which limits the application of the generative adversarial network model. Summary of the invention

[0005] The present disclosure provides a training method and device for an image processing model, and an electronic device, to at least solve the problem in the related art that the generative adversarial network model is difficult to run on devices with limited computing and storage capabilities due to the large number of parameters and calculations, resulting in limited application of the generative adversarial network model. The technical solution of the present disclosure is as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, a method for training an image processing model is provided, comprising: inputting a sample image into a target model for prediction to obtain a first predicted image, wherein the target model is obtained by training a first preset machine learning model; obtaining a first target image having an image frequency higher than a preset frequency from the first predicted image, wherein the image frequency is used to characterize the variation of pixel grayscale values ​​of the image in space; training a second preset machine learning model based on the sample image and the first target image to obtain an image processing model, wherein the number of parameters and the amount of calculation of the image processing model are less than those of the target model, and the image processing model is used for predictive processing of images.

[0007] Optionally, the first preset machine learning model and the second preset machine learning model both include: a generative adversarial network model.

[0008] Optionally, a second preset machine learning model is trained based on the sample image and the first target image to obtain a second target model, including: inputting the sample image and the first target image into the second preset machine learning model for prediction to obtain a second predicted image; obtaining a second target image having an image frequency higher than a preset frequency from the second predicted image; determining a loss function of the first target image and the second target image; and optimizing the loss function to obtain an image processing model when the value of the loss function is lower than a preset threshold.

[0009] Optionally, obtaining a first target image having an image frequency higher than a preset frequency from the first predicted image, and obtaining a second target image having an image frequency higher than a preset frequency from the second predicted image, include: performing frequency decomposition on the first predicted image and the second predicted image, respectively, to obtain first predicted sub-images of different frequencies and second predicted sub-images of different frequencies; and filtering out the first target image and the second target image from the first predicted sub-images of different frequencies and the second predicted sub-images of different frequencies, respectively.

[0010] Optionally, the first predicted image and the second predicted image are subjected to frequency decomposition respectively, which is achieved by one of the following methods: discrete wavelet transform, discrete Fourier transform and discrete cosine transform.

[0011] Optionally, when the first predicted image and the second predicted image are subjected to frequency decomposition by discrete wavelet transform, the above method further includes: performing frequency decomposition on the first predicted image and the second predicted image respectively by using a plurality of wavelet basis functions; performing single frequency decomposition or multiple frequency decomposition on the first predicted image and the second predicted image respectively by using any one of the plurality of wavelet basis functions.

[0012] Optionally, after obtaining the image processing model, the method further includes: replacing the target model with the image processing model; and performing predictive processing on the image with the image processing model.

[0013] According to a second aspect of an embodiment of the present disclosure, a training device for an image processing model is also provided, including: a prediction module, configured to execute image prediction by inputting a sample image into a target model to obtain a first predicted image, wherein the target model is obtained by training a first preset machine learning model; an acquisition module, configured to execute acquisition of a first target image having an image frequency higher than a preset frequency from the first predicted image, wherein the image frequency is used to characterize the change in the pixel grayscale value of the image in space; a training module, configured to execute training of a second preset machine learning model based on the sample image and the first target image to obtain an image processing model, wherein the parameter amount and calculation amount of the image processing model are less than the parameter amount and calculation amount of the target model, and the image processing model is used for predictive processing of the image.

[0014] Optionally, the first preset machine learning model and the second preset machine learning model both include: a generative adversarial network model.

[0015] Optionally, the training module includes: a prediction unit, configured to input the sample image and the first target image into a second preset machine learning model for prediction to obtain a second predicted image; an acquisition unit, configured to acquire a second target image having an image frequency higher than a preset frequency from the second predicted image; a determination unit, configured to determine a loss function of the first target image and the second target image; and a processing unit, configured to optimize the loss function and obtain an image processing model when the value of the loss function is lower than a preset threshold.

[0016] Optionally, the above-mentioned device also includes: a first frequency decomposition module, configured to perform frequency decomposition on the first predicted image and the second predicted image respectively to obtain first predicted sub-images with different frequencies and second predicted sub-images with different frequencies; a screening module, configured to perform screening out the first target image and the second target image from the first predicted sub-images with different frequencies and the second predicted sub-images with different frequencies respectively.

[0017] Optionally, the frequency decomposition module is configured to perform frequency decomposition on the first predicted image and the second predicted image respectively by one of the following methods: discrete wavelet transform, discrete Fourier transform and discrete cosine transform.

[0018] Optionally, when the first predicted image and the second predicted image are subjected to frequency decomposition through discrete wavelet transform, the above-mentioned device also includes: a second frequency decomposition module, configured to perform frequency decomposition of the first predicted image and the second predicted image respectively through multiple wavelet basis functions; and a third frequency decomposition module, configured to perform single frequency decomposition or multiple frequency decomposition of the first predicted image and the second predicted image respectively through any one of the multiple wavelet basis functions.

[0019] Optionally, the above-mentioned device also includes: a replacement module, configured to replace the target model using the image processing model; and an image processing module, configured to perform predictive processing on the image using the image processing model.

[0020] According to a third aspect of an embodiment of the present disclosure, there is also provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute instructions to implement the above image processing model training method.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above image processing model training method.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program / instruction, which implements the above image processing model training method when the computer program / instruction is executed by a processor.

[0023] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: achieving compression of a generative adversarial network model, and improving the quality of images generated by the generative adversarial network model.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0026] Figure 1 is a flowchart of a method for training an image processing model according to an exemplary embodiment;

[0027] Figure 2 is a structural block diagram of a training device for an image processing model according to an exemplary embodiment;

[0028] Figure 3 The present invention is a structural block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0031] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0032] GAN (Generative Adversarial Network), a generative adversarial network is a deep neural network architecture, which usually includes a discriminator network and a generator network, where the generator network is used to generate the required images or other types of data, and the discriminator is used to determine whether the data is generated by the generator or sampled from the data set.

[0033] DWT (Discrete Wavelet Transformation), discrete wavelet transform is the result of discretizing the scale and translation of the basic wavelet transform, and is a commonly used tool for frequency analysis.

[0034] FID (Frechet Inception Distance), Frechet Inception distance is a common indicator for measuring the performance of GAN models. Generally speaking, the larger the FID between real pictures and pictures generated by GAN models, the worse the performance of GAN models, and vice versa.

[0035] CycleGAN, the cyclic adversarial generative network is a GAN model based on cycle consistency loss. The model can be trained without paired training data to achieve the effect of image-to-image translation.

[0036] The knowledge distillation algorithm is one of the most effective algorithms for achieving neural network model compression. It first trains a teacher model with a large number of parameters and computational complexity, and then trains a student model with a small number of parameters and computational complexity. During the training of the student model, the knowledge in the teacher network is transferred to the student model to improve the performance of the student model. When applied, the network can be compressed by replacing the teacher model with the student model. However, most of the existing knowledge distillation algorithms are designed for high-level visual tasks, but GAN models are often used for low-level visual tasks. Although a small number of works have attempted to apply knowledge distillation to the compression of GAN models, the results are not ideal.

[0037] In 2020, scholars such as Muyang Li proposed the GAN Compression algorithm, which compresses the GAN model through neural network architecture search and knowledge distillation. The algorithm mainly consists of three parts: First, the L2-norm distillation loss function between the features of the student network and the teacher network in the middle layer of the network is introduced. By optimizing the distillation loss function, the knowledge is transferred from the teacher network to the student network. Second, the neural network architecture search is performed on the structure of the GAN model, and the search is accelerated through weight sharing technology.

[0038] Although the GAN Compression algorithm achieves a certain degree of GAN model compression effect, the use of the knowledge distillation algorithm is extremely simple and ineffective. For example, the knowledge distillation algorithm only brings a very small decrease in the FID index. Secondly, this knowledge distillation algorithm based on intermediate layer features requires the teacher network and the student network to have a similar network architecture, which greatly limits the application of this technology.

[0039] The present disclosure believes that the reason why existing knowledge distillation algorithms do not work well on GAN models is that these algorithms do not consider the differences in different frequencies of the image. Generally speaking, the quality of the generated image is reflected in its high-frequency information. Since high-frequency information often reflects the detailed information of the image, a picture with high-quality high-frequency information is more ideal in the eyes of humans. In contrast, the low-frequency information in the generated image has little effect. The existing knowledge distillation algorithm simply allows the student model to learn all the prediction results of the teacher model, and does not make differentiated adjustments for different frequencies.

[0040] To solve this problem, the present invention extracts information of different frequencies in the teacher and student models through discrete wavelet transform, and then strengthens the student model's learning of high-frequency information of the image, while suppressing the learning of low-frequency information of the image, thereby improving the effect of the knowledge distillation algorithm.

[0041] The above method is described below with a specific embodiment:

[0042] Figure 1 is a flowchart of a method for training an image processing model according to an exemplary embodiment. Figure 1 As shown, the method comprises the following steps:

[0043] In step S11, the sample image is input into the target model for prediction to obtain a first predicted image, wherein the target model is obtained by training a first preset machine learning model;

[0044] It should be noted that the target model is trained using a standard training method with a GAN teacher model that has a large number of parameters and computational complexity. In this step, the sample image is input into the teacher model and the predicted image is output.

[0045] In step S12, a first target image having an image frequency higher than a preset frequency is obtained from the first predicted image, wherein the image frequency is used to characterize the change of the pixel grayscale value of the image in space;

[0046] The frequency of an image refers to the spatial frequency. An image can be considered as a signal defined on a two-dimensional plane, the amplitude of which corresponds to the grayscale of the pixel (color images correspond to the three components of RGB). The frequency of an image is called the spatial frequency, which refers to the number of periodic changes in brightness per unit length. It reflects the change in the grayscale of the image pixels in space.

[0047] The first target image is a high-frequency image extracted from the first predicted image and having an image frequency higher than a preset frequency.

[0048] In step S13, a second preset machine learning model is trained based on the sample image and the first target image to obtain an image processing model, wherein the number of parameters and the amount of calculation of the image processing model are less than the number of parameters and the amount of calculation of the target model, and the image processing model is used for predictive processing of the image;

[0049] In this step, the teacher network trained in the previous step is used to train a GAN student model (i.e., image processing model) with fewer parameters and computational complexity by means of knowledge distillation.

[0050] It should be noted that the trained GAN student model is used to perform image prediction processing tasks.

[0051] Through the above method, by allowing the student model to focus on learning the more important high-frequency information in the image and avoiding the interference of unimportant low-frequency information, the technical effect of improving the compression effect of the adversarial network model and improving the quality of the generated images of the generative adversarial network model is achieved.

[0052] According to an optional embodiment of the present application, the first preset machine learning model and the second preset machine learning model are both generative adversarial network models.

[0053] According to another optional embodiment of the present application, step S13 trains the first target model based on the sample image and the target image to obtain the second target model, which is achieved by the following method: inputting the sample image and the first target image into a second preset machine learning model for prediction to obtain a second predicted image; obtaining a second target image whose image frequency is higher than a preset frequency from the second predicted image; determining the loss function of the first target image and the second target image; optimizing the loss function, and obtaining the image processing model when the value of the loss function is lower than a preset threshold.

[0054] For each image in the sample image, let the student network and the teacher network predict the generated results respectively. And perform discrete wavelet transform on the predicted results of the two. Calculate the L1-norm loss function of the high-frequency information part of the transformed result as the loss function of the wavelet distillation algorithm. And optimize this loss function together with other loss functions, so as to achieve the transfer of knowledge in the teacher network to the student model during the training process of the student model.

[0055] In some optional embodiments of the present application, obtaining a first target image having an image frequency higher than a preset frequency from a first predicted image, and obtaining a second target image having an image frequency higher than a preset frequency from a second predicted image, include: performing frequency decomposition on the first predicted image and the second predicted image, respectively, to obtain first predicted sub-images of different frequencies and second predicted sub-images of different frequencies; and filtering out the first target image and the second target image from the first predicted sub-images of different frequencies and the second predicted sub-images of different frequencies, respectively.

[0056] In some other optional embodiments of the present application, the first predicted image and the second predicted image are subjected to frequency decomposition respectively, which is achieved by one of the following methods: discrete wavelet transform, discrete Fourier transform and discrete cosine transform.

[0057] In the embodiments provided in the present application, in addition to the method of performing frequency decomposition of the generated image by discrete wavelet transform described in the above scheme, discrete Fourier transform and discrete cosine transform can also be used to perform frequency decomposition of the image. The subsequent steps are the same as using discrete wavelet transform.

[0058] According to an optional embodiment of the present application, when the first predicted image and the second predicted image are subjected to frequency decomposition by discrete wavelet transform, the above method further includes: performing frequency decomposition on the first predicted image and the second predicted image respectively by multiple wavelet basis functions; performing single frequency decomposition or multiple frequency decomposition on the first predicted image and the second predicted image respectively by any one of the multiple wavelet basis functions.

[0059] It should be noted that if discrete wavelet transform is used to perform frequency decomposition on an image, the discrete wavelet transform can use different wavelet basis functions. The discrete wavelet transform can be selected to perform single or multiple decompositions.

[0060] The above method provided by this application does not modify the network architecture of the model, nor is it limited to the network architecture of the teacher model and the student model. It can be applied to teacher and student GAN models of any structure and to any image and video generation tasks.

[0061] According to an optional embodiment of the present application, after executing step S13 to obtain the image processing model, the target model is replaced by the image processing model; and the image is predictively processed using the image processing model.

[0062] When deploying the application, the trained student model is used to replace the teacher model to achieve the compression of the GAN model. The student model is then used for image prediction processing.

[0063] The above method provided in the present application can significantly improve the quality of generated images of GAN models with less parameters and computational complexity. For example, in the Horse2Zebra dataset, on the CycleGAN model, in the "horse to zebra" image translation task, a standard-sized model has 11.38MB of network parameters and 47.22 floating-point operations, and its FID is 61.53. A small model compressed 4 times has 2.85MB of parameters and 12.14G of floating-point operations, and its FID is 84.89. The 4-fold compressed small model trained using the distillation technology of the present invention has 2.85MB of parameters and 12.14G of floating-point operations, and its FID is 75.45. Compared with the model trained without using the present invention, there is a 9.44 decrease in FID.

[0064] Figure 2 is a structural block diagram of a training device for an image processing model according to an exemplary embodiment. Figure 2 As shown, the device comprises:

[0065] The prediction module 20 is configured to perform image prediction by inputting the sample image into a target model to obtain a first predicted image, wherein the target model is obtained by training a first preset machine learning model;

[0066] The target model is trained using a GAN teacher model with a large number of parameters and computational complexity using a standard training method.

[0067] An acquisition module 22 is configured to acquire a first target image having an image frequency higher than a preset frequency from the first predicted image, wherein the image frequency is used to characterize a change in a pixel grayscale value of the image in space;

[0068] The first target image is a high-frequency image extracted from the first predicted image and having an image frequency higher than a preset frequency.

[0069] The training module 24 is configured to perform training on a second preset machine learning model based on the sample image and the first target image to obtain an image processing model, wherein the number of parameters and the amount of calculation of the image processing model are less than the number of parameters and the amount of calculation of the target model, and the image processing model is used for predictive processing of the image.

[0070] Use the teacher network trained in the previous step to train a GAN student model (i.e., image processing model) with fewer parameters and less computation in a knowledge distillation manner.

[0071] Through the above-mentioned device, the student model is made to focus on learning the more important high-frequency information in the image, while avoiding the interference of unimportant low-frequency information, thereby achieving the technical effect of improving the compression effect of the adversarial network model and improving the quality of the generated image of the generative adversarial network model.

[0072] According to an optional embodiment of the present application, the first preset machine learning model and the second preset machine learning model both include: a generative adversarial network model.

[0073] According to another optional embodiment of the present application, the training module 24 includes: a prediction unit, configured to execute inputting the sample image and the first target image into a second preset machine learning model for prediction to obtain a second predicted image; an acquisition unit, configured to execute acquisition of a second target image having an image frequency higher than a preset frequency from the second predicted image; a determination unit, configured to execute determination of a loss function of the first target image and the second target image; and a processing unit, configured to execute optimization of the loss function to obtain an image processing model when the value of the loss function is lower than a preset threshold.

[0074] For each image in the sample image, let the student network and the teacher network predict the generated results respectively. And perform discrete wavelet transform on the predicted results of the two. Calculate the L1-norm loss function of the high-frequency information part of the transformed result as the loss function of the wavelet distillation algorithm. And optimize this loss function together with other loss functions, so as to achieve the transfer of knowledge in the teacher network to the student model during the training process of the student model.

[0075] In some optional embodiments of the present application, the above-mentioned device also includes: a first frequency decomposition module, configured to perform frequency decomposition on the first predicted image and the second predicted image respectively to obtain first predicted sub-images of different frequencies and second predicted sub-images of different frequencies; a screening module, configured to perform screening out the first target image and the second target image from the first predicted sub-images of different frequencies and the second predicted sub-images of different frequencies respectively.

[0076] In an optional embodiment, the frequency decomposition module is configured to perform frequency decomposition on the first predicted image and the second predicted image respectively by one of the following methods: discrete wavelet transform, discrete Fourier transform and discrete cosine transform.

[0077] In the embodiment provided in the present application, in addition to the method of performing frequency decomposition on the generated image by discrete wavelet transform, discrete Fourier transform and discrete cosine transform can also be used to perform frequency decomposition on the image. The subsequent steps are the same as using discrete wavelet transform.

[0078] In some other optional embodiments of the present application, when the first predicted image and the second predicted image are subjected to frequency decomposition through discrete wavelet transform, the above-mentioned device also includes: a second frequency decomposition module, configured to perform frequency decomposition of the first predicted image and the second predicted image respectively through multiple wavelet basis functions; and a third frequency decomposition module, configured to perform single frequency decomposition or multiple frequency decomposition of the first predicted image and the second predicted image respectively through any one of the multiple wavelet basis functions.

[0079] It should be noted that if discrete wavelet transform is used to perform frequency decomposition on an image, the discrete wavelet transform can use different wavelet basis functions. The discrete wavelet transform can be selected to perform single or multiple decompositions.

[0080] According to an optional embodiment of the present application, the above-mentioned device also includes: a replacement module, configured to replace the target model using the image processing model; and an image processing module, configured to perform predictive processing on the image using the image processing model.

[0081] When deploying the application, the trained student model is used to replace the teacher model to achieve the compression of the GAN model. The student model is then used for image prediction processing.

[0082] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0083] Figure 3 is a structural block diagram of an electronic device according to an exemplary embodiment. Figure 3The electronic device may be a terminal 300, for example, the terminal 300 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0084] Terminal 300 may include one or more of the following components: processing component 302 , memory 304 , power component 306 , multimedia component 308 , audio component 310 , input / output (I / O) interface 312 , sensor component 313 , and communication component 316 .

[0085] The processing component 302 generally controls the overall operation of the terminal 300, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 302 may include one or more processors 320 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 302 may include one or more modules to facilitate the interaction between the processing component 302 and other components. For example, the processing component 302 may include a multimedia module to facilitate the interaction between the multimedia component 308 and the processing component 302.

[0086] The memory 304 is configured to store various types of data to support operations on the device 300. Examples of such data include instructions for any application or method operating on the terminal 300, contact data, phone book data, messages, pictures, videos, etc. The memory 304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0087] Power component 306 provides power to various components of terminal 300. Power component 306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to terminal 300.

[0088] The multimedia component 308 includes a screen that provides an output interface between the terminal 300 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 308 includes a front camera and / or a rear camera. When the device 300 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0089] The audio component 310 is configured to output and / or input audio signals. For example, the audio group 310 includes a microphone (MIC), and when the terminal 300 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 304 or sent via the communication component 316. In some embodiments, the audio component 310 also includes a speaker for outputting audio signals.

[0090] I / O interface 312 provides an interface between processing component 302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0091] The sensor assembly 314 includes one or more sensors for providing various aspects of status assessment for the terminal 300. For example, the sensor assembly 314 can detect the open / closed state of the device 300, the relative positioning of the components, such as the display and keypad of the terminal 300, and the sensor assembly 314 can also detect the position change of the terminal 300 or a component of the terminal 300, the presence or absence of contact between the user and the terminal 300, the orientation or acceleration / deceleration of the terminal 300, and the temperature change of the terminal 300. The sensor assembly 314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 314 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 314 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0092] The communication component 316 is configured to facilitate wired or wireless communication between the terminal 300 and other devices. The terminal 300 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 3G, 3G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 316 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 316 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0093] In an exemplary embodiment, the terminal 300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0094] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 304 including instructions, which can be executed by a processor 320 of an electronic device to perform the above method. The computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0095] In an exemplary embodiment, a computer program product is also provided, including a computer program / instruction, which implements the above image processing model training method when the computer program / instruction is executed by a processor.

[0096] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0097] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image processing model, characterized in that: include: Inputting the sample image into a target model for image prediction to obtain a first predicted image, wherein the target model is obtained by training a first preset machine learning model; Acquire a first target image having an image frequency higher than a preset frequency from the first predicted image, wherein the image frequency is used to characterize a change in a pixel grayscale value of the image in space; A second preset machine learning model is trained based on the sample image and the first target image to obtain an image processing model, including: inputting the sample image and the first target image into the second preset machine learning model for image prediction to obtain a second predicted image; obtaining a second target image whose image frequency is higher than the preset frequency from the second predicted image; determining the loss function of the first target image and the second target image; optimizing the loss function, and obtaining the image processing model when the value of the loss function is lower than a preset threshold, wherein the loss function includes: a loss function of a high-frequency information part in a transformation result, the transformation result is obtained by performing discrete wavelet transform on a prediction result, the prediction result includes: a prediction result of the sample image by a student network, a prediction result of the sample image by a teacher network, the parameter amount and the calculation amount of the image processing model are less than the parameter amount and the calculation amount of the target model, and the image processing model is used for image prediction processing.

2. The method according to claim 1, characterized in that: Both the first preset machine learning model and the second preset machine learning model include: a generative adversarial network model.

3. The method according to claim 1, characterized in that Acquiring a first target image having an image frequency higher than a preset frequency from the first predicted image, and acquiring a second target image having an image frequency higher than the preset frequency from the second predicted image, comprising: Performing frequency decomposition on the first predicted image and the second predicted image respectively to obtain first predicted sub-images with different frequencies and second predicted sub-images with different frequencies; The first target image and the second target image are respectively selected from the first predicted sub-images of different frequencies and the second predicted sub-images of different frequencies.

4. The method according to claim 3, characterized in that The first predicted image and the second predicted image are subjected to frequency decomposition respectively, which is achieved by one of the following methods: discrete wavelet transform, discrete Fourier transform and discrete cosine transform.

5. The method according to claim 4, characterized in that In the case where the first predicted image and the second predicted image are subjected to frequency decomposition by the discrete wavelet transform, the method further comprises: Performing frequency decomposition on the first predicted image and the second predicted image respectively by using a plurality of wavelet basis functions; By using any one of the plurality of wavelet basis functions, single frequency decomposition or multiple frequency decomposition is performed on the first predicted image and the second predicted image respectively.

6. The method according to claim 1, characterized in that After obtaining the image processing model, the method further includes: replacing the target model with the image processing model; The image processing model is used to perform predictive processing on the image.

7. A training device for an image processing model, characterized in that: include: A prediction module is configured to perform image prediction by inputting a sample image into a target model to obtain a first predicted image, wherein the target model is obtained by training a first preset machine learning model; an acquisition module, configured to acquire, from the first predicted image, a first target image having an image frequency higher than a preset frequency, wherein the image frequency is used to characterize a change in a pixel grayscale value of the image in space; a training module configured to perform training on a second preset machine learning model based on the sample image and the first target image to obtain an image processing model, wherein the number of parameters and the amount of calculation of the image processing model are less than the number of parameters and the amount of calculation of the target model, and the image processing model is used for prediction processing of the image; The training module includes: a prediction unit, configured to input the sample image and the first target image into the second preset machine learning model for prediction to obtain a second predicted image; an acquisition unit, configured to acquire a second target image with an image frequency higher than the preset frequency from the second predicted image; a determination unit, configured to determine the loss function of the first target image and the second target image; a processing unit, configured to optimize the loss function, and obtain the image processing model when the value of the loss function is lower than a preset threshold, wherein the loss function includes: a loss function of a high-frequency information part in a transformation result, the transformation result is obtained by performing a discrete wavelet transform on a prediction result, and the prediction result includes: a prediction result of the sample image by a student network and a prediction result of the sample image by a teacher network.

8. The device according to claim 7, characterized in that Both the first preset machine learning model and the second preset machine learning model include: a generative adversarial network model.

9. The device according to claim 7, characterized in that The device further comprises: A first frequency decomposition module is configured to perform frequency decomposition on the first predicted image and the second predicted image respectively to obtain first predicted sub-images with different frequencies and second predicted sub-images with different frequencies; The screening module is configured to screen out the first target image and the second target image from the first predicted sub-images of different frequencies and the second predicted sub-images of different frequencies, respectively.

10. The device according to claim 9, characterized in that The frequency decomposition module is configured to perform frequency decomposition on the first predicted image and the second predicted image respectively by one of the following methods: discrete wavelet transform, discrete Fourier transform and discrete cosine transform.

11. The device according to claim 10, characterized in that When the first predicted image and the second predicted image are subjected to frequency decomposition by the discrete wavelet transform, the device further comprises: A second frequency decomposition module is configured to perform frequency decomposition on the first predicted image and the second predicted image respectively through a plurality of wavelet basis functions; The third frequency decomposition module is configured to perform single frequency decomposition or multiple frequency decomposition on the first predicted image and the second predicted image respectively through any one of the multiple wavelet basis functions.

12. The device according to claim 7, characterized in that The device also includes: A replacement module, configured to replace the target model with the image processing model; The image processing module is configured to perform prediction processing on the image using the image processing model.

13. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; Wherein, the processor is configured to execute the instructions to implement the training method of the image processing model as described in any one of claims 1 to 6.

14. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the training method of the image processing model as described in any one of claims 1 to 6.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the training method of the image processing model described in any one of claims 1 to 6 is implemented.

Images