Image processing method, apparatus, medium, and device

Abstract

This application discloses an image processing method, apparatus, medium, and device. The method involves acquiring a first image, inputting the first image into a resolution optimization model for resolution optimization, and obtaining a second image. The second resolution of the second image is greater than the first resolution of the first image. The resolution optimization model is trained jointly with a teacher model using a joint loss function. This application allows for knowledge distillation from both local and global frequency dimensions, making the image super-resolution output of the knowledge-distilled student model more similar to that of the teacher model in terms of local and global details. This enables efficient knowledge distillation of image super-resolution by the teacher and student models with different model structures, improving the image super-resolution processing capability and optimization effect of the knowledge-distilled student model (i.e., the resolution optimization model).

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an image processing method, apparatus, medium, and device. Background Technology

[0002] Image super-resolution refers to the process of converting low-resolution (LR) images into high-resolution (HR) images through a series of techniques. Its main purpose is to improve image sharpness and detail, making the image more visually realistic and usable.

[0003] Currently, image super-resolution can be achieved using neural network models, but these models are complex and slow. To improve processing speed and conserve computational resources, knowledge distillation has been proposed. For example, knowledge about image super-resolution processing from a complex, high-performance teacher model can be transferred to a simpler student model. This allows the student model to learn key features and behaviors of the teacher model, thereby reducing model complexity and improving processing speed while maintaining a certain level of image super-resolution processing capability.

[0004] However, related technologies require that the teacher and student models be combinations of models with the same network structure. But with the development of artificial intelligence, various models have emerged, often resulting in combinations of teacher and student models with different network structures. Due to the different network structures, the two models process information differently, which leads to a decrease in the efficiency of knowledge distillation, or even the inability to perform knowledge distillation at all. This results in poor super-resolution processing capabilities for the student model after knowledge distillation. Summary of the Invention

[0005] This application provides an image processing method, apparatus, medium, and device that can achieve efficient knowledge distillation for image super-resolution when the teacher model and student model have different model structures. This improves the ability of the student model (i.e., the resolution optimization model) after knowledge distillation to perform subsequent image super-resolution processing, thereby improving the resolution of the image obtained by the student model after knowledge distillation through image super-resolution processing and enhancing the optimization effect of image processing.

[0006] To address the aforementioned technical problems, this application provides the following technical solutions:

[0007] An image processing method, comprising:

[0008] Acquire a first image, the first image having a first resolution;

[0009] The first image is input into the resolution optimization model to obtain a second image with a second resolution greater than the first resolution. The resolution optimization model is obtained by jointly training the student model and the teacher model using the first sample image based on the joint loss function. The first sample image has an optimized image label, which is the training target image of the first sample image. The joint loss function is a function that includes a first loss, a second loss, and a third loss.

[0010] Wherein, the first loss corresponds to the discrete wavelet subband difference between the first optimized image and the second optimized image, the first optimized image is obtained by the student model by optimizing the resolution of the first sample image, and the second optimized image is obtained by the teacher model by optimizing the resolution of the first sample image; the second loss corresponds to the frequency domain difference between the first optimized image and the second optimized image; and the third loss corresponds to the pixel difference between the optimized image label and the first optimized image.

[0011] An image processing apparatus, comprising:

[0012] The acquisition unit is used to acquire a first image, wherein the first image has a first resolution;

[0013] The resolution optimization unit is used to input the first image into the resolution optimization model to obtain a second image with a second resolution greater than the first resolution. The resolution optimization model is obtained by jointly training the student model and the teacher model using the first sample image based on the joint loss function during training. The first sample image has an optimized image label, which is the training target image of the first sample image. The joint loss function is a function including a first loss, a second loss, and a third loss.

[0014] Wherein, the first loss corresponds to the discrete wavelet subband difference between the first optimized image and the second optimized image, the first optimized image is obtained by the student model by optimizing the resolution of the first sample image, and the second optimized image is obtained by the teacher model by optimizing the resolution of the first sample image; the second loss corresponds to the frequency domain difference between the first optimized image and the second optimized image; and the third loss corresponds to the pixel difference between the optimized image label and the first optimized image.

[0015] In some embodiments, the apparatus further includes a training unit, the training unit comprising:

[0016] A subunit is used to acquire a first sample image and the corresponding optimized image label of the first sample image;

[0017] The resolution optimization subunit is used to input the first sample image into the student model for resolution optimization to obtain a first optimized image, and input the first sample image into the teacher model for resolution optimization to obtain a second optimized image. The teacher model and the student model have different model structures.

[0018] The first loss generation subunit is used to perform discrete wavelet transform on the first optimized image to obtain a first discrete wavelet subband, and to perform discrete wavelet transform on the second optimized image to obtain a second discrete wavelet subband, and to generate a first loss based on the difference between the second discrete wavelet subband and the first discrete wavelet subband.

[0019] The second loss generation subunit is used to perform a discrete Fourier transform on the first optimized image to obtain a first frequency domain, and to perform a discrete Fourier transform on the second optimized image to obtain a second frequency domain, and to generate a second loss based on the difference between the second frequency domain and the first frequency domain.

[0020] The third loss generation subunit is used to generate a third loss based on the difference between the first pixel of the optimized image label and the second pixel of the first optimized image;

[0021] The calculation subunit is used to input the first loss, the second loss, and the third loss into the joint loss function for calculation to obtain the joint loss;

[0022] The training subunit is used to train the student model using the joint loss to obtain a resolution-optimized model.

[0023] In some embodiments, the first loss generation subunit includes:

[0024] The first discrete wavelet transform module is used to perform discrete wavelet transform on the first optimized image through a discrete wavelet transform template to obtain a first discrete wavelet sub-band. The first discrete wavelet sub-band includes at least a first low-frequency discrete wavelet sub-band, a first high-frequency discrete wavelet sub-band, a second high-frequency discrete wavelet sub-band, and a third high-frequency discrete wavelet sub-band.

[0025] The second discrete wavelet transform module is used to perform discrete wavelet transform on the second optimized image through the discrete wavelet transform template to obtain a second discrete wavelet sub-band. The second discrete wavelet sub-band includes at least a second low-frequency discrete wavelet sub-band, a fourth high-frequency discrete wavelet sub-band, a fifth high-frequency discrete wavelet sub-band, and a sixth high-frequency discrete wavelet sub-band.

[0026] Among them, the first high-frequency discrete wavelet subband, the second high-frequency discrete wavelet subband, and the third high-frequency discrete wavelet subband represent different frequency ranges, and the fourth high-frequency discrete wavelet subband, the fifth high-frequency discrete wavelet subband, and the sixth high-frequency discrete wavelet subband represent different frequency ranges.

[0027] The first loss generation module is used to generate a first loss based on the difference between the second discrete wavelet subband and the first discrete wavelet subband.

[0028] In some embodiments, the first loss generation module includes:

[0029] The first sub-loss generation submodule is used to obtain the first sub-loss based on the difference between the first low-frequency discrete wavelet subband and the second low-frequency discrete wavelet subband.

[0030] The second sub-loss generation submodule is used to obtain the second sub-loss based on the difference between the first high-frequency discrete wavelet sub-band and the fourth high-frequency discrete wavelet sub-band;

[0031] The third sub-loss generation submodule is used to obtain the third sub-loss based on the difference between the second high-frequency discrete wavelet sub-band and the fifth high-frequency discrete wavelet sub-band;

[0032] The fourth sub-loss generation submodule is used to obtain the fourth sub-loss based on the difference between the third high-frequency discrete wavelet sub-band and the sixth high-frequency discrete wavelet sub-band;

[0033] The combined submodule is used to combine the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss to obtain the first loss.

[0034] In some embodiments, the combining submodule is used for:

[0035] Multiply the first sub-loss by the first weight to obtain the first target sub-loss;

[0036] Multiply the second sub-loss by the second weight to obtain the second target sub-loss;

[0037] Multiplying the third sub-loss and the third weight yields the third target sub-loss;

[0038] Multiply the fourth sub-loss and the fourth weight to obtain the fourth target sub-loss;

[0039] By combining the first target sub-loss, the second target sub-loss, the third target sub-loss, and the fourth target sub-loss, the first loss is obtained;

[0040] The sum of the first weight, the second weight, the third weight, and the fourth weight is one.

[0041] In some embodiments, the first loss generation module further includes a weight determination submodule, the weight determination submodule being configured to:

[0042] Determine the image type information of the first sample image;

[0043] Based on the image type information, the values ​​of the first weight, the second weight, the third weight, and the fourth weight are determined respectively.

[0044] In some embodiments, the second loss generation subunit is configured to:

[0045] The first optimized image is subjected to a discrete Fourier transform using a discrete Fourier transform template to obtain the first frequency domain.

[0046] The second optimized image is subjected to a discrete Fourier transform using a discrete Fourier transform template to obtain the second frequency domain.

[0047] The second loss is obtained based on the difference between the second frequency domain and the first frequency domain.

[0048] In some embodiments, the third loss generation subunit is configured to:

[0049] Obtain the first pixel value of the first pixel corresponding to the optimized image label;

[0050] Obtain the second pixel value of the second pixel corresponding to the first optimized image;

[0051] A third loss is generated based on the difference between the first pixel value and the second pixel value.

[0052] In some embodiments, the computing subunit includes:

[0053] The first weighting module obtains a first preset weight value and weights the second loss using the first preset weight value to obtain the weighted second loss;

[0054] The calculation module is used to input the first loss, the weighted second loss, and the third loss into the joint loss function for calculation to obtain the joint loss.

[0055] In some embodiments, the calculation subunit further includes a second weighting module for:

[0056] Obtain a second preset weight value, which is greater than the first preset weight value; weight the third loss using the second preset weight value to obtain a weighted third loss;

[0057] The calculation module is also used to input the first loss, the weighted second loss, and the weighted third loss into the joint loss function for calculation to obtain the joint loss.

[0058] In some embodiments, the training subunit is configured to:

[0059] The student model is iteratively trained using the joint loss until the training termination condition is met, resulting in a resolution-optimized model.

[0060] The training termination condition includes at least the convergence of the joint loss or the number of training iterations reaching a preset threshold.

[0061] A computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to perform the image processing method described above.

[0062] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image processing method described above.

[0063] A computer program product or computer program includes computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions to implement the image processing method described above.

[0064] Although efficient knowledge distillation is difficult between teacher and student models with different model structures, this embodiment adds the learning of discrete wavelet subband differences and frequency domain differences to fully utilize the knowledge contained in the teacher model. Specifically, during training, this embodiment uses a joint loss function that includes a first loss, a second loss, and a third loss. The first loss is determined based on the discrete wavelet subband differences between the first and second optimized images. The first optimized image is obtained by the student model through resolution optimization of the first sample image, and the second optimized image is obtained by the teacher model through resolution optimization of the first sample image. The first loss enables knowledge distillation at local frequencies, allowing the student model to mimic the teacher model's representation at local frequencies.

[0065] The second loss is determined based on the frequency domain difference between the first and second optimized images. This second loss enables knowledge distillation at the global frequency level, allowing the student model to mimic the teacher model's representation at the global frequency level. The third loss is determined based on the pixel difference between the optimized image label and the first optimized image. This third loss enables the student model to learn the pixel value representation of the optimized image label. Therefore, compared to knowledge distillation methods based on related knowledge, this embodiment achieves knowledge distillation from both global and local frequency dimensions. This makes the image super-resolution output of the student model after knowledge distillation closer to the image super-resolution output of the teacher model in terms of both local and global details. It allows for efficient knowledge distillation of image super-resolution by the teacher and student models with different model structures, improving the ability of the student model (i.e., the resolution optimization model) to perform subsequent image super-resolution processing. Thus, inputting the first image to be optimized into the resolution optimization model yields a second image with higher image super-resolution quality, improving the optimization effect of image processing.

[0066] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a scene diagram of the image processing system provided in the embodiments of this application.

[0069] Figure 2 This is a schematic diagram of a scene for the image processing method provided in the embodiments of this application.

[0070] Figure 3 This is a schematic flowchart of the image processing method provided in an embodiment of this application.

[0071] Figure 4 This is another scenario diagram of the image processing method provided in the embodiments of this application.

[0072] Figure 5 This is a schematic diagram of the SRCNN model architecture provided in an embodiment of this application.

[0073] Figure 6This is another schematic flowchart of the image processing method provided in the embodiments of this application.

[0074] Figure 7 This is a schematic diagram of a scenario for the resolution optimization model training method provided in an embodiment of this application.

[0075] Figure 8 A schematic flowchart of an optional image processing method provided in an embodiment of this application.

[0076] Figure 9 Experimental data diagrams for the image processing method provided in the embodiments of this application.

[0077] Figure 10 Another experimental data diagram for the image processing method provided in the embodiments of this application.

[0078] Figure 11 This is another experimental data diagram of the image processing method provided in the embodiments of this application.

[0079] Figure 12 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0080] Figure 13 This is a schematic diagram of the terminal structure provided in the embodiments of this application.

[0081] Figure 14 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0082] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] It is understood that in the specific embodiments of this application, data such as images are involved. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the subject is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0084] Furthermore, when this application embodiment needs to obtain images and other related data, it will obtain separate permission or separate consent for the images and other related data through pop-up windows or redirection to a confirmation page. After clearly obtaining separate permission or separate consent for the images and other related data, it will then obtain the necessary medical images and other related data for enabling the application embodiment to operate normally.

[0085] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0086] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0087] Frequency: Frequency refers to the number of periodic events that occur per unit of time. In physics and engineering, for periodic signals such as sine waves, frequency is usually represented by a symbol, and the unit is Hertz (Hz). One Hertz means that one periodic change occurs per second. For example, a sine wave completes 10 full cycles in one second.

[0088] Discrete wavelets are a mathematical tool with important applications in signal processing and image processing. They are the discrete form of wavelet transform, representing signals or images as combinations of different frequency subbands by decomposing them.

[0089] Discrete Wavelet Transform (DWT) is a signal processing technique used to decompose a signal into subbands of different frequencies. The basic principle is to achieve signal decomposition through a series of filter banks. It uses low-pass filters (LPFs) and high-pass filters (HPFs) to decompose the signal into approximation coefficients (low-frequency components) and detail coefficients (high-frequency components). Multiresolution analysis is a key technology of DWT, which can decompose a signal or image into discrete wavelet subbands at different resolution levels. Its basic idea is to continuously approximate and separate details of the original signal to obtain characteristic representations of the signal at different scales.

[0090] The Discrete Fourier Transform (DFT) is a mathematical tool that converts discrete time series (usually finite-length series) into a frequency domain representation.

[0091] Frequency domain: The frequency domain is a mathematical representation used to analyze signals (including image signals). It describes the characteristics of a signal from the perspective of frequency. Through specific mathematical transformations (such as the Discrete Fourier Transform, DFT), the signal is transformed from the time domain (or, in the case of an image, the spatial domain) to the frequency domain, allowing the signal to be represented by its frequency components. In the frequency domain, the signal is decomposed into components of different frequencies, and these frequency components together constitute the characteristic description of the signal in the frequency domain.

[0092] In this application embodiment, the application scenario is the frequency domain of images. The frequency domain of images is a mathematical representation for analyzing and processing images. It describes the rate and periodicity of pixel value changes in an image from a frequency perspective. An image can be viewed as a two-dimensional signal. In the spatial domain (i.e., the time domain), an image consists of the grayscale or color values ​​of pixels, with each pixel corresponding to the brightness or color information at a specific location in the image. The frequency domain, however, transforms the image from the spatial domain to another representation through mathematical transformations (such as the Discrete Fourier Transform, DFT), where the frequency component becomes a key element in describing the image.

[0093] In the frequency domain, frequency represents how quickly grayscale or color values ​​change in an image. High-frequency components correspond to rapidly changing parts of the image, such as edges, details, and noise; low-frequency components correspond to slowly changing parts, such as large smooth areas and the background. Frequency domain representation reveals the components of the image signal at each frequency, thus showcasing the image's fundamental characteristics. By analyzing frequency domain information, we can gain a deeper understanding of the structure and composition of an image.

[0094] Spectrum: The spectrum is the specific representation of the frequency components of a signal in the frequency domain. It shows how the signal is distributed across different frequencies, including the amplitude (or intensity) and phase information of each frequency component. For image signals, each point or frequency range in the spectrum corresponds to a characteristic of a specific frequency component in the image. For example, in the spectrum of an image, the high-frequency components are typically associated with rapidly changing information such as edges, details, and textures, while the low-frequency components correspond to slowly changing information such as the background and smooth areas of the image.

[0095] The spectral convolution theorem in Fourier theory is a crucial theorem in Fourier analysis, revealing the correspondence between the time and frequency domains. It establishes the equivalence between convolution operations in the time (or spatial) domain and multiplication operations in the frequency domain, finding wide application in signal processing, image processing, and communications. The spectral convolution theorem plays a key role here. It demonstrates that changes in elements of the spectrum (frequency domain) have a global impact on all input features involved in the Fourier transform. Simply put, any change in the frequency domain does not merely affect the frequency components of a local region of the image, but rather has a holistic effect on the entire image (the image after inverse transformation from the frequency domain back to the spatial domain). This is because the frequency domain describes the frequency components of the image from a global perspective; these components are interconnected. Changing one frequency component may trigger a series of chain reactions in the spatial domain through inverse transformation, altering the overall appearance of the image. Therefore, learning in the frequency domain influences the global appearance.

[0096] Image resolution primarily refers to the number of pixels contained per unit length (or unit area) in an image. It measures the ability of an image to render details. It is usually expressed in "pixels per inch (ppi)" or "pixels per centimeter." Since the physical dimensions of an image (such as its length and width in inches) are fixed, the higher the resolution, the more pixels the image contains, resulting in a clearer image, richer details, and more delicate textures and color transitions.

[0097] Currently, image super-resolution can be achieved using neural network models, but these models are complex and slow. To improve the processing speed of image super-resolution, knowledge distillation methods have been proposed. However, in these methods, the teacher and student models are a combination of models with the same network structure. This ensures that they process information in the same way, resulting in similar vector dimensions in their output feature maps. Only then can knowledge distillation be performed based on the differences between the feature maps output by the teacher and student models.

[0098] However, with the rapid development of artificial intelligence, the speed of model updates and iterations is getting faster and faster. Often, teacher models and student models are combinations of models with different network structures (i.e., model structures). Due to the different network structures, the efficiency of knowledge distillation will decrease, resulting in poor image super-resolution processing capabilities of the student model after knowledge distillation.

[0099] To address the aforementioned issues, this application proposes a method for knowledge distillation that utilizes both global and local frequencies. This results in the image super-resolution output of the knowledge-distilled student model being more closely approximated by the image super-resolution output of the teacher model in terms of both local and global details. This allows for efficient knowledge distillation of image super-resolution by both the teacher and student models, even with different model structures. It enhances the subsequent image super-resolution processing capabilities of the knowledge-distilled student model (i.e., the resolution optimization model). Thus, by inputting the first image to be optimized into the resolution optimization model, a second image with high-quality image super-resolution can be obtained, improving the optimization effect of image processing. Please refer to the following specific embodiments for details.

[0100] Please see Figure 1 , Figure 1 This is a schematic diagram of a scene of an image processing system provided in an embodiment of this application. It includes a terminal 140, an Internet 130, a gateway 120, a server 110, etc.

[0101] Terminal 140 includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Furthermore, it can be a single device or a collection of multiple devices. Terminal 140 can communicate with the Internet 130 via wired or wireless means to exchange data.

[0102] Server 110 refers to a computer system that can provide certain services to terminal 140. Compared to ordinary terminal 140, server 110 has higher requirements in terms of stability, security, and performance. Server 110 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines).

[0103] Gateway 120, also known as an internetwork connector or protocol converter, is a computer system or device that acts as a translator, enabling network interconnection at the transport layer. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateways can also provide filtering and security functions. Messages sent from terminal 140 to server 110 are forwarded to the corresponding server 110 via gateway 120. Messages sent from server 110 to terminal 140 are also forwarded to the corresponding terminal 140 via gateway 120.

[0104] The image processing method of this disclosure can be implemented on server 110.

[0105] When the image processing method is implemented on server 110, a first image with a first resolution is acquired on server 110. This first image is then input into a resolution optimization model to obtain a second image with a second resolution greater than the first resolution. The resolution optimization model is trained by jointly training a student model and a teacher model using the first sample image based on a joint loss function. The first sample image has an optimized image label, which serves as the training target image for the first sample image. The joint loss function includes a first loss, a second loss, and a third loss. Specifically, the first loss corresponds to the discrete wavelet subband difference between the first and second optimized images; the first optimized image is obtained by the student model performing resolution optimization on the first sample image; the second optimized image is obtained by the teacher model performing resolution optimization on the first sample image; the second loss corresponds to the frequency domain difference between the first and second optimized images; and the third loss corresponds to the pixel difference between the optimized image label and the first optimized image. It should be noted that the above-described resolution optimization model training method can also be implemented on server 110, as detailed in the following specific embodiments.

[0106] The embodiments disclosed herein can be applied in various scenarios, such as Figure 2 The image optimization application scenarios are shown.

[0107] Image optimization application scenarios:

[0108] Image optimization applications can convert low-resolution images into high-resolution images. That is, given any low-resolution image as input to the resolution optimization model, a high-resolution image can be output, which can help downstream tasks such as autonomous driving or object recognition.

[0109] like Figure 2 As shown, during the processing of a low-resolution image (the first image), the user can use an image optimization system. The image optimization system deployed on the computer can acquire the first image output by the user, perform image super-resolution processing on the first image according to the resolution optimization model, and obtain a high-resolution image (i.e., the second image), thereby quickly optimizing the first image.

[0110] It should be noted that, Figure 1 The schematic diagram of the image processing system shown is merely an example. The image processing system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of image processing and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0111] In this embodiment, the description will be from the perspective of an image processing device, which can be integrated into a computer device that has a storage unit and a microprocessor and thus computing power. The computer device can be a server, and in this embodiment, the computer device will be described as a server.

[0112] Please see Figure 3 , Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of this application. The image processing method includes:

[0113] In step 201, the first image is acquired.

[0114] The first image has a first resolution, which is low resolution. This means the first image is a low-resolution image, indicating insufficient detail and poor texture and color transitions. For example, a common 480x480 pixel image (total pixel count of 230,400 pixels) is considered low-resolution. In some implementations, 230,400 pixels can be used to define a low-resolution image; images with a total pixel count less than or equal to 230,400 pixels are considered low-resolution images. Visually, low-resolution images appear blurry, with jagged edges or a blocky appearance. For instance, in a low-resolution portrait, facial details such as the texture of the eyes and lips may not be clearly displayed, appearing blurry. In a low-resolution landscape image, distant objects may be difficult to distinguish due to insufficient pixels; leaves may appear as color blocks instead of clearly defined leaf shapes.

[0115] Conversely, high-resolution images are images with a large number of pixels, capable of presenting rich details and high clarity. For example, images with 1920×1080 pixels (a total of 2,073,600 pixels) or higher can be considered high-resolution images. In some implementations, this total pixel count of 2,073,600 pixels can be used to define a high-resolution image; images with a total pixel count greater than or equal to 2,073,600 pixels are considered high-resolution images. Visually, high-resolution images are extremely clear, clearly displaying even the smallest details. For example, in a high-resolution landscape photograph, details such as the texture of distant mountains, the veins of leaves, and the ripples on the water's surface can be delicately rendered. In portrait photographs, details such as pores on the skin and individual strands of hair are accurately presented, giving a lifelike impression. Moreover, the edge transitions of high-resolution images are natural and smooth, without the jagged or blocky appearance seen in low-resolution images.

[0116] The first image can be an image captured by the vehicle's camera in an autonomous driving scenario, or an image captured by the terminal's camera in a face recognition scenario. It can be seen that, regardless of whether it is an autonomous driving scenario or a face recognition scenario, a low-resolution first image will cause the task to fail due to the lack of image details.

[0117] For example, in autonomous driving scenarios, road conditions need to be identified based on the acquired first image. If the first image is a low-resolution image, it will cause road condition identification errors, resulting in erroneous autonomous driving and posing a danger to users. Similarly, in facial recognition scenarios, faces need to be identified based on the acquired first image. If the first image is a low-resolution image, facial recognition will fail, causing inconvenience to users. Therefore, it is necessary to perform image super-resolution processing on the first image to improve its resolution and ensure the execution of downstream tasks.

[0118] In step 202, the first image is input into the resolution optimization model to obtain a second image with a second resolution.

[0119] Image super-resolution can be achieved using deep neural network models, such as Convolutional Neural Networks (CNN) models or Transformer models.

[0120] However, current image super-resolution models are complex and slow. Therefore, to improve processing speed and save computational resources, a knowledge distillation method has been proposed. This method uses a student model with a relatively simple structure for knowledge distillation, thereby reducing model complexity while maintaining a certain image super-resolution capability, thus improving processing speed and saving computational resources. However, this requires the teacher and student models to be a combination of models with the same structure, i.e., they must be from the same model family. For example, if the teacher model is a convolutional neural network, then the student model must also be a convolutional neural network. This ensures consistency in how information is processed, and both are equally adept at learning local information, thus enabling effective knowledge distillation. Conversely, if the teacher model is a convolutional neural network and the student model is a Transformer model, they are a combination of models with different structures. Their methods of processing information are inconsistent; convolutional neural networks excel at learning local information, while Transformer models excel at learning global information, making effective knowledge distillation difficult.

[0121] However, with the rapid development of artificial intelligence, the speed of model updates and iterations is getting faster and faster. Often, teacher models and student models are combinations of models with different model structures. Due to the different model structures, the efficiency of knowledge distillation will decrease, resulting in poor super-resolution processing capabilities of the student model after knowledge distillation.

[0122] To address this, this application proposes that knowledge distillation can be implemented from both global and local frequency dimensions. This makes the image super-resolution output of the knowledge-distilled student model more similar to that of the teacher model in terms of both local and global details. This allows for efficient knowledge distillation of image super-resolution by the teacher and student models with different model structures, enhancing the subsequent image super-resolution processing capability of the knowledge-distilled student model (i.e., the resolution optimization model). Thus, by inputting the first image to be optimized into the resolution optimization model, a second image with high super-resolution quality can be obtained. The second resolution of this second image is greater than the first resolution of the first image. For example, if the first image has a first resolution of 100 x 100 pixels and a total of 10,000 pixels, the second image has a second resolution of 2,000 x 3,000 pixels and a total of 6,000,000 pixels. Therefore, the optimization effect of image processing is significantly improved. For example, please refer to [further details omitted]. Figure 4 Front-end A, i.e., the terminal, can upload low-resolution images to back-end B (i.e., the server). Back-end B can quickly obtain high-resolution images with high super-resolution quality and return them to front-end A through the above embodiment, thus realizing fast image super-resolution processing.

[0123] The resolution optimization model is trained as a student model alongside a teacher model using a joint loss function on the first sample image. All three models—the resolution optimization model, the student model, and the teacher model—are deep neural network models. In knowledge distillation, the teacher model is a pre-trained, high-performance (e.g., high-accuracy) complex model. It is a deep neural network model, such as a convolutional neural network or a Transformer model. It has been trained on large-scale data and performs well on specific tasks. In this embodiment, the teacher model effectively performs image super-resolution processing, converting low-resolution images into high-resolution images. The student model is simpler than the teacher model. Its structure may be smaller, with fewer parameters. Due to its smaller structure and fewer parameters, the student model performs poorly in image super-resolution processing; therefore, its purpose is to learn knowledge from the teacher model. The student model can be a small deep neural network model, such as a convolutional neural network (Transformer model). Its task is to mimic the behavior of the teacher model to achieve knowledge distillation, improving inference efficiency while maintaining certain performance. The resolution optimization model is the student model obtained through joint training using a joint loss function.

[0124] In this embodiment, the teacher model and the student model are different models with different model structures (which can also be understood as different network architectures). That is, the teacher model and the student model are not from the same model family. Therefore, their information processing methods are inconsistent. For example, when the teacher model is a Transformer model, the student model can be a convolutional neural network model, or vice versa. Convolutional neural networks excel at learning local information, while Transformer models excel at learning global information. Because they excel at learning different types of information, it is difficult for them to perform effective knowledge distillation. In some implementations, it should be noted that the convolutional neural network is a model family. A convolutional neural network can include multiple models, such as SRCNN, EDSR, or RCAN, etc. No specific limitation is made here. SRCNN (Super Resolution + CNN) refers to using a CNN to directly learn the end-to-end mapping between low-resolution and high-resolution images. For example, please refer to [further details omitted]. Figure 5 As shown, Figure 5 This is a schematic diagram of the SRCNN model architecture provided in the embodiments of this application. Figure 5The SRCNN model structure shown includes a feature extraction layer C, a nonlinear mapping layer D, and a network reconstruction layer E. The feature extraction layer C extracts features from the input low-resolution image, obtaining a feature map corresponding to the low-resolution image. The nonlinear mapping layer D further performs nonlinear mapping on this feature map, mapping it to another high-dimensional feature map, thus obtaining the feature map corresponding to the high-resolution image. Finally, the network reconstruction layer E restores the high-resolution image from the feature map of the high-resolution image for output. This achieves image super-resolution processing of low-resolution images to obtain high-resolution images. RCAN (Residual Channel Attention Network) is a convolutional neural network model used for image super-resolution. EDSR (Enhanced Deep Super-Resolution Network) is an enhanced deep super-resolution network mainly used for image super-resolution reconstruction. It is a convolutional neural network that can convert low-resolution images into high-resolution images, recovering more image details.

[0125] In some implementations, both the teacher and student models can be convolutional neural networks (CNNs), but with different model structures. For example, the teacher model might be a RCAN CNN, while the student model is an EDSR CNN. Although both are CNNs, their specific network designs differ, resulting in structural differences. Therefore, they can still be considered different models. That is, even within the same model family, the teacher and student models may have different structural designs. Due to these structural differences, their learning outcomes differ, and they cannot efficiently distill knowledge, thus they are also considered different models. It should be noted that if both the teacher and student models are EDSRs, then their specific network designs are identical, and their model structures are the same, making them identical models. The above combinations are illustrative; in real-world scenarios, there are many other combinations of different model structures for the teacher and student models. These examples are not intended to limit the scope.

[0126] To better illustrate the embodiments of this application, the training process of the resolution optimization model will be described first; please refer to the following documents. Figure 6 , Figure 6 Another schematic diagram of the image processing method provided in this application embodiment. The training steps of the resolution optimization model are as follows:

[0127] In step 301, the first sample image and the corresponding optimized image label are obtained.

[0128] The first sample image is the training sample image, representing a sample set consisting of N sample images, such as 10,000 images. This first sample image is a low-resolution image, meaning it lacks rich detail and has poor texture and color transitions. It can be a sample image set by the developers or downloaded from the internet. For example, it could include the Set5 dataset, which is relatively small, typically containing 5 images. The images cover different scenes, including people and natural landscapes. Alternatively, it could include the Urban100 dataset, which generally contains 100 images, focusing on urban landscapes, including skyscrapers, streets, and city night scenes. These images typically have complex details and textures, high resolution, and rich structural information.

[0129] In supervised learning, image labels serve as the target output for training the model. These labels are the desired images used to guide model training, also known as training target images (i.e., the optimized image labels in this embodiment). The model learns the correlation between image features and the features of the optimized image labels. For example, when training an image classification model, the input is image data, and the output is the predicted image. By using a large number of images with correct labels, the model can adjust its parameters to make the predicted image match the true image label as closely as possible. The optimized image label is a pre-set high-resolution image, the "image label" corresponding to the first sample image, i.e., the high-resolution image obtained after image super-resolution processing of the expected low-resolution image, which is subsequently used to train the student model.

[0130] In step 302, the first sample image is input into the student model for resolution optimization to obtain the first optimized image, and the first sample image is input into the teacher model for resolution optimization to obtain the second optimized image.

[0131] In this embodiment, both the student model and the teacher model are deep neural network models that can be used to achieve image super-resolution tasks, such as convolutional neural network models or Transformer models. The teacher model and the student model are different models, meaning they have different model structures. To better illustrate this embodiment, we will use a Transformer model as the teacher model and a convolutional neural network model as the student model as an example. The first sample image is input into the student model for resolution optimization, i.e., image super-resolution processing, to obtain a first optimized image. Correspondingly, the first sample image is input into the teacher model for resolution optimization, i.e., image super-resolution processing, to obtain a second optimized image. The first optimized image and the second optimized image can have the same resolution or different resolutions after resolution processing, but their resolutions can both be higher than the original first sample image. In this embodiment, the resolution of the optimized image label is greater than that of the second optimized image, and the resolution of the second optimized image is greater than that of the first optimized image.

[0132] It should be noted that, due to the need for knowledge distillation, the teacher model is a pre-trained Transformer model, and a complex, high-performance large-scale model. The model parameters of the teacher model are fixed, i.e., frozen. Therefore, when the teacher model performs image super-resolution processing on the first sample image, it can obtain a second optimized image with higher image super-resolution processing quality. On the other hand, since the student model is an untrained convolutional neural network model, i.e., the model parameters of the student model are learnable, and the model structure is simpler than that of the teacher model, when the student model performs image super-resolution processing on the first sample image, it obtains a first optimized image with lower image super-resolution processing quality.

[0133] Based on this, the second optimized image is close to the label of the optimized image and is an image with high quality of super-resolution processing. The first optimized image is the image predicted by the student model during the training phase. Therefore, the image display effect of the second optimized image is higher than that of the first optimized image.

[0134] In step 303, a discrete wavelet transform is performed on the first optimized image to obtain a first discrete wavelet sub-band, and a discrete wavelet transform is performed on the second optimized image to obtain a second discrete wavelet sub-band. A first loss is generated based on the difference between the second discrete wavelet sub-band and the first discrete wavelet sub-band.

[0135] In related technologies, it is necessary to limit the teacher model and student model to be a combination of models with the same network structure. When the teacher model is a Transformer model and the student model is a convolutional neural network model, since the network structures of the two are different, their information processing methods are the same, and the vector dimensions of the output predicted feature maps are not similar. Therefore, it is impossible to perform knowledge distillation based on the pixel differences between the feature maps output by the teacher model and the student model. That is, it is impossible to distill the pixel representation knowledge of the model map output by the teacher model into the student model for learning. Even if knowledge distillation is forced, the efficiency of knowledge distillation will be very poor.

[0136] Therefore, this application proposes to assist in knowledge distillation of models with different architectures by using global and local frequency dimensions. Research has shown that frequency describes the rate and periodicity of pixel value changes in an image. In the frequency domain, frequency represents the speed of change of grayscale or color values ​​in an image. High-frequency components correspond to rapidly changing parts of the image, such as edges, details, and noise; low-frequency components correspond to slowly changing parts, such as large smooth areas and the background. For image super-resolution tasks, the basic goal is to reconstruct edges and textures containing high frequencies, while the low-frequency background corresponds to different frequency components. Therefore, even if the teacher and student models have different architectures, this remains a common objective. Based on this principle, it is reasonable to design global and local frequency dimensions to help the outputs of the student and teacher models approximate each other in terms of local and global frequencies. That is, allowing the student model to learn the output of the teacher model in the dimensions of global and local frequencies enables knowledge distillation of heterogeneous models.

[0137] Based on this, the Discrete Fourier Transform (DFT) can convert discrete time series (i.e., images) into a frequency domain representation, and the frequency domain allows signals to be represented using frequency components. In the frequency domain, an image is decomposed into components of different frequencies, which together constitute the feature description of the image in the frequency domain. Combining this with the spectral convolution theorem in Fourier theory, any change made in the frequency domain will not only affect the frequency components corresponding to a local region of the image, but will have a holistic impact on the entire image (the image after inverse transformation from the frequency domain back to the spatial domain). That is, the frequency domain after the DFT can achieve a global frequency representation. Therefore, the DFT can achieve a global frequency representation, and subsequent global frequency distillation learning can be achieved based on the DFT.

[0138] Correspondingly, since discrete wavelet transform can decompose a signal into sub-bands of different frequencies, i.e., different discrete wavelet sub-bands, different frequency sub-bands reflect the performance of the image in different local features in the frequency domain. For example, the low-frequency sub-band can represent the image at low frequencies, i.e., the general outline, overall shape, and relatively smooth areas, while the high-frequency sub-band can represent the image at high frequencies, such as edges, details, and noise. That is, different local frequency representations can be achieved through discrete wavelet transform. Therefore, in order to achieve local frequency distillation learning, this embodiment of the application can perform discrete wavelet transform on the first optimized image to obtain the first discrete wavelet sub-band, and perform discrete wavelet transform on the second optimized image to obtain the second discrete wavelet sub-band. The discrete wavelet transform is implemented through a filter bank. The first discrete wavelet sub-band characterizes the features of the first optimized image at different local frequencies (which can also be understood as different frequency sub-bands), such as the first low-frequency sub-band and the first high-frequency sub-band. The first low-frequency sub-band refers to the part containing the lower frequency components in the first optimized image, and the first high-frequency sub-band refers to the part containing the higher frequency components in the first optimized image. The second discrete wavelet sub-band characterizes the features of the second optimized image at different local frequencies, such as a second low-frequency sub-band and a second high-frequency sub-band. The second low-frequency sub-band refers to the portion containing lower frequency components of the second optimized image, and the second high-frequency sub-band refers to the portion containing higher frequency components of the second optimized image. In this way, different local frequency representations of the first optimized image and the second optimized image can be obtained.

[0139] Furthermore, to make the local frequencies (i.e., features at different local frequencies) of the first optimized image output by the student model close to those of the second optimized image output by the teacher model after the discrete wavelet transform, a first loss can be generated based on the difference between the second and first discrete wavelet subbands. The larger the value of this first loss, the less close the local frequencies of the first optimized image output by the student model are to those of the second optimized image output by the teacher model after the discrete wavelet transform. Conversely, the smaller the value of this first loss, the closer the local frequencies of the first optimized image output by the student model are to those of the second optimized image output by the teacher model after the discrete wavelet transform. These local frequencies can reflect the local details of the image. In some implementations, this first loss can be calculated using the following formula:

[0140]

[0141] Among them, the This is the first loss, where DWT() is the discrete wavelet transform. For the first optimized image, this This is the second optimized image.

[0142] In some implementations, performing a discrete wavelet transform on the first optimized image to obtain a first discrete wavelet sub-band, and performing a discrete wavelet transform on the second optimized image to obtain a second discrete wavelet sub-band, includes:

[0143] (1) Perform discrete wavelet transform on the first optimized image using a discrete wavelet transform template to obtain a first discrete wavelet sub-band. The first discrete wavelet sub-band includes at least a first low-frequency discrete wavelet sub-band, a first high-frequency discrete wavelet sub-band, a second high-frequency discrete wavelet sub-band, and a third high-frequency discrete wavelet sub-band.

[0144] (2) Perform discrete wavelet transform on the second optimized image using the discrete wavelet transform template to obtain the second discrete wavelet sub-band. The second discrete wavelet sub-band includes at least the second low-frequency discrete wavelet sub-band, the fourth high-frequency discrete wavelet sub-band, the fifth high-frequency discrete wavelet sub-band, and the sixth high-frequency discrete wavelet sub-band.

[0145] The discrete wavelet transform template refers to a series of filters used to implement the discrete wavelet transform. By performing the discrete wavelet transform on the first optimized image using the discrete wavelet transform template, the first discrete wavelet subband can be obtained.

[0146] In some implementations, discrete wavelet transform is performed on the two-dimensional image data. By simultaneously performing filtering operations in both the horizontal and vertical directions, the two-dimensional image can be decomposed into four sub-bands with different frequency components. These are a low-frequency sub-band (LL) and three high-frequency sub-bands, namely the LH (horizontal high frequency - vertical low frequency) sub-band, the HL (horizontal low frequency - vertical high frequency) sub-band, and the HH (horizontal high frequency - vertical high frequency) sub-band.

[0147] The LL subband represents the basic structural information of the signal (image) at a coarser granular level. For an image, it contains information such as the general outline, overall shape, and relatively smooth areas, similar to the result of a low-pass filter, which removes high-frequency detail fluctuations while preserving the main framework structure of the image. For example, in a landscape image, the low-frequency subband will reveal basic features such as the shapes of large objects like mountains and lakes, as well as the brightness and color distribution of large areas like the sky and the ground.

[0148] The LH subband primarily captures detailed changes in the horizontal direction of an image, such as horizontal edges and textures. For example, in an image containing buildings, the horizontal edges of the buildings (such as the boundary between the wall and the ground, the horizontal frame of the window, etc.) and the horizontal textures (such as the horizontal arrangement of bricks, horizontal decorative lines, etc.) will be strongly represented in the LH subband.

[0149] The HL sub-band focuses on the vertical details of an image, such as vertical edges and textures. For example, in the same building image, the vertical edges of the building (such as corners of walls and the edges of columns) and vertical textures (such as vertical folds in curtains and vertical architectural decorations) will be highlighted in the HL sub-band.

[0150] The HH subband primarily reflects the details of an image along its diagonal direction, including diagonal edges, textures, and local areas of significant change. For example, sloping roof edges, intersecting texture patterns (such as the patterns on certain fabrics or the diagonal portions of checkerboard patterns), and sudden local changes (such as small bright or dark spots in the image) are more noticeable in the HH subband.

[0151] Therefore, the first discrete wavelet subband may include a first low-frequency discrete wavelet subband, a first high-frequency discrete wavelet subband, a second high-frequency discrete wavelet subband, and a third high-frequency discrete wavelet subband. The first low-frequency discrete wavelet subband is the LL subband, the first high-frequency discrete wavelet subband is the LH subband, the second high-frequency discrete wavelet subband is the HL subband, and the third high-frequency discrete wavelet subband is the HH subband. These subbands can reflect the local features of the first optimized image from different local frequencies.

[0152] Correspondingly, the second optimized image can also be subjected to discrete wavelet transform using the discrete wavelet transform template to obtain a second discrete wavelet sub-band. The second discrete wavelet sub-band can include a second low-frequency discrete wavelet sub-band, a fourth high-frequency discrete wavelet sub-band, a fifth high-frequency discrete wavelet sub-band, and a sixth high-frequency discrete wavelet sub-band. The second low-frequency discrete wavelet sub-band is the LL sub-band, the fourth high-frequency discrete wavelet sub-band is the LH sub-band, the fifth high-frequency discrete wavelet sub-band is the HL sub-band, and the sixth high-frequency discrete wavelet sub-band is the HH sub-band. The local features of the second optimized image can be reflected from different local frequencies.

[0153] It is obvious that the frequency ranges represented by the first, second, and third high-frequency discrete wavelet subbands are different, and the frequency ranges represented by the fourth, fifth, and sixth high-frequency discrete wavelet subbands are also different.

[0154] In some implementations, based on the aforementioned discrete wavelet transform method, local frequency representations of the first and second optimized images can be achieved, thus enabling the construction of a corresponding first loss. Specifically, the first loss is generated based on the difference between the second and first discrete wavelet sub-bands, including:

[0155] (1.1) Based on the difference between the first low-frequency discrete wavelet subband and the second low-frequency discrete wavelet subband, the first sub-loss is obtained;

[0156] (1.2) Based on the difference between the first high-frequency discrete wavelet subband and the fourth high-frequency discrete wavelet subband, the second sub-loss is obtained;

[0157] (1.3) Based on the difference between the second high-frequency discrete wavelet subband and the fifth high-frequency discrete wavelet subband, the third sub-loss is obtained;

[0158] (1.4) Based on the difference between the third high-frequency discrete wavelet subband and the sixth high-frequency discrete wavelet subband, the fourth sub-loss is obtained;

[0159] (1.5) Combining the first sub-loss, the second sub-loss, the third sub-loss and the fourth sub-loss, we obtain the first loss.

[0160] In this embodiment of the application, it is expected that the first optimized image is close to the second optimized image from the sub-bands of four different frequency components. Therefore, the difference between the second low-frequency discrete wavelet sub-band and the first low-frequency discrete wavelet sub-band can be calculated to obtain the first sub-loss corresponding to the local frequency of the low-frequency sub-band.

[0161] The difference between the first and fourth high-frequency discrete wavelet subbands is calculated to obtain the second sub-loss corresponding to the subband representing the local frequency LH. The difference between the second and fifth high-frequency discrete wavelet subbands is calculated to obtain the third sub-loss corresponding to the subband representing the local frequency HL. The difference between the third and sixth high-frequency discrete wavelet subbands is calculated to obtain the fourth sub-loss corresponding to the subband representing the local frequency HH.

[0162] Furthermore, the first sub-loss, second sub-loss, third sub-loss, and fourth sub-loss can be summed to obtain the first loss. In some implementations, the first loss can be calculated using the following loss function:

[0163]

[0164] [LL,LH,HL,HH]=DWT(x) Formula (2)

[0165] Among them, the This is the first loss, where DWT() is the discrete wavelet transform. For the first optimized image, this For the second optimized image, the first loss can be calculated by implementing the above steps through formulas (1) and (2).

[0166] For a better illustration of the embodiments of this application, please refer to the following: Figure 7 As shown, Figure 7This is a schematic diagram of a scenario for the resolution optimization model training method provided in this application embodiment. A first sample image containing a face is input into student model 1 for image super-resolution processing to obtain a first optimized image. Correspondingly, the first sample image is input into teacher model 2 for image super-resolution processing to obtain a second optimized image. Further, by performing discrete wavelet transform on the first optimized image, discrete wavelet sub-bands corresponding to four different frequency components of the first optimized image are obtained. Discrete wavelet transform is also performed on the second optimized image to obtain discrete wavelet sub-bands corresponding to four different frequency components of the second optimized image. Thus, a first loss is calculated based on the difference between the discrete wavelet sub-bands corresponding to the four different frequency components of the two images.

[0167] In some implementations, since the discrete wavelet sub-bands of the four different frequency components focus on different local image features, the target can also assign corresponding weights to the four sub-losses according to the characteristics of the image, so as to flexibly focus on learning the local image features that need attention. Specifically, the first loss obtained by combining the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss may include:

[0168] (2.1) Multiply the first sub-loss by the first weight to obtain the first target sub-loss;

[0169] (2.2) Multiply the second sub-loss by the second weight to obtain the second target sub-loss;

[0170] (2.3) Multiply the third sub-loss and the third weight to obtain the third objective sub-loss;

[0171] (2.4) Multiply the fourth sub-loss by the fourth weight to obtain the fourth objective sub-loss;

[0172] (2.5) Combining the first target sub-loss, the second target sub-loss, the third target sub-loss and the fourth target sub-loss, we obtain the first loss.

[0173] Among them, the four discrete wavelet subbands with different frequency components focus on different local image features. The LL subband focuses on the general outline, overall shape, and relatively smooth areas of the image; the LH subband focuses on detailed features in the horizontal direction, such as horizontal edges and textures; the HL subband focuses on detailed features in the vertical direction, such as vertical edges and textures; and the HH subband focuses on detailed changes in the image in the diagonal direction, including diagonal edges, textures, and locally changing areas in the image.

[0174] Therefore, the object can also set corresponding weights for the four sub-losses based on the characteristics of the first sample image to be processed, and the sum of the first weight, second weight, third weight, and fourth weight is one. For example, for an image whose main content is the sky, the first weight value of the first sub-loss corresponding to the LL subband can be set to be greater than the other weights, for example, the first weight value is set to 0.4, the second weight is set to 0.2, the third weight is set to 0.2, and the fourth weight is set to 0.2. In this way, in subsequent training, the focus will be on the general outline, overall shape, and relatively smooth areas of the image reflected in the LL subband, so as to achieve more accurate local feature learning. Based on this, the first sub-loss and the first weight can be multiplied to obtain the first target sub-loss; the second sub-loss and the second weight can be multiplied to obtain the second target sub-loss; the third sub-loss and the third weight can be multiplied to obtain the third target sub-loss; the fourth sub-loss and the fourth weight can be multiplied to obtain the fourth target sub-loss; finally, the first target sub-loss, the second target sub-loss, the third target loss, and the fourth target sub-loss are added together to obtain the first loss.

[0175] In some implementations, the first, second, third, and fourth weights are manually set and need to be adjusted according to the scenario. Therefore, the setting relies heavily on expert experience. To solve this problem, this application proposes a scheme for automatically setting the first, second, third, and fourth weights. Specifically, the first, second, third, and fourth weights can be determined in the following way:

[0176] (3.1) Determine the image type information of the first sample image;

[0177] (3.2) Based on the image type information, determine the values ​​of the first weight, the second weight, the third weight and the fourth weight respectively.

[0178] Images can be categorized into different image types based on their content, such as landscape images, architectural images, and facial images. Understanding this, different image types have different local features. Therefore, based on the image type information, we can determine the local features that require more attention. Furthermore, based on the image type information, we can determine the corresponding first, second, third, and fourth weights. For example, when the image type information indicates landscape, the first weight value of the first sub-loss corresponding to the LL subband can be automatically set higher than other weights. For instance, the first weight could be set to 0.4, the second to 0.2, the third to 0.2, and the fourth to 0.2. This ensures that subsequent training focuses on the general outline, overall shape, and relatively smooth areas of the image reflected in the LL subband, achieving more accurate local feature learning. When the image type information indicates building type information, the weights of the second sub-loss corresponding to the LH subband and the third sub-loss corresponding to the HL subband can be automatically set to be greater than other weights. For example, the first weight value can be set to 0.2, the second weight to 0.3, the third weight to 0.3, and the fourth weight to 0.2. This ensures that subsequent training focuses on the horizontal detailed features reflected in the LH and HL subbands, such as horizontal edges and textures, and the vertical detailed features, such as vertical edges and textures. For images with building type information, more accurate local feature learning can be achieved. In some implementations, a mapping relationship between different image type information and the corresponding values ​​of the first, second, third, and fourth weights can be established in advance. When matching is needed later, the mapping relationship can be looked up to quickly determine the values ​​of the first, second, third, and fourth weights corresponding to the image type information, saving manual costs.

[0179] In step 304, a discrete Fourier transform is performed on the first optimized image to obtain a first frequency domain, and a discrete Fourier transform is performed on the second optimized image to obtain a second frequency domain. A second loss is generated based on the difference between the second frequency domain and the first frequency domain.

[0180] In the frequency domain, the image is decomposed into components of different frequencies. These frequency components together constitute the feature description of the image in the frequency domain. That is, the frequency domain after Discrete Fourier Transform (DFT) can realize the global frequency representation. Therefore, DFT can achieve global frequency representation. Based on this, in order to achieve global frequency distillation learning, this embodiment of the application can perform DFT on the first optimized image to obtain the first frequency domain, and perform DFT on the second optimized image to obtain the second frequency domain. This DFT can be implemented using the DFT formula, as follows:

[0181]

[0182] Among them, the Representing complex frequencies, it indicates the amplitude of the signal at different frequencies, reflecting the relative intensity of the signal at each frequency component. H and W are the number of pixels in the image in the horizontal and vertical directions, respectively. u and v are the frequency components in the frequency domain in the horizontal and vertical directions, respectively. h and w represent the pixel coordinates of the image in the horizontal and vertical directions, respectively. Using the above formula, a first frequency domain can be obtained by performing a discrete Fourier transform on the first optimized image, and a second frequency domain can be obtained by performing a discrete Fourier transform on the second optimized image. The first frequency domain characterizes the features of the first optimized image at the global frequency, and the second frequency domain characterizes the features of the second optimized image at the global frequency.

[0183] Furthermore, to ensure that the global frequencies of the first optimized image output by the student model after the Discrete Fourier Transform (DFT) are close to those of the second optimized image output by the teacher model after the DFT, a second loss can be generated based on the difference between the second and first frequency domains. A larger value of this second loss indicates a less close global frequency between the first optimized image output by the student model and the second optimized image output by the teacher model; conversely, a smaller value of this second loss indicates a closer global frequency between the first optimized image output by the student model and the second optimized image output by the teacher model. In some implementations, this second loss can be calculated using the following loss function:

[0184]

[0185] Among them, the This is the second loss, where DFT() is the Discrete Fourier Transform. For the first optimized image, this This is the second optimized image.

[0186] For a better illustration of the embodiments of this application, please refer to [link / reference needed]. Figure 7 As shown, a discrete Fourier transform can be performed on the first optimized image to obtain a first frequency domain, and a discrete Fourier transform can be performed on the second optimized image to obtain a second frequency domain. A second loss is generated based on the difference between the second frequency domain and the first frequency domain.

[0187] In some implementations, the first optimized image is subjected to a Discrete Fourier Transform to obtain a first frequency domain, and the second optimized image is subjected to a Discrete Fourier Transform to obtain a second frequency domain, and a second loss is generated based on the difference between the second frequency domain and the first frequency domain, including:

[0188] (1) Perform a discrete Fourier transform on the first optimized image using a discrete Fourier transform template to obtain the first frequency domain;

[0189] (2) Perform a discrete Fourier transform on the second optimized image using a discrete Fourier transform template to obtain the second frequency domain;

[0190] (3) The second loss is obtained based on the difference between the second frequency domain and the first frequency domain.

[0191] The Discrete Fourier Transform template is the Discrete Fourier Transform formula mentioned above. The first optimized image is subjected to Discrete Fourier Transform using the Discrete Fourier Transform formula to obtain the first frequency domain. The second optimized image is subjected to Discrete Fourier Transform using the Discrete Fourier Transform formula to obtain the second frequency domain.

[0192] Furthermore, according to the second frequency domain and the first frequency domain The absolute value of the difference is calculated to obtain the second loss.

[0193] In step 305, a third loss is generated based on the difference between the first pixel of the optimized image label and the second pixel of the first optimized image.

[0194] Since the optimized image label is the label of the first optimized image output by the student model, a third loss needs to be established based on the difference in pixel representation between the two. Specifically, the first pixel represents each pixel on the optimized image label, and the second pixel represents each pixel on the first optimized image. In this way, the difference in pixel value between the first pixel and the second pixel can be calculated to determine the difference between the first pixel of the optimized image label and the second pixel of the first optimized image to generate the third loss. The larger the value of the third loss, the greater the difference in pixel representation between the first pixel of the optimized image label and the second pixel of the first optimized image; the smaller the value of the third loss, the smaller the difference in pixel representation between the first pixel distribution of the optimized image label and the second pixel of the first optimized image.

[0195] Please continue reading. Figure 7 As shown, a third loss can be generated based on the difference between the first pixel of the optimized image label and the second pixel of the first optimized image.

[0196] In some implementations, generating a third loss based on the difference between a first pixel of the optimized image label and a second pixel of the first optimized image includes:

[0197] (1) Obtain the first pixel value of the first pixel corresponding to the optimized image label;

[0198] (2) Obtain the second pixel value of the second pixel corresponding to the first optimized image;

[0199] (3) Generate a third loss based on the difference between the first pixel value and the second pixel value.

[0200] In this embodiment, the first pixel corresponding to the optimized image label can be obtained. This first pixel is each pixel on the optimized image label, and its value is the pixel value of each pixel on the optimized image label. Correspondingly, the second pixel corresponding to the first optimized image is obtained. This second pixel is each pixel on the first optimized image, and its value is the pixel value of each pixel on the first optimized image. The number of pixels on the optimized image label is the same as the number of pixels in the first optimized image. For example, if the first optimized image has 1024 x 1024 pixels, the number of pixels on the optimized image label is also 1024 x 1024 pixels. It should be noted that the third loss can be calculated using the minimum absolute value deviation calculation method or a loss calculation method such as mean square error.

[0201] In some implementations, taking the minimum absolute value deviation calculation method as an example, the purpose of the minimum absolute value deviation calculation method is to minimize the sum of the absolute differences between the target value (i.e., the first pixel value of the first pixel of the optimized image label) and the predicted value (the second pixel value of the second pixel of the first optimized image). In this way, the absolute difference between the first pixel value and the second pixel value at each image pixel position can be calculated according to the image pixel position, and statistical summation can be performed to obtain the sum of absolute differences and generate the third loss.

[0202] In some implementations, taking the mean squared error calculation method as an example, the purpose of the mean squared error calculation method is to minimize the sum of squares of the differences between the predicted value (the second pixel value of the second pixel of the first optimized image) and the target value (i.e., the first pixel value of the first pixel of the optimized image label). In this way, the square of the difference between the pixel values ​​of the first pixel and the second pixel at each image pixel position can be calculated according to the image pixel position, and statistical summation can be performed to generate the third loss.

[0203] In step 306, the first loss, the second loss, and the third loss are input into the joint loss function for calculation to obtain the joint loss.

[0204] It should be noted that the loss function is a function that measures the difference between the model's prediction and the true value. In machine learning and deep learning tasks, the model's goal is to minimize this difference. It takes the model's output and the target (the true value, i.e., the image optimization label in this embodiment) as input, and outputs a non-negative real number representing the degree of prediction error. The joint loss function refers to a function that combines multiple loss functions. In complex machine learning and deep learning tasks, a single loss function may not be able to comprehensively measure the model's performance. It integrates the advantages of different types of loss functions, enabling optimization of the model from multiple perspectives. In this application, the joint loss function refers to the function that combines the first loss, the second loss, and the third loss.

[0205] The student model needs to consider multiple tasks simultaneously (i.e., three losses), each with its own corresponding loss measurement method. To comprehensively optimize these objectives, a joint loss function is constructed, combining the first, second, and third losses by summing them. This directly trains the student model's default training loss (i.e., the third loss), the discrete wavelet transform loss (achieving local frequency knowledge transfer, i.e., the first loss), and the discrete Fourier transform domain loss (achieving global frequency knowledge transfer, i.e., the second loss). In other words, the first, second, and third losses can be input into the joint loss function for calculation to obtain the joint loss.

[0206] In some implementations, the impact of these different loss terms on the overall model training can be balanced by using different weighting coefficients. Specifically, the first loss, second loss, and third loss are input into the joint loss function for calculation to obtain the joint loss, which includes:

[0207] (1) Obtain a first preset weight value, and weight the second loss using the first preset weight value to obtain the weighted second loss;

[0208] (2) Input the first loss, the weighted second loss and the third loss into the joint loss function to calculate the joint loss.

[0209] Since the frequency domain describes the frequency components of an image from a global perspective, and these components are interconnected, changing one frequency component can trigger a series of chain reactions in the spatial domain through inverse transformation, altering the overall appearance of the image. Therefore, the second loss has a significant impact on training. To balance the impact of the second loss on the overall model training, a first preset weight coefficient can be used to weaken its influence. This first preset weight coefficient is any value greater than 0 and less than 0.2, for example, 0.1. The second loss can then be weighted using this first preset weight value to obtain a weighted second loss. Because the first preset weight value is relatively small, the first loss, the weighted second loss, and the third loss are then input into a joint loss function and added together to obtain the joint loss. This balances the impact of the second loss on model training, making the model training process more stable. For example, the following formula can be used for understanding:

[0210]

[0211] Among them, the That is, the target loss. For the first optimized image, this For the second optimized image, this I HR To optimize image labels, this As the first loss, As the second loss, The third loss is represented by λ, which is the first preset weighting coefficient.

[0212] In some implementations, the third loss represents the difference in pixel values ​​between the first pixel of the optimized image label and the second pixel of the first optimized image. As a direct reference, it plays a significant role in guiding the student model to perform image super-resolution processing during training. Therefore, it is necessary to enhance the role of the third loss in training. Based on this, in some implementations, a second preset weight value can also be obtained. The third loss is then weighted using the second preset weight value to obtain the weighted third loss.

[0213] Furthermore, a second preset weight value can be obtained, which is greater than the first preset weight value. The second preset weight value is greater than 1 and less than 1.5, for example, it can be 1.2. The impact of the third loss on the overall model training can be enhanced by the second preset weight value.

[0214] In some implementations, the first loss, the weighted second loss, and the third loss are input into a joint loss function for calculation to obtain the joint loss, including: inputting the first loss, the weighted second loss, and the weighted third loss into a joint loss function for calculation to obtain the joint loss.

[0215] Specifically, the first loss, the weighted second loss, and the weighted third loss can be input into the joint loss function for calculation to obtain the joint loss. This joint loss weakens the impact of the second loss on training and strengthens the impact of the third loss on training. This makes the training process more stable and also enhances the resolution of the student model's training output optimized image, thereby improving the model training effect.

[0216] In step 307, the student model is trained using a joint loss to obtain a resolution-optimized model.

[0217] This can be achieved by iteratively training the student model using the joint loss function (i.e., iteratively training the student model using the joint loss calculated by the joint loss function). Alternatively, it can be understood as iteratively training both the student and teacher models using the joint loss function. Training ends when the number of training iterations reaches a preset threshold, such as 1000, yielding the resolution-optimized model; or training ends when the joint loss converges, also yielding the resolution-optimized model. Since the resolution-optimized model achieves knowledge distillation at local frequencies based on the first loss, knowledge distillation at global frequencies based on the second loss, and the student model learns and optimizes the pixel representation of image labels based on the third loss, during training, the results of image super-resolution output from the resolution-optimized model and the teacher model continuously approach each other in terms of local and global details. This achieves efficient knowledge distillation for image super-resolution using different model structures, enabling the resolution-optimized model to be jointly trained using the first sample image based on the joint loss function. Please continue reading. Figure 7 As shown, the first loss, second loss, and third loss can be input into the joint loss function for calculation to obtain the joint loss. The learnable student model can then be iteratively trained based on this joint loss to end the training and obtain the trained student model, i.e., the resolution optimization model.

[0218] Therefore, by inputting the first image to be optimized into the resolution optimization model, a second image with high super-resolution quality can be directly obtained, thus improving the optimization effect of image processing.

[0219] In some implementations, the student model is trained using a joint loss to obtain a resolution-optimized model. This includes iteratively training the student model using the joint loss until a training termination condition is met, thereby obtaining the resolution-optimized model.

[0220] The training termination condition is that the joint loss converges or the number of training iterations reaches a preset threshold. The preset threshold can be set according to actual needs, such as 1000 or 2000. That is, the student model can be continuously iterated and trained by the joint loss calculated based on the joint loss function until the target loss converges or the number of training iterations reaches the preset threshold. Then the training stops, the joint training is completed, and the finally trained student model is determined as the resolution optimization model.

[0221] As described above, although efficient knowledge distillation is difficult between teacher and student models with different model structures, this embodiment adds the learning of discrete wavelet subband differences and frequency domain differences to fully utilize the knowledge contained in the teacher model. Specifically, during training, this embodiment uses a joint loss function that includes a first loss, a second loss, and a third loss. The first loss is determined based on the discrete wavelet subband differences between the first and second optimized images. The first optimized image is obtained by the student model through resolution optimization of the first sample image, and the second optimized image is obtained by the teacher model through resolution optimization of the first sample image. The first loss enables knowledge distillation at local frequencies, allowing the student model to mimic the teacher model's representation at local frequencies.

[0222] The second loss is determined based on the frequency domain difference between the first and second optimized images. This second loss enables knowledge distillation at the global frequency level, allowing the student model to mimic the teacher model's representation at the global frequency level. The third loss is determined based on the pixel difference between the optimized image label and the first optimized image. This third loss enables the student model to learn the pixel value representation of the optimized image label. Therefore, compared to general knowledge distillation methods, this embodiment achieves knowledge distillation from both global and local frequency dimensions. This makes the image super-resolution output of the student model after knowledge distillation closer to the image super-resolution output of the teacher model in terms of both local and global details. It allows for efficient knowledge distillation of image super-resolution by the teacher and student models with different model structures, improving the ability of the student model (i.e., the resolution optimization model) to perform subsequent image super-resolution processing. Thus, inputting the first image to be optimized into the resolution optimization model yields a second image with high image super-resolution quality, improving the optimization effect of image processing.

[0223] Additionally, refer to Figure 8 , Figure 8 This is an optional flowchart of an image processing method provided in an embodiment of this application. The image processing method can be executed by a server and includes, but is not limited to, the following steps 401 to 409.

[0224] In step 401, is it training or testing?

[0225] In step 402, the low-resolution image and the pre-trained teacher model are loaded.

[0226] In step 403, the student model parameters are updated based on the training objective function and the Adam optimization algorithm.

[0227] In step 404, it is determined whether the set number of iterations has been reached.

[0228] In step 405, the model is saved.

[0229] In step 406, the low-resolution image to be tested.

[0230] In step 407, the pre-trained student model is loaded.

[0231] In step 408, a high-resolution image is obtained through forward computation.

[0232] In step 409, a high-resolution image is output.

[0233] This application embodiment includes two phases: training and testing. When the testing phase is detected in step 401, training begins. The training batch size (batch size refers to the number of samples used in each parameter update) and input patch are set to 16 and 48×48 pixels, respectively. Training is implemented using the PyTorch framework, loading a low-resolution image and a pre-trained teacher model. The training is then performed based on the training objective function (i.e., the joint loss function) and the Adam optimization algorithm. This Adam optimization algorithm is provided by the Adam optimizer; for example, the initial learning rate can be set to 1×10⁻⁶. -4 After every 105 iterations, the number of iterations is reduced by half, and the number of iterations is set to 106. The student model parameters are updated, and these parameters are the model parameters. Each time the student model parameters are updated, step 404 is used to determine whether the predetermined number of iterations has been reached. When the predetermined number of iterations has been reached, it means that the training has ended, the training has stopped, and the model has been saved as the resolution-optimized model. If the predetermined number of iterations has not been reached, the process returns to step 402 and continues iterative training.

[0234] Correspondingly, when the test phase is detected in step 402, the low-resolution image to be tested can be directly loaded into the pre-trained student model (i.e., the resolution optimization model). Through forward computation, i.e. model operation, a high-resolution image is output. Testers can refer to the quality of the high-resolution image to determine the effect of the resolution optimization model training, so as to replace the training samples in a timely manner.

[0235] The specific implementation of each of the above training steps can be found in the previous examples, and will not be repeated here.

[0236] For a better illustration of the embodiments of this application, please refer to the following: Figures 9 to 11 Please refer to the diagram for comprehension.

[0237] First, let's explain the parameters in the diagram. Scale represents the scaling factor, and SwinIR is an image restoration model based on the SwinTransformer architecture. The SwinTransformer has wide applications in computer vision and is characterized by its window-based self-attention mechanism. In image restoration tasks, SwinIR can handle problems such as super-resolution and denoising. It improves image quality by learning a mapping relationship from low-quality images to high-quality images.

[0238] RCAN is a convolutional neural network model used for image super-resolution.

[0239] EDSR, or Enhanced Deep Super-Resolution Network, is mainly used for image super-resolution reconstruction. It is a convolutional neural network that can convert low-resolution images into high-resolution images and recover more image details.

[0240] PSNR (Peak Signal-to-Noise Ratio) is an objective metric used to measure image quality. A higher PSNR value indicates a smaller difference between two images, meaning the processed image quality is higher.

[0241] SSIM (Structural Similarity Index) measures the similarity between two images based on three aspects: brightness, contrast, and structure. SSIM values ​​range from -1 to 1; the closer the value is to 1, the more structurally similar the two images are, indicating higher image quality.

[0242] I. Set5 dataset

[0243] Data size: Relatively small, typically containing 5 images.

[0244] Image content: It covers different scenes, including people, natural landscapes, etc., but the number of images is limited, so the coverage of specific scenes is not comprehensive enough.

[0245] Characteristics: Due to the small number of images, it is often used for quickly verifying the effectiveness of algorithms and for preliminary performance evaluation. In super-resolution tasks, the resolution of the images is relatively low, and the challenge lies in how to recover more detailed information from low-resolution images.

[0246] II. Set14 dataset

[0247] Data size: Slightly larger than Set5, containing 14 images.

[0248] Image content: It includes more types of scenes, such as buildings, animals, and landscapes, and the complexity of the images has also increased.

[0249] Features: It allows for better testing of the performance of super-resolution algorithms in different scenarios. Images contain richer details and textures, demanding a higher level of detail recovery capability from the algorithm.

[0250] III. B100 Dataset

[0251] Data size: Typically contains 100 images.

[0252] Image content: Covers a wide range of scenes and themes, including indoor and outdoor scenes, people, animals, etc., and has a high degree of diversity.

[0253] Features: Enables a more comprehensive evaluation of the performance of super-resolution algorithms under various conditions. The varying resolution and quality of images present further challenges to the algorithms. This dataset can be used to test the generalization ability and robustness of the algorithms.

[0254] IV. Urban100 Dataset

[0255] Data size: Generally contains 100 images, the same as the B100 size.

[0256] Image content: Primarily focuses on urban landscapes, including skyscrapers, streets, and city nightscapes. The images typically feature complex details and textures, high resolution, and rich structural information.

[0257] Features: Particularly suitable for testing the performance of super-resolution algorithms when processing urban landscape images. Because urban landscape images typically have complex structures and textures, the algorithm's ability to restore details and preserve structure is crucial.

[0258] KDHA (Knowledge Distillation of Heterogeneous Architectures) is an example of this application.

[0259] It can be seen that, regardless of Figure 9 Table 11 shows the CNN-Transformer. Figure 10 Table 12 shows Transformer-CNN, or... Figure 11 Table 13 shows that the student model obtained after knowledge distillation in the CNN-CNN embodiment of this application has the best peak signal-to-noise ratio and structural similarity index. In other words, we have achieved better knowledge distillation results regardless of whether the knowledge distillation is heterogeneous or homogeneous, so that the student model after knowledge distillation has the best image super-resolution processing capability.

[0260] To facilitate better implementation of the image processing method provided in the embodiments of this application, the embodiments of this application also provide an apparatus based on the above-described image processing method. The meanings of the terms used are the same as in the image processing method described above, and specific implementation details can be found in the descriptions in the method embodiments.

[0261] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The image processing apparatus is applied to a computer device and may include an acquisition unit 501 and a resolution optimization unit 502, etc.

[0262] The acquisition unit 501 is used to acquire a first image, which has a first resolution.

[0263] The resolution optimization unit 502 is used to input the first image into the resolution optimization model to obtain a second image with a second resolution greater than the first resolution. The resolution optimization model is obtained by jointly training the student model and the teacher model using the first sample image based on the joint loss function during training. The first sample image has an optimized image label, which is the training target image of the first sample image. The joint loss function is a function that includes the first loss, the second loss, and the third loss.

[0264] The first loss corresponds to the discrete wavelet subband difference between the first optimized image and the second optimized image. The first optimized image is obtained by the student model by optimizing the resolution of the first sample image, and the second optimized image is obtained by the teacher model by optimizing the resolution of the first sample image. The second loss corresponds to the frequency domain difference between the first optimized image and the second optimized image. The third loss corresponds to the pixel difference between the optimized image label and the first optimized image.

[0265] In some embodiments, the device further includes a training unit (unidentified), which includes:

[0266] Obtain a sub-unit (unidentified) to obtain the first sample image and the corresponding optimized image label of the first sample image;

[0267] The resolution optimization subunit (unlabeled) is used to input the first sample image into the student model for resolution optimization to obtain the first optimized image, and to input the first sample image into the teacher model for resolution optimization to obtain the second optimized image. The teacher model and the student model have different model structures.

[0268] The first loss generation subunit (unidentified) is used to perform discrete wavelet transform on the first optimized image to obtain a first discrete wavelet subband, and to perform discrete wavelet transform on the second optimized image to obtain a second discrete wavelet subband, and to generate a first loss based on the difference between the second discrete wavelet subband and the first discrete wavelet subband.

[0269] The second loss generation subunit (unidentified) is used to perform a discrete Fourier transform on the first optimized image to obtain a first frequency domain, and to perform a discrete Fourier transform on the second optimized image to obtain a second frequency domain, and to generate a second loss based on the difference between the second frequency domain and the first frequency domain.

[0270] The third loss generation subunit (unidentified) is used to generate a third loss based on the difference between the first pixel of the optimized image label and the second pixel of the first optimized image;

[0271] The calculation sub-unit (unidentified) is used to input the first loss, second loss, and third loss into the joint loss function for calculation to obtain the joint loss;

[0272] The training subunit (unlabeled) is used to train the student model using this joint loss to obtain a resolution-optimized model.

[0273] In some embodiments, the first loss generation subunit (unidentified) includes:

[0274] The first discrete wavelet transform module is used to perform discrete wavelet transform on the first optimized image through a discrete wavelet transform template to obtain a first discrete wavelet sub-band. The first discrete wavelet sub-band includes at least a first low-frequency discrete wavelet sub-band, a first high-frequency discrete wavelet sub-band, a second high-frequency discrete wavelet sub-band, and a third high-frequency discrete wavelet sub-band.

[0275] The second discrete wavelet transform module is used to perform discrete wavelet transform on the second optimized image through the discrete wavelet transform template to obtain a second discrete wavelet sub-band. The second discrete wavelet sub-band includes at least a second low-frequency discrete wavelet sub-band, a fourth high-frequency discrete wavelet sub-band, a fifth high-frequency discrete wavelet sub-band, and a sixth high-frequency discrete wavelet sub-band.

[0276] Among them, the frequency ranges represented by the first high-frequency discrete wavelet subband, the second high-frequency discrete wavelet subband, and the third high-frequency discrete wavelet subband are different, and the frequency ranges represented by the fourth high-frequency discrete wavelet subband, the fifth high-frequency discrete wavelet subband, and the sixth high-frequency discrete wavelet subband are different.

[0277] The first loss generation module (unidentified) is used to generate a first loss based on the difference between the second discrete wavelet subband and the first discrete wavelet subband.

[0278] In some embodiments, the first loss generation module (unidentified) includes:

[0279] The first sub-loss generation submodule (unidentified) is used to obtain the first sub-loss based on the difference between the first low-frequency discrete wavelet subband and the second low-frequency discrete wavelet subband.

[0280] The second sub-loss generation submodule (unidentified) is used to obtain the second sub-loss based on the difference between the first high-frequency discrete wavelet subband and the fourth high-frequency discrete wavelet subband.

[0281] The third sub-loss generation submodule (unidentified) is used to obtain the third sub-loss based on the difference between the second high-frequency discrete wavelet subband and the fifth high-frequency discrete wavelet subband.

[0282] The fourth sub-loss generation submodule (unidentified) is used to obtain the fourth sub-loss based on the difference between the third high-frequency discrete wavelet subband and the sixth high-frequency discrete wavelet subband;

[0283] The combined submodule is used to combine the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss to obtain the first loss.

[0284] In some embodiments, the combining submodule (unidentified) is used for:

[0285] Multiply the first sub-loss by the first weight to obtain the first target sub-loss;

[0286] Multiply the second sub-loss by the second weight to obtain the second target sub-loss;

[0287] Multiply the third sub-loss by the third weight to obtain the third target sub-loss;

[0288] Multiply the fourth sub-loss by the fourth weight to obtain the fourth objective sub-loss;

[0289] By combining the first target sub-loss, the second target sub-loss, the third target sub-loss, and the fourth target sub-loss, the first loss is obtained;

[0290] The sum of the first target sub-loss, the second target sub-loss, the third target sub-loss, and the fourth target sub-loss is one.

[0291] In some embodiments, the first loss generation module (unidentified) further includes a weight determination submodule (unidentified), which is used to:

[0292] Determine the image type information of the first sample image;

[0293] Based on the image type information, determine the values ​​of the first weight, second weight, third weight, and fourth weight respectively.

[0294] In some embodiments, the second loss generation subunit (unidentified) is used for:

[0295] The first optimized image is subjected to a discrete Fourier transform using a discrete Fourier transform template to obtain the first frequency domain.

[0296] The second optimized image is subjected to a Discrete Fourier Transform using a Discrete Fourier Transform template to obtain the second frequency domain.

[0297] The second loss is obtained based on the difference between the second frequency domain and the first frequency domain.

[0298] In some embodiments, the third loss generation subunit (unidentified) is used for:

[0299] Obtain the first pixel value of the first pixel corresponding to the label of the optimized image;

[0300] Obtain the second pixel value of the second pixel corresponding to the first optimized image;

[0301] A third loss is generated based on the difference between the first pixel value and the second pixel value.

[0302] In some embodiments, the computing subunit (unidentified) includes:

[0303] The first weighting module obtains a first preset weight value and weights the second loss using the first preset weight value to obtain the weighted second loss;

[0304] The calculation module is used to input the first loss, the weighted second loss, and the third loss into the joint loss function for calculation to obtain the joint loss.

[0305] In some embodiments, the calculation subunit (unidentified) further includes a second weighting module for:

[0306] Obtain a second preset weight value, which is greater than the first preset weight value; then weight the third loss using the second preset weight value to obtain the weighted third loss.

[0307] This calculation module is also used to input the first loss, the weighted second loss, and the weighted third loss into the joint loss function for calculation to obtain the joint loss.

[0308] In some embodiments, the training subunit (unidentified) is used for:

[0309] The student model is iteratively trained using this joint loss until the training termination condition is met, resulting in a resolution-optimized model.

[0310] The training termination condition includes at least the convergence of the joint loss or the number of training iterations reaching a preset threshold.

[0311] The specific implementation of each of the above units can be found in the previous embodiments, and will not be repeated here.

[0312] Reference Figure 13 , Figure 13 To implement the structural block diagram of a portion of the terminal 140 according to an embodiment of this disclosure, the terminal 140 includes: a radio frequency (RF) circuit 610, a memory 615, an input unit 630, a display unit 640, a sensor 660, an audio circuit 660, a wireless fidelity (WiFi) module 670, a processor 680, and a power supply 690, among other components. Those skilled in the art will understand that... Figure 13 The terminal 140 structure shown does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0313] The RF circuit 610 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 680; in addition, it transmits uplink data to the base station.

[0314] The memory 615 can be used to store software programs and modules. The processor 680 executes various terminal functions and image processing by running the software programs and modules stored in the memory 615.

[0315] The input unit 630 can be used to receive input numeric or character information, and to generate key signal inputs related to the terminal's settings and function control. Specifically, the input unit 630 may include a touch panel 631 and other input devices 632.

[0316] The display unit 640 can be used to display input or provided information, as well as various menus of the terminal. The display unit 640 may include a display panel 641.

[0317] Audio circuit 660, speaker 661, and microphone 662 provide an audio interface.

[0318] In this embodiment, the processor 680 included in the terminal 140 can execute the image processing method of the previous embodiment.

[0319] The terminal 140 in this disclosure includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0320] Figure 14 This is a partial structural block diagram of a server 110 implementing an embodiment of the present disclosure. The server 110 can vary significantly due to different configurations or performance characteristics, and may include one or more Central Processing Units (CPUs) 722 (e.g., one or more processors) and a memory 732, and one or more storage media 730 (e.g., one or more mass storage devices) for storing application programs 742 or data 744. The memory 732 and storage media 730 may be temporary or persistent storage. The program stored in the storage media 730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 700. Furthermore, the CPU 722 may be configured to communicate with the storage media 730 and execute the series of instruction operations in the storage media 730 on the server 700.

[0321] Server 110 may also include one or more power supplies 727, one or more wired or wireless network interfaces 750, one or more input / output interfaces 758, and / or one or more operating systems 741, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0322] The central processing unit 722 in server 110 can be used to execute the image processing method of the embodiments of this disclosure.

[0323] This disclosure also provides a computer-readable storage medium for storing program code for executing the image processing methods of the foregoing embodiments.

[0324] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the image processing method described above.

[0325] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0326] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0327] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0328] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0329] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0330] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0331] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0332] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0333] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0334] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. An image processing method, characterized in that, include: Acquire a first image, the first image having a first resolution; The first image is input into the resolution optimization model to obtain a second image with a second resolution greater than the first resolution. The resolution optimization model is obtained by jointly training the student model and the teacher model using the first sample image based on the joint loss function. The first sample image has an optimized image label, which is the training target image of the first sample image. The joint loss function is a function that includes a first loss, a second loss, and a third loss. Wherein, the first loss corresponds to the discrete wavelet subband difference between the first optimized image and the second optimized image, the first optimized image is obtained by the student model by optimizing the resolution of the first sample image, and the second optimized image is obtained by the teacher model by optimizing the resolution of the first sample image; the second loss corresponds to the frequency domain difference between the first optimized image and the second optimized image; and the third loss corresponds to the pixel difference between the optimized image label and the first optimized image.

2. The image processing method according to claim 1, characterized in that, The training steps for the resolution optimization model are as follows: Obtain the first sample image and its corresponding optimized image label; The first sample image is input into the student model for resolution optimization to obtain a first optimized image, and the first sample image is input into the teacher model for resolution optimization to obtain a second optimized image. The teacher model and the student model have different model structures. Perform discrete wavelet transform on the first optimized image to obtain a first discrete wavelet sub-band, and perform discrete wavelet transform on the second optimized image to obtain a second discrete wavelet sub-band, and generate a first loss based on the difference between the second discrete wavelet sub-band and the first discrete wavelet sub-band; Perform a Discrete Fourier Transform on the first optimized image to obtain a first frequency domain, and perform a Discrete Fourier Transform on the second optimized image to obtain a second frequency domain, and generate a second loss based on the difference between the second frequency domain and the first frequency domain; A third loss is generated based on the difference between the first pixel of the optimized image label and the second pixel of the first optimized image; The first loss, the second loss, and the third loss are input into the joint loss function for calculation to obtain the joint loss; The student model is trained using the joint loss to obtain a resolution-optimized model.

3. The image processing method according to claim 2, characterized in that, The step of performing discrete wavelet transform on the first optimized image to obtain a first discrete wavelet sub-band, and performing discrete wavelet transform on the second optimized image to obtain a second discrete wavelet sub-band, includes: The first optimized image is subjected to discrete wavelet transform using a discrete wavelet transform template to obtain a first discrete wavelet sub-band. The first discrete wavelet sub-band includes at least a first low-frequency discrete wavelet sub-band, a first high-frequency discrete wavelet sub-band, a second high-frequency discrete wavelet sub-band, and a third high-frequency discrete wavelet sub-band. The second optimized image is subjected to discrete wavelet transform using a discrete wavelet transform template to obtain a second discrete wavelet sub-band. The second discrete wavelet sub-band includes at least a second low-frequency discrete wavelet sub-band, a fourth high-frequency discrete wavelet sub-band, a fifth high-frequency discrete wavelet sub-band, and a sixth high-frequency discrete wavelet sub-band. The frequency ranges represented by the first, second, and third high-frequency discrete wavelet subbands are different, as are the frequency ranges represented by the fourth, fifth, and sixth high-frequency discrete wavelet subbands.

4. The image processing method according to claim 3, characterized in that, The step of generating the first loss based on the difference between the second discrete wavelet sub-band and the first discrete wavelet sub-band includes: The first sub-loss is obtained based on the difference between the first low-frequency discrete wavelet sub-band and the second low-frequency discrete wavelet sub-band; The second sub-loss is obtained based on the difference between the first high-frequency discrete wavelet sub-band and the fourth high-frequency discrete wavelet sub-band; The third sub-loss is obtained based on the difference between the second high-frequency discrete wavelet sub-band and the fifth high-frequency discrete wavelet sub-band; The fourth sub-loss is obtained based on the difference between the third high-frequency discrete wavelet sub-band and the sixth high-frequency discrete wavelet sub-band; The first loss is obtained by combining the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss.

5. The image processing method according to claim 3 or 4, characterized in that, The first high-frequency discrete wavelet subband is a discrete wavelet subband with high frequency in the horizontal direction and low frequency in the vertical direction; the second high-frequency discrete wavelet subband is a discrete wavelet subband with low frequency in the horizontal direction and high frequency in the vertical direction; and the third high-frequency discrete wavelet subband is a discrete wavelet subband with high frequency in the horizontal direction and high frequency in the vertical direction. The fourth high-frequency discrete wavelet subband is a discrete wavelet subband with high frequency in the horizontal direction and low frequency in the vertical direction; the fifth high-frequency discrete wavelet subband is a discrete wavelet subband with low frequency in the horizontal direction and high frequency in the vertical direction; and the sixth high-frequency discrete wavelet subband is a discrete wavelet subband with high frequency in the horizontal direction and high frequency in the vertical direction.

6. The image processing method according to claim 5, characterized in that, The first loss is obtained by combining the first sub-loss, the second sub-loss, the third sub-loss, and the fourth sub-loss, including: Multiply the first sub-loss by the first weight to obtain the first target sub-loss; Multiply the second sub-loss by the second weight to obtain the second target sub-loss; Multiplying the third sub-loss and the third weight yields the third target sub-loss; Multiply the fourth sub-loss and the fourth weight to obtain the fourth target sub-loss; By combining the first target sub-loss, the second target sub-loss, the third target sub-loss, and the fourth target sub-loss, the first loss is obtained; The sum of the first weight, the second weight, the third weight, and the fourth weight is one.

7. The image processing method according to claim 6, characterized in that, The method further includes: Determine the image type information of the first sample image; Based on the image type information, the values ​​of the first weight, the second weight, the third weight, and the fourth weight are determined respectively.

8. The image processing method according to any one of claims 2 to 7, characterized in that, The step of performing a discrete Fourier transform on the first optimized image to obtain a first frequency domain, and performing a discrete Fourier transform on the second optimized image to obtain a second frequency domain, and generating a second loss based on the difference between the second frequency domain and the first frequency domain, includes: The first optimized image is subjected to a discrete Fourier transform using a discrete Fourier transform template to obtain the first frequency domain. The second optimized image is subjected to a discrete Fourier transform using a discrete Fourier transform template to obtain the second frequency domain. The second loss is obtained based on the difference between the second frequency domain and the first frequency domain.

9. The image processing method according to any one of claims 2 to 8, characterized in that, The step of generating a third loss based on the difference between the first pixel of the optimized image label and the second pixel of the first optimized image includes: Obtain the first pixel value of the first pixel corresponding to the optimized image label; Obtain the second pixel value of the second pixel corresponding to the first optimized image; A third loss is generated based on the difference between the first pixel value and the second pixel value.

10. The image processing method according to any one of claims 1 to 9, characterized in that, The step of inputting the first loss, the second loss, and the third loss into the joint loss function for calculation to obtain the joint loss includes: Obtain a first preset weight value, and then weight the second loss using the first preset weight value to obtain the weighted second loss; The first loss, the weighted second loss, and the third loss are input into the joint loss function for calculation to obtain the joint loss.

11. The image processing method according to claim 10, characterized in that, Before inputting the first loss, the weighted second loss, and the third loss into the joint loss function for calculation to obtain the joint loss, the method further includes: Obtain a second preset weight value, wherein the second preset weight value is greater than the first preset weight value; The third loss is obtained by weighting it with the second preset weight value; The step of inputting the first loss, the weighted second loss, and the third loss into the joint loss function for calculation to obtain the joint loss includes: The first loss, the weighted second loss, and the weighted third loss are input into the joint loss function for calculation to obtain the joint loss.

12. The image processing method according to any one of claims 2 to 11, characterized in that, The step of training the student model using the joint loss to obtain a resolution-optimized model includes: The student model is iteratively trained using the joint loss until the training termination condition is met, resulting in a resolution-optimized model. The training termination condition includes at least the convergence of the joint loss or the number of training iterations reaching a preset threshold.

13. The image processing method according to any one of claims 1 to 12, characterized in that, The resolution of the optimized image label is greater than that of the second optimized image, and the resolution of the second optimized image is greater than that of the first optimized image.

14. The image processing method according to any one of claims 1 to 13, characterized in that, The teacher model has fixed parameters, while the student model has learnable parameters.

15. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire a first image, wherein the first image has a first resolution; The resolution optimization unit is used to input the first image into the resolution optimization model to obtain a second image with a second resolution greater than the first resolution. The resolution optimization model is obtained by jointly training the student model and the teacher model using the first sample image based on the joint loss function during training. The first sample image has an optimized image label, which is the training target image of the first sample image. The joint loss function is a function including a first loss, a second loss, and a third loss. Wherein, the first loss corresponds to the discrete wavelet subband difference between the first optimized image and the second optimized image, the first optimized image is obtained by the student model by optimizing the resolution of the first sample image, and the second optimized image is obtained by the teacher model by optimizing the resolution of the first sample image; the second loss corresponds to the frequency domain difference between the first optimized image and the second optimized image; and the third loss corresponds to the pixel difference between the optimized image label and the first optimized image.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the image processing method according to any one of claims 1 to 14.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image processing method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the image processing method according to any one of claims 1 to 14.

Images