Image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention discloses an image processing method and device, electronic equipment, a storage medium and a program product, and relates to the field of image processing technologies, artificial intelligence technologies and large model technologies, and the method comprises the steps: obtaining a first image; processing the first image by using a first image processing model to obtain a second image, the resolution of the second image being higher than that of the first image, the first image processing model being determined by performing progressive pruning on a convolutional layer of the second image processing model, the progressive pruning is used for representing that the convolutional layer of the second image processing model is sequentially pruned in the direction from being close to the output layer to being far away from the output layer. According to the method, the first image processing model is obtained by pruning the second image processing model, so that the first image processing model is lighter, and the processing effect of the first image processing model is improved.

Description

Technical Field

[0001] The present application relates to the fields of image processing technology, artificial intelligence technology, and large model technology, and specifically to image processing methods, devices, electronic devices, storage media, and program products. Background Art

[0002] Before using low-resolution images in various applications (such as video playback or remote sensing), they often need to be reconstructed into high-resolution images to ensure optimal processing. Interpolation methods, such as bilinear interpolation and bicubic interpolation, are commonly used for image processing. While these image processing methods can improve resolution, the results are not ideal. Summary of the Invention

[0003] In view of this, the present application provides an image processing method, device, electronic device, storage medium and program product to solve the problem of poor image processing effect.

[0004] In a first aspect, the present application provides an image processing method, comprising:

[0005] acquiring a first image;

[0006] The first image is processed using a first image processing model to obtain a second image, where the resolution of the second image is higher than that of the first image. The first image processing model is determined by progressively pruning the convolutional layers of the second image processing model, where the progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned sequentially from close to the output layer to far away from the output layer.

[0007] In a second aspect, the present application provides an image processing device, comprising:

[0008] An image acquisition module, configured to acquire a first image;

[0009] An image processing module is configured to process the first image using a first image processing model to obtain a second image, where the resolution of the second image is higher than that of the first image, and the first image processing model is determined by progressively pruning the convolutional layers of the second image processing model, where the progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned sequentially from close to the output layer to far away from the output layer.

[0010] In a third aspect, the present application provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0011] In a fourth aspect, the present application provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image processing method of the first aspect or any corresponding embodiment thereof.

[0012] In a fifth aspect, the present application provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the image processing method of the first aspect or any corresponding embodiment thereof.

[0013] The image processing method provided by the embodiment of the present application obtains a first image, and processes the first image using a first image processing model to obtain a second image, wherein the resolution of the second image is higher than the resolution of the first image, and the first image processing model is determined by progressively pruning the convolutional layers of the second image processing model, and the progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned in sequence from close to the output layer to far away from the output layer. In this method, since the first image processing model is obtained by pruning the second image processing model, the first image processing model is made more lightweight; in addition, since the progressive pruning starts from the last convolutional layer of the second image processing model and the pruning process is performed sequentially, the deeper features can supervise the learning of the shallower features during the model iteration process, ensuring that the first image processing model can obtain more robust features, thereby improving the processing effect of the first image processing model. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the specific implementation methods of this application or the technical solutions in related technologies, the following is a brief introduction to the drawings required for use in the specific implementation methods or related technical descriptions. Obviously, the drawings described below are some implementation methods of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present application;

[0016] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of target weight parameters according to an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of initial weight parameters according to an embodiment of the present application;

[0019] Figure 5a to Figure 5dis a schematic diagram of progressive pruning according to an embodiment of the present application;

[0020] Figure 6 is a schematic diagram of training a third image processing model according to an embodiment of the present application;

[0021] Figure 7 is a structural block diagram of an image processing device according to an embodiment of the present application;

[0022] Figure 8 Schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] To make the purpose, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0024] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0025] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0026] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0028] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0029] In related technologies, image processing models are constructed based on deep learning models. The original image is input into the image processing model to produce a processed target image. This processing method can improve image processing results, but the performance of the hardware devices deploying the image processing model may not meet the requirements of the image processing model. In other words, the parameters or computational complexity of the image processing models in related technologies are large, making them difficult to deploy on hardware devices with limited performance, thereby limiting the scope of application of the image processing models.

[0030] Based on this, an embodiment of the present application provides an image processing method, which obtains a first image, processes the first image using a first image processing model to obtain a second image, wherein the resolution of the second image is higher than the resolution of the first image, and the first image processing model is determined by progressively pruning the convolutional layers of the second image processing model, and the progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned in sequence from close to the output layer to far away from the output layer. In this method, since the first image processing model is obtained by pruning the second image processing model, the first image processing model is made more lightweight; in addition, since the progressive pruning starts from the last convolutional layer of the second image processing model and prunes forward in sequence, deeper features can supervise the learning of shallower features during the model iteration process, ensuring that the first image processing model can obtain more robust features, thereby improving the processing effect of the first image processing model.

[0031] The image processing method in the embodiment of the present application can be applied to a single image or a video. If applied to a video, each frame of the video is processed sequentially.

[0032] As an optional application scenario of the embodiment of the present disclosure, Figure 1 As shown, the terminal device 110 has an application 101 installed therein, and the user 130 can interact with the application 101 through the terminal device 110 and / or an access device of the terminal device 110 .

[0033] For example, the application 101 can be any application that can provide video playback. For example, the application 101 can be an image processing application, etc. Figure 1 In the application scenario shown, if the application 101 is active, the terminal device 110 can present the interface 102 of the application 101. The interface 102 can include various pages that the application 101 can provide, such as an interaction page, a setting page, a query page, and the like.

[0034] In some embodiments, the terminal device 110 is in communication with the server 120 to provide services for the application 101. The terminal device 110 can be a mobile terminal, a fixed terminal, or a portable terminal, including but not limited to a mobile phone, a desktop computer, a laptop computer, a multimedia tablet, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices, or any combination thereof. In some embodiments, the terminal device 110 can also support any type of interface, and the server 120 can be any type of computing system or server that can provide computing capabilities, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, and the like.

[0035] It should be noted that Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of the present disclosure.

[0036] The embodiments of the present disclosure will be described below with reference to the accompanying drawings. It should be understood that the pages shown in the accompanying drawings are merely examples, and various page designs may actually exist. The various graphic elements in the page may have different arrangements and different visual representations, one or more of which may be omitted or replaced, and one or more other elements may also exist, which are not limited in the embodiments of the present disclosure. In addition, the embodiments are described below mainly with respect to the terminal device 110. It should be understood that the actions described with respect to the terminal device 110 may be performed by the application 101 on the terminal device 110, or may be performed by the application 101 in collaboration with its service end (e.g., server 120).

[0037] According to an embodiment of the present application, an image processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] In this embodiment, an image processing method is provided, which can be used in a terminal device. Figure 2 is a flowchart of an image processing method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0039] Step S201: Acquire a first image.

[0040] The first image may be a single image, or an image frame in a video, etc. The specific form of the first image is not limited herein.

[0041] In the application scenario shown above, application 101 is a video playback application. Before or during video playback, the image frames in the video are processed using the image processing method provided in the embodiment of the present application, and the processed video is played. For another example, application 101 is an image processing application. The user selects a first image by interacting with an image control added to the application. After selecting the first image, the first image is processed and displayed using the image processing method provided in the embodiment of the present application.

[0042] Of course, the first image may have other sources and application scenarios, which are not limited here and can be set according to actual needs.

[0043] Step S202: Process the first image using the first image processing model to obtain a second image.

[0044] The resolution of the second image is higher than that of the first image, and the first image processing model is determined by progressively pruning the convolutional layers of the second image processing model. Progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned in sequence from close to the output layer to far away from the output layer.

[0045] The first image processing model and the second image processing model are used to improve the image quality of the input image, including but not limited to resolution. The input of the first image processing model includes the first image, and the output includes the second image. The first image processing model is constructed based on a neural network model and includes multiple convolutional units, each of which includes at least one convolutional layer.

[0046] The network structure of the first image processing model is similar to that of the second image processing model, except that the number of convolutional layers in the first image processing model is less than the number of convolutional layers in the second image processing model. Specifically, for the trained second image processing model, its convolutional layers are progressively pruned to ultimately determine the first image processing model. The progressive pruning shown is used to characterize that during the training process of the first image processing model, pruning is performed starting from the last convolutional layer of the second image processing model based on the second image processing model, and the model is trained until convergence after pruning; the last convolutional layer of the converged model is pruned again, and the model is trained until convergence after pruning; ...; and so on, until the scale parameter of the pruned model reaches the target scale parameter.

[0047] Progressive pruning describes the approach to the model's convolutional layers during training, pruning layers sequentially from those closest to the output layer to those further away. After each pruning step, the model is trained until convergence, and the model's target scale parameter can be used as a stopping condition for pruning. Of course, the stopping condition for pruning is not limited to the target scale parameter and can also be other conditions, which are not specified here.

[0048] Since the first image processing model is determined after progressive pruning based on the second image processing model, the parameter scale of the first image processing model is smaller than that of the second image processing model. In other words, the first image processing model is a lightweight representation of the second image processing model.

[0049] In the image processing method provided by this embodiment, since the first image processing model is obtained by pruning the second image processing model, the first image processing model is made more lightweight. In addition, since the progressive pruning starts from the last convolutional layer of the second image processing model and prunes forward in sequence, deeper features can supervise the learning of shallower features during the model iteration process, ensuring that the first image processing model can obtain more robust features, thereby improving the processing effect of the first image processing model.

[0050] In some optional embodiments, the convolution kernel of the convolution layer in the first image processing model is configured as a conversion result of the target weight parameter, the target weight parameter is configured as a parallel first convolution branch, the first convolution branch is configured as a serial first convolution unit, and the first convolution unit is configured as a serial of convolution kernels of at least one size.

[0051] The convolution kernel of the convolution layer in the first image processing model is obtained after the target re-parameterization. The re-parameterization can be understood as compressing the trained network structure during testing. For example, the sum of the results of three parallel convolutions (with the same convolution kernel size) is actually equal to the result of a single convolution using the summed convolution kernel. Since multi-branch networks are conducive to network convergence and learning more information, targeted branches can be designed to learn texture and edge information for super-resolution tasks. Therefore, configuring the convolution kernel of the convolution layer in the first image processing model as the conversion result of the target re-parameterization can improve the processing performance of the first image processing model.

[0052] The target re-parameter structure includes parallel first convolution branches, each of which includes a series of first convolution units, each of which includes a series of convolution kernels of at least one size. It should be noted that the sizes of the first convolution branches, first convolution units, and convolution kernels in each first convolution unit are all set according to actual needs and are not limited herein.

[0053] For example, Figure 3 As shown in the figure, the target re-parameter structure includes three parallel first convolution branches, which are called first convolution branch 1 to first convolution branch 3 from left to right. Among them, the first convolution branch 1 includes one first convolution unit, and the first convolution unit includes one 1*1 convolution kernel; the first convolution branch 2 includes four identical first convolution units, and each first convolution unit includes two serial convolution kernels, namely 1*1 convolution and 3*3 convolution; the first convolution branch 3 includes one first convolution unit, and the first convolution unit includes one 3*3 convolution kernel.

[0054] By using target re-parameters, the learning capability of the first image processing model is enhanced, thereby improving the resolution of the second image. Furthermore, it should be noted that the target re-parameters can be applied to all or some of the convolutional layers in the first image processing model. The specific scope of application is determined based on actual needs and is not limited here.

[0055] In some optional implementations, a target weight parameter determination process is further included. Specifically,

[0056] Step a1: Acquire a second sample set, where the second sample set is used to determine target weight parameters.

[0057] Step a2, determining initial re-parameters, the initial re-parameters are configured as multiple parallel optional convolution branches, the optional convolution branches are configured as a series including a corresponding number of optional convolution units, and the optional convolution units are configured as a series of convolution kernels of at least one size.

[0058] Step a3: obtain the number combination of optional convolution branches.

[0059] Step a4: determining, based on the second sample set, the performance of the optional re-parameters corresponding to the combination of the number of optional convolution branches.

[0060] Step a5: If the performance meets the preset requirements, the optional re-parameter is determined as the target re-parameter.

[0061] The second sample set includes input feature-output feature pairs for evaluating the performance of each re-parameterized structure. Specifically, initial re-parameters are determined, wherein the initial re-parameters include a plurality of parallel optional convolution branches, each of the optional convolution branches including a series of a corresponding number of optional convolution units, and each optional convolution unit is configured as a series of convolution kernels of at least one size.

[0062] For example, Figure 4As shown, the initial re-parameters include seven optional convolution branches, from left to right, optional convolution branch 1 to optional convolution branch 7. The number of optional convolution units in optional convolution branch 1 is k1, the number of optional convolution units in optional convolution branch 2 is k2, and so on. The number of optional convolution units in optional convolution branch 6 is k6, and optional convolution branch 7 is the identity mapping. Different values ​​of k1 to k6 can result in various combinations of the number of optional convolution branches.

[0063] For various optional convolution branch number combinations, corresponding optional re-parameters are formed. The performance of the optional re-parameters is then evaluated in combination with the second sample set. By comparing the performance evaluation results corresponding to all optional convolution branch number combinations, the optional convolution branch number combination with the best performance evaluation result is determined as the convolution branch combination in the target re-parameter.

[0064] The performance of the optional re-parameter can be determined by using the difference between the predicted feature obtained after the optional re-parameterization of the same input feature and the target output feature corresponding to the input feature in the second sample set to determine the performance evaluation result.

[0065] On the basis of the general initial reparameters, various optional combinations of the number of convolution groups are enumerated to obtain target reparameters with better effects. That is, by enumerating the hyperparameter settings, a reparameter structure with better effects can be obtained.

[0066] In some optional implementations, the training process of the first image processing model is further included. Specifically,

[0067] Step b1: Obtain a first sample set and a target scale parameter. The first sample set includes a first sample image pair. The two sample images in the first sample image pair have different resolutions.

[0068] Step b2: Starting from the last convolution layer of the second image processing model, after reducing the preset number of convolution layers in sequence, use the first sample set to train until convergence.

[0069] Step b3: If the scale parameter of the converged image processing model reaches the target scale parameter, the converged image processing model is determined as the first image processing model.

[0070] The first sample set is used to train the progressively pruned second image processing model. The first sample set includes two image pairs of different image qualities, namely, first sample image pairs. The two sample images in the first sample image pairs have different resolutions. The target scale parameter is used to represent the target value of the scale parameter of the pruned model. For example, the target scale parameter can be used to represent the number of layers in the first image processing model. The specific constraint object is set based on actual needs and is not limited here.

[0071] Since the second image processing model is a trained model, it is possible to start from the last convolution layer on the basis of the second image processing model, reduce a preset number of convolution layers, and use the model as the first pruned model, and then use the second sample set to train the model until convergence. After convergence, determine whether the model parameters of the model have reached the target model parameters. If not, continue to perform post-pruning training on the basis of the model until convergence. And so on, until the scale parameters of the pruned model can reach the target scale parameters. Among them, the number of convolution layers pruned each time can be the same or different. The specific number is set according to actual needs and is not limited here. For example, one convolution layer is removed each time pruning.

[0072] For example, Figure 5a As shown, the second image processing model includes a 9*9 convolution (9*9Conv) and 8 Swift Parameter-free Attention Blocks (SPABs). Each SPAB includes 3 convolutional layers (Conv). The last SPAB module 501 includes 3 convolutional layers. The second image processing model also includes a connection layer (Concat), a convolution layer (Conv), and an upsampling layer (Up Sample).

[0073] For the last SPAB module 501, after reducing one convolutional layer, the resulting model structure is as follows Figure 5b For the convenience of description, the model after reducing the convolution layer is collectively referred to as the third image processing model. Among them, the last SPAB module 502 in the third image processing model includes 2 convolution layers. Figure 5b The third image processing model shown is trained until convergence, and the Figure 5b The scale parameter of the third image processing model shown in FIG. 1 is used to determine the image processing model. If the target scale parameter is met, Figure 5b The third image processing model shown is determined as the first image processing model.

[0074] like Figure 5b The scale parameter of the third image processing model shown does not meet the target scale parameter, then continue to Figure 5b The convolutional layer of the third image processing model shown is pruned to obtain Figure 5c The third image processing model shown. Figure 5c The last SPAB module 503 in the third image processing model shown includes one convolution layer. Continue to train and determine the scale parameters of the model in the same way as above. If the parameters are met, Figure 5c The third image processing model shown is determined as the first image processing model, otherwise Figure 5cThe progressive pruning is continued based on the third image processing model shown.

[0075] After multiple pruning processes, Figure 5d If the scale parameter of the third image processing model shown can meet the target scale parameter, the third image processing model is determined as the first image processing model.

[0076] During the training of the first image processing model, since the convolutional layers are pruned progressively starting from the last convolutional layer of the second image processing model, deeper features can be used to supervise the learning of shallower features, and then the number of layers is gradually reduced from the deep features to obtain a model with fewer layers.

[0077] In some optional embodiments, the third image processing model is used to represent the image processing model obtained after reducing the preset number of convolutional layers. Accordingly, the training using the first sample set in step b2 includes:

[0078] Step b21: input the first sample image in the first sample image pair into the third image processing model to obtain a first predicted image.

[0079] Step b22: obtaining a first loss value based on a pixel difference between the first predicted image and a second sample image in the first sample image pair, wherein a resolution of the first sample image is smaller than a resolution of the second sample image.

[0080] Step b23: Obtain a second loss value based on the frequency domain information difference between the first predicted image and the second sample image.

[0081] Step b24: Obtain a total loss value based on the fusion result of the first loss value and the second loss value.

[0082] Step b25: Update the parameters of the third image processing model based on the total loss value.

[0083] Specifically, the third image processing model after each pruning is trained using the first sample set, and the first sample image of the first sample image pair in the first sample set is input into the third image processing model to obtain a first predicted image. A loss is calculated based on the second sample image of the first sample image pair and the first predicted image to obtain a total loss value, and the total loss value is used to update the parameters of the third image processing model.

[0084] The loss calculation process includes two types of losses: pixel-level loss and frequency-domain loss. Pixel-level loss represents the loss between the first predicted image and the second sample image at the pixel level, while frequency-domain loss represents the loss between the first predicted image and the second sample image at the frequency level. It should be noted that frequency-domain loss can be one or more types, and can be set based on actual needs.

[0085] The pixel-level loss is called the first loss value, and the frequency-domain loss is called the second loss value. The first and second loss values ​​are fused to obtain the total loss value. The fusion method can be weighted or other processing methods, which are not limited here.

[0086] By setting the second loss value corresponding to the pixel loss and the frequency domain information, it can be ensured that the trained first image processing model is more focused on restoring important areas.

[0087] In some optional implementations, the frequency domain level loss may include frequency domain loss. Based on this, the above step b32 includes:

[0088] Step b3211: Convert the first predicted image and the second sample image into the frequency domain respectively to obtain first frequency domain information and second frequency domain information.

[0089] Step b3212: obtaining a frequency domain loss value based on the difference between the first frequency domain information and the second frequency domain information, wherein the second loss value includes the frequency domain loss value.

[0090] The first predicted image and the second sample image are transformed in the frequency domain to obtain first frequency domain information corresponding to the first predicted image and second frequency domain information corresponding to the second sample image. The frequency domain transformation methods include but are not limited to discrete cosine transform, discrete wavelet transform, and fast Fourier transform.

[0091] After obtaining the first frequency domain information and the second frequency domain information, the difference between the two is used to calculate the loss to obtain the frequency domain loss value. The frequency domain loss value can be directly used as the second loss value, or the second loss value can also include other loss values ​​based on the frequency domain loss value.

[0092] By converting the image to the frequency domain and calculating the frequency domain loss, the model can better distinguish between high-frequency information and low-frequency information, making it easier for the trained first image processing model to perceive the difference between high and low frequencies.

[0093] In some other optional implementations, the frequency domain level loss may include frequency domain loss. Based on this, the above step b32 includes:

[0094] Step b3221: perform high-pass filtering on the first predicted image and the second sample image respectively to obtain a third image and a fourth image.

[0095] Step b3222: Obtain first edge information based on the difference between the first predicted image and the third image.

[0096] Step b3223: Obtain second edge information based on the difference between the second sample image and the fourth image.

[0097] Step b3224: Perform loss calculation based on the first edge information and the second edge information to obtain edge loss, where the second loss value includes the edge loss.

[0098] High-pass filtering is performed on the first predicted image and the second sample image to obtain a third image corresponding to the first predicted image and a fourth image corresponding to the second sample image. For the first predicted image, first edge information is obtained based on the difference between the first predicted image and the third image; for the second predicted image, second edge information is obtained based on the difference between the second sample image and the fourth image.

[0099] Since the first edge information and the second edge information are obtained through high-pass filtering, they may also be referred to as first high-frequency edge information and second high-frequency edge information, respectively.

[0100] A loss calculation is then performed based on the first edge information and the second edge information to obtain an edge loss, wherein the edge loss can be directly used as the second loss value; or the second loss value can also include the edge loss and the frequency domain loss value described above.

[0101] High-frequency information is extracted through high-pass filtering, and high-frequency information is used to extract edges in the image. Since edges in images are important and difficult to learn, edge loss is introduced during model training to enable the first image processing model to more easily perceive the edges of the image.

[0102] For example, the loss used to train the model after progressive pruning can be called a frequency-aware loss. The frequency-aware loss consists of three parts: the first loss described in step b22 above, the frequency domain loss described in steps b3211 to b3212, and the edge loss described in steps b3221 to b3224. The frequency domain loss is calculated by converting the image to the frequency domain, while the edge loss is calculated by extracting high-frequency information through high-pass filtering.

[0103] In some optional implementations, a training process of a second network model is further included. Specifically,

[0104] Step c1, determining a fourth image processing model, where the fourth image processing model is configured to include a teacher network branch and a student network branch, where the number of convolution modules in the teacher network branch is greater than the number of convolution modules in the student network branch.

[0105] Step c2: Using the third sample set to train the fourth image processing model to obtain a fifth image processing model. The third sample set includes a second sample image pair. The two sample images in the second sample image pair have different resolutions.

[0106] Step c3: determine the teacher network branch in the fifth image processing model as the second image processing model.

[0107] The fourth image processing model is configured to include a teacher network branch and a student network branch, where the teacher network branch is similar to the student network branch, except that the number of convolution modules in the teacher network branch is greater than the number of convolution modules in the student network branch.

[0108] The fourth image processing model is trained using the third sample set to obtain a fifth image processing model. The third sample set and the first sample set may be the same or different sample sets, and there is no limitation on this. The third sample set includes a second sample image pair, and the two sample images in the second sample image pair have different resolutions.

[0109] For example, Figure 6 As shown in the figure, the student network branch includes 5 SPAB modules, while the teacher network branch includes 8 SPAB modules. Both the teacher network branch and the student network branch are configured with a connection layer (Concat), a convolution layer (Conv), and an upsampling layer (Up sample). In addition, a student network output layer (StudentOutput) is set for the student network branch, and a teacher network output layer (TeacherOutput) is set for the teacher network branch.

[0110] By configuring the third image processing to include a teacher network branch and a student network branch, and jointly training these two branches, deeper features are used to guide the learning of shallow features, so that the second image processing model used for progressive pruning can learn more robust features.

[0111] In some optional implementations, the above step c2 includes:

[0112] Step c21: input the third sample image in the third sample set into the fourth image processing model to obtain the second predicted image corresponding to the student network branch and the third predicted image corresponding to the teacher network branch.

[0113] Step c22: Calculate frequency perception loss using the second predicted image and the fourth sample image in the third sample set to obtain a third loss value.

[0114] Step c23: Calculate frequency perception loss using the third predicted image and the fourth sample image in the third sample set to obtain a fourth loss value.

[0115] In step c24, a self-distillation loss is calculated using the second predicted image and the third predicted image to obtain a fifth loss value.

[0116] Step c25: Based on the third loss value, the fourth loss value, and the fifth loss value, update the parameters of the fourth image processing model to obtain a fifth image processing model.

[0117] The image pair in the third sample image set is called the third sample image-fourth sample image. The third sample image is input into the fourth image processing model to obtain the second prediction image corresponding to the student network branch and the third prediction image corresponding to the teacher network branch.

[0118] like Figure 6 As shown, the frequency perception loss is calculated using the second predicted image and the fourth sample image to obtain the third loss value L_stu; the frequency perception loss is calculated using the third predicted image and the fourth sample image to obtain the fourth loss value L_tea; the self-distillation loss is calculated using the second predicted image and the third predicted image to obtain the fifth loss value L_std.

[0119] The calculation of the frequency-aware loss can be seen above and will not be described here in detail. The self-distillation loss calculation can be implemented based on the KL divergence (Kullback-Leibler Divergence), or it can be implemented in other ways, which are not limited here.

[0120] During the training of the fourth image processing model, frequency-aware loss is used to allow the model to focus more on recovering important areas. Self-distillation loss is used to allow the model to learn from itself, thereby exploring deeper knowledge representations and improving model performance and generalization capabilities.

[0121] As a specific application embodiment of the embodiment of the present application, an image processing application is installed on a mobile phone, and the user activates the image processing application by interacting with the image processing application. A control for adding a picture is provided on the interface of the image processing application, and the selection of the first image is achieved by interacting with the control. Accordingly, the image processing application is connected to a server corresponding to the application through communication, and the image processing method described in the embodiment of the present application is deployed in the server. The server obtains a second image with better image quality by executing the image processing method, feeds the second image back to the image processing application, and displays the second image on the interface of the image processing application.

[0122] This embodiment also provides an image processing device for implementing the above-mentioned embodiments and preferred implementations. Details already described will not be repeated. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0123] This embodiment provides an image processing device, such as Figure 7 Shown, including:

[0124] The image acquisition module 701 is configured to acquire a first image.

[0125] Image processing module 702 is used to process the first image using the first image processing model to obtain a second image, where the resolution of the second image is higher than that of the first image. The first image processing model is determined by progressively pruning the convolutional layers of the second image processing model. Progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned in sequence from close to the output layer to far away from the output layer.

[0126] In some optional embodiments, the convolution kernel of the convolution layer in the first image processing model is configured as a conversion result of the target weight parameter, the target weight parameter is configured as a parallel first convolution branch, the first convolution branch is configured as a serial first convolution unit, and the first convolution unit is configured as a serial of convolution kernels of at least one size.

[0127] In some optional embodiments, the method further includes:

[0128] The second sample set acquisition module is used to acquire a second sample set, and the second sample set is used to determine the target weight parameter.

[0129] A re-parameter determination module is used to determine initial re-parameters, where the initial re-parameters are configured as multiple parallel optional convolution branches, the optional convolution branches are configured as a series including a corresponding number of optional convolution units, and the optional convolution units are configured as a series of convolution kernels of at least one size.

[0130] The quantity combination acquisition module is used to obtain the quantity combination of optional convolution branches.

[0131] The performance determination module is used to determine the performance of the optional re-parameters corresponding to the combination of the number of optional convolution branches based on the second sample set.

[0132] The re-parameter determination module is used to determine the optional re-parameter as the target re-parameter if the performance meets the preset requirements.

[0133] In some optional embodiments, the method further includes:

[0134] The first sample set acquisition module is used to acquire a first sample set and a target scale parameter. The first sample set includes a first sample image pair. The two sample images in the first sample image pair have different resolutions.

[0135] The training module is used to start from the last convolution layer of the second image processing model, reduce the preset number of convolution layers in sequence, and then use the first sample set to train until convergence.

[0136] The model determination module is configured to determine the converged image processing model as the first image processing model if the scale parameter of the converged image processing model reaches the target scale parameter.

[0137] In some optional embodiments, the third image processing model is used to represent the image processing model obtained after reducing a preset number of convolutional layers, and the training module includes:

[0138] The prediction unit is configured to input the first sample image in the first sample image pair into the third image processing model to obtain a first predicted image.

[0139] The first loss unit is configured to obtain a first loss value based on a pixel difference between the first predicted image and a second sample image in the first sample image pair, where a resolution of the first sample image is smaller than a resolution of the second sample image.

[0140] The second loss unit is configured to obtain a second loss value based on a frequency domain information difference between the first predicted image and the second sample image.

[0141] The loss fusion unit is used to obtain a total loss value based on the fusion result of the first loss value and the second loss value.

[0142] The first parameter updating unit is used to update the parameters of the third image processing model based on the total loss value.

[0143] In some optional embodiments, the second loss unit includes:

[0144] The frequency domain conversion subunit is used to convert the first predicted image and the second sample image into the frequency domain respectively to obtain first frequency domain information and second frequency domain information.

[0145] The frequency domain loss subunit is configured to obtain a frequency domain loss value based on a difference between the first frequency domain information and the second frequency domain information, where the second loss value includes the frequency domain loss value.

[0146] In some optional embodiments, the second loss unit includes:

[0147] The filtering subunit is configured to perform high-pass filtering on the first predicted image and the second sample image respectively to obtain a third image and a fourth image.

[0148] The first edge subunit is configured to obtain first edge information based on a difference between the first predicted image and the third image.

[0149] The second edge subunit is configured to obtain second edge information based on a difference between the second sample image and the fourth image.

[0150] The edge loss subunit is configured to perform loss calculation based on the first edge information and the second edge information to obtain the edge loss, wherein the second loss value includes the edge loss.

[0151] In some optional embodiments, the method further includes:

[0152] A model determination module is used to determine a fourth image processing model, where the fourth image processing model is configured to include a teacher network branch and a student network branch, and the number of convolution modules in the teacher network branch is greater than the number of convolution modules in the student network branch.

[0153] The model training module is used to train the fourth image processing model using the third sample set to obtain a fifth image processing model. The third sample set includes a second sample image pair, and the resolutions of the two sample images in the second sample image pair are different.

[0154] An image processing model determination module is used to determine the teacher network branch in the fifth image processing model as the second image processing model.

[0155] In some optional embodiments, the model training module includes:

[0156] The image prediction unit is used to input the third sample image in the third sample set into the fourth image processing model to obtain the second predicted image corresponding to the student network branch and the third predicted image corresponding to the teacher network branch.

[0157] The third loss unit is configured to perform frequency perception loss calculation using the second predicted image and the fourth sample image in the third sample set to obtain a third loss value.

[0158] The fourth loss unit is configured to perform frequency perception loss calculation using the third predicted image and the fourth sample image in the third sample set to obtain a fourth loss value.

[0159] The fifth loss unit is configured to perform self-distillation loss calculation using the second predicted image and the third predicted image to obtain a fifth loss value.

[0160] The second parameter updating unit is used to update the parameters of the fourth image processing model based on the third loss value, the fourth loss value and the fifth loss value to obtain the fifth image processing model.

[0161] The image processing device provided in the embodiments of the present disclosure can execute the image processing method provided in any embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method. The further functional description of each of the above modules and units is the same as that of the corresponding embodiment above, and will not be repeated here.

[0162] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present disclosure.

[0163] The following specific reference Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a memory 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device are also stored in the RAM 803. The processor 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0164] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and more or fewer devices may be implemented or possessed instead.

[0165] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 809, or installed from the memory 808, or installed from the ROM 802. When the computer program is executed by the processor 801, the above-mentioned functions defined in the image processing method of the embodiment of the present disclosure are performed.

[0166] Figure 8 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0167] The embodiments of the present application also provide a computer-readable storage medium. The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the image processing method shown in the above embodiment is implemented.

[0168] Part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes but is not limited to a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0169] Although the embodiments of the present application have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present application, and such modifications and variations shall fall within the scope defined by the appended claims.

Claims

1. An image processing method, characterized in that: include: acquiring a first image; The first image is processed using a first image processing model to obtain a second image, where the resolution of the second image is higher than that of the first image. The first image processing model is determined by progressively pruning the convolutional layers of the second image processing model, where the progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned sequentially from close to the output layer to far away from the output layer.

2. The method according to claim 1, characterized in that The convolution kernel of the convolution layer in the first image processing model is configured as a conversion result of a target weight parameter, the target weight parameter is configured as a parallel first convolution branch, the first convolution branch is configured as a serial first convolution unit, and the first convolution unit is configured as a serial of convolution kernels of at least one size.

3. The method according to claim 2, characterized in that Also includes: Acquire a second sample set, where the second sample set is used to determine the target weight parameter; Determining initial re-parameters, wherein the initial re-parameters are configured as a plurality of parallel selectable convolution branches, wherein the selectable convolution branches are configured as a series including a corresponding number of selectable convolution units, wherein the selectable convolution units are configured as a series of convolution kernels of at least one size; Obtaining a quantity combination of the optional convolution branches; Determining, based on the second sample set, performance of optional re-parameters corresponding to the combination of the number of optional convolution branches; If the performance meets the preset requirement, the optional re-parameter is determined as the target re-parameter.

4. The method according to claim 1, wherein Also includes: Acquire a first sample set and a target scale parameter, wherein the first sample set includes a first sample image pair, and two sample images in the first sample image pair have different resolutions; Starting from the last convolutional layer of the second image processing model, after successively reducing a preset number of convolutional layers, training with the first sample set until convergence; If the scale parameter of the converged image processing model reaches the target scale parameter, the converged image processing model is determined as the first image processing model.

5. The method according to claim 4, characterized in that The third image processing model is used to represent an image processing model obtained by reducing a preset number of convolutional layers, and the training using the first sample set includes: inputting the first sample image of the first sample image pair into the third image processing model to obtain a first predicted image; obtaining a first loss value based on a pixel difference between the first predicted image and a second sample image in the first sample image pair, wherein a resolution of the first sample image is smaller than a resolution of the second sample image; Obtaining a second loss value based on a difference in frequency domain information between the first predicted image and the second sample image; Obtaining a total loss value based on a fusion result of the first loss value and the second loss value; Parameters of the third image processing model are updated based on the total loss value.

6. The method according to claim 5, characterized in that The obtaining a second loss value based on a frequency domain information difference between the first predicted image and the second sample image includes: Converting the first predicted image and the second sample image into frequency domains respectively to obtain first frequency domain information and second frequency domain information; A frequency domain loss value is obtained based on a difference between the first frequency domain information and the second frequency domain information, where the second loss value includes the frequency domain loss value.

7. The method according to claim 5 or 6, characterized in that The obtaining a second loss value based on a frequency domain information difference between the first predicted image and the second sample image includes: performing high-pass filtering on the first predicted image and the second sample image respectively to obtain a third image and a fourth image; obtaining first edge information based on a difference between the first predicted image and the third image; obtaining second edge information based on a difference between the second sample image and the fourth image; Loss calculation is performed based on the first edge information and the second edge information to obtain edge loss, and the second loss value includes the edge loss.

8. The method according to claim 1, characterized in that Also includes: Determining a fourth image processing model, where the fourth image processing model is configured to include a teacher network branch and a student network branch, wherein the number of convolution modules in the teacher network branch is greater than the number of convolution modules in the student network branch; training the fourth image processing model using a third sample set to obtain a fifth image processing model, wherein the third sample set includes a second sample image pair, and two sample images in the second sample image pair have different resolutions; The teacher network branch in the fifth image processing model is determined as the second image processing model.

9. The method according to claim 8, characterized in that The method of training the fourth image processing model using the third sample set to obtain a fifth image processing model includes: Inputting the third sample image in the third sample set into the fourth image processing model to obtain a second predicted image corresponding to the student network branch and a third predicted image corresponding to the teacher network branch; performing frequency perception loss calculation using the second predicted image and the fourth sample image in the third sample set to obtain a third loss value; performing frequency perception loss calculation using the third predicted image and a fourth sample image in the third sample set to obtain a fourth loss value; performing a self-distillation loss calculation using the second predicted image and the third predicted image to obtain a fifth loss value; Based on the third loss value, the fourth loss value and the fifth loss value, the parameters of the fourth image processing model are updated to obtain the fifth image processing model.

10. An image processing device, characterized in that: include: An image acquisition module, configured to acquire a first image; An image processing module is configured to process the first image using a first image processing model to obtain a second image, where the resolution of the second image is higher than that of the first image, and the first image processing model is determined by progressively pruning the convolutional layers of the second image processing model, where the progressive pruning is used to characterize that the convolutional layers of the second image processing model are pruned sequentially from close to the output layer to far away from the output layer.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image processing method according to one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the image processing method according to any one of claims 1 to 9.

Images