An image processing method, apparatus, electronic device, and storage medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to an image processing method and device, electronic equipment and a storage medium, which are used for realizing more targeted specific degradation processing of an image. The method comprises the following steps: inputting an image to be processed into an initial degradation processing model to obtain an initial output image; performing gradient integration on network parameters corresponding to each neuron in a tree pool degradation processing model based on the initial output image to obtain an influence factor of each neuron on a target degradation type; selecting a target neuron corresponding to the target degradation type based on the influence factors; updating the target neuron in the first degradation processing model to obtain a target degradation processing model; and performing image processing on the image to be processed based on the target degradation processing model to obtain a target output image. The application performs gradient integration on network parameters to obtain a more targeted target neuron, so that more targeted specific degradation processing of an image is realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In related technologies, the interpretability of high-level tasks mainly focuses on "attribution," that is, identifying which parts (pixels) or neurons in the input image have the greatest impact on the model's final prediction. However, the learnable neurons (filters) identified by these methods often fail to provide satisfactory results for modifying the network's specific functions (such as simply changing the deblurring function). In other words, the identified filters are not the most important filters for the current degradation. Therefore, how to obtain more targeted neurons to achieve more targeted specific degradation processing of images is an urgent problem to be solved. Summary of the Invention

[0003] This application provides an image processing method, apparatus, electronic device, and storage medium to achieve more targeted specific degradation processing of images.

[0004] An image processing method provided in this application includes:

[0005] The image to be processed is input into the initial degradation processing model to obtain the initial output image of the initial degradation processing model. The initial degradation processing model is generated based on the first degradation processing model and the second degradation processing model. The first degradation processing model is a model for restoring degraded images of at least two degradation types, and the second degradation processing model is a model for restoring degraded images of at least one degradation type.

[0006] Based on the initial output image, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on the target degradation type.

[0007] Based on the obtained influencing factors, target neurons corresponding to the target degradation type are selected from the neurons, and the target neurons in the first degradation processing model are updated to obtain the corresponding target degradation processing model.

[0008] Based on the target degradation processing model, image processing is performed on the image to be processed to obtain the target output image.

[0009] An image processing apparatus provided in this application includes:

[0010] The first processing unit is used to input the image to be processed into an initial degradation processing model and obtain an initial output image of the initial degradation processing model. The initial degradation processing model is generated based on a first degradation processing model and a second degradation processing model. The first degradation processing model is a model for restoring degraded images of at least two degradation types, and the second degradation processing model is a model for restoring degraded images of at least one degradation type.

[0011] An integration unit is used to perform gradient integration on the network parameters corresponding to each neuron in the initial degradation processing model based on the initial output image, so as to obtain the influence factor of each neuron on the target degradation type.

[0012] A screening unit is used to select the target neuron corresponding to the target degradation type from the neurons based on the obtained influencing factors, and update the target neuron in the first degradation processing model to obtain the corresponding target degradation processing model.

[0013] The second processing unit is used to perform image processing on the image to be processed based on the target degradation processing model to obtain the target output image.

[0014] Optionally, the filtering unit is specifically used for:

[0015] The neurons are sorted according to their respective influencing factors;

[0016] The neurons whose sorting results are within a preset order range are selected as the target neurons.

[0017] Optionally, the second processing unit is specifically used to perform at least one of the following operations:

[0018] Replace the network parameters corresponding to the target neuron in the first degradation processing model with the network parameters corresponding to the associated neuron in the second degradation processing model to obtain the target degradation processing model;

[0019] The network parameters corresponding to the target neurons in the first degradation processing model are subjected to at least one of pruning and quantization processing to obtain the target degradation processing model;

[0020] Linear interpolation is performed on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model to obtain the target degradation processing model.

[0021] Optionally, when performing linear interpolation on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model, the second degradation processing model and the first degradation processing model have different parameter combination ratios. The parameter combination ratios are used to balance the regulatory ability of the target degradation processing model to handle different degradation types.

[0022] Optionally, the device further includes:

[0023] The model generation unit is used to obtain the first degradation processing model in the following manner:

[0024] The network parameters corresponding to the target neuron in the second degradation processing model are replaced with the network parameters corresponding to the associated neurons in the sample model to obtain the first degradation processing model. The sample model is a model for restoring at least the degradation image of the target degradation type, and the second degradation processing model is a model for restoring degradation images of at least one degradation type. The first degradation processing model, the second degradation processing model, and the sample model have the same number of network parameters.

[0025] Optionally, the second degradation processing model is a model for processing degraded images of the downsampled degradation type, wherein the at least two degradation types include the downsampled degradation type.

[0026] Optionally, the device further includes:

[0027] A classification unit is used to acquire multiple reference images, each containing a degradation type;

[0028] Based on the multiple reference images, the reference neurons corresponding to each type of degradation are obtained;

[0029] The target degradation type corresponding to the image to be processed is determined based on the overlap ratio between the target neuron and the reference neuron.

[0030] Optionally, the classification unit is specifically used for:

[0031] The plurality of reference images are respectively input into the initial degradation processing model to obtain each reference output image of the initial degradation processing model;

[0032] Based on each of the reference output images, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on each of the various degradation types.

[0033] Based on the influence factors of each neuron on the various degeneration types, reference neurons corresponding to each degeneration type are selected from the various neurons.

[0034] An electronic device provided in this application includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor performs the steps of any of the above-described image processing methods.

[0035] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of any of the above-described image processing methods.

[0036] This application provides a computer-readable storage medium including program code. When the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps of any of the above-described image processing methods.

[0037] The beneficial effects of this application are as follows:

[0038] This application provides an image processing method, apparatus, electronic device, and storage medium. By integrating the gradients of the network parameters corresponding to each neuron in the initial degradation processing model, this application obtains the influence factor of each neuron on the target degradation type. Based on this, the selected target neurons are more targeted to the target degradation type, enabling the identification of crucial neurons corresponding to different degradation types. This application can alter the specific function of the network by only changing the selected target neurons; without introducing new parameters, it updates the network parameters corresponding to the target neurons in the first degradation processing model, resulting in a target degradation processing model capable of handling only specific degradation types. This allows for more targeted degradation processing of the image to be processed.

[0039] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0041] Figure 1 This is a schematic diagram illustrating one application scenario in an embodiment of this application;

[0042] Figure 2 This is a flowchart illustrating an image processing method according to this application;

[0043] Figure 3A This is a schematic diagram of a high-definition image in an embodiment of this application;

[0044] Figure 3B This is a schematic diagram of a degraded image according to one embodiment of this application;

[0045] Figure 3C This is a schematic diagram of another degraded image in an embodiment of this application;

[0046] Figure 4 This is a flowchart illustrating a method for calculating an impact factor in an embodiment of this application.

[0047] Figure 5 This is a comparative diagram of the output effects of different models in an embodiment of this application;

[0048] Figure 6 This is a schematic diagram illustrating the denoising effect of a model generated based on different parameters in an embodiment of this application;

[0049] Figure 7 This is a schematic diagram illustrating a comparison of the results of updating neurons obtained by different methods in an embodiment of this application;

[0050] Figure 8 This is a flowchart illustrating a method for determining the degradation type of an image to be processed, as described in an embodiment of this application.

[0051] Figure 9 This is a flowchart illustrating a method for determining a reference neuron in an embodiment of this application;

[0052] Figure 10 This is a schematic diagram comparing the effects of different methods and different proportions of masks in an embodiment of this application.

[0053] Figure 11A This is a schematic diagram illustrating the relationship between the proportion of the first type of target neurons and the changes in model performance in the embodiments of this application;

[0054] Figure 11B This is a schematic diagram illustrating the relationship between the proportion of the second type of target neurons and the changes in model performance in the embodiments of this application;

[0055] Figure 11C This is a schematic diagram illustrating the relationship between the proportion of the third type of target neurons and the changes in model performance in the embodiments of this application;

[0056] Figure 11D This is a schematic diagram illustrating the relationship between the proportion of the fourth type of target neurons and the changes in model performance in the embodiments of this application;

[0057] Figure 12 This is a flowchart illustrating a complete image processing method in an embodiment of this application;

[0058] Figure 13 This is a schematic diagram of the composition structure of an image processing device according to an embodiment of this application;

[0059] Figure 14 This is a schematic diagram of the composition structure of an electronic device according to an embodiment of this application;

[0060] Figure 15 This is a schematic diagram of the composition structure of another electronic device using an embodiment of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0062] The following describes some of the concepts involved in the embodiments of this application.

[0063] Neural networks are widely interconnected, parallel networks composed of adaptive, simple units. Their organization simulates the interactive responses of biological nervous systems to real-world objects. Neural networks can mimic the nervous system's response to input and are a type of machine learning technique that simulates the neural network of the human brain in an attempt to achieve artificial intelligence-like results.

[0064] Neuron: Its function is very similar to that of a human neuron. It is the most basic component of a neural network and has two states: "excitation" and "inhibition." A neural network is composed of neurons arranged in a certain structure, with each neuron connected to other neurons. When a neuron is "excited," it transmits chemical substances to the next connected neuron, changing the potential of the next neuron. When the potential exceeds a certain threshold, the stimulated neuron becomes "excited," which is neuron activation. In applications, this activation process is represented by an activation function. The neuron model is a model that includes input, output, and computation functions. Input can be analogized to the dendrites of a neuron, output to the axon of a neuron, and computation to the cell nucleus. In this application, each K in the convolutional layer... A K-weight is defined as a neuron, where K is the size of the convolution kernel.

[0065] Image degradation refers to the general concept of something deteriorating from its best to worst. During the formation, recording, processing, and transmission of an image, imperfections in the imaging system, recording equipment, transmission medium, and processing methods can lead to a decline in image quality; this phenomenon is called image degradation. Typical manifestations of image degradation include blurring, distortion, and noise. Many factors can cause degradation, such as aberrations in the optical imaging system, imaging diffraction, imaging nonlinearity, geometric distortion, relative motion between the imaging system and the subject, and system noise. Correspondingly, degradation types include noise, blurring, and bicubic downsampling.

[0066] Degradation processing models: These are models used in this application to restore degraded images of different degradation types. The first degradation processing model is used to restore degraded images of at least two degradation types, such as for deblurring, noise reduction, and removal of bicubic subsampling. The second degradation processing model is primarily used to restore degraded images of at least one degradation type. In this application, the example of the second degradation model only being able to handle bicubic subsampling is used for illustration.

[0067] Super-resolution (SLR) is a process of increasing the resolution of an original image using hardware or software methods. It involves creating a high-resolution image from a series of low-resolution images. SLR has wide applications in smart cities, big data healthcare, multimedia social networking, autonomous driving, and many other fields, making it a crucial digital image processing technology.

[0068] Pruning: Neural networks have numerous parameters, but some parameters contribute little to the final output image and are therefore redundant. Pruning, as the name suggests, involves removing these redundant parameters. First, the neurons in the model need to be ranked according to their contribution to the final output image. Then, neurons with low contribution are discarded, resulting in faster model execution and smaller model file size.

[0069] Linear interpolation: An interpolation method that uses a linear function as the interpolation function. Specifically, it refers to the method of using a straight line connecting two known quantities to determine the value of an unknown quantity between the two known quantities.

[0070] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0071] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0072] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0073] The solutions provided in this application embodiment relate to machine learning technology in artificial intelligence. The degradation processing model proposed in this application embodiment is mainly used for image restoration and classification. The training and application methods of the degradation processing model can be divided into two parts: a training part and an application part. The training part involves the field of machine learning. In the training part, the degradation processing model is trained using machine learning technology, so that sample images in the training sample set are processed by the degradation processing model to restore degraded images. The application part is used to restore and classify images to be processed using the degradation processing model trained in the training part.

[0074] The design concept of the embodiments of this application is briefly introduced below:

[0075] With the development of information technology, people have increasingly higher requirements for the quality of digital images, especially in computer vision fields such as medicine, autonomous driving, and astronomy, where high-resolution, detailed, and high-definition images are needed. However, in actual image acquisition, the spatial resolution of the obtained images is often affected by factors such as the spatial resolution of the imaging system itself, the intensity of light or rays, spatial distance, and system noise, resulting in low spatial resolution.

[0076] In the field of image processing, high-level interpretable methods mainly fall into two categories: 1) Perturbation-based methods rely on extensive sampling to ensure the final effect, which is very time-consuming and computationally expensive. 2) Gradient integral (IG) methods, which accumulate the gradients of all pixels in the input image along the path between the reference and target images, using the gradients to determine which regions in the image or which neurons in the network are most important for the final prediction result. Since these methods calculate gradients by altering the network input, and changes in the blind super-resolution network function are achieved through changes in filter weights, it is difficult to directly attribute functional changes to filters when applied directly to image processing scenarios. In terms of results, the filters identified by these methods often fail to achieve satisfactory results in modifying the network's specific function (such as simply changing the deblurring function), meaning the identified filters are not the most important filters for the current degradation.

[0077] In view of this, this application proposes an image processing method, apparatus, electronic device, and storage medium. By integrating the gradients of the network parameters corresponding to each neuron in the initial degradation processing model, this application obtains the influence factor of each neuron on the target degradation type. Based on this, the selected target neurons are more targeted to the target degradation type, enabling the identification of crucial neurons corresponding to different degradation types. This application can alter the specific function of the network by only changing the selected target neurons; without introducing new parameters, it updates the network parameters corresponding to the target neurons in the first degradation processing model, resulting in a target degradation processing model capable of handling only specific degradation types, thereby achieving more targeted specific degradation processing for the image to be processed.

[0078] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0079] like Figure 1 The diagram illustrates an application scenario of an embodiment of this application. The application scenario diagram includes two terminal devices 110 and one server 120. The terminal device 110 in this embodiment may have an image processing-related client installed. The server 120 may include an image processing-related server. Furthermore, the client in this application can be software, a webpage, a mini-program, etc., and the server is a backend server corresponding to the software, webpage, mini-program, etc., or a server specifically used for image processing, model training, etc. This application does not impose specific limitations.

[0080] It should be noted that the image processing method in this application embodiment can be executed by the server or the terminal device alone, or by both the server and the terminal device. For example, the terminal device inputs the image to be processed into the initial degradation processing model to obtain the initial output image of the initial degradation processing model; based on the initial output image, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on the target degradation type; then, based on the obtained influence factors, the target neuron corresponding to the target degradation type is selected from each neuron, and the target neuron in the first degradation processing model is updated to obtain the corresponding target degradation processing model; based on the target degradation processing model, image processing is performed on the image to be processed to obtain the target output image. Alternatively, the server executes the above steps. Or, the server obtains the target processing model based on the above steps, and then the terminal device obtains the target output image based on the target processing model and displays it to the user, etc. This application does not make specific limitations here, and the following mainly uses the terminal device as an example for illustration.

[0081] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network.

[0082] In one alternative implementation, the communication network is a wired network or a wireless network.

[0083] In this embodiment, the terminal device 110 is a computer device used by a user. This computer device can be a personal computer, mobile phone, tablet computer, laptop, e-book reader, in-vehicle terminal, or any other computer device with a certain computing power that runs instant messaging software and websites or social networking software and websites. Each terminal device 110 is connected to the server 120 via a wireless network. The server 120 is a single server, a server cluster composed of several servers, a cloud computing center, or a virtualization platform.

[0084] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0085] The image processing method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way.

[0086] See Figure 2 The diagram shown is a flowchart illustrating an image processing method provided in this application, using a terminal device as the execution subject for illustrative purposes. The specific implementation flow of this method is as follows:

[0087] S21: The terminal device inputs the image to be processed into the initial degradation processing model and obtains the initial output image of the initial degradation processing model. The initial degradation processing model is generated based on the first degradation processing model and the second degradation processing model. The first degradation processing model is a model for restoring degraded images of at least two degradation types, and the second degradation processing model is a model for restoring degraded images of at least one degradation type. The image to be processed is a degraded image containing the target degradation type.

[0088] In this embodiment, image degradation specifically refers to degradations such as blurring, noise, and bicubic downsampling. The first degradation processing model can be a model used to handle multiple degradations, and can also be referred to as the target model. For example, this model can be used to restore fuzziness, noise, bicubic downsampling and other degradations.

[0089] Furthermore, considering that bicubic downsampling is the most commonly used degradation method in image restoration, when the input image is a bicubic sampled image, the degradation processing model can output a high-resolution result; however, when the input image has other degradations (such as blur), although the degradation processing model cannot process it, it can still guarantee that its output is an image. Therefore, in an optional implementation, the second degradation processing model is a model for processing degraded images of the downsampling degradation type, and at least two degradation types include the downsampling degradation type.

[0090] In this embodiment, the second degradation processing model is mainly used as an example, which can only process degradation images with bicubic downsampling. The second degradation processing model can also be called the initial model, denoted as... .

[0091] For example Figure 3A As shown, this is a high-definition image, also known as a super-resolution image, in an embodiment of this application. Figure 3B and Figure 3C These are two types of degraded images corresponding to the high-resolution image. Among them, Figure 3B This indicates the blurred image corresponding to the high-resolution image. Figure 3C This indicates the image containing noise that corresponds to the high-resolution image.

[0092] Taking the target degradation type as fuzzy as an example, at this time Figure 3B This can serve as a schematic diagram of an image to be processed in an embodiment of this application; taking noise as an example of target degradation type, Figure 3C This can serve as a schematic diagram of an image to be processed in an embodiment of this application.

[0093] It should be noted that the above are merely illustrative examples, and this application does not specifically limit the types of degradation.

[0094] S22: Based on the initial output image, the terminal device performs gradient integration on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on the target degradation type.

[0095] The influence factor is used to characterize the importance of filters in the network for a certain type of degradation. In this application, the gradient value corresponding to the filter can be used as the influence factor.

[0096] Unlike the IG method in related technologies, the gradient integral in this application adopts a newly proposed method of gradient integration on learnable neurons in the network (Filter Attribution Integrated Gradients, FAIG), which mainly integrates the gradient of the filter in the parameter space.

[0097] One alternative implementation is to proceed as follows: Figure 4 The flowchart shown below implements step S22, which is a schematic diagram of a method for calculating an impact factor in an embodiment of this application, including the following steps:

[0098] S401: The terminal device constructs a loss function corresponding to the initial degradation processing model based on the difference between the initial output image and the sample image. The sample image is the super-resolution image corresponding to the image to be processed.

[0099] In the embodiments of this application, it is possible to use This represents the initial output image of the initial degradation processing model after the image to be processed, x, is input into it. gt If we represent a sample image, then the loss function can be expressed as:

[0100] (Formula 1)

[0101] Among them, in the above formula The loss function represents the distance between the initial output image and the sample image. The sample image can also be called the target image, for example... Figure 3A The image shown is a target image illustrated in an embodiment of this application.

[0102] Specifically, the network parameters θ in the initial degradation treatment model are obtained by modifying the initial model. With the target model It is generated by linear interpolation between them.

[0103] S402: The terminal device performs gradient integration on the network parameters corresponding to the neurons of the initial degradation processing model according to the preset integration path based on the loss function, and obtains the influence factor of each neuron in the initial degradation processing model on the target degradation type.

[0104] The preset integration path is generated by linear interpolation of the network parameters of the second and first degenerate processing models. The second and first degenerate processing models have the same parameter space structure, and the corresponding neuron positions, connections, etc. are consistent. The number of network parameters is the same, but the specific values ​​of the network parameters may not be the same.

[0105] Specifically, the calculation formula for FAIG, the tool proposed in this application for analyzing blind super-resolution models, is as follows:

[0106] (Formula 2)

[0107] Among them, the integration path By using the initial model With the target model The integral is generated by linear interpolation between the parameters, where N represents the number of integration sub-intervals. The integration sub-intervals are divided according to a certain integration step size, which is generally empirically 100. This method mainly performs gradient integration on the filter in the parameter space. By sampling a large number of points on the integration path (e.g., sampling 100 points), it can approximate the integration result as closely as possible.

[0108] In one optional implementation, based on the obtained influencing factors, target neurons corresponding to the target degradation type are selected from each neuron, including:

[0109] The neurons are sorted according to their respective influencing factors; the neurons whose sorting results fall within the preset order range are selected as target neurons.

[0110] For example, neurons can be sorted in descending order of their respective influence factors, and the top 1% (1% of all learnable neurons in the entire network model) of neurons can be selected as target neurons; or, 5% (5% of all learnable neurons in the entire network model) of neurons can be selected as target neurons, etc. This preset order range can be set according to actual needs and is not specifically limited here.

[0111] It should be noted that the above sorting method is just an example. It can also be sorted in ascending order to select the bottom 1% of neurons, etc. There is no specific limitation here.

[0112] In this embodiment, by sorting the gradient values ​​calculated by all filters, the importance of each filter in the network for a specific degradation is obtained; a larger gradient value indicates greater importance. Furthermore, these identified filters (the top 1%, i.e., the target neurons) play a crucial role in preserving the specific function of the target model and enabling the initial model to handle a certain degradation, rather than being unable to do so. Based on these filters, unsupervised classification of degradation in the input image can be achieved, and the network's capabilities can be adjusted without increasing any parameters.

[0113] S23: Based on the obtained influencing factors, the terminal device selects the target neuron corresponding to the target degradation type from each neuron and updates the target neuron in the first degradation processing model to obtain the corresponding target degradation processing model.

[0114] S24: The terminal device performs image processing on the image to be processed based on the target degradation processing model to obtain the target output image.

[0115] In the above embodiments, since this application obtains the influence factor of each neuron on the target degradation type by performing gradient integration on the network parameters corresponding to each neuron in the initial degradation processing model, the target neurons selected based on this are more targeted to the target degradation type, and can find the corresponding important neurons according to different degradations. This application can change the specific function of the network by only changing the selected target neurons; without introducing new parameters, the network parameters corresponding to the target neurons in the first degradation processing model are updated to obtain a target degradation processing model that can only handle specific degradation types, thereby achieving more targeted specific degradation processing of the image to be processed.

[0116] Step S23 will be described in detail below. In this embodiment, the target neuron in the first degradation processing model is updated to obtain the corresponding target degradation processing model, including any one of the following operations:

[0117] Method 1: Erasure based on the found filters Its specific functions.

[0118] Optionally, the network parameters corresponding to the target neuron in the first degradation processing model can be replaced with the network parameters corresponding to the associated neuron in the second degradation processing model to obtain the target degradation processing model.

[0119] The second degradation treatment model can be one of the models listed above. For the locations of the target neuron filters found by the FAIG method, Replace the values ​​of the network parameters at these locations with those at the same locations. The value can be used for The function can be specifically modified. For example, for target neuron filters found for the target degradation type of blur, when based on the above method... After replacing these filters, the resulting new model (which can be called the target degradation processing model 1) loses its deblurring function but retains its noise reduction function; for example, the target neuron filters found for the noise type of target degradation, when applied to the above method... After replacing these filters, the resulting new model (which can be called the target degradation processing model 2) loses its denoising function but retains its deblurring function, and its effect is as follows: Figure 5 As shown.

[0120] See Figure 5The diagram shown is a comparison of the output effects of different models in an embodiment of this application. The diagram includes the results of masking filters found for different degradation methods. The images are arranged in two rows and four columns, totaling eight images, numbered 1-8. The first column represents the input image, i.e., the image to be processed; the second column represents the network output, i.e., ... The initial output image; the third column is the mask 1% deblurring neuron (Mask). 1% The output of the new model obtained by deblurring filters is the target output image of the target degradation processing model 1; the fourth column is the mask 1% denoising neuron (Mask 1% The output of the new model obtained by denoising filters is the target output image output by the target degradation processing model 2; in addition, the first row represents the blurry input and its corresponding output results, and the second row represents the noise input and its corresponding output results.

[0121] Based on the above, the image numbered 1 (which can be called image 1, and the other images can be processed in the same way) is a blurred image to be processed, and image 5 is a noisy image to be processed; the three output results corresponding to image 1 are image 2, image 3 and image 4, and the three output results corresponding to image 5 are image 6, image 7 and image 8.

[0122] Specifically, It has the ability to deblur and remove noise, such as Figure 5 As shown in Images 2 and 6, when 1% of the target neuron filters found for deblurring are masked using the above method, the resulting target degradation processing model 1 lacks deblurring capability, as shown in Image 3, but still possesses noise reduction capability, as shown in Image 7. Similarly, when 1% of the target neuron filters found for noise reduction are masked using the above method, the resulting target degradation processing model 2 lacks noise reduction capability, as shown in Image 8, but still possesses deblurring capability, as shown in Image 4. Here, "mask" refers to... Replace the value of filters at a specific location with The corresponding value in the middle.

[0123] In the above embodiments, the FAIG method proposed in this application can identify the most important filters based on different degradations. When the weights of these filters are changed, this application can effectively preserve other functions (such as noise reduction) while erasing certain functions of the target model (such as deblurring). This method can serve as a good tool for analyzing blind super-resolution.

[0124] Method 2: Pruning and Quantification.

[0125] Optionally, the target degradation model can be obtained by performing at least one of pruning and quantization processing on the network parameters corresponding to the target neurons in the first degradation model.

[0126] In this approach, the network parameters corresponding to neurons that are not important for the target degradation type need to be pruned or quantized. Here, the target neuron can refer to the bottom 1% of neurons selected after sorting all neurons according to their influence factors from largest to smallest; that is, the 1% of neurons least important for the target degradation type.

[0127] In the above embodiments, the FAIG proposed in this application has good reference value for model pruning and quantization. The small number of filters (such as 1%) found based on FAIG play a very important role in adding specific functions to the network.

[0128] Method 3: Adjust the network's capabilities based on the found filters.

[0129] Optionally, linear interpolation is performed on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model to obtain the target degradation processing model.

[0130] Specifically, based on the locations of the filters found by FAIG, linear interpolation is performed at these locations on the parameters of the initial model and the target model to obtain a new model, namely the target degradation processing model, whose formula is as follows:

[0131] (Formula 3)

[0132] Among them, the second degradation treatment model and the first degradation treatment model correspond to different parameter combination ratios, and the first degradation treatment model... The corresponding parameter combination ratio is λ, the second degradation treatment model. The corresponding parameter combination ratio is 1-λ.

[0133] In this embodiment, the parameters combined with the ratio are used to balance the adjustment capability of the target degradation treatment model in handling different degradation types. That is, It can be used to represent the weight combination ratio between the initial model and the target model, and can be adjusted without adding any parameters. It can adjust the strength of the target degradation processing model in handling degradation, and the results are as follows: Figure 6 As shown.

[0134] Figure 6 This diagram illustrates the denoising effect of a model generated based on different parameters in an embodiment of this application. It represents the denoising effect of a target degradation processing model generated based on different λ values ​​and is divided into two rows and nine columns. The first row represents the different target output images corresponding to different λ values ​​when the input image is grass, and the second row represents the different target output images corresponding to different λ values ​​when the input image is sky.

[0135] The first column represents the input image, i.e. the image to be processed, specifically the part within the white rectangle in the image; the second column represents the output image when λ=0.0, i.e. when the target degradation processing model is the initial model, which obviously cannot be denoised; the third column represents the output image corresponding to the target degradation processing model when λ=0.2, ..., and so on, to obtain the output images corresponding to the target degradation processing models generated based on different λ.

[0136] When the input image is a grassland, by Figure 6 It can be seen that the output image is optimal when λ=0.8, as this output image removes noise while also preserving image details; when the input image is the sky, the optimal output image is achieved by... Figure 6 It can be seen that the output image is optimal when λ=1.0.

[0137] In the above implementation, based on the filters identified by FAIG, the degree to which the model removes a certain type of degradation can be controlled without introducing any new parameters, thus balancing the model's ability to adjust for different types of degradation.

[0138] In this embodiment, in addition to the methods for obtaining the target degradation processing model listed above, target neuron filters can also be obtained based on the above methods. Based on the found filters, additional... Its specific function is to update the target neurons in the second degradation processing model to obtain the target degradation processing model.

[0139] Specifically, for the initial model In the training dataset using blind super-resolution, only the filters found by FAIG (1% and 5%) are updated. Taking 1% as an example, assume that the top 1% of target neurons most important for the type of blur degradation are identified, and then... Replace the network parameters at the corresponding position of the target neuron with By analyzing the network parameters at the corresponding positions, a model with deblurring capabilities can be obtained, i.e., a target degradation processing model.

[0140] Compared to other methods of finding filters and methods of randomly selecting filters in the same proportion, FAIG's performance is closer to that of updating filters. The results of all parameters, and their quantitative results are as follows: Figure 7 As shown.

[0141] in, Figure 7 This is one of the embodiments of this application. The following diagram illustrates the comparison of results obtained from filters based on different methods. Figure 7 As can be seen, this comparison examined the results of different update methods when the input images were blurred and noisy, specifically including: FAIG, IG, and... There are several methods, including randomization. The model performance is mainly measured by the peak signal-to-noise ratio (PSNR), which is measured in decibels (dB). The higher the value, the better the model performance.

[0142] Taking a blurred image as the input as an example, the upper bound is 29.203, where ±0.021 represents the variance obtained through three experiments. When retraining 1% of the neurons for deblurring, the results for the four methods listed above are 28.047 (±0.023), 26.474 (±0.295), 26.758 (±0.103), and 27.028 (±0.154). Clearly, FAIG performs best. When the input is a noisy image and 1% of the neurons are retrained for denoising, the results for the four methods listed above are 25.793 (±0.021), 25.465 (±0.014), 25.418 (±0.009), and 25.526 (±0.010). Clearly, FAIG is the most effective.

[0143] In one alternative implementation, the first degradation processing model can be trained on a training sample dataset containing blurred images, noisy images, and bicubic downsampled images. The images in the training sample dataset can also contain various degradation types, such as noise in blurred images. This is not specifically limited here. The trained first degradation processing model can handle noise, blur, and bicubic downsampled degradation.

[0144] In another optional implementation, the second degradation processing model can be enhanced with noise reduction and deblurring capabilities based on the above ideas to obtain the first degradation processing model. Specifically, the first degradation processing model is obtained by replacing the network parameters corresponding to the target neurons in the second degradation processing model with the network parameters corresponding to the associated neurons in the sample model. The sample model is a model used to restore at least one type of degradation image, the second degradation processing model is a model used to restore at least one type of degradation image, and the first degradation processing model, the second degradation processing model, and the sample model have the same parameter space and the same number of network parameters.

[0145] For example, the second degradation treatment model is The sample model is a model capable of restoring blurred images. Therefore, the target neurons corresponding to the blur type in the sample model can be determined using the methods described above, and then updated... The corresponding target neuron can be used to obtain a new model with deblurring ability. Furthermore, based on this, the target neuron corresponding to the noise type in the sample model can be determined. By updating the target neuron corresponding to the new model obtained in the previous step, a model with denoising ability can be obtained, namely the first degradation processing model.

[0146] In the above embodiments, the filters found by integrating the gradient of the filter along the linear interpolation path in the parameter space of the initial model and the target model can play a very important role in enhancing the functionality of a specific network.

[0147] It should be noted that, in addition to the embodiments listed above, based on the found filters, degradation of the input image can also be judged, achieving the effect of unsupervised classification of degradation of the input image.

[0148] See Figure 8 As shown, it is a flowchart illustrating a method for determining the degradation type of an image to be processed according to an embodiment of this application, specifically including the following steps:

[0149] S81: The terminal device acquires multiple reference images, each of which contains a degradation type;

[0150] S82: The terminal device obtains the reference neurons corresponding to each type of degradation based on multiple reference images;

[0151] S83: The terminal device determines the target degradation type of the image to be processed based on the overlap ratio between the target neuron and the reference neuron.

[0152] For example, based on the filters corresponding to different degradations obtained from the Set14 dataset and the filters of the image to be processed, after calculating the intersection of these filters, the degradation type of the image to be processed can be determined in an unsupervised manner by designing a threshold.

[0153] The Set14 dataset is a commonly used dataset in the field of image processing. Blurred images from the Set14 dataset can be used as input images for the initial degradation processing model, and a set of filters-1 targeting blur can be obtained using the FAIG method listed above. Similarly, noisy images from the Set14 dataset can be used as input images for the initial degradation processing model, and a set of filters-2 targeting noise can be obtained using the FAIG method listed above. Additionally, the image to be processed can be used as input images for the initial degradation processing model, and a set of filters-3 targeting blur can be obtained using the FAIG method listed above. By comparing the intersection of filters-1 and filters-3, and the intersection of filters-2 and filters-3, the specific degradation type of the image to be processed can be determined: blur, noise, or a combination of blur and noise. For example, if the threshold is 70%, the intersection of filters-1 and filters-3 accounts for 80%, and the intersection of filters-2 and filters-3 accounts for 60%, the target degradation type can be determined to be blurry; if the intersection of filters-1 and filters-3 accounts for 20%, and the intersection of filters-2 and filters-3 accounts for 90%, the target degradation type can be determined to be noise.

[0154] In the above implementation, based on the filters identified by FAIG, the degradation type of the input image can be determined in an unsupervised manner.

[0155] An optional implementation method is to carry out step S82 through the following steps, such as... Figure 9 The diagram shown is a flowchart illustrating a method for determining a reference neuron according to an embodiment of this application, comprising the following steps:

[0156] S901: The terminal device inputs multiple reference images into the initial degradation processing model and obtains each reference output image of the initial degradation processing model;

[0157] S902: The terminal device performs gradient integration on the network parameters corresponding to each neuron in the initial degradation processing model based on each reference output image to obtain the influence factor of each neuron on various degradation types.

[0158] S903: The terminal device selects reference neurons corresponding to each type of degradation from each neuron based on the influence factors of each neuron on various degradation types.

[0159] For example, the reference images include noisy images and blurred images. Each reference image is input into the initial degradation processing model, and gradient integration is performed based on Formula 2 listed above to determine a set of reference neurons corresponding to the noise type and a set of reference neurons corresponding to the blur type.

[0160] The following is combined Figure 10 A brief overview of the effects of the FAIG method in the embodiments of this application is provided below:

[0161] This application compares the model's performance with other methods. For ambiguous (noisy) inputs, this application masks the corresponding deblurred (denoising) neurons. Larger values ​​indicate a greater performance degradation. Tested on Set14.

[0162] See Figure 10 The diagram illustrates a comparison of the effects of different methods and mask ratios in an embodiment of this application, demonstrating that the FAIG method of this application can find more important filters compared to other methods. Specifically, this includes: FAIG, IG, and... There are several methods, including random. The input images are also divided into two categories: blurred images and noisy images. Figure 10 The results demonstrate that filters found using FAIG exhibit significantly better specificity for different types of degradation compared to filters obtained through random selection. Figure 10 The number before the ± sign in the model indicates the model performance; the larger the value, the better the performance. The number before the ± sign indicates the variance obtained from three experiments. For example, FAIG:6.68±0.63, where 6.68 represents the model performance and 0.63 represents the variance of the three experimental results.

[0163] The following is combined Figures 11A-11D The process of analyzing the proportion of target neurons in the embodiments of this application is briefly summarized as follows:

[0164] This application allows for the measurement of the target model. The difference between the output of the target processing model and the output of the target processing model, such as the mean squared error (MSE), is used to quantify the contribution of the target neuron to the network function. Figures 11A-11D These are schematic diagrams illustrating the relationship between the proportion of four target neurons and changes in model performance.

[0165] See Figure 11A and Figure 11B The figure shows the results of deblurring neurons discovered based on different proportion masks when the input image is a blurred image. The horizontal axis represents the neuron replacement ratio, and the vertical axis represents the model performance. The dashed line represents the performance degradation curve corresponding to the determination and masking of deblurring neurons based on the FAIG method in this application, and the solid line represents the performance degradation curve corresponding to random masking.

[0166] Specifically, by Figure 11A It can be seen that when using the FAIG method in this application to determine and mask deblurred neurons, the deblurring performance decreases significantly when the target neuron accounts for a small proportion of all neurons, and its contribution becomes more significant. Similarly, from Figure 11B It is evident that when using the FAIG method described in this application to determine and mask deblurred neurons, the denoising performance does not significantly decrease when the target neurons constitute a small proportion of all neurons. That is, when the target neurons constitute a small proportion of all neurons, for example, when only the first 1% of deblurred neurons are masked, the model can lose its deblurring ability while maintaining its denoising ability.

[0167] See Figure 11C and Figure 11D The diagram shows the results of identifying denoising neurons based on different proportion masks when the input image is a blurred image. Similarly, the horizontal axis represents the neuron replacement ratio, and the vertical axis represents the model performance. The dashed line represents the performance degradation curve corresponding to the determination and masking of denoising neurons based on the FAIG method in this application, and the solid line represents the performance degradation curve corresponding to random masking.

[0168] Specifically, by Figure 11C It can be seen that when using the FAIG method in this application to determine and mask denoising neurons, the denoising performance does not decrease significantly when the target neuron accounts for a small proportion of all neurons. Similarly, from Figure 11D It is evident that when using the FAIG method described in this application to determine and mask denoising neurons, the denoising performance degrades significantly when the target neuron accounts for a small proportion of all neurons, while its contribution becomes more pronounced. That is, when the target neuron accounts for a small proportion of all neurons, for example, when only the first 1% of denoising neurons are masked, the model can lose its denoising ability while retaining its deblurring ability.

[0169] See Figure 12 The diagram shown is a flowchart illustrating a complete image processing method according to an embodiment of this application. The specific implementation flow of this method is as follows:

[0170] Step S1201: The terminal device inputs the image to be processed into the initial degradation processing model and obtains the initial output image of the initial degradation processing model;

[0171] Step S1202: The terminal device constructs the loss function corresponding to the initial degradation processing model based on the difference value between the initial output image and the sample image;

[0172] Step S1203: Based on the loss function, the terminal device performs gradient integration on the network parameters corresponding to the neurons of the initial degradation processing model according to the preset integration path to obtain the influence factor of each neuron in the initial degradation processing model on the fuzzy type.

[0173] Step S1204: The terminal device sorts each neuron according to its corresponding influencing factor;

[0174] Step S1205: The terminal device selects the top 1% of neurons in the ranking results as target neurons.

[0175] Step S1206: The terminal device performs linear interpolation on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model to obtain the target degradation processing model.

[0176] Step S1207: The terminal device performs image processing on the image to be processed based on the target degradation processing model to obtain the target output image.

[0177] It should be noted that the above-described embodiments are merely one implementation method in this application. This embodiment, by performing gradient integration on the parameter space of the blind super-resolution network, can find filters that are more important for specific degradations compared to other methods. FAIG is an effective tool for understanding the intrinsic mechanism of blind super-resolution networks. The filters found based on FAIG can be applied not only to… Figure 12 The adjustments to model recovery capabilities listed above can also be used in various fields such as model pruning, quantization, and judgment of input image degradation, without specific limitations here. Furthermore, based on the FAIG framework mentioned above and the important filters found through FAIG analysis, a clearer understanding of the intrinsic mechanisms of blind super-resolution networks in related technologies can be achieved.

[0178] Based on the same inventive concept, embodiments of this application also provide an image processing apparatus. For example... Figure 13 As shown, it is a structural schematic diagram of an image processing device 1300 according to an embodiment of this application, which may include:

[0179] The first processing unit 1301 is used to input the image to be processed into the initial degradation processing model and obtain the initial output image of the initial degradation processing model. The initial degradation processing model is generated based on the first degradation processing model and the second degradation processing model. The first degradation processing model is a model for restoring degraded images of at least two degradation types, and the second degradation processing model is a model for restoring degraded images of at least one degradation type.

[0180] Integration unit 1302 is used to perform gradient integration on the network parameters corresponding to each neuron in the initial degradation processing model based on the initial output image, so as to obtain the influence factor of each neuron on the target degradation type.

[0181] The screening unit 1303 is used to select the target neurons corresponding to the target degradation type from each neuron based on the obtained influencing factors, and update the target neurons in the first degradation processing model to obtain the corresponding target degradation processing model.

[0182] The second processing unit 1304 is used to perform image processing on the image to be processed based on the target degradation processing model to obtain the target output image.

[0183] Optionally, the filtering unit 1303 is specifically used for:

[0184] Based on the difference between the initial output image and the sample image, a loss function corresponding to the initial degradation processing model is constructed, where the sample image is the super-resolution image corresponding to the image to be processed.

[0185] Based on the loss function, the gradient integral of the network parameters corresponding to the neurons of the initial degradation processing model is performed according to the preset integral path to obtain the influence factor of each neuron in the initial degradation processing model on the target degradation type.

[0186] The preset integration path is generated by linear interpolation of the network parameters of the second degradation model and the first degradation model. The second degradation model and the first degradation model have the same number of network parameters.

[0187] Optionally, when performing gradient integration on the network parameters corresponding to each neuron, the same integration weight is assigned to each sub-integration interval of the preset integration path; or,

[0188] When performing gradient integration on the network parameters corresponding to each neuron, different integration weights are assigned to different sub-integration intervals in the preset integration path based on preset integration rules.

[0189] Optionally, the filtering unit 1303 is specifically used for:

[0190] The neurons are sorted according to their respective influencing factors;

[0191] Neurons whose sorting results fall within a preset order range are selected as target neurons.

[0192] Optionally, the second processing unit 1304 is specifically used to perform at least one of the following operations:

[0193] Replace the network parameters corresponding to the target neuron in the first degradation processing model with the network parameters corresponding to the associated neuron in the second degradation processing model to obtain the target degradation processing model;

[0194] For the network parameters corresponding to the target neuron in the first degradation processing model, perform at least one of pruning and quantization processing to obtain the target degradation processing model;

[0195] Linear interpolation is performed on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model to obtain the target degradation processing model.

[0196] Optionally, when performing linear interpolation on the network parameters corresponding to the target neurons in the first degradation treatment model and the network parameters corresponding to the associated neurons in the second degradation treatment model, the second degradation treatment model and the first degradation treatment model have different parameter combination ratios. The parameter combination ratios are used to balance the regulatory ability of the target degradation treatment model to handle different degradation types.

[0197] Optionally, the device also includes:

[0198] Model generation unit 1305 is used to obtain the first degradation model in the following manner:

[0199] The network parameters corresponding to the target neurons in the second degradation processing model are replaced with the network parameters corresponding to the associated neurons in the sample model to obtain the first degradation processing model. The sample model is a model used at least to restore the degradation image of the target degradation type. The number of network parameters in the first degradation processing model, the second degradation processing model, and the sample model is the same.

[0200] Optionally, the second degradation processing model is a model for processing degraded images of the downsampled degradation type, wherein at least two degradation types include the downsampled degradation type.

[0201] Optionally, the device also includes:

[0202] Classification unit 1306 is used to acquire multiple reference images, each of which contains a degradation type;

[0203] Based on multiple reference images, the reference neurons corresponding to each type of degradation are obtained;

[0204] The target degradation type of the image to be processed is determined based on the overlap ratio between the target neuron and the reference neuron.

[0205] Optionally, classification unit 1306 is specifically used for:

[0206] Multiple reference images are input into the initial degradation processing model to obtain the reference output images of the initial degradation processing model.

[0207] Based on each reference output image, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on various degradation types.

[0208] Based on the influence factors of each neuron on various types of degeneration, reference neurons corresponding to each type of degeneration are selected from each neuron.

[0209] In the above embodiments, since this application obtains the influence factor of each neuron on the target degradation type by performing gradient integration on the network parameters corresponding to each neuron in the initial degradation processing model, the target neurons selected based on this are more targeted to the target degradation type, and can find the corresponding important neurons according to different degradations. This application can change the specific function of the network by only changing the selected target neurons; without introducing new parameters, the network parameters corresponding to the target neurons in the first degradation processing model are updated to obtain a target degradation processing model that can only handle specific degradation types, thereby achieving more targeted specific degradation processing of the image to be processed.

[0210] For ease of description, the above sections are divided into functional units (or modules) and described separately. Of course, in implementing this application, the functions of each unit (or module) can be implemented in one or more software or hardware components.

[0211] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0212] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be a server, such as... Figure 1 The terminal device 110 is shown. In this embodiment, the electronic device can be structured as follows: Figure 14 As shown, it includes a memory 1401, a communication module 1403, and one or more processors 1402.

[0213] The memory 1401 is used to store computer programs executed by the processor 1402. The memory 1401 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0214] Memory 1401 may be volatile memory, such as random-access memory (RAM); memory 1401 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1401 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1401 may be a combination of the above-described memories.

[0215] The processor 1402 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1402 is used to implement the above-described image processing method when it calls a computer program stored in the memory 1401.

[0216] The communication module 1403 is used to communicate with terminal devices and other servers.

[0217] This application embodiment does not limit the specific connection medium between the memory 1401, communication module 1403, and processor 1402. This application embodiment... Figure 14 The memory 1401 and the processor 1402 are connected via a bus 1404, and the bus 1404 is in Figure 14 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1404 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 14 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0218] The memory 1401 stores a computer storage medium, which stores computer-executable instructions for implementing the image processing method of this application embodiment. The processor 1402 is used to execute the above-described image processing method, such as... Figure 2 As shown.

[0219] In another embodiment, the electronic device may also be other electronic devices, such as... Figure 1 The terminal device 110 is shown. In this embodiment, the electronic device can be structured as follows: Figure 15 As shown, it includes components such as: communication component 1510, memory 1520, display unit 1530, camera 1540, sensor 1550, audio circuit 1560, Bluetooth module 1570, processor 1580, etc.

[0220] The communication component 1510 is used to communicate with the server. In some embodiments, it may include a Circuit-Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology. Electronic devices can use the WiFi module to help users send and receive information.

[0221] The memory 1520 can be used to store software programs and data. The processor 1580 executes various functions of the terminal device 110 and performs data processing by running the software programs or data stored in the memory 1520. The memory 1520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 1520 stores an operating system that enables the terminal device 110 to run. In this application, the memory 1520 may store the operating system and various applications, and may also store code that executes the image processing method of the embodiments of this application.

[0222] The display unit 1530 can also be used to display information input by the user or information provided to the user, as well as various menus of the terminal device 110, in a graphical user interface (GUI). Specifically, the display unit 1530 may include a display screen 1532 disposed on the front of the terminal device 110. The display screen 1532 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 1530 can be used to display various images in the embodiments of this application, such as images to be processed and output images.

[0223] The display unit 1530 can also be used to receive input digital or character information and generate signal inputs related to user settings and function control of the terminal device 110. Specifically, the display unit 1530 may include a touch screen 1531 disposed on the front of the terminal device 110, which can collect touch operations of the user on or near it, such as clicking a button, dragging a scroll box, etc.

[0224] The touchscreen 1531 can be placed over the display screen 1532, or the touchscreen 1531 and the display screen 1532 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen. In this application, the display unit 1530 can display the application program and the corresponding operation steps.

[0225] Camera 1540 can be used to capture still images, which users can then post comments on via the application. There can be one or multiple cameras 1540. An object is projected onto a photosensitive element through a lens, generating an optical image. This photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to the processor 1580 for conversion into a digital image signal.

[0226] The terminal device may also include at least one sensor 1550, such as an accelerometer 1551, a proximity sensor 1552, a fingerprint sensor 1553, and a temperature sensor 1554. The terminal device may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, and motion sensor.

[0227] Audio circuitry 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and terminal device 110. Audio circuitry 1560 converts received audio data into electrical signals, which are then transmitted to speaker 1561, where they are converted into sound signals for output. Terminal device 110 may also be equipped with volume buttons for adjusting the volume of the sound signal. On the other hand, microphone 1562 converts collected sound signals into electrical signals, which are received by audio circuitry 1560, converted into audio data, and then output to communication component 1510 for transmission to, for example, another terminal device 110, or to memory 1520 for further processing.

[0228] The Bluetooth module 1570 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 1570, thereby exchanging data.

[0229] The processor 1580 is the control center of the terminal device, connecting various parts of the terminal through various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 1520 and calling data stored in the memory 1520. In some embodiments, the processor 1580 may include one or more processing units; the processor 1580 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 1580. In this application, the processor 1580 can run the operating system, applications, user interface display and touch response, and the image processing method of the embodiments of this application. Furthermore, the processor 1580 is coupled to the display unit 1530.

[0230] In some possible implementations, various aspects of the image processing methods provided in this application can also be implemented as a program product, which includes program code. When the program product is run on a computer device, the program code causes the computer device to perform the steps of the image processing methods according to the various exemplary embodiments of this application described above. For example, the computer device can perform actions such as... Figure 2 The steps are shown in the figure.

[0231] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0232] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this application, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0233] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0234] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0235] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0236] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0237] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0238] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0239] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0240] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. An image processing method, characterized in that, The method includes: The image to be processed is input into the initial degradation processing model to obtain the initial output image of the initial degradation processing model. The initial degradation processing model is generated based on the first degradation processing model and the second degradation processing model. The first degradation processing model is a model for restoring degraded images of at least two degradation types, and the second degradation processing model is a model for restoring degraded images of at least one degradation type. Based on the initial output image, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on the target degradation type. Based on the obtained influencing factors, target neurons corresponding to the target degradation type are selected from the neurons, and the target neurons in the first degradation processing model are updated to obtain the corresponding target degradation processing model. Based on the target degradation processing model, image processing is performed on the image to be processed to obtain the target output image.

2. The method as described in claim 1, characterized in that, Based on the initial output image, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on the target degradation type, including: Based on the difference between the initial output image and the sample image, a loss function corresponding to the initial degradation processing model is constructed, wherein the sample image is the super-resolution image corresponding to the image to be processed. Based on the loss function, the network parameters corresponding to the neurons of the initial degradation processing model are integrated by gradient according to the preset integration path to obtain the influence factor of each neuron in the initial degradation processing model on the target degradation type. The preset integration path is generated by linear interpolation of the network parameters of the second degradation model and the first degradation model, and the second degradation model and the first degradation model have the same number of network parameters.

3. The method as described in claim 2, characterized in that, When performing gradient integration on the network parameters corresponding to each neuron, the same integration weight is assigned to each sub-integration interval of the preset integration path. or, When performing gradient integration on the network parameters corresponding to each neuron, different integration weights are assigned to different sub-integration intervals in the preset integration path based on preset integration rules.

4. The method as described in claim 1, characterized in that, The step of selecting the target neuron corresponding to the target degradation type from the various neurons based on the obtained influencing factors includes: The neurons are sorted according to their respective influencing factors; The neurons whose sorting results are within a preset order range are selected as the target neurons.

5. The method as described in claim 1, characterized in that, Updating the target neuron in the first degradation processing model to obtain the corresponding target degradation processing model includes any one of the following operations: Replace the network parameters corresponding to the target neuron in the first degradation processing model with the network parameters corresponding to the associated neuron in the second degradation processing model to obtain the target degradation processing model; The network parameters corresponding to the target neurons in the first degradation processing model are subjected to at least one of pruning and quantization processing to obtain the target degradation processing model; Linear interpolation is performed on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model to obtain the target degradation processing model.

6. The method as described in claim 5, characterized in that, When performing linear interpolation on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model, the second degradation processing model and the first degradation processing model have different parameter combination ratios. The parameter combination ratios are used to balance the regulatory ability of the target degradation processing model to handle different degradation types.

7. The method as described in claim 1, characterized in that, The first degradation treatment model was obtained in the following way: The network parameters corresponding to the target neuron in the second degradation processing model are replaced with the network parameters corresponding to the associated neurons in the sample model to obtain the first degradation processing model. The sample model is a model for restoring at least the degradation image of the target degradation type, and the second degradation processing model is a model for restoring degradation images of at least one degradation type. The first degradation processing model, the second degradation processing model, and the sample model have the same number of network parameters.

8. The method as described in claim 1, characterized in that, The second degradation processing model is a model for processing degraded images of the downsampled degradation type, wherein the at least two degradation types include the downsampled degradation type.

9. The method according to any one of claims 1 to 8, characterized in that, The method further includes: Acquire multiple reference images, each containing a degradation type; Based on the multiple reference images, the reference neurons corresponding to each type of degradation are obtained; The target degradation type corresponding to the image to be processed is determined based on the overlap ratio between the target neuron and the reference neuron.

10. The method as described in claim 9, characterized in that, The step of obtaining reference neurons corresponding to each type of degradation based on the plurality of reference images includes: The plurality of reference images are respectively input into the initial degradation processing model to obtain each reference output image of the initial degradation processing model; Based on each of the reference output images, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on each of the various degradation types. Based on the influence factors of each neuron on the various degeneration types, reference neurons corresponding to each degeneration type are selected from the various neurons.

11. An image processing apparatus, characterized in that, include: The first processing unit is used to input the image to be processed into an initial degradation processing model and obtain an initial output image of the initial degradation processing model. The initial degradation processing model is generated based on a first degradation processing model and a second degradation processing model. The first degradation processing model is a model for restoring degraded images of at least two degradation types, and the second degradation processing model is a model for restoring degraded images of at least one degradation type. An integration unit is used to perform gradient integration on the network parameters corresponding to each neuron in the initial degradation processing model based on the initial output image, so as to obtain the influence factor of each neuron on the target degradation type. A screening unit is used to select the target neuron corresponding to the target degradation type from the neurons based on the obtained influencing factors, and update the target neuron in the first degradation processing model to obtain the corresponding target degradation processing model. The second processing unit is used to perform image processing on the image to be processed based on the target degradation processing model to obtain the target output image.

12. The apparatus as claimed in claim 11, characterized in that, The filtering unit is specifically used for: Based on the difference between the initial output image and the sample image, a loss function corresponding to the initial degradation processing model is constructed, wherein the sample image is the super-resolution image corresponding to the image to be processed. Based on the loss function, the network parameters corresponding to the neurons of the initial degradation processing model are integrated by gradient according to the preset integration path to obtain the influence factor of each neuron in the initial degradation processing model on the target degradation type. The preset integration path is generated by linear interpolation of the network parameters of the second degradation model and the first degradation model, and the second degradation model and the first degradation model have the same number of network parameters.

13. The apparatus as claimed in claim 12, characterized in that, When performing gradient integration on the network parameters corresponding to each neuron, the same integration weight is assigned to each sub-integration interval of the preset integration path. or, When performing gradient integration on the network parameters corresponding to each neuron, different integration weights are assigned to different sub-integration intervals in the preset integration path based on preset integration rules.

14. The apparatus as claimed in claim 11, characterized in that, The filtering unit is specifically used for: The neurons are sorted according to their respective influencing factors; The neurons whose sorting results are within a preset order range are selected as the target neurons.

15. The apparatus as claimed in claim 11, characterized in that, The second processing unit is specifically used to perform at least one of the following operations: Replace the network parameters corresponding to the target neuron in the first degradation processing model with the network parameters corresponding to the associated neuron in the second degradation processing model to obtain the target degradation processing model; The network parameters corresponding to the target neurons in the first degradation processing model are subjected to at least one of pruning and quantization processing to obtain the target degradation processing model; Linear interpolation is performed on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model to obtain the target degradation processing model.

16. The apparatus as claimed in claim 15, characterized in that, When performing linear interpolation on the network parameters corresponding to the target neuron in the first degradation processing model and the network parameters corresponding to the associated neurons in the second degradation processing model, the second degradation processing model and the first degradation processing model have different parameter combination ratios. The parameter combination ratios are used to balance the regulatory ability of the target degradation processing model to handle different degradation types.

17. The apparatus as claimed in claim 11, characterized in that, The device further includes: The model generation unit is used to obtain the first degradation processing model in the following manner: The network parameters corresponding to the target neuron in the second degradation processing model are replaced with the network parameters corresponding to the associated neurons in the sample model to obtain the first degradation processing model. The sample model is a model for restoring at least the degradation image of the target degradation type, and the second degradation processing model is a model for restoring degradation images of at least one degradation type. The first degradation processing model, the second degradation processing model, and the sample model have the same number of network parameters.

18. The apparatus as claimed in claim 11, characterized in that, The second degradation processing model is a model for processing degraded images of the downsampled degradation type, wherein the at least two degradation types include the downsampled degradation type.

19. The apparatus according to any one of claims 11 to 18, characterized in that, The device further includes: A classification unit is used to acquire multiple reference images, each containing a degradation type; Based on the multiple reference images, the reference neurons corresponding to each type of degradation are obtained; The target degradation type corresponding to the image to be processed is determined based on the overlap ratio between the target neuron and the reference neuron.

20. The apparatus as claimed in claim 19, characterized in that, The classification unit is specifically used for: The plurality of reference images are respectively input into the initial degradation processing model to obtain each reference output image of the initial degradation processing model; Based on each of the reference output images, gradient integration is performed on the network parameters corresponding to each neuron in the initial degradation processing model to obtain the influence factor of each neuron on each of the various degradation types. Based on the influence factors of each neuron on the various degeneration types, reference neurons corresponding to each degeneration type are selected from the various neurons.

21. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 10.

22. A computer-readable storage medium, characterized in that, It includes program code that, when the storage medium is running on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 10.

Images