Image processing method, device, computer equipment and storage medium

Abstract

The present application provides an image processing method, apparatus, computer equipment, and storage medium, belonging to the field of artificial intelligence technology. The method includes: obtaining a first super-resolution model through training based on a sample image and a reference image corresponding to the sample image, wherein the resolution of the reference image is greater than the resolution of the sample image; obtaining conversion relationship data, wherein the conversion relationship data is used to represent the correspondence between multiple first processing modules of the first super-resolution model and at least one second processing module of the second super-resolution model, wherein the number of processing modules in the first super-resolution model is greater than the number of processing modules in the second super-resolution model; based on the conversion relationship data, fusing the parameters of the multiple first processing modules to obtain the parameters of at least one second processing module of the second super-resolution model. The present application maintains the original processing effect while reducing the number of parameters, reduces the amount of calculation, and improves image processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] Super-resolution technology reconstructs a high-resolution image from a low-resolution image. Currently, super-resolution technology has been widely used in fields such as remote sensing imaging, medical image processing, and live video streaming to improve image clarity and enhance the user's visual experience.

[0003] With the continuous development of artificial intelligence, super-resolution technology based on deep learning has also developed rapidly. Deep learning-based super-resolution technology uses supervised training with a large number of low-resolution images and their corresponding high-resolution images to obtain a trained super-resolution model. This trained super-resolution model is then applied to image processing to obtain the corresponding high-resolution image.

[0004] In order to improve the image reconstruction effect, the super-resolution model needs to be trained with a highly complex model structure, which will result in a super-resolution model containing a large number of model parameters. During the application of this super-resolution model, the image is processed based on a large number of model parameters, which results in a large amount of calculation and low processing efficiency. Summary of the Invention

[0005] The embodiments of the present application provide an image processing method, apparatus, computer device, and storage medium that can improve processing efficiency. The technical solution is as follows.

[0006] In one aspect, an image processing method is provided, the method comprising:

[0007] Performing training based on a sample image and a reference image corresponding to the sample image to obtain a first super-resolution model, wherein the resolution of the reference image is greater than the resolution of the sample image;

[0008] Obtaining conversion relationship data, where the conversion relationship data is used to represent a correspondence between a plurality of first processing modules of the first super-resolution model and at least one second processing module of the second super-resolution model, wherein the number of processing modules in the first super-resolution model is greater than the number of processing modules in the second super-resolution model;

[0009] Based on the conversion relationship data, the parameters of the multiple first processing modules are fused to obtain parameters of at least one second processing module of the second super-resolution model.

[0010] In one aspect, an image processing apparatus is provided, the apparatus comprising:

[0011] a model training unit, configured to perform training based on a sample image and a reference image corresponding to the sample image to obtain a first super-resolution model, wherein the resolution of the reference image is greater than the resolution of the sample image;

[0012] a data acquisition unit, configured to acquire conversion relationship data, the conversion relationship data being used to represent a correspondence between a plurality of first processing modules of the first super-resolution model and at least one second processing module of the second super-resolution model, wherein the number of processing modules in the first super-resolution model is greater than the number of processing modules in the second super-resolution model;

[0013] A parameter fusion unit is used to fuse the parameters of the multiple first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module of the second super-resolution model.

[0014] In one possible implementation, the at least one second processing module includes a first target convolution module, and the plurality of first processing modules corresponding to the first target convolution module include a first convolution module and a first feature processing module; the first feature processing module is configured to perform other processing on the image features in addition to the convolution processing;

[0015] The parameters of the first convolution module include a first convolution weight and a first convolution bias, and the parameters of the first feature processing module include a first processing weight and a first processing bias;

[0016] The parameter fusion unit includes:

[0017] a convolution weight determination subunit, configured to determine the product of the first convolution weight and the first processing weight as the convolution weight of the first target convolution module;

[0018] A convolution bias determination subunit is used to determine the convolution bias of the first target convolution module based on at least one of the first convolution weight and the first processing weight, the first processing bias and the first convolution bias.

[0019] In another possible implementation, the convolution bias determination subunit is configured to, when the output of the first convolution module is the input of the first feature processing module, determine the sum of the first processing bias and a first value as the convolution bias of the first target convolution module, where the first value is the product of the first processing weight and the first convolution bias; or

[0020] The convolution bias determination subunit is used to, when the output of the first feature processing module is the input of the first convolution module, sum the first convolution bias and a second value, where the second value is the product of the first convolution weight and the first processing bias.

[0021] In another possible implementation, the first feature processing module includes a pixel normalization module; the pixel normalization module is used to normalize the pixel values ​​of the input image to obtain normalized image features; and the first convolution module is used to perform convolution processing on the normalized image features.

[0022] In another possible implementation, the first feature processing module includes a batch normalization module; the batch normalization module is used to perform batch normalization processing on the image features output by the first convolution module.

[0023] In another possible implementation, the at least one second processing module includes a second target convolution module, and the multiple first processing modules corresponding to the second target convolution module include a first point-by-point addition module and multiple second convolution modules; the outputs of the multiple second convolution modules are inputs of the first point-by-point addition module; the multiple second convolution modules are respectively used to perform convolution processing on the same image feature to obtain multiple convolution-processed image features; the first point-by-point addition module is used to perform point-by-point addition on the multiple convolution-processed image features to obtain a fused image feature;

[0024] The parameter fusion unit is used to:

[0025] Determining the sum of the convolution weights of the plurality of second convolution modules as the convolution weight of the second target convolution module;

[0026] The sum of the convolution offsets of the multiple second convolution modules is determined as the convolution offset of the second target convolution module.

[0027] In another possible implementation, the at least one second processing module includes a third target convolution module, and the multiple first processing modules corresponding to the third target convolution module include a jump connection module, a second point-by-point addition module, and at least one third convolution module; the at least one third convolution module is respectively used to perform convolution processing on the first image feature to obtain at least one second image feature; the output of the jump connection module and the output of the at least one third convolution module are input to the second point-by-point addition module; the second point-by-point addition module is used to perform point-by-point addition on the first image feature and the at least one second image feature to obtain a fused image feature;

[0028] The parameter fusion unit is used to:

[0029] Determining the sum of the parameters of the skip connection module and the convolution weight of the at least one third convolution module as the convolution weight of the third target convolution module, where the parameter of the skip connection module is 1;

[0030] The sum of the convolution offsets of the at least one third convolution module is determined as the convolution offset of the third target convolution module.

[0031] In another possible implementation,

[0032] The at least one second processing module includes a fourth target convolution module, and the plurality of first processing modules corresponding to the fourth target convolution module include a second feature processing module and a fourth convolution module; the fourth convolution module is the last convolution module in the first super-resolution model; the processing module of the first super-resolution model also includes a first pixel reorganization module; the first pixel reorganization module is used to reorganize the image features processed by the fourth convolution module to obtain target image features; the output of the first pixel reorganization module is the input of the second feature processing module; the second feature processing module is used to process the target image features to obtain an image of target resolution;

[0033] The parameters of the fourth convolution module include a second convolution weight and a second convolution bias, and the parameters of the second feature processing module include a second processing weight and a second processing bias;

[0034] The parameter fusion unit is used to:

[0035] Determining a product of the second convolution weight and the second processing weight as a convolution weight of the fourth target convolution module;

[0036] The sum of the second processing bias and a third value is determined as the convolution bias of the fourth target convolution module, and the third value is the product of the second processing weight and the second convolution bias.

[0037] In another possible implementation, the second feature processing module includes a pixel denormalization module; the pixel denormalization module is used to perform denormalization processing on the target image features to obtain an image with the target resolution.

[0038] In another possible implementation, the conversion relationship data is further used to represent a correspondence between multiple third processing modules in the first super-resolution model and multiple fourth processing modules in the second super-resolution model;

[0039] Among them, the multiple third processing modules include an upsampling module, a first pixel reorganization module and a third point-by-point addition module; the upsampling module is used to upsample the image features of the input image to obtain third image features of the target scale; the input of the first pixel reorganization module is the fourth image feature processed by the last layer convolution module of the first super-resolution model, and the first pixel reorganization module is used to reorganize the fourth image feature to obtain the fifth image feature of the target scale; the output of the upsampling module and the output of the first pixel reorganization module are input to the third point-by-point addition module; the third point-by-point addition module is used to add the third image feature and the fifth image feature point by point to obtain the target image feature, which is the feature of the image of the target resolution;

[0040] The multiple fourth processing modules include a broadcast addition module and a second pixel reassembly module, the input of the broadcast addition module is the image features of the input image and the image features processed by the last layer convolution module of the second super-resolution model, and the image features processed by the last layer convolution module of the second super-resolution model include sub-features of multiple channels; the broadcast addition module is used to add the image features of the input image to the sub-features of the multiple channels point by point to obtain superimposed features of the multiple channels; the output of the broadcast addition module is the input of the second pixel reassembly module; the second pixel reassembly module is used to reassemble the superimposed features of the multiple channels to obtain the target image features;

[0041] The device further comprises:

[0042] A parameter determination unit is configured to determine, based on the conversion relationship data, parameters of a first pixel reconstruction module in the plurality of third processing modules as parameters of the second pixel reconstruction module in the plurality of fourth processing modules.

[0043] On the one hand, a computer device is provided, comprising a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image processing method described in any one of the possible implementations above.

[0044] On the one hand, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the image processing method described in any one of the possible implementations above.

[0045] On the one hand, a computer program product or computer program is provided, which includes computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the image processing method described in any one of the possible implementations above.

[0046] The technical solution provided in the embodiment of the present application is based on training of low-resolution images and corresponding high-resolution images to obtain a first super-resolution model, and by converting multiple processing modules of the first super-resolution model into a smaller number of processing modules, a second super-resolution model is obtained, thereby reducing the complexity of the model. Based on the correspondence between the processing modules before and after the conversion, parameter fusion is performed, thereby maintaining the original processing effect while reducing the number of parameters, and thus performing image processing based on a small number of parameters contained in the second super-resolution model, which can reduce the amount of calculation while ensuring high-quality image processing effects and improve image processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0049] Figure 2 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0050] Figure 3 Schematic diagram of the structure of a first super-resolution model provided in an embodiment of the present application;

[0051] Figure 4 is a schematic diagram of a pixel recombination process provided in an embodiment of the present application;

[0052] Figure 5 Schematic diagram of a nearest neighbor upsampling process provided in an embodiment of the present application;

[0053] Figure 6 This is a schematic diagram of the structure of a convolutional network provided in an embodiment of the present application;

[0054] Figure 7 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0055] Figure 8 This is a schematic diagram of the structure of a simplified convolutional network provided in an embodiment of the present application;

[0056] Figure 9 is a schematic diagram of generating target image features provided by an embodiment of the present application;

[0057] Figure 10 is a schematic diagram of generating target image features provided by an embodiment of the present application;

[0058] Figure 11 is a schematic diagram of generating target image features provided by an embodiment of the present application;

[0059] Figure 12 is a schematic structural diagram of a second super-resolution model provided in an embodiment of the present application;

[0060] Figure 13 is a schematic diagram of a super-resolved image provided in an embodiment of the present application;

[0061] Figure 14 is a block diagram of an image processing device provided in an embodiment of the present application;

[0062] Figure 15 This is a block diagram of a terminal provided in an embodiment of the present application;

[0063] Figure 16 This is a block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0065] The terms "first", "second", "third" and "fourth" in the specification, claims and drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, for the terms "at least one", "a plurality" and "any" used in this application, at least one refers to one, two or more, a plurality refers to two or more, and any refers to any one of a plurality. For example, the plurality of second processing modules includes three second processing modules, and any refers to any one of the three second processing modules, which can be the first, the second, or the third.

[0066] Furthermore, the terms "comprise," "comprises," and "comprising," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0067] For ease of understanding, the relevant technologies involved in the embodiments of this application are described below.

[0068] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0069] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0070] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Furthermore, it performs image processing to transform the computer-generated images into images more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0071] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0072] Image resolution is an important indicator of image quality, representing the number of pixels per inch. The higher the image resolution, the more detailed information it contains, the clearer the image, and the more conducive it is to subsequent observation and analysis. Super-resolution technology reconstructs a corresponding high-resolution image (HR) from a low-resolution image (LR). Currently, super-resolution technologies can be divided into three categories: interpolation-based super-resolution, reconstruction-based super-resolution, and learning-based super-resolution. Learning-based super-resolution has made significant progress with the development of artificial intelligence. Learning-based super-resolution, which learns the mapping relationship between low-resolution and high-resolution images by training a super-resolution model, can restore texture details in images without introducing edge jaggedness. It has demonstrated significant application value in scenarios such as cloud video conferencing, live video streaming, video surveillance, medical image processing, and remote sensing image processing.

[0073] The image processing method provided in the embodiments of the present application involves technologies such as image processing and deep learning in the field of artificial intelligence, and is specifically illustrated by the following embodiments.

[0074] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1 , the implementation environment includes a computer device 101 and a terminal 102.

[0075] In some embodiments, the computer device 101 is configured as a server. Optionally, the server is an independent physical server; or, the server is a server cluster or distributed system composed of multiple physical servers; or, the server is a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, the computer device 101 is configured as a terminal. Optionally, the terminal is a desktop computer, a laptop computer, a tablet computer, or a smartphone, but is not limited thereto.

[0076] The computer device 101 and the terminal 102 are directly or indirectly connected via wired or wireless communication, which is not limited in this application. Optionally, the terminal 102 is a smartphone, tablet computer, laptop computer, desktop computer, smart TV, or smart speaker, but is not limited to these. Optionally, the terminal 102 is provided with a client, such as a video client, live broadcast client, social client, image editing client, medical client, or browser client, and the terminal displays the corresponding image or video by running the client.

[0077] Computer device 101 has image processing, model training, and model processing functions, and terminal 102 has a display function. In some embodiments, computer device 101 processes a low-resolution image based on a super-resolution model to obtain a corresponding high-resolution image, and sends the high-resolution image to terminal 102; terminal 102 displays the received high-resolution image. In some embodiments, computer device 101 sends a super-resolution model to terminal 102; terminal 102 processes the low-resolution image based on the received super-resolution model to obtain a corresponding high-resolution image, and displays the high-resolution image.

[0078] Figure 2 This is a flow chart of an image processing method provided by an embodiment of the present application. Figure 2 , the image processing method is briefly described, and the image processing method includes the following steps:

[0079] 201. A computer device performs training based on a sample image and a reference image corresponding to the sample image to obtain a first super-resolution model, wherein the resolution of the reference image is greater than the resolution of the sample image.

[0080] The sample image is a low-resolution image, and the reference image corresponding to the sample image is also the high-resolution image corresponding to the low-resolution image. The sample image corresponds one-to-one with the reference image, and the reference image has richer image details than the sample image. Optionally, the sample image is an image obtained by downsampling the reference image by a computer device. In some embodiments, the first super-resolution model is used to increase the resolution of the image by a factor of n. If the first super-resolution model increases the resolution of the image by a factor of n, the resolution of the reference image is n times the resolution of the sample image, where n is a value greater than 1.

[0081] A sample image and a reference image corresponding to the sample image constitute a sample data pair. The computer device performs training based on multiple sample data pairs, and obtains a first super-resolution model when the training stop condition is reached. Optionally, the training stop condition is that the difference between the image output by the first super-resolution model and the reference image corresponding to the sample image input to the first super-resolution model is less than a difference threshold; or, the training stop condition is that the number of training iterations reaches a preset number. For example, during any training iteration, the computer device inputs a sample image in a sample data pair into the current super-resolution model, processes the sample image based on the parameters of the current super-resolution model, and obtains an output image corresponding to the sample image; based on the difference between the output image and the reference image in the sample data pair, the parameters of the current super-resolution model are updated to reduce the difference between the output image and the reference image. The difference between the output image and the reference image in the sample data pair is represented by a loss function, and the computer device trains the super-resolution model by reducing the loss value obtained based on the loss function. The process is represented by the following formula 1:

[0082] Formula 1:

[0083] Among them, G fat represents the super-resolution model in the training phase, LR represents the low-resolution image, G(LR) represents the high-resolution image output by the super-resolution model in the training phase, HR represents the high-resolution image in the sample data pair to which the low-resolution image belongs, and Loss(G(LR), HR) represents the difference between G(LR) and HR. Indicates that the super-resolution model is trained to minimize the value of Loss(G(LR), HR). The loss function is any loss function used to measure the difference between images, for example, the loss function is a mean square error loss function or a cross entropy loss function, etc., which is not limited in the present embodiment.

[0084] The first super-resolution model is a trained super-resolution model. When any image is input into the first super-resolution model, the image is processed based on the first super-resolution model to obtain the image at a target resolution, where the target resolution is greater than the original resolution of the image. In some embodiments, the first super-resolution model increases the resolution of the image by a factor of n. If the first super-resolution model increases the resolution of the image by a factor of n, the target resolution is n times the original resolution, where n is a number greater than 1.

[0085] 202. A computer device obtains conversion relationship data, where the conversion relationship data is used to represent a correspondence between multiple first processing modules of a first super-resolution model and at least one second processing module of a second super-resolution model, wherein the number of processing modules in the first super-resolution model is greater than the number of processing modules in the second super-resolution model.

[0086] The complexity and number of parameters of the first super-resolution model are positively correlated with the image generation effect. The higher the complexity of the first super-resolution model and the more parameters it contains, the higher the quality of the image output by the first super-resolution model and the higher the degree of restoration compared to the input image. Therefore, as image quality improves, the computational complexity of image processing also increases, and image processing efficiency decreases. To improve image processing efficiency, the multiple processing modules of the first super-resolution model are converted into a smaller number of processing modules to obtain a second super-resolution model.

[0087] 203. The computer device fuses the parameters of the multiple first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module of the second super-resolution model.

[0088] In order to improve image processing efficiency while ensuring that the second super-resolution model and the first super-resolution model have the same processing effect on the image, the computer device fuses the parameters of multiple first processing modules based on the correspondence between the first processing module and the second processing module to obtain the parameters of at least one second processing module.

[0089] The technical solution provided in the embodiment of the present application is based on training of low-resolution images and corresponding high-resolution images to obtain a first super-resolution model, and by converting multiple processing modules of the first super-resolution model into a smaller number of processing modules, a second super-resolution model is obtained, thereby reducing the complexity of the model. Based on the correspondence between the processing modules before and after the conversion, parameter fusion is performed, thereby maintaining the original processing effect while reducing the number of parameters, and thus performing image processing based on a small number of parameters contained in the second super-resolution model, which can reduce the amount of calculation while ensuring high-quality image processing effects and improve image processing efficiency.

[0090] Figure 3This is a schematic diagram of the structure of a first super-resolution model provided in an embodiment of the present application, see Figure 3 The first super-resolution model includes at least one convolutional network and a first pixel reassembly module 301. The convolutional network includes a convolution module and an activation module. The convolution module is configured to perform convolution processing on the input image features to obtain convolutional image features. The activation module receives the convolutional image features as input, processes the convolutional image features to obtain activated image features, and uses the activated image features as the output of the convolutional network, thereby introducing nonlinear characteristics into the model. If the at least one convolutional network is a single convolutional network, the output of the convolutional network serves as the input to the first pixel reassembly module 301, which is configured to reassembly the image features output by the convolutional network to obtain reassembly image features. If the at least one convolutional network is a plurality of convolutional networks, for any two convolutional networks in the plurality of convolutional networks, the output of one convolutional network serves as the input to the other convolutional network; and the output of the last convolutional network 302 in the plurality of convolutional networks serves as the input to the first pixel reassembly module 301. The first pixel reassembly module 301 is used to reassemble the image features output by the last convolutional network 302 to obtain reassembled image features.

[0091] In some embodiments, the input of the first convolutional network 303 in the at least one convolutional network is a pixel value of an image, and the pixel value of the image includes pixel values ​​of multiple pixel points of the image.

[0092] In some embodiments, the input of the first convolutional network 303 in the at least one convolutional network is image features obtained by preprocessing the pixel values ​​of the image. In some embodiments, to reduce the difficulty of model training, the input of the first convolutional network 303 is normalized image features. Accordingly, the first super-resolution model further includes a pixel normalization module 304, which is used to normalize the pixel values ​​of the input image to obtain normalized image features. The output of the pixel normalization module 304 is the input of the first convolutional network 303.

[0093] The pixel values ​​of the image are in the range of [0, 255]. In some embodiments, the pixel normalization module 304 is used to normalize the pixel values ​​of the image to obtain image features in the range of [-0.5, 0.5]. Optionally, the process of the pixel normalization module 304 normalizing the pixel values ​​of the image is represented as h norm =x÷255-0.5; where x represents the pixel value of the input image, h norm Represents the normalized image features.

[0094] In some embodiments, the pixel normalization module 304 is used to normalize the pixel values ​​of the image to obtain image features with a value range of [0, 1]. Optionally, the process of the pixel normalization module 304 normalizing the pixel values ​​of the image is represented as h norm =x÷255; where x represents the pixel value of the input image, h norm Represents the normalized image features.

[0095] It should be noted that for each of the multiple pixels in the image, the pixel normalization module 304 performs normalization processing on the pixel values ​​of each pixel to obtain a normalized numerical value corresponding to each pixel. Thus, the normalized numerical values ​​corresponding to the multiple pixels constitute the normalized image features corresponding to the image. Furthermore, the process of normalizing the pixel values ​​of the image by the pixel normalization module 304 can also be understood as the process of performing an affine transformation on the image. Through normalization processing, the pixel normalization module 304 maps the image from the interval [0, 255] to the normalized interval, converting the image from being represented by pixel values ​​to being represented by normalized image features, thereby achieving an affine transformation.

[0096] In some embodiments, the first super-resolution model pre-processes the pixel values ​​of the input image to obtain image features. Accordingly, after a series of processing is performed on the image features to obtain the final image features, the obtained image features are further subjected to inverse processing corresponding to the pre-processing to obtain the pixel values ​​of the final output image. In some embodiments, the first super-resolution model further includes a pixel normalization module 304, and accordingly, the first super-resolution model further includes a pixel denormalization module 305, which is used to perform denormalization processing on the image features to obtain an image of the target resolution.

[0097] In some embodiments, the process of normalizing the pixel values ​​of the image by the pixel normalization module 304 is represented as h norm =x÷255-0.5; where x represents the pixel value of the input image, h norm Represents the normalized image features. Accordingly, the process of pixel denormalization module 305 performing denormalization processing on the input image features is represented as y=255×h out +127.5; where h out Represents the feature value of the input image feature, h out The value range of is [-0.5, 0.5], y represents the denormalized pixel value, and the value range of y is [0, 255]. The pixel denormalization module 305 performs denormalization processing on the feature values ​​of the input image features to obtain denormalized pixel values, so that the denormalized pixel values ​​constitute the image of the target resolution.

[0098] In some embodiments, the process of normalizing the pixel values ​​of the image by the pixel normalization module 304 is represented as h norm =x÷255; where x represents the pixel value of the input image, h norm Represents the normalized image features. Accordingly, the process of pixel denormalization module 305 performing denormalization processing on the input image features is represented as y=255×h out ; Among them, h out represents the input image features, h out The value range of is [0, 1], y represents the denormalized pixel value, and the value range of y is [0, 255].

[0099] In some embodiments, the first super-resolution model further includes an upsampling module 306 and a third point-by-point addition module 307. The upsampling module 306 is used to upsample the image features of the input image to obtain a third image feature of the target scale. The input of the first pixel reorganization module is the output of the last convolutional network of the first super-resolution model, that is, the fourth image feature after processing by the last convolutional module of the first super-resolution model; the first pixel reorganization module 301 is used to reorganize the fourth image feature to obtain a fifth image feature of the target scale. The output of the upsampling module 306 and the output of the first pixel reorganization module 301 are inputs to the third point-by-point addition module 307, and the third point-by-point addition module 307 is used to perform point-by-point addition on the third image feature and the fifth image feature to obtain a target image feature, which is a feature of the image of the target resolution. Among them, point-by-point addition refers to the addition of the feature values ​​of pixel points with the same relative position. For example, the point-by-point addition of the third image feature and the fifth image feature refers to the addition of the feature value of the pixel point in the first row and first column of the third image feature and the feature value of the pixel point in the first row and first column of the fifth image feature; the feature value of the pixel point in the first row and second column of the third image feature and the feature value of the pixel point in the first row and second column of the fifth image feature, and so on.

[0100] The target scale is the desired scale of the target image features. The target scale can be expressed as H×W. Image features at the target scale include H×W feature values. Both H and W can be flexibly configured as positive integers, such as 4×4 or 6×6. The scale of the image features output by the upsampling module 306 is the same as the scale of the image features output by the first pixel reconstruction module 301, both being the target scale.

[0101] It should be noted that the pixel reassembly module is built on the pixel reassembly (PixelShuffle) technology. By reassembling the eigenvalues ​​of the channel dimension into the spatial dimension, the image features are rearranged, thereby realizing the conversion of features in the depth dimension to features in the spatial dimension. If the image feature before pixel reassembly is represented as a tensor with a width of W, a height of H, and a depth of C×r×r, that is, a multidimensional array with a width of W, a height of H, and a depth of C×r×r, then the pixel reassembly module processes the image feature to obtain an image feature with a width of W×r, a height of H×r, and a depth of C, thereby magnifying the image feature by a factor of r in terms of spatial scale.

[0102] Figure 4 This is a schematic diagram of a pixel reorganization process provided in an embodiment of the present application, see Figure 4 , the image feature 401 before pixel reorganization is an image feature with a dimension of 2×2×4, and the image feature includes sub-features of 4 channels, each sub-feature has a width of 2 and a height of 2. Among them, the 4 eigenvalues ​​included in the sub-feature 4011 of the first channel are respectively expressed as 1-1, 1-2, 1-3 and 1-4, the 4 eigenvalues ​​included in the sub-feature 4012 of the second channel are respectively expressed as 2-1, 2-2, 2-3 and 2-4, the 4 eigenvalues ​​included in the sub-feature 4013 of the third channel are respectively expressed as 3-1, 3-2, 3-3 and 3-4, and the 4 eigenvalues ​​included in the sub-feature 4014 of the fourth channel are respectively expressed as 4-1, 4-2, 4-3 and 4-4. After the pixel reorganization module reorganizes the image feature 401, the image feature 402 after pixel reorganization is obtained. It should be noted that, Figure 4 In the figure, the eigenvalues ​​of the four channels are reorganized into the spatial dimension in a clockwise order as an illustration. Of course, the eigenvalues ​​of the four channels can also be reorganized into the spatial dimension in other orders, and the embodiment of the present application does not limit this.

[0103] In some embodiments, the upsampling module 306 is a nearest neighbor upsampling module that upsamples image features based on a nearest neighbor interpolation method, that is, assigns the feature value of the feature point closest to the newly added feature point to the newly added feature point. Figure 5 This is a schematic diagram of a neighbor upsampling process provided in an embodiment of the present application, see Figure 5, the image feature 501 before upsampling is an image feature with a scale of 2×2, and the four feature values ​​included in the image feature 501 are respectively represented as 1, 2, 3, and 4. By upsampling the image feature 501, an image feature 502 with a scale of 4×4 is obtained. By adding the third image feature obtained by upsampling and the fifth image feature obtained by pixel reorganization point by point, the target image feature is obtained, and the target image feature is used as the final output feature. During the training process, the model learns the difference between the high-resolution image and the low-resolution image obtained based on redundant replication, which helps the model learn more detailed feature differences and improve the image reconstruction effect. It should be noted that the upsampling module 306 can also be implemented based on other upsampling methods. For example, the upsampling module 306 upsamples the image features based on the bilinear interpolation method; or, the upsampling module 306 upsamples the image features based on the bicubic interpolation method. The embodiments of the present application do not limit this.

[0104] In some embodiments, the process of performing convolution processing on the input image features by the convolution module in the convolutional network is represented as W d1×d2 *h in +b d1×d2 ;h in Represents the input image feature, which includes multiple feature values; W d1×d2 Represents the convolution weight of the convolution module, which is expressed as a convolution kernel group of size o×i×d1×d2, d1×d2 means that each convolution kernel in the convolution kernel group includes d1×d2 values, d1 and d2 are both positive integers; i represents the number of channels of the input image feature, i is a positive integer; o represents the number of channels of the image feature after processing based on the convolution kernel group, o is a positive integer; W d1×d2 *h in Indicates that the input image features are based on the convolution kernel group W d1×d2 Image features obtained by convolution processing; b d1×d2 Represents the convolution bias of the convolution module; W d1×d2 *h in +b d1×d2 Indicates that the convolution kernel group W will be based on d1×d2 Image features obtained by convolution processing and convolution bias b d1×d2 The output features of the convolution module are obtained by superposition.

[0105] In some embodiments, the activation module in the convolutional network is represented by an activation function. Optionally, the activation function is a ReLu (Rectified Linear Unit, linear rectification function), a Sigmoid (S-shaped growth curve) function, or other activation functions, which is not limited in the embodiments of the present application. In some embodiments, the activation function is represented by max(0, h), where h represents the image feature of the input activation module, and max(0, h) represents that the negative eigenvalues ​​in the image feature are truncated to 0, that is, for any eigenvalue in the image feature, the maximum value between the eigenvalue and 0 is used as the processed eigenvalue.

[0106] In some embodiments, the convolutional network adopts a more complex structure to improve the image reconstruction effect and obtain a high-resolution image with higher restoration. In some embodiments, the convolutional network includes multiple convolution modules, a first point-by-point addition module, and an activation module. The input of the multiple convolution modules is the same image feature, which is a parallel convolution module. The output of the multiple convolution modules is the input of the first point-by-point addition module. The multiple convolution modules respectively perform convolution processing on the image feature to obtain multiple convolution-processed image features. The first point-by-point addition module is used to perform point-by-point addition on the multiple convolution-processed image features to obtain a fused image feature. The output of the first point-by-point addition module is input to the activation module, which is used to process the fused image feature to obtain an activated image feature as the output feature of the convolutional network. The output features of the multiple convolution modules have the same dimension. For example, the dimensions of the output features of the multiple convolution modules are all 2×2×4. Optionally, the sizes of the convolution kernels in the multiple convolution modules are the same or different, which is not limited in this embodiment of the present application. For example, the multiple convolution modules are 3 convolution modules, and the sizes of the convolution kernels of the 3 convolution modules are all 3×3; or, the sizes of the convolution kernels of the 3 convolution modules are 1×3, 3×3 and 3×1 respectively.

[0107] In some embodiments, the convolutional network further includes a residual block. The residual block is composed of at least one convolution module, a jump connection module, and a second point-by-point addition module. Among them, at least one convolution module and the jump connection module are parallel module structures, and the inputs of at least one convolution module and the jump connection module are the same image features, both of which are first image features. The output of at least one convolution module and the output of the jump connection module are inputs of the second point-by-point addition module. The jump connection module is used to pass the first image feature to the second point-by-point addition module; at least one convolution module is used to perform convolution processing on the first image feature, obtain at least one second image feature, and pass the at least one image feature to the second point-by-point addition module; the second point-by-point addition module is used to perform point-by-point addition on the first image feature and the at least one second image feature to obtain a fused image feature. The output of the second point-by-point addition module is the input of the activation module, and the activation module is used to process the fused image feature to obtain an activated image feature as the output feature of the convolutional network.

[0108] In some embodiments, the convolutional network further includes a batch normalization module, the input of the batch normalization module is the output of the convolution module, and the batch normalization module is used to perform batch normalization processing on the image features output by the convolution module. In some embodiments, the convolutional network includes multiple convolution modules, and accordingly, the convolutional network further includes multiple batch normalization modules, the number of multiple batch normalization modules is consistent with the number of multiple convolution modules, the batch normalization modules correspond one-to-one to the convolution modules, and the output of a convolution module is the input of the batch normalization module corresponding to the convolution module. Accordingly, the output of multiple batch normalization modules is the input of the first point-by-point addition module, and the first point-by-point addition module is used to perform point-by-point addition on the batch-normalized image features.

[0109] In some embodiments, see Figure 6The convolutional network includes multiple convolutional modules, namely, a convolutional module 601 with a convolution kernel size of 1×3, a convolutional module 602 with a convolution kernel size of 3×3, and a convolutional module 603 with a convolution kernel size of 3×1. The convolutional network also includes multiple batch normalization modules, namely, a batch normalization module 604, a batch normalization module 605, and a batch normalization module 606. The input of the batch normalization module 604 is the output of the convolution module 601, the input of the batch normalization module 605 is the output of the convolution module 602, and the input of the batch normalization module 606 is the output of the convolution module 603. The convolutional network also includes a skip connection module 607, a point-by-point addition module 608, and an activation module 609. Among them, the input of the jump connection module 607 is the same as the input of the above-mentioned multiple convolution modules, the output of the jump connection module 607 and the output of the above-mentioned multiple batch normalization modules are the input of the point-by-point addition module 608, and the input of the activation module 609 is the output of the point-by-point addition module 608.

[0110] If the first super-resolution model is applied to image processing, during the image processing process, the parameters of the first super-resolution model are fixed, and the image processing flow of the first super-resolution model is also fixed. In order to improve the image processing efficiency, the complexity of the first super-resolution model can be reduced through operations such as parameter fusion, module replacement and rearrangement, and the parameters of the first super-resolution model can be reduced to obtain a second super-resolution model with a processing effect equivalent to that of the first super-resolution model.

[0111] In some embodiments, the plurality of first processing modules in the first super-resolution model can be combined into at least one second processing module in the second super-resolution model. The plurality of first processing modules in the first super-resolution model and the at least one second processing module in the second super-resolution model include at least one corresponding relationship as shown in Table 1 below.

[0112] Table 1

[0113] The first super-resolution model Second super-resolution model Pixel Normalization Module-Convolution Module A convolutional module Convolution module-Pixel denormalization module A convolutional module Multiple convolutional modules with the same input A convolutional module Residual Block A convolutional module Upsampling module - pixel reconstruction module - point-by-point addition module Broadcast addition module-pixel reconstruction module

[0114] In some embodiments, the input of the first convolutional network in the first super-resolution model is the output of the pixel normalization module. Since the convolution module and the pixel normalization module both perform linear processing on image features, the parameters of the pixel normalization module can be integrated into the parameters of the convolution module of the first convolutional network to obtain the parameters of the target convolution module in the first streamlined convolutional network of the second super-resolution model, so that the processing effect of the target convolution module on the image is the same as the processing effect of the pixel normalization module and the convolution module of the first convolutional network on the image, that is, the pixel normalization module in the first super-resolution model and the convolution module in the first convolutional network can be equivalently merged into the target convolution module in the first streamlined convolutional network in the second super-resolution model. Compared with the first super-resolution model, the second super-resolution model integrates the pixel normalization processing into the convolution processing of the first layer, eliminating the process of performing pixel normalization processing on each pixel in the image separately.

[0115] In some embodiments, the first super-resolution model also includes a pixel denormalization module, and the pixel denormalization module processes image features as the last layer of processing. Since the pixel denormalization module processes image features in the same way as the convolution module, both of which are linear processing, the parameters of the pixel denormalization module can be integrated into the parameters of the convolution module of the last convolutional network to obtain the parameters of the target convolution module in the last streamlined convolutional network of the second super-resolution model, so that the processing effect of the target convolution module on the image is the same as the processing effect of the pixel denormalization module and the convolution module of the last convolutional network on the image, that is, the pixel denormalization module in the first super-resolution model and the convolution module in the last convolutional network can be equivalently merged into the target convolution module in the last streamlined convolutional network of the second super-resolution model. Compared with the first super-resolution model, the second super-resolution model integrates the pixel denormalization processing into the last layer of convolution processing, eliminating the process of performing pixel denormalization processing on the features of each pixel point separately.

[0116] In some embodiments, the convolutional network in the first super-resolution model includes multiple convolution modules and a first point-by-point addition module. The inputs of the multiple convolution modules are the same, and the outputs are added point by point by the first point-by-point addition module. That is, the processing effects of the multiple convolution modules on the image are cumulative. Then, the parameters of the multiple convolution modules are accumulated to obtain the parameters of the target convolution module in the simplified convolutional network of the second super-resolution model, so that the processing effect of the target convolution module on the image features is the same as the processing effect of the multiple convolution modules and the first point-by-point addition module of the first super-resolution model. That is, the multiple convolution modules and the first point-by-point addition module of the first super-resolution model can be merged into a target convolution module in the second super-resolution model. Among them, the input of the target convolution module is the same as the input of the multiple convolution modules, and the output of the target convolution module is the same as the output of the first point-by-point addition module. Compared with the first super-resolution model, the second super-resolution model merges multiple convolution processes and point-by-point addition processes into one convolution process, reducing the number of convolution processes and the number of point-by-point additions.

[0117] In some embodiments, the convolutional network in the first super-resolution model includes a residual block consisting of at least one convolutional module, a skip connection module, and a second point-by-point addition module, wherein the input of at least one convolutional module and the skip connection module are the same, and the output of at least one convolutional module and the output of the skip connection module are added point by point by the second point-by-point addition module, that is, the processing effect of the at least one convolutional module and the skip connection module on the image is cumulative, then the parameters of the at least one convolutional module and the skip connection module are accumulated to obtain the parameters of the target convolutional module in the simplified convolutional network of the second super-resolution model, so that the processing effect of the target convolutional module on the image features is the same as the processing effect of the at least one convolutional module, the skip connection module, and the second point-by-point addition module of the first super-resolution model, that is, the at least one convolutional module, the skip connection module, and the second point-by-point addition module of the first super-resolution model can be merged into a target convolutional module in the second super-resolution model. Wherein, the input of the target convolutional module is the same as the input of the at least one convolutional module, and the output of the target convolution module is the same as the output of the second point-by-point addition module. Compared with the first super-resolution model, the second super-resolution model integrates the residual processing process into a convolution process, reducing the complexity of the model.

[0118] In some embodiments, the first super-resolution model includes an upsampling module, a first pixel reassembly module, and a third point-by-point addition module, wherein the upsampling module is a nearest neighbor upsampling module. Since the nearest neighbor upsampling module takes the feature value of the nearest feature point and assigns it to the newly added feature point, the image features obtained by the nearest neighbor upsampling module are equivalent to redundantly copying the original image features. Point-by-point addition of the image features output by the upsampling module and the image features output by the first pixel reassembly module is equivalent to broadcasting and adding the image features input to the upsampling module and the image features input to the first pixel reassembly module before performing pixel reassembly. Therefore, the first pixel reassembly module in the first super-resolution model is converted to the second pixel reassembly module in the second super-resolution model, and the upsampling module and the third point-by-point addition module in the first super-resolution model are converted to the broadcast addition module in the second super-resolution model, wherein the input of the broadcast addition module is equivalent to the input of the upsampling module and the input of the first pixel reassembly module, and the input of the second pixel reassembly module is the output of the broadcast addition module. Compared with the first super-resolution model, the second super-resolution model omits the upsampling process.

[0119] Figure 7 This is a flow chart of an image processing method provided by an embodiment of the present application. Figure 7 , the image processing method is described in detail, and the image processing method includes the following steps:

[0120] 701. A computer device performs training based on a sample image and a reference image corresponding to the sample image to obtain a first super-resolution model, where the resolution of the reference image is greater than the resolution of the sample image.

[0121] Step 701 is similar to step 201 .

[0122] 702. The computer device obtains conversion relationship data, where the conversion relationship data is used to represent a correspondence between multiple first processing modules of the first super-resolution model and at least one second processing module of the second super-resolution model.

[0123] The first super-resolution model includes a plurality of first processing modules, the parameters of which can be fused, so that the plurality of first processing modules can be merged into at least one second processing module, and the at least one second processing module is used as a processing module in the second super-resolution model. The correspondence between the plurality of first processing modules and the at least one second processing module is represented by conversion relationship data. Among them, one second processing module corresponds to a plurality of first processing modules, and one second processing module is formed by merging a plurality of first processing modules. The processing effect of the second processing module on the image features is equivalent to the processing effect of the plurality of first processing modules corresponding to the second processing module on the image features. Accordingly, the number of at least one second processing module is less than the number of the plurality of first processing modules.

[0124] In some embodiments, multiple first processing modules include a first convolution module and a first feature processing module, and the first feature processing module and the first convolution module can be combined into one convolution module. Correspondingly, at least one second processing module includes a first target convolution module, and the conversion relationship data includes a correspondence between the first target convolution module, the first convolution module, and the first feature processing module, used to indicate that the parameters of the first feature processing module can be integrated into the parameters of the first convolution module. Optionally, the number of first feature processing modules is one or more. If the number of first feature processing modules is one, the conversion relationship data includes a correspondence between the first target convolution module, the first convolution module, and the first feature processing module. If the number of first feature processing modules is multiple, the conversion relationship data includes a correspondence between the first target convolution module, the first convolution module, and multiple first feature processing modules; or, the conversion relationship data includes multiple correspondences, each of which is used to indicate a correspondence between the first target convolution module, the first convolution module, and one first feature processing module.

[0125] Among them, the first feature processing module is used to perform other processing on the image features in addition to convolution processing. For example, the first feature processing module is used to perform linear processing such as normalization processing on the image features, and the embodiments of the present application are not limited to this. In some embodiments, the first feature processing module includes a pixel normalization module, and the pixel normalization module is used to normalize the pixel values ​​of the input image to obtain normalized image features; the input of the pixel normalization module is the input of the first convolution module, and the first convolution module is used to perform convolution processing on the normalized image features. The parameters of the pixel normalization module are fused with the parameters of the first convolution module to obtain the parameters of the first target convolution module. In some embodiments, the first feature processing module includes a batch normalization module, and the input of the batch normalization module is the output of the first convolution module. The batch normalization module is used to perform batch normalization processing on the image features output by the first convolution module. The parameters of the batch normalization module are fused with the parameters of the first convolution module to obtain the parameters of the first target convolution module.

[0126] In the above technical solution, by merging the first feature processing module and the convolution module into one convolution module, the complexity of the model is reduced, the model parameters are reduced, and the merged model is applied to perform image processing in the inference stage, which can improve the image processing efficiency.

[0127] In some embodiments, multiple first processing modules include multiple second convolution modules and a first point-by-point addition module. The inputs of the multiple second convolution modules are the same, and the outputs are added point-by-point by the first point-by-point addition module. The multiple second convolution modules and the first point-by-point addition module can be merged into at least one convolution module. Accordingly, at least one second processing module includes a second target convolution module. The conversion relationship data includes the correspondence between the second target convolution module and at least two second convolution modules among the multiple second convolution modules and the first point-by-point addition module, which is used to indicate that the parameters of the at least two second convolution modules corresponding to the second target convolution module and the first point-by-point addition module are fused to obtain the parameters of the second target convolution module.

[0128] In the above technical solution, by merging multiple convolution modules into one convolution module, the complexity of the model is reduced, the model parameters are reduced, and the merged model is applied to perform image processing in the inference stage, which can improve the image processing efficiency.

[0129] In some embodiments, the plurality of first processing modules include a jump connection module, a second point-by-point addition module, and at least one third convolution module, wherein the at least one third convolution module is respectively used to perform convolution processing on the first image feature to obtain at least one second image feature; the jump connection module is used to transfer the first image feature to the second point-by-point addition module; the second point-by-point addition module is used to perform point-by-point addition on the first image feature and the at least one second image feature to obtain a fused image feature. The jump connection module, the second point-by-point addition module, and the at least one third convolution module can be fused into one convolution module. Accordingly, at least one second processing module includes a third target convolution module, and the conversion relationship data includes the correspondence between the third target convolution module and the jump connection module, the second point-by-point addition module, and the at least one third convolution module, indicating that the parameters of the jump connection module, the second point-by-point addition module, and the at least one third convolution module corresponding to the third target convolution module are fused to obtain the parameters of the third target convolution module.

[0130] In the above technical solution, by merging a residual block into a convolution module, the complexity of the model is reduced, the model parameters are reduced, and the merged model is applied to perform image processing in the inference stage, which can improve the image processing efficiency.

[0131] In some embodiments, the pixel values ​​of the input image are preprocessed by the processing module in the first super-resolution model to obtain image features. After a series of processing is performed on the image features to obtain the final image features, the obtained image features are also subjected to inverse processing corresponding to the above preprocessing by the second feature processing module in the first super-resolution model to obtain the pixel values ​​of the final output image.

[0132] Among them, the input of the second feature processing module is the output of the first pixel reorganization module, and the second feature processing module is used to process the target image features output by the first pixel reorganization module to obtain an image of the target resolution. The first pixel reorganization module is used to reorganize the image features processed by the fourth convolution module to obtain the target image features. Accordingly, the parameters of the second feature processing module can be integrated into the fourth convolution module, and the second feature processing module and the fourth convolution module can be combined into one convolution module. Accordingly, at least one second processing module includes a fourth target convolution module, and the conversion relationship data includes the corresponding relationship between the fourth target convolution module and the second feature processing module and the fourth convolution module, which is used to indicate that the parameters of the second feature processing module can be integrated into the parameters of the fourth convolution module.

[0133] In some embodiments, the second feature processing module includes a pixel denormalization module configured to perform denormalization processing on target image features to obtain an image of target resolution. Parameters of the pixel denormalization module are fused with parameters of the fourth convolution module to obtain parameters of the fourth target convolution module.

[0134] In the above technical solution, by merging the second feature processing module and its preceding convolution module into one convolution module, the complexity of the model is reduced, the model parameters are reduced, and the merged model is applied for image processing in the inference stage, which can improve the image processing efficiency.

[0135] 703. The computer device fuses the parameters of the multiple first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module.

[0136] In some embodiments, the conversion relationship data includes a correspondence between the first target convolution module, the first convolution module, and the first feature processing module, which is used to indicate that the parameters of the first feature processing module can be integrated into the parameters of the first convolution module.

[0137] Among them, the parameters of the first convolution module include the first convolution weight and the first convolution bias, the processing object of the first convolution module is the image feature input to the first convolution module, the first convolution weight is the parameter multiplied by the image feature, the first convolution weight is represented as a multidimensional array, and the first convolution bias is the bias added after the first convolution weight and the product of the image feature. The parameters of the first feature processing module include the first processing weight, the processing object of the first feature processing module is the image feature input to the first feature processing module, and the first processing weight is the parameter multiplied by the image feature. Optionally, the parameters of the first feature processing module also include the first processing bias, and the first processing bias is the bias added after the first processing weight and the product of the image feature. Accordingly, the above step 703 includes the following steps 7031 to 7032:

[0138] 7031. The computer device determines the product of the first convolution weight and the first processing weight as the convolution weight of the first target convolution module.

[0139] Optionally, the input of the first feature processing module is the output of the first convolution module; or, the input of the first convolution module is the output of the first feature processing module, then, a certain feature is processed by the first convolution module and then by the first feature processing module; or, a certain feature is processed by the first feature processing module and then by the first convolution module, the comprehensive processing effect of the first convolution module and the first feature processing module is the accumulation of the processing effect of the first convolution module and the processing effect of the first feature processing module, therefore, the product of the first convolution weight and the first processing weight is determined as the convolution weight of the first target convolution module, so that the effect of processing based on the convolution weight of the first target convolution module is equivalent to the effect of processing based on the first convolution weight and the first processing weight.

[0140] 7032. The computer device determines a convolution bias of a first target convolution module based on at least one of the first convolution weight and the first processing weight, the first processing bias, and the first convolution bias.

[0141] The above technical solution determines the convolution weight of the first target convolution module as the product of the first processing weight of the feature processing module and the first convolution weight of the convolution module. On the basis of reducing the number of parameters, the effect of processing based on the convolution weight of the first target convolution module is made equivalent to the effect of processing based on the first convolution weight and the first processing weight. This improves the image processing efficiency while maintaining the consistency of the processing effect as much as possible. In addition, since the number of parameters is reduced, the consumption of computing resources is also reduced, and the computing cost is reduced. In addition, since the number of parameters is reduced, the complexity of the model is reduced, and the storage space occupied by the model is also reduced, and the lightweight model is achieved in the inference stage.

[0142] When the input of the first feature processing module is the output of the first convolution module, the first convolution bias and the first processing weight also have a multiplication relationship. Therefore, based on the first processing bias, the first convolution bias and the first processing weight, the convolution bias of the first target convolution module is determined, that is, the sum of the first processing bias and the first numerical value is determined as the convolution bias of the first target convolution module, and the first numerical value is the product of the first processing weight and the first convolution bias.

[0143] When the input of the first convolution module is the output of the first feature processing module, the first processing bias and the first convolution weight also have a multiplication relationship. Therefore, based on the first processing bias, the first convolution bias and the first convolution weight, the convolution bias of the first target convolution module is determined, that is, the sum of the first convolution bias and the second value is determined as the convolution bias of the first target convolution module, and the second value is the product of the first convolution weight and the first processing bias.

[0144] In some embodiments, the first feature processing module is a pixel normalization module, and the process of the pixel normalization module normalizing the pixel values ​​of the image is represented as h norm =x÷255-0.5; where x represents the pixel value of the input image, h norm Represents the normalized image features. If the process of the first convolution module performing convolution processing on the normalized image features is represented by h out =W*h norm +b; where W represents the first convolution weight, b represents the first convolution bias, and h out Represents the image features after convolution processing. Correspondingly, the process of convolution processing of the pixel value of the input image by the first target convolution module is represented as h out =W*(x÷255-0.5)+b=(W÷255)*x+(b-0.5*W), then the convolution weight of the first target convolution module is also (W÷255), and the convolution bias of the first target convolution module is also (b-0.5*W).

[0145] The above technical solution also determines equivalent processing biases based on the relationship between the convolution module and the feature processing module, ensuring that the processing weights and processing biases are equivalent before and after the module merger. This reduces the number of parameters, improves image processing efficiency, reduces computational costs, and reduces the storage space occupied by the model while maintaining consistent processing effects before and after the module merger.

[0146] Assuming that an image consists of m pixels, if the pixel normalization module and the first convolution module of the first super-resolution model are used to process the image, it is necessary to first perform m normalization processes on the m pixels based on the pixel normalization module, and then perform k convolution processes on the image based on the first convolution module. However, when the first target convolution module of the second super-resolution model is used to process the image, only k convolution processes are required to achieve the same processing effect as the first super-resolution model. It can be seen that processing the image based on the second super-resolution model significantly improves processing efficiency while maintaining consistent processing effects, significantly reduces computational costs, and reduces the storage space occupied by the model.

[0147] In some embodiments, the conversion relationship data includes a corresponding relationship between the second target convolution module and multiple second convolution modules and the first point-by-point addition module, which is used to represent that the parameters of the multiple second convolution modules and the first point-by-point addition module are fused to obtain the parameters of the second target convolution module. Wherein, the multiple second convolution modules are a parallel network structure, and the multiple image features obtained by processing the multiple second convolution modules are added point by point as the output features of the multiple second convolution modules. Then, the multiple second convolution modules can be merged into a second target convolution module, and the sum of the convolution weights of the multiple second convolution modules is determined as the convolution weight of the second target convolution module; the sum of the convolution biases of the multiple second convolution modules is determined as the convolution bias of the second target convolution module, so that the second target convolution module can achieve the same processing effect as the multiple second convolution modules and the first point-by-point addition module.

[0148] The above technical solution merges multiple convolution modules into one convolution module, so that the processing effect of the merged convolution module is consistent with that of the multiple convolution modules. On the basis of maintaining the consistency of processing effects before and after the module merger, the number of parameters is reduced, the image processing efficiency is improved, the computational cost is reduced, and the storage space occupied by the model is reduced.

[0149] In some embodiments, the conversion relationship data includes the correspondence between the third target convolution module and the jump connection module, the second point-by-point addition module and at least one third convolution module, which is used to indicate that the parameters of the jump connection module, the second point-by-point addition module and at least one third convolution module are fused to obtain the parameters of the third target convolution module.

[0150] Among them, at least one third convolution module is a parallel network structure, the jump connection module has the same input as the at least one third convolution module, and the jump connection module directly passes the input features to the second point-by-point addition module. The parameter of the jump connection module is 1, which is used to indicate that the input features are superimposed on the output features. Then, the sum of the parameters of the jump connection module and the convolution weight of the at least one third convolution module is determined as the convolution weight of the third target convolution module; the sum of the convolution bias of the at least one third convolution module is determined as the convolution bias of the third target convolution module, so that the third target convolution module can achieve the same processing effect as the jump connection module, the second point-by-point addition module and the at least one fourth convolution module.

[0151] The above technical solution merges the residual blocks into a convolution module, so that the processing effect of the convolution module is consistent with that of the residual block. On the basis of maintaining the consistent processing effect before and after the module merger, the number of parameters is reduced, the image processing efficiency is improved, the computational cost is reduced, and the storage space occupied by the model is reduced.

[0152] In some embodiments, the conversion relationship data includes a correspondence between the fourth target convolution module and the second feature processing module and the fourth convolution module, which is used to indicate that the parameters of the second feature processing module can be integrated into the parameters of the fourth convolution module.

[0153] Among them, the parameters of the fourth convolution module include a second convolution weight and a second convolution bias, the processing object of the fourth convolution module is the image feature input to the fourth convolution module, the second convolution weight is a parameter multiplied by the image feature, and the second convolution bias is a bias added after the second convolution weight and the product of the image feature. The parameters of the second feature processing module include a second processing weight, the processing object of the second feature processing module is the image feature input to the second feature processing module, and the second processing weight is a parameter multiplied by the image feature. Optionally, the parameters of the second feature processing module also include a second processing bias, and the second processing bias is a bias added after the second processing weight and the product of the image feature.

[0154] Since the second feature processing module is located after the fourth convolution module and is the linear processing module closest to the fourth convolution module, the comprehensive processing effect of the fourth convolution module and the second feature processing module is the accumulation of the processing effect of the fourth convolution module and the processing effect of the second feature processing module. Therefore, the product of the second convolution weight and the second processing weight is determined as the convolution weight of the fourth target convolution module; the sum of the second processing bias and the third numerical value is determined as the convolution bias of the fourth target convolution module, and the third numerical value is the product of the second convolution weight and the second processing weight, so that the fourth target convolution module can achieve the same processing effect as the second feature processing module and the fourth convolution module.

[0155] In some embodiments, the second feature processing module is a pixel denormalization module, which is used to perform denormalization processing on the target image features output by the first pixel reconstruction module to obtain an image of target resolution. The process of denormalization processing of the pixel values ​​of the image by the pixel denormalization module is expressed as y=255×h out +127.5; where h out Represents the feature value of the image feature of the input pixel denormalization module, and y represents the denormalized pixel value. If the process of convolution processing of the input feature by the fourth convolution module is represented by h out =W*h in +b; where h in represents the image features input to the fourth convolution module, W represents the fourth convolution weight, b represents the fourth convolution bias, h out Represents the image features after convolution processing. Correspondingly, the process of the fourth target convolution module processing the input image features is expressed as y = 255 × h out +127.5=255×(W*h in +b)+127.5=(255×W)*h in +(255*b+127.5), then the convolution weight of the fourth target convolution module is (255×W), and the convolution bias of the fourth target convolution module is also (255*b+127.5).

[0156] The above technical solution merges the second feature processing module and the fourth convolution module, which are located after the fourth convolution module and are closest to the fourth convolution module, into one convolution module, so that the processing effect of the merged convolution module is consistent with that of the fourth convolution module and the second feature processing module. On the basis of maintaining the consistency of the processing effect before and after the module merger, the number of parameters is reduced, the image processing efficiency is improved, the computational cost is reduced, and the storage space occupied by the model is reduced.

[0157] In some embodiments, the first super-resolution model includes at least two of the above-mentioned multiple network structures that can be merged; or, a part of the network structure in the first super-resolution model is a combination of at least two of the above-mentioned multiple network structures. Accordingly, the computer device combines the corresponding at least two module merging methods and the corresponding at least two parameter fusion methods to fuse the parameters of the multiple first processing modules involved to obtain the parameters of the corresponding second processing module.

[0158] In some embodiments, the structure of the convolutional network is as follows Figure 6 As shown, the output of each convolution module in the convolutional network is processed by a batch normalization module. The convolution weight of the convolution module 601 is represented by W 1×3 , the convolution bias is denoted as b 1×3The convolution weight of the convolution module 602 is represented by W 3×3 , the convolution bias is denoted as b 3×3 The convolution weight of the convolution module 603 is represented by W 3×1 , the convolution bias is denoted as b 3×1 .

[0159] The parameters of the batch normalization module 604 are fused into the parameters of the convolution module 601 to obtain the convolution weight W′ of the target convolution module. 1×3 , convolution bias b′ 1×3 The parameters of the batch normalization module 605 are fused into the parameters of the convolution module 602 to obtain the convolution weight W′ of the target convolution module. 3×3 , convolution bias b′ 3×3 The parameters of the batch normalization module 606 are fused into the parameters of the convolution module 603 to obtain the convolution weight W′ of the target convolution module. 3×1 , convolution bias b′ 3×1 .

[0160] Furthermore, the multiple parallel convolution modules, the point-by-point addition module 608, and the jump connection module 607 in the convolution network constitute a residual block. The parameters of the multiple convolution modules, the point-by-point addition module 608, and the jump connection module 607 are fused into the parameters of the target convolution module to obtain the convolution weight W″ of the target convolution module. 3×3 = 1+W′ 1×3 +W′ 3×3 +W′ 3×1 , the convolution bias b″ is b′ 1×3 +b′ 3×3 +b′ 3×1 If the activation function is the ReLU activation function, then the output feature h of the convolutional network is out =ReLU[(1+W′ 1×3 +W′ 3×3 +W′ 3×1 )*h in +(b′ 1×3 +b′ 3×3 +b′ 3×1 )]=ReLU(W″ 3×3 *h in +b″), where h in is the feature of the input convolutional network. In addition, the convolution weight is W″ 3×3 , the target convolution module with convolution bias b″ can also be regarded as a convolution module with a convolution kernel size of 3×3. Accordingly, the convolution network is converted to Figure 8 The simplified convolutional network shown includes a convolution module 801 with a convolution kernel size of 3×3 and an activation module 802.

[0161] In some embodiments, if the convolutional network is the first convolutional network in the first super-resolution model, the input of the convolutional network is the output of the pixel normalization module. Accordingly, the parameters of the convolutional module of the convolutional network are also fused with the parameters of the pixel normalization module. If the pixel normalization module performs normalization processing on the pixel values ​​of the input image as represented by h norm =x÷255-0.5, the parameters of the pixel normalization module are fused into multiple convolution modules, and the convolution weights of the corresponding multiple target convolution modules are W″′ 1×3 =W 1×3 ÷255;W″′ 3×3 =W 3×3 ÷255;W″′ 3×1 =W 3×1 ÷255; the convolution bias is b″′ 1×3 =b 1×3 -0.5*W 1×3 ; b″′ 3×3 =b 3×3 -0.5*W 3×3 ; b″′ 3×1 =b 3×1 -0.5*W 3×1 The multiple target convolution modules are then combined with the batch normalization module to obtain the convolution weight W″ 1×3 , the convolution bias is b″″ 1×3 The convolution module has a convolution weight of W″ 3×3 , the convolution bias is b″″ 3×3 The convolution module, and the convolution weight is W″″ 3×1 , the convolution bias is b″″ 3×1 Convolution module; then perform parameter fusion with the jump connection module and the point-by-point addition module to obtain W″′ 3×3 1+W″″ 1×3 +W″″ 3×3 +W″″ 3×1 , the convolution bias b″′ is b″″ 1×3 +b″″ 3×3 +b″″ 3×1 The target convolution module.

[0162] In some embodiments, if the convolutional network is the last convolutional network in the first super-resolution model, the image features processed by the convolutional network are also used as the input of the pixel denormalization module. If the pixel denormalization module performs denormalization processing on the input image features, the process is expressed as y = 255 × h out +127.5, the parameters of the pixel denormalization module are fused into multiple convolution modules, and the corresponding convolution weights of multiple target convolution modules are W″″′1×3 =255×W 1×3 ;W″″′ 3×3 =255×W 3×3 ;W″″′ 3×1 =255×W 3×1 ; The convolution biases are b″″′ 1×3 =255×b 1×3 +127.5; b″″′ 3×3 =255×b 3×3 +127.5; b″″′ 3×1 =255×b 3×1 +127.5. The multiple target convolution modules are then combined with the batch normalization module to obtain the convolution weight W″″′ 1×3 , convolution bias b″″′ 1×3 The convolution module is W″″′ 3×3 , the convolution bias is b″″′ 3×3 The convolution module, and the convolution weight is W″″′ 3×1 , the convolution bias is b″″′ 3×1 The convolution module is then combined with the skip connection module and the point-by-point addition module to obtain the convolution weight W″″ 3×3 1+W″″″ 1×3 +W″″″ 3×3 +W″″″ 3×1 , the convolution bias b″″ is b″″″ 1×3 +b″″″ 3×3 +b″″″ 3×1 The target convolution module.

[0163] In some embodiments, the conversion relationship data also includes a correspondence between multiple third processing modules in the first super-resolution model and multiple fourth processing modules in the second super-resolution model, and the number of the multiple fourth processing modules is less than the number of the multiple third processing modules.

[0164] The plurality of third processing modules include an up-sampling module, a first pixel reassembly module and a third point-by-point addition module. Figure 9 The upsampling module upsamples the image feature 901 of the input image to obtain a third image feature 902 with a target scale of 4×4; the input of the first pixel reconstruction module is the fourth image feature 903 processed by the last convolution module of the first super-resolution model, and the first pixel reconstruction module is used to reconstruct the fourth image feature 903 to obtain a fifth image feature 904 with a target scale of 4×4; the third image feature 902 and the fifth image feature 904 are added point by point through the point-by-point addition module to obtain the target image feature 905.

[0165] In the case where the upsampling module is a nearest neighbor upsampling module, the third image feature obtained by the nearest neighbor upsampling module through upsampling the input image feature is equivalent to the image feature obtained by copying the input image feature into image features of multiple channels and then performing pixel reassembly. Figure 10 , the process of processing image features through the upsampling module, the first pixel reconstruction module and the third point-by-point addition module can also be replaced by the following steps: copying the image feature 1001 of the input image to multiple channels to obtain image feature 1002; through the point-by-point addition module, the image feature 1002 and the fourth image feature 1003 processed by the last layer convolution module of the first super-resolution model are point-by-point added to obtain image feature 1004; since the image feature 1002 and the fourth image feature 1003 both include sub-features of multiple channels, correspondingly, the image feature 1004 also includes sub-features of multiple channels, the sub-features of multiple channels in the image feature 1004 are recombined through the first pixel reconstruction module to obtain the target image feature 1005.

[0166] Furthermore, the process of copying the image features of the input image to multiple channels and point-by-point adding the copied image features to the fourth image features can also be replaced by the process of point-by-point adding the image features of the input image to the sub-features of multiple channels in the fourth image features. This process of point-by-point adding the image features of the input image to the sub-features of multiple channels in the fourth image features is implemented based on a broadcast addition module. Accordingly, the multiple fourth processing modules include a broadcast addition module and a second pixel reassembly module. The input of the broadcast addition module is the image features of the image input to the second super-resolution model and the image features processed by the last convolution module of the second super-resolution model, where the image features processed by the last convolution module include sub-features of multiple channels. The broadcast addition module uses the image features of the image input to the second super-resolution model as features broadcast to the sub-features of the multiple channels and performs point-by-point addition of the image features of the image input to the second super-resolution model with the sub-features of the multiple channels to obtain superimposed features of the multiple channels. The output of the broadcast addition module is input to the second pixel reassembly module, which reassembles the superimposed features of the multiple channels to obtain target image features.

[0167] To ensure that the image processing effects of the broadcast addition module and the second pixel reconstruction module in the second super-resolution model are consistent with the image processing effects of the upsampling module, the first pixel reconstruction module and the third point-by-point addition module in the first super-resolution model, the image processing method also includes the following step 704.

[0168] 704. The computer device determines, based on the conversion relationship data, parameters of the first pixel reconstruction module in the plurality of third processing modules as parameters of the second pixel reconstruction module in the plurality of fourth processing modules.

[0169] Since both the first pixel reassembly module and the second pixel reassembly module are used to reassemble features of multiple channels into target image features, by determining the parameters of the first pixel reassembly module as the parameters of the second pixel reassembly module, the second super-resolution model can achieve the same processing effect as the first super-resolution model.

[0170] See also Figure 11 The second super-resolution model processes the image features 1101 processed by the last layer of convolution module and the image features 1102 of the image input to the second super-resolution model to obtain the target image features 1103. The process is as follows: the image features 1101 processed by the last layer of convolution module and the image features 1102 of the image input to the second super-resolution model are broadcasted and added through the broadcast addition module to obtain the superimposed features 1104 of multiple channels; the superimposed features 1104 of multiple channels are recombined through the second pixel recombination module to obtain the target image features 1103.

[0171] The above technical solution broadcasts and adds the image features after convolution processing and the image features of the input image through a broadcast addition module, and then obtains the target image features through pixel recombination. Compared with the process of obtaining the target image features through an upsampling module, a point-by-point addition module and a pixel recombination module, and the process of obtaining sub-features of multiple channels by copying and performing point-by-point addition and pixel recombination, the generation of intermediate features is reduced, the number of extra memory reads and writes due to the generation of intermediate features is reduced, and the memory read and write overhead in the inference stage is reduced, thereby further improving the image processing efficiency.

[0172] The technical solution provided by the embodiment of the present application realizes the simplification of the convolutional network by performing module merging and parameter fusion through the above steps 702 and 703, and obtains a simplified convolutional network. The upsampling module and the third point-by-point addition module are replaced with a broadcast addition module through step 704, the first pixel reassembly module is converted into a second pixel reassembly module, and the output of the broadcast addition module is used as the input of the second pixel reassembly module, so as to obtain the following: Figure 12The second super-resolution model shown in FIG. 1 includes at least one streamlined convolutional network, a broadcast addition module 1201 and a second pixel reconstruction module 1202, wherein, in the case where at least one streamlined convolutional network is a plurality of streamlined convolutional networks, for any two streamlined convolutional networks in the plurality of streamlined convolutional networks, the input of one streamlined convolutional network is the output of another streamlined convolutional network, the input of the broadcast addition module 1201 includes the image input to the second super-resolution model and the output of the last streamlined convolutional network 1203, the broadcast addition module 1201 is used to perform point-by-point addition of the features of the image input to the second super-resolution model and the output of the last streamlined convolutional network 1203, and output the features after point-by-point addition; the input of the second pixel reconstruction module 1202 is the output of the broadcast addition module 1201, and the second pixel reconstruction module 1202 is used to perform pixel reconstruction on the features output by the broadcast addition module 1201 to obtain an image of the target resolution.

[0173] The above technical solution obtains a second super-resolution model through parameter fusion and module adjustment. It reduces a large number of processing parameters and reduces memory read and write overhead without losing model accuracy. The second super-resolution model is applied to image processing in the inference stage, achieving a 2 to 3 times speed increase. Figure 13 This is a schematic diagram of a super-resolution image provided by an embodiment of the present application, see Figure 13 , image 1301 is a low-resolution image to be processed, image 1302 is a high-resolution image obtained by processing image 1301 by the third super-resolution model, and image 1303 is a high-resolution image obtained by processing image 1301 by the second super-resolution model. The model structure of the third super-resolution model also adopts Figure 12 The model structure shown, the third super-resolution model is based on Figure 12 The model is obtained by training the model structure shown in the figure, that is, the model training process uses Figure 12 The model structure shown. Combined Figure 13 It can be seen that the clarity of image 1303 is significantly higher than that of images 1301 and 1302. Although the model structure of the third super-resolution model is the same as that of the second super-resolution model, since the parameters of the second super-resolution model are learned through a more complex model structure, it can better fit the mapping from low-resolution images to high-resolution images. Therefore, the degree of image clarity restoration achieved by applying the second super-resolution model is higher than that achieved by the third super-resolution model.

[0174] It should be noted that the above technical solution can be applied in a variety of image processing scenarios. For example, the above technical solution is applied in the scenario of cloud video conferencing, where a second super-resolution model is deployed on the terminal. The terminal processes the video frames in the video stream based on the second super-resolution model, displays higher-resolution video frames, and optimizes the visual effect of the members participating in the video conference. For another example, the above technical solution is applied in the scenario of telemedicine, where a second super-resolution model is deployed on the medical device. The medical device receives the remotely transmitted video stream, processes the video frames in the video stream based on the second super-resolution model, and displays higher-resolution video frames, making it easier for medical staff to observe the details of the lesions in the video stream. Moreover, since the number of parameters of the second super-resolution model is small and the complexity of the model structure is low, while ensuring high-quality processing effects, it also improves processing efficiency, speeds up processing, and shortens processing time. It is more suitable for application in the above-mentioned scenarios with high real-time requirements and image quality requirements, and has important application value in the above-mentioned cloud video conferencing scenarios and telemedicine scenarios.

[0175] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0176] Figure 14 This is a block diagram of an image processing device provided by an embodiment of the present application. Figure 14 , the device comprises:

[0177] The model training unit 1401 is configured to perform training based on the sample image and a reference image corresponding to the sample image to obtain a first super-resolution model, wherein the resolution of the reference image is greater than the resolution of the sample image;

[0178] a data acquisition unit 1402 configured to acquire conversion relationship data, the conversion relationship data being used to represent a correspondence between a plurality of first processing modules of a first super-resolution model and at least one second processing module of a second super-resolution model, wherein the number of processing modules in the first super-resolution model is greater than the number of processing modules in the second super-resolution model;

[0179] The parameter fusion unit 1403 is configured to fuse the parameters of the plurality of first processing modules based on the conversion relationship data to obtain the parameters of at least one second processing module of the second super-resolution model.

[0180] The image processing device provided in the embodiment of the present application is trained based on low-resolution images and corresponding high-resolution images to obtain a first super-resolution model. By converting multiple processing modules of the first super-resolution model into a smaller number of processing modules, a second super-resolution model is obtained, thereby reducing the complexity of the model. Parameter fusion is then performed based on the correspondence between the processing modules before and after the conversion, thereby maintaining the original processing effect while reducing the number of parameters. Image processing is performed based on a small number of parameters contained in the second super-resolution model, which can reduce the amount of calculation while ensuring high-quality image processing effects and improve image processing efficiency.

[0181] In one possible implementation, at least one second processing module includes a first target convolution module, and the plurality of first processing modules corresponding to the first target convolution module include a first convolution module and a first feature processing module; the first feature processing module is configured to perform other processing on the image features in addition to the convolution processing;

[0182] The parameters of the first convolution module include a first convolution weight and a first convolution bias, and the parameters of the first feature processing module include a first processing weight and a first processing bias;

[0183] The parameter fusion unit 1403 includes:

[0184] a convolution weight determination subunit, configured to determine the product of the first convolution weight and the first processing weight as the convolution weight of the first target convolution module;

[0185] A convolution bias determination subunit is used to determine a convolution bias of a first target convolution module based on at least one of the first convolution weight and the first processing weight, the first processing bias, and the first convolution bias.

[0186] In another possible implementation, the convolution bias determination subunit is configured to, when the output of the first convolution module is the input of the first feature processing module, determine the sum of the first processing bias and the first value as the convolution bias of the first target convolution module, where the first value is the product of the first processing weight and the first convolution bias; or,

[0187] The convolution bias determination subunit is used to determine the sum of the first convolution bias and the second value when the output of the first feature processing module is the input of the first convolution module, and the second value is the product of the first convolution weight and the first processing bias.

[0188] In another possible implementation, the first feature processing module includes a pixel normalization module; the pixel normalization module is used to normalize the pixel values ​​of the input image to obtain normalized image features; and the first convolution module is used to perform convolution processing on the normalized image features.

[0189] In another possible implementation, the first feature processing module includes a batch normalization module; the batch normalization module is used to perform batch normalization processing on the image features output by the first convolution module.

[0190] In another possible implementation, at least one second processing module includes a second target convolution module, and the multiple first processing modules corresponding to the second target convolution module include a first point-by-point addition module and multiple second convolution modules; the outputs of the multiple second convolution modules are inputs to the first point-by-point addition module; the multiple second convolution modules are respectively used to perform convolution processing on the same image feature to obtain multiple convolution-processed image features; the first point-by-point addition module is used to perform point-by-point addition on the multiple convolution-processed image features to obtain a fused image feature;

[0191] The parameter fusion unit 1403 is used to:

[0192] Determining the sum of the convolution weights of the plurality of second convolution modules as the convolution weight of the second target convolution module;

[0193] The sum of the convolution offsets of the multiple second convolution modules is determined as the convolution offset of the second target convolution module.

[0194] In another possible implementation, at least one second processing module includes a third target convolution module, and the multiple first processing modules corresponding to the third target convolution module include a skip connection module, a second point-by-point addition module, and at least one third convolution module; the at least one third convolution module is respectively used to perform convolution processing on the first image feature to obtain at least one second image feature; the output of the skip connection module and the output of the at least one third convolution module are input to the second point-by-point addition module; the second point-by-point addition module is used to perform point-by-point addition on the first image feature and the at least one second image feature to obtain a fused image feature;

[0195] The parameter fusion unit 1403 is used to:

[0196] Determine the sum of the parameters of the skip connection module and the convolution weight of at least one third convolution module as the convolution weight of the third target convolution module, where the parameter of the skip connection module is 1;

[0197] The sum of the convolution biases of at least one third convolution module is determined as the convolution bias of the third target convolution module.

[0198] In another possible implementation,

[0199] At least one second processing module includes a fourth target convolution module, and the plurality of first processing modules corresponding to the fourth target convolution module include a second feature processing module and a fourth convolution module; the fourth convolution module is the last convolution module in the first super-resolution model; the processing module of the first super-resolution model also includes a first pixel reorganization module; the first pixel reorganization module is used to reorganize the image features processed by the fourth convolution module to obtain target image features; the output of the first pixel reorganization module is input to the second feature processing module; the second feature processing module is used to process the target image features to obtain an image of target resolution;

[0200] The parameters of the fourth convolution module include a second convolution weight and a second convolution bias, and the parameters of the second feature processing module include a second processing weight and a second processing bias;

[0201] The parameter fusion unit 1403 is used to:

[0202] Determine the product of the second convolution weight and the second processing weight as the convolution weight of the fourth target convolution module;

[0203] The sum of the second processing bias and the third value is determined as the convolution bias of the fourth target convolution module, and the third value is the product of the second processing weight and the second convolution bias.

[0204] In another possible implementation, the second feature processing module includes a pixel denormalization module; the pixel denormalization module is used to perform denormalization processing on the target image features to obtain an image with a target resolution.

[0205] In another possible implementation, the conversion relationship data is further used to represent a correspondence between multiple third processing modules in the first super-resolution model and multiple fourth processing modules in the second super-resolution model;

[0206] Among them, the multiple third processing modules include an upsampling module, a first pixel reorganization module and a third point-by-point addition module; the upsampling module is used to upsample the image features of the input image to obtain a third image feature of the target scale; the input of the first pixel reorganization module is the fourth image feature processed by the last layer convolution module of the first super-resolution model, and the first pixel reorganization module is used to reorganize the fourth image feature to obtain a fifth image feature of the target scale; the output of the upsampling module and the output of the first pixel reorganization module are input to the third point-by-point addition module; the third point-by-point addition module is used to add the third image feature and the fifth image feature point by point to obtain a target image feature, which is a feature of the image of the target resolution;

[0207] The plurality of fourth processing modules include a broadcast addition module and a second pixel reassembly module, wherein the input of the broadcast addition module is the image features of the input image and the image features processed by the last convolution module of the second super-resolution model, and the image features processed by the last convolution module of the second super-resolution model include sub-features of multiple channels; the broadcast addition module is used to add the image features of the input image to the sub-features of the multiple channels point by point to obtain superimposed features of the multiple channels; the output of the broadcast addition module is the input of the second pixel reassembly module; the second pixel reassembly module is used to reassemble the superimposed features of the multiple channels to obtain target image features;

[0208] The device also includes:

[0209] The parameter determination unit is configured to determine, based on the conversion relationship data, the parameters of the first pixel reconstruction module in the plurality of third processing modules as the parameters of the second pixel reconstruction module in the plurality of fourth processing modules.

[0210] It should be noted that the image processing apparatus provided in the above embodiments is merely illustrated by the division of the aforementioned functional units when performing image processing. In actual applications, the aforementioned functions can be assigned to different functional units as needed, i.e., the internal structure of the computer device can be divided into different functional units to perform all or part of the functions described above. Furthermore, the image processing apparatus provided in the above embodiments and the image processing method embodiments are based on the same concept. The specific implementation process is detailed in the method embodiments and will not be further described here.

[0211] In some embodiments, the computer device is configured as a terminal, Figure 15 1 is a block diagram of a terminal provided in an embodiment of the present application. The terminal 1500 may be a desktop computer, a laptop computer, a smartphone, or a tablet computer. The terminal 1500 may also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0212] Typically, the terminal 1500 includes a processor 1501 and a memory 1502 .

[0213] The processor 1501 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0214] Memory 1502 may include one or more computer-readable storage media, which may be non-transitory. Memory 1502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1502 is used to store a computer program, which is executed by processor 1501 to implement the image processing method provided in the method embodiment of the present application.

[0215] In some embodiments, terminal 1500 may optionally include a peripheral device interface 1503 and at least one peripheral device. The processor 1501, memory 1502, and peripheral device interface 1503 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1503 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1504, a display screen 1505, a camera assembly 1506, and a power supply 1507.

[0216] The peripheral device interface 1503 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1501 and the memory 1502. In some embodiments, the processor 1501, the memory 1502, and the peripheral device interface 1503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1501, the memory 1502, and the peripheral device interface 1503 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0217] RF circuit 1504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. RF circuit 1504 communicates with communication networks and other communication devices via electromagnetic signals. RF circuit 1504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. RF circuit 1504 may optionally include an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. RF circuit 1504 may communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, RF circuit 1504 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0218] The display screen 1505 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1505 is a touch screen display, the display screen 1505 also has the ability to collect touch signals on the surface or above the surface of the display screen 1505. The touch signal can be input as a control signal to the processor 1501 for processing. In this case, the display screen 1505 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen 1505, which is set on the front panel of the terminal 1500; in other embodiments, there can be at least two display screens 1505, which are respectively set on different surfaces of the terminal 1500 or in a folding design; in other embodiments, the display screen 1505 can be a flexible display screen, which is set on the curved surface or folding surface of the terminal 1500. Even more, the display screen 1505 can be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1505 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0219] The camera assembly 1506 is used to capture images or videos. Optionally, the camera assembly 1506 includes a front camera and a rear camera. Typically, the front camera is set on the front panel of the terminal 1500, and the rear camera is set on the back of the terminal 1500. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0220] Power supply 1507 is used to power various components in terminal 1500. Power supply 1507 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1507 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0221] Those skilled in the art will understand that Figure 15The structure shown in the figure does not constitute a limitation on the terminal 1500, and the terminal 1500 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0222] In some embodiments, the computer device is configured as a server, Figure 16 1 is a block diagram of a server provided in an embodiment of the present application. The server 1600 may vary significantly due to different configurations or performance, and may include one or more processors (Central Processing Units, CPUs) 1601 and one or more memories 1602. The memories 1602 store computer programs that are loaded and executed by the processors 1601 to implement the image processing methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0223] In an exemplary embodiment, a computer-readable storage medium is further provided, which stores a computer program. The computer program can be executed by a processor in a computer device to perform the image processing method in the above embodiment. For example, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0224] The present application also provides a computer program product or computer program, which includes computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device executes the image processing method in each of the above-mentioned method embodiments.

[0225] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by programs instructing related hardware to accomplish the steps. The above programs may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk, or an optical disk, etc.

[0226] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Performing training based on a sample image and a reference image corresponding to the sample image to obtain a first super-resolution model, wherein the resolution of the reference image is greater than the resolution of the sample image; Obtaining conversion relationship data, where the conversion relationship data is used to represent a correspondence between a plurality of first processing modules of the first super-resolution model and at least one second processing module of the second super-resolution model, wherein the number of processing modules in the first super-resolution model is greater than the number of processing modules in the second super-resolution model; Based on the conversion relationship data, the parameters of the multiple first processing modules are fused to obtain parameters of at least one second processing module of the second super-resolution model.

2. The method according to claim 1, characterized in that The at least one second processing module includes a first target convolution module, and the plurality of first processing modules corresponding to the first target convolution module include a first convolution module and a first feature processing module; the first feature processing module is used to perform other processing on the image features except the convolution processing; The parameters of the first convolution module include a first convolution weight and a first convolution bias, and the parameters of the first feature processing module include a first processing weight and a first processing bias; The fusing the parameters of the plurality of first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module of the second super-resolution model includes: Determining a product of the first convolution weight and the first processing weight as a convolution weight of the first target convolution module; A convolution bias of the first target convolution module is determined based on at least one of the first convolution weight and the first processing weight, the first processing bias, and the first convolution bias.

3. The method according to claim 2, characterized in that Determining the convolution bias of the first target convolution module based on at least one of the first convolution weight and the first processing weight, the first processing bias, and the first convolution bias includes any of the following: In a case where the output of the first convolution module is the input of the first feature processing module, determining the sum of the first processing bias and a first value as the convolution bias of the first target convolution module, where the first value is the product of the first processing weight and the first convolution bias; When the output of the first feature processing module is the input of the first convolution module, the sum of the first convolution bias and a second value is calculated, where the second value is the product of the first convolution weight and the first processing bias.

4. The method according to claim 2, characterized in that The first feature processing module includes a pixel normalization module; the pixel normalization module is used to normalize the pixel values ​​of the input image to obtain normalized image features; the first convolution module is used to perform convolution processing on the normalized image features.

5. The method according to claim 2, characterized in that The first feature processing module includes a batch normalization module; the batch normalization module is used to perform batch normalization processing on the image features output by the first convolution module.

6. The method according to claim 1, characterized in that The at least one second processing module includes a second target convolution module, and the plurality of first processing modules corresponding to the second target convolution module include a first point-by-point addition module and a plurality of second convolution modules; the outputs of the plurality of second convolution modules are inputs of the first point-by-point addition module; the plurality of second convolution modules are respectively used to perform convolution processing on the same image feature to obtain a plurality of convolved image features; the first point-by-point addition module is used to perform point-by-point addition on the plurality of convolved image features to obtain a fused image feature; The fusing the parameters of the plurality of first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module of the second super-resolution model includes: Determining the sum of the convolution weights of the plurality of second convolution modules as the convolution weight of the second target convolution module; The sum of the convolution offsets of the multiple second convolution modules is determined as the convolution offset of the second target convolution module.

7. The method according to claim 1, characterized in that The at least one second processing module includes a third target convolution module, and the plurality of first processing modules corresponding to the third target convolution module include a skip connection module, a second point-by-point addition module, and at least one third convolution module; the at least one third convolution module is respectively used to perform convolution processing on the first image feature to obtain at least one second image feature; the output of the skip connection module and the output of the at least one third convolution module are input to the second point-by-point addition module; the second point-by-point addition module is used to perform point-by-point addition on the first image feature and the at least one second image feature to obtain a fused image feature; The fusing the parameters of the plurality of first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module of the second super-resolution model includes: Determining the sum of the parameters of the skip connection module and the convolution weight of the at least one third convolution module as the convolution weight of the third target convolution module, where the parameter of the skip connection module is 1; The sum of the convolution offsets of the at least one third convolution module is determined as the convolution offset of the third target convolution module.

8. The method according to claim 1, characterized in that The at least one second processing module includes a fourth target convolution module, and the plurality of first processing modules corresponding to the fourth target convolution module include a second feature processing module and a fourth convolution module; the fourth convolution module is the last convolution module in the first super-resolution model; the processing module of the first super-resolution model also includes a first pixel reorganization module; the first pixel reorganization module is used to reorganize the image features processed by the fourth convolution module to obtain target image features; the output of the first pixel reorganization module is the input of the second feature processing module; the second feature processing module is used to process the target image features to obtain an image of target resolution; The parameters of the fourth convolution module include a second convolution weight and a second convolution bias, and the parameters of the second feature processing module include a second processing weight and a second processing bias; The fusing the parameters of the plurality of first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module of the second super-resolution model includes: Determining a product of the second convolution weight and the second processing weight as a convolution weight of the fourth target convolution module; The sum of the second processing bias and a third value is determined as the convolution bias of the fourth target convolution module, and the third value is the product of the second processing weight and the second convolution bias.

9. The method according to claim 8, characterized in that The second feature processing module includes a pixel denormalization module; the pixel denormalization module is used to perform denormalization processing on the target image features to obtain an image with the target resolution.

10. The method according to any one of claims 1 to 9, characterized in that The conversion relationship data is further used to represent the corresponding relationship between the plurality of third processing modules in the first super-resolution model and the plurality of fourth processing modules in the second super-resolution model; Among them, the multiple third processing modules include an upsampling module, a first pixel reorganization module and a third point-by-point addition module; the upsampling module is used to upsample the image features of the input image to obtain third image features of the target scale; the input of the first pixel reorganization module is the fourth image feature processed by the last layer convolution module of the first super-resolution model, and the first pixel reorganization module is used to reorganize the fourth image feature to obtain the fifth image feature of the target scale; the output of the upsampling module and the output of the first pixel reorganization module are input to the third point-by-point addition module; the third point-by-point addition module is used to add the third image feature and the fifth image feature point by point to obtain the target image feature, which is the feature of the image of the target resolution; The multiple fourth processing modules include a broadcast addition module and a second pixel reassembly module, the input of the broadcast addition module is the image features of the input image and the image features processed by the last layer convolution module of the second super-resolution model, and the image features processed by the last layer convolution module of the second super-resolution model include sub-features of multiple channels; the broadcast addition module is used to add the image features of the input image to the sub-features of the multiple channels point by point to obtain superimposed features of the multiple channels; the output of the broadcast addition module is the input of the second pixel reassembly module; the second pixel reassembly module is used to reassemble the superimposed features of the multiple channels to obtain the target image features; The method further comprises: Based on the conversion relationship data, parameters of the first pixel reassembly module in the plurality of third processing modules are determined as parameters of the second pixel reassembly module in the plurality of fourth processing modules.

11. An image processing device, characterized in that: The device comprises: a model training unit, configured to perform training based on a sample image and a reference image corresponding to the sample image to obtain a first super-resolution model, wherein the resolution of the reference image is greater than the resolution of the sample image; a data acquisition unit, configured to acquire conversion relationship data, the conversion relationship data being used to represent a correspondence between a plurality of first processing modules of the first super-resolution model and at least one second processing module of the second super-resolution model, wherein the number of processing modules in the first super-resolution model is greater than the number of processing modules in the second super-resolution model; A parameter fusion unit is used to fuse the parameters of the multiple first processing modules based on the conversion relationship data to obtain parameters of at least one second processing module of the second super-resolution model.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 10.

Images