Image processing method and system, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides an image processing method, an image processing system, electronic equipment, a computer readable storage medium and a computer program product. The image processing method comprises the following steps: constructing an image quality improvement network comprising a structure re-parameterization module; and based on the image quality improvement network, extracting a multi-scale feature map of the to-be-processed image, and improving the image quality of the to-be-processed image based on the multi-scale feature map, the structure re-parameterization module has different structures in the training stage and the reasoning stage, and the construction of the structure re-parameterization module comprises the step of utilizing a structure re-parameterization technology to construct the multi-scale feature map of the to-be-processed image. And converting a plurality of branches in the training structure into a single-branch structure in the reasoning structure. The step of constructing the image quality improvement network comprising the structure re-parameterization module comprises the step of constructing a channel-by-channel convolution layer of a lightweight convolution module in the image quality improvement network based on an inference structure.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to an image processing method, an image processing system, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] As the demands for image quality from terminal devices such as mobile devices continue to increase, traditional algorithms can no longer meet the needs of some application scenarios. While AI (Artificial Intelligence) image enhancement technology can effectively solve the aforementioned problems, the network models used in the field of image quality enhancement often have many parameters, large model size, and long inference time. This has resulted in AI image enhancement technology not being widely adopted on terminal devices, including mobile devices, due to its high computing power requirements and long processing time.

[0003] Therefore, how to improve image quality while reducing the parameters of the image quality enhancement network model, compressing the model's computing power, and reducing the model's inference time has become one of the research hotspots in the field of image processing. Summary of the Invention

[0004] This disclosure presents an image processing method, an image processing system, an electronic device, a computer-readable storage medium, and a computer program product, which improve image quality by reducing the parameters of the network model, compressing the computing power of the network model, and reducing the inference time of the network model.

[0005] In a first aspect, this disclosure proposes an image processing method, which includes: constructing an image quality enhancement network including a structure reparameterization module; and extracting multi-scale feature maps of the image to be processed based on the image quality enhancement network, and enhancing the image quality of the image to be processed based on the multi-scale feature maps. The structure reparameterization module has a training structure and an inference structure in the training and inference phases, respectively. Constructing the structure reparameterization module includes using structure reparameterization techniques to transform multiple branches in the training structure into a single branch structure in the inference structure. Constructing the image quality enhancement network including the structure reparameterization module includes constructing a channel-wise convolutional layer of a lightweight convolutional module in the image quality enhancement network based on the inference structure.

[0006] In some embodiments of this disclosure, the first branch of the plurality of branches includes a 1×1 convolutional kernel, and the second branch of the plurality of branches includes a 1×1 convolutional kernel, an n×n convolutional kernel, and a 1×1 convolutional kernel connected in series, where n is a positive integer greater than 1. The third branch of the plurality of branches is a skip-connection structure, and the training structure is trained, wherein training the training structure includes: summing the outputs of the plurality of branches based on weights to obtain the output of the training structure; and activating the third branch in response to the number of input channels of the training structure being equal to the number of output channels.

[0007] In some embodiments of this disclosure, the computational compression ratio K between the inference structure and the training structure satisfies: K = a / b, where a is the number of input channels of the first 1×1 convolutional kernel in the second branch; and b is the number of output channels of the first 1×1 convolutional kernel in the second branch.

[0008] In some embodiments of this disclosure, the training structure further includes: a first branch and a second branch performing grouped convolution operations on the input features.

[0009] In some embodiments of this disclosure, constructing a lightweight convolution module further includes constructing a depth-separable convolution based on a 1×1 convolution kernel connected in series with a channel-by-channel convolutional layer.

[0010] In some embodiments of this disclosure, constructing a lightweight convolutional module further includes constructing an activation function layer concatenated with a depthwise separable convolution.

[0011] In some embodiments of this disclosure, constructing a lightweight convolutional module further includes constructing an attention submodule, wherein the attention submodule includes a cascaded 1×1 convolutional kernel and an activation function layer.

[0012] In some embodiments of this disclosure, the method further includes: using the output of an activation function layer concatenated with a depthwise separable convolution as input to an attention submodule; and determining the output of a lightweight convolution module based on the output of the attention submodule and the output of the activation function layer concatenated with the depthwise separable convolution.

[0013] In some embodiments of this disclosure, the image quality enhancement network, trained on the image, extracts multi-scale feature maps of the image to be processed, and enhances the image quality of the image to be processed based on the multi-scale feature maps, including: extracting multi-scale feature maps based on a lightweight convolutional module; constructing a residual module based on the inference structure of the skip connection structure and the structure reparameterization module; and enhancing the image quality of the image to be processed based on the output of the residual module and the concatenation output of multiple multi-scale feature maps.

[0014] In some embodiments of this disclosure, training an image quality enhancement network includes: inputting prior information into a pre-trained network to obtain trained prior information; and concatenating the trained prior information with training image data information to form a dataset for training the image quality enhancement network.

[0015] In some embodiments of this disclosure, the pre-trained network includes an inference structure of cascaded structural reparameterization modules and an activation function layer.

[0016] In some embodiments of this disclosure, the prior information includes lens parameters and lens wear information.

[0017] Secondly, this disclosure proposes an image processing system, comprising: a generation unit for constructing an image quality enhancement network including a structure reparameterization module, wherein the structure reparameterization module has a training structure and an inference structure in the training and inference phases, respectively; constructing the structure reparameterization module includes using structure reparameterization technology to transform multiple branches in the training structure into a single branch structure in the inference structure; and constructing the image quality enhancement network including the structure reparameterization module includes constructing a channel-wise convolutional layer of a lightweight convolutional module in the image quality enhancement network based on the inference structure; and a processing unit for extracting multi-scale feature maps of the image to be processed based on the trained image quality enhancement network, and improving the image quality of the image to be processed based on the multi-scale feature maps.

[0018] In some embodiments of this disclosure, the first branch of the plurality of branches includes a 1×1 convolutional kernel, and the second branch of the plurality of branches includes a 1×1 convolutional kernel, an n×n convolutional kernel, and a 1×1 convolutional kernel connected in series, where n is a positive integer greater than 1. The third branch of the plurality of branches is a skip-connected structure. The training structure further includes: summing the outputs of the plurality of branches based on weights to obtain the output of the training structure; and activating the third branch in response to the number of input channels of the training structure being equal to the number of output channels.

[0019] In some embodiments of this disclosure, the computational compression ratio K between the inference structure and the training structure satisfies: K = a / b, where a is the number of input channels of the first 1×1 convolutional kernel in the second branch; and b is the number of output channels of the first 1×1 convolutional kernel in the second branch.

[0020] In some embodiments of this disclosure, the training structure further includes: a first branch and a second branch performing grouped convolution operations on the input features.

[0021] In some embodiments of this disclosure, constructing a lightweight convolution module further includes constructing a depth-separable convolution based on a 1×1 convolution kernel connected in series with a channel-by-channel convolutional layer.

[0022] In some embodiments of this disclosure, the generation unit is also used to construct an activation function layer that is cascaded with a depthwise separable convolution.

[0023] In some embodiments of this disclosure, the generation unit is also used to construct an attention submodule, wherein the attention submodule includes a cascaded 1×1 convolutional kernel and an activation function layer.

[0024] In some embodiments of this disclosure, the generation unit is further configured to take the output of the activation function layer concatenated with the depthwise separable convolution as the input of the attention submodule; and to determine the output of the lightweight convolution module based on the output of the attention submodule and the output of the activation function layer concatenated with the depthwise separable convolution.

[0025] In some embodiments of this disclosure, the processing unit is used to extract multi-scale feature maps based on a lightweight convolution module; construct a residual module based on the inference structure of the skip connection structure and the structure reparameterization module; and improve the image quality of the image to be processed by splicing the output of the residual module and multiple multi-scale feature maps.

[0026] In some embodiments of this disclosure, the training unit is used to input prior information into a pre-trained network to obtain trained prior information; and to concatenate the trained prior information with training image data information to form a dataset for training an image quality enhancement network.

[0027] In some embodiments of this disclosure, the pre-trained network includes an inference structure of cascaded structural reparameterization modules and an activation function layer.

[0028] In some embodiments of this disclosure, the prior information includes lens parameters and lens wear information.

[0029] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the image processing method described in any implementation of the second aspect.

[0030] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to perform the image processing method as described in any implementation of the second aspect.

[0031] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the image processing method as described in any implementation of the second aspect.

[0032] It should be understood that the descriptions in this section are not intended to identify key or important features of the embodiments of this disclosure, nor are they intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the descriptions below. Attached Figure Description

[0033] Other features, objects, and advantages involved in the embodiments of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. Wherein:

[0034] Figure 1 A flowchart of an image processing method provided for embodiments of this disclosure;

[0035] Figure 2 This is a schematic diagram of an image quality enhancement network according to an exemplary embodiment of the present disclosure;

[0036] Figure 3 This is a schematic diagram of a structure reparameterization module according to an exemplary embodiment of this disclosure;

[0037] Figure 4 This is a schematic diagram of a lightweight convolution module according to an exemplary embodiment of the present disclosure;

[0038] Figure 5 This is a schematic diagram of a pre-trained network according to an exemplary embodiment of the present disclosure;

[0039] Figure 6 A schematic diagram of a graphics processing system according to an exemplary embodiment of the present disclosure; and

[0040] Figure 7 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0041] To better understand this disclosure, various aspects of this disclosure will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this disclosure and are not intended to limit the scope of this disclosure in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0042] It should be noted that in this specification, the terms "first," "second," etc., are used only to distinguish one feature from another and do not imply any limitation on the features. Therefore, without departing from the teachings of this disclosure, the first branch discussed below may also be referred to as the second branch.

[0043] It should also be understood that expressions such as “comprising,” “including,” “having,” “containing,” and / or “comprising” are open-ended rather than closed expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Additionally, the use of “exemplary” is intended to refer to an example or illustration.

[0044] Unless otherwise specified, all terms used in this disclosure (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that terms (e.g., those defined in common dictionaries) should be understood to have the meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense, unless expressly so specified in this disclosure.

[0045] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. Furthermore, unless explicitly limited or contradicted by the context, the specific steps included in the methods described in this disclosure are not limited to the order in which they are described, but can be performed in any order or in parallel. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0046] Figure 1 A flowchart of an image processing method 1000 provided for an embodiment of this disclosure.

[0047] like Figure 1 As shown, the image processing method 1000 may include the following steps:

[0048] S1: Construct an image quality enhancement network including a structure reparameterization module, wherein the structure reparameterization module has a training structure and an inference structure in the training phase and the inference phase, respectively. Constructing the structure reparameterization module includes using structure reparameterization technology to transform multiple branches in the training structure into a single branch structure in the inference structure; and constructing an image quality enhancement network including a structure reparameterization module includes constructing a channel-wise convolutional layer of a lightweight convolutional module in the image quality enhancement network based on the inference structure.

[0049] S2: Based on the image quality enhancement network, extract multi-scale feature maps of the image to be processed, and improve the image quality of the image to be processed based on the multi-scale feature maps.

[0050] The following will describe in detail each step of the image processing method 1000 described above, with reference to the accompanying drawings.

[0051] Step S1

[0052] Figure 2This is a schematic diagram of an image quality enhancement network 400 according to an exemplary embodiment of the present disclosure. Figure 3 This is a schematic diagram of a structure reparameterization module 100 according to an exemplary embodiment of the present disclosure.

[0053] like Figures 1-3 As shown, in some embodiments of this disclosure, the image quality enhancement network 400 may include a structure reparameterization module 100. The structure reparameterization module 100 has different structures in the training phase and the inference phase, such as a training structure 101 in the training phase and an inference structure 102 in the inference phase.

[0054] Therefore, constructing a structural reparameterization module involves using structural reparameterization techniques to transform multiple branches in the training structure into a single branch in the inference structure. By reparameterizing the model during the inference phase, the structural reparameterization module uses different network architectures during network training and inference. This significantly reduces the number of parameters in the image enhancement network including the structural reparameterization module, allowing the image enhancement network model to be deployed on low-computing-power devices. Furthermore, replacing conventional convolutional kernel structures with the structural reparameterization module and its lightweight convolutional module in the inference structure enhances the feature extraction capabilities of the image enhancement network. By introducing lightweight reparameterization and convolutional modules, image quality is improved while reducing the number of parameters, compressing computational power, and decreasing inference time.

[0055] Specifically, such as Figure 3 As shown, the training structure 101 of the structure reparameterization module 100 during the training phase may include multiple branches, such as a first branch 1 and a second branch 2. Optionally, the first branch 1 may include a 1×1 convolutional kernel. The second branch 2 may include a 1×1 convolutional kernel, an n×n convolutional kernel, and a 1×1 convolutional kernel connected in series, where n is a positive integer greater than 1. For example, the second branch 2 may include a 1×1 convolutional kernel, a 3×3 convolutional kernel, and a 1×1 convolutional kernel connected in series; or, the second branch 2 may include a 1×1 convolutional kernel, a 5×5 convolutional kernel, and a 1×1 convolutional kernel connected in series.

[0056] The receptive field of a 5×5 convolutional kernel is larger than that of a 3×3 convolutional kernel. However, the 5×5 convolutional kernel requires more parameters. Therefore, considering the model complexity and training speed, it is advisable to cascade 1×1 convolutional kernels at both ends of the 3×3 convolutional kernel to construct the second branch 2. Furthermore, the 3×3 convolutional kernel has a higher degree of non-linearity, allowing it to represent more complex functions and extract finer features, which is beneficial for improving image quality.

[0057] During the training phase, the input training image feature data of the training structure 101 of the structure reparameterization module 100 can be represented by (h*w*a), where h and w can be the height and width of the training image feature data, respectively, and a is the number of input channels of the training structure 101.

[0058] Alternatively, the first branch 1 can be constructed from a single 1×1 convolution. Through this branch, the original input data (e.g., training image feature data) can be convolved from a-channel size to c-channel size feature data. During this process, the height and width of the feature data remain unchanged. Therefore, the first branch 1 can increase the computational power of the training structure 101 of the structure reparameterization module 100 and enhance the function of certain feature combination relationships.

[0059] In the second branch 2, the first 1×1 convolution amplifies the computational power, increasing the feature data from h*w*a to h*w*b, where b is the number of output channels of the first 1×1 convolution. This provides more input data for the subsequent 3×3 convolution, which is the core module of the entire training structure 101 and is used for extracting information from the target features. The input and output feature data sizes of this convolution kernel remain unchanged. A second 1×1 convolution is then concatenated after the 3×3 kernel, which can be used to adjust the b channels of data to c channels, effectively compressing the channels.

[0060] Furthermore, during training, the outputs of multiple branches are summed based on weights to form the output of the training structure. Some of these branches can increase the computational power of the network model, improving its expressive power. Different weight values ​​can be assigned to different branches according to the image quality enhancement requirements of the actual image to be processed, thereby enhancing the network model's performance on image quality enhancement tasks. Additionally, reparameterization techniques are used to transform multiple branches in the training structure into single-branch structures in the inference structure, enabling these single-branch structures to achieve the same results for the aforementioned image quality enhancement tasks.

[0061] Furthermore, both the first branch 1 and the second branch 2 can perform grouped convolution operations on the input features. Grouped convolution can be understood as dividing the feature map into multiple segments and using independent convolution kernels to operate on each segment along the channel dimension, thereby reducing the number of parameters and enabling parallel processing of multiple feature maps. For example, g can be used to represent the number of convolution groups. Taking the first branch 1 as an example, when g = k = 1, the first branch 1 performs normal convolution; when g = k > 1, the first branch 1 can perform grouped convolution operations, dividing the input training image feature data into multiple segments and using independent convolution kernels to operate on each segment along the channel dimension, thereby reducing the number of parameters and enabling parallel processing of multiple feature maps. Through this first branch 1, the original input data can be convolved from a-channel size to c-channel size feature data.

[0062] Furthermore, the training structure 101 of the structure reparameterization module 100 during the training phase may also include a third branch 3, which can be a skip connection structure. The third branch 3 can be activated in response to the number of input channels 'a' of the training structure 101 equaling its number of output channels 'c'. During training, the first branch 1 and the second branch 2 can increase the network computing power of the structure reparameterization module, improve its expressive power, and ultimately enhance the performance of the image quality enhancement network including the structure reparameterization module for image quality enhancement-related tasks. The third branch 3 can reduce the probability of gradient vanishing and gradient exploding problems caused by an excessively large or deep network, provided that the number of input channels and output channels of the training structure of the structure reparameterization module are the same.

[0063] It should be noted that, for the sake of clarity, Figure 3 The structural reparameterization module 100 shown includes its optional third branch 3. Notably, the optional third branch 3 is shown with a dashed line.

[0064] Optionally, the outputs of multiple branches can be summed based on their weights to obtain the output of training structure 101. For example, the outputs of the first branch 1, the second branch 2, and the third branch 3 can be summed by weighting them within their channels to obtain the output h*w*c of training structure 101.

[0065] Furthermore, structural reparameterization techniques can be used to transform multiple branches in the training structure 101 into a single branch structure in the inference structure 102. Optionally, the inference structure 102 may consist of only a structurally reparameterized 3×3 convolution. 3×3 convolutions offer better optimization and computational density on both CPUs and GPUs, thus effectively utilizing computational resources and increasing inference speed while enhancing representation capabilities.

[0066] Alternatively, the inference structure 102 can perform grouped convolution operations on the input features. For example, g can be used to represent the number of convolution groups. When g = k = 1, the reparameterized 3×3 convolution is a normal convolution; when g = k > 1, the reparameterized 3×3 convolution can perform grouped convolution operations, dividing the input feature map feature data into multiple segments and using independent convolution kernels to operate on each segment along the channel dimension, thereby reducing the number of parameters and enabling parallel processing of multiple feature maps.

[0067] The performance of the training and inference structures remains consistent before and after structural reparameterization, but the computational power will be significantly reduced. The specific computational power compression ratio depends on the ratio between the number of input channels of the training structure and the number of output channels of the first 1×1 convolution. In other words, the computational power compression ratio K of the inference structure 102 and the training structure 101 satisfies: K = a / b, where a is the number of input channels of the training structure 101; and b is the number of output channels of the first 1×1 convolution kernel in the second branch 2.

[0068] Figure 4 This is a schematic diagram of a lightweight convolution module 200 according to an exemplary embodiment of the present disclosure.

[0069] like Figure 2 and Figure 4 As shown, in some embodiments of this disclosure, the image quality enhancement network 400 further includes a lightweight convolutional module 200. Optionally, step S1, constructing the image quality enhancement network 400 including the structure reparameterization module 100, further includes: constructing the lightweight convolutional module 200.

[0070] As an option, such as Figures 2-4 As shown, constructing a lightweight convolutional module 200 may include an inference structure 102 based on a structure reparameterization module 100, and constructing channel-wise convolutional layers of depth-separable convolutional submodules 201 in the lightweight convolutional module 200.

[0071] Depthwise separable convolution is a convolutional method used for lightweight models, specifically including channel-wise convolution and pointwise convolution. Therefore, to reduce computational cost and inference time in image enhancement networks, depthwise separable convolution submodules can be incorporated. These submodules can use separate kernels for different channels, reducing the number of kernels and channels, thus sustainably improving the overall performance of the image enhancement network without introducing additional complexity.

[0072] Specifically, the number of convolutional groups in the structure reparameterization module 100 of the lightweight convolutional module 200 can be set to be equal to the number of input channels of the lightweight convolutional module 200, thereby constructing a channel-wise convolutional layer of the depth separable convolutional submodule 201 in the lightweight convolutional module 200.

[0073] In addition, the construction of the lightweight convolution module 200 also includes: constructing a pointwise convolutional layer of the depth-separable convolutional submodule 201 in the lightweight convolution module 200 based on a 1×1 convolutional kernel connected in series with the channel-wise convolutional layer.

[0074] Furthermore, to incorporate nonlinear features into the lightweight convolutional module 200 to improve the feature extraction performance of the image enhancement network 400, which includes the lightweight convolutional module 200, and increase the network's expressive power, an activation function layer can be set in the lightweight convolutional module 200 and connected in series with the depthwise separable convolutional submodule 201. Alternatively, the activation function may include, but is not limited to, ReLU activation function, LeakyReLU activation function, Sigmoid activation function, or Tanh activation function.

[0075] In image enhancement networks, replacing standard convolutions with depthwise separable convolutions significantly reduces model computational complexity. However, this reduction in model parameters and computational cost inevitably leads to a decrease in model expressiveness. To maintain high performance while keeping the image enhancement network lightweight, this implementation provides a lightweight convolution module that, within the depthwise separable convolution model architecture, further introduces a structure reparameterization module transformed from structure reparameterization technology. This achieves the effect of enhancing model expressiveness without introducing additional computational complexity during the inference phase.

[0076] Furthermore, the lightweight convolutional module 200 may also include an attention submodule 202. Optionally, the attention submodule 202 can be constructed by concatenating a 1×1 convolutional kernel and an activation function layer. The attention submodule 202 compresses the number of feature map channels to a single channel through 1×1 convolution and performs further nonlinear transformations on the image features through a nonlinear activation function layer. This can compensate for the performance loss caused by computational compression in the depthwise separable convolutional submodule and improve the expressive power of the lightweight convolutional module.

[0077] Optionally, such as Figure 4 As shown, the output of the lightweight convolutional module 200 can be determined based on the output of the attention submodule 202 and the output of the activation function layer connected in series with the depthwise separable convolutional submodule 201. Alternatively, the product of the nonlinear output of the depthwise separable convolutional submodule 201 and the nonlinear output of the attention submodule 202 can be used as the final output of the lightweight convolutional module 200.

[0078] Refer again Figure 2 and Figure 4In some embodiments of this disclosure, downsampling convolutional layers and lightweight convolutional modules 200 can be used to perform downsampling and convolution processing on the feature data (e.g., W*H*N) of the input image to be processed, to obtain multiple feature maps of different sizes (hereinafter referred to as multi-scale feature maps). A multi-scale feature map can be understood as a visualized feature map containing the extracted features and corresponding to different downsampling scales, obtained after the image to be processed has been downsampled at different scales and then feature extracted by the lightweight convolutional module. The image quality of the image to be processed can be improved based on multiple multi-scale feature maps, where W is the width of the image to be processed, H is the height of the image to be processed, and N is the number of input channels of the image quality enhancement network 400, in the feature data W*H*N of the image to be processed.

[0079] Figure 5 This is a schematic diagram of a pre-trained network 400-1 according to an exemplary embodiment of the present disclosure.

[0080] like Figure 1 and Figure 5 As shown, in some embodiments of this disclosure, the image processing method further includes: training an image quality enhancement network. Training the image quality enhancement network may include: inputting prior information into a pre-trained network 400-1 to obtain trained prior information; and concatenating the trained prior information with training image data information to form a training image quality enhancement network 100 (e.g., ...). Figure 2 The dataset shown is a dataset that...

[0081] In addition to the image information to be processed, the input data of the image enhancement network 100 may also contain a large amount of prior information, which is information added manually. For example, prior information may include lens parameters and lens wear information. Prior information may contain some redundancy. Therefore, to reduce the computational power of the image enhancement network 100 and enable it to be trained more efficiently, the prior information can be preprocessed before training the network.

[0082] Optionally, the pre-trained network 400-1 used for preprocessing may include a cascaded structure reparameterization module 100 and an activation function layer.

[0083] Prior information can be represented as W*H*C_f1, where W is the width of the image to be processed (or the training image), H is the height of the image to be processed, and C_f1 is the number of input channels for the prior information. Training image information can be represented as W*H*C_i, where W is the width of the image to be processed, H is the height of the image to be processed, and C_i is the number of input channels for the training image information. Optionally, the number of input channels for the prior information can be greater than the number of input channels for the training image information. Alternatively, multiple dimensions of data can be used to represent the parameters and wear information of the lens used to capture the image to be processed (or the training image). The prior information representing the lens parameters and wear information can be processed into trained prior information W*H*C_f2 through a pre-trained network. The number of input channels for the trained prior information is denoted by C_f2, which is the same as the number of input channels for the training image information. Therefore, a dataset for training the image quality enhancement network can be formed by concatenating channels.

[0084] Optionally, the pre-trained network 400-1 may include multiple sets of cascaded structural reparameterization modules 100 and activation function layers, thereby generating the trained prior information W*H*C_f2 through iterative processes.

[0085] In addition, such as Figure 3 and Figure 5 As shown, as an option, the structure reparameterization module 100 included in the pre-trained network 400-1 can be the inference structure 102 in the inference stage. This prevents the image enhancement network from generating additional parameters and from increasing computational complexity. Alternatively, the structure reparameterization module 100 included in the pre-trained network 400-1 can also be the training structure 101 in the training stage. This can increase the performance of the image enhancement network and achieve better restoration results. Furthermore, the parameters of the training structure 101 included in the pre-trained network 400-1 need to be adjusted according to the difficulty of the image enhancement task, and this disclosure does not limit this.

[0086] It should be noted that prior information only needs to be considered as part of the dataset for training the image enhancement network after changing the lens. When the prior information is fixed or not updated, it does not need to be considered during the training process, which simplifies the training process.

[0087] Step S2

[0088] Refer again Figure 2In some embodiments of this disclosure, step S2, based on the trained image quality enhancement network, extracts multi-scale feature maps of the image to be processed, and enhances the image quality of the image to be processed based on the multi-scale feature maps. This may include: performing downsampling and convolution processing on the feature data W*H*N of the input image to be processed using multiple downsampling convolutional layers and multiple lightweight convolutional modules 200 to obtain multiple multi-scale feature maps; and processing the multiple multi-scale feature maps based on multiple upsampling convolutional layers and multiple lightweight convolutional modules 200 to reconstruct the image to be processed.

[0089] Compared with the input image to be processed, the image quality of the reconstructed image is improved. The improvement in image quality can be understood as the need for super-resolution or image quality enhancement of the image to be processed has been met. Super-resolution can be understood as magnifying the reconstructed image. The image quality of the reconstructed image remains unchanged or is improved compared with the image quality of the image to be processed. The image quality enhancement can be understood as the size of the reconstructed image being the same as the size of the image to be processed, but the clarity of the reconstructed image is improved.

[0090] In addition, in order to introduce nonlinear features into the lightweight convolutional module 200 to improve the feature extraction effect of the image quality enhancement network 400 including the lightweight convolutional module 200 and increase the expressive power of the network, activation function processing can be applied to the output of each lightweight convolutional module 200 in the image quality enhancement network 400 during the process of obtaining multiple multi-scale feature maps.

[0091] Alternatively, the activation function may include, but is not limited to, the ReLU activation function, the LeakyReLU activation function, the Sigmoid activation function, or the Tanh activation function.

[0092] After obtaining multiple multi-scale feature maps, the multi-scale feature maps obtained in the next layer can be processed by convolution and upsampling based on the upsampling convolutional layer and the lightweight convolutional module 200, and the processed output can be concatenated with the multi-scale feature maps obtained in the previous layer in terms of channel dimension.

[0093] In some embodiments of this disclosure, the image quality enhancement network 400 further includes a residual module 300. Optionally, the inference structure 102 based on the jump connection structure and the structure reparameterization module 100 (e.g., Figure 3 As shown, a residual module 300 can be constructed. The residual module, which includes the structure reparameterization module, is also a lightweight convolutional block that does not generate additional parameters or increase computational complexity. The idea behind residuals is to remove identical main parts, thereby highlighting subtle changes. Therefore, the mapping after introducing the residual module into the image enhancement network is more sensitive to changes in the output, which can improve the image quality enhancement effect of the image enhancement network on the image being processed.

[0094] Optionally, the image quality of the image to be processed can be improved by combining the output of the residual module and the output of multiple multi-scale feature maps. Specifically, the final output after the above concatenation operation can be added to the output of the residual module to obtain the final output of the image quality enhancement network 400, where the number of channels in the final output can be represented by C_O.

[0095] It should be noted that, Figure 3 The "A" in the symbol can be used to represent concatenation operations along the channel dimension; the "+" symbol can be used to represent addition operations of features. Furthermore, Figure 3 The diagram only schematically illustrates three-layer downsampling and three-layer upsampling processes; however, those skilled in the art should understand that the number of downsampling convolutional layers, upsampling convolutional layers, and lightweight convolutional modules can be set according to the actual task of the image enhancement network.

[0096] Therefore, the image processing method according to at least one embodiment of this disclosure utilizes structural reparameterization technology to transform multiple branches in the training structure into a single branch structure in the inference structure. By reparameterizing the model during the inference stage, the structural reparameterization module uses different network architectures in the network training and inference stages. This significantly reduces the number of parameters in the image quality enhancement network including the structural reparameterization module, enabling the image quality enhancement network model to be deployed on low-computing-power terminal devices. Furthermore, replacing the conventional convolutional kernel structure with the structural reparameterization module and a lightweight convolutional module in the inference structure including the structural reparameterization module enhances the feature extraction capability of the image quality enhancement network. By introducing lightweight reparameterization and convolutional modules, image quality improvement is achieved while reducing the number of parameters in the image quality enhancement network model, compressing the model's computational power, and reducing the model's inference time.

[0097] Table 1 below shows a comparison of the model computing power and computing power compression ratio between the image quality enhancement network provided in at least one embodiment of this disclosure and the ResUnet network, while achieving the same image quality improvement effect.

[0098] ResUnet is a U-shaped convolutional neural network structure with residual structure blocks. It is widely used in image recognition and classification and performs well. However, it is not well adapted to image quality improvement tasks. Although it can reduce the computational load of the model at the computing power level, the final image quality improvement effect may be significantly reduced.

[0099] Model computational power (Flops(G)) Computing power compression ratio ResUnet network 2000 - Improved image quality 100 5％

[0100] Table 1

[0101] As shown in Table 1, compared with the ResUnet network, the image quality enhancement network provided in at least one embodiment of this disclosure exhibits better performance in both model computing power and computing power compression ratio while achieving the same image quality improvement effect. For example, the ResUnet network has a model computing power of 2000 GFlops (floats per second), while the image quality enhancement network provided in at least one embodiment of this disclosure has a model computing power of only 100 GFlops. This demonstrates that the image quality enhancement network provided in at least one embodiment of this disclosure significantly reduces computing power while improving image quality. Furthermore, the ResUnet network does not have a high computing power compression ratio, while the image quality enhancement network provided in at least one embodiment of this disclosure, due to the inclusion of a structural reparameterization module, can achieve a computing power compression ratio of 5%.

[0102] Figure 6 This is a schematic diagram of a graphics processing system 500 according to an exemplary embodiment of the present disclosure. Figure 6 As shown, one embodiment of this disclosure provides a graphics processing system 500. The graphics processing system 500 may include a generation unit 510 and a processing unit 530. The generation unit 510 is used to construct an image quality enhancement network including a structure reparameterization module; and the processing unit 530 is used to extract multi-scale feature maps of the image to be processed based on the trained image quality enhancement network, and to enhance the image quality of the image to be processed based on the multi-scale feature maps. The structure reparameterization module has different structures in the training and inference phases, for example, a training structure in the training phase and an inference structure in the inference phase. Constructing the structure reparameterization module includes using structure reparameterization techniques to transform multiple branches in the training structure into a single branch structure in the inference structure. Constructing the image quality enhancement network including the structure reparameterization module includes constructing channel-wise convolutional layers of a lightweight convolutional module in the image quality enhancement network based on the inference structure.

[0103] Therefore, the image processing system according to at least one embodiment of this disclosure utilizes structural reparameterization technology to transform multiple branches in the training structure into a single branch structure in the inference structure. By reparameterizing the model during the inference stage, the structural reparameterization module uses different network architectures in the network training and inference stages. This significantly reduces the number of parameters in the image quality enhancement network including the structural reparameterization module, enabling the image quality enhancement network model to be deployed on low-computing-power terminal devices. Furthermore, replacing the conventional convolutional kernel structure with the structural reparameterization module and a lightweight convolutional module in the inference structure including the structural reparameterization module enhances the feature extraction capability of the image quality enhancement network. By introducing lightweight reparameterization and convolutional modules, image quality improvement is achieved while reducing the number of parameters in the image quality enhancement network model, compressing the model's computational power, and reducing the model's inference time.

[0104] Optionally, the graphics processing system 500 may also include a training unit 520. It should be noted that, for clarity, Figure 6 The illustrated graphics processing system 500 includes, in an optional configuration, a training unit 520. Notably, the optional training unit 520 is shown with a dashed box, and its connections to other units are indicated by dashed lines.

[0105] like Figure 3 and Figure 6 As shown, in some embodiments of this disclosure, the first branch 1 of the multiple branches may include a 1×1 convolution kernel, and the second branch 2 of the multiple branches may include a 1×1 convolution kernel, an n×n convolution kernel and a 1×1 convolution kernel connected in series, where n is a positive integer greater than 1. The training unit 520 is also used to train the training structure, which may include: adding the outputs of the multiple branches based on weights to obtain the output of the training structure.

[0106] For example, the second branch 2 may include a 1×1 convolution kernel, a 3×3 convolution kernel, and a 1×1 convolution kernel connected in series; or, the second branch 2 may include a 1×1 convolution kernel, a 5×5 convolution kernel, and a 1×1 convolution kernel connected in series.

[0107] The receptive field of a 5×5 convolutional kernel is larger than that of a 3×3 convolutional kernel. However, the 5×5 convolutional kernel requires more parameters. Therefore, considering the model complexity and training speed, it is advisable to cascade 1×1 convolutional kernels at both ends of the 3×3 convolutional kernel to construct the second branch 2. Furthermore, the 3×3 convolutional kernel has a higher degree of non-linearity, allowing it to represent more complex functions and extract finer features, which is beneficial for improving image quality.

[0108] During training, the outputs of multiple branches are summed based on weights to form the output of the training structure. Some of these branches can increase the computational power of the network model, improving its expressive power. Different weight values ​​can be assigned to different branches according to the image quality enhancement requirements of the actual image to be processed, thereby enhancing the network model's performance on image quality enhancement tasks. Furthermore, reparameterization techniques are used to transform multiple branches in the training structure into single-branch structures in the inference structure, enabling these single-branch structures to achieve the same results for the aforementioned image quality enhancement tasks.

[0109] Alternatively, the first branch 1 can be constructed from a single 1×1 convolution. Through this branch, the original input data (e.g., training image feature data) can be convolved from a-channel size to c-channel size feature data. During this process, the height and width of the feature data remain unchanged. Therefore, the first branch 1 can increase the computational power of the training structure 101 of the structure reparameterization module 100 and enhance the function of certain feature combination relationships.

[0110] In the second branch 2, the first 1×1 convolution amplifies the computational power, increasing the feature data from h*w*a to h*w*b, where b is the number of output channels of the first 1×1 convolution. This provides more input data for the subsequent 3×3 convolution, which is the core module of the entire training structure 101 and is used for extracting information from the target features. The input and output feature data sizes of this convolution kernel remain unchanged. A second 1×1 convolution is then concatenated after the 3×3 kernel, which can be used to adjust the b channels of data to c channels, effectively compressing the channels.

[0111] Furthermore, both the first branch 1 and the second branch 2 can perform grouped convolution operations on the input features. Grouped convolution can be understood as dividing the feature map into multiple segments and using independent convolution kernels to operate on each segment along the channel dimension, thereby reducing the number of parameters and enabling parallel processing of multiple feature maps. For example, g can be used to represent the number of convolution groups. Taking the first branch 1 as an example, when g = k = 1, the first branch 1 performs normal convolution; when g = k > 1, the first branch 1 can perform grouped convolution operations, dividing the input training image feature data into multiple segments and using independent convolution kernels to operate on each segment along the channel dimension, thereby reducing the number of parameters and enabling parallel processing of multiple feature maps. Through this first branch 1, the original input data can be convolved from a channel size to c channel size feature data.

[0112] In some embodiments of this disclosure, the third branch 3 among multiple branches may be a jump connection structure, and the training structure further includes: in response to the number of input channels of the training structure being equal to the number of output channels, the third branch is started.

[0113] During training, the first branch 1 and the second branch 2 can increase the network computing power of the structure reparameterization module, improve the expressive power of the structure reparameterization module, and ultimately improve the effect of the image quality enhancement network including the structure reparameterization module on image quality enhancement related tasks; the third branch 3 can reduce the probability of gradient vanishing and gradient explosion caused by the network being too large and too deep, when the number of input channels and output channels of the training structure of the structure reparameterization module are the same.

[0114] Furthermore, the computational compression ratio K between the inference structure and the training structure of the structure reparameterization module satisfies: K = a / b, where a is the number of input channels of the first 1×1 convolutional kernel in the second branch; and b is the number of output channels of the first 1×1 convolutional kernel in the second branch.

[0115] Furthermore, structural reparameterization is used to transform multiple branches in the training structure into a single branch structure in the inference structure. Optionally, the inference structure 102 may consist of only a single reparameterized 3×3 convolution. 3×3 convolutions offer better optimization and computational density on both CPUs and GPUs, thus effectively utilizing computational resources and increasing inference speed while enhancing representation capabilities.

[0116] As an alternative, the inference structure can perform grouped convolution operations on the input features. For example, g can be used to represent the number of convolution groups. When g = k = 1, the reparameterized 3×3 convolution is a normal convolution; when g = k > 1, the reparameterized 3×3 convolution can perform grouped convolution operations, dividing the input feature map feature data into multiple segments and using independent convolution kernels to operate on each segment along the channel dimension, thereby reducing the number of parameters and enabling parallel processing of multiple feature maps.

[0117] Optionally, such as Figure 2 , Figure 3 and Figure 6 As shown, the lightweight convolution module 200 also includes: constructing a pointwise convolutional layer with depth separable convolution based on a 1×1 convolutional kernel connected in series with the channel-wise convolutional layer.

[0118] To reduce computational power and inference time in the image enhancement network, a depthwise separable convolutional submodule 201 (e.g., ...) can be added to the image enhancement network. Figure 4 (As shown). The depthwise separable convolutional submodule can use separate kernels for different channels, thereby reducing the number of convolutional kernels and the number of convolutional channels. This can sustainably improve image quality and enhance the overall performance of the network without introducing additional complexity.

[0119] Specifically, the number of convolutional groups in the structure reparameterization module 100 of the lightweight convolutional module 200 can be set to be equal to the number of input channels of the lightweight convolutional module 200, thereby constructing a channel-wise convolutional layer of the depthwise separable convolutional submodule 201 in the lightweight convolutional module 200. Furthermore, a 1×1 convolution can be cascaded after the channel-wise convolutional layer to construct a pointwise convolutional layer of the depthwise separable convolutional submodule 201.

[0120] Furthermore, to incorporate non-linear features into the lightweight convolutional module 200 to improve the feature extraction performance of the image enhancement network 400, which includes the lightweight convolutional module 200, and increase the network's expressive power, an activation function layer connected in series with the pointwise convolutional layer can be set in the lightweight convolutional module 200. Alternatively, the activation function may include, but is not limited to, ReLU activation function, LeakyReLU activation function, Sigmoid activation function, or Tanh activation function.

[0121] In image enhancement networks, replacing standard convolutions with depthwise separable convolutions significantly reduces model computational complexity. However, this reduction in model parameters and computational cost inevitably leads to a decrease in model expressiveness. To maintain high performance while keeping the image enhancement network lightweight, this implementation provides a lightweight convolution module that, within the depthwise separable convolution model architecture, further introduces a structure reparameterization module transformed from structure reparameterization technology. This achieves the effect of enhancing model expressiveness without introducing additional computational complexity during the inference phase.

[0122] In addition, such as Figure 4 As shown, the lightweight convolutional module 200 may further include an attention submodule 202. Optionally, the attention submodule 202 can be constructed by cascading a 1×1 convolutional kernel and an activation function layer. The attention submodule 202 compresses the number of feature map channels to a single channel through 1×1 convolution and performs further nonlinear transformations on the image features through a nonlinear activation function layer. This can compensate for the performance loss caused by computational compression in the depthwise separable convolutional submodule and improve the expressive power of the lightweight convolutional module.

[0123] Optionally, the output of the lightweight convolutional module 200 can be determined based on the output of the attention submodule 202 and the output of the activation function layer connected in series with the depthwise separable convolutional submodule 201. Alternatively, the product of the nonlinear output of the depthwise separable convolutional submodule 201 and the nonlinear output of the attention submodule 202 can be used as the final output of the lightweight convolutional module 200.

[0124] In some embodiments of this disclosure, the processing unit 530 is used to extract multi-scale feature maps based on the lightweight convolution module 200; construct a residual module 300 based on the jump connection structure and the inference structure 102 of the structure reparameterization module 100; and improve the image quality of the image to be processed by splicing the output of the residual module 300 and multiple multi-scale feature maps.

[0125] The residual module of the inference structure, including the structural reparameterization module, is also a lightweight convolutional block that does not generate additional parameters or increase computational complexity. The idea behind residuals is to remove identical main parts, thereby highlighting subtle changes. Therefore, the mapping after introducing the residual module into the image enhancement network is more sensitive to changes in the output, which can improve the image quality enhancement effect of the image enhancement network on the image being processed.

[0126] In some embodiments of this disclosure, such as Figure 2 and Figure 5As shown, the training unit 520 is used to input prior information into the pre-trained network 400-1 to obtain the trained prior information; and to concatenate the trained prior information with the training image data information to form a dataset for training the image quality enhancement network 400.

[0127] Prior information can be represented as W*H*C_f1, where W is the width of the image to be processed (or the training image), H is the height of the image to be processed, and C_f1 is the number of input channels for the prior information. Training image information can be represented as W*H*C_i, where W is the width of the image to be processed, H is the height of the image to be processed, and C_i is the number of input channels for the training image information. Optionally, the number of input channels for the prior information can be greater than the number of input channels for the training image information. Alternatively, multiple dimensions of data can be used to represent the parameters and wear information of the lens used to capture the image to be processed (or the training image). The prior information representing the lens parameters and wear information can be processed into trained prior information W*H*C_f2 through a pre-trained network. The number of input channels for the trained prior information is denoted by C_f2, which is the same as the number of input channels for the training image information. Therefore, a dataset for training the image quality enhancement network can be formed by concatenating channels.

[0128] Optionally, the pre-trained network 400-1 may include multiple sets of cascaded structural reparameterization modules 100 and activation function layers, thereby generating the trained prior information W*H*C_f2 through iterative processes.

[0129] In addition, such as Figure 3 and Figure 5 As shown, as an option, the structure reparameterization module 100 included in the pre-trained network 400-1 can be the inference structure 102 in the inference stage. This prevents the image enhancement network from generating additional parameters and from increasing computational complexity. Alternatively, the structure reparameterization module 100 included in the pre-trained network 400-1 can also be the training structure 101 in the training stage. This can increase the performance of the image enhancement network and achieve better restoration results. Furthermore, the parameters of the training structure 101 included in the pre-trained network 400-1 need to be adjusted according to the difficulty of the image enhancement task, and this disclosure does not limit this.

[0130] Therefore, the image processing system according to at least one embodiment of this disclosure utilizes structural reparameterization technology to transform multiple branches in the training structure into a single branch structure in the inference structure. By reparameterizing the model during the inference stage, the structural reparameterization module uses different network architectures in the network training and inference stages. This significantly reduces the number of parameters in the image quality enhancement network including the structural reparameterization module, enabling the image quality enhancement network model to be deployed on low-computing-power terminal devices. Furthermore, replacing the conventional convolutional kernel structure with the structural reparameterization module and a lightweight convolutional module in the inference structure including the structural reparameterization module enhances the feature extraction capability of the image quality enhancement network. By introducing lightweight reparameterization and convolutional modules, image quality improvement is achieved while reducing the number of parameters in the image quality enhancement network model, compressing the model's computational power, and reducing the model's inference time.

[0131] Figure 7 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0132] like Figure 7 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0133] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0134] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform image processing methods by any other suitable means (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0139] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0140] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0141] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0142] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An image processing method, characterized in that, include: Construct an image quality enhancement network that includes a structure reparameterization module; as well as Based on the image quality enhancement network, multi-scale feature maps of the image to be processed are extracted, and the image quality of the image to be processed is improved based on the multi-scale feature maps. The structure reparameterization module has a training structure and an inference structure in the training and inference phases, respectively. Constructing the structure reparameterization module includes using structure reparameterization technology to transform multiple branches in the training structure into a single branch structure in the inference structure; and Constructing an image quality enhancement network that includes a structure reparameterization module involves building channel-wise convolutional layers of a lightweight convolutional module in the image quality enhancement network based on the inference structure.

2. The method according to claim 1, wherein, The first branch of the plurality of branches includes a 1×1 convolutional kernel, and the second branch of the plurality of branches includes a 1×1 convolutional kernel, an n×n convolutional kernel, and a 1×1 convolutional kernel connected in series, where n is a positive integer greater than 1. The third branch of the plurality of branches is a skip-connected structure. The method further includes: training the training structure, wherein training the training structure includes: The outputs of the multiple branches are summed based on their weights to obtain the output of the training structure; and The third branch is initiated in response to the number of input channels of the training structure being equal to the number of output channels.

3. The method according to claim 2, wherein, The computational compression ratio K between the inference structure and the training structure satisfies: K = a / b. Where a is the number of input channels of the training structure; and b is the number of output channels of the first 1×1 convolutional kernel in the second branch.

4. The method according to claim 2, wherein, The training structure also includes: The first branch and the second branch perform grouped convolution operations on the input features.

5. The method according to claim 1, wherein, The lightweight convolutional module also includes: A depth-separable convolution is constructed based on a 1×1 convolution kernel connected in series with the channel-by-channel convolutional layer.

6. The method according to claim 5, wherein, The lightweight convolutional module also includes: Construct an activation function layer that is cascaded with the depth-separable convolution.

7. The method according to claim 6, wherein, The lightweight convolutional module also includes: Construct an attention submodule, wherein the attention submodule comprises a cascaded 1×1 convolutional kernel and an activation function layer.

8. The method according to claim 7, wherein, The method further includes: The output of the activation function layer concatenated with the depthwise separable convolution is used as the input of the attention submodule; and The output of the lightweight convolution module is determined based on the output of the attention submodule and the output of the activation function layer concatenated with the depthwise separable convolution.

9. The method according to claim 1, wherein, Based on the image quality enhancement network, multi-scale feature maps of the image to be processed are extracted, and the image quality of the image to be processed is improved based on the multi-scale feature maps, including: Based on the lightweight convolutional module, the multi-scale feature map is extracted; Based on the jump connection structure and the inference structure of the structure reparameterization module, a residual module is constructed; and Based on the output of the residual module and the stitched output of multiple multi-scale feature maps, the image quality of the image to be processed is improved.

10. The method according to claim 1, wherein, The method further includes training the image quality enhancement network, wherein training the image quality enhancement network includes: Input the prior information into the pre-trained network to obtain the trained prior information; and The prior information after training is concatenated with the training image data to form a dataset used to train the image quality enhancement network.

11. The method according to claim 10, wherein, The pre-trained network includes the structure reparameterization module and the activation function layer connected in series.

12. The method according to claim 10, wherein, The prior information includes lens parameters and lens wear information.

13. An image processing system, characterized in that, include: A generation unit is used to construct an image quality enhancement network including a structure reparameterization module. The structure reparameterization module has a training structure and an inference structure in the training and inference phases, respectively. Constructing the structure reparameterization module includes using structure reparameterization technology to transform multiple branches in the training structure into a single branch structure in the inference structure. Constructing the image quality enhancement network including the structure reparameterization module includes constructing a channel-wise convolutional layer of a lightweight convolutional module in the image quality enhancement network based on the inference structure. as well as The processing unit is used to extract multi-scale feature maps of the image to be processed based on the image quality enhancement network, and to improve the image quality of the image to be processed based on the multi-scale feature maps.

14. The system according to claim 13, wherein, The first branch of the plurality of branches includes a 1×1 convolutional kernel, and the second branch of the plurality of branches includes a 1×1 convolutional kernel, an n×n convolutional kernel, and a 1×1 convolutional kernel connected in series, where n is a positive integer greater than 1. The third branch of the plurality of branches is a skip-connected structure. The image processing system further includes a training unit, which is used to train the training structure, wherein training the training structure includes: The outputs of the multiple branches are summed based on their weights to obtain the output of the training structure; and The third branch is initiated in response to the number of input channels of the training structure being equal to the number of output channels.

15. The system according to claim 14, wherein, The computational compression ratio K between the inference structure and the training structure satisfies: K = a / b. Where a is the number of input channels of the first 1×1 convolutional kernel in the second branch; and b is the number of output channels of the first 1×1 convolutional kernel in the second branch.

16. The system according to claim 14, wherein, The training structure also includes: The first branch and the second branch perform grouped convolution operations on the input features.

17. The system according to claim 13, wherein, The generation unit is also used to construct the depth-separable convolution of the lightweight convolution module based on a 1×1 convolution kernel connected in series with the channel-by-channel convolutional layer.

18. The system according to claim 17, wherein, The generation unit is also used to construct an activation function layer that is cascaded with the depthwise separable convolution.

19. The system according to claim 18, wherein, The generation unit is also used to construct an attention submodule, wherein the attention submodule includes a cascaded 1×1 convolutional kernel and an activation function layer.

20. The system according to claim 19, wherein, The generation unit is also used to take the output of the activation function layer concatenated with the depthwise separable convolution as the input of the attention submodule; as well as The output of the lightweight convolution module is determined based on the output of the attention submodule and the output of the activation function layer concatenated with the depthwise separable convolution.

21. The system according to claim 13, wherein, The processing unit is used to extract the multi-scale feature map based on the lightweight convolution module; Based on the jump connection structure and the inference structure of the structure reparameterization module, a residual module is constructed. as well as Based on the output of the residual module and the stitched output of multiple multi-scale feature maps, the image quality of the image to be processed is improved.

22. The system according to claim 13, wherein, The image processing system further includes a training unit, which is used to input prior information into a pre-trained network to obtain trained prior information. as well as The prior information after training is concatenated with the training image data to form a dataset used to train the image quality enhancement network.

23. The system according to claim 22, wherein, The pre-trained network includes the inference structure of the structure reparameterization module connected in series and the activation function layer.

24. The system according to claim 22, wherein, The prior information includes lens parameters and lens wear information.

25. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to cause the at least one processor to perform the image processing method according to any one of claims 1-12.

26. A computer-readable storage medium storing a computer program, characterized in that, The computer instructions are used to cause the computer to perform the image processing method according to any one of claims 1-12.

27. A computer program product comprising a computer program that, when executed by a processor, implements the image processing method according to any one of claims 1-12.

Images