Image processing method and device, electronic equipment, chip and medium

By using non-learning convolution kernel and weighted processing of learnable weights in convolutional neural networks, the number and complexity of network parameters are reduced, the problem of poor effect of low-level semantic tasks is solved, and the network's image quality improvement is improved. efficiency.

CN119027362APending Publication Date: 2024-11-26BEIJING X RING TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310596149.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, convolutional neural networks have weak effects in low-level semantic tasks, and the network parameters and complexity are high, which makes training difficult and difficult to effectively solve specific picture quality problems.

Method used

By using non-learning and well-functional convolution kernels and weighted processing with learnable weights, a structural network is built to reduce the amount of network parameters and complexity, improve the network convergence speed, and guide the network to learn corresponding features.

Benefits of technology

It effectively reduces the difficulty of network training, improves network convergence speed, and can effectively solve specific image quality problems, such as improvements in sharpness and clarity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027362A_ABST
    Figure CN119027362A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method and device, electronic equipment, a chip and a medium, and relates to the field of digital image processing, and the method comprises the steps: constructing a first structure network through a first convolution kernel and at least two second convolution kernels, the second convolution kernel comprises a non-learning convolution kernel with a function corresponding to the preset task; the image to be processed is processed through the first structure network to obtain a first output image, the problem that the low-level semantic task effect is weak is solved, the network parameter quantity and complexity are effectively reduced by using a non-learning and function-clear convolution kernel, and therefore the network training difficulty is reduced, the network convergence speed is increased, and the image quality is improved. And meanwhile, the network can be well guided to learn corresponding features, so that the network can effectively solve the specific image quality problem.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of digital image processing, and in particular to an image processing method, device, electronic device, chip and medium. Background Art

[0002] Designing a suitable Convolutional Neural Network (CNN) architecture in a practical application environment usually requires a lot of manual work or a lot of processing time. In the related art, a structural re-parameterization method is designed to improve the performance of CNN in practical applications. Among them, structural re-parameterization refers to first constructing a series of structures (generally used in the training phase) and converting their parameters into another set of parameters (generally used in the inference phase), thereby converting this series of structures into another series of structures. Summary of the invention

[0003] The present disclosure provides an image processing method, device, electronic device, chip and medium to solve the problem of weak effect of low-level semantic tasks in related technologies. By using non-learning and function-defined convolution kernels, the network parameter quantity and complexity are effectively reduced, thereby reducing the difficulty of network training and improving the network convergence speed. At the same time, the network can be well guided to learn corresponding features, so that the network can more effectively solve specific image quality problems.

[0004] A first aspect embodiment of the present disclosure proposes an image processing method, which includes: constructing a first structural network through a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel with a function corresponding to a preset task; processing the image to be processed through the first structural network to obtain a first output image.

[0005] In some embodiments of the present disclosure, processing an image to be processed by a first structural network to obtain a first output image includes: extracting a first feature of the image to be processed based on a first convolution kernel to obtain a first feature image; extracting a second feature of the image to be processed based on at least two second convolution kernels to obtain at least two second feature images; fusing the first feature image with at least two second feature images through a first weight coefficient and a second weight coefficient to obtain a third feature image, the first weight coefficient being the weight coefficient corresponding to the first convolution kernel, and the at least two second weight coefficients being the weight coefficients corresponding to at least two second convolution kernels; obtaining the first output image through an activation function and the third feature image.

[0006] In some embodiments of the present disclosure, the method further includes: determining a first weight coefficient and at least two second weight coefficients according to a gradient direction of the image to be processed.

[0007] In some embodiments of the present disclosure, the method further includes: converting the first structural network into a second structural network equivalently; and processing the image to be processed by the second structural network to obtain a second output image.

[0008] In some embodiments of the present disclosure, converting the first structural network into the second structural network equivalently includes: converting the first structural network into the second structural network equivalently by using a first weight coefficient and a second weight coefficient.

[0009] In some embodiments of the present disclosure, converting the first structural network into the second structural network equivalently through the first weight coefficient and the second weight coefficient includes: based on the first weight coefficient and at least two second weight coefficients, converting the first convolution kernel and at least two second convolution kernels into a third convolution kernel equivalently to obtain the second structural network.

[0010] In some embodiments of the present disclosure, processing the image to be processed by the second structure network to obtain the second output image includes: extracting the third feature of the image to be processed by the third convolution kernel to obtain the fourth feature image; and obtaining the second output image by the activation function and the fourth feature image.

[0011] A second aspect embodiment of the present disclosure proposes an image processing device, which includes: a construction unit, used to construct a first structural network through a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel with a function corresponding to a preset task; a processing unit, used to process the image to be processed through the first structural network to obtain a first output image.

[0012] The third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect embodiment of the present disclosure.

[0013] The fourth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure.

[0014] The fifth aspect embodiment of the present disclosure proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals including computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the method described in the first aspect embodiment of the present disclosure.

[0015] In summary, according to the image processing method proposed in the present invention, a first structural network is constructed by a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel with a function corresponding to a preset task; the image to be processed is processed by the first structural network to obtain a first output image, thereby solving the problem of weak effect of low-level semantic tasks, and by using non-learning and functionally clear convolution kernels, the amount and complexity of network parameters are effectively reduced, thereby reducing the difficulty of network training and improving the network convergence speed. At the same time, it can well guide the network to learn corresponding features, so that the network can more effectively solve specific image quality problems.

[0016] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0018] Figure 1 A flowchart of a method for re-parameterizing the structure of an ACB net[3] provided in an embodiment of the present disclosure;

[0019] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0020] Figure 3 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0021] Figure 4 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0022] Figure 5 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0023] Figure 6 A schematic diagram of a structure reparameterization method provided in an embodiment of the present disclosure;

[0024] Figure 7 A schematic diagram of the results of a structure reparameterization method provided in an embodiment of the present disclosure;

[0025] Figure 8 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure;

[0026] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions.

[0028] like Figure 1 The present invention relates to a structural reparameterization method using ACB net[3] in related technologies.

[0029] In the training phase, three learnable convolution kernels are used: 3x3, 1x3, and 3x1 to extract the features of the image to be processed. The output of the network is obtained by adding the obtained image feature maps point by point and then processing them with the activation function (ReLU).

[0030] In the inference stage, the principle of mathematical equivalence is used to first merge the three convolution kernels of 3x3, 1x3, and 3x1 learned in the training stage into one convolution kernel, and then the merged convolution kernel is used to convolve the image to be processed, and then the output structure of the network after the conversion is obtained after ReLU.

[0031] From the above, we can see that in the related technology, a network layer in the training phase extracts features through three convolution operations, but a network layer in the inference phase only extracts features through one convolution operation, and the weights of different convolution kernels are added together at 1:1:1, which saves the amount of calculation. At the same time, the training phase of the related technology uses three learnable convolution kernels to build a convolutional neural network in the training phase, which is suitable for solving abstract high-level semantic tasks.

[0032] However, the network parameters of learnable convolution kernels increase very quickly during training, which makes training difficult and easily leads to slow network convergence. At the same time, since low-level semantic tasks usually have clear image quality points, such as improving weak textures, sharpness, clarity, noise suppression, etc., related technologies can only constrain network learning through loss, and it is difficult to effectively guide the network to learn specific features to effectively solve specific image quality problems. Therefore, related technologies are less effective in solving low-level semantic tasks.

[0033] In order to solve the problems existing in the related art, the present disclosure proposes an image processing method by using a non-learning and functionally clear convolution kernel. Compared with related solutions, it can effectively reduce the number and complexity of network parameters, thereby reducing the difficulty of network training and improving the network convergence speed. Since the function of the convolution kernel is clear, it can well guide the lightweight intelligent super-resolution network to learn the corresponding features, so that the network can more effectively solve specific image quality problems. At the same time, learnable weights are used for weighted processing, so that the network can obtain better features from multiple feature branches.

[0034] The image processing method proposed in the present disclosure can be applied to image enhancement technology in the field of intelligent super-resolution technology (AISR), for example, in the fields of remote sensing, medical imaging, video surveillance, photography applications, etc. that require image processing; it can also be used for image enhancement technology in other fields such as AI-demo, AI-denoise, etc. The image processing method can be used in the process of structural reparameterization, mainly used to enhance the sharpness and clarity of lightweight intelligent super-resolution networks, and can also be used for other image quality improvements except sharpening and clarity with slight modifications.

[0035] The image processing method provided in this application is described in detail below with reference to the accompanying drawings.

[0036] Figure 2 Flow chart of an image processing method provided by an embodiment of the present disclosure. Figure 2 As shown, the image processing method includes steps 101-102.

[0037] Step 101, constructing a first structural network through a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel having a function corresponding to a preset task.

[0038] In an embodiment of the present disclosure, the first convolution kernel includes a learnable convolution kernel. The first structure network includes a convolution neural network used in the training phase of the reparameterization method.

[0039] The learnable parameters of the first convolution kernel can be determined according to the determined matrix size of the first convolution kernel. For example, when the first convolution kernel is a 3*3 learnable convolution kernel, the number of learnable parameters of the convolution kernel is 9. Similarly, when the first convolution kernel is a 5*5 learnable convolution kernel, the number of learnable parameters of the convolution kernel is 25.

[0040] In the present disclosure, since the second convolution kernel is a non-learning convolution kernel having a function corresponding to a preset task, the value of the parameter in the second convolution kernel is fixed. The preset task may refer to a specific task for improving the image quality of the current image to be processed; the function corresponding to the preset task may refer to a processing function determined for the specific task of improving the image quality. For example, using a high-frequency extraction convolution kernel can improve the sharpness and clarity of the image to be processed; using a bandpass extraction convolution kernel can improve the low-frequency details in the current image to be processed.

[0041] It is understandable that the matrix size of the first convolution kernel and at least two second convolution kernels is not limited in the disclosed embodiment. In the disclosed embodiment, the size of 3*3 is taken as an example, specifically a 3*3 learnable convolution kernel, a non-learning 3*3 high-pass convolution kernel, and a non-learning 3*3 band-pass convolution kernel. The above-mentioned non-learning high-pass convolution kernel and non-learning band-pass convolution kernel can be designed with convolution kernels of different functions according to actual use, and the number may not be limited to 2.

[0042] Step 102: Process the image to be processed through a first structured network to obtain a first output image.

[0043] In the embodiment of the present disclosure, the first output image refers to the output of the current convolution layer of the convolutional neural network in the training phase. The convolutional neural network in the training phase, that is, the first convolution kernel and at least two second convolution kernels in the first structure network, respectively extract features of the image to be processed, and obtain corresponding feature images respectively. The feature images are fused through the corresponding weight coefficients to obtain the first output image.

[0044] In summary, according to the image processing method proposed in the present invention, a first structural network is constructed by a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel with a function corresponding to a preset task; the image to be processed is processed by the first structural network to obtain a first output image, thereby solving the problem of weak effect of low-level semantic tasks, and by using non-learning and functionally clear convolution kernels, the amount and complexity of network parameters are effectively reduced, thereby reducing the difficulty of network training and improving the network convergence speed. At the same time, it can well guide the network to learn corresponding features, so that the network can more effectively solve specific image quality problems.

[0045] based on Figure 2 The embodiment shown, Figure 3 The following further illustrates a flow chart of an image processing method proposed in the present disclosure. Figure 3 based on Figure 2 In the embodiment shown, step 102 is further defined. Figure 3 In the embodiment shown, step 102 includes step 202, step 203 and step 204. The method is applied to the training phase of the result reparameterization method, such as Figure 3 As shown, the method comprises the following steps:

[0046] Step 201, constructing a first structural network through a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel having a function corresponding to a preset task.

[0047] What needs to be explained in this disclosure has been explained in the previous text and will not be repeated here. Because convolution is the basic component of most networks, the first structural network in the training stage can be formed by a 3*3 convolution kernel. In the present disclosure, the training stage is to replace each 3x3 convolution kernel in the existing network with a 3*3 learnable convolution kernel, a non-learning 3*3 high-pass convolution kernel, and a non-learning 3*3 bandpass convolution kernel.

[0048] Step 202: extracting a first feature of the image to be processed based on a first convolution kernel to obtain a first feature image.

[0049] In the embodiments of the present disclosure, in the convolutional neural network, the convolution operation is performed on two matrices. Convolutional neural network (CNN) mainly completes feature extraction through convolution operation. Image convolution operation is mainly performed by setting various feature extraction filter matrices (i.e., convolution kernels, usually set to a size of 3*3 or 5*5), and then using the convolution kernel to "slide" on the original image matrix (the image is actually a matrix composed of pixel values) to implement convolution operation to extract image features.

[0050] In an optional embodiment of the present disclosure, the image to be processed may be processed by a learnable convolution kernel of a 3*3 matrix size to extract the first feature, thereby obtaining a first feature image.

[0051] Step 203: extracting second features of the image to be processed based on at least two second convolution kernels to obtain at least two second feature images.

[0052] In an optional embodiment of the present disclosure, the image to be processed can be processed by a non-learning high-pass convolution kernel and a non-learning bandpass convolution kernel of a 3*3 matrix size, respectively, to extract the corresponding second features, thereby obtaining second feature images respectively.

[0053] It should be noted that the image to be processed is obtained by processing the input image. In other words, in the present disclosure, the image to be processed is the first channel image of the input image.

[0054] In an optional embodiment of the present disclosure, Figure 4 As shown, the image channel includes a first channel and a second channel, and the input image is separated into image channels, wherein the first channel refers to the brightness (Y) channel, which indicates the image brightness; the second channel of the input image refers to the chrominance (UV) channel, which indicates the image color. In the present disclosure, the image to be processed outputted by the first channel is processed by the first structural network, and the image of the second channel is interpolated.

[0055] The second channel of the input image includes: obtaining a second channel image of the input image; performing interpolation processing on the second channel image to obtain a target image; and fusing the target image with the first output image of the first channel. The interpolation processing may be bicubic interpolation, and the resolution may be upsampled to a desired resolution by fast bicubic interpolation.

[0056] Step 204: The first feature image is fused with at least two second feature images by using the first weight coefficient and the second weight coefficient to obtain a third feature image.

[0057] In the present disclosure, the first weight coefficient is the weight coefficient corresponding to the first convolution kernel, and the at least two second weight coefficients are the weight coefficients corresponding to the at least two second convolution kernels. The weight coefficients are used to indicate the fusion weights, which are used to fuse multiple feature images.

[0058] In an embodiment of the present disclosure, the three convolution results in the present disclosure, namely, feature images, are weighted fused to obtain a final convolution result output.

[0059] Wherein, according to the gradient direction of the image to be processed, the first weight coefficient and at least two second weight coefficients are determined. In the present disclosure, the first weight coefficient and at least two second weight coefficients are learnable parameters.

[0060] Step 205: Obtain a first output image through the activation function and the third feature image.

[0061] In the embodiments of the present disclosure, the activation function may be Relu, which is an activation function in deep learning. The activation function is mainly used in the convolution (Convolution) and pooling (MaxPooling) of the CNN convolutional neural network, because the values ​​of the image matrix after the convolution operation and the pooling operation are positive and negative, which inevitably causes the data scale to become very large, so the values ​​in the image matrix after the Relu function only have positive values. That is, the third feature image is passed through Relu, and the convolution output of this layer can be obtained to obtain the first output image.

[0062] In summary, through the method provided by the present disclosure, a first structural network is constructed by a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, the second convolution kernel includes a non-learning convolution kernel having a function corresponding to a preset task, the first feature of the image to be processed is extracted based on the first convolution kernel to obtain a first feature image, the second feature of the image to be processed is extracted based on at least two second convolution kernels to obtain at least two second feature images, the first feature image is fused with at least two second feature images through a first weight coefficient and a second weight coefficient to obtain a third feature image, the first output image is obtained through an activation function and the third feature image, and the network parameters and complexity are effectively reduced by using a non-learning and functionally clear convolution kernel in the training stage, thereby reducing the difficulty of network training and improving the network convergence speed, and at the same time, the network can be well guided to learn the corresponding features, so that the network can more effectively solve specific image quality problems. At the same time, learnable weights are used for weighted processing, so that the network can obtain better features from multiple feature branches.

[0063] It should be noted that the order of acquiring the first feature image and the at least two second feature images in the present disclosure is not limited in the embodiments of the present disclosure, and the method proposed in the present disclosure is only used as an example.

[0064] based on Figure 2 , Figure 3 The embodiment shown, Figure 5 A flowchart of an image processing method proposed in the present disclosure is further shown. The method is applied to the reasoning stage of the result reparameterization method, such as Figure 5 As shown, the method comprises the following steps:

[0065] Step 301: convert the first structural network into a second structural network equivalently.

[0066] In the embodiments of the present disclosure, the first structural network can be equivalently converted into the second structural network by the first weight coefficient and the second weight coefficient. That is, based on the first weight coefficient and at least two second weight coefficients, the first convolution kernel and at least two second convolution kernels are added by utilizing the additivity of convolution to obtain an equivalent third convolution kernel to construct the second structural network.

[0067] The additivity of convolution means that the same image to be processed is convolved with several two-dimensional convolution kernels of compatible sizes with the same step size to obtain output images of the same resolution size, and their output images are added. At this time, these two-dimensional convolution kernels can be added at corresponding positions to obtain an equivalent convolution kernel, and the same output image can be obtained by convolving the same image to be processed.

[0068] The second structure network refers to the convolutional neural network in the reasoning stage, and the second structure network includes a third convolution kernel.

[0069] In an optional embodiment of the present disclosure, a 3*3 learnable convolution kernel, a non-learning 3*3 high-pass convolution kernel, and a non-learning 3*3 bandpass convolution kernel are weighted fused according to a first weight coefficient and at least two second weight coefficients to obtain an inference convolution kernel used in the inference stage, i.e., a third convolution kernel.

[0070] It can be understood that the parameters used in the inference phase are all parameters of the training phase, that is, the first convolution kernel, at least two second convolution kernels, and their corresponding weight coefficients are all parameters used in the training phase.

[0071] Step 302: Process the image to be processed through a second structured network to obtain a second output image.

[0072] In an embodiment of the present disclosure, a third feature of the image to be processed is extracted by a third convolution kernel to obtain a fourth feature image; and a second output image is obtained by an activation function and the fourth feature image.

[0073] It is understandable that in the inference stage, the three convolution kernels are mainly fused. In the implementation process, this part uses the fused convolution kernel parameters to initialize the network provided by the relevant technology. Therefore, in the inference stage, the network parameters use parameters with stronger feature extraction capabilities, namely the fused convolution kernel parameters, so the amount of calculation will not be increased in the inference stage. The first output image and the second output image can be the same or different.

[0074] In summary, the first structure network is equivalently converted to the second structure network, and the image to be processed is processed by the second structure network to obtain the second output image. By fusing the non-learning and functionally clear convolution kernels used in the training phase into one convolution kernel in the inference phase, the network parameters and complexity are effectively reduced, thereby reducing the difficulty of network training and improving the network convergence speed. At the same time, the network can be well guided to learn the corresponding features, so that the network can effectively solve specific image quality problems. At the same time, learnable weights are used for weighted processing, so that the network can obtain better features from multiple feature branches.

[0075] based on Figure 2 , Figure 3 , Figure 5 The embodiment shown, as Figure 6 A schematic diagram of a specific structural reparameterization is shown.

[0076] In the embodiment of the present disclosure, structure reparameterization is divided into two stages: a training stage and a reasoning stage.

[0077] During the training phase, the present invention uses a 3*3 learnable convolution kernel, a non-learned 3*3 high-pass convolution kernel, and a non-learned 3*3 bandpass convolution kernel to extract features from the input image to be processed, respectively, to obtain a feature image, that is, three results: result1, result2, and result3. The three convolution results are weighted and fused by weight values ​​to obtain the final convolution result output, wherein the three weight values ​​wt1, wt2, and wt3 of the weighted fusion are learnable parameters.

[0078] In the inference phase, the 3*3 learnable convolution kernel, the non-learned 3*3 high-pass convolution kernel, and the non-learned 3*3 bandpass convolution kernel in the training phase are weighted and fused according to the weight values ​​of wt1, wt2, and wt3 to obtain the inference convolution kernel used in the inference phase. The convolution kernel is used to convolve the input image to be processed to finally obtain the output of the network layer. The three weight values ​​wt1, wt2, and wt3 are the parameters learned in the training phase.

[0079] The non-learning high-pass convolution kernel and band-pass convolution kernel mentioned above can be designed with different functions according to actual uses, and the number is not limited to 2.

[0080] In summary, if Figure 7 As shown in the figure, the right side is an image processed by the structural reparameterization method proposed in this disclosure, and the left side is an image processed by the structural reparameterization method using ACB net[3]. This scheme can effectively improve the sharpness and clarity of the image. This scheme uses 1 learnable convolution kernel and several non-learning convolution kernels with clear functions. The traditional non-learning method is used to design reparameterized convolution kernels to extract specific low-level features to guide the network to improve the effect, while reducing the network complexity, and the lightweight network is easier to train; in the process of reparameterization, the use of learnable weights to fuse several convolution kernels is more conducive to network feature extraction.

[0081] Figure 8 FIG. 8 is a schematic diagram of the structure of an image processing device 800 provided in an embodiment of the present disclosure. Figure 8 As shown, the image processing device comprises:

[0082] A construction unit 810 is used to construct a first structure network through a first convolution kernel and at least two second convolution kernels, where the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel having a function corresponding to a preset task;

[0083] The processing unit 820 is used to process the image to be processed through the first structure network to obtain a first output image

[0084] In some embodiments, the processing unit 820 is used to: extract the first feature of the image to be processed based on the first convolution kernel to obtain a first feature image; extract the second feature of the image to be processed based on at least two second convolution kernels to obtain at least two second feature images; fuse the first feature image with the at least two second feature images through the first weight coefficient and the second weight coefficient to obtain a third feature image, the first weight coefficient being the weight coefficient corresponding to the first convolution kernel, and the at least two second weight coefficients being the weight coefficients corresponding to the at least two second convolution kernels; obtain the first output image through the activation function and the third feature image.

[0085] In some embodiments, the processing unit 820 is used to determine a first weight coefficient and at least two second weight coefficients according to a gradient direction of the image to be processed.

[0086] In some embodiments, the device 800 further includes: a conversion unit, configured to convert the first structural network into a second structural network equivalently; and to process the image to be processed by the second structural network to obtain a second output image.

[0087] In some embodiments, the conversion unit is used to: convert the first structural network into the second structural network equivalently through the first weight coefficient and the second weight coefficient.

[0088] In some embodiments, the conversion unit is used to: based on the first weight coefficient and at least two second weight coefficients, equivalently convert the first convolution kernel and at least two second convolution kernels into a third convolution kernel to obtain a second structural network.

[0089] In some embodiments, the conversion unit is used to: extract the third feature of the image to be processed through a third convolution kernel to obtain a fourth feature image; and obtain the second output image through an activation function and the fourth feature image.

[0090] In summary, through the image processing device, a first structural network is constructed by a first convolution kernel and at least two second convolution kernels, the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel with a function corresponding to a preset task; the image to be processed is processed by the first structural network to obtain a first output image, so as to solve the problem of weak effect of low-level semantic tasks, and by using non-learning and functionally clear convolution kernels, the amount and complexity of network parameters are effectively reduced, thereby reducing the difficulty of network training and improving the network convergence speed. At the same time, it can well guide the network to learn the corresponding features, so that the network can more effectively solve specific image quality problems.

[0091] Corresponding to the methods provided in the above-mentioned embodiments, the present disclosure also provides an image processing device. Since the device provided in the embodiment of the present disclosure corresponds to the methods provided in the above-mentioned embodiments, the implementation method of the method is also applicable to the device provided in this embodiment and will not be described in detail in this embodiment.

[0092] In the embodiments provided in the present application, the methods and devices provided in the embodiments of the present application are introduced. In order to implement the functions in the methods provided in the embodiments of the present application, the electronic device may include a hardware structure and a software module, and implement the functions in the form of a hardware structure, a software module, or a hardware structure plus a software module. A function of the functions may be executed in the form of a hardware structure, a software module, or a hardware structure plus a software module.

[0093] Fig. 9 1 is a block diagram of an electronic device 900 for implementing the above-mentioned image processing method according to an exemplary embodiment. For example, the electronic device 900 may be a mobile phone, a computer, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0094] Reference Fig. 9 , the electronic device 900 may include one or more of the following components: a processing component 902 , a memory 904 , a power component 906 , a multimedia component 908 , an audio component 910 , an input / output (I / O) interface 912 , a sensor component 914 , and a communication component 916 .

[0095] The processing component 902 generally controls the overall operation of the electronic device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 902 may include one or more processors 920 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 902 may include one or more modules to facilitate the interaction between the processing component 902 and other components. For example, the processing component 902 may include a multimedia module to facilitate the interaction between the multimedia component 908 and the processing component 902.

[0096] The memory 904 is configured to store various types of data to support operations on the electronic device 900. Examples of such data include instructions for any application or method operating on the electronic device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 904 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0097] The power supply component 906 provides power to the various components of the electronic device 900. The power supply component 906 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 900.

[0098] The multimedia component 908 includes a screen that provides an output interface between the electronic device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 908 includes a front camera and / or a rear camera. When the electronic device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0099] The audio component 910 is configured to output and / or input audio signals. For example, the audio component 910 includes a microphone (MIC), and when the electronic device 900 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 904 or sent via the communication component 916. In some embodiments, the audio component 910 also includes a speaker for outputting audio signals.

[0100] I / O interface 912 provides an interface between processing component 902 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0101] The sensor assembly 914 includes one or more sensors for providing various aspects of status assessment for the electronic device 900. For example, the sensor assembly 914 can detect the open / closed state of the electronic device 900, the relative positioning of components, such as the display and keypad of the electronic device 900, and the sensor assembly 914 can also detect the position change of the electronic device 900 or a component of the electronic device 900, the presence or absence of user contact with the electronic device 900, the orientation or acceleration / deceleration of the electronic device 900, and the temperature change of the electronic device 900. The sensor assembly 914 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 914 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 914 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0102] The communication component 916 is configured to facilitate wired or wireless communication between the electronic device 900 and other devices. The electronic device 1000 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio) or a combination thereof. In an exemplary embodiment, the communication component 916 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 916 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0103] In an exemplary embodiment, the electronic device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0104] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 904 including instructions, and the instructions can be executed by the processor 920 of the electronic device 900 to complete the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0105] An embodiment of the present disclosure further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the image processing method described in the above embodiment of the present disclosure.

[0106] An embodiment of the present disclosure further provides a computer program product, including a computer program, which executes the image processing method described in the above embodiment of the present disclosure when a processor is used.

[0107] An embodiment of the present disclosure also proposes a chip, which includes one or more interface circuits and one or more processors; the interface circuit is used to receive signals from a memory of an electronic device and send signals to the processor, the signals include computer instructions stored in the memory, and when the processor executes the computer instructions, the electronic device executes the image processing method described in the above embodiments of the present disclosure.

[0108] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0109] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0110] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0111] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processing module, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (control method), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

[0112] It should be understood that the various parts of the embodiments of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0113] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0114] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0115] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An image processing method, characterized in that: The method comprises: Constructing a first structural network by using a first convolution kernel and at least two second convolution kernels, wherein the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel having a function corresponding to a preset task; The image to be processed is processed by the first structural network to obtain a first output image.

2. The method according to claim 1, characterized in that: The step of processing the image to be processed by the first structure network to obtain a first output image includes: Extracting a first feature of the image to be processed based on the first convolution kernel to obtain a first feature image; Extracting a second feature of the image to be processed based on the at least two second convolution kernels to obtain at least two second feature images; The first feature image is fused with the at least two second feature images by using a first weight coefficient and a second weight coefficient to obtain a third feature image, wherein the first weight coefficient is a weight coefficient corresponding to the first convolution kernel, and the at least two second weight coefficients are weight coefficients corresponding to the at least two second convolution kernels; A first output image is obtained by using the activation function and the third feature image.

3. The method according to claim 2, characterized in that The method further comprises: A first weight coefficient and at least two second weight coefficients are determined according to the gradient direction of the image to be processed.

4. The method according to claims 1-3, characterized in that: The method further comprises: Converting the first structural network into a second structural network; The image to be processed is processed by the second structure network to obtain a second output image.

5. The method according to claim 4, characterized in that The equivalent conversion of the first structural network into a second structural network comprises: The first structural network is equivalently converted into a second structural network by using the first weight coefficient and the second weight coefficient.

6. The method according to claim 5, characterized in that The equivalent conversion of the first structural network into a second structural network by using the first weight coefficient and the second weight coefficient includes: Based on the first weight coefficient and the at least two second weight coefficients, the first convolution kernel and the at least two second convolution kernels are equivalently converted into a third convolution kernel to obtain a second structural network.

7. The method according to claim 4, characterized in that The processing of the image to be processed by the second structure network to obtain a second output image comprises: Extracting the third feature of the image to be processed by using the third convolution kernel to obtain a fourth feature image; A second output image is obtained through the activation function and the fourth feature image.

8. An image processing device, characterized in that: The device comprises: A construction unit, configured to construct a first structural network through a first convolution kernel and at least two second convolution kernels, wherein the first convolution kernel includes a learnable convolution kernel, and the second convolution kernel includes a non-learning convolution kernel having a function corresponding to a preset task; A processing unit is used to process the image to be processed through the first structural network to obtain a first output image.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

11. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the interface circuit is used to receive a signal from a memory of an electronic device and send the signal to the processor, the signal includes a computer instruction stored in the memory, and when the processor executes the computer instruction, the electronic device executes the method described in any one of claims 1 to 7.