Image processing method, device and electronic equipment

Abstract

The present invention discloses an image processing method, device and electronic device, the method comprising: obtaining an original image; performing a first image super-resolution processing on the original image according to a preset magnification to obtain a first super-resolution image; performing a second image super-resolution processing on the original image according to the magnification to obtain a second super-resolution image; obtaining a fusion ratio of corresponding pixel points in the first super-resolution image and the second super-resolution image; performing a fusion processing on corresponding pixel points in the first super-resolution image and the second super-resolution image according to the fusion ratio to obtain fused pixel points; generating a fused image according to the fused pixel points. In this way, while improving the resolution of the original image, the visual effect of the fused image can also be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image super-resolution, and more specifically, to an image processing method, device and electronic equipment. Background Art

[0002] As the resolution of image and video playback devices such as mobile terminals continues to improve, the demand for high-resolution images is increasing day by day. In the prior art, the resolution of low-resolution original images can be improved by image super-resolution technology.

[0003] The quality of the original image can be improved after the image super-resolution processing. When the original image is subjected to super-resolution processing, in order to repair noise such as mosaics and burrs in the original image, the skin of the face and other parts of the person in the original image will be smoothed. Although this can improve the visual clarity of the obtained super-resolution image, it may also cause the loss of image details, making the visual effect of the face and other parts of the skin in the super-resolution image less natural. Summary of the invention

[0004] An object of the present invention is to provide a new technical solution for an image processing method.

[0005] According to a first aspect of the present invention, there is provided an image processing method, comprising:

[0006] Get the original image;

[0007] Performing a first image super-resolution process on the original image according to a preset magnification to obtain a first super-resolution image;

[0008] Performing second image super-resolution processing on the original image according to the magnification factor to obtain a second super-resolution image;

[0009] Obtaining a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image;

[0010] According to the fusion ratio, corresponding pixel points in the first super-resolution image and the second super-resolution image are fused to obtain fused pixel points;

[0011] A fused image is generated according to the fused pixel points.

[0012] Optionally, acquiring a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image includes:

[0013] Determine the difference between the image parameters of a pixel point in the original image and an adjacent pixel point;

[0014] Calculating a fusion ratio corresponding to a pixel point in the original image according to a difference in image parameters between a pixel point in the original image and an adjacent pixel point;

[0015] According to the fusion ratio corresponding to the pixels in the original image, the fusion ratio of the pixels in the first super-resolution image and the second super-resolution image corresponding to the pixels in the original image is determined.

[0016] Optionally, acquiring a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image includes:

[0017] Determine a difference in image parameters between at least one pixel point among corresponding pixels in the first super-resolved image and the second super-resolved image and an adjacent pixel point in the super-resolved image;

[0018] According to the difference in image parameters between the at least one pixel and an adjacent pixel, a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image is calculated.

[0019] Optionally, the corresponding pixel points in the first super-resolved image and the second super-resolved image include: a target pixel point on one of the first super-resolved image and the second super-resolved image, and a pixel point corresponding to the target pixel point on the other of the first super-resolved image and the second super-resolved image;

[0020] The step of calculating the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image according to the difference in image parameters between the at least one pixel and an adjacent pixel comprises:

[0021] Calculating the fusion ratio of the target pixel according to the difference in image parameters between the target pixel and adjacent pixels in the super-resolution image;

[0022] According to the fusion ratio of the target pixel point, the fusion ratio of the pixel point corresponding to the target pixel point is calculated; or, the difference in image parameters between the pixel point corresponding to the target pixel point and the adjacent pixel points in the super-resolution image is calculated, and according to the difference in image parameters between the pixel point corresponding to the target pixel point and the adjacent pixel points in the super-resolution image, the fusion ratio of the pixel point corresponding to the target pixel point is calculated.

[0023] Optionally, the step of calculating the fusion ratio of the target pixel point according to the difference in image parameters between the target pixel point and adjacent pixels in the super-resolution image includes:

[0024] The proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold among all adjacent pixels corresponding to the target pixel is determined as the fusion ratio of the target pixel.

[0025] Optionally, the step of determining, among all adjacent pixels corresponding to the target pixel, the proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold as the fusion ratio of the target pixel includes:

[0026] Determine the number of adjacent pixel points whose image parameter difference is greater than or equal to a preset threshold as a first number;

[0027] A ratio of the first number to the second number is calculated as a fusion ratio of the target pixel, wherein the second number is the number of all adjacent pixel points corresponding to the target pixel point.

[0028] Optionally, the step of calculating the fusion ratio of the target pixel point according to the difference in image parameters between the target pixel point and adjacent pixels in the super-resolution image includes:

[0029] Calculating an average value of the image parameter difference between the target pixel and each adjacent pixel in the super-resolved image;

[0030] The fusion ratio of the target pixel is determined according to the average value.

[0031] Optionally, the step of fusing corresponding pixels in the first super-resolution image and the second super-resolution image to obtain fused pixels includes:

[0032] According to the fusion ratio of the target pixel point and the fusion ratio of the pixel point corresponding to the target pixel point, the image parameters of the target pixel point and the image parameters of the pixel point corresponding to the target pixel point are weighted averaged to obtain the image parameters of the fused pixel point.

[0033] Optionally, the image processing method further includes:

[0034] Extracting a portion corresponding to a human face in the first super-resolution image as a first human face image region;

[0035] Extracting a corresponding face portion in the second super-resolution image as a second face image region;

[0036] The corresponding pixels in the first face image region and the second face image region are fused to obtain the fused face image region.

[0037] Optionally, the image parameter includes a grayscale value or a pixel value.

[0038] According to a second aspect of the present invention, there is provided an image processing apparatus, comprising:

[0039] An image acquisition module, used for acquiring original images;

[0040] A first super-resolution module, used for performing a first image super-resolution processing on the original image according to a preset magnification to obtain a first super-resolution image;

[0041] A second super-resolution module, used for performing a second image super-resolution processing on the original image according to the magnification factor to obtain a second super-resolution image;

[0042] A fusion ratio acquisition module, used to acquire the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image;

[0043] a pixel fusion module, configured to fuse corresponding pixels in the first super-resolution image and the second super-resolution image according to the fusion ratio to obtain fused pixels; and

[0044] The image generation module is used to generate a fused image based on the fused pixel points.

[0045] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0046] The image processing device according to the second aspect of the present invention; or,

[0047] A processor and a memory, wherein the memory is used to store executable instructions, and the instructions are used to control the processor to execute the image processing method according to the first aspect of the present invention.

[0048] In this embodiment, the first super-resolution processing is performed on the original image according to a preset magnification to obtain a first super-resolution image, and the second super-resolution processing is performed on the original image according to the magnification to obtain a second super-resolution image, and the fusion ratio of the corresponding pixel points in the first super-resolution image and the second super-resolution image is obtained. According to the fusion ratio, the corresponding pixel points in the first super-resolution image and the second super-resolution image are fused to obtain fused pixel points, and a fused image is generated according to the fused pixel points. In this way, while improving the resolution of the original image, the visual effect of the fused image can also be improved.

[0049] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0051] Figure 1 is a block diagram showing an example of a hardware configuration of an electronic device that can be used to implement an embodiment of the present invention.

[0052] Figure 2 A flowchart showing an image processing method according to an embodiment of the present invention is shown.

[0053] Figure 3 A flowchart showing a first example of the steps for calculating the fusion ratio according to an embodiment of the present invention.

[0054] Figure 4 A flowchart showing a second example of the steps for calculating the fusion ratio according to an embodiment of the present invention.

[0055] Figure 5 A flowchart showing a third example of the steps for calculating the fusion ratio according to the embodiment of the present invention.

[0056] Figure 6 A block diagram showing an image processing apparatus according to an embodiment of the present invention.

[0057] Figure 7 A block diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0058] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0059] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0060] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0061] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0062] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0063] <Hardware Configuration>

[0064] Figure 1 1 is a block diagram showing a hardware configuration of an electronic device 1000 that can implement an embodiment of the present invention.

[0065] The electronic device 1000 may be a portable computer, a desktop computer, a mobile phone, a tablet computer, etc. Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and the like. Among them, the processor 1100 may be a central processing unit CPU, a microprocessor MCU, and the like. The memory 1200 includes, for example, a ROM (read-only memory), a RAM (random access memory), a non-volatile memory such as a hard disk, and the like. The interface device 1300 includes, for example, a USB interface, a headphone interface, and the like. The communication device 1400 is, for example, capable of wired or wireless communication, and may specifically include Wifi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, and the like. The display device 1500 is, for example, a liquid crystal display screen, a touch display screen, and the like. The input device 1600 may include, for example, a touch screen, a keyboard, a somatosensory input, and the like. The user may input / output voice information through the speaker 1700 and the microphone 1800.

[0066] Figure 1 The electronic device shown is merely illustrative and does not in any way limit the present invention, its application or use. In the embodiments of the present invention, the memory 1200 of the electronic device 1000 is used to store instructions, and the instructions are used to control the processor 1100 to operate to perform any image processing method provided by the embodiments of the present invention. It should be understood by those skilled in the art that although Figure 1 In the electronic device 1000, multiple devices are shown, but the present invention may only involve some of the devices, for example, the electronic device 1000 may only involve the processor 1100 and the storage device 1200. A technician can design instructions according to the scheme disclosed in the present invention. How instructions control the processor to operate is well known in the art, so it will not be described in detail here.

[0067] <Method>

[0068] In this embodiment, an image processing method is provided. The method is implemented by an electronic device. The electronic device may be an electronic product having a processor and a memory. For example, it may be a desktop computer, a laptop computer, a mobile phone, a tablet computer, etc. In one example, the electronic device may be Figure 1 An electronic device 1000 is shown.

[0069] The image processing method, such as Figure 2 As shown, it includes: steps S2100-S2600.

[0070] Step 2100, obtaining the original image.

[0071] The original image in this embodiment may be pre-stored in the electronic device, may be obtained by the electronic device from other devices (for example, a server), or may be obtained in real time by a camera of the electronic device.

[0072] In one example, the original image may be an image containing a human face.

[0073] Step S2200: performing a first image super-resolution process on the original image according to a preset magnification factor to obtain a first super-resolution image.

[0074] Image super-resolution processing refers to the process of restoring a high-resolution image from a low-resolution image or image sequence.

[0075] The magnification factor can be set in advance according to the application scenario or requirements. 2 , where r can be any positive integer greater than 1.

[0076] For example, the resolution in the original image may be W×H. This indicates that the original image may include a pixel array of W columns and H rows. Then, according to the magnification factor r 2 The original image is subjected to a first super-resolution process, and the resolution of the obtained first super-resolution image can be W×H×r 2 The first super-resolution image may be composed of a pixel array of W×r rows and H×r columns.

[0077] The image super-resolution processing method may include: interpolation-based image super-resolution processing, reconstruction-based image super-resolution processing, and learning-based image super-resolution processing.

[0078] Interpolation-based image super-resolution processing methods may include nearest neighbor interpolation, linear interpolation, cubic interpolation, spline interpolation, kernel regression interpolation, and the like.

[0079] Nearest neighbor interpolation, also known as zero-order interpolation, is the simplest difference method. It uses the pixel values ​​of specific pixels in the original image to fill the scaled image. It is easy to implement. For grayscale images, the original image is processed point by point, and the grayscale value of each pixel is interpolated and copied multiple times.

[0080] Linear interpolation, also known as bilinear interpolation, is also a simple difference method, widely used in the field of image reconstruction. It is called bilinear interpolation because the interpolation of the image is obtained by linear interpolation of the row and column pixels twice. The basic idea of ​​linear interpolation is to insert several points between adjacent pixel points A and B in the original image, and the grayscale values ​​of these points make the grayscale values ​​between A and B transition linearly.

[0081] Cubic interpolation is also called bicubic convolution value. It is an improvement on linear interpolation. The basic idea is to increase the number of neighboring pixels to obtain a better interpolation function and use the convolution operator to express the interpolation method.

[0082] The above three interpolation methods belong to polynomial interpolation. The oscillation of data points in polynomial interpolation is called Runge phenomenon, which can be solved by spline interpolation.

[0083] Spline is a special function, and polynomials are defined piecewise. Spline interpolation is easier to use than polynomial interpolation. Low-order spline interpolation can achieve the effect of high-order polynomial interpolation, and can also avoid the Runge phenomenon. Spline interpolation has the characteristic of preserving convexity. Spline interpolation is first-order continuous and differentiable at the interpolation point, and the mean square error with the expected function is the smallest.

[0084] The kernel regression method is from the perspective of a two-dimensional plane. At the edge of the image, the sampling window kernel function will select a suitable pixel range along the edge direction for the regularization solution process of image reconstruction.

[0085] Reconstruction-based image super-resolution processing methods include iterative back-projection, convex set projection and maximum a posteriori probability estimation, which require multiple high- and low-resolution image samples for training to obtain prior knowledge.

[0086] Learning-based image super-resolution processing methods include: SRCNN (Super-Resolution Convolutional Neural Network), DRCN (Deeply-Recursive Convolutional Network for Image Super-Resolution), ESPCN (Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network), VESPCN (Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation) and SRGAN (Photo-Realistic Single Image Super-Resolution Using aGenerative Adversarial Network).

[0087] These deep learning-based image super-resolution processing methods all learn the relationship between low-resolution images and high-resolution images, and then use this relationship to guide the super-resolution processing of images.

[0088] The method for super-resolution processing of the first image in this embodiment may be any super-resolution processing method and is not limited to the method disclosed in this embodiment.

[0089] Step S2300: performing a second image super-resolution process on the original image according to the magnification factor to obtain a second super-resolution image.

[0090] The method for super-resolution processing of the second image in this embodiment can be any super-resolution processing method, and is not limited to the method disclosed in this embodiment. In one example, the method for super-resolution processing of the first image can be different from the method for super-resolution processing of the second image, so that there is a difference between the obtained first super-resolution image and the second super-resolution image.

[0091] The first super-resolution image and the second super-resolution image are obtained by performing super-resolution processing on the original image at the same magnification. Therefore, the resolution of the first super-resolution image and the second super-resolution image is the same, and the number of rows and columns of the pixel array of the first super-resolution image is the same as the number of rows and columns of the pixel array of the second super-resolution image.

[0092] For example, the resolution in the original image may be W×H. This indicates that the original image may include a pixel array of W columns and H rows. Then, according to the magnification factor r 2 The original image is subjected to a first super-resolution process, and the resolution of the obtained first super-resolution image can be W×H×r 2 The first super-resolution image may be composed of a pixel array of H×r rows and W×r columns. 2 The second super-resolution processing is performed on the original image, and the resolution of the obtained second super-resolution image can also be W×H×r 2 The second super-resolution image may also be composed of a pixel array of H×r rows and W×r columns.

[0093] Step S2400, obtaining the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image.

[0094] Then, step S2400 may further include:

[0095] Determine the difference in image parameters between at least one pixel point among the corresponding pixels in the first super-resolution image and the second super-resolution image and the adjacent pixel point in the super-resolution image; and calculate the fusion ratio of the corresponding pixel points in the first super-resolution image and the second super-resolution image according to the difference in image parameters between the at least one pixel point and the adjacent pixel point.

[0096] In one or more embodiments of the present invention, the image parameter may be a grayscale value or a pixel value.

[0097] In this embodiment, at least one pixel point among the corresponding pixels in the first super-resolution image and the second super-resolution image may be used as a target pixel point, and the target pixel point may be a pixel point in the first super-resolution image or a pixel point in the second super-resolution image.

[0098] In the case where at least one of the corresponding pixels in the first super-resolution image and the second super-resolution image is used as a target pixel, the fusion ratio of the corresponding pixels in the first super-resolution image and the second super-resolution image can be calculated according to the difference in image parameters between the target pixel and the adjacent pixel. The corresponding pixels in the first super-resolution image and the second super-resolution image include: the target pixel on one of the first super-resolution image and the second super-resolution image, and the pixel corresponding to the target pixel on the other of the first super-resolution image and the second super-resolution image.

[0099] In one embodiment, the step of calculating the fusion ratio of the corresponding pixels in the first super-resolution image and the second super-resolution image according to the difference in image parameters between the target pixel and the adjacent pixel may include: Figure 3 Steps S2411 to S2412 shown:

[0100] Step S2411, calculating the fusion ratio of the target pixel according to the difference in image parameters between the target pixel and adjacent pixels in the super-resolved image.

[0101] In one example, the method for calculating the fusion ratio of the target pixel point can be: determine the proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold among all adjacent pixels corresponding to the target pixel point, as the fusion ratio of the target pixel point.

[0102] Determining the proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold among all adjacent pixels corresponding to the target pixel as the fusion ratio of the target pixel may further include the following steps S2411-1 to S2411-3:

[0103] Step S2411-1, determining the number of adjacent pixel points whose image parameter difference is greater than or equal to a preset threshold, as the first number.

[0104] The threshold may be set in advance according to the application scenario or requirement, or may be obtained through big data training.

[0105] Step S2411-2, determining the number of all adjacent pixel points corresponding to the target pixel point as the second number.

[0106] Specifically, the first number is less than or equal to the second number. Step S2411-2 may be executed before step S2411-1, after step S2411-1, or simultaneously with step S2411-1. In this embodiment, the execution order of steps S2411-1 and S2411-2 is not limited.

[0107] For example, for a pixel matrix of rH rows and rW columns, if the target pixel is located at the 1st row and 1st column, or the 1st row and the rWth column, or the rHth row and the 1st column, or the rHth row and the rWth column in the pixel matrix, then the number of adjacent pixels in the super-resolution image where the target pixel is located is 3, that is, the second number is 3.

[0108] If the target pixel is located at the 1st row and the 2nd to rW-1th column, or the rHth row and the 2nd to rW-1th column, or the 2nd to rH-1th row and the 1st column, or the 2nd to rH-1th row and the rWth column in the pixel matrix, then the number of adjacent pixel points in the super-resolution image where the target pixel is located is 5, that is, the second number is 5.

[0109] If the target pixel is located at other positions in the pixel matrix (other than the above positions), the number of adjacent pixel points in the super-resolution image where the target pixel is located is 8, that is, the second number is 8.

[0110] Step S2411-3, calculate the ratio of the first number to the second number as the fusion ratio of the target pixel point.

[0111] The fusion ratio of the target pixel points=the first number / the second number. Since the first number is less than or equal to the second number, the fusion ratio may be a positive number less than or equal to 1.

[0112] In another example, the method for calculating the fusion ratio of the target pixel point can also be: according to the difference of image parameters between the target pixel point and each adjacent pixel point in the super-resolution image, the average value of the difference of image parameters is calculated; according to the average value, the fusion ratio of the target pixel point is determined.

[0113] A comparison table reflecting the correspondence between the average value of the difference of the image parameters and the fusion ratio can be pre-stored. Then, by searching the comparison table, the fusion ratio corresponding to the average value of the difference of the image parameters of the target pixel point, that is, the fusion ratio of the target pixel point, can be obtained.

[0114] Alternatively, a comparison table may be stored that reflects the correspondence between the range of the average value of the difference of the image parameters and the fusion ratio. Then, by searching the comparison table, the range of the average value of the difference of the image parameters of the target pixel point can be determined, and the fusion ratio corresponding to the range can be obtained, that is, the fusion ratio of the target pixel point.

[0115] Step S2412, according to the fusion ratio of the target pixel point, calculate the fusion ratio of the pixel points corresponding to the target pixel point.

[0116] According to the fusion ratio of the target pixel, the fusion ratio of the pixel corresponding to the target pixel is calculated by calculating the difference between 1 and the fusion ratio of the target pixel as the fusion ratio of the pixel corresponding to the target pixel.

[0117] Specifically, the fusion ratio of the pixel points corresponding to the target pixel point=1-the fusion ratio of the target pixel point. For example, when the fusion ratio of the target pixel point is 0.6, the fusion ratio of the pixel points corresponding to the target pixel point may be 1-0.6=0.4.

[0118] In another embodiment, the step of calculating the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image according to the difference in image parameters between the target pixel and the adjacent pixel may include: Figure 4Steps S2421 to S2422 shown:

[0119] Step S2421, calculating the fusion ratio of the target pixel according to the difference in image parameters between the target pixel and adjacent pixels in the super-resolved image.

[0120] This step may be specifically referred to the aforementioned step S2411, which will not be described in detail here.

[0121] Step S2422, calculate the difference in image parameters between the pixel point corresponding to the target pixel point and the adjacent pixel points in the super-resolution image, and calculate the fusion ratio of the pixel point corresponding to the target pixel image based on the difference in image parameters between the pixel point corresponding to the target pixel point and the adjacent pixel points in the super-resolution image.

[0122] The difference in image parameters between the pixel corresponding to the target pixel and the adjacent pixel in the super-resolution image is calculated, and the fusion ratio of the pixel corresponding to the target pixel map is calculated based on the difference in image parameters between the pixel corresponding to the target pixel and the adjacent pixel in the super-resolution image. Specifically, the method of calculating the fusion ratio of the target pixel based on the difference in image parameters between the target pixel and the adjacent pixel in the super-resolution image is referred to above, and no further details are given here.

[0123] In another embodiment of the present invention, obtaining the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image may also include: Figure 5 Steps S2431 to S2432 shown:

[0124] Step S2431, determining the difference in image parameters between a pixel point in the original image and an adjacent pixel point, and calculating the fusion ratio corresponding to the pixel point in the original image.

[0125] Determine the difference in image parameters between a pixel in the original image and an adjacent pixel, and calculate the method of the fusion ratio corresponding to the pixel in the original image. Specifically, refer to the aforementioned method of calculating the fusion ratio of the target pixel based on the difference in image parameters between the target pixel and the adjacent pixel in the super-resolution image, which will not be repeated here.

[0126] Step S2432, determining the fusion ratio of the pixels in the first super-resolution image and the second super-resolution image corresponding to the pixels in the original image according to the fusion ratio corresponding to the pixels in the original image.

[0127] Specifically, each pixel in the original image corresponds to a plurality of pixels in the first super-resolution image and a plurality of pixels in the second super-resolution image.

[0128] The fusion ratio of the pixels in the original image may be used as the fusion ratio of the corresponding multiple pixels in the first super-resolution image; and the difference between 1 and the fusion ratio of the pixels in the original image may be used as the fusion ratio of the corresponding multiple pixels in the second super-resolution image.

[0129] Step S2500: fusing the first super-resolution image and the second super-resolution image according to the fusion ratio to obtain fused pixels.

[0130] Specifically, the image parameters of the target pixel and the image parameters of the pixel corresponding to the target pixel may be weighted averaged according to the fusion ratio of the target pixel and the fusion ratio of the pixel corresponding to the target pixel to obtain the image parameters of the fused pixel.

[0131] For example, the image parameter of the target pixel is K, the image parameter of the pixel corresponding to the target pixel is Y, the fusion ratio of the target pixel is a, and the fusion ratio of the pixel corresponding to the target pixel is b, then the image parameter Z of the fused pixel can be:

[0132] The positions of the corresponding pixels in the pixel matrix of the corresponding super-resolution image are the same, so the positions of the obtained fused pixels in the corresponding pixel matrix are also the same as the positions of the corresponding pixels in the pixel matrix of the corresponding super-resolution image. For example, when the positions of the corresponding pixels in the pixel matrix of the corresponding super-resolution image are the nth row and the mth column, the positions of the fused pixels in the corresponding pixel matrix are also the nth row and the mth column.

[0133] In one example, the fusion ratio of the corresponding pixel points can also be adjusted based on experience. For example, when the fusion ratio of the target pixel point is w and the fusion ratio of the pixel point corresponding to the target pixel point is 1-w, the fusion ratio of the target pixel point can be adjusted to a*w, and the fusion ratio of the pixel point corresponding to the target pixel point can be adjusted to 1-a*w. 2 , adjust the fusion ratio of the pixel points corresponding to the target pixel points to 1-a*w 2 .

[0134] When the obtained fusion ratio of the target pixel point is w1 and the fusion ratio of the pixel point corresponding to the target pixel point is w2, the fusion ratio of the target pixel point can be adjusted to w1+b, and the fusion ratio of the pixel point corresponding to the target pixel point can be adjusted to w2-b.

[0135] Step S2600, generating a fused image based on the fused pixels.

[0136] Specifically, the fused image may be generated according to the pixel values ​​of the fused pixel points and their positions in the corresponding pixel matrix.

[0137] In one example, only some corresponding pixels in the first super-resolution image and the second super-resolution image may be fused. Then, the unfused pixels in the first super-resolution image or the second super-resolution image may be used as pixels in the fused image to generate the fused image together with the fused pixels.

[0138] For example, the first super-resolution processing is a super-resolution processing based on interpolation, and the second super-resolution processing is a super-resolution processing based on deep learning. The fusion ratio of the target pixel is determined according to the pixel value difference between the target pixel and the adjacent pixel in the first super-resolution image, and then the fusion ratio of the pixel corresponding to the target pixel in the second super-resolution image is determined according to the fusion ratio of the target pixel. Then, if the fusion ratio of the target pixel corresponding to the first super-resolution image is low, it can be indicated that the target pixel is located in the solid background area or the skin area such as the face in the first super-resolution image. If the fusion ratio of the target pixel is high, it can be indicated that the target pixel is located in the noise area or the edge of the contour. According to the corresponding fusion ratio, the corresponding pixels are fused to different degrees, and the generated fused image can retain the image details of the skin parts such as the face.

[0139] In one embodiment, since noises such as mosaics and burrs are easily generated in the face region when performing image super-resolution processing, the image processing method may further include:

[0140] Extracting a corresponding face portion in the first super-resolution image as a first face image region;

[0141] Extracting a corresponding face portion in the second super-resolution image as a second face image region;

[0142] The corresponding pixel points in the first face image region and the second face image region are fused to obtain a fused face image region.

[0143] In this embodiment, the fused image may be generated by combining the unfused pixels in the first super-resolution image or the second super-resolution image and the fused face image region.

[0144] In this way, the pixels corresponding to the face image area in the first super-resolution image and the second super-resolution image are fused to different degrees according to the corresponding fusion ratios, and the generated fused image can retain the image details of the skin area such as the face.

[0145] In this embodiment, the first super-resolution processing is performed on the original image according to a preset magnification to obtain a first super-resolution image, and the second super-resolution processing is performed on the original image according to the magnification to obtain a second super-resolution image, and the fusion ratio of the corresponding pixel points in the first super-resolution image and the second super-resolution image is obtained. According to the fusion ratio, the corresponding pixel points in the first super-resolution image and the second super-resolution image are fused to obtain fused pixel points, and a fused image is generated according to the fused pixel points. In this way, while improving the resolution of the original image, the visual effect of the fused image can also be improved.

[0146] <Image Processing Device>

[0147] In this embodiment, an image processing device 6000 is provided. Figure 6 As shown, it includes: an image acquisition module 6100, a first super-resolution module 6200, a second super-resolution module 6300, a fusion ratio acquisition module 6400, a pixel fusion module 6500 and an image generation module 6600. The image acquisition module 6100 is used to acquire an original image; the first super-resolution module 6200 is used to perform a first image super-resolution process on the original image according to a preset magnification to obtain a first super-resolution image; the second super-resolution module 6300 is used to perform a second image super-resolution process on the original image according to the magnification to obtain a second super-resolution image; the fusion ratio acquisition module 6400 is used to acquire the fusion ratio of the corresponding pixels in the first super-resolution image and the second super-resolution image; the pixel fusion module 6500 is used to perform a fusion process on the corresponding pixels in the first super-resolution image and the second super-resolution image according to the fusion ratio to obtain fused pixels; the image generation module 6600 is used to generate a fused image according to the fused pixels.

[0148] In one embodiment, the fusion ratio acquisition module 6400 may also be used to:

[0149] Determine the difference between the image parameters of the pixel in the original image and the adjacent pixel;

[0150] Calculate the fusion ratio corresponding to the pixel in the original image according to the difference in image parameters between the pixel in the original image and the adjacent pixel;

[0151] According to the fusion ratio corresponding to the pixels in the original image, the fusion ratio of the pixels in the first super-resolution image and the second super-resolution image corresponding to the pixels in the original image is determined.

[0152] In one embodiment, the fusion ratio acquisition module 6400 may also be used to:

[0153] Determine a difference in image parameters between at least one pixel point among corresponding pixels in the first super-resolved image and the second super-resolved image and an adjacent pixel point in the super-resolved image;

[0154] According to the difference of the image parameters between at least one pixel and an adjacent pixel, a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image is calculated.

[0155] In one embodiment, the corresponding pixels in the first super-resolved image and the second super-resolved image include: a target pixel on one of the first super-resolved image and the second super-resolved image, and a pixel corresponding to the target pixel on the other of the first super-resolved image and the second super-resolved image;

[0156] Calculating the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image according to the difference of image parameters between at least one pixel and an adjacent pixel includes:

[0157] Calculate the fusion ratio of the target pixel according to the difference in image parameters between the target pixel and the adjacent pixels in the super-resolution image;

[0158] According to the fusion ratio of the target pixel, the fusion ratio of the pixel corresponding to the target pixel is calculated; or, the difference in image parameters between the pixel corresponding to the target pixel and the adjacent pixel in the super-resolution image is calculated, and according to the difference in image parameters between the pixel corresponding to the target pixel and the adjacent pixel in the super-resolution image, the fusion ratio of the pixel corresponding to the target pixel is calculated.

[0159] In one embodiment, calculating the fusion ratio of the target pixel according to the difference in image parameters between the target pixel and adjacent pixels in the super-resolved image includes:

[0160] The proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold among all adjacent pixels corresponding to the target pixel is determined as the fusion ratio of the target pixel.

[0161] In one embodiment, determining the proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold among all adjacent pixels corresponding to the target pixel as the fusion ratio of the target pixel includes:

[0162] Determine the number of adjacent pixel points whose image parameter difference is greater than or equal to a preset threshold as a first number;

[0163] A ratio of the first number to the second number is calculated as a fusion ratio of the target pixel, wherein the second number is the number of all adjacent pixel points corresponding to the target pixel point.

[0164] In one embodiment, the fusion ratio acquisition module 6400 may also be used to:

[0165] Calculate the average value of the image parameter difference between the target pixel and each adjacent pixel in the super-resolved image;

[0166] According to the average value, the fusion ratio of the target pixels is determined.

[0167] In one embodiment, the pixel fusion module 6500 may also be used for:

[0168] According to the fusion ratio of the target pixel and the fusion ratio of the pixel corresponding to the target pixel, the image parameters of the target pixel and the image parameters of the pixel corresponding to the target pixel are weighted averaged to obtain the image parameters of the fused pixel.

[0169] In one embodiment, the image processing apparatus may further include:

[0170] A module for extracting a corresponding face portion in the first super-resolution image as a first face image region;

[0171] A module for extracting a corresponding face portion in the second super-resolution image as a second face image region;

[0172] A module for fusing corresponding pixels in a first face image region and a second face image region to obtain a fused face image region.

[0173] In one embodiment, the image parameters include grayscale values ​​or pixel values.

[0174] Those skilled in the art will appreciate that the image processing device 6000 can be implemented in various ways. For example, the image processing device 6000 can be implemented by configuring a processor with instructions. For example, instructions can be stored in a ROM, and when the device is started, the instructions are read from the ROM into a programmable device to implement the image processing device 6000. For example, the image processing device 6000 can be solidified into a dedicated device (such as an ASIC). The image processing device 6000 can be divided into independent units, or they can be combined together for implementation. The image processing device 6000 can be implemented by one of the various implementations described above, or can be implemented by a combination of two or more of the various implementations described above.

[0175] In this embodiment, the image processing device 6000 can have multiple implementation forms. For example, the image processing device 6000 can be a functional module running in any software product or application that provides network access services, or it can be a peripheral embedded component, plug-in, patch, etc. of these software products or applications, or it can also be these software products or applications themselves.

[0176] <Electronic equipment>

[0177] In this embodiment, an electronic device 7000 is further provided. In one aspect, the electronic device 7000 may include the aforementioned image processing device 6000 .

[0178] On the other hand, Figure 7 As shown, it may include a memory 7100 and a processor 7200, wherein the memory 7100 is used to store executable instructions; the instructions are used to control the processor 7200 to execute the aforementioned image processing method.

[0179] In this embodiment, the electronic device 7000 can be any electronic product having a memory 7100 and a processor 7200, such as a mobile phone, a tablet computer, a PDA, a desktop computer, a laptop computer, a workstation, a game console, or the like.

[0180] In this embodiment, the electronic device 7000 may also include other hardware devices, such as Figure 1 An electronic device 1000 is shown.

[0181] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. According to the present embodiment, an image processing method, device and electronic device are provided. According to a preset magnification factor, a first super-resolution processing is performed on the original image to obtain a first super-resolution image. According to the magnification factor, a second super-resolution processing is performed on the original image to obtain a second super-resolution image. The fusion ratio of the corresponding pixel points in the first super-resolution image and the second super-resolution image is obtained. According to the fusion ratio, the corresponding pixel points in the first super-resolution image and the second super-resolution image are fused to obtain fused pixel points, and a fused image is generated according to the fused pixel points. In this way, while improving the resolution of the original image, the visual effect of the fused image can also be improved.

[0182] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0183] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0184] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0185] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0186] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0187] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0188] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0189] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0190] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An image processing method, wherein: include: Get the original image; Performing a first image super-resolution process on the original image according to a preset magnification to obtain a first super-resolution image; Performing second image super-resolution processing on the original image according to the magnification factor to obtain a second super-resolution image; Obtaining a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image; According to the fusion ratio, corresponding pixel points in the first super-resolution image and the second super-resolution image are fused to obtain fused pixel points; generating a fused image according to the fused pixel points; The acquiring the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image comprises: Determine a difference in image parameters between at least one pixel point among corresponding pixels in the first super-resolved image and the second super-resolved image and an adjacent pixel point in the super-resolved image; According to the difference in image parameters between the at least one pixel and an adjacent pixel, a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image is calculated.

2. The image processing method according to claim 1, wherein: The acquiring the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image comprises: Determine the difference in image parameters between a pixel point in the original image and an adjacent pixel point; Calculating a fusion ratio corresponding to a pixel point in the original image according to a difference in image parameters between a pixel point in the original image and an adjacent pixel point; According to the fusion ratio corresponding to the pixels in the original image, the fusion ratio of the pixels in the first super-resolution image and the second super-resolution image corresponding to the pixels in the original image is determined.

3. The image processing method according to claim 1, wherein: The corresponding pixels in the first super-resolved image and the second super-resolved image include: a target pixel on one of the first super-resolved image and the second super-resolved image, and a pixel corresponding to the target pixel on the other of the first super-resolved image and the second super-resolved image; The step of calculating the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image according to the difference in image parameters between the at least one pixel and an adjacent pixel comprises: Calculating the fusion ratio of the target pixel according to the difference in image parameters between the target pixel and adjacent pixels in the super-resolution image; According to the fusion ratio of the target pixel point, the fusion ratio of the pixel point corresponding to the target pixel point is calculated; or, the difference in image parameters between the pixel point corresponding to the target pixel point and the adjacent pixel points in the super-resolution image is calculated, and according to the difference in image parameters between the pixel point corresponding to the target pixel point and the adjacent pixel points in the super-resolution image, the fusion ratio of the pixel point corresponding to the target pixel point is calculated.

4. The image processing method according to claim 3, wherein: The step of calculating the fusion ratio of the target pixel point according to the difference in image parameters between the target pixel point and adjacent pixels in the super-resolution image includes: The proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold among all adjacent pixels corresponding to the target pixel is determined as the fusion ratio of the target pixel.

5. The image processing method according to claim 4, wherein: The step of determining the proportion of adjacent pixels whose image parameter difference is greater than or equal to a preset threshold among all adjacent pixels corresponding to the target pixel as the fusion ratio of the target pixel includes: Determine the number of adjacent pixel points whose image parameter difference is greater than or equal to a preset threshold as a first number; A ratio of the first number to the second number is calculated as a fusion ratio of the target pixel, wherein the second number is the number of all adjacent pixel points corresponding to the target pixel point.

6. The image processing method according to claim 3, wherein: The step of calculating the fusion ratio of the target pixel point according to the difference in image parameters between the target pixel point and adjacent pixels in the super-resolution image includes: Calculating an average value of the image parameter difference between the target pixel and each adjacent pixel in the super-resolved image; The fusion ratio of the target pixel is determined according to the average value.

7. The image processing method according to claim 3, wherein: The step of fusing corresponding pixels in the first super-resolution image and the second super-resolution image to obtain fused pixels comprises: According to the fusion ratio of the target pixel point and the fusion ratio of the pixel point corresponding to the target pixel point, the image parameters of the target pixel point and the image parameters of the pixel point corresponding to the target pixel point are weighted averaged to obtain the image parameters of the fused pixel point.

8. The image processing method according to claim 1, wherein: The image processing method further comprises: Extracting a portion corresponding to a human face in the first super-resolution image as a first human face image region; Extracting a corresponding face portion in the second super-resolution image as a second face image region; The corresponding pixels in the first face image region and the second face image region are fused to obtain the fused face image region.

9. The image processing method according to any one of claims 2 to 7, wherein: The image parameters include grayscale values ​​or pixel values.

10. An image processing device, wherein: include: An image acquisition module, used for acquiring original images; A first super-resolution module, used for performing a first image super-resolution processing on the original image according to a preset magnification to obtain a first super-resolution image; A second super-resolution module is used to perform a second image super-resolution processing on the original image according to the magnification factor to obtain a second super-resolution image; A fusion ratio acquisition module, used to acquire the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image; A pixel fusion module, used for fusing corresponding pixels in the first super-resolution image and the second super-resolution image according to the fusion ratio to obtain fused pixels; as well as, An image generation module, used to generate a fused image based on the fused pixel points; The acquiring the fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image comprises: Determine a difference in image parameters between at least one pixel point among corresponding pixels in the first super-resolved image and the second super-resolved image and an adjacent pixel point in the super-resolved image; According to the difference in image parameters between the at least one pixel and an adjacent pixel, a fusion ratio of corresponding pixels in the first super-resolution image and the second super-resolution image is calculated.

11. An electronic device, wherein: include: The image processing device according to claim 10; or, A processor and a memory, wherein the memory is used to store executable instructions, and the instructions are used to control the processor to execute the image processing method according to any one of claims 1-9.

Images