Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium. The image processing method comprises the following steps: acquiring multiple frames of original images; obtaining a reference frame from the multiple frames of original images; performing registration on the original image according to the reference frame to obtain a registered image; and performing interpolation processing on the registered image to obtain an amplified image. According to the image processing method, the information of the original image can be reserved to the maximum extent, and the resolution of the amplified image is increased.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and in particular relates to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] Image processing is a technique that uses various algorithms to process digital images. Image processing has a wide range of applications in many fields, especially in science and technology. Image enlargement is an important part of image processing. There are two main types of image enlargement techniques: interpolation methods and super-resolution methods. Interpolation methods: This is a simple and commonly used image enlargement technique. It increases the size of the image by inserting new pixels between the original pixels. Common interpolation methods include nearest neighbor interpolation, bilinear interpolation, and cubic spline interpolation. Super-resolution methods: This is a more complex image enlargement technique that uses complex algorithms to restore high-resolution images from low-resolution images. Super-resolution methods can be further divided into learning-based methods and optimization-based methods. Learning-based methods usually use deep learning techniques such as convolutional neural networks (CNNs) to learn the mapping relationship between low-resolution and high-resolution images.

[0003] Image magnification is a very active and growing research field that plays an important role in many applications. However, there is still a lack of an image magnification method that can improve image resolution and reduce the loss of original image information. Summary of the invention

[0004] The purpose of the present disclosure is to provide an image processing method and device, an electronic device and a storage medium, which are used to solve the problem that the image magnification method in the prior art has low resolution and loses the original image information.

[0005] In a first aspect, the present disclosure provides an image processing method. The image processing method comprises: acquiring multiple frames of original images; acquiring a reference frame from the multiple frames of original images; registering the original images according to the reference frame to obtain a registered image; and interpolating the registered image to obtain an enlarged image.

[0006] In an implementation manner of the first aspect, acquiring multiple frames of original images includes acquiring multiple frames of original images in a RAW domain.

[0007] In an implementation of the first aspect, interpolating the registered image to obtain an enlarged image includes: interpolating the registered RAW domain image in the horizontal and vertical directions to obtain a preliminary RGB domain image; enlarging the preliminary RGB domain image to obtain an enlarged RGB domain image; and fusing the enlarged RGB domain image in the horizontal and vertical directions to obtain the enlarged image.

[0008] In an implementation of the first aspect, registering the original image according to the reference frame to obtain a registered image includes: constructing an image pyramid comprising multiple layers for each of the original images, the bottom layer corresponds to the original image, and images with different downsampling ratios are generated layer by layer; starting from the top layer, relative to the reference frame of the current layer, finding matching points from images of each current layer; calculating a motion vector between the reference frame of the current layer and the image of the current layer based on the matching points; and obtaining a motion vector in the next layer of the image pyramid based on the motion vector of the current layer, until a motion vector for the original image is obtained for registering the original image.

[0009] In an implementation of the first aspect, registering the original image according to the reference frame to obtain a registered image includes: obtaining at least one downsampled image of the original image and a downsampled reference frame of the reference frame; obtaining a transformation matrix between the downsampled reference frame and each of the downsampled images according to matching information between the downsampled reference frame and the downsampled image; obtaining a motion vector of a matching area in the downsampled image using the transformation matrix; and registering the original image according to the reference frame using the motion vector to obtain the registered image.

[0010] In an implementation of the first aspect, obtaining a transformation matrix between the downsampled reference frame and the downsampled image according to matching information between the downsampled reference frame and the downsampled image includes: obtaining multiple feature regions in the downsampled reference frame; obtaining the matching regions in the downsampled image that match the feature regions; and obtaining a transformation matrix between the downsampled reference frame and the downsampled image according to a matching relationship between the matching regions and the feature regions.

[0011] In an implementation of the first aspect, there are at least two downsampled images, and using the transformation matrix to obtain the motion vector of the matching area in the downsampled image includes: using the transformation matrix to obtain the motion vector of the matching area in the first downsampled image; and calibrating the motion vector using a second downsampled image, the resolution of the second downsampled image being greater than that of the first downsampled image.

[0012] In an implementation of the first aspect, obtaining a transformation matrix between the downsampled reference frame and the downsampled image includes: obtaining a position of a pixel point in the reference frame; obtaining a corresponding position of the pixel point in the downsampled image; and obtaining a transformation matrix between the downsampled reference frame and the downsampled image according to the position of the pixel point in the downsampled reference frame and the corresponding position of the pixel point in the downsampled image.

[0013] In an implementation of the first aspect, interpolating the registered image to obtain an enlarged image includes: performing interpolation processing on the registered image in the horizontal and vertical directions respectively to obtain a green channel image and a color difference image in the vertical and horizontal directions; enlarging the green channel image and the color difference image in the vertical and horizontal directions to obtain an enlarged green channel image and a color difference image in the vertical and horizontal directions; and fusing the green channel image and the color difference image enlarged in the vertical and horizontal directions to obtain the enlarged image.

[0014] In an implementation of the first aspect, the image processing method further includes: obtaining the positional relationship of the operation points between the multiple frames of images based on the registered image; obtaining the weights of the operation points using optimization estimation based on the imaging system parameters and the positional relationship; performing weighted processing on the operation points using the weights to obtain the pixel coordinates of the enlarged image; and obtaining the enlarged green channel image in the vertical and horizontal directions using the pixel coordinates of the enlarged image.

[0015] In an implementation of the first aspect, the green channel image and the color difference image that are amplified in the vertical and horizontal directions are fused to obtain the enlarged image, including: obtaining a red channel image and a blue channel image that are amplified in the vertical and horizontal directions according to the green channel image and the color difference image that are amplified in the vertical and horizontal directions; and fusing the green channel image, the red channel image, and the blue channel image that are amplified in the vertical and horizontal directions to obtain the enlarged image.

[0016] In an implementation of the first aspect, fusing the green channel image, red channel image, and blue channel image amplified in the vertical and horizontal directions includes: weighting the green channel image, red channel image, and blue channel image amplified in the vertical and horizontal directions according to the horizontal weight and the vertical weight, respectively, to obtain the green channel image, red channel image, and blue channel image weighted in the vertical and horizontal directions; and fusing the green channel image, red channel image, and blue channel image weighted in the vertical and horizontal directions to obtain the weighted fused green channel image, red channel image, and blue channel image; wherein a method for obtaining weights in the weighted processing includes: obtaining pixel coordinates of the registered image and coordinates of pixel points around the pixel points; matching the pixel coordinates with the pixel coordinates around the pixel point to obtain the number of pixel points whose difference between the two is less than a threshold; and obtaining the horizontal weight and the vertical weight respectively according to the number of pixel points whose difference between the horizontal and vertical directions is less than a threshold.

[0017] In an implementation of the first aspect, the image processing method also includes: the horizontal direction weight is the ratio of the number of pixels whose horizontal direction difference is less than a threshold to the number of pixels whose horizontal and vertical direction differences are less than a threshold, and the vertical direction weight is the ratio of the number of pixels whose vertical direction difference is less than a threshold to the number of pixels whose horizontal and vertical directions differences are less than a threshold.

[0018] In an implementation of the first aspect, performing interpolation processing based on the registered image to obtain an enlarged image also includes: obtaining a motion area based on the registered image, processing the motion area to obtain a deblurred and smeared image; and performing interpolation processing based on the deblurred and smeared image to obtain the enlarged image.

[0019] In a second aspect, the present disclosure provides an image processing device. The image processing device includes: a storage module configured to store information associated with an original image; and a processing module electrically coupled to the storage module and configured to: acquire multiple frames of original images; acquire a reference frame from the multiple frames of original images; register the original images according to the reference frame to acquire a registered image; and perform interpolation processing on the registered image to acquire an enlarged image.

[0020] In a third aspect, the present disclosure provides an electronic device, comprising: a memory configured to store a processor-executable program; and a processor configured to execute the program so that the electronic device performs the image processing method according to any one of the first aspects.

[0021] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the image processing method according to any one of the first aspects.

[0022] According to the embodiment of the present disclosure, multiple frames of original images are directly obtained, reference frames are obtained from the multiple frames of original images, the original images are registered according to the reference frames, and then the registered images are interpolated to obtain enlarged images. This image processing method can retain the information of the original image to the maximum extent and increase the resolution of the enlarged image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Shown is a schematic diagram of an application scenario of the image processing system described in the present disclosure.

[0024] Figure 2 Shown is a flowchart of the image processing method described in an embodiment of the present disclosure.

[0025] Figure 3 Shown is a flowchart of the image processing method described in an embodiment of the present disclosure.

[0026] Figure 4 Shown is a flowchart of the image processing method described in an embodiment of the present disclosure.

[0027] Figure 5 Shown is a flowchart of the image processing method described in an embodiment of the present disclosure.

[0028] Figure 6 Shown is a flowchart of the image processing method described in an embodiment of the present disclosure.

[0029] Figure 7 Shown is a flowchart of the image processing method described in an embodiment of the present disclosure.

[0030] Figure 8 Shown is a schematic diagram of a super-resolution grid according to an embodiment of the present disclosure.

[0031] Fig. 9 Shown is a schematic diagram of the weight calculation method described in an embodiment of the present disclosure.

[0032] Fig.10 Shown is a structural schematic diagram of the image processing device described in an embodiment of the present disclosure.

[0033] Fig.11 Shown is a schematic diagram of the structure of an electronic device described in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following is an explanation of the embodiments of the present disclosure by specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0035] It should be noted that the illustrations provided in the following embodiments are only used to schematically illustrate the basic concept of the present disclosure. Therefore, the drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0036] Using multiple frames to achieve image enlargement is also called multi-frame super-resolution restoration. At present, the existing methods of using multiple frames to achieve image enlargement are mainly based on iterative methods or learning-based methods. The iterative method regards image enlargement as an optimization problem, and generally obtains the best solution through iteration, thereby obtaining the enlarged result. The learning-based method mainly obtains a model between a low-resolution image and a high-resolution image through a large amount of training. When the image needs to be enlarged, the image is input into the model to obtain the enlarged result. The algorithm based on iteration is often more complicated to calculate, which is not conducive to real engineering applications. The learning-based method relies too much on the data set used for training and does not take into account the differences between imaging systems, so the enlargement effect is limited. In addition, most of the existing methods are based on RGB (Red green blue, referred to as RGB) domain or YUV (Luma and chrominance, referred to as YUV) domain, and these images have undergone certain processing processes, such as denoising, demosaicing, sharpening, etc., which often lose the most original information of the image.

[0037] At least to address the above-mentioned problems, an embodiment of the present disclosure provides an image processing method, which includes: acquiring multiple frames of original images; acquiring a reference frame from the multiple frames of original images; registering the original images according to the reference frame to obtain a registered image; and interpolating the registered image to obtain an enlarged image.

[0038] In the disclosed embodiment, the image processing method directly obtains multiple frames of original images, obtains reference frames from the multiple frames of original images, registers the original images according to the reference frames, and then performs interpolation processing on the registered images to obtain enlarged images. This image processing method can retain the information of the original image to the maximum extent and increase the resolution of the enlarged image.

[0039] In some embodiments, the RAW domain data captured by the camera is used as input, and the RAW data is interpolated in the horizontal and vertical directions to obtain a preliminary RGB domain image. The RGB image is directly enlarged and then the final enlarged result is obtained by the directional fusion algorithm. Compared with the traditional demosaic and then enlargement, the image information can be restored to the maximum extent and aliasing can be avoided. In some embodiments, while ensuring the operation efficiency, by generating an image pyramid, the tracking of feature points or blocks and the calculation of homography matrices are performed in layers to achieve high-precision registration at the sub-pixel level. In some embodiments, by calculating the distribution of the values ​​of the pixel points at the same position in different images, it is determined whether the point is a moving point, and special processing is performed on the moving points to avoid the blurring and smearing phenomena that occur when the moving objects are superimposed, and better handle the smearing generated when the picture moves. In some embodiments, using multiple frames of images as input, the images are first registered so that the picture information of the multiple frames is aligned. For the target point of the enlarged image, the points around the target point on the original image are used, combined with the imaging system parameter information, and the Wiener filter is used to obtain the value of the target point with the best estimate.

[0040] Hereinafter, the specific technical solutions of the present disclosure will be described through specific exemplary embodiments in conjunction with the accompanying drawings.

[0041] Figure 1 The image processing system can be used to implement the image processing method provided in the embodiment of the present disclosure, but the application scenario of the image processing method provided in the embodiment of the present disclosure is not limited to Figure 1 The image processing system shown in FIG. Figure 1 As shown, the image processing system 1 includes an acquisition device 11, a processor 12 and a display terminal 13. The image processing method provided in the embodiment of the present disclosure can be applied to the processor 12.

[0042] Figure 1 The processor 12 in the embodiment may be a single processor or a processor cluster or a cloud computing center composed of multiple processors, and the specific details are not limited here. Figure 1 Only one acquisition device 11, one processor 12 and one display terminal 13 are shown, but it should be understood that Figure 1 The examples are only used to understand this solution, and the specific number of display terminals and processors should be flexibly determined based on actual conditions.

[0043] In some other implementations, the image processing system 1 may not include the display terminal 13, but only include the acquisition device 11 and the processor 12 with display function. The processor 12 with display function may include a tablet computer, a laptop computer, a PDA, a mobile phone, a personal computer (PC), and a voice interaction device, or a monitoring device, a face recognition device, etc., which is not limited here.

[0044] The technical solutions in the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present disclosure.

[0045] The following embodiments of the present disclosure provide an image processing method, for example, Figure 1 The processor 12 is shown to implement, but the disclosure is not limited thereto. Figure 2 The flowchart of the image processing method according to the embodiment of the present disclosure is shown as follows: Figure 2 As shown, the image processing method includes steps S11 to S14.

[0046] Step S11, acquiring multiple frames of original images. The original images are original images in the RAW domain, and the original images in the RAW domain can be acquired by using a camera.

[0047] Step S12, obtaining a reference frame from the multiple frames of original images.

[0048] In some possible implementations, the reference frame is the sharpest original image and also the clearest original image. Obtaining the reference frame from the multiple original image frames is to obtain the clearest original image from the multiple original image frames as the reference frame. In other possible implementations, a high-frequency operator is used to perform convolution processing on the multiple original image frames respectively, and the image with the largest high-frequency response value is the reference frame.

[0049] Step S13, registering the original image according to the reference frame to obtain a registered image.

[0050] In some possible implementations, there is a slight displacement between the frames of the multiple original images when they are captured, resulting in slight differences between different frames of the original images of the same scene. By registering the original images according to the reference frame, a more accurate registered image can be obtained. The registered image is an image obtained by registering the multiple original images.

[0051] Step S14, interpolating the registered image to obtain an enlarged image. In some possible implementations, interpolating the registered image includes interpolating, enlarging, and fusing to obtain the enlarged image.

[0052] In the disclosed embodiment, the image processing method directly obtains multiple frames of original images, obtains reference frames from the multiple frames of original images, registers the original images according to the reference frames, and then performs interpolation processing on the registered images to obtain enlarged images. This image processing method can retain the information of the original image to the maximum extent and increase the resolution of the enlarged image.

[0053] In one embodiment of the present disclosure, in the process of interpolating the registered image to obtain the enlarged image, the registered RAW domain image is interpolated in the horizontal and vertical directions to obtain a preliminary RGB domain image, the preliminary RGB domain image is enlarged to obtain an enlarged RGB domain image, and the enlarged RGB domain image is fused in the horizontal and vertical directions to obtain the enlarged image.

[0054] In one embodiment of the present disclosure, in the process of registering the original image according to the reference frame to obtain the registered image, an image pyramid comprising multiple layers is constructed for each of the original images, the bottom layer corresponds to the original image, and images with different downsampling ratios are generated layer by layer. Subsequently, starting from the top layer, matching points are found from the images of each current layer relative to the reference frame of the current layer. Next, the motion vector between the reference frame of the current layer and the image of the current layer is calculated based on the matching points. Then, based on the motion vector of the current layer, the motion vector in the next layer of the image pyramid is obtained until the motion vector for the original image is obtained for registering the original image.

[0055] Figure 3 The flowchart of the image processing method described in the embodiment of the present disclosure is shown. Figure 3 As shown, step S13 includes the following steps S131 to S134.

[0056] Step S131, obtaining at least one downsampled image of the original image and a downsampled reference frame of the reference frame.

[0057] In some possible implementations, an image pyramid is constructed for the input original image, and the pyramid includes m layers (m≥2), the bottom layer of the image pyramid is the original image, and each layer upward is a downsampled image of different magnifications, but the present disclosure is not limited to this.

[0058] Step S132: acquiring a transformation matrix between the down-sampled reference frame and each of the down-sampled images according to matching information between the down-sampled reference frame and the down-sampled images.

[0059] Step S133: using the transformation matrix to obtain a motion vector of the matching area in the downsampled image. In some possible implementations, the motion vector includes displacement information between images.

[0060] Step S134: using the motion vector, register the original image according to the reference frame to obtain the registered image.

[0061] Figure 4 The flowchart of the image processing method described in the embodiment of the present disclosure is shown. Figure 4 As shown, step S132 includes the following steps S1321 to S1323.

[0062] Step S1321, obtaining multiple feature regions in the downsampled reference frame.

[0063] Step S1322: Acquire the matching region in the downsampled image that matches the feature region.

[0064] Step S1323: acquiring a transformation matrix between the downsampled reference frame and the downsampled image according to the matching relationship between the matching region and the feature region. The transformation matrix is ​​a homography matrix between the downsampled reference frame and the downsampled image.

[0065] In some possible implementations, the matching relationship is the transformation relationship between the coordinates of the pixel points representing the same object between the two frames of images.

[0066] In a possible implementation of the embodiment of the present disclosure, the image is divided into N×N feature regions on the downsampled reference frame, and a region matching the feature region is obtained in the downsampled image as the matching region. According to the matching relationship between the matching region and the feature region, a transformation matrix T between the downsampled reference frame and the downsampled image is obtained.

[0067] Figure 5 The flowchart of the image processing method described in the embodiment of the present disclosure is shown. Figure 5 As shown, the number of the downsampled images is at least two, and step S133 includes the following steps S1331 to S1332.

[0068] Step S1331: using the transformation matrix to obtain a motion vector of the matching area in the first down-sampled image.

[0069] Step S1332: calibrate the motion vector using a second down-sampled image, where the resolution of the second down-sampled image is greater than that of the first down-sampled image.

[0070] In some possible implementations, there are at least two downsampled images. The displacement vector between the first downsampled image and the downsampled reference frame can be obtained as (x0, y0) through the transformation matrix of the first downsampled image. Then, the preliminary displacement vector of the second downsampled image of the next layer relative to the downsampled reference frame is preset to be (2x0, 2y0). After the second downsampled image is aligned using the preliminary displacement vector, feature matching is performed on the aligned pixels to obtain the feature matching displacement vector (Δx, Δy). The motion vector obtained using the feature matching displacement vector and the preliminary displacement vector is (2x0+Δx, 2y0+Δy).

[0071] In the disclosed embodiment, the first downsampled image is used to obtain a motion vector, and then the motion vector of the second downsampled image is obtained based on the motion vector, and the motion vector is calibrated using the second downsampled image. This method can improve the accuracy of image registration and achieve high-precision registration at the sub-pixel level.

[0072] Figure 6 The flowchart of the image processing method described in the embodiment of the present disclosure is shown. Figure 6 As shown, step S1323 includes the following steps S13231 to S13233.

[0073] Step S13231, obtaining the position of the pixel point in the reference frame.

[0074] Step S13232, obtaining the corresponding position of the pixel point in the downsampled image.

[0075] Step S13233: Acquire a transformation matrix between the downsampled reference frame and the downsampled image according to the position of the pixel point in the downsampled reference frame and the corresponding position of the pixel point in the downsampled image.

[0076] In some possible implementations, for example, two frames of images F1 and F2, for any pixel point P(x, y) on image F1, the transformation matrix between the downsampled reference frame and the downsampled image is matrix T, then (x', y') on image F2 = T×(x, y), (x', y') is the corresponding position of the pixel point P(x, y) in image F2.

[0077] Figure 7 The flowchart of the image processing method described in the embodiment of the present disclosure is shown. Figure 7 As shown, step S14 includes the following steps S141 to S143.

[0078] Step S141, interpolating the registered image in the horizontal and vertical directions respectively to obtain a green channel image and a color difference image in the vertical and horizontal directions. The color difference image is obtained by subtracting the brightness image from the primary color image, for example, the color difference image includes a blue color difference image (BY) and a red color difference image (RY).

[0079] Step S142, amplifying the green channel image and the color difference image in the vertical direction and the horizontal direction to obtain amplified green channel image and color difference image in the vertical direction and the horizontal direction.

[0080] In some possible implementations, the color difference images in the vertical and horizontal directions are subjected to multi-frame superposition and linear amplification to obtain denoised color difference images in the vertical and horizontal directions. The green channel images in the vertical and horizontal directions are subjected to a multi-frame adaptive Wiener method to obtain amplified green channel images in the vertical and horizontal directions.

[0081] Step S143, fusing the green channel image and the color difference image amplified in the vertical direction and the horizontal direction to obtain the amplified image.

[0082] In one embodiment of the present disclosure, the method may further include: obtaining the positional relationship of the operation points between the multiple frames of images according to the registered image. Obtaining the weight of the operation points by using optimization estimation according to the imaging system parameters and the positional relationship. Performing weighted processing on the operation points by using the weights to obtain the pixel coordinates of the enlarged image. Obtaining the enlarged green channel image in the vertical and horizontal directions by using the pixel coordinates of the enlarged image.

[0083] In some possible implementations, the reference frame is placed on a super-resolution grid, and the original images of the remaining frames are set at corresponding positions of the super-resolution grid according to the registration results. The degradation model of the imaging system is obtained based on the imaging system parameter information composed of different cameras, lenses, sensors and other devices. According to the degradation model combined with the positional relationship of the original image, the weights of the operation points on the super-resolution grid are obtained using optimization estimation. The operation points are weighted according to the weights to obtain the pixel coordinates of the enlarged image. The enlarged green channel image in the vertical and horizontal directions is obtained using the pixel coordinates of the enlarged image. It should be noted that the above is only one possible implementation of the present disclosure, and the present disclosure is not limited to this.

[0084] In some other possible implementations, Figure 8The diagram is a schematic diagram of the super-resolution grid according to the embodiment of the present disclosure. Assuming that the original image is three frames of original images, the positions of the pixels of the reference frame and the pixels of the remaining frames are as follows: Figure 8 The calculation formula of the lens transfer function of the imaging system is:

[0085]

[0086] ρ is the optical frequency of the lens, ρ c is the optical cutoff frequency of the lens. c The calculation formula is:

[0087]

[0088] λ is the wavelength of light, and f is the focal length of the lens. The calculation formula of the sensor transfer function of the imaging system is:

[0089] H sensor (u,v)=sinc(mu,nv),

[0090] m and n are the horizontal and vertical dimensions of the sensor. The calculation formula of the transfer function of the imaging system is:

[0091] H s =H lens ×H sensor ,

[0092] For H s By performing inverse Fourier transform, the corresponding point spread function can be obtained.

[0093] The positional relationship of the operation points between the multiple frames of images is obtained according to the registered image. The operation point is p = [p1, p2, p3 ... p k ] T , then the result of the target point to be interpolated is:

[0094] d=W T ×p,

[0095] W=[w1,w2,w3...w k ] T is the weight of the operation point. The operation point is weighted according to the weight to obtain the pixel coordinates of the enlarged image. The weight calculation formula is:

[0096] W=G -1 ×D,

[0097] G is the cross-correlation matrix between input points, obtained using the point spread function, and D is the cross-correlation vector between the input point and the target point to be interpolated.

[0098] In one embodiment of the present disclosure, the green channel image and the color difference image that are amplified in the vertical and horizontal directions are fused to obtain the enlarged image, including: obtaining the red channel image and the blue channel image that are amplified in the vertical and horizontal directions according to the green channel image and the color difference image that are amplified in the vertical and horizontal directions; and fusing the green channel image, the red channel image, and the blue channel image that are amplified in the vertical and horizontal directions to obtain the enlarged image.

[0099] In some possible implementations, according to the green channel image G after vertical enlargement v , horizontally enlarged green channel image G h , vertically enlarged color difference image GB v and GR v , the color difference image GB after horizontal enlargement h and GR h Get the vertically enlarged red channel image R v , horizontally enlarged red channel image R h , vertically enlarged blue channel image B v And the blue channel image B after horizontal enlargement h The calculation formula is:

[0100]

[0101] The green channel image G amplified in the vertical and horizontal directions is v and G h , red channel image R v and R h And the blue channel image B v and B h Fusion is performed to obtain the enlarged image.

[0102] In one embodiment of the present disclosure, fusing the green channel image, the red channel image, and the blue channel image that have been magnified in the vertical direction and the horizontal direction includes:

[0103] The green channel image, red channel image and blue channel image amplified in the vertical direction and horizontal direction are weighted according to the horizontal weight and the vertical weight, so as to obtain the green channel image, red channel image and blue channel image weighted in the vertical direction and horizontal direction.

[0104] The weighted green channel image, red channel image and blue channel image in the vertical direction and the horizontal direction are fused to obtain a weighted fused green channel image, red channel image and blue channel image.

[0105] The method for obtaining weights in the weighted processing includes the following steps S21 to S23.

[0106] Step S21, obtaining the coordinates of the pixel points of the registered image and the coordinates of the pixel points around the pixel points.

[0107] Step S22, matching the coordinates of the pixel point with the coordinates of the pixels around the pixel point, and obtaining the number of pixels whose difference between the two is less than a threshold.

[0108] Step S23, obtaining the horizontal direction weight and the vertical direction weight respectively according to the number of pixel points whose horizontal direction and vertical direction differences are less than a threshold.

[0109] In some possible implementations, weighted processing is performed on the green channel image, red channel image, and blue channel image after vertical and horizontal amplification, respectively, according to the horizontal weight and the vertical weight, so as to obtain the vertical and horizontal weighted green channel image, red channel image, and blue channel image. The method for obtaining the weights in the weighted processing is:

[0110] Fig. 9 The diagram is a schematic diagram of the weight calculation method described in the embodiment of the present disclosure. h Point G at position (i,j) h (i,j), and compare it with the surrounding pixel points G h (i,j),G h (i,j),G h (i,j) and G h (i, j) for comparison, the position relationship of each point is as follows Fig. 9 shown.

[0111] According to the coordinates of the pixel point and the coordinates of the pixels around the pixel point, the number of pixels whose horizontal difference between the two is less than the threshold ε is obtained, which is recorded as N h Similarly, the number of pixels whose vertical difference value is less than the threshold ε is obtained, which is recorded as N v . Then the horizontal weight w h The calculation formula is:

[0112]

[0113] Vertical weight w v The calculation formula is:

[0114]

[0115] According to the horizontal weight w h and the vertical weight wv The green channel image G after vertical enlargement v And the green channel image G after horizontal enlargement h , the red channel image R after vertical enlargement v , horizontally enlarged red channel image R h , vertically enlarged blue channel image B v And the blue channel image B after horizontal enlargement h The calculation formula for weighted fusion is:

[0116]

[0117] The enlarged image is obtained according to the enlarged red channel image R, the enlarged green channel image G and the enlarged blue channel image B.

[0118] In one embodiment of the present disclosure, the horizontal weight is the ratio of the number of pixels whose horizontal difference is less than a threshold to the number of pixels whose horizontal and vertical differences are less than a threshold, and the vertical weight is the ratio of the number of pixels whose vertical difference is less than a threshold to the number of pixels whose horizontal and vertical differences are less than a threshold.

[0119] In one embodiment of the present disclosure, step S14 further includes steps S144 to S145.

[0120] Step S144, acquiring a motion region according to the registered image, and processing the motion region to acquire a deblurred and smeared image.

[0121] Step S145 , performing interpolation processing on the deblurred and smeared image to obtain the enlarged image.

[0122] In the disclosed embodiment, the motion region of the registered image is obtained and processed, which can effectively process the ghost image generated when the picture moves, thereby avoiding the blurred ghost image phenomenon when the moving objects are superimposed.

[0123] Fig.10 The diagram is a schematic diagram of the structure of the image processing device according to the embodiment of the present disclosure. Fig.10 As shown, the image processing device 100 includes a storage module 110 and a processing module 120 .

[0124] The storage module 110 is configured to store information associated with an original image.

[0125] The processing module 120 is electrically coupled to the storage module 110. The processing module 120 is configured to acquire multiple frames of original images, acquire a reference frame from the multiple frames of original images, register the original images according to the reference frame to acquire a registered image, and perform interpolation processing on the registered image to acquire an enlarged image.

[0126] It should be noted that the actions or steps performed by the processing module 120 included in the image processing device 100 may correspond one-to-one to the steps in the image processing method in each embodiment described above, and will not be described in detail here.

[0127] In some embodiments, the image processing device 100 includes a chip. In other embodiments, the image processing device 100 is included in a system-on-chip (SoC).

[0128] In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices or methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules or units, which can be electrical, mechanical or other forms.

[0129] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure. For example, the functional modules / units in the various embodiments of the present disclosure may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0130] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0131] The embodiment of the present disclosure also provides an electronic device. Fig.11 The structure diagram of the electronic device 200 according to the embodiment of the present disclosure is shown. Fig.11 As shown, in this embodiment, the electronic device 200 includes a memory 210 and a processor 220 .

[0132] The memory 210 is used to store computer programs and includes: ROM, RAM, disk, USB flash drive, memory card or optical disk, etc., which can store program codes.

[0133] Specifically, the memory 210 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device 200 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present disclosure.

[0134] The processor 220 is connected to the memory 210 and is used to execute the computer program stored in the memory 210 so that the electronic device 200 executes the image processing method described in any embodiment of the present disclosure.

[0135] In some embodiments, the processor 220 may be a general-purpose processor, including a central processing unit (CPU) and a network processor (NP). In other embodiments, the processor 220 may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0136] In this embodiment, the electronic device 200 may further include a display 230. The display 230 is communicatively connected with the memory 210 and the processor 220, and is used to display a graphical user interface (GUI) interaction interface related to the image processing method described in the embodiment of the present disclosure.

[0137] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method described in any embodiment of the present disclosure is implemented.

[0138] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0139] The above embodiments are merely illustrative of the principles and effects of the present disclosure, and are not intended to limit the present disclosure. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present disclosure. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present disclosure shall still be covered by the claims of the present disclosure.

Claims

1. An image processing method, characterized in that: include: Get multiple frames of original images; Acquire a reference frame from the multiple frames of original images; Registering the original image according to the reference frame to obtain a registered image; as well as An interpolation process is performed on the registered image to obtain an enlarged image.

2. The image processing method according to claim 1, characterized in that: Acquiring multiple frames of original images includes: acquiring multiple frames of original images in the RAW domain.

3. The image processing method according to claim 2, characterized in that: Performing interpolation processing on the registered image to obtain an enlarged image includes: Interpolate the registered RAW domain image in horizontal and vertical directions to obtain a preliminary RGB domain image; Amplifying the preliminary RGB domain image to obtain an amplified RGB domain image; and The enlarged RGB domain image is fused in horizontal and vertical directions to obtain the enlarged image.

4. The image processing method according to claim 1, characterized in that: Registering the original image according to the reference frame to obtain a registered image includes: Constructing an image pyramid comprising multiple layers for each of the original images, wherein the bottom layer corresponds to the original image, and images with different downsampling ratios are generated layer by layer; Starting from the top layer, find matching points from the images of each current layer relative to the reference frame of the current layer; Calculating a motion vector between a reference frame of a current layer and an image of a current layer based on the matching points; and Based on the motion vector of the current layer, the motion vector in the next layer of the image pyramid is obtained, until the motion vector for the original image is obtained, so as to be used for registering the original image.

5. The image processing method according to claim 1, characterized in that: Registering the original image according to the reference frame to obtain a registered image includes: Acquire at least one downsampled image of the original image and a downsampled reference frame of the reference frame; Acquire a transformation matrix between the downsampled reference frame and each of the downsampled images according to matching information between the downsampled reference frame and the downsampled images; Obtaining a motion vector of a matching area in the downsampled image using the transformation matrix; and The original image is registered according to the reference frame using the motion vector to obtain the registered image.

6. The image processing method according to claim 5, characterized in that: According to the matching information between the downsampled reference frame and the downsampled image, acquiring the transformation matrix between the downsampled reference frame and the downsampled image comprises: Acquire a plurality of feature regions in the downsampled reference frame; Acquire the matching region in the downsampled image that matches the feature region; and According to the matching relationship between the matching area and the feature area, a transformation matrix between the downsampled reference frame and the downsampled image is obtained.

7. The image processing method according to claim 5, characterized in that: There are at least two downsampled images, and obtaining the motion vector of the matching area in the downsampled images by using the transformation matrix includes: Obtaining a motion vector of the matching area in the first downsampled image using the transformation matrix; and The motion vector is calibrated using a second down-sampled image, where the second down-sampled image has a greater resolution than the first down-sampled image.

8. The image processing method according to claim 6, characterized in that: Acquiring a transformation matrix between the downsampled reference frame and the downsampled image includes: Obtaining the position of the pixel in the reference frame; Obtaining a corresponding position of the pixel point in the downsampled image; and A transformation matrix between the downsampled reference frame and the downsampled image is obtained according to the position of the pixel point in the downsampled reference frame and the corresponding position of the pixel point in the downsampled image.

9. The image processing method according to claim 1, characterized in that: Performing interpolation processing on the registered image to obtain an enlarged image includes: Performing interpolation processing on the registered image in horizontal and vertical directions respectively to obtain a green channel image and a color difference image in vertical and horizontal directions; Amplifying the green channel image and the color difference image in the vertical direction and the horizontal direction to obtain amplified green channel image and color difference image in the vertical direction and the horizontal direction; and The green channel image and the color difference image amplified in the vertical direction and the horizontal direction are fused to obtain the amplified image.

10. The image processing method according to claim 9, characterized in that: Also includes: Acquire the positional relationship of the operation points between the multiple frames of images according to the registered images; Obtaining the weight of the operation point by using optimization estimation according to imaging system parameters and the position relationship; Performing weighted processing on the operation points using the weights to obtain pixel coordinates of the enlarged image; as well as The green channel image after the enlargement in the vertical direction and the horizontal direction is obtained by using the pixel coordinates of the enlarged image.

11. The image processing method according to claim 9, characterized in that: Performing fusion processing on the green channel image and the color difference image after being magnified in the vertical direction and the horizontal direction to obtain the magnified image includes: Obtaining a red channel image and a blue channel image that are magnified in the vertical direction and the horizontal direction according to the green channel image and the color difference image that are magnified in the vertical direction and the horizontal direction; and The green channel image, the red channel image and the blue channel image amplified in the vertical direction and the horizontal direction are fused to obtain the amplified image.

12. The image processing method according to claim 11, characterized in that: The fusion of the green channel image, the red channel image and the blue channel image after being magnified in the vertical direction and the horizontal direction includes: Performing weighted processing on the green channel image, the red channel image, and the blue channel image amplified in the vertical direction and the horizontal direction respectively according to the horizontal direction weight and the vertical direction weight, so as to obtain the green channel image, the red channel image, and the blue channel image weighted in the vertical direction and the horizontal direction; and The weighted green channel image, red channel image and blue channel image are fused according to the vertical and horizontal directions to obtain a weighted fused green channel image, red channel image and blue channel image; The method for obtaining the weight in the weighted processing includes: Acquire the coordinates of a pixel point of the registered image and the coordinates of pixel points around the pixel point; Matching the coordinates of the pixel point with the coordinates of the pixels around the pixel point to obtain the number of pixels whose difference between the two is less than a threshold; and The horizontal direction weight and the vertical direction weight are respectively obtained according to the number of pixel points whose horizontal direction and vertical direction differences are less than a threshold.

13. The image processing method according to claim 12, characterized in that: Also includes: The horizontal weight is the ratio of the number of pixels whose horizontal difference is less than the threshold to the number of pixels whose horizontal and vertical differences are less than the threshold, and the vertical weight is the ratio of the number of pixels whose vertical difference is less than the threshold to the number of pixels whose horizontal and vertical differences are less than the threshold.

14. The image processing method according to claim 1, characterized in that: Performing interpolation processing according to the registered image to obtain an enlarged image also includes: Acquire a motion region according to the registered image, and process the motion region to acquire a deblurred and smeared image; and An interpolation process is performed based on the deblurred and smeared image to obtain the enlarged image.

15. An image processing device, characterized in that: include: a storage module configured to store information associated with the original image; as well as a processing module electrically coupled to the storage module and configured to: Get multiple frames of original images; Acquire a reference frame from the multiple frames of original images; Registering the original image according to the reference frame to obtain a registered image; as well as An interpolation process is performed on the registered image to obtain an enlarged image.

16. An electronic device, characterized in that: The electronic device comprises: a memory configured to store a processor executable program; and The processor is configured to execute the program so that the electronic device performs the image processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the image processing method according to any one of claims 1 to 14 is implemented.

Images