A multi-image super-resolution imaging method guided by a frequency-domain low-pass filter

Abstract

The present invention belongs to the technical field of super-resolution imaging, and discloses a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter. The method includes: converting a low-resolution source image from the spatial domain to the frequency domain to obtain a spectrogram; performing low-pass filtering on the spectrogram to extract the structural and detailed information in the source image to generate an initial weight map; locating the boundary region and fusing multiple source images; estimating the optical flow based on global motion multi-resolution, capturing the direction and displacement information between adjacent pixels in the fused image, and determining the weight of the boundary region according to the direction and displacement information between adjacent pixels; finally, performing weighted fusion on multiple source images by combining the initial weight map and the weight of the boundary region to generate a high-resolution image. The present invention realizes an efficient super-resolution imaging method, and the generated high-resolution image provides richer and more accurate visual detail information.

Description

Technical Field

[0001] The present invention relates to the technical field of super-resolution imaging, and particularly to a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter. Background Art

[0002] Super-resolution imaging technology has wide application value in multiple industrial fields. With the continuous progress of sensor technology and the popularization of image acquisition devices, multi-source image data can be easily obtained. However, due to the fact that images obtained under different sensors or different conditions often have different characteristics and limitations, such as differences in resolution, brightness, contrast, color distribution, etc., it is difficult to achieve high-quality and high-resolution imaging.

[0003] Existing multi-image super-resolution imaging methods mainly achieve multi-image fusion super-resolution based on the spatial domain. By using certain spatial features of images, such as gradients, spatial frequencies, etc., for optimization and fusion, a high-resolution image is finally generated. However, this method is prone to block effects, has high complexity, and low computational efficiency. To overcome the defects and deficiencies of the existing methods, the present invention constructs a low-pass filter from a frequency-domain simulation to obtain the high-frequency information that is approximately lost after the high-resolution image passes through the imaging system. By designing a guided filter to capture the data structure and detailed information of the low-quality image, and then using appropriate fusion weights to reconstruct a high-resolution image.

[0004] Therefore, the present invention provides a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter. This method effectively extracts the high-frequency information of the low-resolution image, overcomes the problems of large loss of structural detail information, easy occurrence of block effects and jagged edges in the traditional method, and improves the multi-image super-resolution imaging performance and computational efficiency. Summary of the Invention

[0005] The present invention provides a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter to solve the above-mentioned technical problems in the prior art.

[0006] To have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary part is not a general review, nor is it intended to identify key / important constituent elements or delineate the protection scope of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0007] The present invention provides a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter, including the following steps:

[0008] S1. Based on the two-dimensional discrete Fourier transform technology, convert the low-resolution source image from the spatial domain to the frequency domain to obtain a spectrogram;

[0009] S2. Use a Gaussian low-pass filter to perform low-pass filtering on the spectrogram, extract the structural and detailed information in the image based on a guided filter, and generate an initial weight map;

[0010] S3. Based on the regional filtering method, remove the unrecognized pixels in the initial weight map, and use the morphological operation dilation method to combine with the initial weight map to locate the boundary region; fuse multiple images according to the processed initial weight map;

[0011] S4. Use the global motion multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image, and determine the weight of the boundary region according to the direction and displacement information between adjacent pixels;

[0012] S5. Combine the initial weight map and the weight of the boundary region to perform weighted fusion on multiple images to generate the final high-resolution image.

[0013] Preferably, the conversion formula of the spectrogram is:

[0014]

[0015] In the formula, F i (u, v) represents the frequency domain representation of the i-th image, u and v respectively represent the horizontal and vertical frequency components in the frequency domain, x and y respectively represent the horizontal and vertical coordinates of the spatial domain image, M and N respectively represent the length and width of the picture, f i (x, y) represents the spatial domain representation of the i-th image, e represents the base of the natural logarithm, and j represents the imaginary unit.

[0016] Preferably, the step of using a Gaussian low-pass filter to perform low-pass filtering on the spectrogram, extracting the structural and detailed information in the source image based on a guided filter, and generating an initial weight map includes the following steps:

[0017] S21. Use a Gaussian low-pass filter to perform low-pass filtering on the spectrogram, obtain the filtered spectrogram, and inverse transform the filtered spectrogram to the spatial domain to obtain the filtered image;

[0018] S22. Calculate the absolute value of the difference between the source image and the filtered image to obtain a high-frequency information map, and generate an initial metric map according to the high-frequency information map;

[0019] S23. Use the source image and the high-frequency information map as the input images of the guided filter, extract the structural and detailed information in the source image, and obtain the final metric map;

[0020] S24. Based on the final metric map, compare the final metric maps generated by multiple images respectively, and retain the image with the larger pixel value to obtain the initial weight map.

[0021] Preferably, the expression of the filtered spectrogram is as follows:

[0022] Q i (u, v) = F i (u, v) * H(u, v)

[0023] The expression of the inverse transformation is as follows:

[0024]

[0025] The expression of the initial metric map is as follows:

[0026] IAM i (x, y) = |f i (x, y) - q i (x, y)|

[0027] The expression of the final metric map is as follows:

[0028] FAM i (x, y) = GF(f i (x, y), IAM i (x, y))

[0029] The expression of the initial weight map is as follows:

[0030]

[0031] In the formula, Q i (u, v) represents the frequency domain map of the i-th image after filtering, H(u, v) represents the standard form of the Gaussian low-pass filter, q i (x, y) represents the image that is inverse-transformed into the spatial domain after low-pass filtering, IAM i (x, y) represents the input image, f i (x, y) represents the original image as the guidance image, FAM i (x, y) represents the final metric map generated by guided filtering, GF represents the guided filtering formula, IM(x, y) represents the initial weight map, and FAM1(x, y), FAM2(x, y) respectively represent the two final metric maps generated by the original image.

[0032] Preferably, for the regional filtering method, the unrecognized pixels in the initial weight map are removed, and the morphological operation dilation method is used to combine with the initial weight map to locate the boundary region; fusing multiple source images according to the processed initial weight map includes the following steps:

[0033] S31. According to the preset image threshold (R * M * N), the spots in the initial weight map are filtered by adjusting the filtering ratio, where R represents the filtering ratio, and M, N respectively represent the width and height of the image;

[0034] S32. Use the morphological operation dilation method to dilate the initial weight map after removing the spots, subtract the dilated image from the initial weight map to obtain the boundary region, and fuse multiple source images according to the dilated initial weight map.

[0035] Preferably, the expression of the boundary region is:

[0036] BA(x, y) = (IM(x, y) ⊕ B - IM(x, y))

[0037]

[0038] BAM(x, y) = (BA(x, y) * 0.5 + IM(x, y)) * w - IM(x, y)

[0039]

[0040] In the formula, BA(x, y) represents the boundary region, B represents that the dilation direction is to spread around, ⊕ represents the dilation operator, IM(x, y) represents the initial weight map, BAM(x, y) represents the generated boundary region, and w represents the convolution kernel.

[0041] Preferably, the steps of using the global motion multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image and determining the weight of the boundary region according to the direction and displacement information between adjacent pixels are as follows:

[0042] S41. Construct a multi-level Gaussian pyramid based on the fused image and the initial weight map, use the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level, and obtain the direction vector and displacement amplitude of each pixel relative to its neighboring pixels;

[0043] S42. Use the optical flow estimation result of the low-resolution layer as the initial estimate value, transfer it layer by layer to the high-resolution layer, and gradually refine the direction and displacement information; classify the boundary region pixels in the fused image based on the direction consistency and displacement amplitude, and determine the weight of the boundary region according to the classification result.

[0044] Preferably, the steps of constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, using the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels are as follows:

[0045] S411. Construct a multi-level Gaussian pyramid from the fused image and the initial weight map; establish an optical flow equation for the direction and displacement information of each pixel, and use the least squares method to solve the optical flow equation in the neighborhood of each pixel to obtain the displacement vector of the pixel;

[0046] S412. Based on the displacement vector of each pixel, calculate the direction vector and displacement amplitude of each pixel relative to its neighboring pixels;

[0047] Among them, the expression of the optical flow equation is:

[0048] I x s x +I y s y +I t =0

[0049] The calculation formula for the direction vector is:

[0050]

[0051] The calculation formula for the displacement amplitude is:

[0052]

[0053] In the formula, I x 、I y respectively represent the luminance change rates of the image in the x and y directions, I t represents the luminance change rate with respect to time t, s x 、s y respectively represent the velocities in the x and y directions, θ represents the direction vector, and mag represents the displacement amplitude.

[0054] Preferably, using the optical flow estimation result of the low-resolution layer as the initial estimate value, gradually transmitting it layer by layer to the high-resolution layer, and gradually refining the direction and displacement information; classifying the pixels in the boundary region of the fused image based on the direction consistency and displacement amplitude, and determining the weight of the boundary region according to the classification result includes the following steps:

[0055] S421. Use the optical flow estimation result of the low-resolution layer as the initial estimate value and transmit it layer by layer to the high-resolution layer, and use the optical flow result of the previous layer as the initial estimate of the current layer at each layer, and further optimize the optical flow estimation on the current layer; on the high-resolution layer, use the pixel information in the neighborhood to further refine the calculation of the direction vector and displacement amplitude to ensure obtaining the final high-precision optical flow information at the original resolution layer;

[0056] S422. Classify the pixels in the boundary region of the fused image according to the comparison results of the direction consistency and the displacement amplitude with the preset direction consistency threshold and displacement amplitude threshold, and determine the weight of the boundary region according to the classification results;

[0057] Among them, when the direction consistency metric value of the pixels in the boundary region is greater than or equal to the preset direction consistency threshold and the displacement amplitude value is less than the preset displacement amplitude threshold, the pixel is classified as a first-class pixel;

[0058] When the direction consistency metric value of the pixels in the boundary region is greater than or equal to the preset direction consistency threshold and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified as a second-class pixel;

[0059] When the direction consistency metric value of the pixels in the boundary region is less than the preset direction consistency threshold and the displacement amplitude value is less than the preset displacement amplitude threshold, the pixel is classified as a third-class pixel;

[0060] When the direction consistency metric value of the pixels in the boundary region is less than the preset direction consistency threshold and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified as a fourth-class pixel;

[0061] And the weights of the first-class pixels, second-class pixels, third-class pixels, and fourth-class pixels gradually decrease.

[0062] Preferably, the expression of the final high-resolution image is:

[0063]

[0064] In the formula, IF i (x, y) represents the final high-resolution image, f1(x, y) and f2(x, y) respectively represent two low-resolution images, and α and β respectively represent the weights of the boundary regions of the two images f1(x, y) and f2(x, y).

[0065] The technical solution provided by the present invention may include the following beneficial effects:

[0066] The present invention first converts the image into the frequency domain and constructs a low-pass filter in the frequency domain to simulate the low-pass filtering effect of the microscope, and better extracts high-frequency information; secondly, uses the global motion multi-resolution optical flow estimation algorithm to determine the weight of the boundary region, and generates the final high-resolution image by fusing the weights of the boundary region, effectively avoiding the block effect and jagged phenomenon of the high-resolution image in the traditional method. Generally speaking, the present invention generates detailed features of the low-resolution images under different focal lengths to generate a high-resolution image that is clear in all regions, thereby improving the accuracy and calculation efficiency of multi-image super-resolution.

[0067] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present invention and, together with the specification, are used to explain the principles of the present invention.

[0069] Figure 1 is a flowchart of a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0070] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0071] The present invention provides a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter.

[0072] The present invention will be further described below in conjunction with the drawings and specific embodiments. As Figure 1 shown, according to an embodiment of the present invention, there is provided a multi-image super-resolution imaging method guided by a frequency-domain low-pass filter, including the following steps:

[0073] S1. Based on the two-dimensional discrete Fourier transform technology, convert the low-resolution source image from the spatial domain to the frequency domain to obtain a frequency spectrum diagram;

[0074] In this embodiment, by simulating a low-pass filter, the information of the image is blurred again, and the lost information is high-frequency information. To obtain this part of high-frequency information and better process the image without excessive influence on local details.

[0075] Among them, the conversion formula of the frequency spectrum diagram is:

[0076]

[0077] In the formula, F i (u, v) represents the frequency-domain representation of the i-th image, i = 1 and 2 represent the input images 1 and 2, u and v respectively represent the frequency components in the horizontal and vertical directions in the frequency domain, x and y respectively represent the horizontal and vertical coordinates of the spatial-domain image, x = {0, 1, 2,..., M - 1}, y = {0, 1, 2,..., N - 1}, M and N respectively represent the length and width of the picture, fi (x, y) represents the spatial domain representation of the i-th image, e represents the base of the natural logarithm, and j represents the imaginary unit.

[0078] S2. Perform low-pass filtering on the spectrogram using a Gaussian low-pass filter, extract the structural and detailed information in the source image based on a guided filter, and generate an initial weight map.

[0079] Among them, the step of performing low-pass filtering on the spectrogram using a Gaussian low-pass filter, extracting the structural and detailed information in the source image based on a guided filter, and generating an initial weight map includes the following steps:

[0080] S21. Perform low-pass filtering on the spectrogram using a Gaussian low-pass filter (that is, multiply the spectrogram of the original image by the Gaussian low-pass filter) to obtain the filtered spectrogram, and inverse-transform the filtered spectrogram to the spatial domain to obtain the filtered image.

[0081] The expression of the filtered spectrogram is:

[0082] Q i (u, v) = F i (u, v) * H(u, v)

[0083] The expression of the inverse transformation is:

[0084]

[0085] In the formula, Q i (u, v) represents the frequency domain diagram of the i-th image after filtering, H(u, v) represents the standard form of the Gaussian low-pass filter, and q i (x, y) represents the image that is inverse-transformed to the spatial domain after low-pass filtering.

[0086] S22. After frequency domain filtering of the image, calculate the absolute value of the difference between the source image and the filtered image to obtain a high-frequency information map, and generate an initial metric map based on the high-frequency information map.

[0087] The expression of the initial metric map is:

[0088] IAM i (x, y) = |f i (x, y) - q i (x, y)|

[0089] In the formula, IAM i (x, y) represents the input image, f i (x, y) represents the original image as a guidance image.

[0090] S23. Use the source image and the high-frequency information image as the input images of the guided filter to extract the structure and detail information in the source image, and obtain the final metric image;

[0091] Specifically, in order to make reasonable use of the structure and detail information attached to the original image, the guided filter is applied in this embodiment. Its most basic assumption is that the output image is a local linear transformation of the guidance image, that is, as shown in the following formula:

[0092] O i = a k G i + b k , i ∈ w k

[0093] In the formula, G is the guidance image, O is the output image. Within a filtering window w k , the output image and the guidance image are linearly related, and a k , b k represent constant parameters;

[0094] For a given input image I, in order to make the output image O keep the same change as the input image locally, the following optimization objective is used. A regularization parameter ε is introduced to avoid a k from being too large;

[0095]

[0096] In the formula, E(a k , b k ) represents the loss function within the filtering window;

[0097] Take the partial derivatives of the parameters a k , b k as shown in the following formula:

[0098]

[0099] In the formula, mean represents the mean value calculated from all pixels within the window range;

[0100] As shown in the following formula, according to the formulas of expectation, variance and covariance, a k is obtained:

[0101] Var(X) = E[X] 2 - E[Y] 2 ,

[0102] Cov(X,Y) = E[XY] - E[X]E[Y]

[0103]

[0104] Wherein, Var(X) represents variance, E[X] represents expectation, and Cov(X,Y) represents covariance. represents the variance within the guiding image window;

[0105] From the above formula, the following conclusion can be calculated: when the input image I and the guiding image G are the same image, for flat regions, the variance of the image is very small, far less than ε, k a is close to 0, and the output image O is close to the mean filtering of the input image I. For edge regions, the variance is very large k far greater than ε, k a is close to 1, k b is close to 0, which is equivalent to the input image I remaining unchanged, achieving the effect of preserving edges.

[0106] In this embodiment, using the guiding filter, the two input images of the guiding filter are respectively the original image and the high-frequency information image obtained in the above process. By utilizing the structure and detail information of the original image, the final metric map is obtained as shown in the following formula:

[0107] FAM i (x,y) = GF(f i (x,y), IAM i (x,y))

[0108] Wherein, FAM i (x,y) represents, GF represents the guiding filter formula;

[0109] S24. Based on the final metric map, compare the final metric maps generated by multiple images respectively, and retain the image with larger pixel values to obtain the initial weight map.

[0110] The expression of the initial weight map is:

[0111]

[0112] Wherein, IM(x,y) represents the initial weight map, and FAM1(x,y), FAM2(x,y) respectively represent the two final metric maps generated by the original image.

[0113] S3. Based on the regional filtering method, remove the unrecognized pixels in the initial weight map, and use the morphological operation dilation method to combine with the initial weight map to locate the boundary region; fuse multiple source images according to the processed initial weight map;

[0114] Specifically, in the initial weight map generated by the above method, there are still some spots that are not well distinguished. To solve this problem, these spots need to be filtered out for subsequent fusion. In this embodiment, a regional filtering method is adopted to handle this problem. In addition, artifacts may appear on the boundary between the focused area and the defocused area of the obtained fused image. To solve this problem, this embodiment defines a boundary area and adopts a weight fusion method in this part of the area to obtain a better super-resolution effect.

[0115] Among them, the regional filtering method removes the unrecognized pixels in the initial weight map and uses the morphological operation dilation method to combine with the initial weight map to locate the boundary area; fusing multiple source images according to the processed initial weight map includes the following steps:

[0116] S31. According to a preset image threshold (R*M*N), filter out the spots in the initial weight map by adjusting the size of the filtering ratio, where R represents the filtering ratio, and M and N respectively represent the width and height of the image;

[0117] Specifically, regional filtering in image fusion is an image processing technique used to combine the information of multiple images into one image while retaining specific regions or features in each image. This can make the synthesized image retain the advantages of each input image while avoiding unnecessary confusion or distortion.

[0118] The method adopted in this embodiment is to filter out these unrecognized pixel points at a certain image size ratio. Specifically, by setting a threshold, the size of the threshold is set to (R*H*W), where R is the filtering ratio, and H and W are the width and height of the image. The spots are filtered out by adjusting the size of R.

[0119] S32. Use the morphological operation dilation method to perform dilation processing on the initial weight map after filtering out the spots, and subtract the initial weight map from the dilated image to obtain the boundary area. Fuse multiple source images according to the dilated initial weight map.

[0120] Specifically, for the definition method of the boundary area, by means of morphological operation dilation, the above-obtained initial weight map is dilated, and the dilated image is subtracted from the initial weight map to obtain the boundary area BA. The size of the boundary area is determined by the number of dilation times, as shown in the following formula:

[0121] BA(x,y) = (IM(x,y) ⊕ B - IM(x,y))

[0122]

[0123] For the boundary region obtained by the above method, when approaching f1(x, y), the weight of f1(x, y) is higher; conversely, the weight of f2(x, y) is higher, as shown in the following formula:

[0124] BAM(x,y)=(BA(x,y)*0.5+IM(x,y))*w-IM(x,y)

[0125]

[0126] In the formula, BA(x, y) represents the boundary region, The operator representing image dilation, B indicates that the dilation direction is spreading in all directions, ⊕ represents the dilation operator, IM(x, y) represents the initial weight map, BAM(x, y) represents the generated boundary region, and w represents the convolution kernel.

[0127] S4. Use the global motion multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image, and determine the weight of the boundary region according to the direction and displacement information between adjacent pixels;

[0128] Among them, the step of using the global motion multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image, and determining the weight of the boundary region according to the direction and displacement information between adjacent pixels includes the following steps:

[0129] S41. Construct a multi-level Gaussian pyramid based on the fused image and the initial weight map, and use the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level to obtain the direction vector and displacement amplitude of each pixel relative to its neighboring pixels;

[0130] Specifically, the step of constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, and using the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level to obtain the direction vector and displacement amplitude of each pixel relative to its neighboring pixels includes the following steps:

[0131] S411. Construct a multi-level Gaussian pyramid from the fused image and the initial weight map; establish an optical flow equation for the direction and displacement information of each pixel, and use the least squares method to solve the optical flow equation in the neighborhood of each pixel to obtain the displacement vector of the pixel;

[0132] S412. Based on the displacement vector of each pixel, calculate the direction vector and displacement amplitude of each pixel relative to its neighboring pixels;

[0133] Among them, the expression of the optical flow equation is:

[0134] I x s x +Iy s y +I t =0

[0135] The calculation formula for the direction vector is as follows:

[0136]

[0137] The calculation formula for the displacement amplitude is as follows:

[0138]

[0139] In the formula, I x 、I y respectively represent the luminance change rates of the image in the x and y directions, I t represents the luminance change rate at time t, s x 、s y respectively represent the velocities in the x and y directions, θ represents the direction vector, and mag represents the displacement amplitude.

[0140] S42. Use the optical flow estimation result of the low-resolution layer as the initial estimate value, and layer by layer transfer it to the high-resolution layer to gradually refine the direction and displacement information; classify the pixels in the boundary region of the fused image based on the direction consistency and displacement amplitude, and determine the weight of the boundary region according to the classification result.

[0141] Specifically, the step of using the optical flow estimation result of the low-resolution layer as the initial estimate value, layer by layer transfer it to the high-resolution layer to gradually refine the direction and displacement information; classify the pixels in the boundary region of the fused image based on the direction consistency and displacement amplitude, and determine the weight of the boundary region according to the classification result includes the following steps:

[0142] S421. Use the optical flow estimation result of the low-resolution layer as the initial estimate value and layer by layer transfer it to the high-resolution layer, and use the optical flow result of the previous layer as the initial estimate of the current layer at each layer, and further optimize the optical flow estimation on the current layer; on the high-resolution layer, use the pixel information in the neighborhood to further refine the calculation of the direction vector and displacement amplitude to ensure obtaining the final high-precision optical flow information at the original resolution layer;

[0143] S422. Classify the pixels in the boundary region of the fused image according to the comparison results of the direction consistency and displacement amplitude with the preset direction consistency threshold and displacement amplitude threshold, and determine the weight of the boundary region according to the classification result;

[0144] Among them, when the direction consistency measurement value of the pixels in the boundary region is greater than or equal to the preset direction consistency threshold and the displacement amplitude value is less than the preset displacement amplitude threshold, classify the pixel as the first type of pixel;

[0145] When the direction consistency metric value of a pixel in the boundary region is greater than or equal to a preset direction consistency threshold and the displacement amplitude value is greater than or equal to a preset displacement amplitude threshold, the pixel is classified as a second type of pixel;

[0146] When the direction consistency metric value of a pixel in the boundary region is less than the preset direction consistency threshold and the displacement amplitude value is less than the preset displacement amplitude threshold, the pixel is classified as a third type of pixel;

[0147] When the direction consistency metric value of a pixel in the boundary region is less than the preset direction consistency threshold and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified as a fourth type of pixel;

[0148] And the weights of the first type of pixel, the second type of pixel, the third type of pixel, and the fourth type of pixel gradually decrease.

[0149] S5. Combine the initial weight map and the weights of the boundary region to perform weighted fusion on multiple low-resolution images to generate a final high-resolution image.

[0150] Specifically, the expression of the final high-resolution image is:

[0151]

[0152] In the formula, IF i (x, y) represents the final high-resolution image, f1(x, y) and f2(x, y) respectively represent two low-resolution images, and α and β respectively represent the weights of the boundary regions of the two images f1(x, y) and f2(x, y).

[0153] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification. Those of ordinary skill in the art can understand that all or part of the steps in implementing the above-described method can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps described in the above method. The storage medium, such as: ROM / RAM, magnetic disk, optical disc, etc.

[0154] The above-described embodiments only represent several implementation manners of the present invention. The description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can still be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent should be subject to the appended claims.

Claims

1. A multi-image super-resolution imaging method based on frequency domain low-pass filter guidance, characterized in that: The following steps are involved: S1. Based on the two-dimensional discrete Fourier transform technology, the low-resolution source image is converted from the spatial domain to the frequency domain to obtain the spectrum diagram; S2, using a Gaussian low-pass filter to perform low-pass filtering on the spectrum map, extracting the structure and detail information in the source image based on the guided filter, and generating an initial weight map; S3, based on the regional filtering method, remove the unrecognized pixels in the initial weight map, and use the morphological operation expansion method combined with the initial weight map to locate the boundary area; fuse multiple source images according to the processed initial weight map; S4. Using a global motion-based multi-resolution optical flow estimation algorithm, the direction and displacement information between adjacent pixels in the fused image is captured, and the weight of the boundary area is determined according to the direction and displacement information between adjacent pixels; specifically, the following steps are included: S41, constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, using the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels; S42, using the optical flow estimation result of the low-resolution layer as the initial estimation value, passing it to the high-resolution layer layer by layer, and gradually refining the direction and displacement information; based on the direction consistency and displacement amplitude, classifying the boundary area pixels in the fused image, and determining the weight of the boundary area according to the classification result; S5. Combining the initial weight map and the weights of the boundary area, weighted fusion is performed on multiple images to generate a final high-resolution image.

2. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 1, characterized in that: The conversion formula of the spectrum graph is: In the formula, F i (u,v) represents the frequency domain representation of the i-th image, u and v represent the horizontal and vertical frequency components in the frequency domain, x and y represent the horizontal and vertical coordinates of the spatial domain image, M and N represent the length and width of the image, respectively. i (x, y) represents the spatial domain representation of the i-th image, e represents the base of the natural logarithm, and j represents the imaginary unit.

3. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 2 is characterized in that: The method of using a Gaussian low-pass filter to perform low-pass filtering on the spectrum map, extracting structure and detail information in the source image based on the guided filter, and generating an initial weight map includes the following steps: S21, performing low-pass filtering on the spectrum graph using a Gaussian low-pass filter to obtain a filtered spectrum graph, and inversely converting the filtered spectrum graph into a spatial domain to obtain a filtered image; S22, calculating the absolute value of the difference between the source image and the filtered image to obtain a high-frequency information graph, and generating an initial metric graph according to the high-frequency information graph; S23, using the source image and the high-frequency information map as input images of the guided filter, extracting the structure and detail information in the source image, and obtaining the final metric map; S24. Based on the final metric map, compare the final metric maps generated for the multiple images respectively, and retain the images with large pixel values ​​to obtain an initial weight map.

4. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 3 is characterized in that: The expression of the filtered spectrum is: Q i (u,v)=F i (u,v)*H(u,v) The inverse conversion expression is: The expression of the initial metric graph is: IAM i (x,y)=|f i (x,y)-q i (x,y)| The expression of the final metric graph is: FAM i (x,y)=GF(f i (x,y),IAM i (x,y)) The expression of the initial weight map is: In the formula, Q i (u, v) represents the frequency domain image after filtering of the i-th image, H(u, v) represents the standard form of the Gaussian low-pass filter, q i (x, y) represents the image inversely transformed into the spatial domain after low-pass filtering, IAM i (x,y) represents the input image, f i (x,y) represents the original image as the guide image, FAM i (x, y) represents the final metric map generated by guided filtering, GF represents the guided filtering formula, IM(x, y) represents the initial weight map, FAM1(x, y) and FAM2(x, y) represent the two final metric maps generated by the original image respectively.

5. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 1, characterized in that: The method based on regional filtering removes unrecognized pixels in the initial weight map, and locates the boundary area by combining the morphological operation expansion method with the initial weight map; fusing multiple source images according to the processed initial weight map includes the following steps: S31, according to a preset image threshold (R*M*N), the spots in the initial weight map are filtered out by adjusting the size of the filtering ratio, where R represents the filtering ratio, and M and N represent the width and height of the image respectively; S32, using a morphological operation dilation method to dilate the initial weight map after filtering out spots, and subtracting the initial weight map from the dilated image to obtain a boundary area, and fusing multiple source images according to the dilated initial weight map.

6. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 5, characterized in that: The expression for the boundary region is: BAM(x,y)=(BA(x,y)×0.5+IM(x,y))*w-IM(x,y) In the formula, BA(x,y) represents the boundary area, B represents the direction of expansion, represents the dilation operator, IM(x,y) represents the initial weight map, BAM(x,y) represents the generated boundary area, and w represents the convolution kernel.

7. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 1, characterized in that: The method of constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, analyzing the direction and displacement information of adjacent pixels at each resolution level using the optical flow estimation algorithm, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels includes the following steps: S411, constructing a multi-level Gaussian pyramid from the fused image and the initial weight map; establishing an optical flow equation for the direction and displacement information of each pixel, and solving the optical flow equation in the neighborhood of each pixel using the least squares method to obtain a displacement vector of the pixel; S412, based on the displacement vector of each pixel, calculating the direction vector and displacement amplitude of each pixel relative to the neighboring pixels; Among them, the expression of the optical flow equation is: I x s x +I y s y +I t =0 The direction vector is calculated as: The calculation formula of displacement amplitude is: In the formula, I x ,I y Respectively represent the brightness change rate of the image in the x and y directions, I t Indicates the rate of change of brightness at time t, s x 、s y Represent the speed in the x and y directions respectively, θ represents the direction vector, and mag represents the displacement amplitude.

8. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 1, characterized in that: The optical flow estimation result of the low-resolution layer is used as the initial estimation value, and is transmitted to the high-resolution layer layer by layer, and the direction and displacement information is gradually refined; based on the direction consistency and displacement amplitude, the boundary area pixels in the fused image are classified, and the weight of the boundary area is determined according to the classification result, which includes the following steps: S421, passing the optical flow estimation result of the low-resolution layer as the initial estimation value to the high-resolution layer layer by layer, and taking the optical flow result of the previous layer as the initial estimation of the current layer at each layer, and further optimizing the optical flow estimation at the current layer; at the high-resolution layer, using the pixel information in the neighborhood to further refine the calculation of the direction vector and the displacement amplitude, to ensure that the final high-precision optical flow information is obtained at the original resolution layer; S422, classifying the boundary area pixels in the fused image according to the comparison results of the direction consistency and the displacement amplitude with the preset direction consistency threshold and the displacement amplitude threshold, and determining the weight of the boundary area according to the classification result; Wherein, when the direction consistency metric value of a pixel in the boundary area is greater than or equal to a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a first type of pixel; When the direction consistency metric value of the pixel in the boundary area is greater than or equal to the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the second type of pixel; When the direction consistency metric value of a pixel in the boundary area is less than a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a third type of pixel; When the direction consistency metric value of the pixel in the boundary area is less than the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the fourth category of pixels; And the weights of the first type of pixels, the second type of pixels, the third type of pixels, and the fourth type of pixels gradually decrease.

9. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 6, characterized in that: The final high-resolution image is expressed as: In the formula, IF i (x, y) represents the final high-resolution image, f1(x, y) and f2(x, y) represent two low-resolution images respectively, and α and β represent the weights of the boundary areas of the two images f1(x, y) and f2(x, y) respectively.

Images