Multi-image super-division imaging method based on guidance of frequency domain low-pass filter

Abstract

The invention belongs to the technical field of super-division imaging, and discloses a multi-image super-division imaging method based on the guidance of a frequency domain low-pass filter, and the method comprises the steps: converting a low-resolution source image from a spatial domain to a frequency domain, and obtaining a spectrogram; performing low-pass filtering processing on the spectrogram, and extracting structure and detail information in the source image to generate an initial weight map; positioning a boundary area, and fusing the plurality of source images; estimating the optical flow based on the global motion multi-resolution, capturing direction and displacement information between adjacent pixels in the fused image, and determining the weight of a boundary region according to the direction and displacement information between the adjacent pixels; and finally, performing weighted fusion on the plurality of source images in combination with the initial weight map and the weight of the boundary region to generate a high-resolution image. According to the invention, an efficient super-resolution imaging method is realized, and the generated high-resolution image provides richer and more accurate visual detail information.

Description

Technical Field

[0001] The present invention relates to the technical field of super-resolution imaging, and in particular to a multi-image super-resolution imaging method based on frequency domain low-pass filter guidance. Background Art

[0002] Super-resolution imaging technology has wide application value in many industrial fields. With the continuous advancement of sensor technology and the popularization of image acquisition equipment, multi-source image data can be easily obtained. However, since images obtained from different sensors or under different conditions often have different characteristics and limitations, such as differences in resolution, brightness, contrast, color distribution, etc., it is difficult to achieve high-quality and high-resolution imaging.

[0003] The existing multi-image super-resolution imaging method mainly realizes multi-image fusion super-resolution based on the spatial domain. It uses certain spatial features of the image, such as gradient, spatial frequency, etc. to optimize the fusion and finally generate a high-resolution image. However, this method is prone to block effects, high complexity and low computational efficiency. In order to overcome the defects and shortcomings of the existing methods, the present invention constructs a low-pass filter from the frequency domain simulation to obtain the high-frequency information lost after the high-resolution image passes through the imaging system. The data structure and detail information of the low-quality image are captured by designing a guided filter, and then the high-resolution image is reconstructed using appropriate fusion weights.

[0004] Therefore, the present invention provides a multi-image super-resolution imaging method based on frequency domain low-pass filter guidance. This method effectively extracts high-frequency information of low-resolution images, overcomes the problems of large loss of structural detail information, easy occurrence of block effects and jagged edges in traditional methods, and improves the multi-image super-resolution imaging performance and computational efficiency. Summary of the invention

[0005] The present invention provides a multi-image super-resolution imaging method based on frequency domain low-pass filter guidance to solve the above-mentioned technical problems in the prior art.

[0006] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be a general review, nor is it intended to identify key / important components or to delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0007] The present invention provides a multi-image super-resolution imaging method based on frequency domain low-pass filter guidance, comprising the following steps:

[0008] S1. Based on the two-dimensional discrete Fourier transform technology, the low-resolution source image is converted from the spatial domain to the frequency domain to obtain the spectrum diagram;

[0009] S2, using a Gaussian low-pass filter to perform low-pass filtering on the spectrum graph, extracting the structure and detail information in the image based on the guided filter, and generating an initial weight map;

[0010] S3, based on the regional filtering method, remove the unrecognized pixels in the initial weight map, and use the morphological operation expansion method combined with the initial weight map to locate the boundary area; fuse multiple images according to the processed initial weight map;

[0011] S4, using a global motion-based multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image, and determining the weight of the boundary area according to the direction and displacement information between adjacent pixels;

[0012] S5. Combining the initial weight map and the weights of the boundary area, weighted fusion is performed on multiple images to generate a final high-resolution image.

[0013] Preferably, the conversion formula of the spectrum graph is:

[0014]

[0015] In the formula, F i (u,v) represents the frequency domain representation of the i-th image, u and v represent the horizontal and vertical frequency components in the frequency domain, x and y represent the horizontal and vertical coordinates of the spatial domain image, M and N represent the length and width of the image, respectively. i (x, y) represents the spatial domain representation of the i-th image, e represents the base of the natural logarithm, and j represents the imaginary unit.

[0016] Preferably, the method of performing low-pass filtering on the spectrum map using a Gaussian low-pass filter, extracting structure and detail information in the source image based on a guided filter, and generating an initial weight map comprises the following steps:

[0017] S21, performing low-pass filtering on the spectrum graph using a Gaussian low-pass filter to obtain a filtered spectrum graph, and inversely converting the filtered spectrum graph into a spatial domain to obtain a filtered image;

[0018] S22, calculating the absolute value of the difference between the source image and the filtered image to obtain a high-frequency information graph, and generating an initial metric graph according to the high-frequency information graph;

[0019] S23, using the source image and the high-frequency information map as input images of the guided filter, extracting the structure and detail information in the source image, and obtaining the final metric map;

[0020] S24. Based on the final metric map, compare the final metric maps generated for the multiple images respectively, and retain the images with large pixel values ​​to obtain an initial weight map.

[0021] Preferably, the expression of the filtered spectrum graph is:

[0022] Q i (u,v)=F i (u,v)*H(u,v)

[0023] The inverse conversion expression is:

[0024]

[0025] The expression of the initial metric graph is:

[0026] IAM i (x,y)=|f i (x,y)-q i (x,y)|

[0027] The expression of the final metric graph is:

[0028] FAM i (x,y)=GF(f i (x,y),IAM i (x,y)

[0029] The expression of the initial weight map is:

[0030]

[0031] In the formula, Q i (u, v) represents the frequency domain image after filtering of the i-th image, H(u, v) represents the standard form of the Gaussian low-pass filter, q i (x, y) represents the image inversely transformed into the spatial domain after low-pass filtering, IAM i (x,y) represents the input image, f i (x,y) represents the original image as the guide image, FAM i (x, y) represents the final metric map generated by guided filtering, GF represents the guided filtering formula, IM(x, y) represents the initial weight map, FAM1(x, y) and FAM2(x, y) represent the two final metric maps generated by the original image respectively.

[0032] Preferably, the method based on regional filtering removes unrecognized pixels in the initial weight map, and locates the boundary area by using the morphological operation expansion method in combination with the initial weight map; fusing multiple source images according to the processed initial weight map comprises the following steps:

[0033] S31, according to a preset image threshold (R*M*N), the spots in the initial weight map are filtered out by adjusting the size of the filtering ratio, where R represents the filtering ratio, and M and N represent the width and height of the image respectively;

[0034] S32, using a morphological operation dilation method to dilate the initial weight map after filtering out spots, and subtracting the initial weight map from the dilated image to obtain a boundary area, and fusing multiple source images according to the dilated initial weight map.

[0035] Preferably, the boundary region is expressed as:

[0036] BA(x,y)=(IM(x,y)⊕B-IM(x,y))

[0037]

[0038] BAM(x,y)=(BA(x,y)*0.5+IM(x,y))*w-IM(x,y)

[0039]

[0040] In the formula, BA(x,y) represents the boundary area, B represents the direction of expansion is to diffuse to the surroundings, ⊕ represents the expansion operator, IM(x,y) represents the initial weight map, BAM(x,y) represents the generated boundary area, and w represents the convolution kernel.

[0041] Preferably, the method of capturing the direction and displacement information between adjacent pixels in the fused image by using a global motion-based multi-resolution optical flow estimation algorithm, and determining the weight of the boundary area according to the direction and displacement information between adjacent pixels comprises the following steps:

[0042] S41, constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, using the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels;

[0043] S42. The optical flow estimation result of the low-resolution layer is used as the initial estimation value, and is passed to the high-resolution layer layer by layer to gradually refine the direction and displacement information; based on the direction consistency and displacement amplitude, the boundary area pixels in the fused image are classified, and the weight of the boundary area is determined according to the classification result.

[0044] Preferably, the method of constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, analyzing the direction and displacement information of adjacent pixels at each resolution level using an optical flow estimation algorithm, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels comprises the following steps:

[0045] S411, constructing a multi-level Gaussian pyramid from the fused image and the initial weight map; establishing an optical flow equation for the direction and displacement information of each pixel, and solving the optical flow equation in the neighborhood of each pixel using the least squares method to obtain a displacement vector of the pixel;

[0046] S412, based on the displacement vector of each pixel, calculating the direction vector and displacement amplitude of each pixel relative to the neighboring pixels;

[0047] Among them, the expression of the optical flow equation is:

[0048] I x s x +I y s y +I t =0

[0049] The direction vector is calculated as:

[0050]

[0051] The calculation formula of displacement amplitude is:

[0052]

[0053] In the formula, I x ,I y Respectively represent the brightness change rate of the image in the x and y directions, I t Indicates the rate of change of brightness at time t, s x 、s y Represent the speed in the x and y directions respectively, θ represents the direction vector, and mag represents the displacement amplitude.

[0054] Preferably, the optical flow estimation result of the low-resolution layer is used as the initial estimation value, and is transferred to the high-resolution layer layer by layer to gradually refine the direction and displacement information; based on the direction consistency and displacement amplitude, the boundary area pixels in the fused image are classified, and the weight of the boundary area is determined according to the classification result, which includes the following steps:

[0055] S421, passing the optical flow estimation result of the low-resolution layer as the initial estimation value to the high-resolution layer layer by layer, and taking the optical flow result of the previous layer as the initial estimation of the current layer at each layer, and further optimizing the optical flow estimation at the current layer; at the high-resolution layer, using the pixel information in the neighborhood to further refine the calculation of the direction vector and the displacement amplitude, to ensure that the final high-precision optical flow information is obtained at the original resolution layer;

[0056] S422, classifying the boundary area pixels in the fused image according to the comparison results of the direction consistency and the displacement amplitude with the preset direction consistency threshold and the displacement amplitude threshold, and determining the weight of the boundary area according to the classification result;

[0057] Wherein, when the direction consistency metric value of a pixel in the boundary area is greater than or equal to a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a first type of pixel;

[0058] When the direction consistency metric value of the pixel in the boundary area is greater than or equal to the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the second type of pixel;

[0059] When the direction consistency metric value of a pixel in the boundary area is less than a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a third type of pixel;

[0060] When the direction consistency metric value of the pixel in the boundary area is less than the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the fourth category of pixels;

[0061] And the weights of the first type of pixels, the second type of pixels, the third type of pixels, and the fourth type of pixels gradually decrease.

[0062] Preferably, the final high-resolution image is expressed as:

[0063]

[0064] In the formula, IF i (x, y) represents the final high-resolution image, f1(x, y) and f2(x, y) represent two low-resolution images respectively, and α and β represent the weights of the boundary areas of the two images f1(x, y) and f2(x, y) respectively.

[0065] The technical solution provided by the present invention may include the following beneficial effects:

[0066] The present invention first converts the image to the frequency domain and constructs a low-pass filter in the frequency domain to simulate the low-pass filtering effect of a microscope, so as to better extract high-frequency information; secondly, the weight of the boundary area is determined by using a global motion multi-resolution optical flow estimation algorithm, and the final high-resolution image is generated by fusing the boundary area weights, which effectively avoids the block effect and jagged phenomenon of high-resolution images in traditional methods. In general, the present invention generates detailed features for low-resolution images under different focuses, and generates a high-resolution image that is clear in all areas, thereby improving the accuracy and computational efficiency of multi-image super-resolution.

[0067] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0069] Figure 1 The present invention is a flowchart of a multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to an exemplary embodiment. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The invention provides a multi-image super-resolution imaging method based on the guidance of a frequency domain low-pass filter.

[0072] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to one embodiment of the present invention, a multi-image super-resolution imaging method based on frequency domain low-pass filter guidance is provided, comprising the following steps:

[0073] S1. Based on the two-dimensional discrete Fourier transform technology, the low-resolution source image is converted from the spatial domain to the frequency domain to obtain the spectrum diagram;

[0074] In this embodiment, the image information is blurred again by simulating a low-pass filter, and the lost information is the high-frequency information. This part of high-frequency information is intended to be obtained to better process the image without excessively affecting local details.

[0075] Among them, the conversion formula of the spectrum diagram is:

[0076]

[0077] In the formula, F i (u,v) represents the frequency domain representation of the i-th image, i=1 and 2 represent the input image 1 and image 2, u and v represent the horizontal and vertical frequency components in the frequency domain, x and y represent the horizontal and vertical coordinates of the spatial domain image, x={0,1,2,…,M-1}, y={0,1,2,…,N-1}, M and N represent the length and width of the image, respectively, and fi (x, y) represents the spatial domain representation of the i-th image, e represents the base of the natural logarithm, and j represents the imaginary unit.

[0078] S2, using a Gaussian low-pass filter to perform low-pass filtering on the spectrum map, extracting the structure and detail information in the source image based on the guided filter, and generating an initial weight map;

[0079] The process of performing low-pass filtering on the spectrum graph using a Gaussian low-pass filter, extracting structure and detail information in the source image based on the guided filter, and generating an initial weight map comprises the following steps:

[0080] S21, performing low-pass filtering on the spectrum graph using a Gaussian low-pass filter (i.e., multiplying the spectrum graph of the original image by the Gaussian low-pass filter) to obtain a filtered spectrum graph, and inversely transforming the filtered spectrum graph into a spatial domain to obtain a filtered image;

[0081] The expression of the filtered spectrum is:

[0082] Q i (u,v)=F i (u,v)*H(u,v)

[0083] The inverse conversion expression is:

[0084]

[0085] In the formula, Q i (u, v) represents the frequency domain image after filtering of the i-th image, H(u, v) represents the standard form of the Gaussian low-pass filter, q i (x, y) represents the image inversely transformed into the spatial domain after low-pass filtering;

[0086] S22, after the image is filtered in the frequency domain, the absolute value of the difference between the source image and the filtered image is calculated to obtain a high-frequency information graph, and an initial metric graph is generated according to the high-frequency information graph;

[0087] The expression of the initial metric graph is:

[0088] IAM i (x,y)=|f i (x,y)-q i (x,y)|

[0089] In the formula, IAM i (x,y) represents the input image, f i (x,y) represents the original image as the guide image;

[0090] S23, using the source image and the high-frequency information map as input images of the guided filter, extracting the structure and detail information in the source image, and obtaining the final metric map;

[0091] Specifically, in order to reasonably utilize the structure and detail information attached to the original image, guided filtering is applied in this embodiment. Its most basic assumption is that the output image is a local linear transformation of the guided image, as shown in the following formula:

[0092] O i =a k G i +b k ,i∈w k

[0093] In the formula, G is the guide image, O is the output image, and in a filtering window of w k In the inner, the output image is linearly related to the guidance image, a k 、b k represents a constant parameter;

[0094] Given an input image I, in order to make the output image O maintain the same change as the input image locally, the optimization objective is as follows: a regularization parameter ε is introduced to avoid a k Too large;

[0095]

[0096] In the formula, E(a k ,b k ) represents the loss function within the filter window;

[0097] For parameter a k 、b k The partial derivative is shown below:

[0098]

[0099] In the formula, mean represents the mean value calculated from all pixels within the window range;

[0100] As shown in the following formula, according to the formula of expectation, variance and covariance, we get a k :

[0101] Var(X)=E[X] 2 -E[Y] 2 ,

[0102] Cov(X,Y)=E[XY]-E[X]E[Y]

[0103]

[0104] In the formula, Var(X) represents variance, E[X] represents expectation, Cov(X,Y) represents covariance, represents the variance within the guidance image window;

[0105] The following conclusions can be drawn from the above formula: When the input image I and the guide image G are the same image, for flat areas, the variance of the image is very small, much smaller than ε, a k Close to 0, the output image O is close to the mean filter of the input image I. For the edge area, the variance is large. k Much larger than ε, a k Close to 1, b k Close to 0, it is equivalent to the input image I remaining unchanged, which has the effect of maintaining the edge.

[0106] In this embodiment, a guided filter is used, and the two input images of the guided filter are the original image and the high-frequency information image obtained by the above process, respectively. The structure and detail information of the original image are used to obtain the final metric map, as shown in the following formula:

[0107] FAM i (x,y)=GF(f i (x,y),IAM i (x,y)

[0108] In the formula, FAM i (x,y) represents, GF represents the guided filtering formula,;

[0109] S24. Based on the final metric map, compare the final metric maps generated for the multiple images respectively, and retain the images with large pixel values ​​to obtain an initial weight map.

[0110] The expression of the initial weight map is:

[0111]

[0112] Where IM(x,y) represents the initial weight map, FAM1(x,y) and FAM2(x,y) represent the two final metric maps generated by the original image respectively.

[0113] S3, based on the regional filtering method, remove the unrecognized pixels in the initial weight map, and use the morphological operation expansion method combined with the initial weight map to locate the boundary area; fuse multiple source images according to the processed initial weight map;

[0114] Specifically, in the initial weight map generated by the above method, there are still some spots, which are not well distinguished. In order to solve this problem, these spots need to be filtered out for subsequent fusion. This embodiment adopts a regional filtering method to deal with this problem. In addition, the obtained fused image may have artifacts on the boundary between the focused area and the defocused area. In order to solve this problem, this embodiment defines the boundary area and adopts the weight fusion method in this area to obtain a better super-resolution effect.

[0115] The method based on regional filtering removes unrecognized pixels in the initial weight map, and locates the boundary area by combining the morphological operation expansion method with the initial weight map; and fusing multiple source images according to the processed initial weight map includes the following steps:

[0116] S31, according to a preset image threshold (R*M*N), the spots in the initial weight map are filtered out by adjusting the size of the filtering ratio, where R represents the filtering ratio, and M and N represent the width and height of the image respectively;

[0117] Specifically, regional filtering in image fusion is an image processing technique used to merge the information of multiple images into one image while retaining specific regions or features in each image. This allows the composite image to retain the advantages of each input image while avoiding unnecessary confusion or distortion.

[0118] The method adopted in this embodiment is to filter out these unrecognized pixels at a certain image size ratio, specifically by setting a threshold, and the size of the threshold is set to (R*H*W), where R is the filtering ratio, H and W are the width and height of the image, and the spot filtering is achieved by adjusting the size of R.

[0119] S32, using a morphological operation dilation method to dilate the initial weight map after filtering out spots, and subtracting the initial weight map from the dilated image to obtain a boundary area, and fusing multiple source images according to the dilated initial weight map.

[0120] Specifically, for the definition of the boundary area, the initial weight map obtained above is expanded by morphological operation expansion, and the initial weight map is subtracted from the expanded image to obtain the boundary area BA. The size of the boundary area is determined by the number of expansions, as shown in the following formula:

[0121] BA(x,y)=(IM(x,y)⊕B-IM(x,y))

[0122]

[0123] The boundary area obtained by the above method has a higher weight for f1(x,y) when it is close to f1(x,y), and vice versa, the weight for f2(x,y) is higher, as shown in the following formula:

[0124] BAM(x,y)=(BA(x,y)*0.5+IM(x,y))*w-IM(x,y)

[0125]

[0126] Where BA(x,y) represents the boundary area, Represents the operator of image expansion, B indicates that the direction of expansion is to diffuse in all directions, ⊕ indicates the expansion operator, IM(x,y) indicates the initial weight map, BAM(x,y) indicates the generated boundary area, and w indicates the convolution kernel.

[0127] S4, using a global motion-based multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image, and determining the weight of the boundary area according to the direction and displacement information between adjacent pixels;

[0128] The method of using a global motion-based multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image and determining the weight of the boundary area according to the direction and displacement information between adjacent pixels includes the following steps:

[0129] S41, constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, using the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels;

[0130] Specifically, the method of constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, analyzing the direction and displacement information of adjacent pixels at each resolution level using the optical flow estimation algorithm, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels includes the following steps:

[0131] S411, constructing a multi-level Gaussian pyramid from the fused image and the initial weight map; establishing an optical flow equation for the direction and displacement information of each pixel, and solving the optical flow equation in the neighborhood of each pixel using the least squares method to obtain a displacement vector of the pixel;

[0132] S412, based on the displacement vector of each pixel, calculating the direction vector and displacement amplitude of each pixel relative to the neighboring pixels;

[0133] Among them, the expression of the optical flow equation is:

[0134] I x s x +Iy s y +I t =0

[0135] The direction vector is calculated as:

[0136]

[0137] The calculation formula of displacement amplitude is:

[0138]

[0139] In the formula, I x ,I y Respectively represent the brightness change rate of the image in the x and y directions, I t Indicates the rate of change of brightness at time t, s x 、s y Represent the speed in the x and y directions respectively, θ represents the direction vector, and mag represents the displacement amplitude.

[0140] S42. The optical flow estimation result of the low-resolution layer is used as the initial estimation value, and is passed to the high-resolution layer layer by layer to gradually refine the direction and displacement information; based on the direction consistency and displacement amplitude, the boundary area pixels in the fused image are classified, and the weight of the boundary area is determined according to the classification result.

[0141] Specifically, the optical flow estimation result of the low-resolution layer is used as the initial estimation value, and is transferred to the high-resolution layer layer by layer to gradually refine the direction and displacement information; based on the direction consistency and displacement amplitude, the boundary area pixels in the fused image are classified, and the weight of the boundary area is determined according to the classification result, which includes the following steps:

[0142] S421, passing the optical flow estimation result of the low-resolution layer as the initial estimation value to the high-resolution layer layer by layer, and taking the optical flow result of the previous layer as the initial estimation of the current layer at each layer, and further optimizing the optical flow estimation at the current layer; at the high-resolution layer, using the pixel information in the neighborhood to further refine the calculation of the direction vector and the displacement amplitude, to ensure that the final high-precision optical flow information is obtained at the original resolution layer;

[0143] S422, classifying the boundary area pixels in the fused image according to the comparison results of the direction consistency and the displacement amplitude with the preset direction consistency threshold and the displacement amplitude threshold, and determining the weight of the boundary area according to the classification result;

[0144] Wherein, when the direction consistency metric value of a pixel in the boundary area is greater than or equal to a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a first type of pixel;

[0145] When the direction consistency metric value of the pixel in the boundary area is greater than or equal to the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the second type of pixel;

[0146] When the direction consistency metric value of a pixel in the boundary area is less than a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a third type of pixel;

[0147] When the direction consistency metric value of the pixel in the boundary area is less than the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the fourth category of pixels;

[0148] And the weights of the first type of pixels, the second type of pixels, the third type of pixels, and the fourth type of pixels gradually decrease.

[0149] S5. Combining the initial weight map and the weights of the boundary area, weighted fusion is performed on multiple low-resolution images to generate a final high-resolution image.

[0150] Specifically, the expression of the final high-resolution image is:

[0151]

[0152] In the formula, IF i (x, y) represents the final high-resolution image, f1(x, y) and f2(x, y) represent two low-resolution images respectively, and α and β represent the weights of the boundary areas of the two images f1(x, y) and f2(x, y) respectively.

[0153] The technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. A person of ordinary skill in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program. The program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps described in the above method, and the storage medium, such as ROM / RAM, a disk, an optical disk, etc.

[0154] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-image super-resolution imaging method based on frequency domain low-pass filter guidance, characterized in that: The following steps are involved: S1. Based on the two-dimensional discrete Fourier transform technology, the low-resolution source image is converted from the spatial domain to the frequency domain to obtain the spectrum diagram; S2, using a Gaussian low-pass filter to perform low-pass filtering on the spectrum map, extracting the structure and detail information in the source image based on the guided filter, and generating an initial weight map; S3, based on the regional filtering method, remove the unrecognized pixels in the initial weight map, and use the morphological operation expansion method combined with the initial weight map to locate the boundary area; fuse multiple source images according to the processed initial weight map; S4, using a global motion-based multi-resolution optical flow estimation algorithm to capture the direction and displacement information between adjacent pixels in the fused image, and determining the weight of the boundary area according to the direction and displacement information between adjacent pixels; S5. Combining the initial weight map and the weights of the boundary area, weighted fusion is performed on multiple images to generate a final high-resolution image.

2. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 1, characterized in that: The conversion formula of the spectrum graph is: In the formula, F i (u,v) represents the frequency domain representation of the i-th image, u and v represent the horizontal and vertical frequency components in the frequency domain, x and y represent the horizontal and vertical coordinates of the spatial domain image, M and N represent the length and width of the image, respectively. i (x, y) represents the spatial domain representation of the i-th image, e represents the base of the natural logarithm, and j represents the imaginary unit.

3. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 2 is characterized in that: The method of using a Gaussian low-pass filter to perform low-pass filtering on the spectrum map, extracting structure and detail information in the source image based on the guided filter, and generating an initial weight map includes the following steps: S21, performing low-pass filtering on the spectrum graph using a Gaussian low-pass filter to obtain a filtered spectrum graph, and inversely converting the filtered spectrum graph into a spatial domain to obtain a filtered image; S22, calculating the absolute value of the difference between the source image and the filtered image to obtain a high-frequency information graph, and generating an initial metric graph according to the high-frequency information graph; S23, using the source image and the high-frequency information map as input images of the guided filter, extracting the structure and detail information in the source image, and obtaining the final metric map; S24. Based on the final metric map, compare the final metric maps generated for the multiple images respectively, and retain the images with large pixel values ​​to obtain an initial weight map.

4. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 3 is characterized in that: The expression of the filtered spectrum is: Q i (u,v)=F i (u),v)*H(u,v) The inverse conversion expression is: The expression of the initial metric graph is: IAM i (x,y)=|f i (x,y)-q i (x,y)| The expression of the final metric graph is: FAM i (x,y)=GF(f i (x,y),IAM i (x,y)) The expression of the initial weight map is: In the formula, Q i (u, v) represents the frequency domain image after filtering of the i-th image, H(u, v) represents the standard form of the Gaussian low-pass filter, q i (x, y) represents the image inversely transformed into the spatial domain after low-pass filtering, IAM i (x,y) represents the input image, f i (x,y) represents the original image as the guide image, FAM i (x, y) represents the final metric map generated by guided filtering, GF represents the guided filtering formula, IM(x, y) represents the initial weight map, FAM1(x, y) and FAM2(x, y) represent the two final metric maps generated by the original image respectively.

5. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 1, characterized in that: The method based on regional filtering removes unrecognized pixels in the initial weight map, and locates the boundary area by combining the morphological operation expansion method with the initial weight map; fusing multiple source images according to the processed initial weight map includes the following steps: S31, according to a preset image threshold (R*M*N), the spots in the initial weight map are filtered out by adjusting the size of the filtering ratio, where R represents the filtering ratio, and M and N represent the width and height of the image respectively; S32, using a morphological operation dilation method to dilate the initial weight map after filtering out spots, and subtracting the initial weight map from the dilated image to obtain a boundary area, and fusing multiple source images according to the dilated initial weight map.

6. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 5, characterized in that: The expression for the boundary region is: BAM(x,y)=(BA(x,y)×0.5+IM(x,y))*w-IM(x,y) In the formula, BA(x,y) represents the boundary area, B represents the direction of expansion, represents the dilation operator, IM(x,y) represents the initial weight map, BAM(x,y) represents the generated boundary area, and w represents the convolution kernel.

7. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 1, characterized in that: The method of capturing the direction and displacement information between adjacent pixels in the fused image by using a global motion multi-resolution optical flow estimation algorithm and determining the weight of the boundary area according to the direction and displacement information between adjacent pixels comprises the following steps: S41, constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, using the optical flow estimation algorithm to analyze the direction and displacement information of adjacent pixels at each resolution level, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels; S42. The optical flow estimation result of the low-resolution layer is used as the initial estimation value, and is passed to the high-resolution layer layer by layer to gradually refine the direction and displacement information; based on the direction consistency and displacement amplitude, the boundary area pixels in the fused image are classified, and the weight of the boundary area is determined according to the classification result.

8. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 7, characterized in that: The method of constructing a multi-level Gaussian pyramid based on the fused image and the initial weight map, analyzing the direction and displacement information of adjacent pixels at each resolution level using the optical flow estimation algorithm, and obtaining the direction vector and displacement amplitude of each pixel relative to its neighboring pixels includes the following steps: S411, constructing a multi-level Gaussian pyramid from the fused image and the initial weight map; establishing an optical flow equation for the direction and displacement information of each pixel, and solving the optical flow equation in the neighborhood of each pixel using the least squares method to obtain a displacement vector of the pixel; S412, based on the displacement vector of each pixel, calculating the direction vector and displacement amplitude of each pixel relative to the neighboring pixels; Among them, the expression of the optical flow equation is: I x s x +I y s y +I t =0 The direction vector is calculated as: The calculation formula of displacement amplitude is: In the formula, I x ,I y Respectively represent the brightness change rate of the image in the x and y directions, I t Indicates the rate of change of brightness at time t, s x 、s y Represent the speed in the x and y directions respectively, θ represents the direction vector, and mag represents the displacement amplitude.

9. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 7, characterized in that: The optical flow estimation result of the low-resolution layer is used as the initial estimation value, and is transferred to the high-resolution layer layer by layer to gradually refine the direction and displacement information; based on the direction consistency and displacement amplitude, the boundary area pixels in the fused image are classified, and the weight of the boundary area is determined according to the classification result, which includes the following steps: S421, passing the optical flow estimation result of the low-resolution layer as the initial estimation value to the high-resolution layer layer by layer, and taking the optical flow result of the previous layer as the initial estimation of the current layer at each layer, and further optimizing the optical flow estimation at the current layer; at the high-resolution layer, using the pixel information in the neighborhood to further refine the calculation of the direction vector and the displacement amplitude, to ensure that the final high-precision optical flow information is obtained at the original resolution layer; S422, classifying the boundary area pixels in the fused image according to the comparison results of the direction consistency and the displacement amplitude with the preset direction consistency threshold and the displacement amplitude threshold, and determining the weight of the boundary area according to the classification result; Wherein, when the direction consistency metric value of a pixel in the boundary area is greater than or equal to a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a first type of pixel; When the direction consistency metric value of the pixel in the boundary area is greater than or equal to the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the second type of pixel; When the direction consistency metric value of a pixel in the boundary area is less than a preset direction consistency threshold, and the displacement amplitude value is less than a preset displacement amplitude threshold, the pixel is classified into a third type of pixel; When the direction consistency metric value of the pixel in the boundary area is less than the preset direction consistency threshold, and the displacement amplitude value is greater than or equal to the preset displacement amplitude threshold, the pixel is classified into the fourth category of pixels; And the weights of the first type of pixels, the second type of pixels, the third type of pixels, and the fourth type of pixels gradually decrease.

10. The multi-image super-resolution imaging method based on frequency domain low-pass filter guidance according to claim 6, characterized in that: The final high-resolution image is expressed as: In the formula, IF i (x, y) represents the final high-resolution image, f1(x, y) and f2(x, y) represent two low-resolution images respectively, and α and β represent the weights of the boundary areas of the two images f1(x, y) and f2(x, y) respectively.

Images