Super-resolution image reconstruction method and device

Abstract

The present invention provides a method and device for reconstructing a super-resolution image, the method comprising: obtaining A primary images; extracting pixels of pixels at the same relative position in the A primary images to synthesize D intermediate images; obtaining pixel offsets relative to a target image; setting a truncation parameter according to D to obtain a first offset distribution matrix; obtaining discrete Fourier transform results of the intermediate image; obtaining sampling results of continuous Fourier transform of each pixel in the intermediate image; dividing each pixel in the target image, extracting pixels of the divided points in each pixel to synthesize an advanced image; obtaining a second offset distribution matrix about the divided points in each pixel; obtaining discrete Fourier transform results of each divided point in the advanced image; synthesizing a super-resolution image. Compared with the prior art, the present invention reduces the cost and the difficulty of installing the required shooting instruments.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a super-resolution image reconstruction method and device. Background Art

[0002] In the process of semiconductor manufacturing and production, in order to improve the yield rate, it is very important to use semiconductor inspection equipment to detect defects in samples. However, due to the limited resolution of optical inspection, as the characteristic size of semiconductor devices decreases, the defect detection of samples will gradually become difficult, and it is often necessary to improve various aspects of the system to ensure the smooth detection of small defects.

[0003] Super Resolution Reconstruction (SRR) technology usually refers to a method of generating higher resolution images by reconstructing high-resolution images based on a single or multiple low-resolution images through digital image processing. Super-resolution reconstruction technology is a very important research issue in the field of computer vision and image processing. There are many different implementation methods, such as spatial domain methods, frequency domain methods, interpolation methods, statistical methods, etc. In recent years, there have been many developments in the technology of super-resolution reconstruction using machine learning. Super-resolution reconstruction technology has received widespread attention in the fields of satellite remote sensing, medical imaging, biometric information recognition, intelligent transportation, security monitoring, etc.

[0004] In order to obtain higher resolution images to improve the detection effect of sample defects, the semiconductor inspection field has gradually begun to pay attention to super-resolution reconstruction technology. Image super-resolution reconstruction can usually be divided into: resolution amplification of a single image, and reconstruction of a single high-resolution image from multiple low-resolution images. Image super-resolution reconstruction is also divided into the following ways: image super-resolution reconstruction in the spatial domain and image super-resolution reconstruction in the frequency domain.

[0005] Generally speaking, the resolution enlargement of a single image includes super-resolution reconstruction technology based on interpolation and super-resolution reconstruction technology based on deep learning. The super-resolution reconstruction methods based on interpolation include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and other methods. Although the nearest neighbor interpolation is simple, the aliasing phenomenon is obvious and the enlargement effect is not ideal; the high-resolution image enlarged by the bilinear interpolation method has improved aliasing, but the image edge is blurred; the bicubic interpolation effect is better, but the computational complexity is high, and when the output image is discontinuous, the enlarged high-resolution image will have ringing noise and edge blur. In order to solve the problem of ringing noise, some nonlinear interpolation methods for image edge enhancement are needed, such as edge-based interpolation methods and wavelet transform-based interpolation methods, specifically, such as the NEDI algorithm, but this will further increase the complexity of the operation. In summary, the degree of improvement in the resolution of higher-resolution images generated by interpolation methods is usually limited.

[0006] Although the deep learning method has better improvement effect, it consumes huge computational resources and is not suitable for real-time super-resolution reconstruction. The specific reasons are: 1) The deep learning super-resolution reconstruction method usually requires a lot of time to train the algorithm using a large number of samples in order to obtain a model of the correspondence between low-resolution images and high-resolution images; 2) In the process of training the model, it is necessary to carefully select various parameters such as the model structure, neural network depth, training samples, penalty function, etc., otherwise a failed model may be trained; 3) When using the model for super-resolution reconstruction algorithm, it is also necessary to carefully select some parameters to avoid super-resolution images that do not conform to the actual situation; 4) In deep learning, if the selected model structure is too shallow, the training effect is often not good enough. If the selected model structure is deep, the computational resources are often huge and it is not suitable for real-time super-resolution reconstruction.

[0007] Most multi-frame image resolution technologies are super-resolution technologies for videos and are not suitable for static image scenarios in semiconductor production inspection. In static image scenarios, multi-frame image spatial domain image super-resolution technologies are often used in the field of satellite remote sensing and are more suitable for under-sampling situations. When the sampling frequency of a low-resolution image is equal to the Nyquist sampling frequency, the degree of improvement in the resolution of a higher-resolution image generated using this low-resolution image will be relatively limited.

[0008] Traditional frequency domain image super-resolution technology retains the characteristics of the frequency domain, so the improvement effect is better and it has the advantage of faster speed. However, due to the limitation of the algorithm, traditional frequency domain image super-resolution technology requires at least 4 low-resolution images, and accordingly, 4 TDI cameras and their supporting equipment are required, which significantly increases the difficulty and cost of installation.

[0009] Therefore, the present invention proposes a super-resolution image reconstruction method and device to reduce the cost of super-resolution image reconstruction and the difficulty of installing the required equipment. Summary of the invention

[0010] The present invention provides a super-resolution image reconstruction method and device, which solves the problems in the prior art of high cost of super-resolution image reconstruction and high difficulty in installing the required equipment.

[0011] In a first aspect, the present invention provides a super-resolution image reconstruction method, comprising:

[0012] S1, obtaining A primary images of the target area, where there is a sub-pixel offset between pixels at the same position on the A primary images, and A is 2 or 3;

[0013] S2, dividing the primary image A into a plurality of sub-image parts according to every B pixel points in the horizontal direction of the primary image A and every C pixel points in the vertical direction of the primary image A, where both B and C are integers and range from 2 to 6;

[0014] S3, extracting pixels of the plurality of sub-images of each of the A primary images at the same relative position to synthesize D intermediate images, where D is equal to the product of A, B and C;

[0015] S4, selecting one of the D intermediate images as a target image, and obtaining a pixel offset of a pixel point of the intermediate image at the same relative position relative to the target image;

[0016] S5, setting a truncation parameter according to D, and obtaining a first offset distribution matrix for each pixel in the intermediate image according to the truncation parameter and the pixel offset of the pixel of the intermediate image at the same relative position relative to the target image obtained by S4;

[0017] S6, performing discrete Fourier transform on the intermediate image to obtain a discrete Fourier transform result of the intermediate image;

[0018] S7, obtaining a sampling result of a continuous Fourier transform of each pixel in the intermediate image according to the truncation parameter, the first offset distribution matrix of each pixel in the intermediate image, and the result of discrete Fourier transform of the intermediate image;

[0019] S8, dividing each pixel point in the target image at intervals of F pixels in the horizontal direction and G pixels in the vertical direction, obtaining E division points in each pixel point in the target image, where E is an integer greater than 1, the values ​​of F and G are numbers divisible by 1 between 1 / 12 and 1 / 3, the product of E, F and G is equal to 1, the product of B and F is equal to the product of C and G, and extracting the pixels of the division points at the same relative position in each pixel point to synthesize E high-level images;

[0020] S9, selecting one of the E division points in each pixel point in S8 as a reference division point, obtaining the offset of all division points in each pixel point relative to the reference division point; obtaining a second offset distribution matrix about the division point in each pixel point according to the offset and the division point;

[0021] S10, obtaining a discrete Fourier transform result of each of the division points in the E advanced images according to the second offset distribution matrix and the sampling result of the continuous Fourier transform of each pixel point;

[0022] S11. Process the discrete Fourier transform results of the division points at the same relative position in the E high-level images to synthesize a super-resolution image.

[0023] The beneficial effects are as follows: compared with the prior art, the present application requires at most 3 low-resolution images, that is, at least 3 primary images, which reduces the cost and the difficulty of installing the shooting instrument (such as TDI equipment); the present application divides the A primary image into every B pixels in the horizontal direction of the A primary image and every C pixels in the vertical direction of the A primary image, and the B and the C are both integers, and the value range is 2 to 6, so that the intermediate image with a resolution smaller than the primary image can be achieved, but the difference is not too large to avoid affecting the subsequent generation of super-resolution images; the present invention can obtain a first offset distribution matrix of appropriate size by setting a truncation parameter, and the data in the first offset distribution matrix is ​​determined by the pixel offset; the present invention can select a subsequent appropriate matrix solution method according to the size relationship between the truncation parameter and the D by setting the truncation parameter according to the D, which is more flexible; the present invention divides each pixel in the target image into F pixels in the horizontal direction and G pixels in the vertical direction to select division points with uniform position distribution, which is conducive to the subsequent generation of super-resolution images.

[0024] Optionally, in S5, the truncation parameter L is set according to D x and the cutoff parameter L y, to obtain a finite number of computational relationships between each pixel point (m, n) in the discrete Fourier transform image of the intermediate image and the sampling result of the continuous Fourier transform of the intermediate image; through the finite number of computational relationships and the pixel offset (δ xk1 , δ yk1 ), and obtain the offset distribution of each pixel point (m, n) in the k1th intermediate image in the intermediate image Wherein r is an integer, and the value range of r is 1 to 4LxLy; the corresponding The first offset distribution matrix of each pixel in the intermediate image is formed Wherein, (m, n) is the coordinate of the intermediate image in the frequency domain, and the value range of h is -L x ~L x –1, l’s value range is -L y ~L y -1, δ x，k1 is the horizontal displacement of the pixel point in the k1th intermediate image in the horizontal direction of the spatial domain relative to the pixel point in the target image, δ y，k1 is the vertical offset of the pixel points in the k1th intermediate image in the vertical direction of the spatial domain relative to the pixel points in the target image. The beneficial effect is that the present invention sets the truncation parameter L x and the cutoff parameter L y , a finite number of calculation relationships between each pixel point (m, n) in the intermediate image and the Fourier transform of the target image is obtained, which is conducive to the subsequent acquisition of the first offset distribution matrix.

[0025] Optionally, in S6, by calculating the truncation parameter T x and the truncation parameter T y The reciprocal of the product of The sampling result F of the continuous Fourier transform of the target image c The product of , obtains the discrete Fourier transform result F of the Pth image in the intermediate image p , p is an integer and its value range is 1 to D, the sampling period T x is the B, the sampling period T y is the C. Its beneficial effect is that the discrete Fourier transform result F of the Pth image in the intermediate image is effectively obtained. p .

[0026] Optionally, in S7, when the truncation parameter L x and the cutoff parameter Ly When four times the product of is less than D, the first offset distribution matrix is ​​obtained. The pseudo-inverse matrix By calculating the pseudo-inverse matrix and the discrete Fourier transform result F of the intermediate image P p The product of the continuous Fourier transform of each pixel (m, n) in the P-th intermediate image is obtained. The beneficial effect thereof is that by obtaining the pseudo-inverse matrix, it is helpful to obtain a super-resolution image with the best effect.

[0027] Optionally, in S7, when the truncation parameter L x and the cutoff parameter L y When the product of four times is equal to D, the first offset distribution matrix is ​​obtained. The inverse matrix By calculating the inverse matrix and the discrete Fourier transform result F of the intermediate image P p The product of the continuous Fourier transform of each pixel (m, n) in the P-th intermediate image is obtained. (r) The beneficial effect is that by obtaining the inverse matrix, it can be ensured that the obtained super-resolution image does not lose frequency domain information.

[0028] Optionally, in S9, the number of rows of the second offset distribution matrix is ​​determined by the truncation parameter Lx and the truncation parameter Ly, and the number of columns of the second offset distribution matrix is ​​determined by E; and the offsets (δ x，k2 , δ y，k2 ), the δ x，k2 represents the offset of the dividing point in the horizontal direction relative to the reference dividing point, the δ y，k2 represents the offset of the division point relative to the reference division point in the vertical direction, and the value range of k2 is 1 to E. The beneficial effect is that an effective second offset distribution matrix is ​​obtained.

[0029] Optionally, in S10, by calculating the second offset distribution matrix The sampling result of the continuous Fourier transform of each pixel (m, n) in the intermediate image The product of , obtains the discrete Fourier transform result F' of each of the division points (a, b) in the E advanced images k2(a, b); wherein the value range of k2 is 1 to E. The beneficial effect is that the effective discrete Fourier transform results of each division point in all the advanced images are obtained.

[0030] Optionally, in S4, the intermediate image is compared with the target image one by one to respectively obtain pixel offsets of pixel points of the intermediate image at the same relative position relative to the target image.

[0031] Optionally, in S4, the pixel offset relationship between the first intermediate image and the target image is obtained through B and C, and the pixel offset of the pixel points at the same relative position of the first intermediate image and the target image is obtained, and the first intermediate image is the same intermediate image as the primary image corresponding to the target image; the pixel offset of the pixel points at the same relative position of the second intermediate image and the target image is obtained through the offset between B and C, and A sheets of the primary images, and the second intermediate image is an intermediate image different from the primary image corresponding to the target image.

[0032] In a second aspect, the present invention provides a super-resolution image reconstruction device, which is used to perform the super-resolution image reconstruction method as described in any one of the first aspects, comprising: an acquisition module, a division module, an extraction module, a processing module, and a synthesis module; the division module comprises a first division unit and a second division unit; the extraction module comprises a first extraction unit and a second extraction unit; the processing module comprises an offset acquisition unit, a discrete Fourier transform unit, a parameter setting unit, a matrix generation unit, and a continuous Fourier processing unit; the synthesis module comprises a first synthesis unit, a second synthesis unit, and a third synthesis unit;

[0033] The acquisition module is used to acquire A primary images of the target area, where there is a sub-pixel offset between pixels at the same position on the A primary images, and A is 2 or 3;

[0034] The first division unit is used to divide the A primary images into a plurality of sub-image parts by dividing them into every B pixels in the horizontal direction of the A primary images and every C pixels in the vertical direction of the A primary images, where both B and C are integers and range from 2 to 6;

[0035] The first extraction unit is used to extract pixels of the pixel points of the plurality of sub-image parts of each of the A primary images at the same relative position, and the first synthesis unit synthesizes D intermediate images according to the extraction results of the first extraction unit, where D is equal to the product of A, B and C;

[0036] The offset acquisition unit is used to select one of the D intermediate images as the target image, and acquire the pixel offset of the pixel point of the intermediate image at the same relative position relative to the target image;

[0037] The parameter setting unit is used to set the truncation parameter according to the D; the matrix generating unit is used to obtain a first offset distribution matrix of each pixel point in the intermediate image according to the truncation parameter and the pixel offset of the pixel point at the same relative position of the intermediate image relative to the target image;

[0038] The discrete Fourier transform unit is used to perform discrete Fourier transform on the intermediate image to obtain a result of performing discrete Fourier transform on the intermediate image;

[0039] The continuous Fourier processing unit is used to obtain a sampling result of a continuous Fourier transform of each pixel point in the intermediate image according to the truncation parameter, the first offset distribution matrix of each pixel point in the intermediate image, and the result of discrete Fourier transform of the intermediate image;

[0040] The second division unit is used to divide each pixel point in the target image at intervals of F pixels in the horizontal direction and G pixels in the vertical direction to obtain E division points in each pixel point in the target image, where E is an integer greater than 1, and the values ​​of F and G are numbers divisible by 1 between 1 / 12 and 1 / 3, the product of E, F and G is equal to 1, and the product of B and F is equal to the product of C and G; the second extraction unit is used to extract the pixels of the division points at the same relative position in each pixel point; the second synthesis unit is used to synthesize E high-level images according to the extraction results of the second extraction unit;

[0041] The matrix generating unit is further used to select one of the E division points in each pixel point as a reference division point, obtain the offset of all the division points in each pixel point relative to the reference division point; and obtain a second offset distribution matrix about the division point in each pixel point according to the offset and the division point;

[0042] The continuous Fourier processing unit is further used to obtain a discrete Fourier transform result of each of the division points in the E advanced images according to the second offset distribution matrix and the sampling result of the continuous Fourier transform of each pixel point;

[0043] The third synthesis unit is used to process the discrete Fourier transform results of the division points at the same relative position in the E high-level images to synthesize a super-resolution image.

[0044] The beneficial effects are as follows: compared with the prior art, the present application only needs 2 to 3 low-resolution images, that is, only 2 to 3 primary images, that is, only 2 to 3 sets of equipment are needed to obtain low-resolution images, which reduces the cost and the difficulty of installing the shooting instruments (such as TDI cameras and their supporting equipment); the present application divides the A primary image into every B pixels in the horizontal direction of the A primary image and every C pixels in the vertical direction of the A primary image, and the B and C are both integers with a value range of 2-6, so as to achieve an intermediate image with a resolution smaller than that of the primary image. However, the difference will not be too big to avoid affecting the subsequent generation of super-resolution images; the present invention can obtain a first offset distribution matrix of appropriate size by setting a truncation parameter, and the data in the first offset distribution matrix is ​​determined by the pixel offset; the present invention can select a subsequent appropriate matrix solution method according to the size relationship between the truncation parameter and the D by setting the truncation parameter according to the D, which is more flexible; the present invention divides each pixel point in the target image by spacing F pixels in the horizontal direction and G pixels in the vertical direction to select division points with uniform position distribution, which is conducive to the subsequent generation of super-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a process flow of a super-resolution image reconstruction method provided by the present invention;

[0046] Figure 2 A schematic diagram of a primary image embodiment provided by the present invention;

[0047] Figure 3 A schematic diagram of an intermediate image embodiment provided by the present invention;

[0048] Figure 4 A schematic diagram of an embodiment of pixel offset distribution of an intermediate image relative to a target image provided by the present invention;

[0049] Figure 5 A schematic diagram of the relationship between i, l and r provided by the present invention;

[0050] Figure 6 A schematic diagram of the distribution of division points within a pixel provided by the present invention;

[0051] Figure 7 A schematic diagram of an embodiment of a super-resolution image reconstruction device provided by the present invention. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "the", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear indication to the contrary in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in a "or" relationship.

[0053] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0054] In the embodiments of the present application, the words "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0055] In order to reduce the cost and installation difficulty in the prior art while ensuring that a super-resolution image is obtained, the present invention provides a super-resolution image reconstruction method.

[0056] In some embodiments, the process of the super-resolution image reconstruction method is as follows: Figure 1 As shown, including:

[0057] S101, acquiring two primary images of a target area, wherein there is a sub-pixel offset between pixels at the same position on the two primary images;

[0058] S102, dividing the two primary images at every 2 pixels in the horizontal direction of the two primary images and at every 2 pixels in the vertical direction of the two primary images to obtain a plurality of sub-image parts;

[0059] S103, extracting pixels of the plurality of sub-images of each of the two primary images at the same relative position to synthesize eight intermediate images; wherein each primary image corresponds to four intermediate images;

[0060] S104, selecting one of the eight intermediate images as the target image, for example, selecting Figure 3 (1) in the figure is used as the target image, and the pixel offsets of the pixel points of the other intermediate images at the same relative position relative to the target image are obtained;

[0061] S105, setting two truncation parameters, and four times the product of the two truncation parameters is less than or equal to 8; obtaining a first offset distribution matrix for each pixel in the intermediate image according to the truncation parameters and the pixel offsets of the pixel points at the same relative position of the intermediate image relative to the target image obtained by S104;

[0062] S106, performing discrete Fourier transform on the intermediate image to obtain a discrete Fourier transform result of the intermediate image;

[0063] S107, obtaining a sampling result of a continuous Fourier transform of each pixel in the intermediate image according to the truncation parameter, the first offset distribution matrix of each pixel in the eight intermediate images, and the result of discrete Fourier transform of the intermediate image;

[0064] S108, dividing each pixel point in the target image at intervals of 0.25 pixels in the horizontal direction and the vertical direction to obtain 16 division points in each pixel point in the target image, and extracting pixels of the division points at the same relative position in each pixel point to synthesize 16 high-level images;

[0065] S109, selecting one of the 16 division points in each pixel in S108 as a reference division point, obtaining the offsets of all division points in each pixel relative to the reference division point; obtaining a second offset distribution matrix about the division point in each pixel according to the offsets and the division points;

[0066] S110, obtaining a discrete Fourier transform result of each of the division points in the E advanced images according to the second offset distribution matrix and the sampling result of the continuous Fourier transform of each pixel point;

[0067] S111, performing processing according to the discrete Fourier transform results of the division points at the same relative position in the 16 high-level images to synthesize a super-resolution image.

[0068] In order to introduce this embodiment in more detail, further explanations are given in some specific embodiments.

[0069] In S101, the primary images are respectively as follows: Figure 2 As shown in (1) and (2) in .

[0070] In S92 and S103, the Figure 2 The intermediate image corresponding to (1) is as follows Figure 3 As shown in (1), (2), (3), and (4); Figure 2 The intermediate image corresponding to (2) is as follows Figure 3 As shown in (5), (6), (7), and (8), each square represents a pixel. Figure 2 The pixels with the same letters in the squares in (1) and (2) are extracted into different mid-level images.

[0071] In S94, for example, select Figure 3 (1) in is taken as the target image, then we get Figure 3 (2), (3), (4), (5), (6), (7), (8) relative to Figure 3 The pixel offset of the pixel point at the same relative position in (1) is obtained. In S94, there are two processing methods. The first method is to obtain Figure 3 (2), (3), (4), (5), (6), (7), (8) relative to Figure 3 The second is through the offset of (1) Figure 3 (2), (3) and (4) in the above Figure 3 The fixed pixel offset relationship of (1) in Figure 3 (2), (3), and (4) are relative to Figure 3The pixel offset of the pixel point at the same relative position in (1); Figure 2 (2) relative to Figure 2 The pixel offset of (1) in Figure 3 (6), (7) and (8) in Figure 3 The fixed pixel offset relationship of (5) in is obtained Figure 3 (5), (6), (7), (8) relative to Figure 3 The pixel offset of the pixel point at the same relative position in (1). Figure 3 (5) relative to Figure 3 The pixel offset of (1) is (δ x , δ y ), the Figure 3 (1), (2), (3), (4), (5), (6), (7), (8) relative to Figure 3 The offset of (1) is as follows Figure 4 The numbers in the same row as k1 correspond to Figure 3 (1), (2), (3), (4), (5), (6), (7), and (8) in the text.

[0072] In the step S105, the truncation parameter L is set according to the D x and the cutoff parameter L y , to get Figure 3 Each pixel (m, n) in the discrete Fourier transform of (1), (2), (3), (4), (5), (6), (7), (8) is Figure 3 A finite number of computational relationships between the sampling results of successive Fourier transforms of the corresponding intermediate images in .

[0073] By formula Get the offset distribution of each pixel (m, n) in the k1th intermediate image in the intermediate image Where r is an integer, and r is the one-dimensional mapping result of (h, l), and the value range is 1 to 4L x L y ; Through the formula Get the first offset distribution matrix of each pixel (m, n) in the intermediate image Among them, the value range of h is -L x ~L x –1, l’s value range is -L y ~L y -1, δ x，k1 is the horizontal displacement of the pixel point in the k1th intermediate image in the horizontal direction of the spatial domain relative to the pixel point in the target image, δy，k1 is the vertical offset of the pixel points in the k1th intermediate image in the vertical direction of the spatial domain relative to the pixel points in the target image. x is equal to 2, the T y is equal to 2, and the value range of k1 is 1 to 8. Formula Simplifying, we get:

[0074] The value of r ranges from 1 to 4L x L y =4. When L x =2,L y = 2, the corresponding h, l and r are as follows Figure 5 shown.

[0075] In S106, by calculating the sampling period T x and sampling period T y The reciprocal of the product of The sampling result F of the continuous Fourier transform of the target image c The product of , obtains the discrete Fourier transform result F of the Pth image in the intermediate image p , p is an integer and its value range is 1 to 8, the sampling period T x is the B, the sampling period T y For the C.

[0076] In said S107,

[0077] Where L x and L y are the truncation parameters in the horizontal and vertical directions respectively. Let the truncation parameter L x =L y =1; F c k1 represents the sampling result of the continuous Fourier transform of each pixel (m, n) in the k1th intermediate image, F c Represents the sampling result of the continuous Fourier transform of the target image; by setting

[0078] Therefore, the discrete Fourier transform results corresponding to all intermediate images can be written in matrix form:

[0079]

[0080]

[0081]

[0082]

[0083] It is abbreviated to the following form: By formula Obtain the discrete Fourier transform result F of the Pth image in the intermediate image p The value range of p is 1 to D. In this embodiment, the maximum value of p is 8.

[0084] In S107, when the truncation parameter L x and the cutoff parameter L y When both are set to 1, the truncation parameter L x and the cutoff parameter L y The product of four times is less than 8, according to the formula Get the first offset distribution matrix The pseudo-inverse matrix Said That is, By calculating the pseudo-inverse matrix and the discrete Fourier transform result F of the intermediate image P p The product of the continuous Fourier transform of each pixel (m, n) in the P-th intermediate image is obtained.

[0085] Alternatively, in S107, when the truncation parameter L x Set to 2, the truncation parameter L y Set to 1, or the truncation parameter L x Set to 1, the truncation parameter L y When set to 2, the truncation parameter L x and the cutoff parameter L y The product of four times is equal to 8, obtaining the first offset distribution matrix The inverse matrix By calculating the inverse matrix and the discrete Fourier transform result F of the intermediate image P p The product of the continuous Fourier transform of each pixel (m, n) in the P-th intermediate image is obtained.

[0086] In the step S109, the second offset distribution matrix It is a matrix with 16 rows and 4 columns, and the number of columns is 4L x L y; The number of rows is 16, and the offsets (δ x，k2 , δ y，k2 ), the δ x，k2 represents the offset of the dividing point in the horizontal direction relative to the reference dividing point, the δ y，k2 represents the vertical offset of the division point relative to the reference division point, and the value range of k2 is 1 to 16. The positional relationship between the 16 division points in each pixel is as follows: Figure 6 As shown in (1), the squares represent pixel points, and the numbers in the squares represent the serial numbers of the corresponding pixel points. When the serial number of the reference division point is 1, and the offset between the division points is 0.25 pixels, the offsets of the 16 division points in each pixel relative to the reference division point are as follows: Figure 6 As shown in (2) and (3), Figure 6 The v in represents the offset of the division point in the horizontal direction relative to the reference division point. Figure 6 The u in represents the offset of the division point relative to the reference division point in the vertical direction.

[0087] In S110, by formula Calculate the second offset distribution matrix The sampling result of the continuous Fourier transform of each pixel (m, n) in the intermediate image The product of , obtains the discrete Fourier transform result F' of each of the division points (a, b) in the 16 advanced images k2 (a, b). The value range of k2 is 1-16.

[0088] In the step S111, the discrete Fourier transform result F' of each of the division points (a, b) in the 16 advanced images is obtained. k2 (a, b), the discrete Fourier transform results of the divided points are discretely inverted Fourier transformed at the corresponding positions, and the discrete inverted Fourier transform results are superimposed to synthesize a super-resolution image.

[0089] In some other embodiments, the super-resolution image reconstruction method includes: acquiring three primary images of the target area, and processing them in the same manner as the above embodiment to acquire a super-resolution image, which will not be described again.

[0090] Optionally, the values ​​of B and C may be 2, 3, 4, 5 or 6. More preferably, the values ​​of B and C are 2 or 3.

[0091] Optionally, the value of F may be 0.1, 0.2, 0.25 or 0.5. More preferably, the value of F is 0.25.

[0092] Based on the super-resolution image reconstruction method provided in any of the above embodiments, the present application also provides a super-resolution image reconstruction device, which is used to execute the super-resolution image reconstruction method as described in any of the above embodiments, such as Figure 7 As shown, it includes: an acquisition module 701, a division module 702, an extraction module 703, a processing module 704, and a synthesis module 705; the division module 702 includes a first division unit 7021 and a second division unit 7022; the extraction module 703 includes a first extraction unit 7031 and a second extraction unit 7032; the processing module 704 includes an offset acquisition unit 7041, a discrete Fourier transform unit 7042, a parameter setting unit 7043, a matrix generation unit 7044, and a continuous Fourier processing unit 7045; the synthesis module 705 includes a first synthesis unit 7051, a second synthesis unit 7052 and a third synthesis unit 7053.

[0093] The acquisition module 701 is used to acquire A primary images of the target area, where there is a sub-pixel offset between pixels at the same position on the A primary images, and A is 2 or 3.

[0094] The first division unit 7021 is used to divide the A primary image into every B pixels in the horizontal direction of the A primary image and every C pixels in the vertical direction of the A primary image to obtain a plurality of sub-image parts, where B and C are both integers and range from 2 to 6.

[0095] The first extraction unit 7031 is used to extract pixels of the pixel points of the several sub-image parts of each of the A primary images at the same relative periodic position, and the first synthesis unit 7051 synthesizes D intermediate images according to the extraction results of the first extraction unit 7031, where D is equal to the product of A, B and C.

[0096] The offset acquisition unit 7041 is used to select one of the D intermediate images as the target image, and acquire pixel offsets of pixel points of the remaining intermediate images at the same relative position relative to the target image;

[0097] The parameter setting unit 7043 is used to set the truncation parameters; the matrix generating unit 7044 is used to obtain a first offset distribution matrix for each pixel point in the intermediate image according to the truncation parameters and the pixel offset of the pixel points at the same relative position of the intermediate image relative to the target image.

[0098] The discrete Fourier transform unit 7042 is used to perform Fourier transform on the target image to obtain a result of the Fourier transform of the target image.

[0099] The continuous Fourier processing unit 7045 is used to obtain a sampling result of a continuous Fourier transform of each pixel point in the intermediate image according to the result of discrete Fourier transform performed on the first offset distribution matrix and the intermediate image.

[0100] The second division unit 7022 is used to divide each pixel point in the target image at an interval of F pixels in the horizontal direction and G pixels in the vertical direction to obtain E division points in each pixel point in the target image, where E is an integer greater than 1, the value range of F and G is 1 / 12 to 1 / 3, and the number is divisible by 1, the product of E, F and G is equal to 1, and the product of B and F is equal to the product of C and G; the second extraction unit 7032 is used to extract the pixels of the division points at the same relative position in each pixel point; the second synthesis unit 7052 is used to synthesize E high-level images according to the extraction results of the second extraction unit 7032.

[0101] The matrix generation unit 7044 is also used to select one of the E division points within each pixel point as a reference division point, and obtain the offset of all division points within each pixel point relative to the reference division point; based on the offset and the division point, obtain a second offset distribution matrix about the division point within each pixel point.

[0102] The continuous Fourier processing unit 7045 is also used to obtain the discrete Fourier transform results of each of the division points in the E advanced images according to the second offset distribution matrix and the sampling results of the continuous Fourier transform of each pixel point.

[0103] The third synthesis unit 7053 is used to process the discrete Fourier transform results of the division points at the same relative position in the E high-level images to synthesize a super-resolution image.

[0104] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding unit module, and will not be repeated here.

[0105] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A super-resolution image reconstruction method, It is characterized in that include: S1, obtaining A primary images of the target area, where there is a sub-pixel offset between pixels at the same position on the A primary images, and A is 2 or 3; S2, dividing the primary image A into a plurality of sub-image parts according to every B pixel points in the horizontal direction of the primary image A and every C pixel points in the vertical direction of the primary image A, where both B and C are integers and range from 2 to 6; S3, extracting pixels of the plurality of sub-images of each of the A primary images at the same relative position to synthesize D intermediate images, where D is equal to the product of A, B and C; S4, selecting one of the D intermediate images as a target image, and obtaining a pixel offset of a pixel point of the intermediate image at the same relative position relative to the target image; S5, setting a truncation parameter according to D, and obtaining a first offset distribution matrix for each pixel in the intermediate image according to the truncation parameter and the pixel offset of the pixel of the intermediate image at the same relative position relative to the target image obtained by S4; S6, performing discrete Fourier transform on the intermediate image to obtain a discrete Fourier transform result of the intermediate image; S7, obtaining a sampling result of a continuous Fourier transform of each pixel in the intermediate image according to the truncation parameter, the first offset distribution matrix of each pixel in the intermediate image, and the result of discrete Fourier transform of the intermediate image; S8, dividing each pixel point in the target image at intervals of F pixels in the horizontal direction and G pixels in the vertical direction, obtaining E division points in each pixel point in the target image, where E is an integer greater than 1, the values ​​of F and G are numbers divisible by 1 between 1 / 12 and 1 / 3, the product of E, F and G is equal to 1, the product of B and F is equal to the product of C and G, and extracting the pixels of the division points at the same relative position in each pixel point to synthesize E high-level images; S9, selecting one of the E division points in each pixel point in S8 as a reference division point, obtaining the offset of all division points in each pixel point relative to the reference division point; obtaining a second offset distribution matrix about the division point in each pixel point according to the offset and the division point; S10, obtaining a discrete Fourier transform result of each of the division points in the E advanced images according to the second offset distribution matrix and the sampling result of the continuous Fourier transform of each pixel point; S11. Process the discrete Fourier transform results of the division points at the same relative position in the E high-level images to synthesize a super-resolution image.

2. The method for reconstructing a super-resolution image according to claim 1, It is characterized in that In S5, the truncation parameter L is set according to D x and the cutoff parameter L y , so as to obtain a finite number of computational relationships between each pixel point (m, n) in the discrete Fourier transform image of the intermediate image and the sampling result of the continuous Fourier transform of the intermediate image; Through the finite number of calculation relationships and the pixel offset (δ x，k1 , δ y，k1 ), and obtain the offset distribution of each pixel point (m, n) in the k1th intermediate image in the intermediate image Where r is an integer, and r is the one-dimensional mapping result of (h, l), and the value range is 1 to 4L x L y ; The corresponding The first offset distribution matrix of each pixel in the intermediate image is formed Wherein, (m, n) is the coordinate of the intermediate image in the frequency domain, and the value range of h is -L x ~L x –1, l’s value range is -L y ~L y -1, δ x，k1 is the horizontal displacement of the pixel point in the k1th intermediate image in the horizontal direction of the spatial domain relative to the pixel point in the target image, δ y，k1 is the vertical offset of the pixel points in the k1th intermediate image in the vertical direction of the spatial domain relative to the pixel points in the target image in the vertical direction.

3. The super-resolution image reconstruction method according to claim 2, It is characterized in that In said S6, By calculating the sampling period T x and sampling period T y The reciprocal of the product of The sampling result F of the continuous Fourier transform of the target image c The product of , obtains the discrete Fourier transform result F of the Pth image in the intermediate image p , p is an integer and its value range is 1 to D, the sampling period T x is the B, the sampling period T y For the C.

4. The method for reconstructing a super-resolution image according to claim 3, It is characterized in that In S7, when the cutoff parameter L x and the cutoff parameter L y When the product of four times is less than D, Get the first offset distribution matrix The pseudo-inverse matrix By calculating the pseudo-inverse matrix and the discrete Fourier transform result F of the intermediate image P p The product of the continuous Fourier transform of each pixel (m, n) in the P-th intermediate image is obtained.

5. The method for reconstructing a super-resolution image according to claim 3, It is characterized in that In S7, when the cutoff parameter L x and the cutoff parameter L y When the product of four times is equal to D, Get the first offset distribution matrix The inverse matrix By calculating the inverse matrix and the discrete Fourier transform result F of the intermediate image P p The product of the continuous Fourier transform of each pixel (m, n) in the P-th intermediate image is obtained.

6. The method for reconstructing a super-resolution image according to claim 4 or 5, It is characterized in that In said S9, By the truncation parameter L x , truncation parameter L y Determine the number of rows of the second offset distribution matrix, and determine the number of columns of the second offset distribution matrix through E; Calculate the offset (δ) of all the partition points within each pixel relative to the reference partition point x，k2 , δ y，k2 ), the δ x，k2 represents the offset of the dividing point in the horizontal direction relative to the reference dividing point, the δ y，k2 It represents the vertical offset of the division point relative to the reference division point. The value range of k2 is 1 to E.

7. The method for reconstructing a super-resolution image according to claim 6, It is characterized in that In said S10, The second offset distribution matrix is ​​calculated by The sampling result of the continuous Fourier transform of each pixel (m, n) in the intermediate image The product of , obtains the discrete Fourier transform result F' of each of the division points (a, b) in the E advanced images k2 (a, b); wherein the value range of k2 is 1~E.

8. The super-resolution image reconstruction method according to claim 1, It is characterized in that In S4, the intermediate image is compared with the target image one by one to obtain pixel offsets of pixel points of the intermediate image at the same relative position relative to the target image.

9. The super-resolution image reconstruction method according to claim 1, It is characterized in that In said S4, By using B and C, a pixel offset relationship between a first intermediate image and the target image is obtained, and a pixel offset of pixel points at the same relative position between the first intermediate image and the target image is obtained, wherein the first intermediate image is the same intermediate image as the primary image corresponding to the target image; By using the offset between B and C, and A, the pixel offset of the pixel points of the second intermediate image and the target image at the same relative position is obtained. The second intermediate image is an intermediate image that is different from the primary image corresponding to the target image.

10. A super-resolution image reconstruction device, It is characterized in that A method for reconstructing a super-resolution image as described in any one of claims 1 to 9, comprising: an acquisition module, a division module, an extraction module, a processing module, and a synthesis module; the division module comprises a first division unit and a second division unit; the extraction module comprises a first extraction unit and a second extraction unit; the processing module comprises an offset acquisition unit, a discrete Fourier transform unit, a parameter setting unit, a matrix generation unit, and a continuous Fourier processing unit; the synthesis module comprises a first synthesis unit, a second synthesis unit, and a third synthesis unit; The acquisition module is used to acquire A primary images of the target area, where there is a sub-pixel offset between pixels at the same position on the A primary images, and A is 2 or 3; The first division unit is used to divide the A primary images into a plurality of sub-image parts by dividing them into every B pixels in the horizontal direction of the A primary images and every C pixels in the vertical direction of the A primary images, where both B and C are integers and range from 2 to 6; The first extraction unit is used to extract pixels of the pixel points of the plurality of sub-image parts of each of the A primary images at the same relative position, and the first synthesis unit synthesizes D intermediate images according to the extraction results of the first extraction unit, where D is equal to the product of A, B and C; The offset acquisition unit is used to select one of the D intermediate images as the target image, and acquire the pixel offset of the pixel point of the intermediate image at the same relative position relative to the target image; The parameter setting unit is used to set the truncation parameter according to the D; the matrix generating unit is used to obtain a first offset distribution matrix of each pixel point in the intermediate image according to the truncation parameter and the pixel offset of the pixel point at the same relative position of the intermediate image relative to the target image; The discrete Fourier transform unit is used to perform discrete Fourier transform on the intermediate image to obtain a result of performing discrete Fourier transform on the intermediate image; The continuous Fourier processing unit is used to obtain a sampling result of a continuous Fourier transform of each pixel point in the intermediate image according to the truncation parameter, the first offset distribution matrix of each pixel point in the intermediate image, and the result of discrete Fourier transform of the intermediate image; The second division unit is used to divide each pixel point in the target image at intervals of F pixels in the horizontal direction and G pixels in the vertical direction to obtain E division points in each pixel point in the target image, where E is an integer greater than 1, and the values ​​of F and G are numbers divisible by 1 between 1 / 12 and 1 / 3, the product of E, F and G is equal to 1, and the product of B and F is equal to the product of C and G; the second extraction unit is used to extract the pixels of the division points at the same relative position in each pixel point; the second synthesis unit is used to synthesize E high-level images according to the extraction results of the second extraction unit; The matrix generating unit is further used to select one of the E division points in each pixel point as a reference division point, obtain the offset of all the division points in each pixel point relative to the reference division point; and obtain a second offset distribution matrix about the division point in each pixel point according to the offset and the division point; The continuous Fourier processing unit is further used to obtain a discrete Fourier transform result of each of the division points in the E advanced images according to the second offset distribution matrix and the sampling result of the continuous Fourier transform of each pixel point; The third synthesis unit is used to process the discrete Fourier transform results of the division points at the same relative position in the E high-level images to synthesize a super-resolution image.

Images