Multi-frame image super-resolution reconstruction method, apparatus, equipment and storage medium

Abstract

The present application relates to a method, device, equipment and storage medium for super-resolution reconstruction of multiple frames of images, which are applied in the field of image processing, including obtaining multiple frames of low-resolution sequence images according to a preset super-resolution scale; setting the first frame image in the multiple frames of low-resolution sequence images as a reference frame image, and calculating the relative offset between the remaining frames of low-resolution sequence images and the reference frame image; selecting a low-resolution reconstructed image that meets the super-resolution grid condition in the multiple frames of low-resolution sequence images, and the super-resolution grid condition is set according to the preset super-resolution scale and the calculated relative offset; rearranging the pixels of the low-resolution reconstructed image to generate an initial estimated image; inputting the initial estimated image into the trained image super-resolution reconstruction model for processing to generate the final super-resolution image. The technical effects of the present application are: reducing the complexity of the operation, simplifying the operation, reducing the cost, and improving the accuracy of imaging.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for super-resolution reconstruction of multiple frames of images. Background Art

[0002] Image super-resolution reconstruction refers to the technology of restoring one or more low-resolution images into a high-resolution image. The enhancement of image resolution by super-resolution reconstruction technology is not just to increase the number of image pixels, but more importantly to increase the detail information that the low-resolution image itself does not have. A single frame image cannot provide such detail information. Through multiple sampling and using the sub-pixel complementary information between multiple frames of images, the high-frequency details in the image can be estimated and reconstructed more accurately.

[0003] The current mainstream super-resolution technologies include micro-scanning technology reconstruction algorithms. Micro-scanning technology requires control of tiny scanning steps and precise positioning to ensure that the displacement is less than one pixel. The image information lost due to downsampling can be restored by using the sub-pixel information between two frames. However, due to limitations of mechanical structure, electronic control and other factors, the micro-scanning method is difficult to control the accuracy and stability of the instrument, which may lead to errors or distortions in the final imaging results. Summary of the invention

[0004] In order to help solve the problem that micro-scanning methods are difficult to control the accuracy and stability of instruments due to limitations of mechanical structure, electronic control and other factors, which may lead to errors or distortions in the final imaging results, the present application provides a multi-frame image super-resolution reconstruction method, device, equipment and storage medium.

[0005] In a first aspect, the present application provides a multi-frame image super-resolution reconstruction method, which adopts the following technical solution: the method comprises:

[0006] Acquire multiple frames of low-resolution sequence images according to the preset super-resolution;

[0007] The first frame image in the plurality of frames of low-resolution sequence images is set as a reference frame image, and the relative offset between the remaining frames of low-resolution sequence images and the reference frame image is calculated;

[0008] Selecting a low-resolution reconstructed image that meets a super-resolution grid condition from among the multiple frames of low-resolution sequence images, wherein the super-resolution grid condition is set according to the preset super-resolution scale and the calculated relative offset;

[0009] Rearranging pixels of the low-resolution reconstructed image to generate an initial estimated image;

[0010] The initial estimated image is input into the trained image super-resolution reconstruction model for processing to generate a final super-resolution image.

[0011] In a specific implementation manner, after selecting the low-resolution reconstructed image that meets the super-grid condition, the method further includes:

[0012] Determining whether the number of the selected low-resolution reconstructed images is less than a preset reconstruction number value;

[0013] If the number of the low-resolution reconstructed images is less than a preset reconstruction number, reacquiring multiple frames of low-resolution sequence images;

[0014] The relative offset of the re-acquired multiple frames of low-resolution sequence images is calculated, the super-resolution grid condition is updated, and the low-resolution reconstructed image that meets the super-resolution grid condition is reselected.

[0015] In a specific possible implementation scheme, the number of frames of the multiple low-resolution sequence images is a preset lower limit threshold, and the preset lower limit threshold is selected according to the super-resolution;

[0016] After selecting the low-resolution reconstructed image that meets the super-resolution grid condition, the method further includes:

[0017] Determining whether the number of the selected low-resolution reconstructed images is less than a preset reconstruction number value;

[0018] If the number of the low-resolution reconstructed images is less than a preset reconstruction number, reacquiring multiple frames of low-resolution sequence images, wherein the number of frames of the multiple frames of low-resolution sequence images is a preset step number increased on the basis of a preset lower limit threshold;

[0019] Calculating the relative offset of the re-acquired multiple frames of low-resolution sequence images, updating the super-resolution grid condition and reselecting the low-resolution reconstructed image that meets the super-resolution grid condition;

[0020] Circularly determining whether the number of the selected low-resolution reconstructed images is less than a preset reconstruction number value;

[0021] If the number of the low-resolution reconstructed images is less than a preset reconstruction number, multiple frames of low-resolution sequence images are reacquired until the number of frames of the multiple frames of low-resolution sequence images reaches a preset upper limit threshold.

[0022] In a specific implementation manner, the calculating of the relative offset between the remaining low-resolution sequence images and the reference frame image comprises:

[0023] Performing center cropping on the plurality of frames of low-resolution sequence images according to a preset size to generate a center area sequence image corresponding to the plurality of frames of low-resolution sequence images;

[0024] Using an optical flow neural network to calculate the optical flow vector of each pixel in the central area sequence image;

[0025] Calculate the median of the optical flow vectors of all pixels in the horizontal and vertical directions according to the calculated optical flow vector of each pixel, and generate the median of the horizontal optical flow vector and the median of the vertical optical flow vector;

[0026] Calculating the absolute deviation of the optical flow vector of each pixel from the median of the horizontal optical flow vector and the median of the vertical optical flow vector in the horizontal direction and the vertical direction respectively, and comparing the absolute deviation with a preset deviation threshold;

[0027] Set the pixel points whose absolute deviation is greater than the preset deviation threshold as singular points, and delete the singular points;

[0028] Calculate the average optical flow vectors of the remaining pixels except the singular points in the horizontal and vertical directions to generate the horizontal offset and vertical offset.

[0029] In a specific implementation scheme, the relative offset includes a sub-pixel offset, and the selecting of a low-resolution reconstructed image that meets the super-grid condition includes:

[0030] Setting a super-resolution grid according to the super-resolution scale, and setting a screening condition for each grid according to the sub-pixel offset and the position of each grid, wherein the size of the super-resolution grid is the square of the super-resolution scale;

[0031] According to the screening conditions of each grid, a low-resolution image meeting the screening conditions of each grid is selected;

[0032] Determine whether there are multiple low-resolution images in each grid that meet the screening conditions of the corresponding grid;

[0033] If there are multiple low-resolution images in a single grid that meet the screening conditions of the corresponding grid, the distance between the sub-pixel offset of the low-resolution image and the preset offset is calculated, and the low-resolution image with the smallest distance is selected;

[0034] The low-resolution image screened out from each grid is set as the low-resolution reconstructed image;

[0035] The screening conditions include:

[0036]

[0037] in, and is an integer, represents the Ith grid in the horizontal direction of the super-resolution grid, represents the Ith grid in the vertical direction of the super-resolution grid; S represents the super-resolution scale; Indicates the sub-pixel offset in the horizontal direction, Indicates the sub-pixel offset in the vertical direction; represents the sub-pixel error threshold, Represents the alignment error threshold.

[0038] In a specific implementation manner, the performing pixel rearrangement on the low-resolution reconstructed image to generate an initial estimated image comprises:

[0039] translating the low-resolution reconstructed image;

[0040] According to the grid number corresponding to the low-resolution reconstructed image in the super-resolution grid, rearrange the pixels of the low-resolution reconstructed image in a preset manner to generate a rearranged image;

[0041] Determining whether there are blank pixels in the rearranged image;

[0042] If there are blank pixels in the rearranged image, an interpolation algorithm is used to fill the blank pixels and generate the initial estimated image;

[0043] If there are no blank pixels in the rearranged image, setting the rearranged image as the initial estimated image;

[0044] The translation method includes:

[0045]

[0046] in, Indicates the relative offset in the horizontal direction. Indicates the relative offset in the vertical direction; Express Round down, Express Round down; Point to Round off, I image Indicates the super-resolution grid number;

[0047] The pixel rearrangement method includes:

[0048]

[0049] Wherein, r represents the rth grid in the horizontal direction of the super-resolution grid, and c represents the cth grid in the vertical direction of the super-resolution grid; LR irepresents the low-resolution reconstructed image corresponding to the i-th grid in the super-resolution grid; h represents the height of the low-resolution reconstructed image; w represents the width of the low-resolution reconstructed image; mod represents the modulo operation; represents the rearranged image.

[0050] In a specific implementation scheme, the training method of the image super-resolution reconstruction model includes:

[0051] Acquire a low-resolution image training set and a high-resolution image training set corresponding to the low-resolution image training set;

[0052] Using the multi-frame image super-resolution reconstruction method to process the low-resolution image training set to generate an initial estimation image training set;

[0053] Inputting the initial estimated image training set into a pre-trained super-resolution reconstruction model, and comparing the output result of the pre-trained super-resolution reconstruction model with the high-resolution image training set;

[0054] The pre-trained super-resolution reconstruction model is optimized according to the comparison result to generate a trained image super-resolution reconstruction model.

[0055] In a second aspect, the present application provides a multi-frame image super-resolution reconstruction device, which adopts the following technical solution: the device comprises:

[0056] An image acquisition module, used for acquiring multiple frames of low-resolution sequence images according to a preset super-resolution;

[0057] An offset calculation module, used for setting the first frame image in the plurality of frames of low-resolution sequence images as a reference frame image, and calculating the relative offset between the remaining frames of low-resolution sequence images and the reference frame image;

[0058] An image screening module, used for selecting a low-resolution reconstructed image that meets a super-resolution grid condition from the plurality of frames of low-resolution sequence images, wherein the super-resolution grid condition is set according to the preset super-resolution scale and the calculated relative offset;

[0059] A pixel rearrangement module, used for performing pixel rearrangement on the low-resolution reconstructed image to generate an initial estimated image;

[0060] The super-resolution image generation module is used to input the initial estimated image into the trained image super-resolution reconstruction model for processing to generate a final super-resolution image.

[0061] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute any of the multi-frame image super-resolution reconstruction methods described above.

[0062] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute any of the above-mentioned multi-frame image super-resolution reconstruction methods.

[0063] In summary, this application has the following beneficial technical effects:

[0064] 1. By directly rearranging the pixels of images with sub-pixel offsets into high-resolution images, the complementary information between images can be better utilized. Compared with the existing method of fusing multiple frames of image information into one image through alignment and fusion, it is more reliable and interpretable;

[0065] 2. By improving the image resolution according to the super-resolution in the image selection and pixel rearrangement stages, only one image reconstruction model can be trained to be applicable to image reconstruction at multiple super-resolution scales. Compared with the existing method that requires training a corresponding super-resolution model for each super-resolution scale, it has higher flexibility and applicability;

[0066] 3. The present application does not require the use of sophisticated instruments such as micro-scanning equipment to acquire images. When acquiring images, there is no need to control the accuracy and stability of the equipment during the image acquisition stage, which can reduce the difficulty and complexity of operation and reduce costs. At the same time, it can be used for mass-produced products such as mobile phones, cameras, and handheld telescopes. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a flowchart of a multi-frame image super-resolution reconstruction method in an embodiment of the present application;

[0068] Figure 2 This is a flow chart of an implementation method of a multi-frame image super-resolution reconstruction method based on pixel rearrangement in an embodiment of the present application;

[0069] Figure 3 is another implementation flow chart of the multi-frame image super-resolution reconstruction method based on pixel rearrangement in an embodiment of the present application;

[0070] Figure 4 is a schematic diagram of the image screening process in an embodiment of the present application;

[0071] Figure 5 is a schematic diagram of a pixel rearrangement process in an embodiment of the present application;

[0072] Figure 6This is a flow chart of the image super-resolution reconstruction model training process in an embodiment of the present application;

[0073] Figure 7 is a schematic diagram of a multi-frame image super-resolution reconstruction device in an embodiment of the present application;

[0074] Figure 8 It is a schematic diagram used to embody a computer device in an embodiment of the present application.

[0075] Reference numerals: 701, image acquisition module; 702, offset calculation module; 703, image screening module; 704, pixel rearrangement module; 705, super-resolution image generation module. DETAILED DESCRIPTION

[0076] The following combination Figure 1-Figure 8 This application is described in further detail.

[0077] The embodiment of the present application discloses a method for super-resolution reconstruction of multi-frame images, which can be used to super-resolution reconstruct low-resolution images in the fields of medicine, remote sensing, military security, and face recognition to obtain high-resolution images, and provide clear high-quality images for subsequent image recognition, detection and other operations. In the embodiment of the present application, the offset of the acquired multi-frame low-resolution sequence images is calculated, and the offset includes the sub-pixel offset. Then, the images that meet the super-grid conditions are screened according to the calculated sub-pixel offset. For the screened images, the pixels are rearranged according to the positional relationship between the images to obtain the initial estimated image. Finally, the initial estimated image is input into the trained image reconstruction model to eliminate the aliasing effect. The model outputs the final super-resolution image as the reconstruction result. The super-resolution reconstruction algorithm is an algorithm that restores low-resolution images to high-resolution images, and is widely used in the fields of medicine, remote sensing, military security, and face recognition. The enhancement of image resolution by super-resolution reconstruction technology is not just increasing the number of image pixels. What is more important is to increase the detail information that the low-resolution image itself does not have. A single frame image cannot provide such detail information. Through multiple sampling and utilizing the sub-pixel complementary information between multiple frames of images, the high-frequency details in the image can be more accurately estimated and reconstructed.

[0078] The current mainstream super-resolution technologies include micro-scanning technology and deep learning reconstruction algorithms. Micro-scanning technology requires control of tiny scanning steps and precise positioning to ensure that the displacement is less than one pixel. By using the sub-pixel information between two frames, the image information lost due to downsampling can be restored. The existing multi-frame super-resolution network model paradigm consists of three steps: alignment, fusion, and reconstruction. During fusion, weighted fusion is performed based on the correlation of the images.

[0079] Deep learning super-resolution reconstruction algorithms are divided into single-frame reconstruction methods and multi-frame reconstruction methods. The difference between multi-frame reconstruction and single-frame reconstruction is that multi-frame reconstruction not only introduces prior knowledge of external samples of the image, but also needs to consider the time-space correlation between images and use it as prior knowledge for reconstruction. Then, the mapping relationship between high-resolution images and low-resolution images is trained through the network to achieve super-resolution, further improving the accuracy of the reconstructed image and the subjective visual effect of the image. However, the deep learning super-resolution reconstruction model algorithm is highly dependent on the data set. It is necessary to first establish a model and use a large number of data sets to train the model. When faced with scenes where it is difficult to establish a data set in advance, the lack of data sets will lead to insufficient model training and limited generalization ability, which in turn affects the quality and effect of image generation.

[0080] Micro-scanning imaging technology is a technology that samples the same target scene multiple times along a certain path during image acquisition, generates multiple frames of low-resolution images with tiny displacements or even sub-pixel displacements, and then reconstructs them into a frame of high-resolution images through super-resolution. As long as the displacement is less than one pixel, image upsampling can be achieved; using the sub-pixel information between two frames, the image information lost due to downsampling can be reconstructed and restored. According to different micro-displacement generation methods, micro-scanning imaging technology can be divided into controllable and uncontrollable micro-scanning. Uncontrollable scanning technology usually comes from the random jitter of some vibration platforms, such as detectors installed on tanks and helicopters. Due to the vibration of the platform, the images generated have a certain pixel offset from each other, which meets the necessary conditions for super-resolution reconstruction; uncontrollable micro-scanning is easy to implement, does not require too many special devices, and has broad application prospects. However, the accuracy of controllable micro-scanning imaging technology is difficult to control. Due to the limitations of mechanical structure, electronic control and other factors, accuracy control often faces challenges, which may lead to errors or distortions in imaging results. Secondly, micro-scanning equipment is expensive and has poor long-term stability. Micro-scanning technology involves precise mechanical structures, highly sensitive detectors, and complex electronic control systems, which makes the overall manufacturing cost high. At the same time, some micro-scanning equipment, such as torsional scanning grating micromirrors, although they have good optical properties and a large spectral scanning range, may degrade in performance due to fatigue effects after long-term operation, affecting the stability and reliability of imaging. The high cost and complex technical requirements of micro-scanning technology make it difficult to mass-produce.

[0081] In order to help reduce the complexity of operations, while improving the accuracy and reliability of super-resolution reconstruction results and reducing manufacturing costs, the present application provides a multi-frame image super-resolution reconstruction method.

[0082] Reference Figure 1 , the method comprises the following steps:

[0083] S10, acquiring multiple frames of low-resolution sequence images according to a preset super-resolution.

[0084] Specifically, the user can pre-set a desired super-resolution scale, and shoot multiple low-resolution sequence images according to the set super-resolution scale. The user inputs the captured multiple low-resolution sequence images into the system, and the system obtains multiple low-resolution sequence images. The user can shoot multiple low-resolution sequence images by hand-held cameras, mobile phones and other devices, and shoot a set of low-resolution (LR) sequence images containing M frames by slight shaking of the hand; because the hand tremor is highly periodic, the movement of the hand is uniform and random in all angles and directions, and the offset sub-pixel value can have sufficient coverage for super-resolution. Among them, M-frame images are also multi-frame images, M is an integer, when the image is enlarged by 2 times, M can be preferably 14; when the image is enlarged by 4 times, M can be preferably 50; the value of M is determined by the super-resolution scale, and the specific value can be set by the user, and there is no restriction here.

[0085] S20, setting the first frame image in the plurality of frames of low-resolution sequence images as a reference frame image, and calculating the relative offset between the remaining frames of the low-resolution sequence images and the reference frame image.

[0086] Specifically, the first frame of the acquired M-frame LR sequence image is selected as the reference frame image f0, and the optical flow neural network PWCNet is used to calculate the relative offset V between the central part a*b (central part image resolution size) of the other M-1 frame images and the central part a*b of the reference frame image f0 i =(d x , d y ), where d x Indicates the offset in the x direction, that is, the horizontal direction, d y Indicates the offset in the y direction, that is, the vertical direction. The method used in the embodiment of the present application does not require that the difference between the M-1 frame image and the reference frame image f0 is only a sub-pixel offset, such as 0.25pixel, 0.5pixel, etc., but allows the offset between the two frames to include an integer pixel offset, such as 0.25pixel, 1.5pixel, 2.75pixel, etc., that is, only the relative offset V of the two frames is required. i =(d x , d y ) are the sub-pixel offsets U modulo pixel 1 i =( , ) are all less than 1, then the relative offset V i =(d x , d y ) can be expressed as:

[0087]

[0088] in, and Indicates integer pixel offsets in the x and y directions, and does not provide sub-pixel information; and Indicates the sub-pixel offset in the x and y directions.

[0089] S30, selecting a low-resolution reconstructed image that meets a super-resolution grid condition from a plurality of low-resolution sequence images, wherein the super-resolution grid condition is set according to a preset super-resolution scale and a calculated relative offset.

[0090] Specifically, in the M-frame LR sequence image, the pictures that meet the super-resolution grid conditions are screened out according to the super-resolution scale S. The super-resolution scale can also be understood as the magnification factor. When the super-resolution scale is S, the set super-resolution grid is S*S. S*S conditions are set to screen the corresponding pictures. Each grid position corresponds to a different sub-pixel offset condition, and pictures with different sub-pixel information are selected according to the grid position; wherein the super-resolution grid condition is set according to the preset super-resolution scale and the calculated relative offset.

[0091] S40, rearrange pixels of the low-resolution reconstructed image to generate an initial estimated image.

[0092] Specifically, after screening and obtaining a low-resolution reconstructed image that meets the conditions, the pixels in the low-resolution reconstructed image are rearranged. The rearrangement method can refer to the pixel rearrangement method used in the micro-scanning technology. After the pixel rearrangement operation, the initial estimated image H0 is obtained.

[0093] S50, inputting the initial estimated image into the trained image super-resolution reconstruction model for processing to generate a final super-resolution image.

[0094] Specifically, since a certain error threshold is set when selecting images in the super-resolution grid, the generated initial estimated image H0 will have a serious aliasing effect; in order to improve the visual perception quality of the reconstructed image and eliminate the aliasing effect, an image super-resolution reconstruction model is introduced to obtain the final super-resolution image SR. Among them, the image super-resolution reconstruction model is a pre-trained model that can be used directly in practical applications.

[0095] In the present application scheme, there is no need to use precise equipment such as micro-scanning equipment, so there is no need to control the accuracy and stability of the equipment during the image acquisition stage when acquiring images, which can reduce the difficulty and complexity of operation and reduce costs; in addition, by directly rearranging the pixels of images with sub-pixel offsets into high-resolution images, the complementary information between images can be better utilized, thereby improving the accuracy of the imaging results, which is more reliable and interpretable than the existing method of fusing multiple frames of image information into one image through alignment and fusion.

[0096] In one embodiment, considering that the number of images obtained by screening through super-resolution grid conditions may be insufficient, which may affect the subsequent pixel rearrangement process, after selecting the low-resolution reconstructed image that meets the super-resolution grid conditions, the following steps may be performed:

[0097] Determine whether the number of selected low-resolution reconstructed images is less than a preset value of the reconstruction number; if the number of low-resolution reconstructed images is less than the preset value of the reconstruction number, reacquire multiple frames of low-resolution sequence images; calculate the relative offset of the reacquired multiple frames of low-resolution sequence images, update the super-resolution grid condition and reselect low-resolution reconstructed images that meet the super-resolution grid condition; if the number of low-resolution reconstructed images is not less than the preset value of the reconstruction number, execute the step of pixel rearrangement.

[0098] Reference Figure 2 , is a flowchart of an implementation method of a multi-frame image super-resolution reconstruction method based on pixel rearrangement. After acquiring M frames of LR sequence images, set the offset between the reference frame image and the remaining M-1 frame images and the reference frame image, perform conditional screening on the M frames of LR sequence images, obtain low-resolution reconstructed images that meet the conditions, and the number of screened low-resolution reconstructed images is P. The number of low-resolution reconstructed images P is compared with the preset value of the reconstruction number N. If the number of low-resolution reconstructed images P is greater than or equal to the preset value of the reconstruction number N (that is, P is not less than N), the low-resolution reconstructed images are pixel rearranged; if the number of low-resolution reconstructed images P is less than the preset value of the reconstruction number N, then re-acquire M frames of low-resolution sequence images for screening. Among them, when the super-resolution scale is 2, the value of N can be preferably 3; when the super-resolution scale is 4, the value of N can be preferably 12; the value of N can be set by the user according to actual conditions. It should be noted that the value of N should not be too small. If the value of N is too small, there will be too many blank pixels, the amount of interpolation calculation will increase and the reconstruction effect will be poor, or blank pixels will still exist after interpolation.

[0099] It should be noted that in the embodiment of the present application, the number M of LR sequence images initially acquired is generally the maximum threshold value. It can also be understood that the acquired M-frame LR sequence images generally meet the number of super-resolution grid conditions after screening and processing. If the number is not met, the number of acquired images can be re-acquired to continue to increase, that is, the value of M is increased.

[0100] In the present application scheme, considering that the number of images obtained through super-resolution grid condition screening may be insufficient, which may affect the subsequent pixel rearrangement process, therefore, the number of images after screening is judged to ensure the number of images obtained by screening to avoid affecting the subsequent pixel rearrangement process.

[0101] In one embodiment, the number of frames of the multiple low-resolution sequence images is a preset lower threshold, and the preset lower threshold is selected according to the super-resolution scale; considering that the number of images obtained by screening through the super-resolution grid condition may be insufficient, which may affect the subsequent pixel rearrangement process, therefore, after selecting the low-resolution reconstructed image that meets the super-resolution grid condition, the following steps may be performed:

[0102] Determine whether the number of selected low-resolution reconstructed images is less than a preset value of the reconstruction number; if the number of low-resolution reconstructed images is less than the preset value of the reconstruction number, reacquire multiple frames of low-resolution sequence images, wherein the number of frames of the multiple frames of low-resolution sequence images is a preset lower limit threshold plus a preset step number; calculate the relative offset of the reacquired multiple frames of low-resolution sequence images, update the super-resolution grid condition and reselect low-resolution reconstructed images that meet the super-resolution grid condition; cyclically determine whether the number of selected low-resolution reconstructed images is less than a preset value of the reconstruction number; if the number of low-resolution reconstructed images is less than the preset value of the reconstruction number, reacquire multiple frames of low-resolution sequence images until the number of frames of the multiple frames of low-resolution sequence images reaches a preset upper limit threshold.

[0103] Reference Figure 3, is a flow chart of another implementation of a multi-frame image super-resolution reconstruction method based on pixel rearrangement. In the embodiment of the present application, the value of M is determined by a test shot, and M frames of LR sequence images are obtained. The first set M is a preset lower limit threshold, which can also be understood as the number of pictures that need to be taken at least. After that, the offset between the reference frame image and the remaining M-1 frame images and the reference frame image is set, and the M frames of LR sequence images are conditionally screened to obtain low-resolution reconstructed images that meet the conditions. The number of low-resolution reconstructed images screened out is P, and the number of low-resolution reconstructed images P is compared with the preset reconstruction number N. If the number of low-resolution reconstructed images P is greater than or equal to the preset reconstruction number N (that is, P is not less than N), the low-resolution reconstructed images are pixel-rearranged. If the number of low-resolution reconstructed images P is less than the preset reconstruction number N, the step size M' is increased on the basis of the original M, that is, (M+M') frames of low-resolution sequence images are re-acquired for screening. After that, the number of reconstructed images obtained by screening the acquired resolution sequence images is cyclically judged. If the requirement is not met, the step size M' is increased each time until the number of acquired multiple frames of low-resolution sequence images reaches the preset upper limit threshold.

[0104] For example, when the super-resolution scale is 2, assuming that N is 3, the step size M' is set to 4, the minimum threshold Mmin (preset lower threshold) is set to 8, and the maximum threshold Mmax (preset upper threshold) is set to 16; the first test shot acquires 8 (minimum threshold Mmin) low-resolution images by default. If there are less than 3 images that meet the super-resolution grid conditions, 12 images are acquired again for screening. If there are still less than 3 images that meet the conditions, the total number of acquired images is increased by 4, that is, 16 (maximum threshold Mmax) images are acquired again for screening. When the super-resolution scale S is 4, assume that N is 12, the step size M' is set to 10, the minimum threshold Mmin (preset lower threshold) is set to 30, and the maximum threshold Mmax (preset upper threshold) is set to 50; the first test shot will acquire 30 (minimum threshold Mmin) low-resolution images by default. If there are less than 12 pictures that meet the super-resolution grid conditions, the total number of acquired pictures will increase by 10, that is, 40 pictures will be acquired again for screening. If there are still less than 12 pictures that meet the conditions, 50 (maximum threshold Mmin) pictures will be acquired again for screening. Among them, the minimum threshold, maximum threshold, and step size can be set by the photographer himself. Considering that the more pictures are acquired, the greater the calculation amount and the longer the reconstruction time, it is necessary to select the appropriate number of low-resolution pictures in different scenarios.

[0105] In the present application scheme, considering that the number of images obtained through super-resolution grid condition screening may be insufficient, which may affect the subsequent pixel rearrangement process, therefore, the number of images after screening is judged to ensure the number of images obtained by screening to avoid affecting the subsequent pixel rearrangement process.

[0106] In one embodiment, the step of calculating the relative offset between the remaining low-resolution sequence images and the reference frame image may be specifically performed as follows:

[0107] First, the center of the multi-frame low-resolution sequence image is cropped according to a preset size to generate a center area sequence image corresponding to the multi-frame low-resolution sequence image. The size of the cropped center area is a*b. The center cropping is to avoid hand shaking, which causes the image to pitch and rotate, resulting in a significant difference in the optical flow vector between the edge and the center of the image, resulting in a large error when calculating the offset of the entire image. After the center cropping is completed, the optical flow neural network is used to calculate the optical flow vector of each pixel in the center area sequence image; in the embodiment of the present application, the accuracy of the sub-pixel offset calculation is required to be high, so PWCNet, which can estimate the optical flow more accurately at the sub-pixel level, is used. It uses a pyramid structure to extract multi-scale features of the input image, gradually estimates the optical flow from low resolution to high resolution, and performs feature fusion at each resolution level.

[0108] Afterwards, the median of the optical flow vectors of all pixels in the horizontal and vertical directions is calculated based on the calculated optical flow vector of each pixel to generate the median of the horizontal optical flow vector and the median of the vertical optical flow vector; the absolute deviation of the optical flow vector of each pixel in the horizontal and vertical directions from the median of the horizontal optical flow vector and the median of the vertical optical flow vector is calculated respectively, and the absolute deviation is compared with the preset deviation threshold. Pixels whose absolute deviations are greater than the preset deviation threshold are set as singular points. It can also be understood that when the absolute deviation of a pixel is greater than the threshold set in that direction, the optical flow vector of the pixel in the x or y direction is judged to be a singular value. If the x or y optical flow vector of a pixel is a singular value, the pixel is a singular point and the singular point is deleted. After deleting the singular point, the average values ​​of the optical flow vectors in the horizontal and vertical directions of the remaining pixels except the singular point are calculated to generate the horizontal offset d. x and vertical offset d y Among them, the purpose of deleting singular points is to prevent the optical flow vectors of the sky, water surface, etc. in the image from being inconsistent with the optical flow vectors of the buildings in the foreground, etc., which leads to a large error in the calculated offset.

[0109] In the present application scheme, an optical flow neural network PWCNet is used to calculate the optical flow vector of each pixel, so that the optical flow vector of each pixel can be estimated more accurately, and the sub-pixel complementary information between multiple frame images can be used to more accurately estimate and reconstruct the high-frequency details in the image.

[0110] In one embodiment, the relative offset includes a sub-pixel offset, referring to Figure 4 , the step of selecting a low-resolution reconstructed image that meets the super-resolution grid condition can be specifically performed as follows:

[0111] The relative offsets in the x- and y-directions are composed of the integer pixel offsets and sub-pixel offsets in their respective directions. After the pixel offsets are calculated, first, the super-resolution grid is set according to the super-resolution scale S, and the filtering conditions of each grid are set according to the sub-pixel offset and the position of each grid. Each grid position corresponds to a different sub-pixel offset condition. Images with different sub-pixel information are selected according to the grid position. The super-resolution grid size is the square of the super-resolution scale, that is, the size of the super-resolution grid is S*S; after each grid condition is set, a low-resolution image that meets the filtering conditions of each grid is selected according to the filtering conditions of each grid. Among them, the set filtering conditions can be expressed as:

[0112]

[0113] in, and is an integer, represents the Ith grid in the horizontal direction of the super-resolution grid, represents the Ith grid in the vertical direction of the super-resolution grid; S represents the super-resolution scale; Indicates the sub-pixel offset in the horizontal direction, Indicates the sub-pixel offset in the vertical direction; represents the sub-pixel error threshold, Represents the alignment error threshold.

[0114] The error threshold is used to ensure that all images with sub-pixel offsets within the allowable error range can be selected into the super-resolution grid, so as to avoid setting selection conditions that are too strict, resulting in too few or even no images that meet the requirements, which will lead to the inability to generate the initial estimated image H0.

[0115] After filtering out the qualified images, determine whether there are multiple low-resolution images in each grid that meet the filtering conditions of the corresponding grid; due to the setting of the error threshold, there may be multiple images that meet the filtering conditions in one grid, so it is necessary to first determine whether the number of images in each grid is greater than 1; if there are multiple low-resolution images in a single grid that meet the filtering conditions of the corresponding grid, calculate the sub-pixel offset of the low-resolution image ( , ) and the preset offset , select the low-resolution image with the smallest distance; the preset offset is the set optimal offset ( , ), It can be expressed as:

[0116]

[0117] Among them, S represents super-resolution; represents the Ith grid in the horizontal direction of the super-resolution grid, Represents the Ith grid in the vertical direction in the super-resolution grid; Indicates the sub-pixel offset in the horizontal direction, Indicates the sub-pixel offset in the vertical direction.

[0118] For example, when the super-resolution scale is 2, the optimal image offset ( , ) is (0pixel, 0.5pixel), (0.5pixel, 0pixel), (0.5pixel, 0.5pixel); when the super-resolution scale is 4, the optimal offset of the image is (0.25pixel, 0pixel), (0.5pixel, 0pixel), (0.75pixel, 0pixel) (0pixel, 0.25pixel), (0.25pixel, 0.25pixel), and so on.

[0119] Finally, the low-resolution image filtered out from each grid is set as the low-resolution reconstructed image.

[0120] In the solution of the present application, a screening condition is set for each super-resolution grid according to the sub-pixel offset to screen out low-resolution images that meet the requirements, which is helpful for subsequent rearrangement operations of the images.

[0121] In one embodiment, referring to Figure 5 , the step of rearranging pixels of the low-resolution reconstructed image and generating an initial estimated image can be specifically performed as follows:

[0122] After the low-resolution reconstructed image is obtained through screening, the low-resolution reconstructed image is first translated; since the remaining M-1 frames have left-right and up-down offsets relative to the reference frame, that is, the offset of the image is positive or negative, it is necessary to correspond to the label I of the super-resolution grid according to the screened image. image , move each image ( , ), so that the image in the super-resolution grid has only a positive sub-pixel offset with the reference frame image f0, which is convenient for the subsequent pixel rearrangement of the image according to the super-resolution grid position; where the reference frame image f0 is Figure 4 and Figure 5 As shown, the super-grid is labeled I image Indicates the number of the grid. For example, when the super-resolution is 2, refer to Figure 5 As shown, I image =1, 2, 3, 4; when the super-resolution is 4, refer to Figure 4 As shown in the number in the upper right corner of the grid, I image =1, 2,…, 15, 16, and so on.

[0123] The translation mode can be expressed as:

[0124]

[0125] in, Indicates the relative offset in the horizontal direction. Indicates the relative offset in the vertical direction; Express Round down, Express Round down; Point to Round off, I image Indicates the super-resolution grid number. For example, when the image super-resolution scale is 4, I image =2 and the relative offset between the reference frame f0 is Vi = (-2.78pixel, -1.97pixel). After translation, the image should be moved by (3pixel, 2pixel). After translation, the relative offset between the image and the reference frame image f0 becomes (0.22pixel, 0.03pixel). image =13 and the relative offset of the reference frame image f0 is Vi = (0.08pixel, 1.78pixel). After translation, the image should move (0pixel, -1pixel). After the movement, the relative offset of the image and the reference frame image f0 becomes (0.08pixel, 0.78pixel).

[0126] Afterwards, according to the grid number corresponding to the low-resolution reconstructed image in the super-resolution grid, the low-resolution reconstructed image is pixel-rearranged in a preset manner to generate a rearranged image; the rearrangement method can be expressed as:

[0127]

[0128] Among them, r is the row index and c is the column index, which respectively represent the rth point in the x direction and the cth point in the y direction in the super-resolution grid. i is the i-th grid in the super-resolution grid (i=I image ), h is the height of the low-resolution reconstructed image, w is the width of the low-resolution reconstructed image; mod represents the remainder after dividing two numerical expressions; represents the rearranged image, that is, the initial estimated image without interpolation. For example, when the image super-resolution scale is 4, I image =2 image row index r i is 2, column index c i =3; I image =13 image row index r i is 3, column index c i is 0.

[0129] Finally, determine whether there are blank pixels in the rearranged image; if there are blank pixels in the rearranged image, use an interpolation algorithm to fill the blank pixels and generate an initial estimated image H0, where the interpolation algorithm can use a linear interpolation algorithm to replace the blank pixel with the average value of the 8 pixels around the blank pixel, or other methods can be used for interpolation. The interpolation algorithm used here is not limited; if there are no blank pixels in the rearranged image, set the rearranged image as the initial estimated image H0.

[0130] In the present application scheme, by directly rearranging the pixels of an image with sub-pixel offset into a high-resolution image, the complementary information between images can be better utilized, which is more reliable and interpretable than the existing method of fusing multiple frames of image information into one image through alignment and fusion.

[0131] In one embodiment, referring to Figure 6 , the steps of training the image super-resolution reconstruction model can be specifically described as follows:

[0132] A low-resolution image training set and a high-resolution image training set corresponding to the low-resolution image training set are obtained, wherein the obtained training data set can be obtained by shooting, or a data set in an existing database can be directly used. Afterwards, a multi-frame image super-resolution reconstruction method is used to process the low-resolution image training set to generate an initial estimated image training set; the initial estimated image training set is input into a pre-trained super-resolution reconstruction model, and the output result of the pre-trained super-resolution reconstruction model is compared with the high-resolution image training set; the pre-trained super-resolution reconstruction model is optimized according to the comparison result to generate a trained image super-resolution reconstruction model.

[0133] Specifically, the generated initial estimated image H0 and its corresponding high-resolution image Hhr As training samples, H0 and H hr The resolution size is the same as that of the training samples, and the model is trained using the training samples. The trained model can be applied to different super-resolution scales S such as 2, 3, and 4, which effectively solves the problem that the existing super-resolution algorithm needs to train a super-resolution model for each super-resolution scale.

[0134] In the present application scheme, by improving the image resolution according to the super-resolution in the image selection and pixel rearrangement stages, only one image reconstruction model can be trained to be applicable to image reconstruction of multiple super-resolution scales. Compared with the existing method that requires training a corresponding super-resolution model for each super-resolution scale, it has higher flexibility and applicability.

[0135] Figure 1 FIG. 1 is a flow chart of a multi-frame image super-resolution reconstruction method in one embodiment. It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified in this document, there is no strict order restriction for the execution of the steps, and the steps may be executed in other orders; and Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0136] Based on the above method, an embodiment of the present application also discloses a multi-frame image super-resolution reconstruction device.

[0137] Reference Figure 7 , the device includes the following modules:

[0138] The image acquisition module 701 is used to acquire multiple frames of low-resolution sequence images according to a preset super-resolution method;

[0139] The offset calculation module 702 is used to set the first frame image in the multiple frames of low-resolution sequence images as the reference frame image, and calculate the relative offset between the remaining frames of low-resolution sequence images and the reference frame image;

[0140] An image screening module 703 is used to select a low-resolution reconstructed image that meets a super-resolution grid condition from a plurality of low-resolution sequence images, where the super-resolution grid condition is set according to a preset super-resolution scale and a calculated relative offset;

[0141] A pixel rearrangement module 704 is used to rearrange pixels of the low-resolution reconstructed image to generate an initial estimated image;

[0142] The super-resolution image generation module 705 is used to input the initial estimated image into the trained image super-resolution reconstruction model for processing to generate a final super-resolution image.

[0143] In one embodiment, the image screening module 703 is also used to determine whether the number of selected low-resolution reconstructed images is less than a preset value of the reconstruction number; if the number of low-resolution reconstructed images is less than the preset value of the reconstruction number, reacquire multiple frames of low-resolution sequence images; calculate the relative offset of the reacquired multiple frames of low-resolution sequence images, update the super-resolution grid condition and reselect the low-resolution reconstructed image that meets the super-resolution grid condition.

[0144] In one embodiment, the image screening module 703 is also used to determine whether the number of selected low-resolution reconstructed images is less than a preset value of the reconstruction number; if the number of low-resolution reconstructed images is less than the preset value of the reconstruction number, then reacquire multiple frames of low-resolution sequence images, wherein the number of frames of the multiple frames of low-resolution sequence images is a preset lower limit threshold plus a preset step number; calculate the relative offset of the reacquired multiple frames of low-resolution sequence images, update the super-resolution grid condition and reselect low-resolution reconstructed images that meet the super-resolution grid condition; cyclically determine whether the number of selected low-resolution reconstructed images is less than the preset value of the reconstruction number; if the number of low-resolution reconstructed images is less than the preset value of the reconstruction number, then reacquire multiple frames of low-resolution sequence images until the number of frames of the multiple frames of low-resolution sequence images reaches the preset upper limit threshold.

[0145] In one embodiment, the offset calculation module 702 is specifically used to perform center cropping on multiple frames of low-resolution sequence images according to a preset size to generate a central area sequence image corresponding to the multiple frames of low-resolution sequence images; use an optical flow neural network to calculate the optical flow vector of each pixel in the central area sequence image; calculate the median of the optical flow vectors of all pixels in the horizontal and vertical directions based on the calculated optical flow vector of each pixel, and generate the median of the horizontal optical flow vector and the median of the vertical optical flow vector; calculate the absolute deviation of the optical flow vector of each pixel in the horizontal and vertical directions from the median of the horizontal optical flow vector and the median of the vertical optical flow vector, respectively, and compare the absolute deviation with a preset deviation threshold; set the pixel point whose absolute deviation is greater than the preset deviation threshold as a singular point, and delete the singular point; calculate the average value of the optical flow vectors of the remaining pixels except the singular point in the horizontal and vertical directions to generate the horizontal offset and the vertical offset.

[0146] In one embodiment, the image screening module 703 is further used to set a super-resolution grid according to the super-resolution scale, and set the screening condition of each grid according to the sub-pixel offset and the position of each grid, and the super-resolution grid size is the square of the super-resolution scale; select a low-resolution image that meets the screening condition of each grid according to the screening condition of each grid; determine whether there are multiple low-resolution images in each grid that meet the screening condition of the corresponding grid; if there are multiple low-resolution images in a single grid that meet the screening condition of the corresponding grid, calculate the distance between the sub-pixel offset of the low-resolution image and the preset offset, and select the low-resolution image with the smallest distance; set the low-resolution image screened out of each grid as the low-resolution reconstructed image;

[0147] Filters include:

[0148]

[0149] in, and is an integer, represents the Ith grid in the horizontal direction of the super-resolution grid, represents the Ith grid in the vertical direction of the super-resolution grid; S represents the super-resolution scale; Indicates the sub-pixel offset in the horizontal direction, Indicates the sub-pixel offset in the vertical direction; represents the sub-pixel error threshold, Represents the alignment error threshold.

[0150] In one embodiment, the pixel rearrangement module 704 is specifically used to translate the low-resolution reconstructed image; rearrange the pixels of the low-resolution reconstructed image in a preset manner according to the grid number corresponding to the low-resolution reconstructed image in the super-resolution grid to generate a rearranged image; determine whether there are blank pixels in the rearranged image; if there are blank pixels in the rearranged image, fill the blank pixels with an interpolation algorithm and generate an initial estimated image; if there are no blank pixels in the rearranged image, set the rearranged image as the initial estimated image;

[0151] The translation modes include:

[0152]

[0153] in, Indicates the relative offset in the horizontal direction. Indicates the relative offset in the vertical direction; Express Round down, Express Round down; Point to Round off, Iimage Indicates the super-resolution grid number;

[0154] Pixel rearrangement methods include:

[0155]

[0156] Where r represents the rth grid in the horizontal direction of the super-resolution grid, and c represents the cth grid in the vertical direction of the super-resolution grid; LR i represents the low-resolution reconstructed image corresponding to the i-th grid in the super-resolution grid; h represents the height of the low-resolution reconstructed image; w represents the width of the low-resolution reconstructed image; mod represents the modulo operation; Rearranges the image.

[0157] In one embodiment, the super-resolution image generation module 705 is also used to obtain a low-resolution image training set and a high-resolution image training set corresponding to the low-resolution image training set; use a multi-frame image super-resolution reconstruction method to process the low-resolution image training set to generate an initial estimated image training set; input the initial estimated image training set into a pre-trained super-resolution reconstruction model, and compare the output result of the pre-trained super-resolution reconstruction model with the high-resolution image training set; optimize the pre-trained super-resolution reconstruction model according to the comparison result to generate a trained image super-resolution reconstruction model.

[0158] The multi-frame image super-resolution reconstruction device provided in the embodiment of the present application can be applied to the multi-frame image super-resolution reconstruction method provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0159] It should be noted that: when the multi-frame image super-resolution reconstruction device provided in the embodiment of the present application performs multi-frame image super-resolution reconstruction, only the division of the above-mentioned functional modules / functional units is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the multi-frame image super-resolution reconstruction device is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the multi-frame image super-resolution reconstruction method provided in the above method embodiment and the implementation method of the multi-frame image super-resolution reconstruction device provided in this embodiment belong to the same concept. The specific implementation process of the multi-frame image super-resolution reconstruction device provided in this embodiment is detailed in the above method embodiment, which will not be repeated here.

[0160] The embodiment of the present application also discloses a computer device.

[0161] Specifically, if Figure 8As shown, the computer device can be a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. The processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above chips.

[0162] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0163] The embodiment of the present application also discloses a computer-readable storage medium.

[0164] Specifically, the computer-readable storage medium is used to store a computer program, and when the computer program is executed by the processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art can understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it may include the process of the implementation of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0165] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A multi-frame image super-resolution reconstruction method, characterized in that: The method comprises: Acquire multiple frames of low-resolution sequence images according to the preset super-resolution; The first frame image in the plurality of frames of low-resolution sequence images is set as a reference frame image, and the relative offset between the remaining frames of low-resolution sequence images and the reference frame image is calculated; Selecting a low-resolution reconstructed image that meets a super-resolution grid condition from among the multiple frames of low-resolution sequence images, wherein the super-resolution grid condition is set according to the preset super-resolution scale and the calculated relative offset; Rearranging pixels of the low-resolution reconstructed image to generate an initial estimated image; Inputting the initial estimated image into the trained image super-resolution reconstruction model for processing to generate a final super-resolution image; The step of rearranging pixels of the low-resolution reconstructed image to generate an initial estimated image comprises: translating the low-resolution reconstructed image; According to the grid number corresponding to the low-resolution reconstructed image in the super-resolution grid, rearrange the pixels of the low-resolution reconstructed image in a preset manner to generate a rearranged image; Determining whether there are blank pixels in the rearranged image; If there are blank pixels in the rearranged image, an interpolation algorithm is used to fill the blank pixels and generate the initial estimated image; If there are no blank pixels in the rearranged image, setting the rearranged image as the initial estimated image; The translation modes include: in, Indicates the relative offset in the horizontal direction. Indicates the relative offset in the vertical direction; Express Round down, Express Round down; Point to Round off, I image Indicates the super-resolution grid number; The pixel rearrangement method includes: Wherein, r represents the rth grid in the horizontal direction of the super-resolution grid, and c represents the cth grid in the vertical direction of the super-resolution grid; LR i represents the low-resolution reconstructed image corresponding to the i-th grid in the super-resolution grid; h represents the height of the low-resolution reconstructed image; w represents the width of the low-resolution reconstructed image; mod represents the modulo operation; represents the rearranged image.

2. The method according to claim 1, characterized in that: After selecting the low-resolution reconstructed image that meets the super-resolution grid condition, the method further includes: Determining whether the number of the selected low-resolution reconstructed images is less than a preset reconstruction number value; If the number of the low-resolution reconstructed images is less than a preset reconstruction number, reacquiring multiple frames of low-resolution sequence images; The relative offset of the re-acquired multiple frames of low-resolution sequence images is calculated, the super-resolution grid condition is updated, and the low-resolution reconstructed image that meets the super-resolution grid condition is reselected.

3. The method according to claim 1, characterized in that: The number of frames of the plurality of low-resolution sequence images is a preset lower limit threshold, and the preset lower limit threshold is selected according to the super-resolution; After selecting the low-resolution reconstructed image that meets the super-resolution grid condition, the method further includes: Determining whether the number of the selected low-resolution reconstructed images is less than a preset reconstruction number value; If the number of the low-resolution reconstructed images is less than a preset reconstruction number, reacquiring multiple frames of low-resolution sequence images, wherein the number of frames of the multiple frames of low-resolution sequence images is a preset step number increased on the basis of a preset lower limit threshold; Calculating the relative offset of the re-acquired multiple frames of low-resolution sequence images, updating the super-resolution grid condition and reselecting the low-resolution reconstructed image that meets the super-resolution grid condition; Circularly determining whether the number of the selected low-resolution reconstructed images is less than a preset reconstruction number value; If the number of the low-resolution reconstructed images is less than a preset reconstruction number, multiple frames of low-resolution sequence images are reacquired until the number of frames of the multiple frames of low-resolution sequence images reaches a preset upper limit threshold.

4. The method according to claim 1, characterized in that: The calculating of the relative offset between the remaining low-resolution sequence images and the reference frame image comprises: Performing center cropping on the plurality of frames of low-resolution sequence images according to a preset size to generate a center area sequence image corresponding to the plurality of frames of low-resolution sequence images; Using an optical flow neural network to calculate the optical flow vector of each pixel in the central area sequence image; Calculate the median of the optical flow vectors of all pixels in the horizontal and vertical directions according to the calculated optical flow vector of each pixel, and generate the median of the horizontal optical flow vector and the median of the vertical optical flow vector; Calculating the absolute deviation of the optical flow vector of each pixel from the median of the horizontal optical flow vector and the median of the vertical optical flow vector in the horizontal direction and the vertical direction respectively, and comparing the absolute deviation with a preset deviation threshold; Set the pixel points whose absolute deviation is greater than the preset deviation threshold as singular points, and delete the singular points; Calculate the average optical flow vectors of the remaining pixels except the singular points in the horizontal and vertical directions to generate the horizontal offset and vertical offset.

5. The method according to claim 1, characterized in that: The relative offset includes a sub-pixel offset, and the selecting of a low-resolution reconstructed image that meets the super-resolution grid condition includes: Setting a super-resolution grid according to the super-resolution scale, and setting a screening condition for each grid according to the sub-pixel offset and the position of each grid, wherein the size of the super-resolution grid is the square of the super-resolution scale; Selecting a low-resolution image that meets the filtering conditions of each grid according to the filtering conditions of each grid; Determine whether there are multiple low-resolution images in each grid that meet the screening conditions of the corresponding grid; If there are multiple low-resolution images in a single grid that meet the screening conditions of the corresponding grid, the distance between the sub-pixel offset of the low-resolution image and the preset offset is calculated, and the low-resolution image with the smallest distance is selected; The low-resolution image screened out from each grid is set as the low-resolution reconstructed image; The screening conditions include: in, and is an integer, represents the Ith grid in the horizontal direction of the super-resolution grid, represents the Ith grid in the vertical direction of the super-resolution grid; S represents the super-resolution scale; Indicates the sub-pixel offset in the horizontal direction, Indicates the sub-pixel offset in the vertical direction; represents the sub-pixel error threshold, Represents the alignment error threshold.

6. The method according to claim 1, characterized in that: The training method of the image super-resolution reconstruction model includes: Acquire a low-resolution image training set and a high-resolution image training set corresponding to the low-resolution image training set; Using the multi-frame image super-resolution reconstruction method to process the low-resolution image training set to generate an initial estimation image training set; Inputting the initial estimated image training set into a pre-trained super-resolution reconstruction model, and comparing the output result of the pre-trained super-resolution reconstruction model with the high-resolution image training set; The pre-trained super-resolution reconstruction model is optimized according to the comparison result to generate a trained image super-resolution reconstruction model.

7. A multi-frame image super-resolution reconstruction device, applied to the multi-frame image super-resolution reconstruction method according to any one of claims 1 to 6, characterized in that: The device comprises: An image acquisition module, used for acquiring multiple frames of low-resolution sequence images according to a preset super-resolution; An offset calculation module, used for setting the first frame image in the plurality of frames of low-resolution sequence images as a reference frame image, and calculating the relative offset between the remaining frames of low-resolution sequence images and the reference frame image; An image screening module, used for selecting a low-resolution reconstructed image that meets a super-resolution grid condition from the plurality of frames of low-resolution sequence images, wherein the super-resolution grid condition is set according to the preset super-resolution scale and the calculated relative offset; A pixel rearrangement module, used for performing pixel rearrangement on the low-resolution reconstructed image to generate an initial estimated image; The super-resolution image generation module is used to input the initial estimated image into the trained image super-resolution reconstruction model for processing to generate a final super-resolution image.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6.

Images