A method and system for multi-frame subpixel super-resolution image reconstruction

Abstract

This invention relates to the field of image reconstruction technology and discloses a method and system for multi-frame subpixel super-resolution image reconstruction. The method includes: acquiring a multi-frame low-resolution image sequence and corresponding subpixel displacement vectors; the low-resolution image sequence is obtained by generating a stacked spot array from a spatial radio frequency marker microscopy module to scan the sample under test, and performing multiple displacement samplings on the sample; each spot in the stacked spot array has a different frequency; the multi-frame low-resolution image sequence is upsampled to a high-resolution grid, and subpixel alignment is performed using the subpixel displacement vectors to obtain an aligned high-resolution grid image sequence, which is then input into a pre-constructed image super-resolution neural network model. The model sequentially passes through a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module to output a reconstructed high-resolution image. This scheme achieves good accuracy in realizing image reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, and specifically to a method and system for multi-frame subpixel super-resolution image reconstruction. Background Technology

[0002] Currently, imaging technology based on spatial radio frequency markers has overcome the speed limitations of traditional point scanning by utilizing radio frequency multiplexing, achieving high-speed image acquisition. Building on this, microscopic imaging schemes based on the stacking of polychromatic or monochromatic light spots have emerged, aiming to transcend the limitations of light spot size.

[0003] However, after scanning the sample using stacked light spots, the acquired image information is essentially a sequence of images with known subpixel displacements determined by hardware timing. Related technologies have failed to effectively utilize this prior displacement information to achieve high-quality super-resolution reconstruction.

[0004] Therefore, how to reconstruct a high-resolution image based on this image information has become an urgent problem to be solved. Summary of the Invention

[0005] This invention provides a method and system for reconstructing multi-frame subpixel super-resolution images to solve the problem of how to reconstruct high-resolution images based on image information obtained after scanning with stacked light spots.

[0006] In a first aspect, the present invention provides a method for reconstructing multi-frame subpixel super-resolution images, the method comprising: A multi-frame low-resolution image sequence is acquired, along with sub-pixel displacement vectors corresponding to each of the multi-frame low-resolution image sequences. The low-resolution image sequence is obtained by generating a stacked spot array from a spatial radio frequency marker microscopy imaging module to scan the sample under test and performing multiple displacement samplings on the sample under test. Each spot in the stacked spot array corresponds to a different frequency. The multi-frame low-resolution image sequence is upsampled to a high-resolution grid, and sub-pixel alignment is performed using the sub-pixel displacement vector to obtain an aligned high-resolution grid image sequence. The aligned high-resolution grid image sequence is input into a pre-constructed image super-resolution neural network model, and then processed sequentially by a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module to output a reconstructed high-resolution image.

[0007] In one optional implementation, the sub-pixel alignment using the sub-pixel displacement vector includes: Each low-resolution image in the multi-frame low-resolution image sequence is bilinearly upsampled to a high-resolution grid. Based on the subpixel displacement vector, a bilinear grid sampling function is used to map each upsampled low-resolution image frame to a high-resolution grid, resulting in an aligned high-resolution grid image for each frame.

[0008] In one optional implementation, the process of sequentially passing through a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module to output a reconstructed high-resolution image includes: The stacked spot image feature extraction module extracts features from each frame of the aligned high-resolution grid image sequence by sequentially using the first convolutional layer, the first activation function, and the second convolutional layer to obtain a feature map for each frame. The stacked spot image feature fusion module performs cascaded operations on the feature map channels of each frame to obtain a merged feature map. The merged feature map is then fused sequentially using the third convolutional layer, the second activation function, the fourth convolutional layer, and the third activation function to obtain a fused feature map. The image reconstruction module uses a fifth convolutional layer to map the fused feature map back to the image space, outputting a single-channel high-resolution image.

[0009] In an alternative implementation, after outputting the reconstructed high-resolution image, the method further includes: The reconstructed high-resolution image is subjected to a no-reference quality assessment to obtain the assessment results; If the evaluation result does not meet the preset requirements, a reacquisition control command is generated to control the spatial radio frequency marker microscopy imaging module to re-execute the step of acquiring multiple frames of low-resolution image sequences.

[0010] In one optional implementation, the method further includes: Construct a training dataset; the training dataset contains multiple sets of training samples, each set of training samples includes a multi-frame low-resolution sample image sequence with sub-pixel displacement, sub-pixel displacement vectors corresponding to the multi-frame low-resolution sample image sequence respectively, and a single-frame high-resolution image patch as a supervised learning label; An initial image super-resolution neural network model is constructed; the initial image super-resolution neural network model includes a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module; Using the training dataset, the initial image super-resolution neural network model is trained by optimizing the loss function to obtain the image super-resolution neural network model.

[0011] In one alternative implementation, constructing the training dataset includes: Read the original high-resolution image and randomly crop it to obtain a single frame of high-resolution image patch; The single-frame high-resolution image block is downsampled using bicubic interpolation to obtain a low-resolution reference frame; For each non-reference frame, a sub-pixel displacement vector is randomly generated relative to the low-resolution reference frame. An affine transformation is then used to perform a displacement operation on the original high-resolution image corresponding to the sub-pixel displacement vector to generate a displacementd high-resolution image. The high-resolution image after displacement is cropped in the same way as the single-frame high-resolution image block, and then downsampled to obtain a low-resolution sample image; The low-resolution sample images corresponding to each non-reference frame are combined with the low-resolution reference frame to form a low-resolution sample image sequence; The low-resolution sample image sequence, the sub-pixel displacement vectors corresponding to each non-reference frame, and the single-frame high-resolution image block constitute a set of training samples.

[0012] In a second aspect, the present invention provides a multi-frame subpixel super-resolution image reconstruction system, the system comprising: A spatial radio frequency labeling microscopy imaging module is used to acquire a multi-frame low-resolution image sequence and sub-pixel displacement vectors corresponding to the multi-frame low-resolution image sequence; the low-resolution image sequence is obtained by the spatial radio frequency labeling microscopy imaging module generating a stacked spot array to scan the sample under test and performing multiple displacement samplings on the sample under test; each spot in the stacked spot array corresponds to a different frequency. The image super-resolution reconstruction module is used to execute the multi-frame sub-pixel super-resolution image reconstruction method of the first aspect or any corresponding embodiment described above, and to convert the multi-frame low-resolution image sequence and the sub-pixel displacement vectors corresponding to the multi-frame low-resolution image sequence into reconstructed high-resolution images. The image evaluation and reacquisition module is used to evaluate the reconstructed high-resolution image, and when the evaluation fails, it controls the spatial radio frequency marker microscopy module to re-execute the step of acquiring a multi-frame low-resolution image sequence.

[0013] Thirdly, the present invention provides a multi-frame subpixel super-resolution image reconstruction apparatus, the apparatus comprising: The image acquisition module is used to acquire a multi-frame low-resolution image sequence and sub-pixel displacement vectors corresponding to the multi-frame low-resolution image sequence; the low-resolution image sequence is obtained by generating a stacked spot array by a spatial radio frequency marker microscopy imaging module to scan the sample under test and performing multiple displacement samplings on the sample under test; each spot in the stacked spot array corresponds to a different frequency. The alignment module is used to upsample the multi-frame low-resolution image sequence to a high-resolution grid and perform sub-pixel alignment using the sub-pixel displacement vector to obtain an aligned high-resolution grid image sequence. The image reconstruction module is used to input the aligned high-resolution grid image sequence into a pre-constructed image super-resolution neural network model, and then process it sequentially through the stacked spot image feature extraction module, the stacked spot image feature fusion module, and the image reconstruction module to output the reconstructed high-resolution image.

[0014] Fourthly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the multi-frame sub-pixel super-resolution image reconstruction method described in the first aspect or any corresponding embodiment thereof.

[0015] Fifthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the multi-frame sub-pixel super-resolution image reconstruction method described in the first aspect or any corresponding embodiment thereof.

[0016] In a sixth aspect, the present invention provides a computer program product, including computer instructions for causing a computer to execute the multi-frame subpixel super-resolution image reconstruction method of the first aspect or any corresponding embodiment thereof.

[0017] The technical solution provided by this invention may include the following beneficial effects: The multi-frame subpixel super-resolution image reconstruction method provided by this invention, after acquiring a multi-frame low-resolution image sequence and the corresponding subpixel displacement vectors, directly upsamples and aligns the multi-frame low-resolution image sequence using the corresponding subpixel displacement vectors. This eliminates the complex motion estimation process in related technologies, fundamentally avoiding the problem of motion estimation error accumulation caused by the similarity of stacked spot image features, and significantly reducing the computational complexity of alignment. By processing the aligned high-resolution grid image sequence sequentially through the stacked spot image feature extraction module and the stacked spot image feature fusion module, it can effectively mine and integrate the complementary high-frequency details brought about by the small displacements between the multi-frame aligned images. Finally, the image reconstruction module maps and outputs the result, thereby achieving accurate reconstruction from stacked spot image features to a high-quality super-resolution image without changing the original grayscale distribution. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the first process of a multi-frame subpixel super-resolution image reconstruction method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the second process of the multi-frame subpixel super-resolution image reconstruction method according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the generation of the training dataset according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an image super-resolution neural network model according to an embodiment of the present invention; Figure 5 This is a schematic diagram illustrating the reconstruction effect according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a multi-frame subpixel super-resolution image reconstruction system according to an embodiment of the present invention; Figure 7 This is a structural block diagram of a multi-frame subpixel super-resolution image reconstruction apparatus according to an embodiment of the present invention; Figure 8 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0022] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0023] Currently, imaging technology based on spatial radio frequency markers has overcome the speed limitations of traditional point scanning by utilizing radio frequency multiplexing, making it highly effective for high-speed single-pixel imaging in biomedical applications and enabling high-speed image acquisition. Building upon this, microscopic imaging schemes based on the stacking of polychromatic or monochromatic light spots have emerged, aiming to transcend the limitations of light spot size.

[0024] However, after scanning the sample using stacked light spots, the acquired image information is essentially a sequence of images with known subpixel displacements determined by hardware timing. Related technologies have failed to effectively utilize this prior displacement information to achieve high-quality super-resolution reconstruction.

[0025] Therefore, how to reconstruct a high-resolution image based on this image information has become an urgent problem to be solved.

[0026] According to an embodiment of the present invention, a method for reconstructing multi-frame subpixel super-resolution images is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] This embodiment provides a multi-frame subpixel super-resolution image reconstruction method, which is applied to a multi-frame subpixel super-resolution image reconstruction system. Figure 1 This is a flowchart of a multi-frame subpixel super-resolution image reconstruction method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain a multi-frame low-resolution image sequence and a sub-pixel displacement vector corresponding to the multi-frame low-resolution image sequence.

[0028] This low-resolution image sequence refers to a sequence of multiple consecutive frames of images that fail to capture the microscopic details of the sample due to limitations in the physical size of the light spot. This low-resolution image sequence is obtained by a spatial radio frequency marker microscopy module that generates a stacked light spot array to scan the sample and performs multiple displacement samplings on the sample. Each light spot in the stacked light spot array corresponds to a different frequency, generated based on radio frequency multiplexing technology. By making the frequencies of each light spot in the stacked light spot array different, the optical signals at different spatial positions are encoded and modulated, thus overcoming the speed bottleneck of single-point scanning in related technologies. Simultaneously, by manipulating the hardware to perform multiple displacement samplings on the sample, extremely small (sub-pixel level) relative positional movements occur between adjacent samplings and are recorded as sub-pixel displacement vectors. These sub-pixel displacement vectors are determined by at least one of the following parameters of the spatial radio frequency marker microscopy module: scanning orientation, scanning step size, light spot overlap rate, and displacement stage control parameters. The number of frames, magnification, or number of fusion channels can be set according to the light spot overlap rate or scanning redundancy. Among them, the scanning orientation is used to indicate the scanning path or direction when the space radio frequency marker microscopy imaging module scans the sample under test; the scanning step distance is used to indicate the set distance of a single step of the motion axis when the space radio frequency marker microscopy imaging module scans the sample under test; the spot overlap rate is used to indicate the proportion of the overlapping area between adjacent spots in the stacked spot array to the total area of ​​a single spot; the sample under test is placed on the displacement stage, and the displacement stage control parameters are used to indicate the parameters for the displacement stage to move the sample under test, such as the direction of movement, the distance of movement, and the speed of movement.

[0029] In other words, the subpixel displacement vector in this embodiment is not obtained by inversion of image content through a registration algorithm, but is jointly or assisted in determining by the scanning path, scanning direction, displacement control parameters, spot array arrangement, and spot overlap rate of the spatial radio frequency marker microscopy imaging module. Since the displacement information can be obtained during the acquisition stage, motion estimation instability caused by texture repetition, low contrast, or structural similarity in stacked spot images can be avoided.

[0030] Therefore, this step not only obtains the resolution-limited image sequence, but also simultaneously obtains the absolutely accurate sub-pixel displacement vector, which is strictly guaranteed by hardware timing, providing a data foundation for skipping the complex motion estimation algorithms in related technologies.

[0031] Step S102: Upsample the multi-frame low-resolution image sequence to a high-resolution grid, and use the sub-pixel displacement vector to perform sub-pixel alignment to obtain the aligned high-resolution grid image sequence.

[0032] Because there are slight sub-pixel misalignments between the multiple low-resolution image frames obtained in step S101, directly superimposing these low-resolution images would result in a blurred image. Therefore, this step first performs an upsampling operation on the multiple low-resolution image sequence to a high-resolution grid, that is, using an interpolation algorithm to enlarge the physical size of the low-resolution images to the coordinate space of the target high-resolution image. Based on this, sub-pixel alignment is performed directly using the known sub-pixel displacement vectors corresponding to the multiple low-resolution image sequence obtained in step S101. That is, using the known precise displacement values, the misaligned low-resolution images are reverse-translated in the enlarged high-resolution coordinate grid, so that the low-resolution images achieve precise sub-pixel overlap in spatial position. This achieves rigid mapping entirely based on the known displacement values ​​provided by the hardware, avoiding the step of guessing and estimating the motion trajectory between images through software algorithms in related super-resolution methods. This eliminates the accumulation of motion estimation errors and greatly improves the absolute accuracy of alignment and the stability of subsequent image reconstruction.

[0033] Step S103: The aligned high-resolution grid image sequence is input into the pre-constructed image super-resolution neural network model, and is processed sequentially by the stacked spot image feature extraction module, the stacked spot image feature fusion module, and the image reconstruction module to output the reconstructed high-resolution image.

[0034] After aligning the high-resolution grid image sequence in step S102, the multiple low-resolution images are spatially overlapped. However, the resulting high-resolution grid image sequence is only physically enlarged to the high-resolution grid size; its own resolution remains low. At this point, the aligned high-resolution grid image sequence is organized and input according to the reference frame coordinate system, thereby enhancing consistency with the scanning trajectory of the spatial radio frequency marker microscopy module. Then, the high-resolution grid image sequence is fed into a pre-trained image super-resolution neural network model, which processes the data sequentially through three collaborative modules. Specifically, firstly, the stacked spot image feature extraction module identifies and extracts the spatial features left by the scanning of specific spot from the aligned single-frame high-resolution grid images. Next, since there is misalignment between the high-resolution grid images, each high-resolution grid image contains detailed features from different angles corresponding to the original low-resolution image. Through the stacked spot image feature fusion module, the different features corresponding to each high-resolution grid image are complemented and fused across frames, converging the scattered high-frequency details while suppressing the noise introduced by the scanning process, and obtaining the fused abstract feature data. Finally, through the image reconstruction module, the fused abstract feature data is remapped back to the real image pixel space, and the reconstructed high-resolution image is finally output, breaking through the physical limitations of the original spot size and restoring the clear microstructure of the sample under test.

[0035] In space radio frequency marker microscopy, the overlap rate of adjacent light spots affects the information redundancy and complementarity distribution among low-resolution image sequences obtained from multiple scans. This embodiment focuses on multi-frame low-resolution image sequences formed by stacked light spot scanning in space radio frequency marker microscopy. Utilizing the sub-pixel displacement relationships determined by the scanning prior, it collaboratively designs the input organization, alignment, and cross-frame fusion methods of the image super-resolution neural network model. By introducing the sub-pixel displacement vector corresponding to the scanning parameters into the high-resolution grid alignment process, and performing cross-frame feature extraction and fusion on the aligned multi-frame high-resolution grid image sequences, the complementary high-frequency information brought by the overlapping sampling of stacked light spots can be utilized in a targeted manner, thereby improving the stability and detail recovery capability of super-resolution reconstruction in this type of imaging scenario.

[0036] The multi-frame subpixel super-resolution image reconstruction method provided in this embodiment, after acquiring a multi-frame low-resolution image sequence and the corresponding subpixel displacement vectors, directly upsamples and aligns the multi-frame low-resolution image sequence using the corresponding subpixel displacement vectors. This eliminates the complex motion estimation process in related technologies, fundamentally avoiding the problem of motion estimation error accumulation caused by the similarity of stacked spot image features, and significantly reducing the computational complexity of alignment. By processing the aligned high-resolution grid image sequence sequentially through the stacked spot image feature extraction module and the stacked spot image feature fusion module, it can effectively mine and integrate the complementary high-frequency details brought about by the small displacements between the multi-frame aligned images. Finally, the image reconstruction module maps and outputs the result, thereby achieving accurate reconstruction from stacked spot image features to a high-quality super-resolution image without changing the original grayscale distribution.

[0037] This embodiment provides a multi-frame subpixel super-resolution image reconstruction method, which is applied to a multi-frame subpixel super-resolution image reconstruction system. Figure 2 This is a flowchart of a multi-frame subpixel super-resolution image reconstruction method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps: Step S201: Obtain the image super-resolution neural network model.

[0038] An initial image super-resolution neural network model to be trained is obtained in advance, and the initial image super-resolution neural network model is used to obtain an image super-resolution neural network model for subsequent steps through the training dataset.

[0039] Specifically, firstly, a training dataset is constructed, containing multiple sets of training samples. The displacement distribution range in the training samples matches the achievable displacement range of the actual spatial radio frequency marker microscopy imaging module. Each set of training samples includes a sequence of multi-frame low-resolution sample images with sub-pixel displacement, sub-pixel displacement vectors corresponding to each multi-frame low-resolution sample image sequence, and a single-frame high-resolution image patch as a supervised learning label. Next, an initial image super-resolution neural network model is constructed. This initial model includes a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module. In other words, the initial model and the trained model have the same structure but different model parameters. Finally, using this training dataset, the initial model is trained by optimizing the loss function, enabling the network to learn the mapping from low-resolution image sequences to high-resolution images and continuously updating the optimizer parameters to obtain the final image super-resolution neural network model, which is then deployed.

[0040] For example, during the training phase, the image super-resolution neural network model is trained end-to-end by optimizing the supervised loss function. This supervised loss function can be the L1 loss function, or a combined loss function including both L1 loss and robust error terms, to simultaneously consider reconstruction accuracy and the ability to suppress outlier errors. Model parameter updates can employ the Adam (Adaptive Moment Estimation) optimizer, the AdamW (Adam combined with Weight Decay) optimizer, or other adaptive optimization algorithms; the learning rate can employ a pre-defined annealing strategy, a cosine annealing strategy, or an adaptive decay strategy based on validation set performance; the optimal model can be selected based on the validation set loss, the PSNR (Peak Signal-to-Noise Ratio) metric, or a combination thereof.

[0041] In one alternative implementation, the combined loss It can be represented as:

[0042] in, and These are non-negative weighting coefficients. Represents the L1 loss term. This represents the robust error term.

[0043] Optionally, a stacked spot array is generated using a spatial radio frequency marker microscopy imaging module. Multiple displacement sampling is performed on the test sample, which has known high-resolution images (single-frame high-resolution image patches). Each set of sampled images constitutes a set of training data. By adjusting the overlap rate between the spots in the stacked spot array, training data acquisition with different super-resolution factors can be achieved. Simultaneously, existing high-resolution sample image datasets can be downsampled to obtain low-resolution sample image sequences with known sub-pixel displacement vectors to expand the dataset. This existing high-resolution sample image dataset supports multiple formats, such as png, jpg, jpeg, bmp, tif, and tiff. The high-resolution sample images in this dataset can be 16-bit color images, and the downsampled low-resolution sample images can be 8-bit grayscale images to maintain format consistency. It should be noted that the test sample in the training phase (step S201) and the test sample in the inference phase (steps S202-S205) are not limited to the same sample.

[0044] In one optional implementation, when constructing the training dataset, the original high-resolution image is first read and randomly cropped to obtain a single-frame high-resolution image patch. Then, this single-frame high-resolution image patch is downsampled to obtain a low-resolution reference frame. The downsampling method can be selected according to actual needs, such as bilinear interpolation or bicubic interpolation. Then, for each non-reference frame, a sub-pixel displacement vector (e.g., -1 to 1 low-resolution pixel) relative to the low-resolution reference frame is randomly generated. An affine transformation is then used to perform a displacement operation corresponding to this sub-pixel displacement vector on the original high-resolution image, generating a displacementd high-resolution image. Next, the displacementd high-resolution image undergoes the same cropping operation as the single-frame high-resolution image patch, and is downsampled to obtain low-resolution sample images. Then, the low-resolution sample images corresponding to each non-reference frame and the low-resolution reference frame are combined to form a low-resolution sample image sequence. Finally, the low-resolution sample image sequence, the sub-pixel displacement vectors corresponding to each non-reference frame, and the single-frame high-resolution image patch constitute a set of training samples.

[0045] For example, Figure 3 This is a schematic diagram illustrating the generation of a training dataset according to an embodiment of the present invention. When constructing the training dataset, an N-frame low-resolution grayscale image sequence (low-resolution sample image sequence) of the same test sample is acquired through a spatial radio frequency marker microscopy imaging module. The 0th frame is used as the reference frame, and the sub-pixel displacement vectors of the remaining frames relative to the reference frame are precisely known by the system hardware parameters, with a displacement range of […]. [1,+1] low-resolution pixels. Specifically, first read the original high-resolution image. ,in, , H O Indicates the height of the original high-resolution image. W O Indicates the width of the original high-resolution image. R This represents the set of real numbers. Next, through linear grayscale normalization, the original high-resolution image is mapped from a 16-bit image to an 8-bit image, and from a color image to a grayscale image, to unify the data format and maintain the consistency of the image's grayscale information. Then, the converted original high-resolution image is randomly cropped to generate a size of... High-resolution image patches ,in, This represents the high-resolution image patch generated by cropping. H crop Indicates the height of a high-resolution image patch. W crop Indicates the width of a high-resolution image patch; For the trimming operator, The randomly selected starting coordinates for cropping satisfy... , This high-resolution image patch, used as a supervised learning label, is denoted as: ,in, This represents the true high-resolution labeled image. Next, interpolation downsampling (subsampling) is performed on this high-resolution image patch to obtain a low-resolution reference frame. .in, This indicates the 0th frame as a low-resolution reference frame. The scale factor is represented as s The downsampling operator, s It is a positive integer. This can be achieved using bilinear interpolation, bicubic interpolation, or other image resampling methods. For example, if... This indicates that the horizontal and vertical directions are both magnified or reduced by a factor of 4, corresponding to an area scale magnified or reduced by a factor of 16.

[0046] For each non-reference frame, randomly generate its sub-pixel displacement vector relative to the low-resolution reference frame:

[0047] in, Indicates the first i The horizontal displacement of a frame relative to a low-resolution reference frame, in units of low-resolution pixels; Indicates the first iThe vertical displacement of the frame relative to the low-resolution reference frame, in units of low-resolution pixels. For the low-resolution reference frame, its corresponding sub-pixel displacement vector is 0, i.e.:

[0048] Then, based on this subpixel displacement vector, the corresponding subpixel translation resampling (displacement operation) is performed on the high-resolution image patch to obtain the displaced high-resolution image patch:

[0049] in, Indicates the first i The displacement of the high-resolution image patch corresponding to the frame. Represents the subpixel displacement vector The determined affine translation resampling operator. Because... The unit is low-resolution pixels, therefore the corresponding displacement scale in the high-resolution image patch coordinate system is... .

[0050] Then, downsampling is performed on the shifted high-resolution image patch to obtain the first... i Frame low-resolution sample image:

[0051] Then, the low-resolution reference frame Low-resolution sample images corresponding to each non-reference frame Composition of low-resolution sample image sequence :

[0052] At the same time, the sub-pixel displacement vectors corresponding to each frame are combined into a displacement vector set. :

[0053] Finally, the low-resolution sample image sequence, the corresponding set of sub-pixel displacement vectors, and the single-frame high-resolution image patch are used to form a set of training samples. .

[0054] By repeating the above process, multiple sets of training samples can be obtained, forming a training dataset.

[0055] Optionally, during the training phase, for a given sequence of low-resolution sample images... and its corresponding set of subpixel displacement vectors The image super-resolution neural network model learns the following mapping relationship:

[0056] in, This represents the reconstructed high-resolution image predicted by the image super-resolution neural network model. Indicates that there are trainable parameters The image super-resolution neural network model is characterized. It should be noted that the trainable parameters... The set of trainable parameters used to indicate the super-resolution neural network model for images includes parameters for the stacked spot image feature extraction module, the stacked spot image feature fusion module, and the image reconstruction module. For ease of description, a unified set of parameters is used. express.

[0057] In one implementation, the supervised loss function may be the L1 loss function, or a combined loss function including an L1 loss term and a robust error term. Taking the L1 loss function as an example, for a high-resolution label image of size... For the image, the expression for the L1 loss function is:

[0058] in, This represents the height of the actual high-resolution label image. Indicates the width of the high-resolution label image. This indicates that the image super-resolution neural network model is in coordinates The predicted pixel value output at that location. This represents the true high-resolution label image in coordinates. The pixel value at that location.

[0059] Meanwhile, the trainable parameters are continuously optimized using the gradient descent algorithm. .

[0060] For example, the Adam optimizer is used to update trainable parameters. :

[0061] in, The loss function represents the loss function with respect to trainable parameters. gradient, This indicates a parameter update operation. This represents the learning rate. For example, the learning rate... An adaptive learning rate adjustment strategy based on validation set PSNR performance is adopted. When the validation set PSNR metric does not improve in several consecutive training rounds (e.g., 8 consecutive rounds), the learning rate decays to a preset proportion (e.g., half). An early stopping mechanism can also be set to terminate training when the validation set PSNR metric does not improve for a preset number of consecutive rounds (e.g., 15 consecutive rounds).

[0062] Selecting optimal model parameters based on the PSNR metric of the validation set. :

[0063] in, This represents the true high-resolution labeled images in the validation set. This represents a sequence of low-resolution sample images in the validation set. This represents the set of subpixel displacement vectors in the validation set. Indicates that there are trainable parameters The super-resolution neural network model representing the image.

[0064] The formula for calculating the PSNR index is as follows:

[0065]

[0066] in, This indicates the maximum possible value of an image pixel; for an 8-bit grayscale image, For grayscale images normalized to [0,1], MSE represents mean squared error. This represents the height of the actual high-resolution label image. Represents the width of the actual high-resolution label image; This represents the reconstructed high-resolution image output by the image super-resolution neural network model in coordinates. Pixel value at; This represents the true high-resolution label image in coordinates. The pixel value at that location.

[0067] For example, training employs mini-batch stochastic gradient descent with a batch size of B. Each training round iterates through the entire training set, calculating gradients and updating parameters. After each round, performance is evaluated on the validation set, and the best model is saved based on the PSNR metric, with the learning rate adjusted accordingly. Through end-to-end training, the network learns the ability to recover high-frequency details from low-resolution observations of multi-frame subpixel displacement, with optimization specifically for fine structure reconstruction in microscopic images.

[0068] Step S202: Obtain a multi-frame low-resolution image sequence and a sub-pixel displacement vector corresponding to the multi-frame low-resolution image sequence.

[0069] The low-resolution image sequence was obtained by generating a stacked spot array from a spatial radio frequency marker microscopy imaging module, scanning the sample under test, and performing multiple displacement samplings on the sample under test; each spot in the stacked spot array corresponds to a different frequency.

[0070] Optionally, the stacked spot array can be a one-dimensional or two-dimensional spot array. Adjacent spots in the stacked spot array overlap, and the overlap area, i.e., the spot overlap rate, can be set as needed. Specifically, the spatial radio frequency marker microscopy imaging module obtains each spot through heterodyne interference generated by two linearly polarized beams. One beam of linearly polarized light is modulated using spatial radio frequency marker technology, and the other beam serves as a reference beam, resulting in a stacked spot array with overlapping characteristics that is incident on the sample under test. After transmission through the sample, the resulting optical signal is modulated by the intensity of the sample, thus carrying the spatial characteristics of the sample. Further, the sample under test is placed on a high-precision displacement device. By manipulating the high-precision displacement device, the sample can be repeatedly scanned to obtain a sequence of optical signals carrying the spatial characteristics and relative displacement information of the sample, which serves as the raw signal for subsequent imaging. The sub-pixel displacement vector corresponding to the low-resolution image is determined by the spot overlap rate and the scanning orientation.

[0071] For example, similar to the processing during the construction of the training dataset, the inference phase also requires obtaining a multi-frame low-resolution image sequence of the test sample and the corresponding sub-pixel displacement vectors. N Taking frames as an example, with frame 0 as the reference frame and the remaining frames as non-reference frames, the low-resolution image sequence can be represented as:

[0072] The corresponding set of sub-pixel displacement vectors can be represented as:

[0073] in, , Indicates the first i The subpixel displacement vector of a frame relative to a low-resolution reference frame. Indicates the first i The horizontal displacement of a frame relative to a low-resolution reference frame, in units of low-resolution pixels; Indicates the first i The vertical displacement of a frame relative to a low-resolution reference frame, in units of low-resolution pixels; , indicating that the sub-pixel displacement vector corresponding to the low-resolution reference frame is zero. It should be noted that, for ease of description, both the N low-resolution sample images in the training phase and the N low-resolution images in the inference phase are represented as . The subpixel displacement vectors corresponding to N frames in both the training and inference phases are represented as... However, in practical applications, the same image and subpixel displacement vector are not required for the training and inference phases.

[0074] Step S203: Upsample the multi-frame low-resolution image sequence to a high-resolution grid, and use the sub-pixel displacement vector to perform sub-pixel alignment to obtain an aligned high-resolution grid image sequence.

[0075] In one alternative implementation, when using the subpixel displacement vector for subpixel alignment, each low-resolution image in the multi-frame low-resolution image sequence is first bilinearly upsampled to a high-resolution grid; then, based on the subpixel displacement vector, a bilinear grid sampling function is used to map each upsampled low-resolution image to a high-resolution grid, resulting in a high-resolution grid image after alignment for each frame.

[0076] For example, for the first i Frame low-resolution image sequence Bilinear upsampling to high-resolution grid:

[0077] in, Indicates the first i High-resolution grid intermediate image after frame upsampling, The scale factor is represented as s The upsampling operation, i.e., the super-resolution magnification is s .

[0078] Then, subpixel alignment is performed based on the subpixel displacement vector to obtain the aligned high-resolution mesh image. :

[0079] in, Indicates the first i High-resolution grid image after frame alignment Represents pixel coordinates in a high-resolution grid. Represents the bilinear grid sampling function. This represents the displacement after converting the low-resolution pixel unit to a high-resolution grid. The above formula uses backsampling, meaning that the output pixel coordinates... From the upsampled high-resolution grid intermediate image coordinates Interpolation sampling is performed at the location.

[0080] Step S204: The aligned high-resolution grid image sequence is input into the pre-constructed image super-resolution neural network model, and then processed by the stacked spot image feature extraction module, the stacked spot image feature fusion module, and the image reconstruction module in sequence to output the reconstructed high-resolution image.

[0081] In one optional implementation, when the reconstructed high-resolution image is output after sequential processing by the stacked spot image feature extraction module, the stacked spot image feature fusion module, and the image reconstruction module, the stacked spot image feature extraction module first performs multi-layer convolutional feature extraction on each frame of the aligned high-resolution grid image sequence to obtain the corresponding single-frame feature map. For example, the stacked spot image feature extraction module includes two convolutional layers and a non-linear activation function set after at least one of the convolutional layers. For instance, a first convolutional layer, a first activation function, and a second convolutional layer are used sequentially for feature extraction to obtain each frame's feature map. The stacked spot image feature extraction module is also used to keep the pixel value range of the input high-resolution grid image within a preset range. Next, the stacked spot image feature fusion module performs cascaded operations on the feature map channels of each frame to obtain a merged feature map. Then, a third convolutional layer, a second activation function are sequentially used to initially compress and fuse multi-frame information, a fourth convolutional layer, and a third activation function to further refine the fused features, thereby performing feature fusion on the merged feature map to obtain a fused feature map. This stacked spot image feature fusion module enhances cross-frame complementarity and suppresses noise by fusing feature maps from various frames using two convolutional layers, ultimately outputting a single-frame scale fused feature map. Finally, the image reconstruction module performs deep feature reconstruction on the fused feature map and maps it back to image space, outputting a single-channel high-resolution image. This image reconstruction module may include one or more residual reconstruction units and image mapping convolutional layers. For example, the image reconstruction module uses a fifth convolutional layer to map the fused feature map back to image space, outputting a single-channel high-resolution image.

[0082] For example, the first convolutional layer is a 3×3 convolutional network; the second convolutional layer is a 3×3 convolutional network; the third convolutional layer is a 3×3 convolutional network; the fourth convolutional layer is a 3×3 convolutional network; the first activation function is a ReLU activation function; the second activation function is a ReLU activation function; the third activation function is a ReLU activation function; and the stacked spot image feature extraction module outputs a 64-channel feature map.

[0083] For example, the expression for the stacked spot image feature extraction module is as follows:

[0084] in, The kernel of the first convolutional layer, This is the bias of the first convolutional layer. The kernel of the second convolutional layer. This is the bias for the second convolutional layer. , , For the first i Frame feature map , sH Indicates the height of the feature map. sW This represents the width of the feature map. It should be noted that in this embodiment, the convolutional layer uses a convolution method that maintains the same spatial dimensions; therefore, the single-frame feature map, the fused feature map, and the high-resolution mesh image have the same spatial dimensions. In other words, , .

[0085] For example, the expression for the stacked spot image feature fusion module is as follows:

[0086] in, Represents the fused feature map. The kernel of the third convolutional layer, The kernel of the fourth convolutional layer. , , The bias of the third convolutional layer, This is the bias of the fourth convolutional layer.

[0087] For example, the fifth convolutional layer is a 3×3 convolutional network; the image reconstruction module outputs a single-channel high-resolution image, which can be represented as an 8-bit grayscale image, a 16-bit grayscale image, or a floating-point grayscale image, depending on storage, display, or subsequent analysis requirements.

[0088] For example, the image reconstruction result can be represented as:

[0089] in, Indicates that there are trainable parameters Residual reconstruction mapping of representation, This indicates that the low-resolution reference frame has been magnified by a factor of 1. s ( s The upsampling result (for the scaling factor).

[0090] Optionally, the reconstructed image can be directly output using only convolutional mapping. Taking an image reconstruction module containing a fifth convolutional layer as an example, the expression for this image reconstruction module is:

[0091] in, The kernel of the fifth convolutional layer. , The bias of the fifth convolutional layer, Represents the reconstructed high-resolution image. .

[0092] In an optional implementation, the function implemented in step S203 can also be integrated into the image super-resolution neural network model. For example, an image upsampling module can be set in the image super-resolution neural network model to upsample the multi-frame low-resolution image sequence to a high-resolution grid, and a sub-pixel alignment module can be set to perform sub-pixel alignment using the sub-pixel displacement vector to obtain an aligned high-resolution grid image sequence. For example, Figure 4 This is a schematic diagram of the structure of an image super-resolution neural network model according to an embodiment of the present invention. Figure 4 The subpixel alignment module is deployed in the super-resolution neural network model shown.

[0093] Step S205: Perform a no-reference quality assessment on the reconstructed high-resolution image to obtain the assessment result; if the assessment result does not meet the preset requirements, generate a reacquisition control command to control the spatial radio frequency marker microscopy imaging module to re-execute the step of acquiring multiple frames of low-resolution image sequence.

[0094] For example, this no-reference quality assessment uses the Natural Image Quality Evaluator (NIQE) to evaluate the naturalness and distortion of an image. Alternatively, metrics such as the Laplacian variance can also be used as the basis for image quality assessment.

[0095] The multi-frame subpixel super-resolution image reconstruction method provided in this embodiment significantly reduces the number of model parameters and inference time by constructing a lightweight image super-resolution neural network model and adopting an optimized training strategy. This enables the super-resolution reconstruction process to be processed in real time on most hardware devices, solving the technical obstacle of integrating super-resolution methods into real-time analysis systems of microscope imagers in related technologies, and providing practical value for real-time analysis.

[0096] As one or more specific application embodiments of the present invention, the optimal implementation scheme or the scheme that the inventors most want to embody is described in combination with the specific application scenario.

[0097] To verify the effectiveness of the present invention, Figure 5 The diagram below illustrates the reconstruction effect according to an embodiment of the present invention. As shown in Figure 5, (a) is one of the input low-resolution images, (b) is the high-resolution image reconstructed using the multi-frame sub-pixel super-resolution image reconstruction method, and (c) is the original high-resolution image. It can be seen that the present invention has performed good restoration of cell edges in low-resolution images and has increased the details inside cells to a certain extent, making its texture and edge features more obvious and closer to the original high-resolution image.

[0098] Table 1 shows a quantitative performance comparison between the multi-frame subpixel super-resolution image reconstruction method used in this invention and the traditional nearest neighbor interpolation algorithm. It can be seen that the PSNR and SSIM (Structure Similarity Index Measure) of this method are both better, with PSNR reaching 30.03 dB and SSIM reaching 0.9555.

[0099] Table 1: Quantitative performance comparison of reconstruction algorithms.

[0100]

[0101] This embodiment provides a multi-frame subpixel super-resolution image reconstruction system. Figure 6 This is a schematic diagram of the structure of a multi-frame subpixel super-resolution image reconstruction system according to an embodiment of the present invention, as shown below. Figure 6 As shown, the system includes: The spatial radio frequency labeling microscopy imaging module is used to acquire a multi-frame low-resolution image sequence and a sub-pixel displacement vector corresponding to the multi-frame low-resolution image sequence. The low-resolution image sequence is obtained by the spatial radio frequency labeling microscopy imaging module generating a stacked spot array to scan the sample under test and performing multiple displacement samplings on the sample under test. Each spot in the stacked spot array corresponds to a different frequency. The image super-resolution reconstruction module is used to perform tasks such as... Figure 1 Examples or any of their corresponding implementations and Figure 2 The multi-frame subpixel super-resolution image reconstruction method of the embodiment or any of its corresponding implementations converts the multi-frame low-resolution image sequence and the subpixel displacement vectors corresponding to the multi-frame low-resolution image sequence into a reconstructed high-resolution image. The image evaluation and reacquisition module is used to evaluate the reconstructed high-resolution image. If the evaluation fails, it controls the spatial radio frequency marker microscopy module to re-execute the step of acquiring a sequence of multiple low-resolution images.

[0102] Optionally, the spatial radio frequency marker microscopy imaging module includes a light source module, a radio frequency drive module, a heterodyne interferometry module, and a high-precision displacement stage. The light source module generates a beam that is incident on the heterodyne interferometry module. The heterodyne interferometry module splits the beam into two linearly polarized beams. One of these beams is modulated according to multiple frequency radio frequency drive signals generated by the radio frequency drive module. This modulated beam, along with the other linearly polarized beam, forms a reference beam. These beams are then combined in the heterodyne interferometry module to generate a stacked spot array. This stacked spot array is used to scan the sample on the high-precision displacement stage to obtain sample information. By repeatedly scanning the high-precision displacement stage, a sequence of sample information with relative displacement information (sub-pixel displacement vectors) can be obtained.

[0103] Optionally, the light source module, radio frequency drive module, heterodyne interference module, and high-precision displacement stage are integrated into a microscopic imager.

[0104] Optionally, the space radio frequency tagging microscopy imaging module also includes an image acquisition module. This image acquisition module has high-speed data acquisition and time-frequency domain calculation capabilities. It acquires the continuous time-domain light signal generated by the space radio frequency tagging microscopy imaging module scanning the sample under test, and obtains image information through frequency domain calculation to obtain a multi-frame low-resolution image sequence. For example, the image acquisition module converts the continuous time-domain light signal obtained by the space radio frequency tagging microscopy imaging module scanning the sample under test into a grayscale image sequence through an image acquisition board, and stores it along with relative displacement information in a storage or computing device for further processing.

[0105] The image super-resolution reconstruction module ( Figure 6 The image reconstruction module is used to build a super-resolution neural network model and deploy the trained super-resolution neural network model on a computing device. It takes relative displacement information (sub-pixel displacement vector) and a sequence of multiple low-resolution images as input and calculates to obtain the reconstructed high-resolution image.

[0106] The image evaluation and re-acquisition module evaluates the reconstructed high-resolution image. If the pre-set requirements are not met or the user inputs re-acquisition information, the acquisition and reconstruction process is re-executed, and the high-resolution image is output again.

[0107] In one alternative implementation, the multi-frame subpixel super-resolution image reconstruction system can be integrated into a flow cytometer, digital pathology scanner, or medical imaging workstation to generate high-resolution cell images in real time.

[0108] This embodiment also provides a multi-frame subpixel super-resolution image reconstruction apparatus, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0109] This embodiment provides a multi-frame subpixel super-resolution image reconstruction device, such as... Figure 7 As shown, it includes: The image acquisition module 701 is used to acquire a multi-frame low-resolution image sequence and sub-pixel displacement vectors corresponding to the multi-frame low-resolution image sequence; the low-resolution image sequence is obtained by generating a stacked spot array to scan the sample under test by a spatial radio frequency marker microscopy imaging module and performing multiple displacement samplings on the sample under test; each spot in the stacked spot array corresponds to a different frequency. Alignment module 702 is used to upsample the multi-frame low-resolution image sequence to a high-resolution grid and perform sub-pixel alignment using the sub-pixel displacement vector to obtain an aligned high-resolution grid image sequence. The image reconstruction module 703 is used to input the aligned high-resolution grid image sequence into a pre-constructed image super-resolution neural network model, and process it sequentially through the stacked spot image feature extraction module, the stacked spot image feature fusion module and the image reconstruction module to output the reconstructed high-resolution image.

[0110] In one optional implementation, the sub-pixel alignment using the sub-pixel displacement vector includes: Each low-resolution image in the multi-frame low-resolution image sequence is bilinearly upsampled to a high-resolution grid. Based on the subpixel displacement vector, a bilinear grid sampling function is used to map each upsampled low-resolution image frame to a high-resolution grid, resulting in an aligned high-resolution grid image for each frame.

[0111] In one optional implementation, the process of sequentially passing through a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module to output a reconstructed high-resolution image includes: The stacked spot image feature extraction module extracts features from each frame of the aligned high-resolution grid image sequence by sequentially using the first convolutional layer, the first activation function, and the second convolutional layer to obtain a feature map for each frame. The stacked spot image feature fusion module performs cascaded operations on the feature map channels of each frame to obtain a merged feature map. The merged feature map is then fused sequentially using the third convolutional layer, the second activation function, the fourth convolutional layer, and the third activation function to obtain a fused feature map. The image reconstruction module uses a fifth convolutional layer to map the fused feature map back to the image space, outputting a single-channel high-resolution image.

[0112] In an alternative implementation, after outputting the reconstructed high-resolution image, the method further includes: The reconstructed high-resolution image is subjected to a no-reference quality assessment to obtain the assessment results; If the evaluation result does not meet the preset requirements, a reacquisition control command is generated to control the spatial radio frequency marker microscopy imaging module to re-execute the step of acquiring multiple frames of low-resolution image sequences.

[0113] In one optional implementation, the method further includes: Construct a training dataset; the training dataset contains multiple sets of training samples, each set of training samples includes a multi-frame low-resolution sample image sequence with sub-pixel displacement, sub-pixel displacement vectors corresponding to the multi-frame low-resolution sample image sequence respectively, and a single-frame high-resolution image patch as a supervised learning label; An initial image super-resolution neural network model is constructed; the initial image super-resolution neural network model includes a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module; Using the training dataset, the initial image super-resolution neural network model is trained by optimizing the loss function to obtain the image super-resolution neural network model.

[0114] In one alternative implementation, constructing the training dataset includes: Read the original high-resolution image and randomly crop it to obtain a single frame of high-resolution image patch; The single-frame high-resolution image block is downsampled using bicubic interpolation to obtain a low-resolution reference frame; For each non-reference frame, a sub-pixel displacement vector is randomly generated relative to the low-resolution reference frame. An affine transformation is then used to perform a displacement operation on the original high-resolution image corresponding to the sub-pixel displacement vector to generate a displacementd high-resolution image. The high-resolution image after displacement is cropped in the same way as the single-frame high-resolution image block, and then downsampled to obtain a low-resolution sample image; The low-resolution sample images corresponding to each non-reference frame are combined with the low-resolution reference frame to form a low-resolution sample image sequence; The low-resolution sample image sequence, the sub-pixel displacement vectors corresponding to each non-reference frame, and the single-frame high-resolution image block constitute a set of training samples.

[0115] The multi-frame subpixel super-resolution image reconstruction apparatus provided in this embodiment of the invention can execute the multi-frame subpixel super-resolution image reconstruction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the above modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0116] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0117] The following is a detailed reference. Figure 8 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 802 or a program loaded from memory 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device. The processor 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0118] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0119] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a memory 808, or installed from a ROM 802. When the computer program is executed by the processor 801, it performs the functions defined in the multi-frame sub-pixel super-resolution image reconstruction method of the embodiments of the present invention.

[0120] Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0121] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the multi-frame sub-pixel super-resolution image reconstruction method shown in the above embodiments is implemented.

[0122] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0123] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the invention.

Claims

1. A method for reconstructing multi-frame subpixel super-resolution images, characterized in that, The method includes: A multi-frame low-resolution image sequence is acquired, along with sub-pixel displacement vectors corresponding to each of the multi-frame low-resolution image sequences. The low-resolution image sequence is obtained by generating a stacked spot array from a spatial radio frequency marker microscopy imaging module to scan the sample under test and performing multiple displacement samplings on the sample under test. Each spot in the stacked spot array corresponds to a different frequency. The multi-frame low-resolution image sequence is upsampled to a high-resolution grid, and sub-pixel alignment is performed using the sub-pixel displacement vector to obtain an aligned high-resolution grid image sequence. The aligned high-resolution grid image sequence is input into a pre-constructed image super-resolution neural network model, and then processed sequentially by a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module to output a reconstructed high-resolution image.

2. The multi-frame subpixel super-resolution image reconstruction method according to claim 1, characterized in that, The sub-pixel alignment using the sub-pixel displacement vector includes: Each low-resolution image in the multi-frame low-resolution image sequence is bilinearly upsampled to a high-resolution grid. Based on the subpixel displacement vector, a bilinear grid sampling function is used to map each upsampled low-resolution image frame to a high-resolution grid, resulting in an aligned high-resolution grid image for each frame.

3. The multi-frame subpixel super-resolution image reconstruction method according to claim 2, characterized in that, The image is processed sequentially by a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module, outputting a reconstructed high-resolution image, including: The stacked spot image feature extraction module extracts features from each frame of the aligned high-resolution grid image sequence by sequentially using the first convolutional layer, the first activation function, and the second convolutional layer to obtain a feature map for each frame. The stacked spot image feature fusion module performs cascaded operations on the feature map channels of each frame to obtain a merged feature map. The merged feature map is then fused sequentially using the third convolutional layer, the second activation function, the fourth convolutional layer, and the third activation function to obtain a fused feature map. The image reconstruction module uses a fifth convolutional layer to map the fused feature map back to the image space, outputting a single-channel high-resolution image.

4. The multi-frame subpixel super-resolution image reconstruction method according to any one of claims 1 to 3, characterized in that, After outputting the reconstructed high-resolution image, the method further includes: The reconstructed high-resolution image is subjected to a no-reference quality assessment to obtain the assessment results; If the evaluation result does not meet the preset requirements, a reacquisition control command is generated to control the spatial radio frequency marker microscopy imaging module to re-execute the step of acquiring multiple frames of low-resolution image sequences.

5. The multi-frame subpixel super-resolution image reconstruction method according to claim 1, characterized in that, The method further includes: Construct a training dataset; the training dataset contains multiple sets of training samples, each set of training samples includes a multi-frame low-resolution sample image sequence with sub-pixel displacement, sub-pixel displacement vectors corresponding to the multi-frame low-resolution sample image sequence respectively, and a single-frame high-resolution image patch as a supervised learning label; An initial image super-resolution neural network model is constructed; the initial image super-resolution neural network model includes a stacked spot image feature extraction module, a stacked spot image feature fusion module, and an image reconstruction module; Using the training dataset, the initial image super-resolution neural network model is trained by optimizing the loss function to obtain the image super-resolution neural network model.

6. The multi-frame subpixel super-resolution image reconstruction method according to claim 5, characterized in that, The construction of the training dataset includes: Read the original high-resolution image and randomly crop it to obtain a single frame of high-resolution image patch; The single-frame high-resolution image block is downsampled using bicubic interpolation to obtain a low-resolution reference frame; For each non-reference frame, a sub-pixel displacement vector is randomly generated relative to the low-resolution reference frame. An affine transformation is then used to perform a displacement operation on the original high-resolution image corresponding to the sub-pixel displacement vector to generate a displacementd high-resolution image. The high-resolution image after displacement is cropped in the same way as the single-frame high-resolution image block, and then downsampled to obtain a low-resolution sample image; The low-resolution sample images corresponding to each non-reference frame are combined with the low-resolution reference frame to form a low-resolution sample image sequence; The low-resolution sample image sequence, the sub-pixel displacement vectors corresponding to each non-reference frame, and the single-frame high-resolution image block constitute a set of training samples.

7. A multi-frame subpixel super-resolution image reconstruction system, characterized in that, The system includes: A spatial radio frequency labeling microscopy imaging module is used to acquire a multi-frame low-resolution image sequence and sub-pixel displacement vectors corresponding to the multi-frame low-resolution image sequence; the low-resolution image sequence is obtained by the spatial radio frequency labeling microscopy imaging module generating a stacked spot array to scan the sample under test and performing multiple displacement samplings on the sample under test; each spot in the stacked spot array corresponds to a different frequency. The image super-resolution reconstruction module is used to execute the multi-frame sub-pixel super-resolution image reconstruction method as described in any one of claims 1-6, and to convert the multi-frame low-resolution image sequence and the sub-pixel displacement vectors corresponding to the multi-frame low-resolution image sequence into reconstructed high-resolution images. The image evaluation and reacquisition module is used to evaluate the reconstructed high-resolution image, and when the evaluation fails, it controls the spatial radio frequency marker microscopy module to re-execute the step of acquiring a multi-frame low-resolution image sequence.

8. An electronic device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to perform the multi-frame subpixel super-resolution image reconstruction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the multi-frame subpixel super-resolution image reconstruction method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes computer instructions for causing a computer to perform the multi-frame subpixel super-resolution image reconstruction method according to any one of claims 1 to 6.

Images