A super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration

Abstract

The application discloses a kind of super-resolution SIM-FRET image reconstruction methods based on GPU parallel acceleration, including steps: FRET original image is collected using acquisition card and is stored;Program initialization is carried out;FRET single-channel original image is obtained and transmitted asynchronously to GPU global memory;Parallel image preprocessing is carried out to FRET single-channel original image group by GPU;Normalizing separation image is iterated by the heterogeneous computing of CPU and GPU Correlation Parameter Estimation, and the modulation intensity and initial phase of illumination light vector are estimated;GPU parallel SR-SIM linear wiener reconstruction;Super-resolution FRET three-channel image is obtained;FRET three-channel super-resolution image is processed based on channel sensitization intensity measurement method E-FRET data by GPU.The application improves the parallel algorithm structure suitable for GPU acceleration, and the heterogeneous computing of CPU and GPU is used for the iterative correlation parameter estimation algorithm which cannot be completely parallelized, improves the GPU histogram summation algorithm of biological image, and improves the time resolution of super-resolution SIM-FRET image reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fluorescence resonance energy transfer FRET detection, and particularly to a super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration. BACKGROUND

[0002] The fluorescence resonance energy transfer FRET microscopic imaging technology can be used to analyze the interaction of biological macromolecules in a living cell in real time and quantitatively. The distance of 1-10 nm is the size of some biological macromolecules and intracellular proteins, and is also the approximate range of the thickness of biological membranes, which makes the fluorescence resonance energy transfer FRET microscopic imaging technology play a wide role in studying the interaction of biological macromolecules and proteins, the conformational changes and the regulatory network between proteins in the signal pathway in cell biology. The traditional wide-field-based FRET microscopic imaging is limited by the diffraction limit and is not sufficient to reveal the FRET signal of the molecular interaction located in the sub-diffraction region. The structured light super-resolution SR-SIM microscopic imaging technology can break through the limit of the optical diffraction limit, so that it can reveal the function and regulation mechanism of biological molecules more accurately when combined with SIM-FRET.

[0003] The traditional CPU-based super-resolution SIM-FRET image reconstruction speed is slow. According to the principle of super-resolution SIM-FRET image reconstruction, the improvement of resolution is realized from the frequency domain operation, and the image data needs to be converted between the complex plane and the real plane; the parameter estimation of the image needs to obtain accurate reconstruction parameters through iterative cross-correlation; and the FRET analysis of the image needs histogram calculation. Therefore, the super-resolution SIM-FRET image reconstruction involves a large number of Fourier transform, cross-correlation, histogram, matrix point-by-point addition, subtraction, multiplication and division operations. These operations take a lot of time, and it usually takes about 1 minute to reconstruct 27 original images with a size of 512*512, which cannot meet the needs of researchers for rapid feedback of experimental results, quality monitoring and dynamic intervention of the experimental process.

[0004] The time-consuming repetitive operations involved in the super-resolution SIM-FRET image reconstruction can be accelerated by the graphics processing unit (GPU). The GPU is mainly composed of a large number of arithmetic units and caches, has a large number of computing units and a super-long pipeline, and processes each pixel on the graph in a similar way. For a large number of same type of calculations, the GPU can exert its powerful computing performance. Compared with the CPU, the GPU is configured with a large number of arithmetic units and a small amount of cache, and the GPU is better at large amount of calculation and repetitive operation. SUMMARY

[0005] In order to solve the technical problems in the prior art, the application provides a super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration, wherein a traditional CPU-based serial super-resolution SIM-FRET image reconstruction algorithm is improved into a parallel algorithm structure suitable for GPU acceleration, further, a CPU and a GPU are used for heterogeneous calculation on an iterative cross-correlation parameter estimation algorithm which cannot be completely parallelized, and a GPU histogram summation algorithm of a biological image is improved, so that the time resolution of the super-resolution SIM-FRET image reconstruction is improved.

[0006] The application is implemented by adopting the following technical scheme: a super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration, comprising the following steps:

[0007] S101, a FRET original image modulated by a structured light is collected by a collection card in a single image unit and stored;

[0008] S102, program initialization is performed, including CPU end program memory application and GPU end CUDA environment initialization;

[0009] S103, a FRET single-channel original image is acquired and transmitted to a GPU global memory asynchronously; wherein the FRET single channel is one of a donor excitation donor emission channel DD, a donor excitation acceptor emission channel DA and an acceptor excitation acceptor emission channel AA;

[0010] S104, a FRET single-channel original image group is subjected to parallel image preprocessing by a GPU; the FRET single-channel original image is subjected to two-dimensional Fourier transform first, then phase difference separation is performed to obtain a separated image, and the separated image is subjected to normalization processing to obtain a normalized separated image;

[0011] S105, the normalized separated image is subjected to iterative cross-correlation parameter estimation by heterogeneous calculation of a CPU and a GPU, and the modulation intensity and the initial phase of an illumination light vector are estimated;

[0012] S106, GPU parallel SR-SIM linear Wiener reconstruction; the GPU performs parallel decomposition of a FRET single-channel spectrum according to the estimated initial phase to obtain a decomposition image, then the GPU performs frequency shift on the decomposition image according to the estimated illumination light vector to obtain a frequency shift image, and an optical transfer function corresponding to the frequency shift is generated in parallel, and the GPU performs linear Wiener filter calculation according to the obtained result to obtain a FRET single-channel super-resolution image;

[0013] S107, if the FRET three-channel super-resolution image reconstruction is completed, step S108 is entered, otherwise, steps S103-S106 are repeated in a loop;

[0014] S108, acquire a super-resolution FRET three-channel image;

[0015] S109, perform E-FRET data processing on the FRET three-channel super-resolution image based on channel sensitization intensity measurement by GPU.

[0016] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0017] 1. The present application improves the traditional CPU-based serial super-resolution SIM-FRET image reconstruction algorithm into a parallel algorithm structure suitable for GPU acceleration, further uses CPU and GPU for heterogeneous computing on the iterative cross-correlation parameter estimation algorithm which cannot be completely parallelized, and improves the GPU histogram summation algorithm of biological images, thereby improving the time resolution of super-resolution SIM-FRET image reconstruction.

[0018] 2. The method of the present application shortens the time consumption of super-resolution SIM-FRET reconstruction of 27 images with a traditional resolution of 512*512 from about 36 seconds to 1 second on a platform with CPU i5-12450H, GPU RTX-3060-Laptop and memory of 16G, and the speed is improved by about 36 times. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of the method of the present application;

[0020] Figure 2 is a flowchart of CPU and GPU heterogeneous parallel computing iterative cross-correlation parameter estimation;

[0021] Figure 3 is a flowchart of GPU parallel SR-SIM linear Wiener reconstruction;

[0022] Figure 4 is a flowchart of GPU parallel E-FRET calculation;

[0023] Figure 5 is a histogram block partitioning schematic diagram;

[0024] Figure 6 is a histogram interlaced partitioning schematic diagram;

[0025] Figure 7 is a histogram optimization program flowchart;

[0026] Figure 8 is a FRET three-channel image comparison diagram;

[0027] Figure 9 is a FRET quantitative analysis result comparison diagram. DETAILED DESCRIPTION

[0028] The application will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the application are not limited thereto. Embodiments

[0029] As shown in the figure, the embodiment of the application is a GPU parallel acceleration-based super-resolution SIM-FRET image reconstruction method, comprising the following steps: Figure 1

[0030] S101, using a capture card to collect FRET original images modulated by structured light in units of single images and store them;

[0031] S102, program initialization, including CPU program memory application and GPU CUDA environment initialization; wherein CUDA is a unified computing device architecture (Compute Unified Device Architecture, CUDA), which is a parallel computing platform launched by NVIDIA, which provides related APIs for developers to use GPU to complete general-purpose computing acceleration; CUDA environment initialization includes selecting a GPU device, allocating GPU global memory, allocating CPU page-locked memory, configuring cuFFT library parameters, etc.; wherein the cuFFT library is a GPU-based fast Fourier transform library provided by NVIDIA;

[0032] S103, obtaining FRET single-channel original images and transmitting them to GPU global memory asynchronously; wherein the FRET single channel is one of the donor excitation donor emission channel DD, the donor excitation acceptor emission channel DA, and the acceptor excitation acceptor emission channel AA;

[0033] S104, parallel image preprocessing of the FRET single-channel original image group by GPU; first, two-dimensional Fourier transform is performed on the FRET single-channel original image, then phase difference separation is performed to obtain a separated image, and normalization processing is performed on the separated image to obtain a normalized separated image;

[0034] ​S105, the normalized separation image is iteratively correlated by heterogeneous computing of the CPU and the GPU to estimate the modulation intensity and the initial phase of the illumination light vector; the iteratively correlated parameter estimation is to implement the part with high step complexity in the iteratively correlated algorithm by the CPU in series, implement the part with low step complexity by the GPU in parallel, and implement the part of loop iteration by OpenMP multi-thread distribution CUDA stream; OpenMP (Open Multi-Processing) is a programming model of CPU parallel computing, which allows developers to write parallel programs in a shared memory system; CUDA stream is a very important concept in CUDA programming, and stream is a mechanism for asynchronous execution of CUDA command sequences, which allows the use of device parallelism to improve the performance of the application;

[0035] S106, GPU parallel SR-SIM linear Wiener reconstruction; the GPU performs parallel decomposition FRET single-channel spectrum according to the estimated initial phase to obtain a decomposition image, then the GPU performs frequency shift on the decomposition image according to the estimated illumination light vector to obtain a frequency shift image, while generating an optical transfer function corresponding to the frequency shift in parallel, and then the GPU performs linear Wiener filter calculation according to the obtained results to obtain a FRET single-channel super-resolution image;

[0036] S107, if the FRET three-channel super-resolution image reconstruction is completed, step S108 is entered, otherwise, steps S103-S106 are repeated; the repeated operation is executed three times, because the image transmission of the acquisition card is line by line, a plurality of images of one channel are transmitted first, then a plurality of images of the next channel are transmitted, and the CPU can perform subsequent operation after receiving the image of one channel, so that transmission and calculation are overlapped, and the overall time consumption is reduced;

[0037] S108, obtain the super-resolution FRET three-channel image;

[0038] S109, perform E-FRET data processing on the FRET three-channel super-resolution image based on the channel sensitization intensity measurement method by the GPU.

[0039] In the embodiment, the specific process of step S102 is as follows:

[0040] The system parameters such as objective numerical aperture, objective magnification, optical transfer function and camera pixel size are imported, then the CUDA function cudaSetDevice() is called to select the GPU device to be used, cudaMallocHost is called to allocate CPU page-locked memory, cudaMalloc() is called to allocate GPU global memory, cudaStreamCreate() is called to create a CUDA stream, cufftPlan2d() is called to create a two-dimensional Fourier transform handle, and cufftSetStream() is called to bind the two-dimensional Fourier transform handle to the created CUDA stream.

[0041] In this embodiment, the specific process of step S103 is as follows:

[0042] The CUDA function cudaMemcpyAsync() is called to realize asynchronous data transmission from the CPU page-locked memory to the GPU global memory, and the GPU performs kernel function calculation while executing the asynchronous data transmission, so as to realize the overlap of data transmission and kernel operation and reduce the overall time consumption; specifically, the CUDA function cudaMemcpyAsync() performs parallel asynchronous data transmission of FRET single-channel 9 images from the CPU to the GPU global memory; wherein the FRET single-channel image is one of the following three-channel images, donor excitation donor emission channel image group , donor excitation acceptor emission channel image group and acceptor excitation acceptor emission channel image group , wherein, is the spatial coordinate of the fluorescence image pixel, each group of original image data group contains several different structural light direction angles, and each structural light direction angle contains several cosine structured light illumination wide-field fluorescence images with different phase differences.

[0043] Preferably, the FRET single-channel super-resolution original image, each group of original image data group contains three different structural light direction angles, and each structural light direction angle contains three different phase differences, a total of 9 images.

[0044] In this embodiment, the FRET single-channel original image group in step S104 is taken as a unit of a single image, the FRET single-channel 9 images are parallel Fourier transformed by calling the GPU function cufftExecC2C(), 9 FRET single-channel original image spectrum images are obtained, 9 FRET single-channel original image spectrum images are input, then they are parallel separated according to the phase difference, 9 separated images are obtained, and finally 9 normalized separated images are obtained through parallel normalization, and the specific implementation process includes:

[0045] S1041, parallel Fourier transform of FRET single-channel 9 images, and the specific formula is as follows:

[0046]

[0047] wherein, represents the value of the pixel point of the Fourier transform of ; is the spectrum of the FRET single-channel 9 images, ], i is an integer; k represents the frequency coordinate of the Fourier transform; , respectively represent the donor excitation donor emission channel DD, the donor excitation acceptor emission channel DA, and the acceptor excitation acceptor emission channel AA, wherein, is the spatial coordinate of the fluorescence image pixel;

[0048] S1042, the 9 FRET single-channel original image spectrum is separated according to the known 120° phase difference, and the specific process is as follows:

[0049]

[0050] wherein, j is an imaginary number, represents the value of the pixel point of the separated image of different components and different directions, , ;

[0051] S1043, the separated image is normalized to obtain a normalized separated image, and the normalized separated image is divided by its module pixel by pixel:

[0052]

[0053] wherein, represents the value of the pixel point of the normalized separated image.

[0054] As shown in Figure 2 , in the embodiment, the specific process of step S105 is as follows:

[0055] S201, input the normalized separated image output by the GPU parallel image preprocessing ;

[0056] S202, for the dth direction of the structured light, the normalized separated image corresponding to the direction is extracted ;

[0057] S203, the cross-correlation function of the dth direction normalized separated image is calculated by GPU parallel calculation, and the corresponding illumination light vector when the cross-correlation function maximum value is obtained;

[0058] ​S204, the illumination light vector corresponding to the maximum of the cross-correlation function is taken as the initial value of the estimated illumination light vector ;

[0059] S205, the CPU normalizes the separated images according to the d-th direction , translates the illumination light vector, and calculates the overlapping area of the separated images in the d-th direction, i.e. the overlapping area of and :

[0060]

[0061]

[0062]

[0063] wherein, is a weight coefficient, usually 0.15; is the optical transfer function; is the value of after the restriction of the weight coefficient ; and are the part of and the part of on the overlapping area of and ;

[0064] S206-S213, the CUDA stream is distributed in the overlapping area and by OpenMP for parallel cross-correlation iteration; taking the initial value of the illumination light vector as the independent variable, and as the function, the first iteration is performed between ±2.5 pixels with a step of 0.5, all values are calculated by GPU, and the CPU calculates the complex linear regression of and in the overlapping area according to the independent variable to estimate the modulation intensity and initial phase of multiple illumination light vectors, so that reaches the maximum The estimated modulation intensity and initial phase are taken as the final values of this iteration; the second iteration is performed between ±0.5 pixels with a step of 0.1 taking as the independent variable, and the third iteration is performed between ±0.1 pixels with a step of 0.02 taking as the independent variable, and finally the illumination light vector value of sub-pixel level alignment is estimated The modulation intensity and initial phase of the illumination light vector are given by the following formulas:

[0065]

[0066] S214. Repeat steps S202-S213 three times to estimate the illumination light vectors in the three directions, as well as the modulation intensity and initial phase of the illumination light vectors.

[0067] Specifically, in this embodiment, the normalized separated images are estimated through iterative cross-correlation parameters via heterogeneous computing using CPU and GPU. This is based on the characteristic that CPUs are suitable for processing algorithms with high step complexity, low parallelization, and numerous logical operations. The CPU is used for tasks such as calculating overlapping regions of different separated images, complex linear regression calculations, and loop logic control. Figure 2 The solid-line diagram shows that GPUs are suitable for algorithms with few steps, high parallelization, and few logical operations, such as... Figure 2 The portion within the dashed box diagram represents a preferred embodiment of this algorithm.

[0068] Specifically, CUDA streams are distributed in overlapping regions via OpenMP. and Perform parallel cross-correlation iterations, where N is the number of iterations, determined by the independent variable. The iteration range and iteration step are calculated, and in this embodiment, N=100. Independent variable The preferred embodiment is to select a range of ±2.5 pixels and an iteration step of 0.5 in the first iteration, a range of ±0.5 pixels and an iteration step of 0.1 in the second iteration, and a range of ±0.1 pixels and an iteration step of 0.02 in the third iteration.

[0069] Specifically, since the CPU single-threaded CUDA stream distribution takes a lot of time to process steps S208, S209, and S210, it will block the main thread from distributing the next GPU CUDA stream, resulting in low GPU utilization. Therefore, CPU threads n are used to distribute CUDA stream n, where n is the number of CPU threads used for computation. After extensive testing, the number of threads n used for computation by an 8-thread CPU is set to 5, the number of threads n used for computation by a 16-thread CPU is set to 10, and the number of threads n used for computation by a 32-thread CPU is set to 20. The selection of CPU threads n for distributing CUDA stream n and the number of CPU threads used for computation is a preferred solution in this embodiment.

[0070] like Figure 3 As shown, in this embodiment, the specific process of step S106 is as follows:

[0071] S301, GPU parallelly decomposes the FRET single channel spectrum according to the initial phase estimated in step S105 to obtain a decomposition image:

[0072]

[0073] wherein, is the initial phase estimated in step S105, , and are the spectrum images obtained by Fourier transform of the FRET single channel images , and respectively; represents the value of the pixel point of the separation image of different components and different directions, , ;

[0074] S302, GPU performs frequency shift on the decomposition image according to the illumination light vector estimated in step S105 to obtain a frequency shift image , and simultaneously generates the optical transfer function corresponding to the frequency shift in parallel ;

[0075] S303, GPU multiplies the frequency shift image with the corresponding optical transfer function , and calculates the square sum of each optical transfer function ;

[0076] S304, GPU calculates the Wiener filter according to the previous calculation results, and then performs inverse Fourier transform to obtain the FRET single channel super-resolution image:

[0077]

[0078] wherein, is the gray value of the pixel point of the reconstructed FRET single channel super-resolution image, iFFT represents inverse Fourier transform, represents a Gaussian-type apodization function, represents a Wiener filter parameter, is the spatial coordinate of the pixel of the fluorescence image.

[0079] Specifically, in the embodiment, steps S107-S108 are implemented by repeating steps S103-S106 three times to obtain super-resolution FRET three channel images , and .

[0080] As​​Figure 4 As shown, in this embodiment, the specific process of step S109 is as follows:

[0081] S401-S403, the GPU first processes the input super-resolution FRET three-channel image. , and Perform parallel histogram calculations to calculate the histogram. Figure 1 and histogram Figure 2 Among them, the histogram Figure 1 statistics Pixels with grayscale values ​​of 8-1500 and 8-12 are included in the first histogram bar, pixels with grayscale values ​​of 13-16 are included in the second histogram bar, and so on, for a total of 373 histogram bars, thus obtaining the histogram. This indicates that the nth column counted f pixels; histogram Figure 2 statistics Pixels with grayscale values ​​between 0 and 5000 are included in the first histogram bar. Pixels with grayscale values ​​between 0 and 1 are included in the second histogram bar, and so on, for a total of 5000 histogram bars, thus obtaining the histogram. , indicating that the nth column has counted f pixels;

[0082] S404, GPU via histogram Figure 1 Parallel computation of FRET three-channel image background subtraction, using histograms of donor and acceptor channels. Figure 2 Calculate the mask template:

[0083]

[0084]

[0085] in, This is the background value that needs to be subtracted, and its value is equal to the corresponding histogram. The n corresponding to the maximum value of f; Its value is equal to the donor channel histogram. When f is at its maximum, n corresponds to Its value is equal to the receptor channel histogram. When f is maximized, n corresponds to the mask template. This represents a coefficient, determined based on actual system measurements; in this system, it is typically set to 3.

[0086] S405-S406 and GPU perform parallel E-FRET data processing to obtain the FRET donor-acceptor concentration ratio under structured light super-resolution. and FRET apparent efficiency :

[0087]

[0088]

[0089]

[0090] wherein, is the fluorescence intensity of the acceptor sensitized emission, G is the sensitized quenching conversion factor, K is the donor-acceptor concentration conversion factor, a, b, c, d are the system crosstalk coefficients.

[0091] Specifically, the sensitized quenching conversion factor G, the donor-acceptor concentration conversion factor K, and the system crosstalk coefficients a, b, c, d can be determined by preparing two different standard plasmid samples with different fixed FRET efficiencies and a donor-acceptor concentration ratio of 1:1, and separately transfecting the donor and acceptor plasmid samples. For different FRET donor-acceptor and FRET measurement systems, the system parameters a, b, c, d, G, K, etc. are also different. In this embodiment, a = 0.080625, b = 0.015753, c = 0.066790, d = 0.166357, G = 0.581469, and k = 1.344915 are obtained by pre-measurement.

[0092] Specifically, the GPU parallel computing histogram Figure 1 and the histogram Figure 2 When the histogram is calculated, the array storing the histogram is modified by all threads. In order to prevent interference between threads, the relationship between threads must be coordinated when the threads update the output elements, and atomic operations are usually used to serialize the update of each element. Atomic operations solve the thread competition problem, but cause a decrease in parallelism.

[0093] Specifically, the local histogram technique is used in this embodiment to solve the output interference problem in parallel computing. The highly competitive output data structure is transferred to the private registers of the threads, so that each thread subset can update its private registers. After the calculation is completed, the private registers need to be merged into the shared memory of the corresponding thread block. As shown in Figure 7 When the histogram is calculated, each thread block has a private local histogram array, and the local histogram array is calculated first and then merged together. After testing and balancing the thread competition degree and the merging cost, the fastest speed is achieved when a thread grid contains 256 thread blocks and a thread block contains 512 threads.

[0094] Specifically, the global memory transfers data in different regions to the corresponding private local histogram array. In order to maximize the bandwidth of the global memory, grid-stride loops are used here to implement the access to the global memory, that is, the image partitioning method uses the interleaved partitioning method instead of the regular block partitioning method, as shown in Figure 5The diagram shows histogram block partitioning, which divides elements in memory into contiguous blocks of 4 elements each, and assigns each block to a thread. This type of partitioning is called block partitioning. When block partitioning is applied to a GPU, the performance of the program is largely limited by the bandwidth of the global memory because accesses to global memory cannot be merged. Figure 6 It is a histogram with staggered partitions, and the size of the staggered partitions in the figure is shown. Before the first iteration of memory processing The second iteration processes the following elements. This partitioning method, where there are multiple elements, is called staggered partitioning. For example... Figure 7 S504-S506 is a grid stepping cycle, and its total image pixels are The size of the staggered partition is The number of partitions is .

[0095] Specifically, when multiple threads access the same location in the same private local histogram array through atomic operations, contention arises, leading to a decrease in parallelism. Since this embodiment processes biological microscopic images, the grayscale values ​​in local regions of these images change continuously and gradually, with small fluctuations. Therefore, this characteristic of biological microscopic images is utilized to aggregate multiple local shared memory accesses into a single access, thereby reducing thread contention and improving the computational throughput of the parallel histogram. Figure 7 In the S505, if a thread reads pixels with the same value consecutively, it will first update the thread's private register instead of directly updating the local histogram. The local histogram will only be updated when the read pixel values ​​are different. This aggregates multiple accesses to the local histogram into one, which greatly reduces thread contention.

[0096] In order to test the GPU acceleration effect of the algorithm, taking GFP-mCherry as the donor and acceptor pair, combining the actual parameters in the above examples a=0.080625, b=0.015753, c=0.066790, d=0.166357, G=0.581469, k=1.344915, the existing SIM-FRET system is used to image test the human breast cancer cell (Michigan Cancer Foundation-7, MCF-7) sample expressing the FRET standard plasmid (ActA-G17M) targeting the outer membrane of mitochondria, 27 original images of FRET three channels with a size of 512*512 are obtained, and the original images are reconstructed into SIM-FRET images. The test platform is: Inteli5-12450H CPU 4.4GHz, NVIDIA Geforce RTX 3060 Laptop GPU, memory 16GB, Visual Studio 2019, cuda12.0.

[0097] As shown in Figure 8 , it is a FRET three-channel image comparison, in which (a) is a FRET three-channel wide-field image, (b) is a FRET three-channel super-resolution image calculated by a traditional CPU algorithm, and (c) is a FRET three-channel super-resolution image calculated by a GPU algorithm. The resolution of the wide-field image is only about 200 nm, and the resolution of the super-resolution images calculated by the CPU and the GPU is about 90-120 nm. The comparison image shows that the resolution of the wide-field image is low and relatively blurred. The structural similarity of the FRET three-channel super-resolution images calculated by the CPU and the GPU is compared, the structural similarity of the DD channel image is 0.9605, the structural similarity of the DA channel image is 0.9565, and the structural similarity of the AA channel image is 0.9711. The results show that the FRET three-channel super-resolution images calculated by the GPU and the CPU are relatively similar.

[0098] As shown in Figure 9 , it is a FRET quantitative analysis result comparison, in which (a) is the FRET efficiency Ed and the donor and acceptor concentration ratio Rc of the wide-field image, (b) is the FRET efficiency Ed and the donor and acceptor concentration ratio Rc calculated by the traditional CPU algorithm, and (c) is the FRET efficiency Ed and the donor and acceptor concentration ratio Rc calculated by the GPU algorithm. The analysis results show that the accuracy of the FRET efficiency calculated by the GPU algorithm and the CPU algorithm is the same, and the average donor and acceptor concentration ratio avRc error is only a few ten-thousandths. Within the allowable error range, the results can be considered consistent.

[0099] The comparison results of the calculation time of the method of the application and the traditional CPU-based super-resolution SIM-FRET reconstruction algorithm are shown in Table 1:

[0100] Table 1

[0101]

[0102] It can be seen that the GPU algorithm has almost the same accuracy as the SIM-FRET image calculated by the traditional CPU algorithm, and the calculation speed is increased by about 36 times.

[0103] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration, characterized in that, Includes the following steps: S101. Use the acquisition card to acquire and store the original FRET images modulated by structured light, one image at a time. S102. Perform program initialization, including CPU-side program memory allocation and GPU-side CUDA environment initialization. S103. Acquire the raw image of the FRET single channel and asynchronously transmit it to the GPU global memory; where the FRET single channel is one of the three channels: donor-excited donor emission channel DD, donor-excited acceptor emission channel DA, and acceptor-excited acceptor emission channel AA. S104. Parallel image preprocessing is performed on the FRET single-channel original image group using GPU; the FRET single-channel original image is first subjected to two-dimensional Fourier transform, then separated according to phase difference to obtain the separated image, and the separated image is normalized to obtain the normalized separated image. S105. The normalized separated image is subjected to iterative cross-correlation parameter estimation through heterogeneous computing of CPU and GPU to estimate the modulation intensity and initial phase of the illumination light vector. S106, GPU parallel SR-SIM linear Wiener reconstruction: The GPU performs parallel decomposition of the FRET single-channel spectrum based on the estimated initial phase to obtain the decomposed image. Then, the GPU performs frequency shifting on the decomposed image based on the estimated illumination light vector to obtain the frequency-shifted image, and generates the corresponding optical transfer function in parallel. The GPU then performs linear Wiener filtering calculation based on the obtained results to obtain the FRET single-channel super-resolution image. S107. If the FRET three-channel super-resolution image reconstruction is completed, proceed to step S108; otherwise, repeat steps S103-S106. S108. Acquire super-resolution FRET three-channel image; S109. Perform E-FRET data processing on the FRET three-channel super-resolution image using the GPU based on the channel sensitization intensity measurement method.

2. The super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration according to claim 1, characterized in that, The specific process of step S102 is as follows: Import the objective lens numerical aperture, objective lens magnification, optical transfer function, and camera pixel size system parameters. Then, call the CUDA function cudaSetDevice() to select the GPU device to be used, call cudaMallocHost to allocate CPU page-locked memory, call cudaMalloc() to allocate GPU global memory, call cudaStreamCreate() to create a CUDA stream, call cufftPlan2d() to create a 2D Fourier transform handle, and call cufftSetStream() to bind the 2D Fourier transform handle to the created CUDA stream.

3. The super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration according to claim 1, characterized in that, The specific process of step S103 is as follows: By calling the CUDA function cudaMemcpyAsync(), asynchronous data transfer is initiated from CPU page-locked memory to GPU global memory. The GPU performs kernel function calculations while executing this asynchronous data transfer, allowing data transfer and kernel computation to overlap. The CUDA function cudaMemcpyAsync() transfers several FRET single-channel images from the CPU to GPU global memory in parallel and asynchronously. The FRET single-channel image is one of three types of images: donor-excited and donor-emitted channel image sets. Image group of donor-excited receptor emission channels and receptor-activated receptor emission channel imaging group ,in, The spatial coordinates of the fluorescence image pixels are represented by the original image data set. Each set of original image data contains several different structured light direction angles, and each structured light azimuth angle contains several cosine structured light illumination wide-field fluorescence images with different phase differences.

4. The super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration according to claim 1, characterized in that, In step S104, the FRET single-channel raw image group is processed by taking a single image as a unit. The GPU function `cufftExecC2C()` is called to perform parallel Fourier transform on several FRET single-channel images, resulting in several FRET single-channel raw image spectrograms. These spectrograms are then input and separated in parallel according to phase difference, yielding several separated images. Finally, parallel normalization is performed to obtain several normalized separated images. The specific implementation process is as follows. include: S1041. Perform parallel Fourier transform on several images from a single FRET channel. The specific formula is as follows: in, Indicates to The value of the pixel at frequency coordinate k obtained by performing a Fourier transform; It is a spectrum of several images from a single FRET channel; k represents the frequency coordinates of the Fourier transform; , respectively, represent donor-induced donor emission channel DD, donor-induced acceptor emission channel DA, and acceptor-induced acceptor emission channel AA, wherein, The spatial coordinates of the pixels in the fluorescence image; S1042. Separate the spectrograms of several FRET single-channel raw images according to the known 120° phase difference, as follows: Where j is an imaginary number, The pixel values ​​representing the separated images of different components and different orientations. , ; S1043. Normalize the separated image to obtain a normalized separated image. Divide each pixel by its modulus: in, This represents the pixel value of the normalized separated image.

5. The super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration according to claim 1, characterized in that, The specific process of step S105 is as follows: S201, Input the normalized separated image from the GPU parallel image preprocessing output. ; S202. For the d-th direction of the structured light, extract the normalized separation image of the corresponding direction. ; S203, Normalize and separate the image in the d-th direction using GPU parallel computation. The cross-correlation function is used to obtain the illumination light vector corresponding to the maximum value of the cross-correlation function; S204. Use the illumination light vector corresponding to the maximum value of the cross-correlation function as the initial value of the estimated illumination light vector. ; S205, The CPU normalizes and separates the image according to the d-th direction. The images are translated according to the light vector, and the separation images in different directions of the d-th direction are calculated. The overlapping area, i.e. and Overlapping areas: in, It is a weighting coefficient; It is the optical transfer function; yes After weighting coefficients The value after the restricted area; and They are and On the overlapping area Parts and Part of; S206-S213, Distributing CUDA streams in overlapping regions via OpenMP and Perform parallel cross-correlation iterations; using the initial values ​​of the illumination light vectors. As the independent variable, with For function, The first iteration is performed within ±2.5 pixels with a step size of 0.5, and all values ​​are calculated using the GPU. The value, the CPU then determines based on the independent variable. Calculate the overlapping region and Complex linear regression is used to estimate the modulation intensity and initial phase of multiple illumination light vectors, making... To reach the maximum The estimated modulation intensity and initial phase are the final values ​​of the iteration; then... The second iteration is performed with the independent variable ranging between ±0.5 pixels and a step size of 0.1, and finally... The third iteration, with the independent variable set to ±0.1 pixels and a step size of 0.02, is performed to finally estimate the sub-pixel aligned illumination vector values. The modulation intensity and initial phase of the illumination light vector are given by the following formulas: S214. Repeat steps S202-S213 three times to estimate the illumination light vectors in the three directions, as well as the modulation intensity and initial phase of the illumination light vectors.

6. The super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration according to claim 1, characterized in that, The specific process of step S106 is as follows: S301, The GPU performs parallel decomposition of the FRET single-channel spectrum based on the initial phase estimated in step S105 to obtain the decomposed image: in, The initial phase estimated in step S105, , and These represent FRET single-channel images. , and The spectrum obtained by Fourier transform; The pixel values ​​representing the separated images of different components and different orientations. , ; S302, GPU estimates the illumination light vector based on S105. Frequency-shifted image is obtained by frequency shifting the decomposed image. And generate the corresponding frequency shift optical transfer function in parallel. ; S303, GPU shifts image frequency With corresponding optical transfer function Multiply and calculate the individual optical transfer functions. Sum of squares ; S304 and GPU calculate Wiener filtering based on the previous calculation results, and then perform inverse Fourier transform to obtain the FRET single-channel super-resolution image: in, These are the grayscale values ​​of the pixels in the reconstructed FRET single-channel super-resolution image. iFFT stands for Inverse Fourier Transform. This represents the Gaussian apodization function. Represents the Wiener filter parameters. These are the spatial coordinates of the pixels in the fluorescence image.

7. The super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration according to claim 6, characterized in that, Steps S107-S108 involve repeating steps S103-S106 three times to obtain a super-resolution FRET three-channel image. , and .

8. The super-resolution SIM-FRET image reconstruction method based on GPU parallel acceleration according to claim 6, characterized in that, The specific process of step S109 is as follows: S401-S403, the GPU first processes the input super-resolution FRET three-channel image. , and Parallel histogram calculations were performed, resulting in histogram 1 and histogram 2; histogram 1 was statistically analyzed. Pixels with grayscale values ​​of 8-1500 and 8-12 are included in the first histogram bar, and pixels with grayscale values ​​of 13-16 are included in the second histogram bar, thus obtaining the histogram. This indicates that the nth column counts f pixels; histogram 2 counts... Pixels with grayscale values ​​between 0 and 5000 are included in the first histogram bar, and pixels with grayscale values ​​between 0 and 1 are included in the second histogram bar, thus obtaining the histogram. , indicating that the nth column has counted f pixels; The S404 and GPU compute the FRET three-channel image in parallel using histogram 1, and compute the mask using histograms 2 of the donor and acceptor channels. in, This is the background value that needs to be subtracted, and its value is equal to the corresponding histogram. The n corresponding to the maximum value of f; Its value is equal to the donor channel histogram. When f is at its maximum, n corresponds to Its value is equal to the receptor channel histogram. When f is maximized, n corresponds to the mask template. Represents a coefficient; S405-S406 and GPU perform parallel E-FRET data processing to obtain the FRET donor-acceptor concentration ratio under structured light super-resolution. and FRET apparent efficiency : in, denoted as α, where α is the receptor-sensitized emission fluorescence intensity, G is the sensitization quenching conversion factor, K is the donor-receptor concentration conversion factor, and a, b, c, and d are system crosstalk coefficients.

Images