Volume electron microscope microscopic image sequence registration method based on Gaussian filtering

Abstract

The invention discloses a volume electron microscope microscopic image sequence registration method based on Gaussian filtering, and belongs to the technical field of volume electron microscope images. According to the method, the 3D registration problem is regarded as a challenge of high-frequency and low-frequency superposition: natural deformation is represented as progressive low-frequency change, and nonlinear distortion is represented as sudden and violent high-frequency change. To this end, an optical flow network is employed to calculate a deformation field between adjacent slices, and a one-dimensional Gaussian filter is extended to a three-dimensional image stack to integrate neighborhood deformation fields. In this way, high-frequency distortion can be smoothed on the premise that the integrity of a natural structure is not damaged, and therefore high-precision and rapid 3D registration is achieved. And practical verification shows that the registration precision and speed of the method are better than those of the existing method, and the method is suitable for high-resolution large-scale real data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of volume electron microscope image technology, and particularly relates to a method for registering a sequence of volume electron microscope microscopic images based on Gaussian filtering. Background Art

[0002] Volume electron microscopy (vEM) is a powerful technique that can be used for high-resolution 3D visualization of biological tissues and cells, revealing complex nanoscale cellular structures and generating detailed models of macromolecular complexes. It has a wide range of applications in the fields of life science, medicine, and clinical diagnosis. The vEM workflow begins with serial section scanning of a biological sample, followed by staining and digital imaging. The generated 2D images are stitched, 3D registered, and segmented to reconstruct the biological structure, where 3D registration is a key step that can correct misalignments and distortions in sample preparation and ensure the accuracy and quality of the final 3D reconstruction. However, how to distinguish natural cell deformation from non-linear distortions introduced during sample preparation remains an urgent problem to be solved. In addition, complex non-linear distortions, error accumulation in long sequences of sections, and massive amounts of image data pose further challenges for fast and high-precision 3D registration.

[0003] Traditional 3D registration methods mainly rely on optimizing energy functions. Among them, TrakEM2 is widely used, which performs 3D registration by iteratively optimizing a spring-connected particle system modeled by a triangular spring mesh. However, it has a high computational cost and a long processing time. Gaffling et al. proposed a method based on Gauss-Seidel, which separates high-frequency and low-frequency information, but it requires multiple iterations and there is a risk of weakening natural deformation. Recently, the progress of deep learning has led to an increasing interest in CNN-based 3D registration methods. Xin et al. used a pre-trained encoder to calculate the slice similarity loss and an optical flow network to estimate the deformation field, introducing a structured regression method to minimize error accumulation; however, their method is difficult to generalize to long sequences. Liu et al. proposed a correlation-based coarse-to-fine elastic registration network that uses a residual estimator to update the displacement field, but it only focuses on the pixel differences between image pairs and is therefore vulnerable to over-registration.

[0004] Recently, Popovyc et al. proposed SEAMLeSS, a trillion-scale computational pipeline for metric learning using self-supervised convolutional networks. Their method iteratively refines the initial alignment, combines vector voting to enhance robustness, divides the image into overlapping blocks for distributed computing, and achieves global registration by combining decay transformations. Although these deep learning-based methods have improved in accuracy and speed compared to traditional methods, they often ignore the natural deformations between slices. Focusing only on pixel-level similarity between adjacent slices may inadvertently eliminate the natural deformations between them, thus destroying the axial structure of biological tissues. Summary of the Invention

[0005] The object of the present invention is to provide a method for registering a sequence of volume electron microscopy micrographs based on Gaussian filtering, aiming to correct non-linear distortions while preserving the natural morphological changes of biological samples, so as to solve the problems of slow speed, low accuracy, damage to natural deformations and difficulty in extending to long-sequence large-scale data sets in existing methods.

[0006] The present invention regards the 3D registration problem as a challenge of superposition of high-frequency and low-frequency components: natural deformations are manifested as gradual low-frequency changes, while non-linear distortions are manifested as sudden and drastic high-frequency changes. For this reason, an optical flow network is used to calculate the deformation field between adjacent slices, and a one-dimensional Gaussian filter is extended to a three-dimensional image stack to integrate the neighborhood deformation field. In this way, high-frequency distortions can be smoothed without damaging the integrity of the natural structure, thereby achieving high-precision and fast 3D registration.

[0007] To achieve the above object, the present invention is implemented through the following technical solutions:

[0008] A method for registering a sequence of volume electron microscopy micrographs based on Gaussian filtering, comprising the following steps:

[0009] S1: Obtain a stack of consecutive sequence micrographs, sequentially select multiple adjacent slices and place them within the receptive field of a Gaussian filter, and use its powerful smoothing ability to effectively eliminate non-linear distortions while preserving natural deformations;

[0010] S2: Use a cascaded pyramid optical flow estimation network to estimate the deformation field between adjacent slices;

[0011] S3: Use a Gaussian filter to integrate the deformation field within the receptive field, and then perform weighted fusion to obtain the deformation field for the current slice. Use the obtained deformation field to distort the current slice and replace the original slice in the image stack;

[0012] S4: Sequentially perform the above operations until all slices in the image stack are registered, and the estimated deformation field is integrated by a Gaussian filter to achieve precise distortion elimination.

[0013] Furthermore, in S1, in the problem of volume electron microscopy microscopic image sequence registration, the processed image stack is distorted due to non - linear deformation caused by sample slicing or optical effects. The goal of 3D registration is to determine the deformation field between a series of adjacent slices to correct these distortions and thus restore the accurate 3D structure. By using a one - dimensional Gaussian filter, the optimal deformation field Φ i is found to smooth the non - linear distortion, and the following objective function is defined:

[0014]

[0015] where I i represents the slice to be registered, N(I i ) represents the adjacent slice of I i . The function F(·) estimates the difference between the I i deformed by Φ i and the adjacent slice I j , which serves as the regularization term.

[0016] Furthermore, in S2, the cascaded pyramid optical flow estimation network includes: a pyramid encoder module, a GRU - based decoder module, and a fusion module; the encoder module adopts a four - layer pyramid structure, including two 2D convolutional encoder modules for constructing a feature pyramid and and a context feature pyramid . The output channels of each convolutional layer of the encoder are 8, 16, 32, 64 respectively; each layer of the pyramid structure also includes a GRU decoder module, which adopts a design of multiple convolutional layers, inputs the encoded features of this layer, and outputs the deformation field of this layer . The fusion module of this network is a fusion module with a U - net - like structure. This fusion module accepts the deformation field and features of the last layer of the pyramid and outputs the final deformation field Φ m,f,t ; the output channels of each convolutional layer of this fusion module are 8, 16, 32 respectively.

[0017] Furthermore, in S2, the specific calculation process of the cascaded pyramid optical flow estimation network is as follows: The network adopts a cascaded design, and the inputs are a fixed image I f and a moving image I m , and the deformation field is estimated step by step; the generation of the deformation field Φ m,f is divided into T iterations; in each iteration t, the inputs include the fixed image I f and the deformed image registered by the deformation field Φ m,f,t-1 in the previous iteration First, a context feature pyramid is constructed using a 2D convolutional encoder Feature pyramid and Among them, the feature pyramid is used to generate a deformation field for each layer, and the context feature pyramid provides prior information for field generation; the deformation field of the previous layer is upsampled and input into the next layer to obtain an updated deformation field Finally, and are together input into the fusion module to generate the final deformation field Φ of the current iteration m,f,t ; after multiple iterations, the deformation field Φ is finally generated m,f .

[0018] Furthermore, the specific content of S3 is as follows:

[0019] S3-1: First, demonstrate this step on a simple 1D model and then extend it to a 3D image stack; given a 1D signal f of length N and a Gaussian kernel w with a radius r of 1, the filtering operation at position i is expressed as:

[0020]

[0021] where w k represents the weight of the k-th element in the Gaussian kernel, f i represents the i-th value in the signal, represents the filtered value at position i;

[0022] S3-2: Let d i,j = f j - f i , for any indices i and j, then formula (2) is rewritten as:

[0023]

[0024] S3-3: Consider applying a similar 1D Gaussian filter to a 3D image stack, where each point f in the 1D signal i is similar to the image I in the image stack i ; here, d i,j represents the deformation field Φ i,j that registers the slice I i to the slice I j ; therefore, the Gaussian filtering operation of the slice I i is expressed as:

[0025]

[0026] where ° represents the warping operation;

[0027] S3-4: According to formula (4), includes the weighted contributions from the deformed I i and the deformed I i+1 , and the calculation of is changed to be as follows:

[0028]

[0029] wherein, the deformed {I i , I i+1} is called the intermediate image and its corresponding normalized weight is expressed as

[0030] S3-5: Register I i to to obtain After being weighted by , is used for the final registration to obtain

[0031] Compared with the prior art, the advantages and beneficial effects of the present invention are:

[0032] The present invention defines the 3D registration problem as a superposition problem of low-frequency components and high-frequency components. This accurate and correct modeling fully decouples natural deformation and non-linear distortion. Adopting simple but effective Gaussian filtering combined with a lightweight optical flow estimation network is conducive to achieving fast and accurate 3D registration. The present invention can quickly and accurately correct non-linear distortion without damaging natural deformation, and can avoid the problem of error accumulation in the registration of long sequence image stacks.

[0033] The present invention realizes accurate and fast 3D elastic registration by extending 1D Gaussian filtering to the field of 3D image stacks and then integrating the neighborhood deformation field estimated by the optical flow network. The present invention can eliminate non-linear distortion in the image stack and retain the natural deformation between adjacent slices, thereby restoring the correct 3D structure of the organism. And through actual verification, the registration accuracy and speed of the present invention are superior to existing methods and are applicable to high-resolution large-scale real datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is (a) a one-dimensional signal with high-frequency noise (original signal) and a Gaussian filtered signal (denoised signal). (b) A side view (y-z plane) of the original vEM data (original data) and the registered vEM data (registered data).

[0035] Figure 2 is the overall flowchart of the present invention.

[0036] Figure 3It is a schematic diagram of the optical flow network framework for estimating the optical flow between adjacent slices in the present invention.

[0037] Figure 4 It is a 3D visualization result diagram of the registration results of various methods on the mouse kidney, mouse liver, and mouse skin data sets in Example 2. Detailed implementation manners

[0038] The technical solution of the present invention will be further described and illustrated below in conjunction with embodiments.

[0039] Example 1:

[0040] In this example, the 3D registration problem is defined as a superposition problem of high-frequency components and low-frequency components, and the natural deformation and non-linear distortion are successfully decoupled in the frequency domain. As Figure 1 shown, the natural deformation presents smooth and slow changes, while the non-linear distortion is often rapid and fine jitter.

[0041] The present invention proposes a method for registering a sequence of volume electron microscopy images based on Gaussian filtering, which combines an optical flow estimation network to achieve fast and accurate 3D registration. The specific flow chart is as Figure 2 shown. Given a continuous sequence of microscopic image stacks, first, multiple adjacent slices are sequentially selected and placed within the receptive field of a Gaussian filter. Then, an optical flow network is used to estimate the deformation field between adjacent slices. Next, the Gaussian filter is used to integrate the deformation field within the receptive field, and the deformation field for the current slice is obtained through weighted fusion. The current slice is distorted using this deformation field and replaced with the original slice in the image stack. Finally, the above operations are sequentially executed until the registration of all slices in the image stack is completed. Specifically, it includes:

[0042] I. In the 3D elastic registration problem, the processed image stack is distorted due to non-linear deformation caused by sample slicing or optical effects. The goal of 3D registration is to determine the deformation field between a series of adjacent slices to correct these distortions and thus restore the accurate 3D structure. To this end, the deformation field Φ i not only needs to effectively eliminate non-linear distortion but also maintain the natural deformation in the axial direction. From the perspective of the frequency domain, this problem is processed by considering the axial natural deformation as the low-frequency component and the non-linear distortion as the high-frequency component. This perspective can find the optimal deformation field Φ i by using the idea of a one-dimensional Gaussian filter to smooth the non-linear distortion. Based on this method, the following objective function is defined:

[0043]

[0044] where I i represents the slice to be registered, N(Ii ) represents I i 's adjacent slices, and the function F(·) estimates the difference between the deformed I i through φ i and the adjacent slice I j as a regularization term. As a regularization term.

[0045] Second, this embodiment uses an optical flow estimation network to calculate the deformation field between slices. The network architecture is as Figure 3 shown. The network mainly includes three major modules: a pyramid encoder module, a GRU-based decoder module, and a fusion module. The encoder module adopts a four-layer pyramid structure, including two 2D convolutional encoder modules to construct a feature pyramid and and a context feature pyramid The output channels of each convolutional layer of the encoder are 8, 16, 32, and 64 respectively. Each layer of the pyramid also includes a GRU decoder module, which adopts a design of multiple convolutional layers, inputs the encoded features of this layer, and outputs the deformation field of this layer In addition, the network also includes a fusion module with a U-net-like structure, which accepts the deformation field and features of the last layer of the pyramid and outputs the final deformation field Φ m,f,t . The output channels of each convolutional layer of this fusion module are 8, 16, and 32 respectively.

[0046] The specific calculation process of this network is as follows: The network adopts a cascade design, and the inputs are the fixed image I f and the moving image I m , and gradually estimates the deformation field. The generation of the deformation field Φ m,f is divided into T iterations. In each iteration t, the inputs include the fixed image I f and the deformed image registered through the deformation field Φ m,f,t-1 in the previous iteration First, use a 2D convolutional encoder to construct a context feature pyramid Feature pyramid and Among them, the feature pyramid is used to generate the deformation field for each layer, while the context feature pyramid provides prior information for field generation. The deformation field of the previous layer is upsampled and input to the next layer to obtain the updated deformation field Finally, and are input to the fusion module together to generate the final deformation field Φ m,f,t of the current iteration. After multiple iterations, the final deformation field Φ m,f is finally generated.

[0047] III. 3D Elastic Registration Based on Gaussian Filtering

[0048] In the step of integrating the deformation field within the receptive field, the idea of Gaussian filtering is proposed to effectively weight and fuse the deformation field. First, this step is demonstrated on a simple 1D model and then extended to 3D image stacks. Given a 1D signal f of length N and a Gaussian kernel w with a radius r of 1, the filtering operation at position i can be expressed as:

[0049]

[0050] where w k represents the weight of the k-th element in the Gaussian kernel, f i represents the i-th value in the signal, represents the filtered value at position i.

[0051] Let d i,j = f j - f i , for any indices i and j, then formula (2) can be rewritten as:

[0052]

[0053] Consider applying a similar 1D Gaussian filter to 3D image stacks, where each point f in the 1D signal i is similar to the image I in the image stack i . Here, d i,j represents the deformation field Φ i,j that registers slice i i to slice i j . Therefore, the Gaussian filtering operation for slice i i can be expressed as:

[0054]

[0055] where ° represents the warping operation.

[0056] According to formula (4), it can be observed that contains the weighted contributions from the deformed I i and the deformed i i+1 , which inevitably introduces the image information from i i+1 . To retain only the true information from i i , we adjusted formula (4). Specifically, we use the weighted deformation field calculated between multiple adjacent slices to deform i i , rather than directly using the information of adjacent slices. The deformation field of I i is calculated based on the registered adjacent slices. Therefore, The calculation is changed as follows:

[0057]

[0058] Among them, the deformed {I i , I i+1} is called the intermediate image and its corresponding normalized weight is denoted as Then, I i is registered to to obtain After passing through and being weighted, is used for the final registration to obtain

[0059] IV. Finally, the above operations are sequentially performed on each slice of the entire image stack until the registration of all slices is completed.

[0060] Example 2:

[0061] To prove the effectiveness of the method of the present invention, the method of the present invention was compared with four widely used 3D registration methods, including a commonly used traditional method TrakEM2 and three recent deep learning methods EMReg, EFSR, and SEMLeSS. By statistically analyzing the NCC(GT), SSIM(GT), and MI(GT) between the registered image stacks of each method and the ground truth, the performance of 3D elastic registration was evaluated from different perspectives.

[0062] The method of the present invention was evaluated on three publicly available datasets on the OpenOrganelle platform. These datasets include electron microscopy images of various tissues such as mouse kidneys, mouse livers, and mouse skins. These images were cropped into slices of size 1184×1184 px and randomly elastic deformations were added to simulate non-linear distortions. These datasets provide diverse and representative biological samples for comprehensively evaluating the applicability and robustness of the method of the present invention in different biological tissues. As shown by the quantitative results in Table 1, the method of the present invention consistently showed excellent performance in all metrics for all datasets. In addition, Figure 4 The 3D visualization results of various methods on the three datasets are shown. It can be seen that the method of the present invention achieved better and smoother registration results, closest to the true value. This is particularly evident in the mouse skin dataset, which involves cell structures with obvious inter-slice gaps. In this case, the method of the present invention demonstrated that even for structures with a large span, the method can provide smooth and continuous registration results, highlighting its advantage in long-sequence registration.

[0063] Table 1. Quantitative results of various methods on mouse kidney, mouse liver, and mouse skin datasets

[0064]

[0065] Finally, although this specification is described in terms of embodiments, not every embodiment contains only an independent technical solution. This narrative style of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A volume electron microscope image sequence registration method based on Gaussian filtering, characterized in that: The steps include: S1: Obtain a continuous microscopic image stack, sequentially select multiple adjacent slices and place them within the receptive field of the Gaussian filter; S2: Estimate the deformation field between adjacent slices using a cascaded pyramid optical flow estimation network; S3: Use a Gaussian filter to integrate the deformation field in the receptive field, and then perform weighted fusion to obtain the deformation field for the current slice. Use the obtained deformation field to distort the current slice and replace the original slice in the image stack. S4: The above operations are performed sequentially until all slices in the image stack are registered, and the estimated deformation field is integrated by a Gaussian filter to achieve accurate distortion elimination.

2. The volume electron microscope image sequence registration method according to claim 1, characterized in that: In S1, in the problem of volume electron microscopy image sequence registration, the image stack processed Distortion occurs due to nonlinear deformation caused by sample slicing or optical effects. By using a one-dimensional Gaussian filter, the optimal deformation field Φ is found. i , to smooth the nonlinear distortion, the following objective function is defined: Among them, I i represents the slice to be registered, N(I i ) indicates I i The adjacent slices of the function F(·) are estimated by Φ i Deformed I i With adjacent slice I j The difference between as a regularization term.

3. The volume electron microscope image sequence registration method according to claim 1, characterized in that: In S2, the cascade pyramid optical flow estimation network includes: a pyramid encoder module, a GRU-based decoder module and a fusion module; the encoder module adopts a four-layer pyramid structure, including two 2D convolution encoder modules to construct a feature pyramid and and context feature pyramid Each layer of the pyramid structure also includes a GRU decoder module, which uses a design of multiple convolutional layers to input the encoding features of the layer and output the deformation field of the layer. The fusion module of the network is a fusion module of a U-net structure, which accepts the deformation field and feature of the last layer of the pyramid. Output the final deformation field Φ m,f,t .

4. The volume electron microscope image sequence registration method according to claim 1, characterized in that: In S2, the specific calculation process of the cascade pyramid optical flow estimation network is as follows: The network input is a fixed image I f and moving image I m , and gradually estimate the deformation field; the deformation field Φ m,f The generation of is divided into T iterations; At each iteration t, the input consists of a fixed image I f and the deformation field Φ in the previous iteration m,f,t-1 Deformed image after registration First, use a 2D convolutional encoder to build a context feature pyramid Feature Pyramid and Among them, the feature pyramid is used to generate the deformation field for each layer, and the context feature pyramid provides the prior information for field generation; the deformation field of the previous layer is upsampled and input to the next layer to obtain the updated deformation field final, and Input together into the fusion module to generate the final deformation field Φ of the current iteration m,f,t ; After multiple iterations, the deformation field Φ is finally generated m,f .

5. The volume electron microscope image sequence registration method according to claim 1, characterized in that: The S3 is specifically: S3-1: We first demonstrate the procedure on a simple 1D model and then extend it to a 3D image stack; given a 1D signal f of length N and a Gaussian kernel w of radius r 1, the filtering operation at position i is expressed as: where w k represents the weight of the kth element in the Gaussian kernel, f i represents the i-th value in the signal, represents the filtered value at position i; S3-2: Let d i,j =f j -f i , for any index i and j, formula (2) can be rewritten as: S3-3: Consider a similar 1D Gaussian filter applied to a 3D image stack, where each point f in the 1D signal i Similar to image I in the image stack i ;d i,j Denotes the deformation field Φ i,j , the deformation field will slice I i Register to Slice I j ; Therefore, slice I i The Gaussian filtering operation is expressed as: Where ° represents the twist operation; S3-4: According to formula (4), Contains the transformed I i and the deformed I i+1 The weighted contribution of The calculation is changed to the following: Among them, the deformed {I i ,I i+1 } is called the intermediate image And its corresponding normalized weight is expressed as S3-5: I i Register to To obtain In passing After weighting, is used for the final registration and obtains

Images