Tandem electron microscope three-dimensional registration trajectory optimization method based on Shenchang differential equation

Abstract

The invention discloses a string-slice electron microscope three-dimensional registration trajectory optimization method based on a Shenxual differential equation, and belongs to the technical field of electron microscope three-dimensional reconstruction. According to the method, each pixel is regarded as a particle in a continuous medium, and a single-particle trajectory optimization objective function and a global consistency trajectory constraint objective function are set; estimating a displacement field between adjacent slices by using a two-dimensional U-shaped convolutional neural network, and constructing a time sequence track body of each pixel; performing trajectory smoothing in combination with Gaussian-Seidel iteration and a Sheng differential equation; and inverting the optimized trajectory into a displacement field on a pixel grid, carrying out network training based on a final objective function, and finally realizing registration of the image stack. According to the method, unsupervised training is completely adopted, the method is particularly suitable for a series-film electron microscope scene with scarce annotations, and a large number of experiments carried out on a plurality of data sets covering different tissue types show that the method is superior to an existing method in the aspects of structure recovery precision and cross-tissue robustness.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology of electron microscopy, and specifically to an unsupervised trajectory optimization method for three-dimensional registration of serial electron microscopes based on neural ordinary differential equations. Background Technology

[0002] Serial electron microscopy has become a powerful nanoscale three-dimensional visualization technique, widely used in fields such as neuroconnectomics, developmental biology, and clinical diagnostics. The standard workflow of serial electron microscopy typically includes computational steps such as two-dimensional image stitching, three-dimensional registration, and three-dimensional segmentation. Among these, three-dimensional registration is particularly critical, as it corrects axial misalignment and nonlinear distortions generated during slicing, directly determining the accuracy and reliability of the final three-dimensional structure.

[0003] The fundamental challenge of accurate 3D registration lies in the topological entanglement between biological signals and technical noise—that is, the difficulty in distinguishing between natural cell morphological deformations and nonlinear distortions introduced during sample preparation. The blurred boundaries between these deformation types significantly increase the difficulty of recovering the true 3D structure of the sample, making 3D registration a highly complex and critical task. Furthermore, with the continuous advancement of slicing techniques, modern equipment can generate trillions of image data points daily. This massive data volume and the accumulation of errors in long sequences further exacerbate the complexity and computational burden of 3D registration.

[0004] Several effective methods have been proposed. For example, 3D registration can be achieved by optimizing a particle system connected by springs (modeled as a triangular mesh) or by using self-supervised networks for metric learning. However, these methods often ignore the natural deformation between slices and overemphasize pixel-level similarity, thus affecting the realism of the reconstruction results. Summary of the Invention

[0005] The purpose of this invention is to provide a method for optimizing the three-dimensional registration trajectory of a series electron microscope based on neural ordinary differential equations, so as to overcome the shortcomings of the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A method for optimizing the 3D registration trajectory of a series-slice electron microscope based on the ordinary differential equation of the neuron is proposed. The 3D registration problem is redefined as a continuous manifold trajectory optimization problem, and includes the following steps: S1: Treat each pixel as a particle in a continuous medium, and set a single-particle trajectory optimization objective function and a globally consistent trajectory constraint objective function; S2: Use a two-dimensional U-shaped convolutional neural network to estimate the displacement field between adjacent slices and construct the temporal trajectory volume of each pixel; S3: Combining Gauss-Seidel iteration with the God ordinary differential equation for trajectory smoothing; S4: The optimized trajectory is inverted into a displacement field on the pixel grid, and the network is trained based on the final objective function to finally achieve image stack registration.

[0007] Furthermore, S1 specifically includes: S1-1: The trajectory of a single particle is defined as follows: ; in Represents the original trajectory. It is an axial Laplacian operator; S1-2: Define the objective function for globally consistent constraints: ; in It is the total number of trajectories. It is the spatial consistency coefficient, used to maintain the topological relationship between neighboring pixels within a slice, and is defined as: .

[0008] Furthermore, in step S2, a trajectory volume is introduced to model the trajectory of each pixel. ,in Used to store slices The spatial coordinates and trajectory are initialized as follows: ; Next, the displacement field of the previous slice is recursively propagated to determine the position, and the sub-position flow is estimated using bilinear interpolation. Trajectory propagation is then performed using the transport equation: ; in This represents the interpolated displacement field.

[0009] Furthermore, in S2, the two-dimensional U-shaped network structure consists of two parts: a hybrid convolutional feature encoder and an upsampled dense feature decoder. The encoder passes through four convolutional layers with a stride of 2, and the number of channels is 8, 16, 32, and 64 respectively. Each convolutional layer includes two convolution operations, and the kernel sizes are as follows: and The decoder consists of four dense transposed convolutional layers with kernel sizes of 2, strides of 2, and channel sizes of 64, 32, 16, and 8 respectively. The final displacement field is generated by dual... Convolutional kernels and the Softsign activation function are used for generation. Image registration utilizes a spatial transformation network to create spatial twisting. The Adam optimizer is used with the following parameters: , The learning rate is There are a total of 500 rounds, with each batch size being 16. Smoothness parameter. The Gauss-Seidel method iterates 200 times. .

[0010] Furthermore, the process in S3 involves modeling trajectory smoothing as a weighted minimization of discrete curvature energy and the original trajectory deviation, with the objective of minimizing the deviation between the discrete curvature energy and the original trajectory. To solve this optimization objective, an iterative trajectory smoothing method is employed.

[0011] Furthermore, S3 specifically includes: S3-1: The Gauss-Seidel method uses the latest updated value immediately in each iteration to accelerate convergence. Its iterative formula is as follows: ; Among them, the boundary value is fixed as ; in This refers to the total number of slices and their positions. Each iteration starts from 1 to... ; S3-2: Compared to the Gauss-Seidel method, the God ordinary differential equations perform better in modeling continuous dynamic systems. They can effectively capture complex nonlinear deviations through implicit dynamic modeling, satisfying the following formula: ; in, It is a neural network with trainable parameters.

[0012] S3-3: The trajectory smoothing is reformulated as a dynamic process of trajectory evolution over time, and the explicit iterative convergence of Gauss-Seidel is organically integrated with the implicit dynamic modeling capability of the God ordinary differential equation: ; in It is the Gauss-Seidel operator, with the initial time... The trajectory is taken as input, with The time-smoothed trajectory is the output. ODESolver represents the neural dynamic correction module. These are adaptive weights that evolve over time, and the evolution and update method is as follows: ; in This represents the activation function used for smoothing. Controlling response rate sensitivity. The distance between the smoothed trajectory and the original trajectory is represented by: ; The differential equations used in the study employ a 3D U-shaped network, with parameters consistent with the kernel of the trajectory tracking network. The training batch size is 4, corresponding to the parameters of the objective function. , Other parameters remain the same as the default.

[0013] Furthermore, S4 includes: S4-1: To transform the smoothed trajectory into a discrete displacement field, the trajectory is optimized to obtain the following trajectory. Each of the components Represents a slice The trajectory coordinates The displacement vector of the trajectory is ; S4-2: Using bilinear scatter interpolation, the trajectory of each pixel is inverted into integer grid point positions: ; This represents the bilinear scatter interpolation method. Representation of sliced ​​images Related deformation field; S4-3: The image registration is obtained by performing a deformation operation on the deformation field and the slice image as follows: ; S4-4: The final objective function is: .

[0014] Compared with the prior art, the present invention has at least the following beneficial effects:

[0015] 1. This invention analyzes the formation mechanism and intrinsic characteristics of nonlinear distortion from the perspective of biological motion consistency. Based on this, three-dimensional registration is redefined as a continuous manifold trajectory optimization problem, and a continuous trajectory dynamics modeling method for three-dimensional registration is proposed for the first time.

[0016] 2. This invention employs unsupervised training entirely, without relying on manual annotation, making it particularly suitable for serial electron microscopy scenarios where annotations are scarce.

[0017] 3. Extensive experiments conducted on multiple datasets covering different tissue types demonstrate that this method outperforms existing methods in both structural recovery accuracy and cross-tissue robustness. Attached Figure Description

[0018] Figure 1 A diagram comparing the principles of this invention with existing methods.

[0019] Figure 2 This is the overall flowchart of the present invention.

[0020] Figure 3 This is a visualization comparison of the present invention with other methods on skin and pancreas datasets.

[0021] Figure 4 This is a visualization comparison of the present invention with other methods on a liver dataset.

[0022] Figure 5 This is a visualization comparing the present invention with other methods on heart and kidney datasets. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description, in conjunction with specific embodiments and accompanying drawings, further illustrates the invention. Obviously, the described embodiments are only a portion, not all, of the embodiments disclosed in this invention. All other embodiments obtained by those skilled in the art based on the embodiments disclosed in this invention without inventive effort are within the scope of protection of this invention.

[0024] This invention, starting from the fact that anatomical structures typically exhibit smooth manifold characteristics and maintain evolutionary consistency during morphological transformation, and that nonlinear distortion is essentially caused by abrupt changes in local curvature due to mechanical slicing, proposes a paradigm shift optimization framework. This framework redefines 3D registration as a motion trajectory optimization problem, preserving local structural information while maintaining overall trajectory smoothness, and systematically integrating biophysical priors with data-driven deformation compensation mechanisms. The motivation for this invention can be found in [link to invention]. Figure 1 Comparison with previous methods and motivation for the present invention: (a) Previous methods focused on local registration of adjacent slices, while the present invention improves upon this by introducing a global trajectory optimization that distinguishes the position, velocity, acceleration and curvature of the trajectory before and after smoothing; (b) The figure shows the curvature change of the original slice and the slice with nonlinear twist. Based on this observation, the present invention explores the possible internal correlation between nonlinear twist and trajectory curvature abrupt change.

[0025] Example 1 The overall process of this invention is as follows: Figure 2 The following technical solution is adopted: An unsupervised trajectory optimization method for 3D registration in serial electron microscopy based on neural ordinary differential equations redefines the 3D registration problem as a continuous manifold trajectory optimization problem, including the following steps: S1: Treat each pixel as a particle in a continuous medium, and set a single-particle trajectory optimization objective function and a globally consistent trajectory constraint objective function; specifically including: S1-1: The trajectory of a single particle is defined as follows: ; in Represents the original trajectory. It is an axial Laplacian operator; S1-2: Define the objective function for globally consistent constraints: ; in It is the total number of trajectories. It is the spatial consistency coefficient, used to maintain the topological relationship between neighboring pixels within a slice, and is defined as: .

[0026] S2: A two-dimensional U-shaped convolutional neural network is used to estimate the displacement field between adjacent slices and construct the temporal trajectory volume for each pixel; to model the trajectory of each pixel, a trajectory volume is introduced. ,in Used to store slices The spatial coordinates and trajectory are initialized as follows: ; Next, the displacement field of the previous slice is recursively propagated to determine the position, and the sub-position flow is estimated using bilinear interpolation. Trajectory propagation is then performed using the transport equation: , in This represents the interpolated displacement field.

[0027] The U-shaped convolutional neural network structure consists of a hybrid convolutional feature encoder and an upsampled dense feature decoder. The encoder consists of four convolutional layers with a stride of 2, and the number of channels is 8, 16, 32, and 64 respectively. Each convolutional layer includes two convolution operations, and the kernel size is as follows: and The decoder consists of four dense transposed convolutional layers with kernel sizes of 2, strides of 2, and channel sizes of 64, 32, 16, and 8 respectively. The final displacement field is generated by dual... Convolutional kernels and the Softsign activation function are used for generation. Image registration utilizes a spatial transformation network to create spatial twisting. The Adam optimizer is used with the following parameters: , The learning rate is There are a total of 500 rounds, with each batch size being 16. Smoothness parameter. The Gauss-Seidel method iterates 200 times. .

[0028] S3: Trajectory smoothing is performed by combining Gauss-Seidel iteration with the God constant differential equation. Trajectory smoothing is modeled as a weighted minimization of discrete curvature energy and the original trajectory deviation, with the objective of minimizing the deviation between the discrete curvature energy and the original trajectory. To solve this optimization objective, an iterative trajectory smoothing method based on the Gauss-Seidel method is used. Specifically, this includes: S3-1: The Gauss-Seidel method uses the latest updated value immediately in each iteration to accelerate convergence. Its iterative formula is as follows: ; The boundary values ​​are fixed as follows: ; in This refers to the total number of slices and their positions. Each iteration starts from 1 to... .

[0029] S3-2: Compared to the Gauss-Seidel method, the God ordinary differential equations perform better in modeling continuous dynamic systems. They can effectively capture complex nonlinear deviations through implicit dynamic modeling, satisfying the following formula: ; in, It is a neural network with trainable parameters.

[0030] S3-3: This invention reformulates trajectory smoothing as a dynamic process of trajectory evolution over time, and organically integrates the explicit iterative convergence of Gauss-Seidel with the implicit dynamic modeling capability of the God ordinary differential equation: ; in It is the Gauss-Seidel operator, with the initial time... The trajectory is taken as input, with The time-smoothed trajectory is the output. ODESolver represents the neural dynamic correction module. These are adaptive weights that evolve over time, and the evolution and update method is as follows: ; in This represents the activation function used for smoothing. Controlling response rate sensitivity. The distance between the smoothed trajectory and the original trajectory is represented by: .

[0031] The differential equations used in the study employ a 3D U-shaped network, with parameters consistent with the kernel of the trajectory tracking network. The training batch size is 4, corresponding to the parameters of the objective function. , Other parameters remain the same as the default.

[0032] S4: The optimized trajectory is inverted into a displacement field on a pixel grid and used for image stack registration; and network training is performed based on the final objective function. In S4, to transform the smoothed trajectory into a discrete displacement field, the trajectory optimization yields the following trajectory: Each of the components Represents a slice The trajectory coordinates The displacement vector of the trajectory is Bilinear scatter interpolation is used to invert the trajectory of each pixel into integer grid point positions: ; This represents the bilinear scatter interpolation method. Representation of sliced ​​images The relevant deformation field. The image registration is obtained by performing deformation operations on the deformation field and the sliced ​​image as follows: ; And the final objective function is: .

[0033] Example 2:

[0034] This embodiment is based on Example 1 and performs specific experimental verification, utilizing six publicly available datasets collected from the OpenOrganelle platform, including mouse heart, kidney, liver, skin, and pancreatic tissues.

[0035] Table 1 compares the present invention with existing 3D image registration methods. Here, MI represents interaction information, which measures the statistical dependence between the grayscale distributions of images; the higher the degree of registration between two images, the larger the MI coefficient. SSIM represents structural similarity index, which measures the structural similarity of images; the larger the SSIM coefficient, the more similar the images are in visual quality. NCC represents normalized cross-correlation, which measures the similarity at the pixel level; the larger the NCC coefficient, the higher the pixel similarity. Table 1 shows that the present invention outperforms previous methods in comparisons across various datasets and metrics.

[0036] Table 1: Performance of different 3D registration methods on six datasets .

[0037] In addition to comparing the inventive performance, the present invention was also compared in terms of visualization, and the results are as follows: Figure 3-5The present invention is superior to existing methods in terms of visualization, and the image texture is restored more smoothly and delicately while maintaining the original image texture and grayscale.

[0038] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing the three-dimensional registration trajectory of a serial electron microscope based on neural ordinary differential equations, characterized in that, The optimization method includes the following steps: S1: Treat each pixel as a particle in a continuous medium, and set a single-particle trajectory optimization objective function and a globally consistent trajectory constraint objective function; S2: Use a two-dimensional U-shaped convolutional neural network to estimate the displacement field between adjacent slices and construct the temporal trajectory volume of each pixel; S3: Combining Gauss-Seidel iteration with the God ordinary differential equation for trajectory smoothing; S4: The optimized trajectory is inverted into a displacement field on the pixel grid, and the network is trained based on the final objective function to finally achieve image stack registration.

2. The method for optimizing the three-dimensional registration trajectory of a serial electron microscope as described in claim 1, characterized in that, S1 specifically includes: S1-1: The trajectory of a single particle is defined as follows: ； in Represents the original trajectory. It is an axial Laplacian operator; S1-2: Define the objective function for globally consistent constraints: ； in It is the total number of trajectories. It is the spatial consistency coefficient, used to maintain the topological relationship between neighboring pixels within a slice, and is defined as: 。 3. The method for optimizing the three-dimensional registration trajectory of a serial electron microscope as described in claim 1, characterized in that, In step S2, a trajectory volume is introduced to model the trajectory of each pixel. ,in Used to store slices The spatial coordinates and trajectory are initialized as follows: ； Next, the displacement field of the previous slice is recursively propagated to determine the position, and the sub-position flow is estimated using bilinear interpolation. Trajectory propagation is then performed using the transport equation: ； in This represents the interpolated displacement field.

4. The method for optimizing the three-dimensional registration trajectory of a serial electron microscope as described in claim 1, characterized in that, In S2, the two-dimensional U-shaped network structure consists of a hybrid convolutional feature encoder and an upsampled dense feature decoder. The encoder passes through four convolutional layers with a stride of 2, and the number of channels is 8, 16, 32, and 64 respectively. Each convolutional layer includes two convolution operations, and the kernel size is as follows: and ; The decoder consists of four dense transposed convolutional layers with kernel sizes of 2, strides of 2, and channel sizes of 64, 32, 16, and 8 respectively; the final translation field is composed of dual... Convolutional kernels and Softsign activation functions are used for generation; image registration uses a spatial transformation network to create spatial twist.

5. The method for optimizing the three-dimensional registration trajectory of a serial electron microscope as described in claim 1, characterized in that, The process of S3 is to model trajectory smoothing as a weighted minimization of discrete curvature energy and original trajectory deviation, with the goal of minimizing the deviation between discrete curvature energy and original trajectory. To solve this optimization objective, the Gauss-Seidel method is used for iterative trajectory smoothing.

6. The method for optimizing the three-dimensional registration trajectory of a serial electron microscope as described in claim 5, characterized in that, S3 specifically includes: S3-1: The Gauss-Seidel method uses the latest updated value immediately in each iteration to accelerate convergence. Its iterative formula is as follows: ； The boundary values ​​are fixed as follows: ； in This refers to the total number of slices and their positions. Each iteration starts from 1 to... ; S3-2: Compared to the Gauss-Seidel method, the neural ordinary differential equation can effectively capture complex nonlinear deviations through implicit dynamic modeling, satisfying the following formula: ； in, It is a neural network with trainable parameters; S3-3: The trajectory smoothing is reformulated as a dynamic process of trajectory evolution over time, and the explicit iterative convergence of Gauss-Seidel is organically integrated with the implicit dynamic modeling capability of the God ordinary differential equation: ； in It is the Gauss-Seidel operator, with the initial time... The trajectory is taken as input, with The time-smoothed trajectory is the output; ODESolver represents the neural dynamic correction module. These are adaptive weights that evolve over time, and the evolution and update method is as follows: ； in Indicates the activation function; Controlling response rate sensitivity; The distance between the smoothed trajectory and the original trajectory is represented by: ； The differential equations used by the God often employ a three-dimensional U-shaped network, with parameters consistent with the kernel of the trajectory tracking network.

7. The method for optimizing the three-dimensional registration trajectory of a serial electron microscope as described in claim 1, characterized in that, S4 includes: S4-1: To transform the smoothed trajectory into a discrete displacement field, the trajectory is optimized to obtain the following trajectory. Each of the components Represents a slice The trajectory coordinates The displacement vector of the trajectory is ; S4-2: Using bilinear scatter interpolation, the trajectory of each pixel is inverted into integer grid point positions: ； This represents the bilinear scatter interpolation method. Representation of sliced ​​images Related deformation field; S4-3: The image registration is obtained by performing a deformation operation on the deformation field and the slice image as follows: ； S4-4: The final objective function is: 。

Images