A Cost-Volume-Based Electron Microscopy Image Registration Method

Abstract

The present invention discloses a cost volume-based electron microscope image registration method, including: first, constructing a multi-level feature extraction and feature enhancement network branch; second, estimating the deformation field residual step by step and updating the prediction result of the deformation field, including: cost volume calculation, deformation field residual estimation, and deformation field update. The present invention can encode more globally semantic information in the image, thereby improving the processing ability for large deformations in electron microscope images and achieving an improvement in the registration accuracy of electron microscope images.

Description

Technical Field

[0001] The present invention relates to the technical field of electron microscope image processing, and particularly to an electron microscope image registration method based on cost volume. Background Art

[0002] With the development of biotechnology and brain science, the importance of exploring the mesoscopic structure of biological tissues has become increasingly prominent. As an important technology for obtaining nanoscale image data of biological tissues, electron microscope imaging has developed rapidly in recent years. To obtain electron microscope images, the general procedure is to first cut a biological sample into ultra-thin sections with a nanoscale thickness, and then image the sections with an electron microscope to obtain image data. Since biological tissues are soft in texture, the cutting process will inevitably change the original shape of the sample; for each section, cutting will introduce independent deformations as well as various types of distortions such as fractures, folds, and impurity contaminations. These distortions will hinder subsequent studies such as neural circuit reconstruction and anatomical analysis of biological tissues.

[0003] In the above context, electron microscope image registration technology has emerged, and its purpose is to eliminate the deformations existing in the images and obtain high-quality registered images. Given two electron microscope images to be registered, namely a fixed image and a floating image, the goal of electron microscope image registration is to calculate a spatial transformation to map the floating image to the fixed image, striving to make the pixel points on the floating image correspond one by one to the pixel points on the fixed image, so as to achieve the purpose that image features with the same semantic information have the same coordinates in space. Here, this spatial transformation is the deformation field.

[0004] Existing methods for electron microscope image registration are mainly divided into traditional registration methods and deep learning-based registration methods. Most traditional registration methods rely on feature extraction and feature matching of images: first, image features are extracted and feature descriptors are generated, and then feature matching is performed according to the similarity degree of the descriptors to obtain the transformation parameters for image registration. Such methods rely on the accurate calculation and matching of image features to obtain reliable registration results, and usually the process of feature extraction and matching is extremely time-consuming. In real scenarios with actual requirements for registration speed, in order to reduce the registration time, the accuracy of feature extraction and matching has to be reduced, which in turn leads to a decrease in registration accuracy. In addition, in order to obtain satisfactory registration results, traditional methods need to repeatedly adjust the registration parameters and manually intervene in the registration process, which is not conducive to improving the registration efficiency.

[0005] Currently, the electron microscope image registration method based on deep learning mainly constructs a regression network model based on a convolutional neural network. Such methods greatly reduce the requirements for manual work in the registration process by using an end-to-end deep learning framework. However, due to the inherent limitation that a convolutional neural network can only focus on local semantic information, such methods can only obtain local image feature matching relationships through convolutional operations, and thus can only handle image deformations within a small range, and cannot correct larger deformations. Therefore, in view of these problems existing in the existing electron microscope image registration technology, it is necessary to conduct further research on obtaining more global semantic information and improving the registration accuracy of large-deformation images. Summary of the Invention

[0006] The present invention is to overcome the above-mentioned deficiencies existing in the prior art, and proposes an electron microscope image registration method based on cost volume, in order to encode more globally semantic information in the image, so as to improve the processing ability for large deformations in electron microscope images and achieve the improvement of the registration accuracy of electron microscope images.

[0007] In order to achieve the above-mentioned invention purpose, the present invention adopts the following technical solutions:

[0008] The characteristics of an electron microscope image registration method based on cost volume according to the present invention include the following steps:

[0009] Step 1: Construct input data, including data acquisition and data preprocessing;

[0010] Step 1.1: Obtain a number of fixed images and their corresponding registered floating images from a registered electron microscope image dataset through random cropping operations; among them, the set of fixed images is denoted as F ∈ R h×w , and the set of registered floating images is denoted as M ∈ R h×w , h represents the height of the image, and w represents the width of the image;

[0011] Step 1.2: Apply random deformations to all the floating images in the set of floating images M to synthesize corresponding unregistered images, thereby obtaining a set of deformed floating images, denoted as W ∈ R h×w ;

[0012] Step 1.3: Construct an input data pair set {(F i , W i )|i = 1,…,n} from the set of fixed images F and the set of deformed floating images W; where, F i represents the i-th fixed image, W i represents the i-th deformed floating image, and n represents the total number of image pairs in the set;

[0013] Step 2: Construct an electron microscope image registration network based on cost volume, including: a feature extraction and enhancement branch, and a deformation field prediction branch;

[0014] Step 2.1: The feature extraction and enhancement branch contains N levels, and each level is composed of a cascaded feature extraction sub-network and a feature enhancement sub-network; among them, the j-th level feature extraction sub-network is denoted as Encoder j , and the j-th level feature enhancement sub-network is denoted as Enhancer j , j = 1, …, N;

[0015] When j = 1, the i-th image pair (F i , W i ) is input into the j-th level feature extraction sub-network Encoder j for processing, and the i-th intermediate feature of the j-th level is output Among them, represents the i-th intermediate fixed feature output by Encoder j , represents the i-th intermediate floating feature output by Encoder j ; After being processed by the j-th level feature enhancement sub-network Enhancer j , the i-th final output feature of the j-th level is obtained Among them, represents the i-th enhanced fixed feature output by the j-th level, represents the i-th enhanced floating feature output by the j-th level;

[0016] When j = 2, …, N, the i-th feature finally output by the j-1-th level is input into the j-th level feature extraction sub-network Encoder j for processing and then input into the j-th level feature enhancement sub-network Enhancer j for processing to obtain the i-th final output feature of the j-th level Thus, the i-th final output feature of the N-th level is output by the N-th level feature enhancement sub-network Enhancer N

[0017] Step 2.2: The deformation field prediction branch contains N levels of deformation field residual estimation sub-networks, and the j-th level deformation field residual estimation sub-network is denoted as DFRE j , j = 1, …, N;

[0018] Use Equation (1) to determine the i-th initial deformation field of the j-th level ​

[0019]

[0020] In Equation (1), represents the i-th deformation field which is the final output of the deformation field residual estimation sub-network at the (j + 1)-th level;

[0021] When j = N, the final output feature at the j-th level and the initial deformation field are input into the deformation field residual estimation sub-network DFRE at the j-th level j for processing to obtain the i-th deformation field residual at the j-th level and the i-th deformation field output at the j-th level is obtained using Equation (2)

[0022]

[0023] In Equation (2), ⊕ represents the element-wise addition operation, and ↑ represents the upsampling operation;

[0024] When j = 1, …, N - 1, the i-th final output feature at the (N - 1)-th level and the i-th initial deformation field at the (N - 1)-th level are input into the deformation field residual estimation sub-network DFRE at the (N - 1)-th level N-1 for processing to obtain the i-th deformation field residual at the (N - 1)-th level and the i-th deformation field output at the (N - 1)-th level is obtained using Equation (2) Thus, the i-th deformation field output at the 1st level is obtained from the deformation field residual estimation sub-network DFRE1 at the 1st level

[0025] Step 2.3: Using the i-th deformation field output at the 1st level to perform spatial interpolation on the i-th deformed floating image W i in the i-th image pair (F i ), W i to obtain the i-th registered floating image R i ;

[0026] Step 3: Training and optimization of the electron microscope image registration network based on cost volume;

[0027] Step 3.1: Perform upsampling operations on the i-th deformation fields at the 2nd to N-th levels respectively, so that they have the same pixel resolution as to obtain the upsampled deformation fields, denoted as where, represents the i-th upsampled deformation field at the j-th level; when j = 1, let

[0028] Step 3.2: Use to interpolate the $i$-th warped floating image $W$ i to obtain a set of registered images obtained by registering the warped fields after upsampling at each level where represents the $i$-th registered image at the $j$-th level; when $j = 1$, let

[0029] Step 3.3: Use Equation (4) to establish the mean square error similarity loss function between the $i$-th registered image at the $j$-th level and the $i$-th fixed image $F$ i

[0030]

[0031] In Equation (4), $(F$ i ) k represents the $k$-th pixel value of $F$ i , represents 's $k$-th pixel value, and $n$ represents the total number of pixels in $F$ i ;

[0032] Step 3.4: Use Equation (3) to establish the spatial gradient loss function of the $i$-th upsampled warped field

[0033]

[0034] In Equation (3), represents the spatial gradient operation, and $\|\cdot\|_2$ represents the $L_2$ norm operation;

[0035] Step 3.5: Use Equation (5) to establish the total loss function for the $i$-th image pair $(F$ i , $W$ i )

[0036]

[0037] In Equation (5), $\lambda$ is a hyperparameter;

[0038] Step 3.6: Based on the set of input data pairs $\{(F$ i , $W$ i )|i = 1, …, n}, train the electron microscope image registration network and optimize the total loss function of each image pair using the gradient descent algorithm ​​Stop training until the iterative training reaches the set number of times, so as to obtain an optimal image registration network model for registering electron microscope image pairs.

[0039] Another feature of the electron microscope image registration method based on cost volume according to the present invention is that each hierarchical feature extraction sub-network in step 2.1 includes M two-dimensional convolutional layers and M ReLU activation function layers;

[0040] A ReLU activation function layer is connected after each two-dimensional convolutional layer to form a basic convolutional block, and M basic convolutional blocks are connected in sequence to form the feature extraction sub-network; among them, the convolutional kernel size of each two-dimensional convolutional layer is 3×3, the convolutional stride of the first two-dimensional convolutional layer is 2, and the convolutional stride of the remaining two-dimensional convolutional layers is 1.

[0041] Each hierarchical feature enhancement sub-network in step 2.1 includes: three two-dimensional convolutional blocks, three pooling blocks, and two Sigmoid activation function layers;

[0042] The three two-dimensional convolutional blocks are respectively denoted as ConvBlock1, ConvBlock2, and ConvBlock3, and each two-dimensional convolutional block is composed of a two-dimensional convolutional layer and a ReLU activation function layer connected;

[0043] The three pooling blocks are respectively denoted as PoolingBlock1, PoolingBlock2, and PoolingBlock3, and each pooling block is composed of a max pooling layer and an average pooling layer;

[0044] The two Sigmoid activation function layers are respectively denoted as Sigmoid1 and Sigmoid2;

[0045] Connect and in the channel dimension and input it into ConvBlock1 to obtain a fused feature

[0046] Connect and respectively process through the pooling blocks PoolingBlock1, PoolingBlock2, and PoolingBlock3. The max pooling layer and the average pooling layer respectively perform pooling on the input features in the channel dimension, and then connect the results of the two poolings in the channel dimension to obtain the corresponding pooled features and

[0047] Connect and respectively with After being concatenated in the channel dimension, they are respectively input into the two-dimensional convolutional blocks ConvBlock2 and ConvBlock3 for processing to obtain spatial attention features and

[0048] After and are respectively processed by the activation function layers Sigmoid1 and Sigmoid2, the fixed feature spatial attention weights and the floating feature spatial attention weights

[0049] After and are multiplied element-wise, the enhanced fixed features After and are multiplied element-wise, the enhanced floating features

[0050] Each level of the deformation field residual estimation sub-network in step 2.2 includes: a cost volume calculation module and a gated recurrent unit module;

[0051] The cost volume calculation module in the j-th level of the deformation field residual estimation sub-network DFRE j uses Equation (6) to calculate the cost volume between the i-th enhanced fixed feature and the enhanced floating feature in the j-th level

[0052]

[0053] In Equation (6), C represents the number of channels of the enhanced features, and T represents the transpose operation;

[0054] Using the i-th initial deformation field in the j-th level to perform an interpolation operation on the cost volume to obtain the i-th relevant feature

[0055] The initial deformation field the relevant feature and the enhanced fixed feature are input into the gated recurrent unit module for processing, and the i-th deformation field residual

[0056] An electronic device according to the present invention includes a memory and a processor, characterized in that the memory is used to store a program that supports the processor to execute any one of the electron microscope image registration methods, and the processor is configured to execute the program stored in the memory.

[0057] A computer-readable storage medium according to the present invention, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, it executes the steps of any one of the electron microscope image registration methods.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] 1. When constructing the electron microscope image registration network, the present invention introduces a cost volume, which improves the ability of the registration method to process large deformations in images in a novel and effective way, improves the accuracy of electron microscope image registration, and has obvious advantages when there are large deformations in the images. The cost volume records the matching degree of features by calculating the inner product of pixel pairs between features. Compared with the local convolution method adopted by the convolutional neural network, the cost volume can pay attention to the more global semantic connections between features, which is beneficial for the registration network to pay attention to larger image deformations and improve the registration accuracy.

[0060] 2. The present invention adds a feature enhancement module to the branch of extracting the features of the input fixed and floating images and establishing their respective feature pyramids to strengthen the features associated between the fixed features and the floating features. On the one hand, in order to obtain an accurate deformation field, the registration process attempts to strengthen the connection between features with the same semantic information between the fixed features and the floating features and weaken the influence of feature pairs that cannot find semantic correspondence on the registration process. The feature enhancement module applies different weights to the features through a spatial attention mechanism to achieve the above goal. On the other hand, the cost volume records the degree of association between the fixed features and the floating features. After being processed by the feature enhancement module, the features with high correlation are highlighted, making the degree of association of the features recorded by the cost volume more robust and accurate.

[0061] 3. The present invention adopts a multi-level residual estimation strategy in the deformation field prediction branch to continuously correct the estimation result of the deformation field, thereby improving the prediction accuracy of the final deformation field and obtaining a more accurate registration result. Inspired by related methods in optical flow estimation, the present invention uses a gated recurrent unit to calculate the residuals of the deformation fields at each level and constructs a multi-level residual estimation network to estimate the residuals and correct the deformation field at the current level from low resolution to high resolution, continuously optimizing the deformation field prediction result to achieve more accurate registration.

[0062] 4. During the training phase of the network, the present invention uses a multi-level loss function to simultaneously impose constraints on the deformation field prediction results at each level, promoting the deformation field residual estimation sub-network at each level to learn more accurate semantic connections between image features, so as to accelerate the convergence of network training and improve the registration accuracy. Description of the Drawings

[0063] Figure 1 is the flowchart of the method of the present invention;

[0064] Figure 2 is the overall framework diagram of the cost volume-based electron microscope image registration method of the present invention;

[0065] Figure 3 is the structural diagram of the feature extraction sub-network in the present invention;

[0066] Figure 4 is the structural diagram of the j-th level feature enhancement sub-network in the present invention;

[0067] Figure 5 is the structural diagram of the j-th level deformation field residual estimation sub-network in the present invention. Detailed Embodiment

[0068] In this embodiment, as Figure 2 shown, a cost volume-based electron microscope image registration method obtains image features with high semantic relevance and encodes the correlation degree between features through cost volume to obtain a more global semantic correspondence of image features, and adopts a multi-level residual estimation strategy to iteratively update the deformation field prediction result to improve the registration accuracy of electron microscope images. Specifically, as Figure 1 shown, the method includes the following steps:

[0069] Step 1: Construct input data, including data acquisition and data preprocessing;

[0070] Step 1.1: Obtain a number of fixed images and their corresponding registered floating images from the registered electron microscope image dataset through random cropping operations; among them, the fixed image set is denoted as F ∈ R h×w , the registered floating image set is denoted as M ∈ R h×w , h represents the height of the image, and w represents the width of the image;

[0071] Step 1.2: Apply random deformations to all floating images in the floating image set M to synthesize corresponding unregistered images, thereby obtaining a deformed floating image set, denoted as W ∈ R h×w ; in specific implementation, take h = w = 512.

[0072] Step 1.3: Construct an input data pair set \(\{(F i , W i )|i = 1, \ldots, n\}\) from the fixed image set \(F\) and the deformed floating image set \(W\); where \(F i \) represents the \(i\)-th fixed image, \(W i \) represents the \(i\)-th deformed floating image, and \(n\) represents the total number of image pairs in the set;

[0073] In a specific implementation, three publicly available electron microscopy image datasets are used: CREMI, FIB - 25, and FAFB datasets; a data set \(F\) and \(M\) with an image size of \(512\times512\) are obtained from the datasets through cropping operations; the images in the set \(M\) are randomly deformed using an interpolation method based on narrow - band spline functions to obtain the deformed floating image set \(W\), and then the input data pair set \(\{(F i , W i )|i = 1, \ldots, n\}\) is obtained; in this embodiment, the total number of input data pairs obtained from the CREMI, FIB - 25, and FAFB datasets are 992, 1496, and 1074 respectively;

[0074] Step 2: Construct a cost - volume - based electron microscopy image registration network, including: a feature extraction and enhancement branch, and a deformation field prediction branch;

[0075] Step 2.1: The feature extraction and enhancement branch contains \(N\) levels, and each level is composed of a cascaded feature extraction sub - network and a feature enhancement sub - network; where the \(j\) - th level feature extraction sub - network is denoted as Encoder j , and the \(j\) - th level feature enhancement sub - network is denoted as Enhancer j , \(j = 1, \ldots, N\);

[0076] Each level's feature extraction sub - network contains \(M\) two - dimensional convolutional layers and \(M\) ReLU activation function layers; a ReLU activation function layer is connected after each two - dimensional convolutional layer to form a basic convolutional block, and \(M\) basic convolutional blocks are connected in sequence to form the feature extraction sub - network; where the convolutional kernel size of each two - dimensional convolutional layer is \(3\times3\), the convolutional stride of the first two - dimensional convolutional layer is 2, and the convolutional strides of the remaining two - dimensional convolutional layers are 1; in this embodiment, \(M = 3\) is taken; the structure of the feature extraction sub - network is as Figure 3 shown.

[0077] Each level of the feature enhancement sub-network contains: three two-dimensional convolutional blocks, three pooling blocks, and two Sigmoid activation function layers; the three two-dimensional convolutional blocks are denoted as ConvBlock1, ConvBlock2, and ConvBlock3 respectively, and each two-dimensional convolutional block is composed of a two-dimensional convolutional layer and a ReLU activation function layer connected; the three pooling blocks are denoted as PoolingBlock1, PoolingBlock2, and PoolingBlock3 respectively, and each pooling block is composed of a max pooling layer and an average pooling layer; the two Sigmoid activation function layers are denoted as Sigmoid1 and Sigmoid2 respectively;

[0078] The structure of the feature enhancement sub-network at the j-th level is as Figure 4 shown; and are concatenated in the channel dimension and then input into ConvBlock1 to obtain the fused feature For each pooling block, its input is pooled by the max pooling layer and the average pooling layer in the channel dimension respectively, and the obtained results are further concatenated in the channel dimension to get the output of the pooling block; and After being processed by the pooling blocks PoolingBlock1, PoolingBlock2, and PoolingBlock3 respectively, the corresponding pooled features and are obtained; and are respectively concatenated with in the channel dimension and then input into the two-dimensional convolutional blocks ConvBlock2 and ConvBlock3 respectively to obtain the spatial attention features and are obtained; and After being processed by the activation function layers Sigmoid1 and Sigmoid2 respectively, the fixed feature spatial attention weight and the floating feature spatial attention weight are obtained; is multiplied element-wise with to obtain the enhanced fixed feature is multiplied element-wise with and to obtain the enhanced floating feature

[0079] When j = 1, the i-th image pair (F i , W i ) is input into the j-th level feature extraction sub-network Encoder jProcess it and output the i-th intermediate feature of the j-th level in, Represents Encoder j The i-th intermediate fixed feature of the output, Represents Encoder j The i-th intermediate floating feature of the output; After the j-th level feature enhancement sub-network Enhancer j After processing, the final i-th output feature of the j-th level is obtained in, represents the i-th enhanced fixed feature output of the j-th level, represents the i-th enhanced floating feature output of the j-th level;

[0080] When j=2,…,N, the i-th feature of the final output of the j-1th level Input j-th level feature extraction subnetwork Encoder j After processing, it is input into the j-th level feature enhancement sub-network Enhancer j Processing is performed to obtain the final i-th output feature of the j-th level Thus, the N-th level feature enhancer network Enhancer N Output the final i-th output feature of the N-th level In a specific implementation, N=6.

[0081] Step 2.2: The deformation field prediction branch contains N levels of deformation field residual estimation subnetworks, and the j-th level deformation field residual estimation subnetwork is recorded as DFRE j , j=1,…,N;

[0082] The structure of the deformation field residual estimation subnetwork is as follows Figure 5 As shown in the figure, each level of the deformation field residual estimation sub-network includes: a cost volume calculation module and a gated recurrent unit module;

[0083] For the j-th level deformation field residual estimation subnetwork DFRE j , its cost volume calculation module uses formula (6) to calculate the i-th enhanced fixed feature of the j-th level and enhanced floating features The cost volume between

[0084]

[0085] In formula (6), C represents the number of channels of enhanced features, T represents the transposition operation; the cost volume is obtained Then, using the i-th initial deformation field of the j-th level Interpolate the cost volume to obtain the i-th relevant feature at the j-th level The initial deformation field Relevant features and enhanced fixed features are input into the gated recurrent unit module for processing to obtain the i-th deformation field residual at the j-th level As the output of DFRE j ;

[0086] Use Equation (1) to determine the i-th initial deformation field at the j-th level

[0087]

[0088] In Equation (1), represents the i-th deformation field that is the final output of the deformation field residual estimator sub-network at the (j + 1)-th level;

[0089] When j = N, the i-th initial deformation field at the j-th level is obtained from Equation (1) and the final output feature at the j-th level and the initial deformation field are input into the deformation field residual estimator sub-network DFRE at the j-th level j for processing to obtain the i-th deformation field residual at the j-th level and the i-th deformation field output at the j-th level is obtained using Equation (2)

[0090]

[0091] In Equation (2), ⊕ represents the element-wise addition operation, and ↑ represents the upsampling operation;

[0092] When j = 1, …, N - 1, the i-th initial deformation field at the j-th level is obtained from Equation (1) and the i-th final output feature at the (N - 1)-th level and the i-th initial deformation field at the (N - 1)-th level are input into the deformation field residual estimator sub-network DFRE at the (N - 1)-th level N-1 for processing to obtain the i-th deformation field residual at the (N - 1)-th level and the i-th deformation field output at the (N - 1)-th level is obtained using Equation (2) Thus, the i-th deformation field output at the first level is obtained from the deformation field residual estimator sub-network DFRE1 at the first level

[0093] Step 2.3: Use the i-th deformation field output at the first level for the i-th image pair (Fi , W i ) the i-th deformed floating image W i Perform bilinear interpolation in space to obtain the i-th registered floating image R i ;

[0094] Step 3: Training and optimization of the electron microscope image registration network based on cost volume;

[0095] Step 3.1: Perform upsampling operations on the i-th deformation field from the second level to the N-th level respectively, so that it has the same pixel resolution as to obtain the upsampled deformation field, denoted as where represents the i-th upsampled deformation field at the j-th level; when j = 1, let In a specific implementation, the i-th deformation field from the second level to the N-th level is upsampled to a size of 512×512 by the bilinear interpolation method.

[0096] Step 3.2: Use to interpolate the i-th deformed floating image W i to obtain the set of registered images registered by the upsampled deformation fields at each level where represents the i-th registered image at the j-th level; when j = 1, let

[0097] Step 3.3: In order to maximize the visual similarity between the i-th registered image at the j-th level and the i-th fixed image F i and achieve semantic correspondence at the pixel level, in this method, by constraining and F i similarity to achieve the above goal; use Equation (4) to establish and F i the mean square error similarity loss function between

[0098]

[0099] In Equation (4), (F i ) k represents the k-th pixel value of F i , represents the k-th pixel value of, n represents the total number of pixels in F i .

[0100] Step 3.4: Simply constraining the similarity between images without constraining the spatial transformation of the deformation field will result in the registered image having too high a visual similarity to the fixed image, making the spatial transformation of the registered image non-smooth and possibly causing the loss of some information in the deformed floating image. Therefore, it is necessary to impose constraints on the spatial transformation of the deformation field; use Equation (5) to establish the upsampled deformation field of the $i$-th at the $j$-th level of the spatial gradient loss function

[0101]

[0102] In Equation (3), represents the spatial gradient operation, and $\|\cdot\|_2$ represents the L2 norm operation.

[0103] Step 3.5: Use Equation (5) to establish the total loss function i ,W i ) for the $i$-th image pair $(F

[0104]

[0105] In Equation (5), $\lambda$ is a hyperparameter that balances the spatial gradient loss function and the similarity loss function ; in a specific implementation, take $\lambda = 0.5$.

[0106] Step 3.6: Based on the input data pair set $\{(F i ,W i )|i = 1,\ldots,n\}$, train the electron microscope image registration network and use the gradient descent algorithm to optimize the total loss function Stop training after iteratively training to the set number of times, and obtain the optimal image registration network model for registering electron microscope image pairs.

[0107] In a specific implementation, the experimental environment uses the PyTorch deep learning framework; the computing devices used for network training and testing are selected as NVIDIA V100 GPUs and NVIDIA 1080Ti GPUs respectively; the gradient descent algorithm selects the Adam optimization algorithm; the number of training iteration rounds is taken as 500;

[0108] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0109] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is run by a processor, it executes the steps of the above method.

[0110] To quantitatively evaluate the registration effect of the network, in this embodiment, the traditional electron microscope image registration methods Elastic and bUnwarpJ, and the deep learning-based registration methods ssEMnet and Zhou’s are used as comparison methods; the Dice coefficient, cross-correlation coefficient (NCC), and structural similarity (SSIM) are used as evaluation indicators;

[0111] Step 3.6.1: To verify the registration accuracy of the network on the three datasets of CREMI, FIB-25, and FAFB, in this embodiment, the data pair sets obtained from the above datasets are first divided at a ratio of 8:1:1 to obtain their respective corresponding training sets, validation sets, and test sets; the three training sets obtained by the division are used to train the network respectively to obtain the best model corresponding to each training set, and the registration accuracy is compared with the four comparison methods on the test set; the comparison of the results on the three evaluation indicators of Dice coefficient, NCC, and SSIM is shown in Table 1;

[0112] Table 1 Comparison of the results of the cost volume-based registration method and other methods on different datasets:

[0113]

[0114] In Table 1, Ours represents the method of the present invention;

[0115] The comparison of the test results on the three electron microscope image datasets shows that compared with the previous electron microscope image registration methods, the method provided by the present invention has achieved the best results in the quantitative indicators and can effectively improve the registration accuracy of electron microscope images;

[0116] Step 3.6.2: To verify the processing ability of the method of the present invention for large deformations in electron microscope images, in this embodiment, four datasets with different deformation sizes are first constructed based on the CREMI dataset in Step 1.3 and the narrow-band spline function interpolation method, which are respectively denoted as CREMI-1, CREMI-2, CREMI-3, and CREMI-4, and the deformations in the images to be registered in the four datasets increase in sequence; to quantitatively verify the registration performance of the method of the present invention for electron microscope images with large deformations, based on the three indicators of Dice coefficient, SSIM, and NCC, the method of the present invention is compared with the two deep learning-based registration methods of ssEMnet and Zhou’s; the experimental results are shown in Table 2;

[0117] Table 2 Comparison of the results of the cost volume-based registration method and other methods on different deformation datasets:

[0118]

[0119] The results in Table 2 show that, compared with the comparative method, the method of the present invention can capture more global semantic information correspondence when there are large deformations in the image, thereby obtaining a higher ability to process large deformations and improving the accuracy of electron microscope image registration.

Claims

1. A cost volume-based electron microscope image registration method, characterized in that It includes the following steps: Step 1: Construct input data, including data acquisition and data preprocessing; Step 1.1: Obtain a number of fixed images and their corresponding registered floating images from the registered electron microscope image dataset through random cropping operations; among them, denote the set of fixed images as , and denote the set of registered floating images as , where h represents the height of the image and w represents the width of the image; Step 1.2: Apply random deformations to all the floating images in the floating image set M to synthesize corresponding unregistered images, thereby obtaining a deformed floating image set, denoted as W ; Step 1.3: Construct an input data pair set { } from the fixed image set F and the deformed floating image set W; where represents the i-th fixed image, represents the i-th deformed floating image, and n represents the total number of image pairs in the set; Step 2: Construct an electron microscope image registration network based on cost volume, including: a feature extraction and enhancement branch, and a deformation field prediction branch; Step 2.1: The feature extraction and feature enhancement branch contains N levels, and each level is composed of a cascaded feature extraction sub-network and a feature enhancement sub-network; among them, the j-th level feature extraction sub-network is denoted as , and the j-th level feature enhancement sub-network is denoted as , ; When the i-th image pair is input into the level-i feature extraction sub-network for processing and outputs the i-th intermediate feature at level ; where represents the i-th intermediate fixed feature output, represents the i-th intermediate floating feature output; After being processed by the level- feature enhancement sub-network the i-th output feature at the final level is obtained; where represents the i-th enhanced fixed feature output at level represents the i-th enhanced floating feature output at level; When the i-th feature finally output at the (j - 1)-th level is input into the j-th level feature extraction sub-network for processing and then input into the j-th level feature enhancement sub-network for processing to obtain the i-th output feature finally at the j-th level ; thus, the i-th output feature finally at the j-th level is output by the j-th level feature enhancement sub-network ; ; Step 2.2: The deformation field prediction branch includes N levels of deformation field residual estimation sub-networks, and the deformation field residual estimation sub-network at the j-th level is denoted as , ; Determine the $i$-th initial deformation field of the $j$-th level using Equation (1). : (1) In formula (1), represents the i-th deformation field that is the final output of the deformation field residual estimation sub-network at the (j + 1)-th level; When j = N, the final output feature of the j-th level and the initial deformation field are input into the deformation field residual estimation sub-network of the j-th level for processing to obtain the i-th deformation field residual of the j-th level , and the i-th deformation field output of the j-th level is obtained using Equation (2) : (2) In formula (2), represents an element-wise addition operation, represents an upsampling operation; When j = , the i-th final output feature of the -th level and the i-th initial deformation field of the -th level are input into the deformation field residual estimation sub-network of the -th level for processing to obtain the i-th deformation field residual of the -th level , and the i-th deformation field output of the -th level is obtained using Equation (2); thus, the i-th deformation field output of the -th level is obtained from the deformation field residual estimation sub-network of the -th level ; -th level ;​​​​​​ Each level of the deformation field residual estimation sub-network in Step 2.2 includes: a cost volume calculation module and a gated recurrent unit module; The deformation field residual estimation sub-network of the j-th level The cost volume calculation module in it calculates the i-th enhanced fixed feature of the j-th level using Equation (6) and the enhanced floating feature The cost volume between : (6) In Equation (6), C represents the number of channels of the enhanced feature, and T represents the transpose operation; Using the i-th initial deformation field of the j-th level perform an interpolation operation on the cost volume to obtain the i-th correlation feature of the j-th level ; Input the initial deformation field , related features and enhanced fixed features into the gated recurrent unit module for processing, and output the i-th deformation field residual of the j-th level ; Step 2.3: Output using the i-th deformation field of the first level For the i-th image pair Perform spatial interpolation on the i-th deformed floating image in the pair to obtain the i-th registered floating image ; Step 3: Training and optimization of the electron microscope image registration network based on cost volume; Step 3.1: Perform upsampling operations on the $i$-th deformation field from the 2nd level to the $N$-th level respectively, so that it has the same pixel resolution as , thereby obtaining the upsampled deformation field, denoted as ; where represents the $i$-th upsampled deformation field at the $j$-th level; when , let ; Step 3.2: Using to interpolate the i-th warped floating image so as to obtain a set of registered images obtained by registering the warped fields after upsampling at each level ; where represents the i-th registered image at the j-th level; when , let ; Step 3.3: Establish the mean square error similarity loss function between the \(i\)-th registered image of the \(j\)-th level and the \(i\)-th fixed image using Equation (4). and the \(i\)-th fixed image as follows: : (4) In formula (4), represents the k-th pixel value of, represents the k-th pixel value of, and n represents the total number of pixels in; Step 3.4: Establish the upsampled deformation field of the $i$-th in the $j$-th level by using Equation (3). spatial gradient loss function : (3) In formula (3), represents the spatial gradient operation, represents the norm operation; Step 3.5: Establish the total loss function for the \(i\)-th image pair using Equation (5) :[[]]END]]​ (5) In formula (5), is a hyperparameter; Step 3.6: Based on the input data, train the electron microscope image registration network for the set { }, and use the gradient descent algorithm to optimize the total loss function of each image pair , until the iterative training stops after reaching the set number of times, so as to obtain the optimal image registration network model for the registration of electron microscope image pairs.

2. The method for electron microscope image registration based on cost volume according to claim 1, wherein, Each level of the feature extraction sub-network in Step 2.1 includes M two-dimensional convolutional layers and M ReLU activation function layers; After each two-dimensional convolutional layer, a ReLU activation function layer is connected to form a basic convolutional block, and M basic convolutional blocks are sequentially connected to form the feature extraction sub-network; among them, the convolutional kernel size of each two-dimensional convolutional layer is , the convolutional stride of the first two-dimensional convolutional layer is 2, and the convolutional strides of the remaining two-dimensional convolutional layers are 1.

3. A method for registering electron microscope images based on cost volume according to claim 1, characterized in that, Each level of the feature enhancement sub-network in Step 2.1 includes: three two-dimensional convolutional blocks, three pooling blocks, and two Sigmoid activation function layers; Denote the three two-dimensional convolutional blocks as , and respectively, and each two-dimensional convolutional block is composed of a two-dimensional convolutional layer and a ReLU activation function layer connected together; Denote the three pooling blocks as , and respectively, and each pooling block consists of a max pooling layer and an average pooling layer; Denote the two Sigmoid activation function layers as and ; Connect and after connecting them in the channel dimension and input to obtain the fused feature ; Put , and are respectively processed through pooling blocks , and . The maximum pooling layer and the average pooling layer respectively perform pooling on the input features in the channel dimension, and then the results of the two poolings are concatenated in the channel dimension to obtain the corresponding pooled features , and ; Connect and separately with in the channel dimension, and then input them into the two-dimensional convolutional blocks and for processing to obtain the spatial attention features and ; After and are processed by the activation function layers and respectively, the fixed feature space attention weights and the floating feature space attention weights are obtained; Multiply with through element-wise multiplication to obtain enhanced fixed features , and multiply with through element-wise multiplication to obtain enhanced floating features .

4. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program for supporting the processor to execute the electron microscope image registration method described in any one of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor, it executes the steps of the electron microscope image registration method described in any one of claims 1-3.

Images