A bidirectional stitching method and system for electron microscope biological images

Abstract

The present disclosure provides a bidirectional stitching method for electron microscope biological images, the scheme comprising: obtaining images to be stitched with overlapping areas; performing information expansion processing on the overlapping boundaries of the images to be stitched based on an image context exchange strategy; inputting the processed images into a pre-trained convolutional pyramid for multi-scale feature extraction; at each layer of the convolutional pyramid, superimposing the obtained features of the images to be stitched, and simultaneously inputting the deformation field estimated by the previous layer into a bidirectional deformation field estimation module to obtain the deformation field corresponding to each image of the current scale; applying the dense deformation field output by the last layer of the convolutional pyramid to the original images to be stitched, and generating the final stitched image based on a weighted mask method.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a method and system for bidirectional stitching of electron microscope biological images. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] In serial section electron microscopy, stitching multiple images with overlapping areas to generate a high-resolution full view of biological samples is a very powerful cell visualization technology in the biological field. A common process in image stitching is to identify reliable and recognizable patterns or features in the overlapping area of ​​two images, and then estimate the corresponding transformation parameters based on the matched features to align the overlapping areas of the two images.

[0004] For image stitching, a common solution is to use a global homography or affine transformation to characterize the correspondence between overlapping areas. However, a single transformation model cannot handle street view images with large parallax and biological images with local distortion.

[0005] In order to eliminate the mixed artifacts caused by parallax in street view panorama stitching, multiple local deformation transformation models are introduced. The image is gridded according to a pre-set spacing, and the local homography transformation within each grid block is estimated. Finally, these local homography models are combined, and some geometric features or geometric constraints are used to extend the transformation model to non-overlapping areas, thereby effectively alleviating the artifact effect caused by large parallax. However, artificially designed features may ignore detailed structures and produce false deformation models. At the same time, artificially designed features cannot extract enough key points in some smooth images with high signal-to-noise ratio, resulting in failure of local deformation model estimation.

[0006] In biological images with local distortions, the use of deep learning to estimate dense deformation fields has become the mainstream in medical image registration. Dense deformation fields are different from parameter-specific transformation models such as homography matrices and affine matrices. Dense deformation fields estimate the deformation displacement of each pixel, which can better and more meticulously simulate the elastic deformation distortions contained in biological images. Using differentiable spatial transformation components, unsupervised networks that do not require artificial information can effectively extract detailed structural features in images through data-driven models, thereby accurately estimating the corresponding dense deformation field. However, biological electron microscope images are different from medical images. Biological electron microscope images may have large-scale non-uniform deformations, contrast changes, and high noise phenomena. Their stitching is still a challenge, especially, Figure 1As shown, when biological images have large deformations, traditional methods cannot perfectly estimate the deformation parameters of the overlapping areas of the images, resulting in artifacts. Summary of the invention

[0007] In order to solve the above-mentioned problems, the present disclosure provides a method and system for bidirectional stitching of electron microscope biological images. The scheme estimates dense deformation fields based on deep learning, solves the elastic deformation phenomenon that may occur in the stitching of sequential biological images, thereby generating a stitched image with almost no artifacts, presenting a clear overall picture of the biological sample.

[0008] According to a first aspect of an embodiment of the present disclosure, a method for bidirectional stitching of electron microscope biological images is provided, comprising:

[0009] Acquire images to be stitched with overlapping areas;

[0010] Based on the image context exchange strategy, the overlapping boundaries of the stitched images are expanded;

[0011] Input the processed image into the pre-trained convolutional pyramid for multi-scale feature extraction;

[0012] At each layer of the convolutional pyramid, the features of the image to be spliced ​​are superimposed and simultaneously input into the bidirectional deformation field estimation module together with the deformation field estimated by the previous layer to obtain the deformation field corresponding to each image of the current scale;

[0013] The dense deformation field output by the last layer of the convolutional pyramid is applied to the original image to be stitched, and the final stitched image is generated based on the weighted mask method.

[0014] Furthermore, the image context exchange strategy is specifically as follows: directly stitching each image in the images to be stitched with a non-overlapping area of ​​another image according to a stitching boundary.

[0015] Furthermore, the bidirectional deformation field estimation module includes a differentiable space transformation submodule and a registration submodule for each image to be stitched, which are used for bidirectionally fitting the target deformation.

[0016] Furthermore, the differentiable spatial transformation submodule is used to distort the image features obtained by the current base convolution layer of the convolution pyramid using the deformation field obtained by the previous base convolution layer, and obtain the distorted image based on bilinear interpolation calculation.

[0017] Furthermore, the registration submodule is used to stack the distorted image features with the target image features, and sequentially input them into several convolutional layers to obtain the residual of the deformation field; and output the re-estimated deformation field by integrating the residual with the deformation field output by the previous layer.

[0018] According to a second aspect of an embodiment of the present disclosure, a bidirectional stitching system for electron microscope biological images is provided, comprising:

[0019] A data acquisition unit, which is used to acquire images to be stitched with overlapping areas;

[0020] An overlapping boundary expansion unit, which is used to expand the information of the overlapping boundaries of the images to be stitched based on an image context exchange strategy;

[0021] The deformation field estimation unit is used to input the processed image into the pre-trained convolutional pyramid for multi-scale feature extraction; at each layer of the convolutional pyramid, the obtained features of the image to be spliced ​​are superimposed and simultaneously input into the bidirectional deformation field estimation module with the deformation field estimated in the previous layer to obtain the deformation field corresponding to each image of the current scale;

[0022] The stitching unit is used to apply the dense deformation field output by the last layer of the convolutional pyramid to the original image to be stitched, and generate a final stitched image based on a weighted mask method.

[0023] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the method for bidirectional stitching of electron microscope biological images as described above is implemented.

[0024] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for bidirectional stitching of electron microscope biological images as described above is implemented.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] (1) The present disclosure provides a method and system for bidirectional stitching of electron microscope biological images. The scheme estimates dense deformation fields based on deep learning to solve the elastic deformation phenomenon that may occur in the stitching of sequential biological images, thereby generating a stitched image with almost no artifacts and presenting a clear overall picture of the biological sample.

[0027] (2) The scheme uses a feature pyramid to extract multi-scale information of the image. Each layer in the pyramid estimates the residual of the current deformation field, thereby achieving a coarse-to-fine registration and stitching effect. In addition, by using a bidirectional stitching strategy, the large deformation field can be effectively decomposed into two small deformation field estimation problems for each image, so that the overlapping areas can be nearly perfectly registered, thereby eliminating artifacts.

[0028] (3) The scheme can effectively expand the scope of feature extraction by adopting the image context information exchange strategy, while ensuring that the network focuses on the area of ​​target deformation field estimation.

[0029] Advantages of additional aspects of the present disclosure will be given in part in the following description and in part will become apparent from the following description or will be learned through practice of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0031] Figure 1 The traditional method described in the first embodiment of the present disclosure is prone to generate a stitching result with artifacts;

[0032] Figure 2 Schematic diagram of the comparison between the traditional unidirectional estimation strategy described in the first embodiment of the present disclosure and the bidirectional splicing strategy of the present disclosure, wherein (a) is the traditional unidirectional estimation strategy, and (b) is the bidirectional splicing strategy;

[0033] Figure 3 This is a flow chart of the bidirectional stitching method for electron microscope biological images described in the first embodiment of the present disclosure;

[0034] Figure 4 This is a schematic diagram of the image context information exchange strategy described in the first embodiment of the present disclosure;

[0035] Figure 5 Schematic diagram of the structure of the bidirectional deformation field estimation module described in the first embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0037] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0038] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0039] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0040] Embodiment 1:

[0041] The purpose of this embodiment is to provide a bidirectional stitching method for electron microscope biological images.

[0042] like Figure 1 As shown, it is shown that the traditional method is easy to generate a stitching result with artifacts. When the biological image has a large deformation, the traditional method cannot perfectly estimate the deformation parameters of the overlapping area of ​​the image, thereby generating artifacts. In order to solve the above problem, the present disclosure provides a bidirectional stitching method for electron microscope biological images, including:

[0043] Acquire images to be stitched with overlapping areas;

[0044] Based on the image context exchange strategy, the overlapping boundaries of the stitched images are expanded;

[0045] Input the processed image into the pre-trained convolutional pyramid for multi-scale feature extraction;

[0046] At each layer of the convolutional pyramid, the features of the image to be spliced ​​are superimposed and simultaneously input into the bidirectional deformation field estimation module together with the deformation field estimated by the previous layer to obtain the deformation field corresponding to each image of the current scale;

[0047] The dense deformation field output by the last layer of the convolutional pyramid is applied to the original image to be stitched, and a final stitched image is generated based on a weighted mask method.

[0048] Furthermore, the image context exchange strategy is specifically as follows: according to the stitching boundary, each image in the image to be stitched is directly stitched with the non-overlapping area of ​​another image; wherein the stitching boundary can be manually determined according to actual needs, or estimated after rough alignment using a traditional alignment algorithm. The scheme disclosed in the present invention can allow a boundary estimation error of approximately ±10% (relative to the overlapping area itself).

[0049] Among them, Figure 4 (a) and Figure 4 (b) shows the image context exchange strategy, specifically:

[0050] Boundary content expansion: For the two input images to be stitched, each image is directly stitched with the non-overlapping area of ​​the other image according to the stitching boundary, such as Figure 4 (a) Each image simultaneously extracts neighboring information near the boundary to help the network estimate the deformation field around the boundary area.

[0051] Feature mask in the registration module: The image features extracted by the pyramid include the context information of the previous expansion step. Figure 4 As shown in (b), the expanded features in the pyramid backbone network need to be cut by the boundary of the image to be distorted. This operation allows the network to focus on estimating the deformation field of the target area and adapt to the convolutional neural network architecture.

[0052] Furthermore, the bidirectional deformation field estimation module includes a differentiable space transformation submodule and a registration submodule for each image to be stitched, which are used for bidirectionally fitting the target deformation.

[0053] Among them, Figure 4 As shown in Figure 1, the detailed architecture of the bidirectional deformation field estimation module is presented. This module contains two differentiable spatial transformation submodules and corresponding registration submodules for each input image, which are used to fit the target deformation in both directions. In the differentiable spatial transformation submodule, the source image features are distorted by the deformation field of the previous layer after upsampling by 2 times, so that the registration submodule can use the deformation field to propagate gradients during the differential distortion process. Here, we use bilinear interpolation technology to calculate the distorted image. In the feature registration submodule, the distorted source image features are stacked with the target image features and input into 5 convolutional layers (corresponding layers are 32-, 64-, 32-, 16-, and 2 channels) in sequence to calculate the residual of the deformation field. By integrating the residual with the deformation field of the previous layer after upsampling by 2 times, the registration submodule can output the re-estimated deformation field.

[0054] Furthermore, the differentiable spatial transformation submodule is used to distort the image features obtained by the current base convolution layer of the convolution pyramid using the deformation field obtained by the previous base convolution layer, and obtain the distorted image based on bilinear interpolation calculation.

[0055] Furthermore, the registration submodule is used to stack the distorted image features with the target image features, and input them into several convolutional layers in sequence to obtain the residual of the deformation field; by integrating the residual with the deformation field output by the previous layer, a re-estimated deformation field is output; wherein the integration is specifically the addition of the residual and the deformation field of the previous layer.

[0056] Furthermore, the convolutional pyramid network includes several basic convolutional layers of different scales, and the basic convolutional layer includes a two-dimensional convolutional layer, an activation function and a batch normalization layer.

[0057] Among them, Figure 3As shown in the figure, the network architecture and overall data flow we designed are demonstrated. First, the input electron microscope biological image is subjected to an image context information exchange operation, and it is spliced ​​with non-adjacent regions of another image, and then sent to the pyramid backbone to obtain multi-scale features. The deformation field of the bottom-level scale feature starts from a standard grid. The bidirectional deformation field estimation module of each layer accepts the image feature map and deformation field of the previous layer. The last original scale registration module outputs the final dense deformation field, which is directly applied to the input image through a grid sampling method. The distorted image generates the final splicing result through a weighted mask.

[0058] The basic convolution layer consists of a 2D convolution layer, a Leaky-Relu layer (k=0.2) and a batch normalization layer. In the feature pyramid, the i-th layer has (8*i+1) feature channels, and the pyramid is gradually constructed with 2-fold downsampling.

[0059] Pyramid backbone network: Figure 3 As shown in the figure, for each input image, a K-layer pyramid convolutional neural network is used to extract its multi-scale features. In general, the k-th layer of the pyramid performs the following operations: i) a convolution operation of (8*k-3) channels with a step size of 1; ii) a convolution operation with a step size of 2, outputting a feature map of 2-fold downsampled (8*k+1) channels. As the number of layers increases, the feature map is continuously abstracted from detailed to coarse. At the same time, starting from the last layer (the K-th layer), the extracted multi-scale feature map is input into the cascaded bidirectional deformation field estimation module, and the deformation field of each input image from coarse to fine is estimated at the same time. The pyramid backbone network can effectively estimate large-scale deformations step by step by transmitting high-level semantic information and combining it with low-level detail features.

[0060] by and Represents image I A and image I B The k-th layer features of and represents the dense deformation field output from the k+1 layer, then the dense deformation field output from the kth layer is as follows:

[0061]

[0062]

[0063]

[0064] Among them, Up2(·) represents 2 times upsampling, R(·) is a regularization function.

[0065] Furthermore, the weighted mask method is used to generate the final stitched image, specifically using the following formula:

[0066]

[0067] in, It is a linear weighted operator acting on the pixel level, μ is the pixel value on image A, v is the pixel value on image B, and C measures the vertical distance from the pixel to the overlapping boundary.

[0068] Furthermore, the training process of the network model in the bidirectional splicing method is as follows:

[0069] The training of the network is done using a staged training strategy, starting from the coarsest level K bidirectional deformation field estimation module and ending with the most detailed level 1 registration module. For the kth level, we tune the network parameters by minimizing the following cascaded loss function:

[0070]

[0071]

[0072]

[0073] in, is the difference loss term for pixel-level comparison. Here we use MSE(·) mean square error, and Up(·) represents 2 k Upsampling is to upsample the estimated dense deformation field of each layer to the original image size for distortion comparison; represents the gradient operator of the deformation field acting on the x and y axes respectively, J(·) calculates the Jacobian matrix of each pixel displacement; is a smooth constraint term, smoothing the displacement field change; It is mainly used to penalize negative Jacobian determinants (det(·) represents determinant calculation) to alleviate unnatural folding.

[0074] The batch size of the network is set to 16×2×512×512 input size. In this embodiment, K=4 layers of convolutional pyramid are used as the network backbone, and the weighted loss function parameters are set to λ1=0.2, λ2=50. Except for the last layer which requires 40 epochs, each layer only needs to be trained for 5 epochs. It should be noted that the parameters of the previous layer and the pyramid backbone network are not frozen during the current layer training. The model training uses the Adam optimizer with default parameters, and the learning rate is set to 0.0001 for all training stages.

[0075] Embodiment 2:

[0076] The purpose of this embodiment is to provide a bidirectional stitching system for electron microscope biological images.

[0077] A bidirectional stitching system for electron microscope biological images, comprising:

[0078] A data acquisition unit, which is used to acquire images to be stitched with overlapping areas;

[0079] An overlapping boundary expansion unit, which is used to expand the information of the overlapping boundaries of the images to be stitched based on an image context exchange strategy;

[0080] The deformation field estimation unit is used to input the processed image into the pre-trained convolutional pyramid for multi-scale feature extraction; at each layer of the convolutional pyramid, the obtained features of the image to be spliced ​​are superimposed and simultaneously input into the bidirectional deformation field estimation module with the deformation field estimated in the previous layer to obtain the deformation field corresponding to each image of the current scale;

[0081] The stitching unit is used to apply the dense deformation field output by the last layer of the convolutional pyramid to the original image to be stitched, and generate a final stitched image based on a weighted mask method.

[0082] In further embodiments, there is also provided:

[0083] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0084] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0085] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0086] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0087] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0088] Those skilled in the art will appreciate that the units, i.e., algorithm steps, of the various examples described in conjunction with this embodiment can be implemented in electronic hardware or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0089] The above-mentioned embodiment provides a method and system for bidirectional stitching of electron microscope biological images, which can be realized and has broad application prospects.

[0090] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A bidirectional stitching method for electron microscope biological images, characterized in that: include: Acquire images to be stitched with overlapping areas; Performing information expansion processing on the overlapping boundaries of the images to be stitched based on the image context exchange strategy; the image context exchange strategy is specifically: directly stitching each image in the images to be stitched with the non-overlapping area of ​​another image according to the stitching boundary; Input the processed image into the pre-trained convolutional pyramid for multi-scale feature extraction; At each layer of the convolutional pyramid, the obtained features of the image to be spliced ​​are superimposed, and the deformation field estimated by the previous layer is simultaneously input into the bidirectional deformation field estimation module to obtain the deformation field corresponding to each image of the current scale; the bidirectional deformation field estimation module includes a differentiable space transformation submodule and a registration submodule for each image to be spliced, which are used for bidirectional fitting of the target deformation; the differentiable space transformation submodule is used to distort the image features obtained by the current basic convolutional layer of the convolutional pyramid using the deformation field obtained by the previous basic convolutional layer, and obtain the distorted image based on bilinear interpolation calculation; the registration submodule is used to stack the distorted image features and the target image features, and input them into several convolutional layers in sequence to obtain the residual of the deformation field; by integrating the residual with the deformation field output by the previous layer, the re-estimated deformation field is output; The dense deformation field output by the last layer of the convolutional pyramid is applied to the original image to be stitched, and the final stitched image is generated based on the weighted mask method.

2. A bidirectional stitching method for electron microscope biological images as claimed in claim 1, characterized in that: The convolutional pyramid network includes several basic convolutional layers of different scales, and the basic convolutional layer includes a two-dimensional convolutional layer, an activation function and a batch normalization layer.

3. A bidirectional stitching method for electron microscope biological images as claimed in claim 1, characterized in that: The weighted mask method is used to generate the final stitched image, specifically using the following formula: in, is a linear weighting operator acting on the pixel level. is the pixel value on image A, is the pixel value on image B, C Measures the vertical distance from the pixel to the overlap boundary.

4. A bidirectional stitching system for electron microscope biological images, characterized in that: include: A data acquisition unit, which is used to acquire images to be stitched with overlapping areas; An overlapping boundary expansion unit is used to expand the information of the overlapping boundary of the images to be stitched based on an image context exchange strategy; the image context exchange strategy is specifically: according to the stitching boundary, each image in the images to be stitched is directly stitched with the non-overlapping area of ​​another image; A deformation field estimation unit, which is used to input the processed image into a pre-trained convolutional pyramid for multi-scale feature extraction; At each layer of the convolutional pyramid, the obtained features of the image to be spliced ​​are superimposed, and the deformation field estimated by the previous layer is simultaneously input into the bidirectional deformation field estimation module to obtain the deformation field corresponding to each image of the current scale; the bidirectional deformation field estimation module includes a differentiable space transformation submodule and a registration submodule for each image to be spliced, which are used for bidirectional fitting of the target deformation; the differentiable space transformation submodule is used to distort the image features obtained by the current basic convolutional layer of the convolutional pyramid using the deformation field obtained by the previous basic convolutional layer, and obtain the distorted image based on bilinear interpolation calculation; the registration submodule is used to stack the distorted image features and the target image features, and input them into several convolutional layers in sequence to obtain the residual of the deformation field; by integrating the residual with the deformation field output by the previous layer, the re-estimated deformation field is output; The stitching unit is used to apply the dense deformation field output by the last layer of the convolutional pyramid to the original image to be stitched, and generate a final stitched image based on a weighted mask method.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a bidirectional stitching method for electron microscope biological images as described in any one of claims 1 to 3 is implemented.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, a bidirectional stitching method for electron microscope biological images as described in any one of claims 1 to 3 is implemented.

Images