An image inpainting method for serial section electron microscopy data

Abstract

The application discloses an image repairing method for continuous slice electron microscope data and belongs to the technical field of image repairing. The application uses the information of adjacent frames of a missing area, provides effective information for the information missing area of a damaged frame through optical flow estimation and weighted interpolation operation; an encoding network capable of effectively learning the multi-scale fusion features of a damaged image and a reference image is proposed; the skip-connection operation is used in the decoding part to recover the effective information lost in the encoding process, and the final prediction result is output; a structure consistency loss function is introduced when training the model to optimize the repairing result, the difference between the optical flow between the repaired frame and the front and rear frames and the true value is calculated, so that the network can repair the visual and semantic reasonable and clear slice image along the section and slice direction. The application solves the problems that the traditional and existing deep learning methods cannot process the inconsistent semantic information, the insufficient visual clarity and the incoherent structure of the front and rear frames.

Claims

1. An image restoration method for serial section electron microscopy data, characterized in that, This image restoration method includes the following steps: S1: Collect consecutive slices to obtain a numbered image stack. By manually judging and annotating the entire image stack, obtain the damaged slice number k and the original mask M corresponding to the damaged area. Determine the size and position of the cut area of ​​the damaged frame based on the information of the original mask M. Finally, based on the cut area, find the image number and corresponding slice image with the closest interval between the front and back of the corresponding area and the complete image information in the image stack, that is, the slice images of the front and back frames. S2: Extract localized previous frame information larger than the cut area of ​​the damaged frame from the sliced ​​images of the previous and previous frames. and local post-frame information Then, the optical flow estimation model is input to obtain the optical flow information between the two, and weighted interpolation is used to obtain the intermediate interpolated frame. Finally, based on the size and position of the cut area of ​​the damaged frame, from the damaged frame... Intermediate interpolation frames The damaged image I cut out from the original mask M inter Reference Image I ref and local mask m; S3: Transfer the damaged image I inter Reference Image I ref The local mask m is input into the image restoration model: First, the feature encoding module learns the effective feature representations of the damaged image and the reference image with the labeled missing region. Then, the features at different scales are fused and encoded. The fusion module aggregates image features at different hole rates to generate contextual information at different distances. Finally, in the feature decoding part, the skip-connection operation is used to introduce the features of the damaged image at different scales into the decoding process to avoid the loss of effective information and obtain the final output. S4: Repeat S1-S3 until all damaged images detected in the image stack have been repaired.

2. The image restoration method as described in claim 1, characterized in that, S2 includes: S2-1: Prediction Model for Optical Flow Information The PWC-Net model architecture is adopted, and the objective function of the model is... Adjustments were made; since the original model training dataset contained true optical flow values ​​used for supervised learning, while the dataset targeted by this invention does not contain true optical flow values, local previous frame information was used instead. and local post-frame information As input, the model will output N sets of corresponding values ​​at different scales during training. arrive Optical flow information The specific objective function is as follows: in Represents reconstruction losses, Represents structural similarity loss. The input image is downsampled according to the optical flow scale. The projected image is obtained by mapping each pixel of the input image according to the input optical flow. Used to calculate structural similarity between images, with values ​​ranging from [0,1]; S2-2: Prediction Model Using Optical Flow Information Calculate the local previous frame information respectively To local post-frame information and information from local later frames Information from the previous local frame Forward optical flow information and backward optical flow information Then, weighted interpolation is used to obtain the intermediate interpolated frame. The specific calculation method is as follows: where is the deformation function, and each pixel point in the image is distorted to the indicated position according to the input optical flow, is the weight coefficient, representing the degree of information contribution of the previous frame image, with a range of [0, 1]. If the corresponding numbers of the selected previous and next frames and the damaged frame in the image stack are i, j, k (i < k < j), then ; then, for the intermediate interpolated frame perform cropping to obtain the reference image I ref .

3. The image restoration method as described in claim 1, characterized in that, In S3, the image inpainting model is an image inpainting network based on neighboring frame supervision. This model is built on an Encoder-Decoder architecture. The Encoder part uses the segmented, damaged image I... inter and reference image I ref The multi-scale features are fused and then aggregated and learned from the features after passing through different dilated convolutional layers. The Decoder part recovers the effective information lost during the encoding process through the skip-connection operation.

4. The image restoration method as described in claim 3, characterized in that, The specific process of S3 is as follows: (1) The input to the network is a damaged image. Local masking and reference image : The final repair result is: in Represents a real image. Represents a binary mask. This represents multiplication by pixels; (2) Model structure composition: In the image encoder section, a feature aggregation module is proposed to learn effective features of the input image, mainly including traditional convolutional layers. And the aggregated dilated convolutional layer obtained by merging dilated convolutions with different dilation rates (r=1,2,4,8) The encoding part is formulated as follows: in and These represent damaged images. and reference image The Layer features, Represents a damaged image and reference image The Layer aggregation features This represents a splicing operation. This represents the number of feature layers, with a value of 4. In the image decoding section, Combination Continuous Layer deconvolution Decoding operation, to obtain The formula is as follows; (3) Objective function: The final prediction result is: Reconstruction losses for: Structural similarity loss for: in and It is a constant. and These are the image's width and height in pixels, respectively. Perceived loss for: in For the pre-trained network Activation layer of the layer, for The number of elements in; Structural consistency loss for: Generators against loss for: The objective function of the generator is: The objective function of the discriminator for: Where G represents the generator and D represents the discriminator.

Images