A method for slice image registration and annotation

Abstract

The application discloses a kind of slice image registration and marking method, belong to image processing technical field, steps are as follows: based on generative adversarial network, using the first generator of well-trained brain slice image is carried out style migration, obtain the brain slice image after style migration;Improved ConvNext neural network model is used to the brain slice image after style migration carries out atlas level position identification, and is matched with the allen sign picture slice image, constitutes brain slice image matching pair;Several brain slice image matching pairs are input based on Voxelmorph registration model, obtain the nonlinear registration vector field between brain slice image and allen sign picture slice image;Based on nonlinear registration vector field, the atlas information of allen standard atlas slice image is registered to brain slice image, complete brain slice image registration and solve academic annotation.The application solves the problem that mouse brain slice image registration and marking accuracy and efficiency are insufficient.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a slice image registration and labeling method. BACKGROUND

[0002] A core concept in neuroscience is functional localization, which refers to the idea that different functions of the brain are dominated by neural circuits that are physically located in different spatial locations. Therefore, one of the key steps in analyzing neural data is to accurately register brain regions from different data sources.

[0003] In the registration process of mouse brain regions, traditional two-dimensional paper maps and Allen mouse brain volume reference maps developed by the Allen Institute have been widely used in anatomical spatial registration, thereby unifying the coordinate system and realizing semantic labeling of brain regions. However, accurately registering histological slice images into these standard coordinate systems is still a time-consuming and high-load process. Existing registration methods rely on the subjective judgment of operators in actual operation, and the difference in registration accuracy often varies with the experience of the operator. There is often a large accuracy gap between experienced technicians and beginners who are new to registration, further increasing the complexity of the process.

[0004] Therefore, there is an urgent need for a technical solution that can overcome the interference of staining differences, eliminate human subjectivity, and automatically register and label mouse brain slice images with high throughput. SUMMARY

[0005] In view of the above problems in the prior art, the slice image registration and labeling method provided by the application solves the problems of insufficient accuracy and efficiency of mouse brain slice image registration and labeling.

[0006] To achieve the above-mentioned purposes, the technical solution adopted by the application is as follows:

[0007] The slice image registration and labeling method provided by the application comprises the following steps:

[0008] S1, obtaining a brain slice image, and performing style transfer on the brain slice image by using a trained first generator based on a generative adversarial network to obtain a brain slice image after style transfer;

[0009] S2, performing atlas-level position recognition on the brain slice image after style transfer by using a trained improved ConvNext neural network model, and pairing the brain slice image after style transfer with an Allen landmark picture slice image to form a brain slice image matching pair;

[0010] S3, inputting the matched pairs of brain slice images into a Voxelmorph registration model to obtain a nonlinear registration vector field between the brain slice images and the Allen standard atlas slice images;

[0011] S4, registering atlas information of the Allen standard atlas slice images to the brain slice images based on the nonlinear registration vector field to complete registration of the brain slice images and academic annotation.

[0012] Further, the S1 comprises the following steps:

[0013] S11, acquiring brain slice images of different modalities with coordinate information;

[0014] S12, grouping the acquired brain slice images according to bright field and dark field imaging modes to obtain a first data set composed of bright field brain slice images and a second data set composed of dark field brain slice images;

[0015] S13, based on a CycleGAN adversarial generation network, according to the corresponding relationship between the bright field brain slice images in the first data set, the dark field brain slice images in the second data set and the Allen standard atlas slice images, performing unsupervised style transfer learning training on the first generator to obtain a trained first generator;

[0016] S14, performing style transfer processing on the brain slice images based on the trained first generator to obtain style-transferred brain slice images.

[0017] Further, the CycleGAN adversarial generation network comprises a first generator and a second generator based on a ResNet-Unet structure, and a first discriminator and a second discriminator based on a PatchGAN structure;

[0018] The bright field brain slice images in the first data set and the dark field brain slice images in the second data set are taken as source domain images, and the Allen standard atlas slice images are taken as target domain images;

[0019] The first generator is configured to read original source domain images and generate fake target domain images;

[0020] The second generator is configured to read original target domain images and generate fake source domain images;

[0021] The first discriminator is configured to distinguish whether an input image is a real target domain image or a fake target domain image generated by the first generator;

[0022] The second discriminator is configured to distinguish whether an input image is a real source domain image or a fake source domain image generated by the second generator.

[0023] Further, the calculation expression of the loss function of the adversarial generation network CycleGAN is as follows:

[0024] ,

[0025] wherein, denotes the loss function of the adversarial generation network CycleGAN, denotes the first generator, denotes the second generator, denotes the first discriminator, denotes the second discriminator, denotes the first adversarial loss, denotes the source domain, denotes the target domain, denotes the second adversarial loss, denotes the weight of the cycle consistency loss, denotes the cycle consistency loss, denotes the ratio of the identity loss to the cycle consistency loss, denotes the identity loss.

[0026] Further, the first generator and the second generator each comprise a first padding layer, a first convolution layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a first residual module, a dense connection module, a self-attention module, a second residual module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a second padding layer and a second convolution layer connected in sequence, wherein the input end of the first down-sampling layer and the output end of the third up-sampling layer are connected through channel dimension splicing to form a skip connection, the input end of the second down-sampling layer and the output end of the second up-sampling layer are connected through channel dimension splicing to form a skip connection, and the input end of the third down-sampling layer and the output end of the first up-sampling layer are connected through channel dimension splicing to form a skip connection.

[0027] Further, the S2 comprises the following steps:

[0028] S21, using the trained improved ConvNext neural network model to perform atlas layer position recognition on the style-transferred brain slice image, and predicting the spatial position of the style-transferred brain slice image in the Allen 3D brain atlas as a predicted spatial position;

[0029] S22, extracting a corresponding standard two-dimensional atlas image based on the predicted spatial position, and pairing the style-transferred brain slice image with the Allen landmark picture slice image to form a brain slice image matching pair.

[0030] Further, the improved ConvNext neural network model in S21 replaces the last softmax layer of the ConvNext neural network model with a Relu activation function layer, two consecutive dense connection modules connected in sequence, and an output layer outputting 9 coordinates.

[0031] The method for slice image registration and labeling provided by the application has the advantages that: the method for slice image registration and labeling provided by the application performs unsupervised style transfer learning between brain slice images in a bright-field brain slice image dataset and a dark-field brain slice image dataset and Allen standard atlas slice images, retains the structural details of the brain slice images, and simultaneously unifies the styles of brain slice images of different imaging modalities to the style of the standard atlas, so that the cross-modal brain imaging images are standardized; the first generator can complete the style conversion between the bright-field brain slice images and the dark-field brain slice images and the Allen standard atlas slice images without relying on manual screening of a large amount of data and without strictly paired training data; the application has unsupervised learning capability, does not require strictly paired training data, can retain the structural information of the images while completing the style conversion; the application aligns the Allen standard atlas slice images to the brain slice images by learning the deformation field, that is, affine and nonlinear transformation, can realize accurate pixel-level spatial registration, and can map the registered atlas information back to the original slice images, so that high-precision brain slice registration with structural annotations is realized; the application scheme does not require complete and high-definition brain slices, is not affected by bright-field and dark-field changes, has high automation level, can greatly reduce the workload of the registration operator, does not require programming capability, and can effectively improve the registration work efficiency.

[0032] The other advantages of the application will be analyzed in more detail in the subsequent embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed in the embodiments will be briefly introduced as follows. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0034] Figure 1 The step flow chart of the method for slice image registration and labeling in the embodiment of the application.

[0035] Figure 2 The structural diagram of the generative adversarial network in the embodiment of the application.

[0036] Figure 3This is a schematic diagram of the structure of the first generator in an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram of the residual block in an embodiment of the present invention.

[0038] Figure 5 This is a schematic diagram of the dense connection module in an embodiment of the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0040] CCFv3 (Allen Mouse Brain Common Coordinate Framework Version 3) is a three-dimensional standard coordinate atlas of the mouse brain released by the Allen Institute for Brain Science, designed to provide a high-precision, standardized reference framework for brain structure in neuroscience research.

[0041] like Figure 1 As shown, in one embodiment of the present invention, the present invention provides a method for registering and annotating sliced ​​images, comprising the following steps:

[0042] S1. Obtain brain slice images, and use the trained first generator based on the adversarial generative network CycleGAN to perform style transfer on the brain slice images to obtain style-transferred brain slice images.

[0043] S1 includes the following steps:

[0044] S11. Obtain several brain slice images of different modalities with coordinate information;

[0045] In this embodiment, several brain slice images containing CCFv3 standard coordinate information in various modalities were obtained from the database published by the Allen Institute for Brain Science. The obtained brain slice images include approximately 45,000 bright-field images and 33,000 dark-field images. Bright-field images include in situ hybridization (ISH) and Nissl staining, while dark-field images include fluorescence in situ hybridization (FISH), biotinylated dextranamine (BDA), and two-photon imaging.

[0046] S12, group the acquired brain slice images according to the imaging modes of bright field and dark field to obtain a first data set composed of bright field brain slice images and a second data set composed of dark field brain slice images;

[0047] S13, based on the adversarial generative network CycleGAN, according to the corresponding relationship between the bright field brain slice images in the first data set and the dark field brain slice images in the second data set and the Allen standard atlas slice images, the first generator is trained by unsupervised style transfer learning to obtain a trained first generator;

[0048] S14, based on the trained first generator, the brain slice images are processed by style transfer to obtain the brain slice images after style transfer.

[0049] In the scheme, by unsupervised style transfer learning between the brain slice images in the bright field brain slice image data set and the dark field brain slice image data set and the Allen standard atlas slice images, the structural details of the brain slice images can be preserved, and the style of the brain slice images of different imaging modalities is unified to the standard atlas style, thereby realizing the standardization of cross-modal brain imaging images.

[0050] As shown in Figure 2 The adversarial generative network CycleGAN includes a first generator Generator G and a second generator Generator F based on the ResNet-Unet structure, and a first discriminator Discriminator X and a second discriminator Discriminator Y based on the PatchGAN structure;

[0051] The bright field brain slice images in the first data set and the dark field brain slice images in the second data set are used as source domain images, and the Allen standard atlas slice images are used as target domain images;

[0052] The first generator Generator G is used to read the original source domain images and generate fake target domain images;

[0053] As shown in Figure 3As shown, the first generator Generator G and the second generator Generator F each include a first padding layer, a first convolutional layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a first residual module, a dense connection module, a self-attention module, a second residual module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a second padding layer and a second convolutional layer connected in sequence, wherein the input end of the first down-sampling layer and the output end of the third up-sampling layer are connected through channel dimension splicing to form a skip connection, the input end of the second down-sampling layer and the output end of the second up-sampling layer are connected through channel dimension splicing to form a skip connection, and the input end of the third down-sampling layer and the output end of the first up-sampling layer are connected through channel dimension splicing to form a skip connection.

[0054] Through style transfer learning of the adversarial generation network CycleGAN, the first generator Generator G can complete style conversion between bright field brain slice images, dark field brain slice images and Allen standard atlas slice images without relying on manual screening of a large amount of data and without strict pairing of the training set; the first generator Generator G and the second generator Generator F integrate ResNet modules to improve the stability of deep network training, and introduce a U-Net skip connection structure in the up-sampling stage to improve the image detail restoration capability.

[0055] As shown in Figure 4 In the present scheme, the first residual module includes eight residual blocks Residual Block connected in sequence, and the second residual module includes seven residual blocks Residual Block connected in sequence, wherein the residual blocks Residual Block in the first residual module and the second residual module include a third padding layer, a third convolutional layer, a first normalization layer, a first activation layer, a fourth padding layer, a fourth convolutional layer, a second normalization layer and a residual connection layer connected in sequence.

[0056] As shown in Figure 5 The dense connection module includes a first full-channel splicing layer, a third normalization layer, a second activation layer, a fourth convolutional layer and a second full-channel splicing layer connected in sequence.

[0057] The second generator Generator F is configured to read original target domain images and generate fake source domain images;

[0058] The first discriminator Discriminator X is configured to distinguish whether an input image is a real target domain image or a fake target domain image generated by the first generator Generator G;

[0059] The second discriminator Discriminator Y is used to discriminate whether the input image is a real source domain image or a fake source domain image generated by the second generator Generator F.

[0060] In the scheme, the CycleGAN is an image-to-image translation framework without paired training data, which is designed to solve the unsupervised style transfer problem, and the training process of the model does not need strict sample pair dependence, thereby saving the cost of the training model process, and the generated image can maintain the original image structure information while performing style transfer, thereby realizing high-level reconstruction of the style.

[0061] The first generator Generator G provided in the scheme has unsupervised learning ability, does not need strict paired training data, can retain the structure information of the image while completing style conversion, and the discriminator improves the style discrimination ability by focusing on the local texture structure, and is suitable for medical image style conversion tasks.

[0062] The loss function of the CycleGAN is calculated according to the following expression:

[0063] ,

[0064] Among them, L represents the loss function of the CycleGAN, G represents the first generator Generator G, F represents the second generator Generator F, X represents the first discriminator Discriminator X, Y represents the second discriminator Discriminator Y, L represents the first adversarial loss, S represents the source domain, T represents the target domain, L represents the second adversarial loss, represents the weight of the cycle consistency loss, L represents the cycle consistency loss, represents the ratio of the identity loss to the cycle consistency loss, L represents the identity loss.

[0065] The adversarial generative network CycleGAN improves the performance of the generator and the discriminator through adversarial training between the generator and the discriminator, and the ultimate goal is to make the fake target domain image generated by the first generator Generator G indistinguishable from the first discriminator Discriminator X, so the first loss function of the adversarial generative network CycleGAN is set as an adversarial loss; since the fake target domain image generated by the first generator Generator G reading the original source domain image must be able to be read by the second generator Generator F and generate a fake source domain image similar to the original source domain image to ensure that information is not lost during image style transfer, two pairs of generators and two pairs of discriminators are needed to form a cycle network, so the second loss function of the adversarial generative network CycleGAN is set as a cycle consistency loss to ensure that the mapping learned by the generator does not arbitrarily modify the image; for the case where the input image is a real target domain image, in order to avoid unnecessary color or style distortion, the fake target domain image generated by the first generator Generator G should be similar to the input image itself, so the third loss function of the adversarial generative network CycleGAN is identity loss.

[0066] S2, using the improved ConvNext neural network model trained to perform atlas level position recognition on the brain slice image after style transfer, and pairing the brain slice image after style transfer with the Allen landmark picture slice image to form a brain slice image matching pair;

[0067] Since the brain slice image after style transfer and the Allen landmark picture slice image may differ in size, tilt and detail level, further affine and nonlinear registration is needed to improve the matching accuracy.

[0068] The S2 includes the following steps:

[0069] S21, using the improved ConvNext neural network model to perform atlas level position recognition on the brain slice image after style transfer, and predicting the spatial position of the brain slice image after style transfer in the Allen 3D brain atlas as the predicted spatial position.

[0070] The improved ConvNext neural network model in the S21 replaces the softmax layer at the end of the ConvNext neural network model with a Relu activation function layer, two consecutive dense connection modules, and an output layer outputting 9 coordinates.

[0071] In the scheme, the improved ConvNext model can identify the layers of mouse brain slice images from multiple different data sources and generate 3 groups of spatial positioning vectors, each group of spatial positioning vectors corresponding to 3 coordinates, i.e. the first 3 coordinates of the 9 coordinates output by the output layer are used to determine the 2D plane in which the brain slice image after style transfer is located in the Allen 3D brain atlas space, and the last 6 coordinates are used to determine the specific angle of the brain slice image in the 2D plane. Through the QuickNII software, the specific information of the layers in the Allen 3D brain atlas space can be viewed to realize accurate spatial positioning.

[0072] In the embodiment, when training the improved ConvNext neural network model to identify the atlas layer position, the samples in the data set are composed of brain slice images and their true 9 coordinates for pairing 3D brain atlas; the brain slice images of a plurality of samples are input into the improved ConvNext neural network model, so as to output the predicted coordinates of the 9 3D brain atlas for pairing about the brain slice images of the samples; based on the accuracy between the predicted coordinates and the true coordinates, the network parameters of the improved ConvNext neural network model are optimized, so that when the accuracy between the predicted coordinates and the true coordinates reaches a preset accuracy threshold, a trained improved ConvNext neural network model is obtained, which can accurately and effectively extract the spatial position of the brain slice image after style transfer in the Allen 3D brain atlas, i.e. the 9 predicted coordinates corresponding to the brain slice image after style transfer.

[0073] S22, based on the predicted spatial position, extract the corresponding standard two-dimensional atlas image, and pair the brain slice image after style transfer with the Allen landmark picture slice image to form a brain slice image matching pair.

[0074] S3, input a plurality of brain slice image matching pairs into the Voxelmorph registration model to obtain a nonlinear registration vector field between the brain slice image and the Allen landmark picture slice image;

[0075] S4, based on the nonlinear registration vector field, register the atlas information of the Allen standard atlas slice image to the brain slice image to complete the brain slice image registration and academic annotation.

[0076] The voxelmorph registration model is a convolutional neural network model based on the Unet structure. The voxelmorph registration model learns a deformation field (including affine and non-linear transformation) to align the Allen standard atlas slice image to the brain slice image, and can realize accurate pixel-level spatial registration. The process is carried out under semi-supervised conditions, which can effectively reduce the cost of manual annotation. The voxelmorph registration model is used to learn the 2D Allen standard atlas slice image, and a deformation field is generated to make it undergo affine transformation and non-linear transformation, that is, the difference between the atlas image and the style transfer image in size, inclination and details can be reduced, so as to realize the complete fitting and registration of the atlas image to the style transfer image. Since the atlas image contains detailed anatomical structure information, and the style transfer image retains the structural features of the original image, the registration result can finally be mapped back to the original slice image, realizing high-precision brain slice registration with structural annotation.

[0077] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which shall be covered within the protection scope of the present application.

Claims

1. A method for slice image registration and annotation, characterized in that, The method comprises the following steps: S1, acquiring brain slice images, and performing style transfer on the brain slice images based on a generative adversarial network and using a trained first generator to obtain brain slice images after style transfer; The S1 comprises the following steps: S11, acquiring brain slice images of different modalities with coordinate information; S12, grouping the acquired brain slice images according to bright field and dark field imaging modes to obtain a first data set composed of bright field brain slice images and a second data set composed of dark field brain slice images; S13, based on a CycleGAN adversarial generation network, performing unsupervised style transfer learning training on the first generator according to the corresponding relationship between the bright field brain slice images in the first data set and the dark field brain slice images in the second data set and the Allen standard atlas slice images, to obtain the trained first generator; The CycleGAN adversarial generation network comprises a first generator and a second generator based on a ResNet-Unet structure, and a first discriminator and a second discriminator based on a PatchGAN structure; The bright field brain slice images in the first data set and the dark field brain slice images in the second data set are used as source domain images, and the Allen standard atlas slice images are used as target domain images; The first generator is used to read original source domain images and generate fake target domain images; The second generator is used to read original target domain images and generate fake source domain images; The first discriminator is used to distinguish whether the input image is a real target domain image or a fake target domain image generated by the first generator; The second discriminator is used to distinguish whether the input image is a real source domain image or a fake source domain image generated by the second generator; S14, performing style transfer processing on the brain slice images based on the trained first generator to obtain brain slice images after style transfer; S2, using a trained improved ConvNext neural network model to perform atlas layer position recognition on the brain slice images after style transfer, and pairing the brain slice images after style transfer with Allen landmark picture slice images to form brain slice image matching pairs; The S2 comprises the following steps: S21, using the trained improved ConvNext neural network model to perform atlas layer position recognition on the brain slice images after style transfer, and predicting the spatial position of the brain slice images after style transfer in the Allen 3D brain atlas as a predicted spatial position; S22, based on the predicted spatial position, extracting a corresponding standard two-dimensional atlas image as an Allen landmark picture slice image, and pairing the brain slice images after style transfer with the Allen landmark picture slice images to form brain slice image matching pairs; S3, inputting a plurality of brain slice image matching pairs into a Voxelmorph registration model to obtain a nonlinear registration vector field between the brain slice images and the Allen landmark picture slice images; S4, based on the nonlinear registration vector field, registering atlas information of the Allen landmark picture slice images to the brain slice images to complete brain slice image registration and academic annotation.

2. The method of slice image registration and annotation of claim 1, wherein, The calculation expression of the loss function of the adversarial generation network CycleGAN is as follows: ， wherein, denotes a loss function of the adversarial generative network CycleGAN, denotes a first generator, denotes a second generator, denotes a first discriminator, denotes a second discriminator, denotes a first adversarial loss, denotes a source domain, denotes a target domain, denotes a second adversarial loss, denotes a weight of the cycle consistency loss, denotes a cycle consistency loss, denotes a ratio of the identity loss to the cycle consistency loss, denotes an identity loss.

3. The method of slice image registration and annotation of claim 1, wherein, The first generator and the second generator each comprise a first padding layer, a first convolution layer, a first down-sampling layer, a second down-sampling layer, a third down-sampling layer, a first residual module, a dense connection module, a self-attention module, a second residual module, a first up-sampling module, a second up-sampling module, a third up-sampling module, a second padding layer and a second convolution layer connected in sequence, wherein the input end of the first down-sampling layer and the output end of the third up-sampling layer are connected through channel dimension splicing to form a skip connection, the input end of the second down-sampling layer and the output end of the second up-sampling layer are connected through channel dimension splicing to form a skip connection, and the input end of the third down-sampling layer and the output end of the first up-sampling layer are connected through channel dimension splicing to form a skip connection.

4. The method of slice image registration and annotation of claim 1, wherein, The improved ConvNext neural network model in the S21 replaces the softmax layer at the end of the ConvNext neural network model with a Relu activation function layer, two consecutive dense connection modules and an output layer outputting 9 coordinates connected in sequence.

Images