A two-stage method, apparatus, and storage medium for generating brain vascular images and tags.

Abstract

This invention discloses a two-stage method for generating brain vascular images and labels. First, labeled and unlabeled brain vascular images are processed separately to obtain training set 1 and training set 2. Then, a two-stage generative adversarial network (GAN) model for brain vascular image-label generation is constructed and trained using training set 1 and training set 2. Finally, N-dimensional noise is input into the trained model to obtain a complete brain vascular image. This invention utilizes a two-stage GAN model with pseudo-labels and real data input to generate brain vascular images and labels respectively. The generated brain vascular images have clearer voxels around the blood vessels and stronger connectivity of the vascular labels. Data augmentation applied to segmentation tasks can improve the accuracy of brain vascular structures segmented from brain vascular images, facilitating subsequent pathological image diagnosis. Simultaneously, the use of pseudo-labels alleviates the shortage of labeled medical image data.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology, specifically relating to a two-stage method, apparatus, and storage medium for generating brain blood vessel images and labels. Background Technology

[0002] In cerebrovascular image segmentation, supervised deep learning segmentation methods require training network models with a large amount of paired image-label data. However, obtaining such a large amount of training data is difficult, and the segmentation performance of the network model is limited by insufficient data. Although traditional data augmentation methods such as cropping and flipping have solved the problem of insufficient data to some extent, the sample diversity generated by these methods is limited and it is difficult to include anatomical variations outside the training samples. Therefore, an automatic image generation method is needed to generate a large amount of the required training data.

[0003] Existing methods for generating brain vessel images and labels directly generate images and labels by inputting random noise into the generator, i.e., a single-stage generation approach. This method produces image-label data with a distribution close to the overall distribution of the training data; however, while improving sample diversity, it sacrifices some local image information. In brain vessel segmentation tasks, the segmentation network focuses more on voxels of the blood vessels and surrounding areas, as these voxels determine the edges of the segmented vessels. The texture structure of the blood vessels in the image is not clear enough, resulting in poor connectivity of the generated labels and failing to guarantee the accuracy of the generated blood vessel voxels. Furthermore, the training of existing methods still requires a large amount of data. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a two-stage method for generating cerebral vascular images and labels, comprising the following steps:

[0005] Step 1: Preprocess the image data and label data of the labeled cerebral vascular images, and randomly generate N-dimensional noise. Use the N-dimensional noise, the preprocessed image data, and the label data as training set 1.

[0006] Step 2: Perform topological constraint segmentation on the unlabeled cerebral vascular images to obtain pseudo-labels, and use the pseudo-label data as training set 2;

[0007] Step 3: Construct a two-stage generative adversarial network model for cerebral vascular images and labels. The model includes a cascaded label generation network and an image generation network.

[0008] Step 4: Input training set 1 and training set 2 into the two-stage generative adversarial network model of cerebral blood vessel image-label respectively for training, and obtain the trained two-stage generative adversarial network model of cerebral blood vessel image-label.

[0009] Step 5: Input N-dimensional noise into the trained two-stage generative adversarial network model of cerebral blood vessel image-label to obtain image-label data pairs, and then generate complete cerebral blood vessel images.

[0010] Further, step 1 is specifically as follows: normalize and augment the image data and label data, generate N-dimensional noise using random distribution, and convert the N-dimensional noise, augmented image data, and label data into parallel training form to form training set 1.

[0011] Furthermore, the topology constraints described in step 2 employ U-Net combined with hybrid loss function constraints.

[0012] Furthermore, in step 3,

[0013] Label generation network: includes a label generator and a label discriminator; used to map the distribution of random noise to the distribution of cerebral blood vessel labels and train it to generate cerebral blood vessel labels and send them to the image generation network;

[0014] Image generation network: includes an image generator and an image discriminator; used to add texture structure to the received cerebral blood vessel labels, and increase the realism of the texture through training, thereby generating complete cerebral blood vessel images.

[0015] Furthermore, in the aforementioned tag generation network,

[0016] The label generator consists of multiple decoding modules and a feature fusion module. Each decoding module contains a fixed-stride 3D deconvolution and is connected to batch normalization and Leaky ReLU activation functions. The last feature fusion module uses a fixed-stride 3D deconvolution to output an image with a fixed channel size and uses an activation function to limit the range of output values.

[0017] The label discriminator consists of multiple feature extraction modules and a fully connected layer. Each feature extraction module includes a fixed-stride 3D convolution, instance normalization, and a Leaky ReLU activation function. The feature map output by the last module is flattened before being input into the fully connected layer.

[0018] Furthermore, in the image generation network,

[0019] The image generator consists of an encoding module and a decoding module. The encoding module is composed of multiple convolutional blocks connected together. Each convolutional block consists of a fixed-stride 3D convolutional connection, instance normalization, and Leaky ReLU activation function. The decoding module is composed of multiple fixed-stride 3D deconvolutions connected together. The 3D deconvolutions are connected by batch instance normalization and Leaky ReLU activation function. The last decoding module uses a fixed-stride 3D deconvolution, and the output channel values ​​are consistent with the label image channel values, and the output value range is limited by an activation function.

[0020] The image discriminator consists of multiple feature extraction modules and a feature fusion module: each feature extraction module contains a fixed-stride 3D convolution, instance normalization, and a Leaky ReLU activation function; the feature fusion module contains a fixed-stride 3D convolution.

[0021] Furthermore, the number of feature extraction modules in the image discriminator is determined by the output feature map, and the output feature map is not all 1 in all three dimensions.

[0022] Furthermore, step 4 is specifically as follows:

[0023] Step 4.1, parameter initialization, including: setting the training number threshold, training batch size, network parameter generator learning rate, discriminator learning rate, and randomly initializing network weights;

[0024] Step 4.2: Input N-dimensional random noise into the two-stage generative adversarial network model of cerebral vascular image-label for forward propagation to obtain the generated cerebral vascular labels and images;

[0025] Step 4.3 Based on the generated cerebral vascular labels and images, compare them with the real cerebral vascular labels and images, and use the loss function to calculate the training error of the two-stage generative adversarial network model of cerebral vascular image-label.

[0026] Step 4.4: Update and optimize network parameters based on the obtained training error through backpropagation;

[0027] Step 4.5: Determine whether the number of training iterations has reached the set threshold. If yes, the training is complete and the network model parameters are saved. Otherwise, return to step 4.2.

[0028] A two-stage device for generating images and labels of cerebral blood vessels, comprising:

[0029] processor;

[0030] Storage medium: Used to store one or more programs;

[0031] When the one or more programs are executed by a processor, the processor performs the method as described in any one of claims 1 to 8.

[0032] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

[0033] Advantages of this invention:

[0034] This invention presents a two-stage method for generating brain vascular images and labels. It utilizes a two-stage generative adversarial network model with pseudo-labels and real data input to generate brain vascular images and labels respectively. The generated brain vascular images have clearer voxels around the blood vessels, and the vascular labels have strong connectivity. When applied to segmentation tasks, data augmentation can improve the accuracy of brain vascular structures segmented from brain vascular images, which is helpful for subsequent pathological image diagnosis. At the same time, the use of pseudo-labels alleviates the problem of insufficient medical image label data. Attached Figure Description

[0035] Figure 1 This is a flowchart of a two-stage method for generating cerebral vascular images and labels according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of the structure of a two-stage generative adversarial network model for cerebral vascular images and labels according to an embodiment of the present invention;

[0037] Figure 3 This invention provides a single-stage generation method, a two-stage generation method, and an image-label 3D rendering of a real image.

[0038] Figure 4 The images show real labels and 3D renderings of segmentation results after data augmentation using traditional data augmentation, single-stage generation methods, and two-stage generation methods, respectively, according to one embodiment of the present invention. Detailed Implementation

[0039] An embodiment of the present invention will be further described below with reference to the accompanying drawings.

[0040] In this embodiment of the invention, a two-stage method for generating cerebral vascular images and labels is described in the flowchart below. Figure 1 As shown, it includes the following steps:

[0041] Step 1: Preprocess the labeled cerebral vascular image data and label data, and generate N-dimensional noise using a random distribution. Use the N-dimensional noise, the preprocessed image data, and the label data as training set 1, as follows:

[0042] Step 1.1: Read the image data and label data of the labeled cerebral blood vessels and perform normalization processing using the following formula:

[0043]

[0044] Among them, v * This represents the normalized image data or label data, where v represents the input image. min v represents the minimum value in image or label data. max This represents the maximum value in the image or label data.

[0045] Step 1.2: Generate 128-dimensional noise using a Gaussian distribution;

[0046] Step 1.3: Perform data augmentation on the normalized image data and label data; the data augmentation adopts the random cropping method, randomly selects positions from three directions of the pixel image, and crops it according to the set volume of 128×128×64 to obtain voxel data which is input into the neural network for training of subsequent tasks.

[0047] Step 1.4: Set the data precision. Convert the 128-dimensional noise, data-augmented image data, and label data into a parallel training format to form the total training set. Divide the total training set into training set 1 and test set in a 7:3 ratio.

[0048] Step 2: Perform topological constraint segmentation on the unlabeled cerebral vascular images to generate pseudo-labels. Use the pseudo-labeled data to form training set 2, as follows:

[0049] The topology of the pseudo-labels segmented using U-Net combined with a hybrid loss function is calculated as follows:

[0050]

[0051]

[0052] L = Loss Dice +Loss clDice (4)

[0053] Among them, L Dice The value represents the Dice similarity coefficient loss, N represents the sum of all voxels, i represents the number of brain vascular tag voxels, and t represents the total number of voxels. i y represents the i-th voxel value of the cerebral vascular tag. i L represents the i-th voxel value of the segmented pseudo-label, ε represents the smoothing coefficient used to smooth the loss value and gradient, and L represents the i-th voxel value of the segmented pseudo-label. clDice p represents the Dice similarity coefficient loss of the equilibrium centerline. i This represents the centerline extracted from the pseudo-labels, m i Let L represent the centerline extracted from the i-th cerebral vascular tag, and L represent the mixed loss function.

[0054] Step 3: Construct a two-stage generative adversarial network model for cerebral vascular images and labels, such as... Figure 2 As shown, the model includes a cascaded label generation network and an image generation network;

[0055] Label generation network: used to map the distribution of random noise to the distribution of cerebral blood vessel labels using a label generator, and trained with a discriminator to generate cerebral blood vessel labels which are then sent to the image generation network.

[0056] Image generation network: used to add texture structure to the generated brain blood vessel label input image generator, and use image discriminator to train the realism of the added texture, and finally generate a complete brain blood vessel image.

[0057] The aforementioned tag generation network includes a tag generator and a tag discriminator:

[0058] The label generator consists of five decoding modules and one feature fusion module. Each decoding module contains a 4×4×4 three-dimensional deconvolution with a stride of 2, connected with batch normalization and Leaky ReLU activation function. The kernel sizes in each decoding module are 256, 128, 64 and 32 respectively. The last feature fusion module uses a 4×4×4 three-dimensional deconvolution with a stride of 2, outputs a label image with 1 channel, and uses a sigmoid activation function to limit the output value range to [0,1].

[0059] The label discriminator consists of three feature extraction modules, a feature fusion layer, and a fully connected layer. Each feature extraction module contains a 4×4×4 3D convolution with a stride of 2, instance normalization, and a Leaky ReLU activation function. The feature fusion layer uses a 4×4×4 convolution with a stride of 1 to fuse the extracted cerebral vascular label features. The kernel sizes of the convolutions in each feature extraction module are 64, 128, and 256, respectively. The feature map output from the last module is flattened before being input into the fully connected layer.

[0060] The image generation network includes an image generator and an image discriminator:

[0061] The image generator consists of an encoding module and a decoding module. The encoding module is composed of four convolutional blocks connected together. Each convolutional block consists of a 3×3×3 three-dimensional convolution with a stride of 2, followed by instance normalization and the Leaky ReLU activation function. The decoding module consists of four 3×3×3 three-dimensional deconvolutions with a stride of 2 connected together. The three-dimensional deconvolutions are connected by batch instance normalization and the Leaky ReLU activation function. The kernel sizes in each level of the encoding module are 32, 64, 128, and 256, respectively, and the decoding module at the same level has the same number of kernels as the encoding module. The last layer module uses a 3×3×3 three-dimensional deconvolution with a stride of 2, and the output channel values ​​are consistent with the label image channel values. The output value range is limited to [0,1] using the sigmoid activation function.

[0062] The image discriminator consists of four feature extraction modules and one feature fusion module: each feature extraction module contains a 3×3×3 three-dimensional convolution with a stride of 2, instance normalization, and a Leaky ReLU activation function; the feature fusion module contains a 3×3×3 three-dimensional convolution with a stride of 1, and the output feature map size is [1,8,8,4]; the kernel sizes of each feature extraction module are 64, 128, 256, and 512, respectively.

[0063] Step 4: Input training set 1 and training set 2 into the two-stage generative adversarial network model of cerebral vascular images and labels respectively for training, and obtain the trained two-stage generative adversarial network model of cerebral vascular images and labels, as follows:

[0064] Step 4.1: Input training set 1 and training set 2 into the label generation network for training to obtain the trained label generation network, specifically:

[0065] Step 4.1.1, parameter initialization, including: setting the training count threshold to 20000, the training batch size to 6, the learning rate of both the label generator and the label discriminator to 0.0002, and randomly initializing the network weights;

[0066] Step 4.1.2: Input the 128-dimensional noise into the label generation network for forward propagation to obtain the generated cerebrovascular labels;

[0067] Step 4.1.3: Based on the generated cerebral vascular labels and the real cerebral vascular labels, calculate the training error of the label generation network using the loss function;

[0068] The loss functions for the label generator and the label discriminator are as follows:

[0069]

[0070]

[0071] Where E represents the expectation calculation of the data, x represents the cerebral vascular labeled image, and p g p represents the distribution of the generated images. data This represents the distribution of the true labels. D represents the distribution of the sampled data, and D(·) represents the output of the label discriminator.

[0072] Step 4.1.4: Update and optimize the labels and generate network parameters based on the obtained training error through backpropagation;

[0073] Network parameter updates use the Adam optimizer, and the parameter update strategy is as follows:

[0074]

[0075] Where, θ t θ represents the convolution weight parameters in the updated neural network. t-1 The values ​​represent the convolution weights in the neural network before the update, and lr represents the current learning rate. This represents the correction for the first moment estimate of the gradient; This indicates a correction to the second-moment estimate of the gradient;

[0076] Step 4.1.5: Determine whether the number of training iterations has reached the set threshold. If yes, the training is complete and the network model parameters are saved. Otherwise, return to step 4.1.2.

[0077] Step 4.2: Input training set 1 and training set 2 into the image generation network for training to obtain the trained image generation network, as follows:

[0078] Step 4.2.1, parameter initialization, including: setting the training iteration threshold to 20000, the training batch size to 6, the learning rate of both the image generator and the image discriminator to 0.0002, and randomly initializing the network weights;

[0079] Step 4.2.2: Input the 128-dimensional noise into the image generation network and perform forward propagation to obtain the generated cerebral vascular image;

[0080] Step 4.2.3: Based on the generated cerebral vascular images and the real cerebral vascular images, calculate the training error of the image generation network using the loss function;

[0081] The loss functions for the image generator and the image discriminator are as follows:

[0082] D_loss=L MSE (D(x,y),1)+L MSE (D(x,G(z)),0) (8)

[0083] G_loss=L MSE (D(x,G(z),1)) (9)

[0084] Among them, L MSE Let G(z) represent the mean squared loss function, x represent the cerebral blood vessel labeled image, y represent the cerebral blood vessel image corresponding to x, 1 and 0 are both multi-dimensional vectors with the same output dimension as the image discriminator, and G(z) represent the output of the image generator.

[0085] Step 4.2.4: Update and optimize the image generation network parameters based on the obtained training error through backpropagation. The parameter update method is the same as in step 4.1.4.

[0086] Step 4.2.5: Determine whether the number of training iterations has reached the set threshold. If yes, the training is complete and the network model parameters are saved. Otherwise, return to step 4.2.2.

[0087] Step 5: Input 128-dimensional Gaussian random noise into the trained two-stage generative adversarial network model of cerebral blood vessel image-label to obtain image-label data pairs, and then generate complete cerebral blood vessel images.

[0088] In embodiments of the present invention, such as Figure 3 As shown, the three sets of images represent a qualitative analysis comparison of the two-stage generation method, the single-stage generation method, and real images using a test set.

[0089] Compared to real images, the single-stage generation method exhibits several differences in voxel intensity in brain blood vessel images. The diversity of voxel intensity patterns in the generated blood vessels is low, and blurring of voxels around the vessels is observed. Furthermore, due to noise in the pseudo-labels, this single-stage generation method is affected by noise in the generated results, producing noticeable artifacts.

[0090] Compared with single-stage generation methods, the method of this invention has better connectivity of vascular labels, higher diversity of generated vascular voxel intensity, and is closer to the vascular voxel intensity of real images, with higher quality voxels around blood vessels; the method of generating brain blood vessels separately can better resist the interference of noise in pseudo-labels and generate fewer artifacts.

[0091] In this embodiment of the invention, Figure 4This invention reflects the qualitative segmentation performance of the segmentation network after using three data augmentation methods. In this embodiment, Hausdorff distance (95HD) with a maximum distance quantile of 95, Dice similarity coefficient (DSC), and centerline Dice (clDice) are used as quantitative evaluation indicators for segmentation performance. The segmentation results are compared with the gold standard to calculate these indicators, and the segmentation results are then quantitatively analyzed. Through experimental results comparison, compared with traditional data augmentation methods based on random pruning, the method of this invention achieves higher performance improvements in multiple indicators for the segmentation network. Compared with single-stage generation methods, it exhibits better segmentation performance in terms of overall segmentation accuracy and blood vessel connectivity.

[0092] Table 1

[0093]

[0094] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for brain vessel image-label two-stage generation, characterized in that: The method comprises the following steps: Step 1, pre-process the image data and label data of the labeled cerebral vascular image, and randomly generate dimensional noise, and dimensional noise, the pre-processed image data and label data as training set 1; Step 2: topological structure constraint segmentation is performed on the unlabeled cerebral vascular image to obtain pseudo-labels, and the pseudo-label data are taken as a training set 2; Step 3: a cerebral vascular image-label two-stage generative adversarial network model is constructed, and the model comprises a label generator and an image generator in series: The label generator comprises a plurality of decoding modules and a feature fusion module, each decoding module comprises a fixed-step three-dimensional deconvolution, and is connected with a batch normalization and a Leaky ReLU activation function; the last layer feature fusion module adopts a fixed-step three-dimensional deconvolution, outputs an image with a fixed channel size, and uses an activation function to limit the output value range; The label discriminator comprises a plurality of feature extraction modules and a full connection layer, each feature extraction module comprises a fixed-step three-dimensional convolution, an instance normalization and a Leaky ReLU activation function; and the feature map output by the last layer module is subjected to a flattening operation and then input into the full connection layer; The image generator comprises an encoding module and a decoding module; the encoding module comprises a plurality of convolution blocks connected in series; each convolution block comprises a fixed-step three-dimensional convolution connected with an instance normalization and a Leaky ReLU activation function; the decoding module comprises a plurality of fixed-step three-dimensional deconvolutions connected in series; the three-dimensional deconvolutions are connected through an instance normalization-Leaky ReLU activation function; the last layer decoding module adopts a fixed-step three-dimensional deconvolution, outputs an image with a channel value consistent with that of the label image, and uses an activation function to limit the output value range; The image discriminator comprises a plurality of feature extraction modules and a feature fusion module: each feature extraction module comprises a fixed-step three-dimensional convolution, an instance normalization and a Leaky ReLU activation function; and the feature fusion module comprises a fixed-step three-dimensional convolution; The number of the feature extraction modules is determined by the output feature map, and the output feature map is not all 1 in three dimensions; Step 4: the training set 1 and the training set 2 are input into the cerebral vascular image-label two-stage generative adversarial network model for training, and a trained cerebral vascular image-label two-stage generative adversarial network model is obtained; The topological structure constraint in step 2 adopts a U-Net combined with a hybrid loss function constraint. The step 4 is specifically as follows: Step 5, adding The white noise is input into the trained cerebral vascular image-label two-stage generative adversarial network model to obtain an image-label data pair, and then a complete cerebral vascular image is generated.

2. The brain vessel image-label two-stage generation method according to claim 1, characterized in that: Step 1 is specifically as follows: Normalize and augment the image data and label data, and generate data using a random distribution. Dimensional noise, The noise, data augmented image data, and label data are all converted into parallel training data to form training set 1.

3. The brain vessel image-label two-stage generation method according to claim 1, characterized in that: Step 4.1: parameter initialization, including setting a training frequency threshold, a training batch size, a network parameter generator learning rate, a discriminator learning rate and randomly initializing network weights; 4. The brain vessel image-label two-stage generation method according to claim 1, characterized in that: Step 4.3: according to the generated cerebral vascular label and image, the real cerebral vascular label and image are compared, and a loss function is used to calculate the training error of the cerebral vascular image-label two-stage generative adversarial network model; ​ Step 4.2, the Forward propagation of the two-stage generative adversarial network model with random noise input brain blood vessel image-label, to obtain the generated brain blood vessel label and image; ​ Step 4.4, updating the optimization network parameters according to the obtained training error back propagation; Step 4.5, judging whether the training number reaches the set training number threshold, if yes, the training is completed, the network model parameters are saved, otherwise, returning to execute step 4.

2. 5.A brain vessel image-label two-stage generation apparatus, characterized by comprising: Comprising: a processor; a storage medium for storing one or more programs; when the one or more programs are executed by the processor, the processor implements the method as claimed in any one of claims 1 to 4.

6. A computer readable storage medium storing a computer program, characterized in that: The program is executed by the processor to implement the method as claimed in any one of claims 1 to 4.

Images