Fully automatic near-infrared vein segmentation method based on semi-supervised learning

Abstract

This invention discloses a fully automatic near-infrared vein segmentation method based on semi-supervised learning, which relates to the technical field of near-infrared limb imaging in medical image processing. The method comprises: S1: acquiring near-infrared limb imaging to produce vein segmentation images, performing preprocessing, and constructing a dataset; S2: constructing a semantic segmentation model based on a recursive scaling recurrent neural network, supplemented by a reconstructed consistency neural network and a segmentation adversarial neural network; S3: training and iteratively optimizing the semantic segmentation model. Based on the constructed datasets of labeled and unlabeled parts, the method trains the segmentation model by inputting labeled data, generates a prediction graph by inputting unlabeled data, and optimizes the model through the reconstruction of the consistency neural network and the segmentation adversarial neural network, thereby improving the accuracy and speed of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing for vein image segmentation, and in particular to a fully automatic near-infrared vein segmentation method based on semi-supervised learning. Background Art

[0002] Intravenous injection, as an important part of the medical field, currently still relies on manual operation procedures based on the clinician's subjective consciousness. Under the guidance of observation, inspection and palpation of peripheral veins, it is necessary to roughly determine the vascular area, select and locate a suitable vein, and insert the needle into its center. For inexperienced clinicians, the accuracy of the first injection is very low, and multiple injection attempts will increase the patient's pain.

[0003] In the absence of sufficient visible light, peripheral intravenous injections require optical auxiliary equipment and algorithms to correctly guide the injection process. For example, near-infrared optical imaging technology based on deep learning uses near-infrared light from lasers or light-emitting diodes to enhance image contrast to display subcutaneous blood vessels in skin tissue. Deep neural networks contain millions of parameters and require a large amount of labeled data. This leads to the idea of ​​using semi-supervised semantic segmentation to refine the network structure and parameters using limited labeled data and a large amount of unlabeled data to further improve the performance of medical image segmentation. However, existing segmentation methods all have more or less the disadvantages of unsatisfactory segmentation results, low segmentation efficiency, and long segmentation time. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a fully automatic near-infrared vein segmentation method based on semi-supervised learning, constructs a semantic segmentation model, which is based on a recursive neural network. The segmentation model is trained by inputting labeled data and unlabeled data to generate a prediction map. Then, a reconstruction consistency network model is constructed, and the latent coding feature representation and prediction results of the recursive neural network are used to reconstruct the labeled and unlabeled data to optimize the recursive neural network. Finally, the recursive neural network is optimized through the segmentation adversarial model to improve the accuracy of the model.

[0005] The present invention provides a fully automatic near-infrared vein segmentation method based on semi-supervised learning, the method comprising:

[0006] S1: Acquire near-infrared limb imaging to produce a vein segmentation image, segment the vein portion from the original image, and perform preprocessing to construct a dataset, which includes a supervised part and an unsupervised part;

[0007] S2: Constructing a semantic segmentation model, which includes a three-branch semi-supervised deep learning neural network model, namely, a recursive scaling recurrent neural network model, a reconstruction consistency neural network model, and a segmentation adversarial neural network model;

[0008] S3: Training the semantic segmentation model, training and extracting features of the recursive scaled recurrent neural network model based on the data set; and iteratively optimizing the recursive scaled recurrent neural network model through the reconstruction consistency neural network model and the segmentation adversarial neural network model.

[0009] Furthermore, the dataset construction process is as follows:

[0010] S101: calibrating a group of near-infrared camera devices using a baseline distance, and acquiring a series of near-infrared limb vein images through the near-infrared camera devices;

[0011] S102: performing enhancement preprocessing on the near-infrared limb vein image, and annotating the processed image to obtain an annotated image;

[0012] S103: performing binary segmentation on the annotated part and the remaining parts in the annotated image to construct a segmentation data set;

[0013] S104: Using the labeled images and the unlabeled images in the segmentation dataset as datasets of the supervised part and the unsupervised part of the segmentation adversarial model, respectively.

[0014] Furthermore, the binary segmentation is performed by converting the spline curve after adjusting the thickness of each segment in the annotated image into a binary mask in pixel form, thereby achieving segmentation of the vein portion and the remaining portions.

[0015] Furthermore, the recursive scaling recurrent neural network model includes three levels, and the network input resolution of each level increases successively.

[0016] Furthermore, the encoding part of each layer of the recursive scaled recurrent neural network model is connected through a convolutional long short-term memory network.

[0017] Furthermore, the training of the semantic segmentation model specifically includes:

[0018] S301: Inputting the supervised part of the data set and the corresponding pre-processed images into the recursive scaling recurrent neural network model for training;

[0019] S302: Inputting the unsupervised portion of the dataset and the corresponding preprocessed image into the recursive scaling recurrent neural network model trained in step S301 to obtain a segmentation result map and a feature representation of the encoded portion;

[0020] S303: Input the segmentation result image and the feature representation of the encoding part into the reconstruction consistency neural network model, output the reconstruction result image, and optimize the recursive scaling recurrent neural network model based on the reconstruction result image.

[0021] Furthermore, the training of the semantic segmentation model further includes, after step S303:

[0022] S304: Input the supervised part and the unsupervised part of the data set into the segmentation adversarial neural network model, and optimize the recursive scaling recurrent neural network model obtained in step S303 based on the output results.

[0023] The beneficial effects of the present invention are as follows:

[0024] 1. Construct a semantic segmentation model with a recursive scaling recurrent neural network as the main body, and a reconstruction consistency neural network and a segmentation adversarial neural network as auxiliary. Based on the constructed datasets of labeled and unlabeled data, the model is trained and optimized. By refining the resolution layer by layer from low to high, the model realizes adversarial optimization of semantic segmentation from the low-resolution stage to the high-resolution stage. The coherent optimization process makes the network have a high accuracy rate.

[0025] 2. Make full use of unlabeled data. Through the auxiliary role of unlabeled data, under the condition of the same labeled data, the segmentation effect is improved, and the utilization rate and use value of the data set are increased. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0027] Figure 2 is a flowchart of the data set construction of the present invention;

[0028] Figure 3 This is a schematic diagram of the semantic segmentation network model structure proposed in this invention. DETAILED DESCRIPTION

[0029] The following description clearly and completely describes the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts are within the scope of protection of the present invention.

[0030] Example 1

[0031] Embodiment 1 of the present invention provides a fully automatic near-infrared vein segmentation method based on semi-supervised learning, such as Figure 1As shown, the method includes:

[0032] S1: Acquire near-infrared limb imaging to produce a vein segmentation image, segment the vein portion from the original image, and perform preprocessing to construct a dataset, which includes a supervised part and an unsupervised part;

[0033] like Figure 2 As shown, the dataset construction process is as follows:

[0034] S101: calibrating a group of near-infrared camera devices using a baseline distance, and acquiring a series of near-infrared limb vein images through the near-infrared camera devices, wherein the near-infrared limb vein images are images of the upper limbs including veins;

[0035] S102: using a Hessian filter to enhance and pre-process the near-infrared limb vein image, and annotating the processed image using annotation software MicroDicom to obtain an annotated image;

[0036] S103: Binarize and segment the annotated portion and the remaining portion of the annotated image. In this embodiment, the spline curves after adjusting the thickness of each segment in the annotated image are converted into pixel-based binary masks to segment the vein portion and the remaining portion, thereby constructing a segmentation dataset.

[0037] S104: Using the labeled images and the unlabeled images in the segmentation dataset as the datasets of the supervised part and the unsupervised part of the segmentation adversarial model respectively, the datasets including a training sample set, a verification sample set and a test sample set.

[0038] S2: Construct a semantic segmentation model, which includes a three-branch semi-supervised deep learning neural network model, such as Figure 3 As shown, they are, respectively, the recursive scaling recurrent neural network model, the reconstruction consistency neural network model and the segmentation adversarial neural network model;

[0039] The recursive scaling recurrent neural network model is used to segment the vein portion in near-infrared limb images;

[0040] The reconstructed consistency neural network model is used to reconstruct data using the encoding feature representation and prediction results of the recursive scaling recurrent neural network model, optimize the encoding feature representation of the recursive scaling recurrent neural network model, and ensure that the encoding result of the input image is representative;

[0041] The segmentation adversarial neural network model uses adversarial learning to determine the convergence of the recursive scaling recurrent neural network model and improve the model accuracy.

[0042] In this embodiment, the recursive scaling recurrent neural network model structure is divided into three layers, and the input resolution of the three layers increases successively. The feature representation of the encoding part of each layer is connected to the encoding part of the next layer through a convolutional long short-term memory network (ConvLSTM); each layer includes an encoder and a decoder, including a convolutional layer, a residual layer, a replication layer, and a deconvolution layer;

[0043] There are four convolutional layers, with the number of convolution kernels being 32, 64, 128, and 256, respectively. The convolution kernel size is 5×5, and the relu activation function is used with the same padding parameter. Each convolution layer is followed by three residual layers, and each residual layer contains two consecutive convolutional layers. The parameters of the first consecutive convolutional layer are the same as those of the previous convolutional layer, and the second one uses linear activation, and the other parameters are the same as those of the first convolutional layer. During encoding, the convolutional layer is used to obtain the feature map of the input image.

[0044] There are three deconvolution layers, with the number of convolution kernels being 128, 64, and 32, respectively. The convolution kernel size is 2×2, the stride is 2, and the padding parameter is the same. A copy layer is set after the deconvolution layer to copy and bitwise add the feature matrix of the encoder part. The feature map obtained during the concatenation encoding is then subjected to three consecutive residual operations. Each residual layer contains two consecutive convolution layers. The parameters of the first consecutive convolution layer are the same as those of the previous convolution layer, and the second one uses linear activation. The other parameters are the same as those of the first convolution layer. During decoding, the deconvolution network is used to restore pixel information layer by layer. The final segmentation result is obtained by a pixel-level classifier in the last layer.

[0045] The classifiers in this embodiment are as follows:

[0046]

[0047] Among them, a k (X) represents the activation function at feature channel k and pixel position X∈Ω;

[0048] In this embodiment, the convolution kernel in the network model adopts a size of 3×3, the deconvolution kernel adopts a size of 2×2, and the nonlinear mapping between the convolution process and the deconvolution process adopts the Dice activation function as follows:

[0049]

[0050] Among them, p(X) is the true classification result of each pixel value, and θ represents the feature map.

[0051] The reconstruction consistency neural network model includes a convolutional layer, a residual layer, an upsampling layer and a Dense layer;

[0052] The Dense layer has two layers with relu activation function, which are set continuously to receive the encoded feature representation output by the recursive scale recurrent neural network model. After the Dense layer, there are two upsampling layers and convolution layers, as well as three residual layers. The number of convolution kernels in the convolution layer is 16 / 8 respectively, the convolution kernel size is 3×3, the relu activation function is used, and the padding parameter is the same. Each residual layer contains two consecutive convolution layers. The parameters of the first one are the same as the previous convolution layer. The second one uses linear activation, and the other parameters are the same as the first convolution layer. Finally, there is another convolution layer. The number of convolution kernels in this convolution layer is 1, the convolution kernel size is 7×7, the tanh activation function is used, and the padding parameter is the same.

[0053] The segmentation adversarial neural network model includes a convolutional layer and a dense layer;

[0054] There are four convolutional layers, with the number of convolution kernels being 32, 64, 128, and 256 respectively. The convolution kernel size is 3×3, the step size is 2, the leaky-relu activation function is used, the padding parameter is the same, the Batch-Normalization algorithm is used, and the Dense layer has a sigmoid activation function.

[0055] In this embodiment, the network input of the first layer is the low-resolution supervised part data set of limb vein segmentation and the corresponding preprocessing results, and the network input of the next two layers is the successively increasing high-resolution supervised part data set of limb vein segmentation and the segmentation prediction results of the previous layer.

[0056] S3: Training the semantic segmentation model, training and extracting features of the recursive scaled recurrent neural network model based on the data set; and iteratively optimizing the recursive scaled recurrent neural network model through the reconstruction consistency neural network model and the segmentation adversarial neural network model.

[0057] The specific process of training the semantic segmentation model is as follows:

[0058] S301: Inputting the supervised part of the data set and the corresponding pre-processed images into the recursive scaling recurrent neural network model for training;

[0059] During the training process of the recursive scaled recurrent neural network model based on the data set input, the Adam algorithm is finally used for optimization. The specific formula is as follows:

[0060]

[0061] m t =β1·m t-1 +(1-β1)·gt

[0062]

[0063]

[0064]

[0065]

[0066] in, is the gradient operator, β1=0.9, β2=0.999, ε=10 -8 , α=10 -3 , t is the time step, g t is the gradient of the time step, m t and v t They are the exponential moving average of the gradient and the exponential moving average of the square of the gradient, and are initially set to 0.

[0067] S302: Inputting the unsupervised portion of the dataset and the corresponding preprocessed image into the recursive scaling recurrent neural network model trained in step S301 to obtain a segmentation result map and a feature representation of the encoded portion;

[0068] S303: Input the segmentation result image and the feature representation of the encoding part into the reconstruction consistency neural network model, output the reconstruction result image, and optimize the recursive scaling recurrent neural network model based on the reconstruction result image.

[0069] S304: Inputting the supervised part and the unsupervised part of the data set into the segmentation adversarial neural network model, and optimizing the recursive scaling recurrent neural network model obtained in step S303 based on the output results;

[0070] The input of the segmentation adversarial neural network model is the segmentation result and label of the recursive scaling recurrent neural network model, and the corresponding labels are 0 and 1 respectively. Through the adversarial process between the segmentation adversarial model and the recursive scaling recurrent neural network model, the network parameters are continuously improved, thereby optimizing the segmentation result.

[0071] The present invention is not limited to the aforementioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A fully automatic near-infrared vein segmentation method based on semi-supervised learning, characterized in that: include: S1: Acquire near-infrared limb imaging to produce a vein segmentation image, segment the vein portion from the original image, and perform preprocessing to construct a dataset, which includes a supervised part and an unsupervised part; S2: Constructing a semantic segmentation model, which includes a three-branch semi-supervised deep learning neural network model, namely, a recursive scaling recurrent neural network model, a reconstruction consistency neural network model, and a segmentation adversarial neural network model; S3: training the semantic segmentation model, training and extracting features from the recursive scaled recurrent neural network model based on the data set; and iteratively optimizing the recursive scaled recurrent neural network model through the reconstruction consistency neural network model and the segmentation adversarial neural network model; The training of the semantic segmentation model specifically includes: S301: Inputting the supervised part of the data set and the corresponding pre-processed images into the recursive scaling recurrent neural network model for training; S302: Inputting the unsupervised portion of the dataset and the corresponding preprocessed image into the recursive scaling recurrent neural network model trained in step S301 to obtain a segmentation result map and a feature representation of the encoded portion; S303: Inputting the segmentation result image and the feature representation of the encoding part into the reconstruction consistency neural network model, outputting the reconstruction result image, and optimizing the recursive scaling recurrent neural network model based on the reconstruction result image; S304: Input the supervised part and the unsupervised part of the data set into the segmentation adversarial neural network model, and optimize the recursive scaling recurrent neural network model obtained in step S303 based on the output results.

2. The fully automatic near-infrared vein segmentation method according to claim 1, characterized in that: The dataset construction process is as follows: S101: calibrating a group of near-infrared camera devices using a baseline distance, and acquiring a series of near-infrared limb vein images through the near-infrared camera devices; S102: performing enhancement preprocessing on the near-infrared limb vein image, and annotating the processed image to obtain an annotated image; S103: performing binary segmentation on the annotated part and the remaining parts in the annotated image to construct a segmentation data set; S104: Using the labeled images and the unlabeled images in the segmentation dataset as datasets of the supervised part and the unsupervised part of the segmentation adversarial model, respectively.

3. The fully automatic near-infrared vein segmentation method according to claim 2, characterized in that: The binary segmentation is performed by converting the spline curve after adjusting the thickness of each segment in the annotated image into a binary mask in pixel form, thereby achieving segmentation of the vein portion and the remaining portions.

4. The fully automatic near-infrared vein segmentation method according to claim 1, characterized in that: The recursive scaling recurrent neural network model includes three levels, and the network input resolution of each level increases successively.

5. The fully automatic near-infrared vein segmentation method according to claim 4, characterized in that: The recursive scaling recurrent neural network model includes a convolutional layer, a residual layer, a replication layer and a deconvolution layer; There are four convolutional layers and three deconvolutional layers. Each convolutional layer and deconvolutional layer is followed by three residual layers. Each residual layer also contains two consecutive convolutional layers. The parameters of the first consecutive convolutional layer are the same as the previous convolutional layer, and the second one uses linear activation. The reconstruction consistency neural network model structure is two Dense layers connected to two upsampling layers and a convolutional layer, followed by three residual layers and a convolutional layer; The segmentation adversarial neural network model consists of a dense layer and four convolutional layers.

6. The fully automatic near-infrared vein segmentation method according to claim 5, characterized in that: The encoding part of each layer of the recursive scaled recurrent neural network model is connected through a convolutional long short-term memory network.

Images