Method for automatically generating visible light image from medical ultrasonic image

Abstract

The invention discloses a method for automatically generating a visible light image from a medical ultrasonic image, which comprises the following steps of: 1, strictly aligning each ultrasonic image with the visible light image, and resampling the visible light image to match the resolution ratio and the visual field of the ultrasonic image; 2, further correcting the intensity nonuniformity of each ultrasonic image; 3, standardizing the ultrasonic image intensities of different experimenters; 4, separating the uterus area of the visible light image from the background area; 5, establishing a data set; 6, data enhancement: increasing the number of training data; 7, carrying out pre-training through the ImageNet data set; 8, loading the pre-training model, and adding an output layer; and 9, training the TransUNet network, evaluating the performance of the method, inputting the ultrasonic image into the trained TransUNet network, and outputting a corresponding visible light image. The visible light image can be accurately generated in real time by inputting the ultrasonic image.

Description

Technical Field

[0001] The present invention relates to a method for automatically generating a visible light image from a medical ultrasound image, and belongs to the technical field of processing medical images using deep learning methods. Background Art

[0002] Image-based deep learning methods extract image features by training deep neural network models, and use the extracted features to classify or regress images. Compared with traditional manual features, neural networks can self-learn by updating parameters through back propagation through designed loss functions and optimization methods. Deep learning has the advantages of strong learning ability, wide coverage, strong adaptability and good portability.

[0003] Visible light images have high resolution and stronger interactions with organic molecules. The reflected light carries various information such as biochemical changes in the body, cell abnormalities, blood sugar concentration, blood oxygen content, etc., which can help clinicians better diagnose diseases. However, visible light images cannot be obtained through external instruments and require surgery to obtain, which will cause great damage to the human body and pose certain risks. Ultrasound images use ultrasound beams to scan the human body and obtain images of internal organs by receiving and processing reflected signals. Ultrasound equipment is easy to move and non-invasive. It can be used at the bedside for patients with limited mobility. Ultrasound does not radiate the human body and can be used preferentially for special patients. Therefore, we hope to overcome these difficulties by generating visible light images through ultrasound images. So we introduced a deep learning method to train the parameters of the neural network through the experimenter's ultrasound images and visible light images, so that the neural network can learn the end-to-end mapping from ultrasound images to corresponding visible light images. Summary of the invention

[0004] The purpose of the present invention is to provide a method for automatically generating visible light images from medical ultrasound images. The method preprocesses the ultrasound image and visible light image of the experimenter to obtain a data set with the ultrasound image as input data and the visible light image as the real label. The data set is grouped and trained with the TransUNet network through a six-fold cross-validation method. The trained network can generate visible light images in real time by only inputting ultrasound images. This not only reduces the clinical workload, but also avoids the harm of ionizing radiation to the human body when taking visible light images.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for automatically generating a visible light image from a medical ultrasound image comprises the following steps:

[0007] Step 1: Each ultrasound image and visible light image are strictly aligned, and the visible light image is resampled to match the resolution and field of view of the ultrasound image;

[0008] Step 2: further correct the intensity inhomogeneity of each ultrasound image;

[0009] Step 3: Perform histogram matching on all ultrasound images with randomly selected templates to standardize the intensity of ultrasound images from different subjects;

[0010] Step 4: Automatically generate a uterine mask from each visible light image to separate the uterine region from the non-anatomical, background region of the visible light image;

[0011] Step 5: Use ultrasound images as input data and visible light images as true labels to establish a dataset;

[0012] Step 6: Perform data enhancement on the dataset and expand the number of ultrasound images in the dataset;

[0013] Step 7: Pre-train the TransUNet network on the ImageNet dataset;

[0014] Step 8: TransUNet network loads the pre-trained model and adds the output layer;

[0015] Step 9: Group the data set into the TransUNet network for training and evaluate the performance of the method; input the ultrasound image into the TransUNet network and output the corresponding visible light image.

[0016] Furthermore, the method for strictly aligning each ultrasound image and visible light image in step 1 is a mutual information rigid registration algorithm.

[0017] The mutual information rigid registration algorithm is to find a mapping relationship T:X A →X B , where X A is the image to be registered, X B To register the target image, make X A Every point on X B There is a unique point on the image that corresponds to it, and these two points are located at the same anatomical position. The rigid body transformation formula for two-dimensional image registration is: Where x and y represent the pixel coordinates before registration. ′ ,y ′ Indicates the target pixel coordinate position, p, q, θ are the horizontal displacement, vertical displacement and clockwise rotation angle respectively, the mapping relationship

[0018] Furthermore, the method for correcting the intensity non-uniformity of each ultrasound image in step 2 is an N3 bias field correction algorithm.

[0019] Furthermore, the method for standardizing the ultrasound image intensities of different experimenters in step 3 is a dynamic histogram distortion method.

[0020] Furthermore, the method for separating the uterus region from the non-anatomical and background regions of the visible light image in step 4 is the Otsu automatic threshold method.

[0021] Arbitrarily select a threshold t to divide the image into two parts (foreground and background). The proportion of foreground pixels in the image is W0, and the mean is U0. The proportion of background pixels in the image is W1, and the mean is U1. The mean of the entire image is U=W0×U0+W1×U1. Establish the objective function

[0022] g(t)=W0×(U0―U) 2 +W1×(U1―U) 2 ,

[0023] g(t) is the inter-class variance when the segmentation threshold is t. The Otsu algorithm makes the t corresponding to the maximum g(t) the optimal threshold, which can distinguish the foreground and background to the greatest extent.

[0024] Furthermore, the data enhancement method in step 6 is random translation and random flipping.

[0025] Furthermore, in step 7, the TransUNet network weights are randomly initialized to zero-mean Gaussian distribution and the bias parameters are initialized to all zeros, and then the TransUNet network is pre-trained using the ImageNet dataset.

[0026] Furthermore, in step 8, the parameters of the training model are loaded into the TransUNet network, the classification layer is removed and an additional 1×1 convolutional layer is added at the end to map each channel feature vector of the previous layer to a visible light pixel value.

[0027] Furthermore, in step 9, the data set is divided into 6 groups using the six-fold cross-check validation method, one group is retained as a test set each time, and the remaining five groups are used as training data to train the TransUNet network. When the ultrasound image is input into the trained TransUNet network, a real-time visible light image will be obtained.

[0028] The beneficial effects of the present invention are as follows: considering that the acquisition of visible light images requires surgery, which causes great damage to the human body and involves certain risks, a deep learning method is used to generate visible light images through ultrasound images. The quality of ultrasound images and visible light images is improved through data preprocessing, and the number of data sets is increased through data enhancement, so that the TransUNet network can be fully trained with a limited data set. The time for model training can be greatly reduced by loading pre-trained weights trained on the ImageNet data set. The six-fold cross-check verification method can obtain as much effective information as possible from limited data and reduce overfitting to a certain extent. We can use the trained TransUNet model to obtain visible light images in real time when only ultrasound images are available, improve the resolution of the images, obtain many organ-related parameters, help clinicians better diagnose diseases, and avoid the harm of surgery to the human body when taking visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of a method for automatically generating visible light images from medical ultrasound images according to the present invention;

[0030] Figure 2 It is the network structure diagram of TransUNet of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0032] like Figure 1 As shown, a method for automatically generating a visible light image from a medical ultrasound image of the present invention comprises the following steps:

[0033] Step 1: Use the mutual information rigid registration algorithm to strictly align each ultrasound image and visible light image, and resample the visible light image to match the resolution and field of view of the ultrasound image. The mutual information rigid registration algorithm is to find a mapping relationship T:X A →X B , where X A is the image to be registered, X B To register the target image, make X A Every point on X B There is a unique point on the image that corresponds to it, and these two points are located at the same anatomical position. The rigid body transformation formula for two-dimensional image registration is: Where x and y represent the pixel coordinates before registration. ′ ,y ′ Indicates the target pixel coordinate position, p, q, θ are the horizontal displacement, vertical displacement and clockwise rotation angle respectively, the mapping relationship

[0034] Step 2: Use the N4 bias field correction algorithm to further correct the intensity inhomogeneity of each ultrasound image.

[0035] Step 3: All ultrasound images were histogram matched to randomly selected templates to help standardize image intensities across experimenters using a dynamic histogram warping method that acts directly on the intensity histogram by stretching or compressing the intensity.

[0036] Step 4: In order to better evaluate the accuracy of the visible light image, a uterine mask is automatically generated from each visible light image to separate the uterine area from the non-anatomical background area of ​​the visible light image. This is achieved by applying the Otsu automatic thresholding method to each visible light image. An arbitrary threshold t is selected to divide the image into two parts (foreground and background). The proportion of foreground pixels in the image is W0 and the mean is U0. The proportion of background pixels in the image is W1 and the mean is U1. The mean of the entire image is U=W0×U0+W1×U1. The objective function is established.

[0037] g(t)=W0×(U0―U) 2 +W1×(U1―U) 2 ,

[0038] g(t) is the inter-class variance when the segmentation threshold is t. The Otsu algorithm makes the t corresponding to the maximum g(t) the optimal threshold, which can distinguish the foreground and background to the greatest extent.

[0039] Step 5: Use the uterine ultrasound images of multiple subjects as input data and the visible light images as true labels to establish a dataset.

[0040] Step 6: Use random translation and random flipping methods to enhance the data set and expand the number of ultrasound images in the data set.

[0041] Step 7: The TransUNet network weights are randomly initialized to zero-mean Gaussian distribution and the bias parameters are initialized to all zeros, and then the TransUNet network is pre-trained using the ImageNet dataset. The ImageNet dataset consists of 1 million images divided into 1,000 categories. Although the TransUNet network model is trained on a non-medical image dataset, the low-level and mid-level features learned from the model are universal and can help us achieve better performance with limited training data.

[0042] Step 8: Load the parameters of the trained model into the TransUNet network, remove the classification layer and add an additional 1×1 convolutional layer at the end to map each channel feature vector of the previous layer to visible light pixel values. The added output layer parameters are initialized with a zero-mean Gaussian distribution, and through the output layer we can transform the extracted feature map into the visible light image we need.

[0043] Step 9: Use the six-fold cross-validation method to evaluate the performance of the automatic generation method of visible light images based on TransUNet. The 18 experimenters were randomly divided into 6 groups of equal size. Each group was retained as a test set, and the remaining five groups were used as training data to train the TransUNet model. Each ultrasound image has approximately 160 slices, so a group of 15 training subjects can provide 2400 training samples. The training data is divided into 48 small batches, and 50 iterations are required to traverse all training samples once. When the ultrasound image is input into the trained TransUNet network, a visible light image will be obtained in real time.

[0044] like Figure 2 As shown, the implementation of the TransUNet network in the method of the present invention includes the following steps:

[0045] Step 1: TransUNet receives an image. The encoding part receives an ultrasound image. If the image is single-channel, it is copied twice through the repeat function to expand the image to three-channel.

[0046] Step 2: Encode the ultrasound image into a high-level feature representation and learn to extract increasingly complex features from the input ultrasound image. This step is implemented using convolutional layers, each of which performs a 2D convolution on its input with a set of convolution kernels, and the result is passed through a nonlinear activation function. Mathematically, the operation can be expressed as follows:

[0047] h k =max(0,W k *X+b k ),k∈[0,K―1],

[0048] Among them, W k and b k The subscript k, k∈[0,K―1], represents the index of the kth convolution kernel in the set of K convolution kernels, X represents the input, and h represents the weight and bias of the kth convolution kernel. k is the output of the kth convolution kernel. max(0,·) represents the nonlinear activation function rectified linear unit (ReLU). For the first convolutional layer, the input X is the ultrasound image itself, and for subsequent layers, the input is the output feature map from the previous layer.

[0049] Step 3: Introduce a direct connection between the encoding part and the decoding part (such as Figure 2 ), so that the high-resolution features of the encoding part can be used as additional inputs to the convolutional layers of the decoding part. This design makes it easier for the decoding part to generate high-resolution predictions and also makes the model more flexible. This step is achieved by creating a feature list and saving the output feature maps of each convolutional layer into the feature list.

[0050] Step 4: Serialize and map the feature map to the potential D-dimensional embedding space, retaining the position information to encode the global spatial information. Feature map serialization is achieved through convolution operations, and the formula for mapping to the embedding space and adding position information is as follows:

[0051]

[0052] in is the embedding projection of the feature image, E pos Represents positional embedding.

[0053] Step 5: Use Transformer to introduce the self-attention mechanism into the encoding part, so that the network can extract global spatial information and increase the receptive field. The Transformer encoder consists of L layers of multi-head self-attention (MSA) and multi-layer perceptron (MLP) blocks. The calculation formula for self-attention is:

[0054]

[0055] Where Q, K, V are the calculated vectors, d k is the dimension of the embedding space. The output of the Lth layer can be written as follows:

[0056] z′ l =MSA(LN(z l―1 ))+z l―1 ,

[0057] z l =MLP(LN(z′ l ))+z′ l ,

[0058] where LN(·) represents the layer normalization operator, z′ l represents the multi-head self-attention output, z l Represents the output of the Transformer encoder. The Transformer encoder has 12 layers stacked to encode the image.

[0059] Step 6: Convert features, upsample feature maps and combine them with high-resolution features to gradually achieve visible light image prediction from low resolution to high resolution. This step doubles the size of the feature map through bilinear upsampling, concatenates it with the high-resolution output of the convolutional layer of the encoding part in the channel dimension, and finally maps the feature map to a low-dimensional space through the convolutional layer for fusion.

[0060] Step 7: Output the visible light image. This step performs a 1×1 convolution operation on the feature map output by the decoding part to map the 64 feature vectors of the previous layer to visible light pixel values ​​and restore the feature map to the input ultrasound image size.

[0061] Step 8: The complete TransUNet network can be viewed as representing a complex end-to-end mapping function that converts an input ultrasound image into its corresponding visible light image. Learning the end-to-end mapping function requires estimating the network parameters θ = {W1, b1, W2, b2, …}, which is achieved by minimizing the loss or prediction error between the predicted image F(x; θ) and the corresponding ground truth visible light image Y. Given a set of ultrasound images {X i} and their corresponding visible light images {Y i}, we use the mean absolute error (MAE) as the loss function:

[0062]

[0063] Where N is the number of training images.

[0064] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for automatically generating a visible light image from a medical ultrasound image, characterized in that: The following steps are involved: Step 1: Each ultrasound image and visible light image are strictly aligned, and the visible light image is resampled to match the resolution and field of view of the ultrasound image; Step 2: further correct the intensity inhomogeneity of each ultrasound image; Step 3: Perform histogram matching on all ultrasound images with randomly selected templates to standardize the intensity of ultrasound images from different subjects; Step 4: Automatically generate a uterine mask from each visible light image to separate the uterine region from the non-anatomical, background region of the visible light image; Step 5: Use ultrasound images as input data and visible light images as true labels to establish a dataset; Step 6: Perform data enhancement on the dataset and expand the number of ultrasound images in the dataset; Step 7: Pre-train the TransUNet network on the ImageNet dataset; Step 8: TransUNet network loads the pre-trained model and adds the output layer; Step 9: Group the data set into the TransUNet network for training and evaluate the performance of the method; input the ultrasound image into the TransUNet network and output the corresponding visible light image.

2. The method for automatically generating visible light images from medical ultrasound images according to claim 1, characterized in that: The method for strictly aligning each ultrasound image and visible light image in step 1 is a mutual information rigid registration algorithm; The mutual information rigid registration algorithm is to find a mapping relationship T:X A →X B , where X A is the image to be registered, X B To register the target image, make X A Every point on X B There is a unique point on the image that corresponds to it, and these two points are located at the same anatomical position; the rigid body transformation formula for two-dimensional image registration is: Where x and y represent the pixel coordinates before registration, x′ and y′ represent the target pixel coordinates, p, q, and θ represent the horizontal displacement, vertical displacement, and clockwise rotation angle, respectively.

3. The method for automatically generating visible light images from medical ultrasound images according to claim 2, characterized in that: The method for correcting the intensity non-uniformity of each ultrasound image in step 2 is the N3 bias field correction algorithm.

4. The method for automatically generating visible light images from medical ultrasound images according to claim 3, characterized in that: The method for standardizing the ultrasound image intensity of different experimenters in step 3 is the dynamic histogram distortion method.

5. The method for automatically generating visible light images from medical ultrasound images according to claim 4, characterized in that: The method for separating the uterus region from the non-anatomical and background regions of the visible light image in step 4 is the Otsu automatic threshold method; Arbitrarily select a threshold t to divide the image into two parts: foreground and background. The proportion of foreground pixels in the image is W0, and the mean is U0. The proportion of background pixels in the image is W1, and the mean is U1. The mean of the entire image is U=W0×U0+W1×U1. Establish the objective function g(t)=W0×(U0-U) 2 +W1×(U1-U) 2 , g(t) is the inter-class variance when the segmentation threshold is t; the Otsu algorithm makes the t corresponding to the maximum g(t) the optimal threshold, which can distinguish the foreground and background to the greatest extent.

6. The method for automatically generating visible light images from medical ultrasound images according to claim 5, characterized in that: The data enhancement method in step 6 is random translation and random flipping.

7. The method for automatically generating visible light images from medical ultrasound images according to claim 6, characterized in that: In step 7, the TransUNet network weights are randomly initialized to zero-mean Gaussian distribution and the bias parameters are initialized to all zeros, and then the TransUNet network is pre-trained using the ImageNet dataset.

8. The method for automatically generating visible light images from medical ultrasound images according to claim 7, characterized in that: In step 8, the parameters of the training model are loaded into the TransUNet network, the classification layer is removed and an additional 1×1 convolutional layer is added at the end to map each channel feature vector of the previous layer to a visible light pixel value.

9. The method for automatically generating visible light images from medical ultrasound images according to claim 8, characterized in that: In step 9, the data set is divided into 6 groups using the six-fold cross-check validation method, one group is retained as a test set each time, and the remaining five groups are used as training data to train the TransUNet network. When the ultrasound image is input into the trained TransUNet network, a real-time visible light image will be obtained.

Images