A method for automatically generating visible light images from medical ultrasound images

Abstract

This invention discloses a method for automatically generating visible light images from medical ultrasound images, comprising the following steps: 1. Strictly aligning each ultrasound image and visible light image, and resampling the visible light image to match the resolution and field of view of the ultrasound image; 2. Further correcting the intensity non-uniformity of each ultrasound image; 3. Standardizing the intensity of ultrasound images from different subjects; 4. Separating the uterine region from the background region in the visible light image; 5. Establishing a dataset; 6. Data augmentation, increasing the amount of training data; 7. Pre-training using the ImageNet dataset; 8. Loading the pre-trained model and adding an output layer; 9. Training the TransUNet network, evaluating the method's performance, inputting the ultrasound image into the trained TransUNet network, and outputting the corresponding visible light image. This invention can generate visible light images in real time and accurately from input ultrasound images.

Description

Technical Field

[0001] This invention relates to a method for automatically generating visible light images from medical ultrasound images, belonging to the field of deep learning methods for processing medical images. Background Technology

[0002] Image-based deep learning methods train deep neural network models to extract image features and then use these features to classify or regress images. Compared to traditional hand-crafted features, neural networks learn themselves by updating parameters through backpropagation using a designed loss function and optimization methods. Deep learning offers advantages such as strong learning capabilities, broad coverage, high adaptability, and good portability.

[0003] Visible light images offer high resolution and stronger interactions with organic molecules. The reflected light carries various information such as biochemical changes, cellular abnormalities, blood glucose concentration, and blood oxygen levels, aiding clinicians in disease diagnosis. However, visible light images cannot be obtained through external instruments and require surgery, which can be invasive and risky. Ultrasound images, on the other hand, use ultrasound beams to scan the body, receiving and processing reflected signals to obtain images of internal organs. Ultrasound equipment is portable and non-invasive, allowing for bedside diagnosis of patients with limited mobility. Furthermore, ultrasound emits no radiation, making it a preferred option for special patients. Therefore, we aim to overcome these difficulties by generating visible light images from ultrasound images. We introduced a deep learning approach, training the neural network parameters using ultrasound and visible light images of the subjects. This allows the neural network to learn an end-to-end mapping from ultrasound images to corresponding visible light images. Summary of the Invention

[0004] The purpose of this invention is to provide a method for automatically generating visible light images from medical ultrasound images. By preprocessing the ultrasound images and visible light images of the subjects, a dataset is obtained with ultrasound images as input data and visible light images as real labels. The dataset is then used to train a TransUNet network through a six-fold cross-validation method. The trained network can generate visible light images in real time with only ultrasound images as input, which not only reduces the workload of clinical practice but also avoids the harm to the human body caused by ionizing radiation when taking visible light images.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] A method for automatically generating visible light images from medical ultrasound images includes the following steps:

[0007] Step 1: Strictly align each ultrasound image and visible light image, and resample the visible light image to match the resolution and field of view of the ultrasound image;

[0008] Step 2: Further correct the intensity inhomogeneity of each ultrasound image;

[0009] Step 3: Perform histogram matching between all ultrasound images and randomly selected templates to standardize the ultrasound image intensity of different subjects;

[0010] Step 4: Automatically generate a uterine mask from each visible light image to separate the uterine region from the non-anatomical, background region of the visible light image;

[0011] Step 5: Use ultrasound images as input data and visible light images as real labels to build a dataset;

[0012] Step 6: Perform data augmentation on the dataset to increase the number of ultrasound images in the dataset;

[0013] Step 7: Pre-train the TransUNet network on the ImageNet dataset;

[0014] Step 8: Load the pre-trained model into the TransUNet network and add an output layer;

[0015] Step 9: Input the dataset into the TransUNet network for training and evaluate the method performance; input the ultrasound image into the TransUNet network and output the corresponding visible light image.

[0016] Furthermore, the method for strictly aligning each ultrasound image and visible light image in step 1 is the mutual information rigid registration algorithm.

[0017] Mutual information rigid registration algorithm is to find a mapping relationship T:X A →X B , where X A For the image to be registered, X B To register the target image, so that X A Every point on X B Each point on the image corresponds to a unique point, and both points are located in the same anatomical position. The rigid body transformation formula for 2D image registration is: Where x and y represent the pixel coordinates before registration, x ′ y ′ The coordinates of the target pixel are represented by p, q, and θ, which represent the horizontal displacement, vertical displacement, and clockwise rotation angle, respectively. The mapping relationship is as follows:

[0018] Furthermore, the method for correcting the intensity non-uniformity of each ultrasound image in step 2 is the N3 bias field correction algorithm.

[0019] Furthermore, the method for standardizing the ultrasound image intensity of different subjects in step 3 is the dynamic histogram distortion method.

[0020] Furthermore, the method for separating the uterine region from the non-anatomical, background region of the visible light image in step 4 is the Otsu automatic thresholding method.

[0021] Arbitrarily select a threshold t to divide the image into two parts (foreground and background). The proportion of foreground pixels in the image is W0, and the mean is U0. The proportion of background pixels in the image is W1, and the mean is U1. The mean of the entire image is U = W0 × U0 + W1 × U1. Establish the objective function.

[0022] g(t) = W0 × (U0 - U) 2 +W1×(U1―U) 2 ,

[0023] g(t) is the inter-class variance when the segmentation threshold is t. The Otsu algorithm finds the optimal threshold to be t when g(t) is maximized, which can distinguish the foreground and background to the greatest extent.

[0024] Furthermore, the data augmentation method in step 6 is random translation and random flipping.

[0025] Furthermore, in step 7, the TransUNet network weights are randomly initialized to a zero-mean Gaussian distribution and the bias parameters are initialized to all zeros. Then, the TransUNet network is pre-trained using the ImageNet dataset.

[0026] Furthermore, in step 8, the parameters of the trained model are loaded into the TransUNet network, the classification layer is removed, and an additional 1×1 convolutional layer is added at the end to map the feature vector of each channel of the previous layer to the visible light pixel value.

[0027] Furthermore, in step 9, the dataset is divided into 6 groups using a six-fold cross-validation method. Each time, one group is retained as the test set, and the remaining five groups are used as training data to train the TransUNet network. By inputting the ultrasound image into the trained TransUNet network, a real-time visible light image is obtained.

[0028] The beneficial effects of this invention are as follows: Considering that obtaining visible light images requires surgery, which poses significant harm and risks to the human body, this invention utilizes deep learning to generate visible light images from ultrasound images. Data preprocessing improves the quality of both ultrasound and visible light images, and data augmentation increases the dataset size, allowing for sufficient training of the TransUNet network with limited data. Loading pre-trained weights trained on the ImageNet dataset significantly reduces model training time, and the six-fold cross-validation method extracts as much effective information as possible from limited data while mitigating overfitting to some extent. Using the trained TransUNet model, we can obtain visible light images in real-time from ultrasound images alone, improving image resolution, obtaining numerous organ-related parameters, and assisting clinicians in better disease diagnosis, thus avoiding the harm to the human body caused by surgery when capturing visible light images. Attached Figure Description

[0029] Figure 1 This is a flowchart of a method for automatically generating visible light images from medical ultrasound images according to the present invention;

[0030] Figure 2 This is a diagram of the TransUNet network structure of the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0032] like Figure 1 As shown, a method for automatically generating a visible light image from a medical ultrasound image according to the present invention includes the following steps:

[0033] Step 1: Use the mutual information rigid registration algorithm to strictly align each ultrasound image and visible light image, and resample the visible light image to match the resolution and field of view of the ultrasound image. The mutual information rigid registration algorithm seeks a mapping relationship T:X. A →X B , where X A For the image to be registered, X B To register the target image, so that X A Every point on X B Each point on the image corresponds to a unique point, and both points are located in the same anatomical position. The rigid body transformation formula for 2D image registration is: Where x and y represent the pixel coordinates before registration, x ′ y ′ The coordinates of the target pixel are represented by p, q, and θ, which represent the horizontal displacement, vertical displacement, and clockwise rotation angle, respectively. The mapping relationship is as follows:

[0034] Step 2: Use the N4 bias field correction algorithm to further correct the intensity inhomogeneity of each ultrasound image.

[0035] Step 3: Perform histogram matching of all ultrasound images with a randomly selected template to help standardize image intensity across different subjects using dynamic histogram distortion. Dynamic histogram distortion directly affects the intensity histogram by expanding or compressing the intensity.

[0036] Step 4: To better evaluate the accuracy of the visible light images, a uterine mask is automatically generated from each visible light image to separate the uterine region from the non-anatomical background region. This is achieved by applying the Otsu automatic thresholding method to each visible light image. An arbitrary threshold t is selected to divide the image into two parts (foreground and background). The proportion of foreground pixels in the image is W0, with a mean of U0, and the proportion of background pixels in the image is W1, with a mean of U1. The overall mean of the image is U = W0 × U0 + W1 × U1. An objective function is then established.

[0037] g(t) = W0 × (U0 - U) 2 +W1×(U1―U) 2 ,

[0038] g(t) is the inter-class variance when the segmentation threshold is t. The Otsu algorithm finds the optimal threshold to be t when g(t) is maximized, which can distinguish the foreground and background to the greatest extent.

[0039] Step 5: Use uterine ultrasound images of multiple participants as input data and visible light images as real labels to build a dataset.

[0040] Step 6: Use random translation and random flipping methods to augment the dataset and increase the number of ultrasound images in the dataset.

[0041] Step 7: The TransUNet network weights are randomly initialized to a zero-mean Gaussian distribution, and the bias parameters are initialized to all zeros. Then, the TransUNet network is pre-trained using the ImageNet dataset. The ImageNet dataset consists of 1 million images divided into 1000 classes. Although the TransUNet network model is trained on a non-medical image dataset, the low- and mid-level features learned from the model are general and can help us achieve better performance with limited training data.

[0042] Step 8: Load the parameters of the trained model into the TransUNet network, remove the classification layer, and add an extra 1×1 convolutional layer at the end to map the feature vector of each channel of the previous layer to visible light pixel values. The parameters of the added output layer are initialized using a zero-mean Gaussian distribution. Through the output layer, we can transform the extracted feature map into the visible light image we need.

[0043] Step 9: The performance of the TransUNet-based visible light image automatic generation method was evaluated using a six-fold cross-validation method. Eighteen participants were randomly divided into six equal-sized groups. One group was kept as the test set, and the remaining five groups were used as training data to train the TransUNet model. Each ultrasound image has approximately 160 slices, so a group of 15 trainees provided 2400 training samples. The training data was divided into 48 mini-batches, requiring 50 iterations to traverse all training samples once. Inputting the ultrasound image into the trained TransUNet network resulted in a visible light image in real time.

[0044] like Figure 2 As shown, the implementation of the TransUNet network in the method of this invention includes the following steps:

[0045] Step 1: The TransUNet network receives the image. The encoding part receives an ultrasound image. If the image is single-channel, the repeat function is used to copy it twice, expanding the image into a three-channel image.

[0046] Step 2: Encode the ultrasound image into a high-level feature representation, learning to extract progressively more complex features from the input ultrasound image. This step is implemented using convolutional layers, each layer performing a 2D convolution on its input using a set of convolutional kernels, with the result passed through a non-linear activation function. Mathematically, the operation can be represented as follows:

[0047] h k =max(0,W k *X+b k ), k∈[0,K―1],

[0048] Among them, W k and b k These represent the weights and biases of the k-th convolutional kernel, respectively, with "*" indicating the convolution operation. The subscript k, k∈[0,K-1], represents the index of the k-th convolutional kernel in a set of K kernels, X represents the input, and h k This is the output of the k-th convolutional kernel. max(0,·) represents the Corrected Linear Unit (ReLU) nonlinear activation function. For the first convolutional layer, the input X is the ultrasound image itself; for subsequent layers, the input is the output feature map from the previous layer.

[0049] Step 3: Introduce a direct connection between the encoding and decoding parts (e.g., ... Figure 2 (As shown by the arrow in the diagram), so that high-resolution features from the encoding part can be used as additional input to the convolutional layers in the decoding part. This design makes it easier for the decoding part to generate high-resolution predictions and also makes the model more flexible. This step is achieved by creating a feature list, saving the output feature map of each convolutional layer into the feature list.

[0050] Step 4: Serialize the feature maps and map them to a latent D-dimensional embedding space, preserving positional information to encode global spatial information. Feature map serialization is achieved through convolution operations. The formula for mapping to the embedding space and adding positional information is as follows:

[0051]

[0052] in It is the embedding projection of the feature image, E pos This indicates positional embedding.

[0053] Step 5: Introduce a self-attention mechanism into the encoding part using the Transformer, enabling the network to extract global spatial information and increase the receptive field. The Transformer encoder consists of an L-layer multi-head self-attention (MSA) layer and a multilayer perceptron (MLP) block. The formula for calculating self-attention is:

[0054]

[0055] Where Q, K, V are the calculated vectors, and d k Let L be the dimension of the embedding space. The output of the Lth layer can be written as follows:

[0056] z′ l =MSA(LN(z) l―1 ))+z l―1 ,

[0057] z l =MLP(LN(z′) l ))+z′ l ,

[0058] Where LN(·) denotes the layer normalization operator, z′ l This represents the multi-head self-attention output, z l This represents the output of the Transformer encoder. The Transformer encoder uses a total of 12 layers to encode the image.

[0059] Step 6: Feature transformation, upsampling the feature map and combining it with high-resolution features to gradually achieve visible light image prediction from low resolution to high resolution. This step doubles the size of the feature map through bilinear upsampling, concatenates it with the high-resolution output of the encoding part's convolutional layer in the channel dimension, and finally maps the feature map to a low-dimensional space for fusion through a convolutional layer.

[0060] Step 7: Output visible light image. This step involves performing a 1×1 convolution operation on the feature map output from the decoding section. This maps the 64 feature vectors from the previous layer to visible light pixel values ​​and restores the feature map to the size of the input ultrasound image.

[0061] Step 8: The complete TransUNet network can be viewed as representing a complex end-to-end mapping function that transforms an input ultrasound image into its corresponding visible light image. Learning the end-to-end mapping function requires estimating the network parameters θ = {W1, b1, W2, b2, ...}, which is achieved by minimizing the loss or prediction error between the predicted image F(x; θ) and the corresponding ground-based real visible light image Y. Given a set of ultrasound images {X... i} and their corresponding visible light images {Y i We use the mean absolute error (MAE) as the loss function:

[0062]

[0063] Where N is the number of training images.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatically generating a visible light image from a medical ultrasound image, characterized by: The method comprises the following steps: Step 1: Strictly align each ultrasound image and visible light image, and resample the visible light image to match the resolution and field of view of the ultrasound image; Step 2: Further correct the intensity inhomogeneity of each ultrasound image; Step 3: Histogram matching is performed on all ultrasound images with a randomly selected template to standardize the ultrasound image intensity of different experimenters; Step 4: An automatic uterine mask is generated from each visible light image to separate the uterine region from the non-anatomical, background region of the visible light image; Step 5: The ultrasound image is used as input data, and the visible light image is used as a true label to establish a data set; Step 6: Data augmentation is performed on the data set to expand the number of ultrasound images in the data set; Step 7: The TransUNet network is pre-trained in the ImageNet data set; Step 8: The TransUNet network loads the pre-trained model and adds an output layer; Step 9: The data set is grouped and input into the TransUNet network for training, and the performance of the method is evaluated; the ultrasound image is input into the TransUNet network, and the corresponding visible light image is output; In step 7, the weights of the TransUNet network are randomly initialized to a zero-mean Gaussian distribution and the bias parameters are initialized to all zeros, and then the TransUNet network is pre-trained using the ImageNet data set; In step 9, the data set is divided into 6 groups using the six-fold cross-validation method, one group is reserved as the test set each time, and the remaining five groups are used as training data to train the TransUNet network; the ultrasound image is input into the trained TransUNet network to obtain a real-time visible light image.

2. The method for automatically generating a visible light image from medical ultrasound images according to claim 1, characterized in that: In step 1, the method for strictly aligning each ultrasound image and visible light image is mutual information rigid registration algorithm; Mutual information rigid registration algorithm is to find a mapping relationship T: , where is the image to be registered, is the registration target image, so that each point on has a unique point corresponding to it on , and both points are located at the same anatomical position; the rigid transformation formula for two-dimensional image registration is: , where represents the pixel coordinate position before registration, represents the target pixel coordinate position, , , horizontal displacement, vertical displacement and clockwise rotation angle respectively, and the mapping relationship is 3. The method of automatically generating a visible light image from a medical ultrasound image of claim 2, wherein: In step 2, the method for correcting the intensity inhomogeneity of each ultrasound image is N3 bias field correction algorithm.

4. The method for automatically generating a visible light image from medical ultrasound images according to claim 3, wherein: In step 3, the method for standardizing the ultrasound image intensity of different experimenters is dynamic histogram warping method.

5. The method for automatically generating a visible light image from medical ultrasound images according to claim 4, characterized in that: In step 4, the method for separating the uterine region from the non-anatomical, background region of the visible light image is Otsu automatic threshold method; An arbitrary threshold t is selected to divide the image into two parts: foreground and background, the proportion of foreground pixel points in the image is , the mean value is , the proportion of background pixel points in the image is , the mean value is , the mean value of the whole image is , and a target function is established ， That is, the between-class variance when the segmentation threshold is t; the Otsu algorithm makes The t corresponding to the maximum is the optimal threshold, which can distinguish the foreground and background to the greatest extent.

6. The method for automatically generating a visible light image from medical ultrasound images according to claim 5, wherein: In step 6, the data augmentation method is random translation and random flipping.

7. The method of automatically generating a visible light image from a medical ultrasound image of claim 6, wherein: The parameters of the trained model are loaded into the TransUNet network in step 8, the classification layer is removed and an additional convolutional layer is added at the end for mapping each channel feature vector of the previous layer to a visible light pixel value.

Images