Ultrasonic medical image enhancement method based on deep learning

Abstract

The invention belongs to the technical field of image processing, and particularly relates to an ultrasonic medical image enhancement method based on deep learning, which comprises the following steps: establishing an ultrasonic medical image enhancement model, training the ultrasonic medical image enhancement model, and performing image enhancement through the trained ultrasonic medical image enhancement model. According to the method, the visual effect of the ultrasonic image can be effectively improved, key details in the image are reserved, the enhancement effect of the ultrasonic medical image is improved, and the integrity of the contour and the clarity of the structure of the medical image are ensured; through end-to-end training and optimization, the optimal strategy for enhancing the visual effect from the ultrasonic medical image can be automatically learned, manual intervention is not needed, and the convenience and practicability of operation are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to an ultrasonic medical image enhancement method based on deep learning. Background Art

[0002] Image enhancement is a key technology in the field of image processing, which aims to improve the visual quality of images or optimize their feature performance to meet the needs of subsequent processing. Ultrasound medical images are widely used in clinical diagnosis, but due to factors such as equipment limitations, operator skills, and motion artifacts, the quality of the generated ultrasound images is usually low, affecting the accuracy of diagnosis. Unlike ordinary RGB images, ultrasound images are mainly grayscale images, which contain more limited information and have more noise and artifacts, so their enhancement processing faces more challenges.

[0003] In order to solve the problem of medical image enhancement, many literatures have proposed various methods. These methods can be basically divided into two categories: traditional (non-learning-based) methods and learning-based methods. In general, non-learning-based methods can be divided into spatial domain and frequency domain methods according to the different spaces where the enhancement process is located. Most of the traditional methods used for medical image enhancement are based on the spatial domain, the most common of which are histogram equalization methods and some filtering-based methods. Traditional methods have obvious defects. Traditional methods usually require manual adjustment of parameters and lack automatic processing capabilities. Under different imaging conditions and different patients, the robustness and consistency of traditional methods are insufficient. In addition, there are lesion areas with low contrast in medical images. Traditional enhancement methods are difficult to improve the contrast of these areas alone without affecting other parts. More importantly, the structures in medical images are often complex and contain many details and tiny lesions. Traditional enhancement methods are often unable to handle these complex structures and it is difficult to ensure both details and overall image quality at the same time.

[0004] Therefore, in order to overcome the defects of the above-mentioned traditional methods, people have been pursuing a fully automatic and reliable medical image enhancement technology. Until recently, deep learning technology has been introduced into the field of image enhancement. Compared with the problem that traditional methods require a lot of time to adjust parameters, deep learning technology automatically learns important features in images through multi-layer neural networks, can adapt to images of different types and complexities, greatly improves the efficiency of image enhancement, and has achieved excellent results in improving image quality. Although the introduction of deep learning technology has brought great improvements to the field of image enhancement, there are still the following shortcomings for the enhancement of medical images. (1) Most of the existing research is aimed at non-medical images and does not take into account the characteristics of medical images. (2) In the process of neural network training, the existing loss function does not fully constrain the medical image enhancement process. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an ultrasonic medical image enhancement method based on deep learning, comprising:

[0006] Establishing an ultrasonic medical image enhancement model, training the ultrasonic medical image enhancement model, and performing image enhancement using the trained ultrasonic medical image enhancement model;

[0007] The model includes two branches, the upper branch path is composed of a preliminary feature extraction module, three deep feature extraction modules, a multi-scale information fusion module and an ARAM module; the lower branch is composed of a high-frequency component processing module;

[0008] The training of the ultrasonic medical image enhancement model comprises:

[0009] S1: Ultrasonic medical images are processed sequentially by the modules of the upper branch;

[0010] The preliminary feature extraction module obtains basic features and provides initial information input for subsequent modules;

[0011] The three deep feature extraction modules further refine features based on the basic features to obtain features of different scales to enhance the expression of complex structures and details;

[0012] The multi-scale information fusion module integrates features of different scales to ensure that the optimized image has both clear details and complete structure;

[0013] Combining regional channel attention and spatial dimension adaptive aggregation, we design an adaptive regionalized attention mechanism ARAM to gradually filter and enhance important information.

[0014] S2: In the lower branch, the ultrasonic medical image passes through the high-frequency component processing module to process the high-frequency components of the medical image, so that the enhanced image is more complete in contour and clearer in structure;

[0015] S3: The outputs of the upper and lower branches are fused and reconstructed through a convolutional layer to obtain an enhanced image;

[0016] S4: Design texture gradient loss, which designs the loss function by calculating the gradient difference between the enhanced image and the target image in a specific feature space, so as to maintain the clarity and coherence of image texture and edge features during the optimization process;

[0017] S5: Multi-level feature mapping is introduced. Through a convolutional network model with hierarchical feature extraction capabilities, the multi-level structural features of the image are extracted and analyzed, so as to calculate the perceptual loss in the multi-layer feature space.

[0018] Beneficial effects of the present invention:

[0019] The present invention proposes a lightweight image enhancement model, which is particularly suitable for portable medical equipment scenarios. The model extracts feature information of medical images through multi-scale convolution and attention mechanism, and introduces a high-frequency component processing module to process the high-frequency components of the image to retain the integrity of the contour and the clarity of the structure. A special deep feature extraction module is designed to extract deeper abstract features and enhance the robustness and adaptability of the network. In addition, texture gradient loss and perceptual loss are designed according to the characteristics of medical images to ensure the consistency and accuracy of texture during the image generation process, and also make the enhanced image closer to the real image in terms of semantic content and visual perception.

[0020] The present invention can effectively improve the visual effect of ultrasound images and retain key details in the images, which not only improves the effect of ultrasound medical image enhancement, but also ensures the integrity of the medical image contour and the clarity of the structure. Through end-to-end training and optimization, the present invention can automatically learn the best strategy for enhancing the visual effect from ultrasound medical images without manual intervention, thus improving the convenience and practicality of operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the overall framework of the ultrasonic medical image enhancement model of the present invention.

[0022] Figure 2 Schematic diagram of the deep feature extraction module of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] A method for ultrasound medical image enhancement based on deep learning, such as Figure 1 As shown, including:

[0025] Establishing an ultrasonic medical image enhancement model, training the ultrasonic medical image enhancement model, and performing image enhancement using the trained ultrasonic medical image enhancement model;

[0026] The model includes two branches, the upper branch path is composed of a preliminary feature extraction module, three deep feature extraction modules, a multi-scale information fusion module and an ARAM module; the lower branch is composed of a high-frequency component processing module;

[0027] The training of the ultrasonic medical image enhancement model comprises:

[0028] S1: Ultrasonic medical images are processed sequentially by the modules of the upper branch;

[0029] The preliminary feature extraction module obtains basic features and provides initial information input for subsequent modules;

[0030] The three deep feature extraction modules further refine features based on the basic features to obtain features of different scales to enhance the expression of complex structures and details;

[0031] The multi-scale information fusion module integrates features of different scales to ensure that the optimized image has both clear details and complete structure;

[0032] Combining regional channel attention and spatial dimension adaptive aggregation, we design an adaptive regionalized attention mechanism ARAM to gradually filter and enhance important information.

[0033] S2: In the lower branch, the ultrasonic medical image passes through the high-frequency component processing module to process the high-frequency components of the medical image, so that the enhanced image is more complete in contour and clearer in structure;

[0034] S3: The outputs of the upper and lower branches are fused and reconstructed through a convolutional layer to obtain an enhanced image;

[0035] S4: Design texture gradient loss, which designs the loss function by calculating the gradient difference between the enhanced image and the target image in a specific feature space, so as to maintain the clarity and coherence of image texture and edge features during the optimization process;

[0036] S5: Multi-level feature mapping is introduced. Through a convolutional network model with hierarchical feature extraction capabilities, the multi-level structural features of the image are extracted and analyzed, so as to calculate the perceptual loss in the multi-layer feature space.

[0037] Step 1: The image to be processed is analyzed and changed through the preliminary feature extraction module to obtain the corresponding preliminary feature map. The preliminary feature extraction module consists of two convolutional layers. The first layer consists of 16 convolutional kernels of size 3×3 and stride 1, and the second layer consists of 32 convolutional kernels of size 3×3 and stride 1.

[0038] Step 1.1: Input the feature map obtained in step 1 into the deep feature extraction module, such as Figure 2As shown in the figure, a total of three deep feature extraction modules are required. Each deep feature extraction module has two upper and lower paths. The first convolution layer of the upper path consists of 32 convolution kernels of size 3×3 and stride 1, followed by the MaxPool2d pooling layer. The second convolution layer consists of 64 convolution kernels of size 3×3 and stride 1, followed by the MaxPool2d pooling layer. The third layer is first sampled by Up-sampling, followed by a convolution layer consisting of 64 convolution kernels of size 3×3 and stride 1. The fourth layer is also first sampled by Up-sampling, followed by a convolution layer consisting of 32 convolution kernels of size 3×3 and stride 1. The convolution layer can extract local features, while the pooling layer will reduce the size of the feature map, thereby retaining more important features. By gradually reducing the resolution, the model can focus on the core features of the image and capture higher-level semantic information. The structures of the upper and lower paths are exactly the same, and the convolution kernel sizes used are different. The upper path is 3×3 and the lower path is 5×5. Convolution kernels of different sizes correspond to different receptive fields. Smaller convolution kernels focus more on local details, while larger convolution kernels can capture global structural information. This multi-scale feature extraction method can help the model understand the image content more comprehensively, especially for tasks that require precise boundaries such as medical images. After each path passes through the first four layers, the extracted feature information is then passed through the ARAM attention mechanism, and finally the outputs of the two paths are added as the output of the deep feature extraction module.

[0039] Step 1.2: Concatenate the output obtained in step 1.1 and the output in step 1 and input them into the multi-scale information aggregation module. The multi-scale information aggregation module includes two convolutional layers. The first convolutional layer accepts the feature information of all previous modules and consists of 128 convolutional kernels with a size of 3×3 and a stride of 1. The second convolutional layer consists of 32 convolutional kernels with a size of 1×1 and a stride of 1, which is used to adjust the size of the feature map.

[0040] Step 2: Input the input of step 1.2 into the ARAM attention mechanism. The adaptive regionalized attention mechanism (ARAM) can combine regional channel attention and spatial dimension adaptive aggregation to gradually filter and strengthen important information, thereby improving the performance of the network.

[0041] Step 3: At the same time, the image to be processed is passed through the high-frequency component processing module from the second branch. The high-frequency components of the image are first retained by a high-pass filter, and then the high-frequency feature information of the image is extracted through a convolution layer composed of 32 convolution kernels with a size of 3×3 and a step size of 1.

[0042] Step 4: Design the texture gradient loss by calculating the gradient difference between the enhanced image and the real image to maintain clear texture features during the enhancement process. The texture gradient loss is expressed as:

[0043]

[0044] in, represents the generated image, I represents the real label image, and Sobel(.) represents the application of the Sobel operator to calculate the gradient of the image. The specific method is:

[0045] First, enhance the image Apply Sobel operator to the original image I and obtain their gradient maps;

[0046] Then calculate the difference in gradient between the two images;

[0047] Finally, the L1 norm is used to quantify the gradient difference as the loss value

[0048] Step 5: Introduce the pre-trained convolutional network model to extract the high-level feature representation of the image, and use the difference between the enhanced image and the target image in these feature spaces as the perceptual loss. The perceptual loss is expressed as:

[0049]

[0050] in, represents the jth layer of the pre-trained neural network, j∈{1,2…L}, L represents the number of network layers used to calculate the perceptual loss. The specific approach is:

[0051] First, select the pre-trained model and the middle layer of the model as the feature extraction network;

[0052] Then the original image and the generated image are passed through the feature extraction network respectively to extract their feature representations;

[0053] Next, calculate their differences in the high-level feature space;

[0054] Finally, the perceptual losses of different layers are weighted.

[0055] Furthermore, the network is trained and optimized through a combination of total loss functions, which include pixel-level loss, texture gradient loss, and perceptual loss. The total loss function is expressed as:

[0056] L all =λ1L pixel +λ2L ore +λ3L tg

[0057] Where L pixel is the basic loss function, i.e., the pixel-level loss of the image, L all Represents the total loss, λ1, λ2, λ3 represent the weights corresponding to each loss function.

[0058] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for ultrasonic medical image enhancement based on deep learning, characterized in that: include: Establishing an ultrasonic medical image enhancement model, training the ultrasonic medical image enhancement model, and performing image enhancement using the trained ultrasonic medical image enhancement model; The model includes two branches, the upper branch path is composed of a preliminary feature extraction module, three deep feature extraction modules, a multi-scale information fusion module and an ARAM module; the lower branch is composed of a high-frequency component processing module; The training of the ultrasonic medical image enhancement model comprises: S1: Ultrasonic medical images are processed sequentially by the modules of the upper branch; The preliminary feature extraction module obtains basic features and provides initial information input for subsequent modules; The three deep feature extraction modules further refine features based on the basic features to obtain features of different scales to enhance the expression of complex structures and details; The multi-scale information fusion module integrates features of different scales to ensure that the optimized image has both clear details and complete structure; Combining regional channel attention and spatial dimension adaptive aggregation, we design an adaptive regionalized attention mechanism ARAM to gradually filter and enhance important information. S2: In the lower branch, the ultrasonic medical image passes through the high-frequency component processing module to process the high-frequency components of the medical image, so that the enhanced image is more complete in contour and clearer in structure; S3: The outputs of the upper and lower branches are fused and reconstructed through a convolutional layer to obtain an enhanced image; S4: Design texture gradient loss, which designs the loss function by calculating the gradient difference between the enhanced image and the target image in a specific feature space, so as to maintain the clarity and coherence of image texture and edge features during the optimization process; S5: Multi-level feature mapping is introduced. Through a convolutional network model with hierarchical feature extraction capabilities, the multi-level structural features of the image are extracted and analyzed, so as to calculate the perceptual loss in the multi-layer feature space.

2. The method for ultrasonic medical image enhancement based on deep learning according to claim 1, characterized in that: The preliminary feature extraction module consists of two cascaded convolutional layers. By extracting and refining local features layer by layer, the first layer is used to identify basic edge and texture information, and the second layer further deepens the extraction of image details, thereby obtaining richer basic features.

3. The method for ultrasonic medical image enhancement based on deep learning according to claim 1, characterized in that: Each of the three deep feature extraction modules contains two parallel paths, the upper path is biased towards the refinement of local features, and the lower path uses a convolution kernel with a larger receptive field and focuses more on capturing global feature information; both the upper and lower paths are composed of four levels, and image features are gradually extracted and refined through different convolution sizes. The first two levels of each path contain feature extraction and compression operations to identify multi-level image details, and the last two levels further enhance feature expression through amplification operations; After the feature information in the upper and lower paths is processed by the adaptive regionalized attention mechanism ARAM, ARAM enhances and filters the feature information through regional channel attention and spatial dimension adaptive aggregation to generate richer contextual information and regionalized feature expressions. Finally, the outputs of the two paths are merged as the final output of the deep feature extraction module.

4. The method for ultrasonic medical image enhancement based on deep learning according to claim 1, characterized in that: The multi-scale information aggregation module includes two feature processing layers. The first layer further extracts multi-scale information from the input features to enhance the details and edge features of the image; the second layer is used to appropriately compress and adjust the dimensions of the features in order to optimize the expression effect of the feature map and provide a unified feature scale for subsequent processing.

5. The method for ultrasonic medical image enhancement based on deep learning according to claim 1, characterized in that: The adaptive regionalized attention mechanism ARAM combines regional channel attention and spatial dimension adaptive aggregation to gradually filter and strengthen important information, including: Regional channel attention: The image features are divided into several regions, and regional attention is applied in the channel dimension to identify the most discriminative channel features in each region. Regional channel attention extracts the global features of each region through average pooling and maximum pooling to obtain the importance of different regions in the channel. The pooled features are processed through multiple convolutional layers to generate the weight vector of each channel in each region; Adaptive aggregation in spatial dimension: In the spatial dimension, importance maps of different regions are generated through adaptive aggregation strategies: feature maps of different regions are calculated and aggregated with adaptive weights to obtain a two-dimensional feature map, which is then processed by convolution and nonlinear activation functions to generate a spatial attention map, making it more sensitive to the spatial information of different regions, thereby strengthening the feature expression of key areas.

6. The method for ultrasonic medical image enhancement based on deep learning according to claim 1, characterized in that: In the lower branch, the ultrasonic medical image passes through the high-frequency component processing module to process the high-frequency components of the medical image, making the enhanced image more complete in contour and clearer in structure, including: Perform Fourier transform on the input image to obtain the frequency domain image; create a Gaussian high-pass filter, apply the Gaussian high-pass filter to the frequency domain image, and perform inverse Fourier transform to convert the filtered frequency domain image back to the spatial domain; then pass it through two convolutional layers to extract the high-frequency information of the image.

7. The method for ultrasonic medical image enhancement based on deep learning according to claim 1, characterized in that: The texture gradient loss can enhance the edges and local details of the image and make the contour clearer. The texture gradient loss is expressed as: in, represents the texture gradient loss, Represents the generated image, I represents the real label image, Sobel(.) represents the application of the Sobel operator to calculate the gradient of the image, and || ||1 represents the L1 norm.

8. The method for ultrasonic medical image enhancement based on deep learning according to claim 1, characterized in that: The perceptual loss is a loss function based on feature space. The perceptual loss focuses on the similarity of images in high-level features rather than precise pixel-by-pixel matching. It uses a pre-trained convolutional neural network to extract high-level feature representations of images and measures the quality of images by comparing the differences between enhanced images and target images in feature space. The perceptual loss is expressed as: in, represents the perceived loss, represents the generated image, I represents the real label image, represents the jth layer of the pre-trained neural network, j∈{1,2…L}, L represents the number of network layers used to calculate the perceptual loss, Represents the square of the L2 norm.

Images