An ultrasonic medical image enhancement method based on deep learning

Abstract

The application belongs to the technical field of image processing, and particularly relates to an ultrasonic medical image enhancement method based on deep learning, which comprises the following steps: establishing an ultrasonic medical image enhancement model, training the ultrasonic medical image enhancement model, and performing image enhancement through the trained ultrasonic medical image enhancement model; the application can effectively improve the visual effect of the ultrasonic image, retain the key details in the image, improve the ultrasonic medical image enhancement effect, and ensure the integrity of the medical image contour and the clarity of the structure; through end-to-end training and optimization, the application can automatically learn the best strategy for enhancing the visual effect from the ultrasonic medical image, without manual intervention, and the operation convenience and practicality are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an ultrasonic medical image enhancement method based on deep learning. BACKGROUND

[0002] Image enhancement is a key technology in the field of image processing, which aims to improve the visual quality of images or optimize the feature representation to meet the needs of subsequent processing. Ultrasonic medical images are widely used in clinical diagnosis, but due to equipment limitations, operator skills, and motion artifacts, the quality of the generated ultrasonic images is usually low, affecting the accuracy of diagnosis. Unlike ordinary RGB images, ultrasonic images are mainly grayscale images, containing more limited information and more noise and artifacts, so the enhancement processing faces more challenges.

[0003] In order to solve the problem of medical image enhancement, many literatures have successively proposed various methods. These methods can be basically divided into two categories: traditional (non-learning-based) methods and learning-based methods. Generally, non-learning-based methods can be divided into space-based and frequency-based methods according to the space where the enhancement processing process is located. Most of the traditional methods for medical image enhancement are based on spatial domain, the most common one is the histogram equalization method, and some filtering-based methods, etc. The traditional method has obvious defects, and the traditional method usually needs to adjust the parameters manually, and lacks the ability of automatic processing. In different imaging conditions and different patient conditions, the robustness and consistency of the traditional method are insufficient, in addition, there are low-contrast lesion regions in medical images, and the traditional enhancement method is difficult to improve the contrast of these regions without affecting other parts. More importantly, the structure in the medical image is often complex, containing many details and small lesions, and the traditional enhancement method often falls short when dealing with these complex structures, and it is difficult to ensure the quality of details and the overall image.

[0004] Therefore, in order to overcome the defects of the above-mentioned traditional method, people have been pursuing a fully automatic and reliable medical image enhancement technology, and until recently, deep learning technology has been introduced into the field of image enhancement. Compared with the problem of consuming a lot of time to adjust parameters in the traditional method, the deep learning technology automatically learns the important features in the image through a multi-layer neural network, can adapt to different types and complexity of images, greatly improves the efficiency of image enhancement, and also achieves excellent results in the improvement of image quality. Although the introduction of deep learning technology has brought great improvement to the field of image enhancement, there are still the following deficiencies in the enhancement of medical images.(1) Most of the existing researches are for non-medical images, and the characteristics of medical images are not considered.(2) In the process of neural network training, the existing loss function is not comprehensive enough in the constraint of the medical image enhancement process. SUMMARY

[0005] To solve the above technical problems, the application provides an ultrasonic medical image enhancement method based on deep learning, comprising:

[0006] An ultrasonic medical image enhancement model is established, the ultrasonic medical image enhancement model is trained, and image enhancement is performed through the trained ultrasonic medical image enhancement model;

[0007] The model comprises an upper branch and a lower branch, the upper branch path comprises a preliminary feature extraction module, three deep feature extraction modules, a multi-scale information fusion module and an ARAM module; and the lower branch comprises a high-frequency component processing module;

[0008] The ultrasonic medical image enhancement model is trained, comprising:

[0009] S1: The ultrasonic medical image is processed by the modules of the upper branch in turn;

[0010] The preliminary feature extraction module obtains basic features and provides initial information input for subsequent modules;

[0011] The three deep feature extraction modules further refine the features on the basis of the basic features to obtain features of different scales, so as to enhance the expression of complex structures and details;

[0012] The multi-scale information fusion module integrates the features of different scales to ensure that the optimized image has clear details and complete structures;

[0013] The adaptive regional attention mechanism ARAM is designed by combining regional channel attention and spatial dimension adaptive aggregation, and important information is gradually filtered and strengthened;

[0014] S2: In the lower branch, the ultrasonic medical image is processed by the high-frequency component processing module, the high-frequency components of the medical image are processed, the enhanced image is more complete in outline and more clear in structure;

[0015] S3: The outputs of the upper branch and the lower branch are fused and reconstructed by a convolution layer to obtain an enhanced image;

[0016] S4: A texture gradient loss is designed, a loss function is designed by calculating the gradient difference of the enhanced image and the target image in a specific feature space, so that the clarity and continuity of the image texture and edge features are maintained in the optimization process;

[0017] S5: A multi-level feature mapping is introduced, a convolution network model with hierarchical feature extraction capability is used to extract and analyze the multi-level structure features of the image, so as to calculate the perception loss in the multi-level feature space.

[0018] Advantages of the present application:

[0019] The present application provides a lightweight image enhancement model, especially suitable for portable medical device scenarios, which extracts feature information of medical images through multi-scale convolution and attention mechanism, and introduces a high-frequency component processing module to process the high-frequency components of the image, retain the integrity of the outline and the clarity of the structure, and design a special deep feature extraction module to extract deeper abstract features, enhance the robustness and adaptability of the network; In addition, texture gradient loss and perception loss are designed for the features of medical images to ensure the coherence and accuracy of the texture during image generation, and also make the enhanced image closer to the real image in semantic content and visual perception.

[0020] The present application can effectively improve the visual effect of the ultrasound image, retain the key details in the image, not only improve the effect of ultrasound medical image enhancement, but also ensure the integrity of the medical image outline and the clarity of the structure. Through end-to-end training and optimization, the present application can automatically learn the best strategy for enhancing the visual effect from the ultrasound medical image without human intervention, improving the convenience and practicality of operation. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The present application is an ultrasound medical image enhancement model.

[0022] Figure 2 The present application is a deep feature extraction module. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0024] An ultrasound medical image enhancement method based on deep learning, as shown in Figure 1 , comprising:

[0025] An ultrasound medical image enhancement model is established, the ultrasound medical image enhancement model is trained, and the trained ultrasound medical image enhancement model is used for image enhancement;

[0026] The model includes two branches, the upper branch path is composed of a preliminary feature extraction module, three deep feature extraction modules, a multi-scale information fusion module and an ARAM module; the lower branch is composed of a high-frequency component processing module;

[0027] The ultrasound medical image enhancement model is trained, comprising:

[0028] S1: The ultrasound medical image is processed by the upper branch module in turn;

[0029] The preliminary feature extraction module obtains basic features and provides initial information input for subsequent modules;

[0030] The three deep feature extraction modules further refine the features on the basis of the features to obtain features of different scales to enhance the expression of complex structures and details;

[0031] The multi-scale information fusion module integrates features of different scales to ensure that the optimized image has clear details and complete structure;

[0032] The adaptive regional attention mechanism ARAM is designed by combining regional channel attention and spatial dimension adaptive aggregation to gradually filter and strengthen important information;

[0033] S2: In the lower branch, the ultrasound medical image is processed by the high-frequency component processing module to process the high-frequency components of the medical image, so that the enhanced image is more complete in outline and more clear in structure;

[0034] S3: The outputs of the upper and lower branches are fused and reconstructed by a convolution layer to obtain an enhanced image;

[0035] S4: Design texture gradient loss, design loss function by calculating the gradient difference of the enhanced image and the target image in a specific feature space, so as to maintain the clarity and continuity of the image texture and edge features in the optimization process;

[0036] S5: Introduce multi-level feature mapping, extract and analyze the multi-level structure features of the image through a convolution network model with hierarchical feature extraction capability, and calculate the perception loss in the multi-level feature space.

[0037] Step 1: The image to be processed is analyzed and changed by the preliminary feature extraction module to obtain the corresponding preliminary feature mapping. The preliminary feature extraction module is composed of two convolution layers. The first layer is composed of 16 convolution kernels with a size of 3x3 and a step of 1, and the second layer is composed of 32 convolution kernels with a size of 3x3 and a step of 1.

[0038] Step 1.1: Input the feature mapping obtained in step 1 into the deep feature extraction module, such as Figure 2As shown, a total of three deep feature extraction modules are needed, each having an upper and a lower path. The first layer of the upper path is composed of 32 convolution kernels with a size of 3x3 and a step of 1, followed by a MaxPool2d pooling layer. The second layer is composed of 64 convolution kernels with a size of 3x3 and a step of 1, followed by a MaxPool2d pooling layer. The third layer is first up-sampled, followed by a convolution layer composed of 64 convolution kernels with a size of 3x3 and a step of 1. The fourth layer is also first up-sampled, followed by a convolution layer composed of 32 convolution kernels with a size of 3x3 and a step of 1. Convolution layers can extract local features, while pooling layers can reduce the size of feature maps, thus preserving more important features. By gradually reducing the resolution, the model can focus on the core features of the image and capture higher-level semantic information. The structure of the upper and lower paths is completely consistent, with different convolution kernel sizes used in the upper path (3x3) and the lower path (5x5). Different sizes of convolution kernels correspond to different receptive fields, with smaller convolution kernels focusing more on local details and larger convolution kernels capturing global structural information. This multi-scale feature extraction method can help the model understand the image content more comprehensively, especially for tasks such as medical images that require accurate boundaries. After passing through the first four layers of each path, the extracted feature information is then passed through the ARAM attention mechanism, and the outputs of the two paths are added together as the output of the deep feature extraction module.

[0039] Step 1.2: The output obtained in step 1.1 is concatenated with the output in step 1 and input into a multi-scale information aggregation module. The multi-scale information aggregation module includes two convolution layers. The first convolution layer accepts feature information from all previous modules and is composed of 128 convolution kernels with a size of 3x3 and a step of 1. The second convolution layer is composed of 32 convolution kernels with a size of 1x1 and a step of 1, used to adjust the size of the feature map.

[0040] Step 2: The input of step 1.2 is input into an ARAM attention mechanism. The adaptive regional attention mechanism (ARAM) can combine regional channel attention and spatial dimension adaptive aggregation, gradually filter and strengthen important information, and thus improve the performance of the network.

[0041] Step 3: The image to be processed is simultaneously passed through a high-frequency component processing module from the second branch. First, a high-pass filter is used to retain the high-frequency components of the image, and then a convolution layer composed of 32 convolution kernels with a size of 3x3 and a step of 1 is used to extract the high-frequency feature information of the image.

[0042] Step 4: A texture gradient loss is designed by calculating the gradient difference between the enhanced image and the real image to maintain clear texture features during the enhancement process. The texture gradient loss is represented as:

[0043]

[0044] where, denotes the generated image, I denotes the real label image, Sobel(.) denotes the gradient of the image calculated by applying the Sobel operator. The specific method is:

[0045] First, apply the Sobel operator to the enhanced image and the original image I respectively to obtain their gradient images;

[0046] Then calculate the difference between the two images in the gradient;

[0047] Finally, use the L1 norm to quantify the gradient difference as the loss value

[0048] Step 5: Introduce a pre-trained convolutional network model to extract high-level feature representations of the image, and use the difference between the enhanced image and the target image in the feature space as the perceptual loss, which is represented as:

[0049]

[0050] where, denotes the j-th layer of the pre-trained neural network, j ∈ {1, 2, …, L}, L represents the number of network layers used to calculate the perceptual loss. The specific method is:

[0051] First, select a pre-trained model and an intermediate layer of the model as the feature extraction network;

[0052] Then pass the original image and the generated image through the feature extraction network to extract their feature representations;

[0053] Next, calculate the difference between them in the high-level feature space;

[0054] Finally, weight the perceptual loss of different layers.

[0055] Further, the network is trained and optimized by combining the total loss function, which includes the pixel-level loss, the texture gradient loss, and the perceptual loss, and the total loss function is represented as:

[0056] L all = λ1L pixel + λ2L ore + λ3L tg

[0057] where L pixel is the basic loss function, i.e., the pixel-level loss of the image, L all represents the total loss, λ1, λ2, λ3 represent the weights corresponding to each loss function.

[0058] While embodiments of the application have been shown and described, it is to be understood that the embodiments described are merely exemplary of the principles and application of the present application. Numerous modifications and adaptions can be made without departing from the spirit and scope of the present application, which is defined by the following claims and their equivalents.

Claims

1. A deep learning-based method for enhancing ultrasound medical images, characterized in that, include: An ultrasound medical image enhancement model is established, trained, and then used to enhance images. The model consists of two branches: the upper branch consists of a preliminary feature extraction module, three deep feature extraction modules, a multi-scale information fusion module, and an adaptive regional attention mechanism (ARAM) module; the lower branch consists of a high-frequency component processing module. The training of the ultrasound medical image enhancement model includes: S1: Ultrasound medical images are processed sequentially through the modules in the upper branch; The preliminary feature extraction module acquires basic features, providing initial information input for subsequent modules; The three deep feature extraction modules further refine the features on the basis of the basic features to obtain features at different scales, so as to enhance the expression of complex structures and details; The multi-scale information fusion module integrates features at different scales to ensure that the optimized image has both clear details and complete structure. By combining regional channel attention and spatial dimension adaptive aggregation, an adaptive regional attention mechanism (ARAM) is designed to progressively filter and enhance important information. The Adaptive Regionalized Attention Mechanism (ARAM) combines regional channel attention and spatial dimension adaptive aggregation to progressively filter and enhance important information, including: Region Channel Attention: The image features are divided into several regions, and regional attention is applied in the channel dimension to identify the most discriminative channel features in each region. Regional channel attention extracts the global features of each region through average pooling and max pooling to obtain the importance of different regions in the channel. The pooled features are processed through multiple convolutional layers to generate the weight vector of each channel in each region. Spatial Dimension Adaptive Aggregation: In the spatial dimension, an adaptive aggregation strategy is used to generate importance maps of different regions: feature maps of different regions are calculated and aggregated by applying adaptive weights to obtain a two-dimensional feature map. This feature map is then processed by convolution and non-linear activation functions to generate a spatial attention map, which makes it more sensitive to the spatial information of different regions, thereby strengthening the feature representation of key regions. S2: In the lower branch, the ultrasound medical image is processed by the high-frequency component processing module to make the enhanced image more complete in outline and clearer in structure. In the next branch, the ultrasound medical image is processed by a high-frequency component processing module to enhance the high-frequency components of the medical image, making the enhanced image more complete in contour and clearer in structure, including: The input image is subjected to Fourier transform to obtain the frequency domain image; a Gaussian high-pass filter is created and applied to the frequency domain image, and an inverse Fourier transform is performed to convert the filtered frequency domain image back to the spatial domain; then, the high-frequency information of the image is extracted through two convolutional layers. S3: The outputs of the upper and lower branches are fused and reconstructed through a convolutional layer to obtain an enhanced image; S4: Design texture gradient loss. The loss function is designed by calculating the gradient difference between the enhanced image and the target image in the gradient feature space, so as to maintain the clarity and coherence of image texture and edge features during the optimization process. S5: Introducing multi-level feature mapping, a convolutional network model with hierarchical feature extraction capabilities is used to extract and analyze the multi-level structural features of the image, thereby calculating the perceptual loss in a multi-level feature space.

2. The ultrasound medical image enhancement method based on deep learning according to claim 1, characterized in that, The preliminary feature extraction module consists of two cascaded convolutional layers. By extracting and refining local features layer by layer, the first layer is used to identify basic edge and texture information, and the second layer further deepens the extraction of image details, thereby obtaining richer basic features.

3. The ultrasound medical image enhancement method based on deep learning according to claim 1, characterized in that, The three deep feature extraction modules each contain two parallel paths, one above the other. The upper path focuses on refining the extraction of local features, while the lower path uses convolutional kernels with a larger receptive field to capture global feature information. Both the upper and lower paths consist of four levels, which gradually extract and refine image features through different convolutional sizes. The first two levels of each path contain feature extraction and compression operations to identify multi-level image details, while the last two levels further enhance feature expression through magnification operations. After the feature information in the upper and lower paths is processed by the Adaptive Regionalized Attention Mechanism (ARAM), ARAM enhances and filters the feature information through regional channel attention and spatial dimension adaptive aggregation to generate richer contextual information and regionalized feature representation. Finally, the outputs of the two paths are merged as the final output of the deep feature extraction module.

4. The ultrasound medical image enhancement method based on deep learning according to claim 1, characterized in that, The multi-scale information aggregation module includes two feature processing layers. The first layer further extracts multi-scale information from the input features to enhance the details and edge features of the image. The second layer is used to appropriately compress and adjust the dimensions of the features in order to optimize the expression effect of the feature map and provide a unified feature scale for subsequent processing.

5. The ultrasound medical image enhancement method based on deep learning according to claim 1, characterized in that, The texture gradient loss described above can enhance the edges and local details of an image, making the contours clearer. The texture gradient loss is expressed as: in, Indicates texture gradient loss. Let I represent the generated image, I represent the real labeled image, Sobel(.) represents the gradient of the image calculated by applying the Sobel operator, and ||||1 represents the L1 norm.

6. The ultrasound medical image enhancement method based on deep learning according to claim 1, characterized in that, The perceptual loss is a feature space-based loss function. It focuses on the similarity of images in high-level features, rather than pixel-by-pixel precise matching. It utilizes a pre-trained convolutional neural network to extract high-level feature representations of the image and measures image quality by comparing the differences between the enhanced and target images in the feature space. The perceptual loss is expressed as: in, Indicates perceived loss. I represents the generated image, and I represents the actual label image. Let j denote the j-th layer of the pre-trained neural network, j∈{1,2…L}, where L represents the number of network layers used to calculate the perceptual loss. This represents the square of the L2 norm.

Images