Pulmonary nodule image enhancement and restoration method based on deep learning

Abstract

The invention discloses a pulmonary nodule image enhancement and restoration method based on deep learning, and relates to the technical field of medical image processing, and the method comprises the following steps: S1, image preprocessing: employing a self-adaptive image normalization method; according to the method provided by the invention, in an image preprocessing stage, adaptive normalization and generative adversarial network denoising can be realized, the image contrast can be effectively improved, the noise can be accurately removed, high-quality data, multi-scale feature fusion and an attention mechanism can be provided for subsequent processing, pulmonary nodule features can be comprehensively and accurately extracted, the detection and classification precision can be improved, and the method is suitable for popularization and application. The recurrent neural network is used for image restoration, fuzzy distortion areas can be successfully reconstructed, image details can be restored, through model training and optimization strategies, adaptability and overall performance of the model to pulmonary nodule images can be improved, training time and data dependence are reduced, powerful technical support is provided for early diagnosis of lung diseases, and the method is suitable for popularization and application. And improvement of medical level is facilitated.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and specifically to a method for lung nodule image enhancement and restoration based on deep learning. Background Art

[0002] Pulmonary nodules are a term frequently used in medical imaging, which mainly describes a lesion that occurs in the lungs. This lesion usually manifests as one or more focal, rounded or irregularly shaped increased density shadows with a diameter of less than or equal to 3 cm in the lungs. These nodules can be solitary or multiple, and are not accompanied by other symptoms such as atelectasis, hilar lymphadenopathy, and pleural effusion. From the perspective of etiology, pulmonary nodules can be caused by a variety of reasons, including but not limited to lung infections (such as tuberculosis, pulmonary fungal disease, etc.), benign lung lesions (such as old lung lesions, sclerosing hemangiomas, etc.) and lung malignancies (such as primary lung cancer, metastatic lung cancer, etc.). Due to the differences in these causes, the imaging manifestations of pulmonary nodules will also be different, such as shape, size, density and other characteristics. When diagnosing pulmonary nodules, doctors usually make a comprehensive judgment based on the patient's medical history, symptoms and imaging examination results. For some nodules that are difficult to diagnose, further examinations may be required, such as biopsies.

[0003] With the development of medical imaging technology, although CT and other imaging methods can obtain lung nodule images, the image quality is worrying due to the influence of equipment performance, scanning parameters, and patient factors. Equipment aging may lead to increased image noise, and the patient's breathing movement during scanning can easily cause image blur or distortion. These make it difficult to clearly present the characteristics of lung nodules. Traditional image processing methods are unable to cope with these problems. For example, simple grayscale adjustment cannot accurately highlight the characteristics of lung nodules, and traditional filtering and denoising can easily destroy image details. Although deep learning has made progress, traditional methods focus more on single-scale features and do not handle long sequence information well during image restoration. It is difficult to meet the needs of lung nodule image analysis. Innovative solutions are urgently needed to improve the image quality and analysis accuracy of lung nodules. Therefore, it is necessary to propose a lung nodule image enhancement and restoration method based on deep learning to solve the problems in the existing technology. Summary of the invention

[0004] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide a lung nodule image enhancement and restoration method based on deep learning, which can solve the problems existing in the above-mentioned existing technology, significantly improve the quality of lung nodule images, and then improve the accuracy of lung nodule detection and classification, providing strong technical support for the accurate diagnosis of lung diseases.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for lung nodule image enhancement and restoration based on deep learning, the method comprising the following steps:

[0006] S1. Image preprocessing: Adaptive image normalization method is used to define the pixel gray value of a certain area in the lung nodule image as x. ij , the grayscale mean of the area is μ, the standard deviation is σ, and the normalized pixel value is Where k and b are nonlinear adjustment coefficients determined according to the characteristics of the pulmonary nodule image. At the same time, the noise removal based on the generative adversarial network is used to construct the generator network G and the discriminator network D. The generator network learns the noise distribution characteristics through a deep neural network and obtains the original image I in Remove the noise and get the denoised image I out =G(I in ), the discriminator network distinguishes the denoised image from the real noise-free image through adversarial training, and its loss function is where p data (x) is the real image data distribution, x is the real image, p z (z) is the noise data distribution, z is the noise input, Calculate for expectation;

[0007] S2. Feature extraction and enhancement: Using multi-scale feature fusion, through convolution kernels of different sizes K s (s represents different scales) and image I are convolved to obtain a multi-scale feature map F s =K s *I, then concatenate and weightedly fuse these feature maps. When weighted fusion is performed, the weights of feature maps of different scales are set to ω s , the fused feature F = ∑ s ω s F s ; And introduce the attention mechanism, build the attention module A, and calculate the attention weight W for the feature map F A =A(F), and the weighted feature F′=W is obtained A ⊙F, where ⊙ represents element-by-element multiplication;

[0008] S3, Image restoration: Using the recurrent neural network RNN, the lung nodule image sequence S = {s1, s2, ..., s n} Input, define the hidden state of RNN at time t as h t , and its update formula is h t =f RNN (s t ,h t-1 ), where f RNNis the state update function of RNN, which reconstructs the image sequence according to the hidden state to obtain the restored image;

[0009] S4. Model training and optimization: Improve the pre-trained model, adjust the parameters of the layers related to lung nodule image feature extraction, build a multi-task learning framework, integrate lung nodule detection, classification and image restoration tasks, share some network layer parameters and design a joint loss function L total =λ1L detect +λ2L classify +λ3L restore , where λ1, λ2, λ3 are weight coefficients of different character losses, L detect , L classify , L restore The loss functions for detection, classification, and restoration tasks are respectively optimized. The adversarial training strategy is optimized, the discriminator structure is improved, and the loss function that includes the morphology and texture features of lung nodules is designed. At the same time, the training balance of the generator and discriminator is dynamically adjusted.

[0010] Furthermore, in the adaptive image normalization method, the nonlinear adjustment coefficients k and b are determined as follows: first, the lung nodule image is divided into multiple sub-regions, the grayscale histogram of each sub-region is calculated, and the frequency of occurrence of different grayscale levels is counted. Then, the k value is determined according to the difference in grayscale distribution between the lung nodule region and the surrounding tissue region, so that the grayscale contrast of the lung nodule region is significantly improved, and then the b value is determined according to the brightness distribution of the overall image.

[0011] Furthermore, in the noise removal based on generative adversarial network, the network structure of the generator network G includes multiple convolutional layers and deconvolutional layers. The convolutional layer is used to extract image features, and the deconvolutional layer is used to reconstruct the image and remove noise. During the training process, the parameters of the discriminator network D are first fixed, and the generator loss function is minimized. To update the parameters of the generator network, and then fix the parameters of the generator network, by minimizing the discriminator loss function L D To update the discriminator network parameters, the training is repeated alternately until the generator network can stably output high-quality denoised images.

[0012] Furthermore, in the weighted fusion process of multi-scale feature fusion, the weights ω of feature maps of different scales are determined. s The method is to build a feature evaluation network E, input feature maps of different scales into the network, and the network outputs a score s representing the importance of the feature. E , and then normalize these scores to get the weight ω s ,The structure of the feature evaluation network E includes a fully connected layer and an activation function. The fully connected layer is used to integrate the global information of the feature map, and the activation function is used to map the information to an appropriate range in order to determine the weight.

[0013] Furthermore, in the process of introducing the attention mechanism, the structure of the attention module A includes a spatial attention submodule and a channel attention submodule. The spatial attention submodule obtains the spatial attention map M by performing a pooling operation on the feature map in the spatial dimension. s The channel attention submodule obtains the channel attention map M by performing a pooling operation on the feature map in the channel dimension. c , the final attention weight W A =M s ⊙M c ,The spatial pooling operation adopts a combination of average pooling and maximum pooling, and the channel pooling operation adopts global average pooling.

[0014] Furthermore, when the recurrent neural network RNN ​​performs image restoration, the hidden state h t The update function f R NN uses the calculation formula of the long short-term memory network LSTM unit, that is, i t =σ(W i s t +U i h t-1 +b i ), f t =σ(W f s t +U f h t-1 +b f ), o t =σ(W o s t +U o h t-1 +b o ), h t =o t ⊙tanh(c t ), where i t 、f t , c t , o t are the calculation results of the input gate, forget gate, candidate memory unit, memory unit, and output gate respectively. W and U are weight matrices, b is the bias vector, σ is the sigmoid activation function, and ⊙ represents element-by-element multiplication.

[0015] Furthermore, when improving the pre-trained model, for the convolution layer in the pre-trained model, the differences between the features learned on the natural image dataset and the features of the lung nodule image are analyzed, and the initialization parameters of the convolution kernel are redesigned according to the grayscale texture feature distribution of the lung nodule image. The specific operation is to first count the frequency of occurrence of different grayscale texture patterns in the lung nodule image, and map this frequency information to the initialization parameters of the convolution kernel, so that the convolution layer can have a certain adaptability to the features of the lung nodule image in the initial state, and then further adjust the parameters during the training process to optimize the feature extraction effect.

[0016] Furthermore, in constructing a multi-task learning framework, the shared part of the network layer adopts a feature pyramid structure, and the branch networks of lung nodule detection, classification and image restoration tasks are connected at different levels of the feature pyramid. The low-level feature pyramid mainly provides rich local detail information for the image restoration task, and the high-level feature pyramid provides more abstract global feature information for the lung nodule detection and classification tasks. Each branch network designs a different loss function according to its own task requirements, and the loss function L is used in the training process. total Update the parameters.

[0017] Furthermore, in the optimization of adversarial training strategy, when improving the discriminator structure, a multi-scale feature fusion module is added to the discriminator network. This module extracts multi-scale features from the input image before identification. When designing the loss function including lung nodule morphology and texture features, for lung nodule morphology features, the curvature change rate and area-to-perimeter ratio of the contour are calculated as part of the loss function. For texture features, the contrast and correlation statistics of the gray-level co-occurrence matrix are calculated as part of the loss function. In the process of dynamically adjusting the training balance of the generator and the discriminator, a training round threshold T is set. After each training round t, the quality evaluation index Q of the generated image is calculated. G And the accuracy of the discriminator A D , when Q G Below the set lower limit and A D When it is higher than the set upper limit, the learning rate and training step size of the generator and discriminator are adjusted according to the preset rules to avoid mode collapse and ensure the stability of training.

[0018] Compared with the existing technology, this deep learning-based lung nodule image enhancement and restoration method has the following beneficial effects:

[0019] The method provided by the present invention can adaptively perform normalization and generative adversarial network denoising in the image preprocessing stage, effectively improve image contrast and accurately remove noise, and provide high-quality data for subsequent processing. The multi-scale feature fusion and attention mechanism can comprehensively and accurately extract lung nodule features and improve detection and classification accuracy. The recurrent neural network is used for image restoration, which can successfully reconstruct blurred and distorted areas and restore image details. Through model training and optimization strategies, the model's adaptability to lung nodule images and overall performance can be improved, and training time and data dependence can be reduced, providing strong technical support for the early diagnosis of lung diseases, which is conducive to helping improve medical standards.

[0020] Other advantages, objectives and features of the present invention will be set forth in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be taught from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flowchart of a deep learning-based lung nodule image enhancement and restoration method. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Embodiment 1

[0025] The lung disease diagnosis center of a large hospital receives a large amount of CT image data of patients suspected of having lung nodules every day. These images are crucial for accurately judging the nature of lung nodules (benign or malignant) and formulating subsequent treatment plans. However, due to the differences in CT equipment, the patient's respiratory movements during the scan, and the complex interaction of body tissues with X-rays, the acquired lung nodule images often have problems such as noise interference, low contrast, blur or local distortion, which seriously affects the doctor's accurate judgment of lung nodules. Traditional image processing methods and conventional deep learning models cannot fully meet the needs of high-precision diagnosis when processing these images. Therefore, this deep learning-based lung nodule image enhancement and restoration method was introduced into the hospital's image diagnosis process.

[0026] First, the input lung nodule image is divided into multiple sub-regions (for example, divided into 5x5 sub-region grids according to a certain pixel interval). For each sub-region, its grayscale histogram is calculated and the frequency of occurrence of different grayscale levels is counted. Suppose the grayscale value of the pixel in a sub-region is x ij The grayscale mean value of the region is μ, and the standard deviation is σ. The nonlinear adjustment coefficient k is determined according to the difference in grayscale distribution between the pulmonary nodule region and the surrounding tissue region (for example, by comparing the difference multiples of the grayscale mean value between the core region of the pulmonary nodule and the surrounding lung tissue region, and combining the empirical coefficient to determine the k value), and then the b value is determined according to the brightness distribution of the overall image (for example, calculating the median of the overall grayscale value of the image, and determining b according to the deviation between the median and the expected brightness range). The normalized pixel value

[0027] Construct the generator network G and the discriminator network D. The generator network contains multiple convolutional layers and deconvolutional layers. The convolutional layer is used to extract image features, and the deconvolutional layer is used to reconstruct the image and remove noise. The discriminator network distinguishes the denoised image from the real noise-free image through adversarial training. Its loss function is where p data (x) is the real image data distribution, p z (z) is the noise data distribution, z is the noise input, Calculate the expectation.

[0028] During the training process, the parameters of the discriminator network D are fixed first, and the generator loss function is minimized. To update the parameters of the generator network, and then fix the parameters of the generator network, by minimizing the discriminator loss function L D To update the parameters of the discriminator network, the training is performed alternately. For example, the number of training iterations is set to 10,000, and the quality index of the generated image (such as peak signal-to-noise ratio) is checked every 100 iterations. When the index does not improve for 5 consecutive times, the training is stopped. At this time, the generator network can stably output high-quality denoised images.

[0029] Through different sizes of convolution kernel K s (s represents different scales, such as 3x3, 5x5 and 7x7) and image I are convolved to obtain a multi-scale feature map F s =K s *I, determine the weights ω of feature maps of different scales s When , a feature evaluation network E is constructed, and feature maps of different scales are input into the network respectively, and the network outputs a score s representing the importance of the feature E , and then normalize these scores to get the weight ω s The structure of the feature evaluation network E includes a fully connected layer and an activation function (such as ReLU activation function). The fully connected layer is used to integrate the global information of the feature map, and the activation function is used to map the information to an appropriate range in order to determine the weight. Finally, these feature maps are weighted fused, and the fused feature F = ∑ s ω s F s .

[0030] The structure of the attention module A includes a spatial attention submodule and a channel attention submodule. The spatial attention submodule obtains the spatial attention map M by performing a pooling operation on the feature map in the spatial dimension (using a combination of average pooling and maximum pooling, with a pooling kernel size of 2x2). s The channel attention submodule obtains the channel attention map M by performing a pooling operation on the feature map in the channel dimension (using global average pooling). c , the final attention weight W A =M s ⊙M c , and get the weighted feature F ′ =W A ⊙F, where ⊙ represents element-wise multiplication.

[0031] The pulmonary nodule image sequence S = {s1, s2, …, s n} Input, let the hidden state of RNN at time t be h t , and its update formula is h t =f RNN (s t ,h t-1 ), where f RNN The calculation formula of the long short-term memory network LSTM unit is adopted, that is, i t =σ(W i s t +U i h t-1 +b i ), f t =σ(Wf s t +U f h t-1 +b f ), o t =σ(W o s t +U o h t-1 +b o ), h t =o t ⊙tanh(c t ), where i t 、f t , c t , o t are the calculation results of the input gate, forget gate, candidate memory unit, memory unit, and output gate respectively. W and U are weight matrices, b is the bias vector, σ is the sigmoid activation function, ⊙ represents element-by-element multiplication, and the image sequence is reconstructed according to the hidden state to obtain the restored image.

[0032] A model pre-trained on a large-scale natural image dataset (such as a ResNet model) is introduced. For the convolution layer in the pre-trained model, the differences between the features learned on the natural image dataset and the features of the lung nodule image are analyzed. According to the grayscale texture feature distribution of the lung nodule image, the initialization parameters of the convolution kernel are redesigned. The specific operation is to first count the frequency of occurrence of different grayscale texture patterns in the lung nodule image, and map this frequency information to the initialization parameters of the convolution kernel (for example, by constructing a mapping table from grayscale texture frequency to convolution kernel parameters), so that the convolution layer can have a certain adaptability to the features of the lung nodule image in the initial state, and then further adjust the parameters during the training process to optimize the feature extraction effect.

[0033] A multi-task learning framework was constructed to integrate the tasks of lung nodule detection, classification and image restoration. The shared network layers adopted a feature pyramid structure. The branch networks of lung nodule detection, classification and image restoration tasks were connected at different levels of the feature pyramid. The low-level feature pyramid mainly provided rich local detail information for the image restoration task, while the high-level feature pyramid provided more abstract global feature information for the lung nodule detection and classification tasks. Different loss functions were designed for each branch network according to its own task requirements, and the loss function L was used in the training process. total =λ1L detect +λ2L classify +λ3L restoreUpdate the parameters, where λ1, λ2, and λ3 are weight coefficients of different task losses (for example, λ1 = 0.3, λ2 = 0.3, and λ3 = 0.4 are set based on experience), L d etect,L classify , L r estore is the loss function for detection, classification, and restoration tasks respectively.

[0034] When improving the discriminator structure, a multi-scale feature fusion module was added to the discriminator network. This module used a method similar to the previous multi-scale feature fusion to extract multi-scale features from the input image before identification. When designing a loss function that includes lung nodule morphology and texture features, for lung nodule morphology features, the curvature change rate, area-to-perimeter ratio and other indicators of its contour were calculated as part of the loss function. For texture features, the contrast, correlation and other statistics of the gray-level co-occurrence matrix were calculated as part of the loss function. In the process of dynamically adjusting the training balance between the generator and the discriminator, the training round threshold T = 500 was set. After each training round t = 50, the quality evaluation index Q of the generated image was calculated. G (such as structural similarity index) and the accuracy of the discriminator A D , when Q G Below the set lower limit (such as 0.6) or A D When it is higher than the set upper limit (such as 0.95), the learning rate and training step size of the generator and discriminator are adjusted according to the preset rules to avoid mode collapse and ensure the stability of training.

[0035] Effects brought by this embodiment: Through the application of this embodiment, in the image preprocessing stage, adaptive normalization significantly improves the contrast of lung nodule images, generative adversarial network denoising effectively removes various types of noise in the image, and provides a clear image basis for subsequent analysis. The combination of multi-scale feature fusion and attention mechanism accurately extracts the key features of lung nodules, which increases the accuracy of lung nodule detection by about 20% and the accuracy of classification by about 15%. In terms of image restoration, the recurrent neural network successfully reconstructs blurred and distorted areas, making the image details more complete, which helps doctors to more accurately judge the morphology and structural characteristics of lung nodules. The model training and optimization strategy improves the adaptability and efficiency of the entire model, reduces the training time and data requirements by about 30%, greatly improves the accuracy and efficiency of lung disease diagnosis, and provides a more reliable basis for the early diagnosis and treatment of patients. At the same time, it also reduces the pressure on medical resources, and has significant clinical application value and social benefits.

[0036] Embodiment 2

[0037] In the telemedicine diagnosis platform for lung diseases, the quality of lung nodule CT images uploaded by primary medical institutions varies greatly. These images may be compressed to a certain extent during transmission, resulting in loss of image details, increased noise, blurred lung nodule features, and other problems. Remote diagnosis experts need to make accurate diagnostic judgments based on these images, which poses a challenge to the quality and analyzability of the images. This deep learning-based lung nodule image enhancement and restoration method is applied to the telemedicine diagnosis platform to improve image quality and assist experts in making accurate diagnoses.

[0038] For the uploaded lung nodule image, first convert it into a standard grayscale image format, then segment the image into several overlapping regions (for example, use a sliding window of 4x4 pixels with a step size of 2 pixels for segmentation), and for each region, calculate the grayscale mean μ and standard deviation σ, as well as the pixel grayscale value x in the region. ij According to the overall grayscale dynamic range of the lung nodule image and the grayscale difference characteristics between regions, the nonlinear adjustment coefficients k (e.g., k is determined by analyzing the statistical law of grayscale differences between lung nodule regions and normal lung tissue regions in a large number of similar images) and b (e.g., b is determined based on the skewness of the overall grayscale distribution of the image to correct the overall brightness of the image), and the formula is used. Perform normalization.

[0039] Construct the generator network G and the discriminator network D. The generator network has five convolutional layers and four deconvolutional layers. The convolutional layers use convolution kernels of different sizes (such as 3x3 and 5x5 alternately) to extract multi-scale image features. The deconvolutional layers are used to gradually reconstruct the image and remove noise. The discriminator network consists of four convolutional layers to distinguish the denoised image output by the generator from the real noise-free image. Its loss function where p data (x is the real image data distribution, p z (z) is the noise data distribution, z is the noise input, Calculate the expectation.

[0040] In the training phase, the network parameters of the generator and discriminator are initialized first, and then the discriminator parameters are fixed to minimize the generator loss function. Use the stochastic gradient descent algorithm to update the generator parameters. Then, fix the generator parameters and minimize the discriminator loss function L D Update the discriminator parameters and train alternately in this way. Set the total training to 8000 times. Calculate the mean square error between the generated image and the real noise-free image every 200 times. When the change of the mean square error for 10 consecutive times is less than 0.01, stop training. At this time, the generator can output high-quality denoised images.

[0041] Use three different scales of convolution kernel Ks (3x3, 5x5 and 7x7 respectively) Convolution operation is performed on the normalized and denoised image I to obtain a multi-scale feature map F s =K s *I, for determining the weights ω of feature maps of different scales s , construct a three-layer feature evaluation network E, whose input layer receives feature maps of different scales, the middle layer is a fully connected layer with 64 neurons and uses the Sigmoid activation function, and the output layer is a neuron that outputs a score s representing the importance of the feature E , these scores are normalized by the Softmax function to obtain the weight ω s , and finally perform weighted fusion, the fused feature F = ∑ s ω s F s .

[0042] The attention module A consists of a spatial attention submodule and a channel attention submodule. The spatial attention submodule first performs 3x3 average pooling and maximum pooling operations on the feature map, and then concatenates the two results and passes a 1x1 convolution layer and a Sigmoid activation function to obtain the spatial attention map M. s After the channel attention submodule performs global average pooling on the feature map, it passes through two fully connected layers (128 neurons in the first layer, 1 neuron in the second layer, and ReLU activation function in the middle) to obtain the channel attention map M c , the final attention weight W A =M s ⊙M c , and get the weighted feature F ′ =W A ⊙F, where ⊙ represents element-wise multiplication.

[0043] The pulmonary nodule image sequence S = {s1, s2, …, s n} Input, let the hidden state of RNN at time t be h t , and its update formula is h t =f RNN (s t ,h t-1 ), where f RNN The calculation formula of the long short-term memory network LSTM unit is adopted, that is, i t =σ(W i s t +U i h t-1 +b i ), f t =σ(W f s t +Uf h t-1 +b f ), o t =σ(W o s t +U o h t-1 +b o ), h t =o t ⊙tanh(c t ), where i t 、f t , c t , o t are the calculation results of the entry gate, forget gate, candidate memory unit, memory unit, and output gate, respectively. W and U are weight matrices, b is the bias vector, σ is the sigmoid activation function, and G represents element-by-element multiplication. The image sequence is reconstructed according to the hidden state to obtain the restored image. When training RNN, the time backpropagation algorithm is used, the learning rate is set to 0.001, and the training is 3000 time steps. The image restoration error on the validation set is calculated every 500 time steps. When the error change is less than 0.005 for three consecutive times, the training is stopped.

[0044] A VGG model pre-trained on a large-scale natural image dataset was selected to analyze the differences in feature extraction of its convolutional layer on natural images and lung nodule images. The convolutional layer parameters of the pre-trained model were adjusted according to the grayscale distribution characteristics and texture feature patterns of the lung nodule images. For example, for the convolutional layer that extracts edge features, the convolution kernel weight parameters were modified according to the statistical law of grayscale changes at the edge of the lung nodule (e.g., some weight values ​​were adjusted according to the direction of the edge grayscale gradient). Then, fine-tuning training was performed on the lung nodule image dataset. The learning rate of fine-tuning training was set to 0.0001, and training was performed 2000 times. The accuracy of feature extraction was calculated every 400 times. When the accuracy increased by less than 0.5% for three consecutive times, fine-tuning was stopped.

[0045] A multi-task learning framework is constructed to integrate lung nodule detection, classification and image restoration tasks. The shared network layer adopts a feature pyramid structure. The bottom layer (higher resolution) of the pyramid is connected to the image restoration task branch network, the middle layer is connected to the detection task branch network, and the high layer is connected to the classification task branch network. The loss function is designed for each branch network. The detection task adopts the loss function L based on region proposal. detect , the classification task uses the cross entropy loss function L classify , the image restoration task uses a loss function L based on pixel differences restore , through the joint loss function L total =λ1Ldetect +λ2L classify +λ3L restore The parameters are updated, where λ1=0.25, λ2=0.25, and λ3=0.5. During the training process, the stochastic gradient descent algorithm is used, and the initial learning rate is set to 0.001. The learning rate decays to 0.9 times of the original after every 1000 training times, and the total number of training times is 6000 times.

[0046] The discriminator structure was improved, and a multi-scale feature fusion module was added to the discriminator network. The multi-scale features were first extracted in a manner similar to the previous multi-scale feature fusion, and then concatenated and passed through a convolutional layer to obtain the final identification features. A loss function that included the morphological and texture features of lung nodules was designed. For the morphological features of lung nodules, the differences between their circularity, ellipticity and other indicators and the real image were calculated as part of the loss function; for the texture features, the differences between the energy, entropy and other statistics of the grayscale co-occurrence matrix and the real image were calculated as part of the loss function.

[0047] When dynamically adjusting the training balance between the generator and the discriminator, set the training round number threshold T = 400, and calculate the quality evaluation index Q of the generated image after each training round t = 40. G (such as peak signal-to-noise ratio) and the accuracy of the discriminator A D , when Q G Below the set lower limit of 0.55 or A D When it is higher than the set upper limit of 0.93, the learning rate of the generator and discriminator (such as the generator learning rate multiplied by 0.8, the discriminator learning rate multiplied by 1.2) and the training step size (such as the generator training step size increased by 1, the discriminator training step size reduced by 1) are adjusted according to the preset rules to avoid mode collapse and ensure the stability of training.

[0048] The effects brought by this embodiment are as follows: in the application scenario of this remote medical diagnosis platform, after image preprocessing, the image noise is significantly reduced and the grayscale contrast is significantly improved, making the lung nodules more prominent in the image. The application of multi-scale feature fusion and attention mechanism enhances the ability to extract lung nodule features, increases the recall rate of lung nodule detection by about 25%, and increases the classification accuracy by about 20%. The image restoration link effectively restores the detailed information lost due to compression and transmission, allowing experts to observe the internal structure and edge features of lung nodules more clearly. The model training and optimization strategy reduces the training time and data requirements by about 25%, and improves the efficiency and accuracy of the entire remote diagnosis process. This helps primary medical institutions and experts to collaborate better, improves the reliability of remote medical diagnosis, enables more patients to receive timely and accurate diagnosis suggestions locally, and improves the uneven distribution of medical resources, which has important medical and social value.

[0049] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the same elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for lung nodule image enhancement and restoration based on deep learning, characterized in that: The method comprises the following steps: S1. Image preprocessing: Adaptive image normalization method is used to define the pixel gray value of a certain area in the lung nodule image as x. ij , the grayscale mean of the area is μ, the standard deviation is σ, and the normalized pixel value is Where k and b are nonlinear adjustment coefficients determined according to the characteristics of the pulmonary nodule image. At the same time, the noise removal based on the generative adversarial network is used to construct the generator network G and the discriminator network D. The generator network learns the noise distribution characteristics through a deep neural network and obtains the original image I in Remove the noise and get the denoised image I out =G(I in ), the discriminator network distinguishes the denoised image from the real noise-free image through adversarial training, and its loss function is where p data (x) is the real image data distribution, x is the real image, p z (z) is the noise data distribution, z is the noise input, Calculate for expectation; S2. Feature extraction and enhancement: Using multi-scale feature fusion, through convolution kernels of different sizes K s (s represents different scales) and image I are convolved to obtain a multi-scale feature map F s =K s *I, then concatenate and weightedly fuse these feature maps. When weighted fusion is performed, the weights of feature maps of different scales are set to ω s , the fused feature F = ∑ s ω s F s ; And introduce the attention mechanism, build the attention module A, and calculate the attention weight W for the feature map F A =A(F), and the weighted feature F′=W is obtained A ⊙F, where ⊙ represents element-wise multiplication; S3, Image restoration: Using the recurrent neural network RNN, the lung nodule image sequence S = {s1, s2, ..., s n } Input, define the hidden state of RNN at time t as h t , and its update formula is h t =f RNN (s t ,h t-1 ), where f RNN is the state update function of RNN, which reconstructs the image sequence according to the hidden state to obtain the restored image; S4. Model training and optimization: Improve the pre-trained model, adjust the parameters of the layers related to lung nodule image feature extraction, build a multi-task learning framework, integrate lung nodule detection, classification and image restoration tasks, share some network layer parameters and design a joint loss function L total =λ1L detect +λ2L classify +λ3L restore , where λ1, λ2, λ3 are weight coefficients of different character losses, L detect , L classify , L restore The loss functions for detection, classification, and restoration tasks are respectively optimized. The adversarial training strategy is optimized, the discriminator structure is improved, and the loss function that includes the morphology and texture features of lung nodules is designed. At the same time, the training balance of the generator and discriminator is dynamically adjusted.

2. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: In the adaptive image normalization method, the nonlinear adjustment coefficients k and b are determined as follows: first, the lung nodule image is divided into multiple sub-regions, the grayscale histogram of each sub-region is calculated, and the frequency of occurrence of different grayscale levels is counted. Then, the k value is determined according to the difference in grayscale distribution between the lung nodule area and the surrounding tissue area, so that the grayscale contrast of the lung nodule area is significantly improved. Then, the b value is determined according to the brightness distribution of the overall image.

3. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: In noise removal based on generative adversarial networks, the network structure of the generator network G contains multiple convolutional layers and deconvolutional layers. The convolutional layer is used to extract image features, and the deconvolutional layer is used to reconstruct the image and remove noise. During the training process, the parameters of the discriminator network D are first fixed, and the generator loss function is minimized. To update the parameters of the generator network, and then fix the parameters of the generator network, by minimizing the discriminator loss function L D To update the discriminator network parameters, the training is repeated alternately until the generator network can stably output high-quality denoised images.

4. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: In the weighted fusion process of multi-scale feature fusion, the weights ω of feature maps of different scales are determined s The method is to build a feature evaluation network E, input feature maps of different scales into the network, and the network outputs a score s representing the importance of the feature. E , and then normalize these scores to get the weight ω s ,The structure of the feature evaluation network E includes a fully connected layer and an activation function. The fully connected layer is used to integrate the global information of the feature map, and the activation function is used to map the information to an appropriate range in order to determine the weight.

5. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: In the process of introducing the attention mechanism, the structure of the attention module A includes a spatial attention submodule and a channel attention submodule. The spatial attention submodule obtains the spatial attention map M by performing a pooling operation on the feature map in the spatial dimension. s The channel attention submodule obtains the channel attention map M by performing a pooling operation on the feature map in the channel dimension. c , the final attention weight W A =M s ⊙M c ,The spatial pooling operation adopts a combination of average pooling and maximum pooling, and the channel pooling operation adopts global average pooling.

6. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: When the recurrent neural network RNN ​​performs image restoration, the hidden state h t The update function f R NN uses the calculation formula of the long short-term memory network LSTM unit, that is, i t =σ(W i s t +U i h t-1 +b i ), f t =σ(W f s t +U f h t-1 +b f ), o t =σ(W o s t +U o h t-1 +b o ), h t =o t ⊙tanh(c t ), where i t 、f t , c t , o t are the calculation results of the input gate, forget gate, candidate memory unit, memory unit, and output gate respectively. W and U are weight matrices, b is the bias vector, σ is the sigmoid activation function, and ⊙ represents element-by-element multiplication.

7. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: When improving the pre-trained model, for the convolution layer in the pre-trained model, the differences between the features learned on the natural image dataset and the features of the lung nodule image are analyzed, and the initialization parameters of the convolution kernel are redesigned according to the grayscale texture feature distribution of the lung nodule image. The specific operation is to first count the frequency of occurrence of different grayscale texture patterns in the lung nodule image, and map this frequency information to the initialization parameters of the convolution kernel, so that the convolution layer can have a certain adaptability to the features of the lung nodule image in the initial state, and then further adjust the parameters during the training process to optimize the feature extraction effect.

8. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: In constructing a multi-task learning framework, the shared network layer adopts a feature pyramid structure. The branch networks of lung nodule detection, classification and image restoration tasks are connected at different levels of the feature pyramid. The low-level feature pyramid mainly provides rich local detail information for the image restoration task, and the high-level feature pyramid provides more abstract global feature information for the lung nodule detection and classification tasks. Each branch network designs a different loss function according to its own task requirements, and the loss function L is used in the training process. total Update the parameters.

9. The method for lung nodule image enhancement and restoration based on deep learning according to claim 1, characterized in that: In the optimization adversarial training strategy, when improving the discriminator structure, a multi-scale feature fusion module is added to the discriminator network. This module extracts multi-scale features from the input image before identification. When designing the loss function that includes the morphological and texture features of lung nodules, for the morphological features of lung nodules, the curvature change rate of the contour and the area-to-perimeter ratio index are calculated as part of the loss function. For the texture features, the contrast and correlation statistics of the gray-level co-occurrence matrix are calculated as part of the loss function. In the process of dynamically adjusting the training balance of the generator and the discriminator, the training round number threshold T is set. After each training round t, the quality evaluation index Q of the generated image is calculated. G And the accuracy of the discriminator A D , when Q G Below the set lower limit and A D When it is higher than the set upper limit, the learning rate and training step size of the generator and discriminator are adjusted according to the preset rules to avoid mode collapse and ensure the stability of training.

Images