Lung cancer image enhancement method based on generative adversarial network

Abstract

The application discloses a lung cancer image enhancement method based on a generative adversarial network, S1. obtaining a standardized lung cancer patchy image data set; S2. forming a labeled lung cancer patchy image training data; S3. generating a lung cancer patchy multiscale dynamic feature map; S4. outputting a preliminary reconstructed lung cancer patchy image data; S5. constructing a local perception discriminator; S6. obtaining a trained dilated convolution fusion generative adversarial network model; S7. inputting newly obtained low-quality lung cancer patchy image data into the trained dilated convolution fusion generative adversarial network model, outputting reconstructed lung cancer patchy image data, and generating lung cancer patchy image quality evaluation results. The application improves the authenticity and credibility of the generated image in the key lesion area.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of lung cancer imaging, and particularly relates to a lung cancer image enhancement method based on a generative adversarial network. BACKGROUND

[0002] With the continuous development of medical imaging technology, early screening and diagnosis of lung cancer increasingly rely on image data obtained based on computed tomography and magnetic resonance imaging devices, especially lung patchy images collected under low-dose scanning conditions, which become an important data source for lung cancer screening. However, due to the problems of reduced signal-to-noise ratio, increased artifacts and missing image details caused by low-dose imaging itself, the existing lung cancer images have significant deficiencies in actual clinical application, which affects the accurate identification and analysis of lung cancer patchy lesions by doctors.

[0003] At present, the processing of low-quality lung cancer images mainly adopts two types of methods. One type is a traditional image processing-based enhancement method, such as histogram equalization, sharpening filtering and contrast stretching means. Although it can improve the overall clarity of the image to a certain extent, it is difficult to effectively restore the structural details of the lung cancer patchy lesion area, and it is easy to introduce noise, leading to artifact enhancement, which affects subsequent lesion identification and quantitative evaluation. The other type is a deep learning-based image enhancement method, such as using a convolutional neural network or a generative adversarial network to reconstruct the overall image. However, the following problems generally exist: On the one hand, due to the failure to fully consider the local characteristics of the lung cancer patchy lesion area, the model is prone to ignore the lesion details or produce over-smoothing phenomenon during the enhancement process, reducing the recognizability of the lesion area. On the other hand, the existing methods often lack pertinence in network structure design, ignoring the complex change characteristics of the edge texture of the patchy lesion, causing blurred boundaries or structural distortion, which cannot meet the actual needs of clinical high-precision diagnosis support.

[0004] In addition, in the existing deep learning methods, although some studies attempt to introduce a dilated convolution to expand the receptive field, a fixed dilated rate is usually adopted, lacking a mechanism for self-adaptive adjustment of dilated parameters according to the local characteristics of the lesion, resulting in poor feature extraction effect in different scales and different texture complexity areas, further limiting the overall performance of image enhancement. At the same time, the existing generator and discriminator design generally focuses on global discrimination, lacking fine-grained perception and authenticity evaluation of the local area of the lesion, causing obvious defects in the restoration of fine-grained features in the reconstructed image, especially the problem of insufficient authenticity in the lesion edge transition area.

[0005] In summary, the prior art in the enhancement processing of lung cancer patchy images has the following defects: insufficient perception of local lesion characteristics, weak structure detail reconstruction capability, poor boundary preservation effect, and disconnection between overall enhancement and local authenticity evaluation, which seriously restricts the application value of low-quality images in the early diagnosis of lung cancer in clinical practice. Therefore, a new method that takes into account local feature perception and overall quality optimization is urgently needed to improve the reconstruction quality of low-quality lung cancer patchy images and the accuracy of lesion identification, and to meet the urgent demand for high-quality lung image data in precision medicine. SUMMARY

[0006] One object of the present application is to provide a lung cancer image enhancement method based on a generative adversarial network, which improves the authenticity and credibility of the generated image in the key lesion area.

[0007] According to the lung cancer image enhancement method based on the generative adversarial network, the following steps are included:

[0008] S1. Collecting lung cancer patchy image data and establishing a lung cancer patchy image data set, preprocessing the lung cancer patchy image data set, the preprocessing including intensity normalization, artifact removal and interested region cropping, obtaining a standardized lung cancer patchy image data set;

[0009] S2. Generating lung cancer patchy lesion region annotation information based on the radiologist's manual delineation result, and pairing the lung cancer patchy lesion region annotation information with the standardized lung cancer patchy image data set to form labeled lung cancer patchy image training data;

[0010] S3. Constructing a cavity convolution feature extraction module to perform multi-scale feature extraction on the labeled lung cancer patchy image training data, and generating lung cancer patchy multi-scale dynamic feature maps;

[0011] S4. Constructing a lightweight residual generator network, inputting the lung cancer patchy multi-scale dynamic feature maps into the lightweight residual generator network, and outputting preliminary reconstructed lung cancer patchy image data;

[0012] S5. Constructing a local perception discriminator, the local perception discriminator including a global discrimination branch and a local discrimination branch, the global discrimination branch receiving the preliminary reconstructed lung cancer patchy image data, and the local discrimination branch receiving lung cancer patchy lesion region image blocks cropped according to the lung cancer patchy lesion region annotation information;

[0013] S6. Taking the standardized lung cancer patchy image data set as input, lung cancer patchy lesion region annotation information as region weight, a hollow convolution feature extraction module and a lightweight residual generator network as a generator, and a local perception discriminator as a discriminator, a hollow convolution fusion generative adversarial network model is trained by using a structure perception loss, an edge preservation loss and an adversarial loss, and a trained hollow convolution fusion generative adversarial network model is obtained;

[0014] S7. The newly obtained low-quality lung cancer patchy image data is input into the trained hollow convolution fusion generative adversarial network model, and reconstructed lung cancer patchy image data is output. The reconstructed lung cancer patchy image data is input into an image quality evaluation module, and a lung cancer patchy image quality evaluation result is generated.

[0015] Optionally, the S1 specifically comprises the following steps:

[0016] S11. Collecting lung cancer patchy image data obtained by a computed tomography or magnetic resonance imaging device under clinical low-dose scanning conditions, pairing each original lung cancer patchy image collected with its corresponding basic image metadata, establishing a lung cancer patchy image data set, and each data sample in the lung cancer patchy image data set being composed of an original lung cancer patchy image I i and corresponding basic image metadata m i , the total number of lung cancer patchy image data samples being N;

[0017] S12. Performing intensity normalization processing on each original lung cancer patchy image in the lung cancer patchy image data set, linearly mapping the pixel intensity of each original lung cancer patchy image according to an interval, taking the minimum pixel value and the maximum pixel value of the original lung cancer patchy image as the normalization reference in the normalization process, so that the pixel intensity value of the normalized lung cancer patchy image is within the interval range, and obtaining a lung cancer patchy normalized image data set;

[0018] S13. Performing artifact removal processing on each lung cancer patchy normalized image in the lung cancer patchy normalized image data set, setting an artifact removal intensity threshold by using a global histogram threshold segmentation-based method, marking a pixel point in the lung cancer patchy normalized image as a valid pixel if the pixel intensity of the pixel point is greater than or equal to the artifact removal intensity threshold, otherwise marking the pixel point as an invalid pixel, removing the artifact region in the lung cancer patchy normalized image, and obtaining a lung cancer patchy artifact removal image data set;

[0019] S14. Based on the lung cancer patchy artifact removal image dataset, the corresponding lung region is extracted as the region of interest according to the lung anatomical structure detection result in the image, the region of interest is composed of all pixel coordinates (x, y) belonging to the lung region mask, and the extracted region is defined as the lung cancer patchy region of interest R i , forming a standardized lung cancer patchy image dataset D roi .

[0020] Optionally, the S2 specifically comprises the following steps:

[0021] S21. The radiologists perform manual delineation operation based on each lung cancer patchy region of interest image R i in the standardized lung cancer patchy image dataset i , determine the corresponding lung cancer patchy lesion region contour for each lung cancer patchy region of interest image R i , and generate a lung cancer patchy lesion region mask M i :

[0022]

[0023] Wherein, M i (x, y) represents the lung cancer patchy lesion region mask of the coordinate position (x, y) in the i-th lung cancer patchy region of interest image;

[0024] S22. Each lung cancer patchy region of interest image R i in the standardized lung cancer patchy image dataset is paired with the corresponding lung cancer patchy lesion region mask M i , and a lung cancer patchy image and label information pair (R i , M i ) is constructed;

[0025] S23. According to the lung cancer patchy lesion region mask M i , the area proportion A i of the lesion region in each lung cancer patchy region of interest image is calculated:

[0026]

[0027] Wherein, ∑ (x,y) M i (x, y) represents the number of all pixel points belonging to the lung cancer patchy lesion region in the i-th lung cancer patchy region of interest image, and ∑ (x,y) 1 represents the total number of pixels in the entire lung cancer patchy region of interest image;

[0028] S24. According to the area proportion A iThe statistical distribution of the lung cancer patchy images and annotation information pairs are obtained by filtering out abnormal samples whose area ratio is lower than the set threshold, retaining the lung cancer patchy images and annotation information pairs that meet the requirements, and forming the labeled lung cancer patchy image training dataset D with the retained lung cancer patchy images and annotation information pairs. train .

[0029] Optionally, the S3 specifically includes the following steps:

[0030] S31. Read each pair of lung cancer patchy region of interest image and lung cancer patchy lesion area mask (R i ,M i ), for the patchy region of interest image R of lung cancer i Patchy lesion area of ​​lung cancer mask M i Perform pixel-by-pixel product operations to obtain enhanced images of patchy lesions of lung cancer

[0031] S32. Based on enhanced imaging of patchy lesions in lung cancer Calculate the local texture gradient intensity map G i , local texture gradient intensity map G i The gradient intensity value G of each pixel position (x, y) i (x,y) is defined as the enhanced image of the patchy lung cancer lesion along the transverse direction at that location The square of the rate of change and the enhanced image of lung cancer patchy lesions along the longitudinal direction The result of taking the square root of the sum of the squares of the rates of change;

[0032]

[0033] S33. According to the local texture gradient intensity map G i Dynamically generate void ratio map D i , void ratio graph D i The hole rate value D of each pixel position (x, y) i (x,y) is based on the preset minimum void ratio d min and the maximum void ratio d max The difference multiplied by one minus the local texture gradient intensity value G i (x, y) and the maximum value of the local texture gradient intensity map G max After adding the minimum void ratio d min to obtain;

[0034]

[0035] Among them, d min and d max are the preset minimum void rate and maximum void rate, Gmax denotes the maximum value in the local texture gradient intensity map G i .

[0036] S34. based on the lung cancer patchy lesion enhancement image and the dynamic cavity rate map D i (x, y), for each position (x, y) Adaptive application of the corresponding cavity convolution operation, extract lung cancer patchy multiscale dynamic feature map F i (x, y).

[0037]

[0038] wherein, denotes the local neighborhood sampled by the cavity rate D i (x, y), W (u, v) is the convolution kernel weight, and σ (·) is the activation function.

[0039] Optionally, the S4 specifically includes the following steps:

[0040] S41. The lung cancer patchy multiscale dynamic feature map F i extracted by the dynamic cavity rate mechanism is input to the feature input end of the lightweight residual generator network, and the lightweight residual generator network includes a plurality of stacked lightweight residual units.

[0041] S42. In the kth lightweight residual unit, the lung cancer patchy multiscale dynamic feature map F is subjected to convolution transformation operation, and the convolution kernel weight and the bias term are used in the convolution process, and the convolution transformation operation result is processed through the lightweight nonlinear activation function, to obtain the convolution feature output of the kth lightweight residual unit. The convolution feature output and the input lung cancer patchy multiscale dynamic feature map F are added element by element to generate the output feature map O of the kth lightweight residual unit.

[0042] S43. After being processed by all lightweight residual units, the final feature map O is obtained. The final feature map O is input to the feature reconstruction module, and the feature reconstruction module generates the preliminary reconstructed lung cancer patchy image data I through a layer of standard convolution and Tanh activation function.

[0043] Optionally, in each lightweight residual unit, the convolution transformation operation adopts depth separable convolution instead of standard convolution, and the depth separable convolution includes two-step operation of depth convolution and pointwise convolution.

[0044] Optionally, the S5 specifically includes the following steps:

[0045] S51. Preliminary reconstruction of lung cancer patchy imaging data The corresponding lung cancer patchy lesion area mask M i The local perception discriminator is used to identify the patchy lesion area of ​​lung cancer. i Calculating the distance transformation map U of patchy lung cancer lesions i :

[0046]

[0047] in, Represents the boundary between pixel (x,y) and the patchy lesion area of ​​lung cancer The Euclidean distance of

[0048] S52. Determine the distance transformation map U of the patchy lung cancer lesions i The pixel coordinate with the largest normalized Euclidean distance Centered on pixel coordinates, multi-scale image blocks of lung cancer patchy lesions are cropped according to the scale set.

[0049] S53. In the global discriminant branch, the preliminarily reconstructed lung cancer patchy image data is processed by convolution-global average pooling, and then the global authenticity score is obtained by the fully connected layer and Sigmoid activation function.

[0050] S54. In the multi-scale local discrimination branch, each multi-scale image block of the lung cancer patch is Input convolutional network to obtain local feature map Output scale-local authenticity score through fully connected layer and Sigmoid activation function

[0051] S55. Based on the distance transformation map U of the patchy lesions of lung cancer i Calculate the multi-scale image blocks of each lung cancer patch The scale attention weight The scale attention weight is obtained by averaging the normalized Euclidean distance of all pixels in the patchy lesion area of ​​lung cancer and then normalizing it. The weighted sum of each scale-local authenticity score is calculated according to the scale attention weight to obtain the weighted local authenticity score.

[0052] S56. Calculate the average gradient intensity of the edge gradient map of the patchy lung cancer lesions Get the edge integrity adjustment factor γ i , using the edge integrity adjustment factor to score the global authenticity and weighted local authenticity score Perform linear fusion to obtain a joint authenticity score

[0053]

[0054] Optionally, S6 specifically comprises the following steps:

[0055] S61. inputting each lung cancer patchy region of interest image R roi in the standardized lung cancer patchy image dataset D i as a generator input, lung cancer patchy lesion region mask M i as a region weight reference, into a generator composed of a dilated convolution feature extraction module and a lightweight residual generator network, to generate a preliminary reconstructed lung cancer patchy image data

[0056] S62. inputting the preliminary reconstructed lung cancer patchy image data and the corresponding lung cancer patchy region of interest image R i into a local perception discriminator to obtain a joint authenticity score

[0057] S63. calculating a structure perception loss L struct , the structure perception loss aims to maintain the structural continuity of the lung cancer patchy image, and is defined as the weighted square error of the local gradient difference between the lung cancer patchy region of interest image R i and the preliminary reconstructed lung cancer patchy image data :

[0058]

[0059] wherein, and represent the gradient values of the original image and the reconstructed image at position (x, y), respectively, and U i (x, y) is the lung cancer patchy lesion distance transform map weight;

[0060] S64. calculating an edge preservation loss L edge , the edge preservation loss aims to strengthen the boundary clarity of the lung cancer patchy lesion region, and is defined as the edge detection error under the action of the lung cancer patchy lesion region mask M i :

[0061]

[0062] wherein, E i (x, y) and represent the edge gradient values of the original image and the preliminary reconstructed image at position (x, y), respectively;

[0063] S65. calculating an adversarial loss Ladv , the adversarial loss aims to enhance the realism of lung cancer patchy shadows, defined as the negative log loss of the joint realism score ;

[0064] S66. The total loss function L is constructed by combining the structural perception loss, the edge preservation loss and the adversarial loss total :

[0065] L total = λ struct L struct + λ edge L edge + λ adv L adv ,

[0066] Wherein, λ struct , λ edge , λ adv Respectively, the weighting coefficient of each loss term;

[0067] S67. The total loss function L total is used as the optimization target, and the parameters of the dilated convolution feature extraction module, the lightweight residual generator network and the local perception discriminator are jointly optimized until the training converges, and the trained dilated convolution fusion generative adversarial network model is obtained.

[0068] The beneficial effects of the present application are:

[0069] (1) The present application adopts a dynamic dilated rate regulated dilated convolution feature extraction module, which dynamically generates a dilated rate map according to the local texture gradient intensity map of the lung cancer patchy lesion area, and applies dilated convolution operation based on the dilated rate of each pixel position, realizes multi-scale dynamic feature extraction, can adaptively adjust the receptive field size according to the texture complexity of different positions of the lesion area, applies small dilated rate to finely extract features in the detailed rich area, and applies large dilated rate to enhance the context information fusion in the simple structure area, effectively avoiding the problems of detail loss or lack of context information.

[0070] (2) The present application constructs a lightweight residual generator network, which replaces the traditional standard convolution with depth separable convolution in the feature reconstruction process, and introduces a stacked lightweight residual unit, which effectively reduces the model parameter quantity and calculation complexity, while maintaining efficient information flow and feature fusion capability, the lightweight residual generator can more fully retain local detail information and overall structural continuity when processing lung cancer patchy multi-scale dynamic feature map, avoiding the phenomenon of over-smoothing or artifact diffusion of the reconstructed image.

[0071] (3) The application introduces a local perception discriminator and combines a scale attention weighting mechanism, calculates a distance transform graph on a lung cancer patch lesion area, extracts multi-scale local image blocks, and dynamically allocates weights according to the importance of each scale local area, to finely discriminate the local authenticity of the generated image, which can effectively evaluate the authenticity of the lesion edge and internal structure at a fine-grained level, and further improve the authenticity and credibility of the generated image in the key lesion area. BRIEF DESCRIPTION OF DRAWINGS

[0072] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings:

[0073] Figure 1 A flowchart of a lung cancer image enhancement method based on a generative adversarial network is provided. DETAILED DESCRIPTION

[0074] The application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams that show only the basic structure of the application in a schematic manner, and thus only show the components relevant to the application.

[0075] REFERENCE Figure 1 A lung cancer image enhancement method based on a generative adversarial network includes the following steps:

[0076] S1. Collecting lung cancer patch image data and establishing a lung cancer patch image data set, pre-processing the lung cancer patch image data set, the pre-processing including intensity normalization, artifact removal and interested region cropping, obtaining a standardized lung cancer patch image data set;

[0077] S2. Generating lung cancer patch lesion region annotation information based on radiologist manual delineation results, and pairing the lung cancer patch lesion region annotation information with the standardized lung cancer patch image data set to form labeled lung cancer patch image training data;

[0078] S3. Constructing a hollow convolution feature extraction module to perform multi-scale feature extraction on the labeled lung cancer patch image training data, generating lung cancer patch multi-scale dynamic feature maps;

[0079] S4. Constructing a lightweight residual generator network, inputting the lung cancer patch multi-scale dynamic feature maps into the lightweight residual generator network, and outputting preliminary reconstructed lung cancer patch image data;

[0080] S5. Constructing a local perception discriminator, the local perception discriminator comprising a global discrimination branch and a local discrimination branch, the global discrimination branch receiving the preliminary reconstructed lung cancer patch image data, and the local discrimination branch receiving the lung cancer patch lesion region image block cropped according to the lung cancer patch lesion region annotation information;

[0081] S6. Taking the standardized lung cancer patch image data set as input, the lung cancer patch lesion region annotation information as region weight, the hollow convolution feature extraction module and the lightweight residual generator network as the generator, and the local perception discriminator as the discriminator, training the hollow convolution fusion generative adversarial network model by using the structure perception loss, the edge preservation loss and the adversarial loss, and obtaining the trained hollow convolution fusion generative adversarial network model;

[0082] S7. Inputting the newly obtained low-quality lung cancer patch image data into the trained hollow convolution fusion generative adversarial network model, outputting the reconstructed lung cancer patch image data, inputting the reconstructed lung cancer patch image data into the image quality evaluation module, and generating the lung cancer patch image quality evaluation result.

[0083] In the embodiment, S1 specifically comprises the following steps:

[0084] S11. Collecting lung cancer patch image data acquired by a computed tomography or magnetic resonance imaging device under clinical low-dose scanning conditions, pairing each collected original lung cancer patch image with its corresponding basic image metadata, establishing a lung cancer patch image data set, each data sample in the lung cancer patch image data set consisting of an original lung cancer patch image I i and corresponding basic image metadata m i , and the total number of lung cancer patch image data samples being N;

[0085] S12. Performing intensity normalization processing on each original lung cancer patch image in the lung cancer patch image data set, linearly mapping the pixel intensity of each original lung cancer patch image according to the interval, taking the minimum pixel value and the maximum pixel value of the original lung cancer patch image as the normalization reference in the normalization process, so that the pixel intensity value of the normalized lung cancer patch image is within the interval range, and obtaining a lung cancer patch normalized image data set;

[0086] S13. The artifact removal processing is performed on each lung cancer patch normalized image in the lung cancer patch normalized image dataset, a global histogram threshold segmentation-based method is adopted, an artifact removal intensity threshold is set, if the pixel intensity of a pixel point in the lung cancer patch normalized image is greater than or equal to the artifact removal intensity threshold, the pixel point is marked as a valid pixel; otherwise, the pixel point is marked as an invalid pixel, the artifact area in the lung cancer patch normalized image is removed, and a lung cancer patch artifact removal image dataset is obtained;

[0087] S14. Based on the lung cancer patch artifact removal image dataset, the corresponding lung region is extracted as a region of interest according to the lung anatomical structure detection result in the image, the region of interest is composed of all pixel coordinates (x, y) belonging to the lung region mask, and the extracted region is defined as a lung cancer patch region of interest R i , and a standardized lung cancer patch image dataset D roi is formed.

[0088] In this embodiment, S2 specifically includes the following steps:

[0089] S21. The radiologist performs a manual delineation operation based on each lung cancer patch region of interest image R i in the standardized lung cancer patch image dataset i , determines the corresponding lung cancer patch lesion region contour for each lung cancer patch region of interest image R i , and generates a lung cancer patch lesion region mask M i :

[0090]

[0091] wherein M i (x, y) represents the lung cancer patch lesion region mask at the coordinate position (x, y) in the i-th lung cancer patch region of interest image;

[0092] S22. Each lung cancer patch region of interest image R i in the standardized lung cancer patch image dataset is paired with the corresponding lung cancer patch lesion region mask M i to construct a lung cancer patch image and annotation information pair (R i , M i ).

[0093] S23. The area proportion A i of the lesion region in each lung cancer patch region of interest image is calculated according to the lung cancer patch lesion region mask M i :

[0094]

[0095] Among them, ∑ (x,y) M i (x,y) represents the number of all pixels belonging to the patchy lesion area of ​​lung cancer in the i-th patchy region of interest image of lung cancer, ∑ (x,y) 1 represents the total number of pixels in the entire lung cancer patchy region of interest image;

[0096] S24. According to the area ratio A i The statistical distribution of the lung cancer patchy images and annotation information pairs are obtained by filtering out abnormal samples whose area ratio is lower than the set threshold, retaining the lung cancer patchy images and annotation information pairs that meet the requirements, and forming the labeled lung cancer patchy image training dataset D with the retained lung cancer patchy images and annotation information pairs. train .

[0097] In this embodiment, S3 specifically includes the following steps:

[0098] S31. Read each pair of lung cancer patchy region of interest image and lung cancer patchy lesion area mask (R i ,M i ), for the patchy region of interest image R of lung cancer i Patchy lesion area of ​​lung cancer mask M i Perform pixel-by-pixel product operations to obtain enhanced images of patchy lesions of lung cancer

[0099] S32. Based on enhanced imaging of patchy lesions in lung cancer Calculate the local texture gradient intensity map G i , local texture gradient intensity map G i The gradient intensity value G of each pixel position (x, y) i (x,y) is defined as the enhanced image of the patchy lung cancer lesion along the transverse direction at that location The square of the rate of change and the enhanced image of lung cancer patchy lesions along the longitudinal direction The result of taking the square root of the sum of the squares of the rates of change;

[0100]

[0101] S33. According to the local texture gradient intensity map G i Dynamically generate void ratio map D i , void ratio graph D i The hole rate value D of each pixel position (x, y) i (x,y) is based on the preset minimum void ratio d min and the maximum void ratio d max The difference multiplied by one minus the local texture gradient intensity value G i (x, y) and the maximum value of the local texture gradient intensity map Gmax After adding the minimum void ratio d min to obtain;

[0102]

[0103] Among them, d min and d max are the preset minimum void rate and maximum void rate, G max Represents the local texture gradient intensity map G i The maximum value in ;

[0104] S34. Based on enhanced imaging of patchy lesions in lung cancer and dynamic void ratio diagram D i (x, y), adaptively apply the corresponding dilated convolution operation to each position (x, y) to extract the patchy multi-scale dynamic feature map F of lung cancer i (x,y);

[0105]

[0106] in, Indicated by the void ratio D i The local neighborhood of (x,y) is sampled, W(u,v) is the convolution kernel weight, and σ(·) is the activation function.

[0107] In this embodiment, S4 specifically includes the following steps:

[0108] S41. The multi-scale dynamic feature map of lung cancer patchy shape extracted by dynamic void ratio mechanism F i Input to the feature input end of the lightweight residual generator network, the lightweight residual generator network includes multiple stacked lightweight residual units;

[0109] S42. Multi-scale dynamic feature map of lung cancer patchiness in the kth lightweight residual unit Perform convolution transformation operation, use convolution kernel weight and bias term in the convolution process, process the convolution transformation operation result through lightweight nonlinear activation function, obtain the convolution feature output of the kth lightweight residual unit, and compare the convolution feature output with the input lung cancer patchy multi-scale dynamic feature map Add element-wise to generate the output feature map of the kth lightweight residual unit

[0110] S43. After all lightweight residual units are processed, the final feature map is obtained The final feature map The input is sent to the feature reconstruction module, which generates preliminary reconstruction of lung cancer patchy image data through a layer of standard convolution and Tanh activation function.

[0111] In this embodiment, in each lightweight residual unit, the convolutional transformation operation adopts a depthwise separable convolution instead of a standard convolution, which includes two-step operations of depthwise convolution and pointwise convolution.

[0112] In this embodiment, S5 specifically includes the following steps:

[0113] S51. Reconstructing the preliminary lung cancer patchy image data with the corresponding lung cancer patchy lesion region mask M i Synchronously input into the local perception discriminator input end, and pass the lung cancer patchy lesion region mask M through the local perception discriminator i Calculate the lung cancer patchy lesion distance transform graph U i :

[0114]

[0115] wherein, represents the Euclidean distance from the pixel (x, y) to the boundary of the lung cancer patchy lesion region;

[0116] S52. Determine the lung cancer patchy lesion distance transform graph U i with the maximum normalized Euclidean distance of the pixel coordinates Centered on the pixel coordinates, crop the lung cancer patchy lesion multi-scale image block according to the scale set

[0117] S53. After the convolution-global average pooling processing of the preliminary reconstructed lung cancer patchy image data in the global discrimination branch, obtain the global authenticity score through the fully connected layer and the Sigmoid activation function

[0118] S54. In the multi-scale local discrimination branch, respectively input each lung cancer patchy lesion multi-scale image block into the convolutional network to obtain the local feature map Output the scale-local authenticity score through the fully connected layer and the Sigmoid activation function

[0119] S55. According to the lung cancer patchy lesion distance transform graph U i Calculate the scale attention weight of each lung cancer patchy lesion multi-scale image block The scale attention weight is obtained by averaging and normalizing the normalized Euclidean distances of all pixels in the lung cancer patchy lesion region, and the weighted local authenticity score is obtained by weighting and summing the scale-local authenticity scores according to the scale attention weight ​​

[0120] S56. Calculate the average gradient intensity of the lung cancer patch lesion edge gradient map Get the edge integrity adjustment factor γ i , use the edge integrity adjustment factor to score the global authenticity And the weighted local authenticity score Linearly fused to get the joint authenticity score

[0121]

[0122] In this embodiment, S6 specifically includes the following steps:

[0123] S61. Each lung cancer patch region of interest image R roi in the standardized lung cancer patch image dataset D i is input as a generator, and the lung cancer patch lesion region mask M i is input as a region weight reference to the generator composed of the hollow convolution feature extraction module and the lightweight residual generator network, to generate the preliminary reconstructed lung cancer patch image data

[0124] S62. The preliminary reconstructed lung cancer patch image data and the corresponding lung cancer patch region of interest image R i are input into the local perception discriminator to obtain the joint authenticity score

[0125] S63. Calculate the structure perception loss L struct , which aims to maintain the structure continuity of the lung cancer patch image, and is defined as the weighted square error of the local gradient difference between the lung cancer patch region of interest image R i and the preliminary reconstructed lung cancer patch image data :

[0126]

[0127] where, and represent the gradient values of the original image and the reconstructed image at position (x, y), respectively, and U i (x, y) is the lung cancer patch lesion distance transform map weight.

[0128] S64. Calculate the edge preservation loss L edge , which aims to enhance the clarity of the lung cancer patch lesion region boundary, and is defined as the edge detection error under the lung cancer patch lesion region mask M i :

[0129]

[0130] wherein, E i (x, y) and respectively represent the edge gradient values of the original image and the preliminary reconstructed image at position (x, y);

[0131] S65. Calculate the adversarial loss L adv The adversarial loss aims to improve the realism of the lung cancer patchy image, and is defined as the negative logarithmic loss of the joint authenticity score .

[0132] S66. Synthesize the structural perception loss, edge preservation loss and adversarial loss to construct the total loss function L total :

[0133] L total = λ struct L struct + λ edge L edge + λ adv L adv ,

[0134] wherein, λ struct , λ edge , λ adv are the weighting coefficients of each loss term, respectively.

[0135] S67. Take the total loss function L total as the optimization target, jointly optimize the parameters of the dilated convolution feature extraction module, the lightweight residual generator network and the local perception discriminator, until the training converges, to obtain the trained dilated convolution fusion generative adversarial network model.

[0136] In the embodiment, the S7 specifically comprises the following steps:

[0137] S71. Obtain newly obtained low-quality lung cancer patchy image data Input the low-quality lung cancer patchy image data into the trained dilated convolution fusion generative adversarial network model, and output reconstructed lung cancer patchy image data

[0138] S72. Input the reconstructed lung cancer patchy image data into the image quality evaluation module, and extract the following four image quality indicators:

[0139] The structural clarity indicator is used to measure the integrity of the tissue structure of the lung cancer patchy lesion area.

[0140] The edge sharpness indicator is used to measure the clarity of the lung cancer patchy lesion boundary.

[0141] texture fidelity index, used to measure the level of texture detail recovery in the lung cancer patchy lesion area;

[0142] overall visual consistency index, used to measure the overall perceptual consistency between the reconstructed image and the original image;

[0143] S73. Extract classification criteria from historical lung cancer patchy image reconstruction evaluation results, and define high-quality, medium-quality and low-quality image sample classification rules according to the structure definition index, edge sharpness index, texture fidelity index and overall visual consistency index, wherein:

[0144] High-quality sample: structure definition index Q struct,j ≥T struct , edge sharpness index Q edge,j ≥T edge , texture fidelity index Q texture,j ≥T texture , overall visual consistency index Q vis,j ≥T vis ;

[0145] Medium-quality sample: meet the evaluation conditions of any three indicators;

[0146] Low-quality sample: less than three indicators meet the evaluation conditions;

[0147] Wherein, T struct , T edge , T texture , T vis are the preset structure definition, edge sharpness, texture fidelity and overall visual consistency evaluation thresholds, respectively;

[0148] S74. Output the lung cancer patchy image quality evaluation results, and store and record them according to the classification of high-quality samples, medium-quality samples and low-quality samples for subsequent clinical application screening and model performance dynamic optimization

[0149] Example 1:

[0150] In August 2024, the lung cancer early screening special project group in the imaging center of A hospital launched a new technical verification research on the enhancement of lung cancer patchy images under low-dose CT scanning conditions. The project aims to solve the problems of multiple artifacts, blurred lesion boundaries and missing details in current low-dose lung images, improve the visualization effect of lung cancer patchy lesion area, and assist doctors in more accurately identifying early micro lesions.

[0151] In the actual test, the project team selected 527 low-dose chest CT scan data collected from May 2023 to March 2024, the patients were aged 45 to 76 years old, all were high-risk lung cancer screening objects, the image data acquisition device model was Siemens SOMATOM Force, the scanning condition was 120kV, the tube current was set to 25-30mAs, the layer thickness was 1.0mm, and the standard low-dose protocol was used, resulting in large noise and insufficient contrast of the image itself. After preliminary screening, a total of 482 image data met the requirements of patchy lesion characteristics and were included in the study.

[0152] The project team first standardized the original image data. Specifically, intensity normalization was performed on all images to unify the gray scale distribution to the [0, 1] interval. At the same time, a global histogram threshold segmentation method was applied to remove large-area metal artifacts and motion artifact regions. After processing, a standardized lung cancer patchy image data set was formed, and a ROI (Region of Interest) mask was extracted based on the lung anatomy region in each image, retaining only the lung parenchyma region for subsequent analysis.

[0153] In order to obtain reliable lesion region annotation information, the project team invited 5 radiologists with more than 10 years of clinical experience to draw each ROI image frame by frame, draw lesion masks, and label the work from August to September 2024. A total of 482 pairs of labeled data were generated, each image containing detailed lesion contour information, including patchy irregular density shadows, ground glass nodules, and small solid nodules.

[0154] Next, the DR-GAN (Dilated Convolution Fusion Generative Adversarial Network) proposed in the present application was used for image enhancement. First, based on the lesion mask and the original image, a local texture gradient intensity map was calculated to dynamically generate a dilated rate map, which was used to guide the dilated convolution feature extraction module. Compared with the traditional method of using a fixed dilated rate (such as a fixed value of 2, 4 or 8) to extract features, the present application adjusts the dilated rate in real time according to the texture changes in the lesion area, applies a smaller dilated rate (dmin=2) in areas with fine texture, and applies a larger dilated rate (dmax=6) in areas with single texture, effectively maintaining the richness of details.

[0155] After dynamic feature extraction, the multi-scale dynamic feature map is input into a lightweight residual generator. The lightweight residual network uses depthwise separable convolution, which greatly reduces the parameter quantity. When training the entire model on a Tesla V100 GPU, the single batch memory occupation is reduced from 3.8 GB of the traditional method to 2.2 GB, and the training speed is increased by about 1.7 times. The training process uses the Adam optimizer, the initial learning rate is set to 1e-4, the batch size is 16, and the total training period is 120 epochs.

[0156] In order to further improve the local authenticity of the generated image, the project team constructed a local perception discriminator. Unlike traditional discriminators, the discriminator of the application calculates a distance transform map based on a lesion mask, automatically extracts local image blocks of different scales, scores the authenticity of local details, and integrates the scores based on scale attention weights, thereby significantly enhancing the attention to the authenticity of the lesion boundary.

[0157] After complete training, the project team used the model to perform image enhancement processing on 50 low-dose CT images collected in October 2024 for lung cancer screening, and compared the results with those obtained by using a traditional GAN method based on a U-Net structure (referred to as Baseline-GAN). The specific comparison is as follows:

[0158] Table 1 Comparison of the application with the traditional GAN method based on a U-Net structure

[0159] Metrics Baseline-GAN The present invention (DR-GAN) Average PSNR (dB) 27.6 30.2 Average SSIM 0.842 0.893 Lesion region Dice coefficient 0.778 0.846 Detail texture retention rate (custom texture score) 74.5％ 85.3％ Inference speed (fps) 6.2 11.1

[0160] As can be seen from the data, the PSNR of the enhanced lung cancer patch image of the application is improved by 2.6 dB, indicating that the overall signal-to-noise ratio of the image is significantly improved. The SSIM is improved by 5.1%, indicating that the structural information is better preserved. The Dice coefficient of the lesion area is improved by 8.7%, indicating that the reconstructed image is more conducive to lesion identification. In addition, by introducing dynamic dilated convolution and lightweight residual design, the inference speed is increased by nearly 79%, meeting the requirements of actual clinical rapid auxiliary diagnosis.

[0161] In the physician qualitative evaluation link, the project team organized 3 chief physicians and 2 deputy chief physicians to conduct a blind evaluation experiment. Each physician needed to score the image clarity and lesion recognizability generated by Baseline-GAN and the application without prior notice. The final statistics showed that the average score of Baseline-GAN was 3.2 / 5, while the average score of the application method was 4.5 / 5. It was unanimously believed that the application was superior to the traditional method in the enhancement effect of micro lesions and fuzzy boundaries.

[0162] In addition, in the test set, the method of the application successfully improves the sharpness of the lesion edge by an average of 16.4% for patchy lesions with a size less than 5mm, and significantly improves the problem of serious detail loss in small lesions when the traditional method is used.

[0163] In summary, the embodiment fully verifies that the method of the application can effectively solve the problems of lesion detail loss, boundary blurring and poor enhancement quality in low-dose lung cancer patchy images, and has good feasibility and significant practical application value through specific application in real medical scenarios and detailed data comparison.

[0164] The application adopts a dynamic hole rate regulated hole convolution feature extraction module, dynamically generates a hole rate map according to a local texture gradient intensity map of a lung cancer patchy lesion area, and adaptively applies a hole convolution operation based on the hole rate of each pixel position, realizes multi-scale dynamic feature extraction, adaptively adjusts the receptive field size according to the texture complexity of different positions of the lesion area, applies a small hole rate to finely extract features in a detailed rich area, and applies a large hole rate to enhance context information fusion in a simple structure area, effectively avoiding the problems of detail loss or insufficient context information.

[0165] The application constructs a lightweight residual generator network, adopts a depth separable convolution instead of a traditional standard convolution in the feature reconstruction process, and introduces a stacked lightweight residual unit, effectively reducing the model parameter quantity and the calculation complexity, while maintaining the efficient information flow and feature fusion capability, and the lightweight residual generator can more fully retain the local detail information and the overall structure continuity when processing the lung cancer patchy multi-scale dynamic feature map, avoiding the phenomenon of over-smoothing or artifact diffusion of the reconstructed image.

[0166] The application introduces a local perception discriminator, and combines a scale attention weighting mechanism, calculates a distance transform map for the lung cancer patchy lesion area, extracts multi-scale local image blocks, and dynamically allocates weights according to the importance of each scale local area, finely discriminates the local authenticity of the generated image, can effectively evaluate the authenticity of the lesion edge and the internal structure at a fine-grained level, and further improves the authenticity and credibility of the generated image in the key lesion area.

[0167] The above is only the preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can make equivalent replacement or change according to the technical scheme and the inventive concept of the application within the technical range disclosed by the application, which should be covered within the protection scope of the application.

Claims

1.A lung cancer image enhancement method based on a generative adversarial network, characterized in that, Comprising the following steps: S1. Collecting lung cancer patchy image data and establishing a lung cancer patchy image data set, preprocessing the lung cancer patchy image data set to obtain a standardized lung cancer patchy image data set; S2. Generating lung cancer patchy lesion region annotation information based on the radiologist's manual delineation result, and pairing the lung cancer patchy lesion region annotation information with the standardized lung cancer patchy image data set to form labeled lung cancer patchy image training data; S3. Constructing a hollow convolution feature extraction module to perform multi-scale feature extraction on the labeled lung cancer patchy image training data to generate lung cancer patchy multi-scale dynamic feature maps; The S3 specifically comprises the following steps: S31. Read each pair of lung cancer patch-shaped region of interest image and lung cancer patch-shaped lesion region mask (R i , M i ) in the labeled lung cancer patch-shaped image training data set, and perform pixel-by-pixel multiplication operation on the lung cancer patch-shaped region of interest image R i and the lung cancer patch-shaped lesion region mask M i to obtain a lung cancer patch-shaped lesion enhanced image S32. Based on enhanced imaging of patchy lesions in lung cancer Calculate the local texture gradient intensity map G i , local texture gradient intensity map G i The gradient intensity value G of each pixel position (x, y) i (x,y) is defined as the enhanced image of the patchy lung cancer lesion along the transverse direction at that location The square of the rate of change and the enhanced image of lung cancer patchy lesions along the longitudinal direction The result of taking the square root of the sum of the squares of the rates of change; S33. According to the local texture gradient intensity map G i Dynamically generating a void rate map D i , the void rate map D i The void rate value D of each pixel position (x, y) in the image i (x, y) is obtained according to the difference between the preset minimum void rate d min And the maximum void rate d max Multiplied by a minus ratio of the local texture gradient intensity value G i (x, y) and the maximum value of the local texture gradient intensity map G max , and then plus the minimum void rate d min ; S34. lung cancer patchy lesion enhancement image and dynamic cavity rate map D i (x, y), for each position (x, y) adaptively apply the corresponding hole convolution operation, extract lung cancer patchy multi-scale dynamic feature map F i (x, y); S4. Constructing a lightweight residual generator network, inputting the lung cancer patchy multi-scale dynamic feature maps into the lightweight residual generator network, and outputting preliminary reconstructed lung cancer patchy image data; S5. Constructing a local perception discriminator; The S5 specifically comprises the following steps: S51. Reconstructing the preliminary lung cancer patchy image data with the corresponding lung cancer patchy lesion region mask M i synchronously into the local perception discriminator input end, and passing the lung cancer patchy lesion region mask M through the local perception discriminator i calculating the lung cancer patchy lesion distance transform graph U i : wherein, represents the Euclidean distance of pixel (x, y) to the lung cancer patch lesion region boundary ; S52. Determine the lung cancer patch lesion distance transform map U i The pixel coordinate with the largest normalized Euclidean distance Crop the lung cancer patch lesion multi-scale image block centered at the pixel coordinate according to the scale set S53. After the convolution-global average pooling processing of the preliminary reconstructed lung cancer patchy shadow image data in the global discriminant branch, the global authenticity score is obtained through the full connection layer and the Sigmoid activation function S54. In the multi-scale local discriminant branch, respectively, each lung cancer patchy lesion multi-scale image block is input into the multi-scale local discriminant branch An input convolutional network is used to obtain a local feature map A full connection layer and a Sigmoid activation function are used to output a scale-local authenticity score S55. The lung cancer patch lesion distance transform map U is obtained according to the lung cancer patch lesion distance transform map U i Calculate the multi-scale image block of each lung cancer patch lesion The scale attention weight The scale attention weight is obtained by normalizing and averaging the normalized Euclidean distance of all pixels in the lung cancer patch lesion area, and the weighted local authenticity score is obtained by weighted sum of the scale-local authenticity score according to the scale attention weight S56. Obtaining the average gradient intensity of the lung cancer patchy lesion edge gradient map by calculation obtaining an edge integrity adjustment factor γ i using the edge integrity adjustment factor on the global authenticity score and the weighted local authenticity score performing linear fusion to obtain a joint authenticity score S6. Taking the standardized lung cancer patchy image data set as input, the lung cancer patchy lesion region annotation information as region weight, the hollow convolution feature extraction module and the lightweight residual generator network as generator, and the local perception discriminator as discriminator to obtain a trained hollow convolution fusion generative adversarial network model; S7. Inputting newly obtained low-quality lung cancer patchy image data into the trained hollow convolution fusion generative adversarial network model to output reconstructed lung cancer patchy image data and generate lung cancer patchy image quality evaluation results. 2.The lung cancer image enhancement method based on a generative adversarial network according to claim 1, characterized in that, The S1 specifically comprises the following steps: S11. Collecting lung cancer patchy image data obtained by a computed tomography or magnetic resonance imaging device under clinical low-dose scanning conditions, pairing each original lung cancer patchy image with its corresponding basic image metadata to establish a lung cancer patchy image data set; S12. Performing intensity normalization processing on each original lung cancer patchy image in the lung cancer patchy image data set to obtain a lung cancer patchy normalized image data set; S13. Performing artifact removal processing on each lung cancer patchy normalized image in the lung cancer patchy normalized image data set, setting an artifact removal intensity threshold using a global histogram threshold segmentation-based method, and if the pixel intensity of a pixel point in the lung cancer patchy normalized image is greater than or equal to the artifact removal intensity threshold, marking the pixel point as a valid pixel; otherwise, marking the pixel point as an invalid pixel, removing the artifact region in the lung cancer patchy normalized image, and obtaining a lung cancer patchy artifact-removed image data set; S14. Based on the lung cancer patchy artifact, the image dataset is removed, and the corresponding lung region is extracted as the region of interest according to the detection result of the lung anatomical structure in the image. The region of interest is composed of all pixel coordinates (x, y) belonging to the lung region mask. The extracted region is defined as the lung cancer patchy region of interest R i , forming a standardized lung cancer patchy image dataset D roi . 3.The lung cancer image enhancement method based on a generative adversarial network according to claim 2, characterized in that, The S2 specifically comprises the following steps: S21. determining, by the radiologist, a corresponding lung cancer patchy lesion region contour based on each lung cancer patchy region of interest image R i S21. determining, by the radiologist, a corresponding lung cancer patchy lesion region contour based on each lung cancer patchy region of interest image R i S21. determining, by the radiologist, a corresponding lung cancer patchy lesion region contour based on each lung cancer patchy region of interest image R i : wherein M i (x,y) represents the lung cancer patch lesion region mask of the coordinate position (x,y) in the i-th lung cancer patch region of interest image. S22. Each lung cancer patchy region of interest image R in the standardized lung cancer patchy image dataset i The corresponding lung cancer patchy lesion area mask M i Pairing is performed to construct a pair of lung cancer patchy images and annotation information (R i ,M i ); S23. Depending on the lung cancer patchy lesion region mask M i Calculate the area proportion A of the lesion region in each lung cancer patchy region of interest image i : wherein∑ (x,y) M i (x, y) represents the number of all pixel points belonging to the lung cancer patch lesion region in the i-th lung cancer patch region of interest image,∑ (x,y) 1 represents the total number of pixels of the whole lung cancer patch region of interest image; S24. According to the statistical distribution of area proportion A i , abnormal samples with area proportion lower than the set threshold are screened out, the required lung cancer patchy image and label information pairs are retained, and the retained lung cancer patchy image and label information pairs are combined to form a labeled lung cancer patchy image training dataset D train . 4.The lung cancer image enhancement method based on a generative adversarial network of claim 1, wherein, The S4 specifically comprises the following steps: S41. Inputting the lung cancer patchy multi-scale dynamic feature maps extracted by the dynamic hole rate mechanism into the feature input end of the lightweight residual generator network, and the lightweight residual generator network comprises a plurality of stacked lightweight residual units; S42. In the kth lightweight residual unit, the lung cancer patchy multiscale dynamic feature map is processed The convolution transformation operation is performed, the convolution kernel weight and the bias term are used in the convolution process, the result of the convolution transformation operation is processed through the lightweight nonlinear activation function, and the convolution feature output of the kth lightweight residual unit is obtained. The lung cancer patchy multiscale dynamic feature map is inputted into the kth lightweight residual unit The output feature maps of the kth lightweight residual unit are generated by element-wise addition S43. After all the lightweight residual unit processing, the final feature map is obtained The final feature map is input to the feature reconstruction module, and the feature reconstruction module generates a preliminary reconstructed lung cancer patchy shadow image data through one layer of standard convolution and Tanh activation function 5.The lung cancer image enhancement method based on a generative adversarial network according to claim 4, characterized in that, In each lightweight residual unit, the convolution transformation operation adopts a depth separable convolution instead of a standard convolution, and the depth separable convolution includes two-step operations of depth convolution and point-by-point convolution. 6.The lung cancer image enhancement method based on a generative adversarial network of claim 1, wherein, The S6 specifically comprises the following steps: S61. Each lung cancer patchy region of interest image R in the standardized lung cancer patchy image dataset D roi i The lung cancer patchy lesion region mask M as the generator input i As a region weight reference, input into the generator composed of the cavity convolution feature extraction module and the lightweight residual generator network to generate the preliminary reconstructed lung cancer patchy image data ​ S62. Reconstructing the preliminary lung cancer patchy image data With the corresponding lung cancer patchy region of interest image R i Input the local perception discriminator to obtain the joint authenticity score S63. Compute structure-aware loss L struct The structure-aware loss aims to preserve the structure continuity of lung cancer patchy image, and is defined as the weighted square error of local gradient difference between the lung cancer patchy image R i and the preliminary reconstructed lung cancer patchy image data ​ wherein, and respectively represent the gradient value of the original image and the reconstructed image at position (x, y), U i (x, y) is the weight of the lung cancer patch lesion distance transform map. S64. Calculate the edge preservation loss L edge The edge preservation loss aims to enhance the boundary clarity of the lung cancer patch lesion region, and is defined as the edge detection error of the lung cancer patch lesion region mask M i under the action of the edge detection function where E i (x,y) and respectively represent the edge gradient values of the original image and the preliminary reconstructed image at position (x,y). S65. Compute adversarial loss L adv The adversarial loss aims to improve the realism of lung cancer patchy shadows, defined as the negative log loss of the joint realism score . S66. The total loss function L is constructed by integrating the structure-aware loss, the edge-preserving loss, and the adversarial loss. total : L total = λ struct L struct + λ edge L edge + λ adv L adv , wherein, λ struct , λ edge , λ adv are the weighting coefficients of each loss term, respectively. S67. with a total loss function L total As an optimization objective, the parameters of the cavity convolution feature extraction module, the lightweight residual generator network and the local perception discriminator are jointly optimized until the training converges, and a trained cavity convolution fusion generative adversarial network model is obtained.

Images