Image video super-resolution enhancement method based on degradation generative adversarial network
By using a method based on degradation generative adversarial networks, a multi-scale degradation kernel generation module and a dynamic upsampling mechanism are constructed, which solves the problem of poor image processing effect in existing technologies, achieves high-quality super-resolution image reconstruction, and improves the fidelity of key structures and texture authenticity.
Patent Information
- Application Number
- CN202510790225.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-13
- Publication Date
- 2025-10-03
- Estimated Expiration
- 2045-06-13
AI Technical Summary
Existing super-resolution methods cannot accurately simulate the complex degradation patterns of images under different devices and tissues, lack the fidelity and edge clarity of key anatomical structures, and are difficult to meet clinical needs.
A method based on degradation generative adversarial network is adopted. By constructing a multi-scale degradation kernel generation module and dynamic convolution, the local gradient distribution and frequency domain characteristics of CT images are combined to generate degradation kernels related to the image content. A degradation-aware residual correction generator is used for dynamic upsampling, and a multi-scale gradient discriminator is constructed for image enhancement.
It significantly improves the detail restoration capability and structural fidelity of super-resolution images, enhances the image texture authenticity and key tissue density consistency, and achieves high-quality reconstruction of images under different devices and tissues.
Smart Images

Figure CN120746830A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to an image and video super-resolution enhancement method based on a degenerate generative adversarial network. Background Art
[0002] With the rapid development of imaging technology, the role of image processing technology has become increasingly important. However, due to factors such as imaging equipment resolution, radiation dose control, and scanning speed, the acquired images often have problems such as insufficient resolution, blurred details, and noise interference. Especially in critical areas, low-resolution images may lead to unclear tissue structure and reduced judgment accuracy. Therefore, how to perform high-quality super-resolution reconstruction of images without changing the original acquisition conditions has become an important problem that needs to be solved in the current image processing field. Most existing super-resolution methods are based on fixed degradation models or direct interpolation strategies, lack the ability to model the real imaging degradation process, and perform insufficiently in terms of the fidelity and edge clarity of key anatomical structures, making it difficult to meet the high clinical requirements for structural consistency and density restoration. To this end, it is urgent to propose an image super-resolution enhancement method that can adaptively simulate the actual degradation mechanism, enhance the ability to restore key structures, and take into account both image clarity and diagnostic value.
[0003] Existing technologies, such as patents with publication numbers CN114627009B, CN119228750B, and CN110136217B, have problems such as the inability to accurately simulate the complex degradation patterns of images under different devices and tissues, blurred details and distortion of key anatomical structures, lack of multi-level modeling capabilities for image edges and texture details, and easy falling into local optimality or poor convergence. Summary of the Invention
[0004] In order to solve the technical problem of poor image processing effect in the prior art, the present invention provides an image and video super-resolution enhancement method based on a degenerate generative adversarial network.
[0005] The present invention is achieved through the following technical solutions:
[0006] A method for image and video super-resolution enhancement based on a degenerate generative adversarial network, comprising:
[0007] Image acquisition: collect CT images and video sequences as high-resolution raw sample data;
[0008] Image and video preprocessing; including image standardization, smoothing filtering and local motion compensation between video frames;
[0009] Super-resolution enhancement model construction and training; super-resolution enhancement of CT images based on the degradation generative adversarial network model;
[0010] Perform super-resolution enhancement of images; use the trained degraded generative adversarial network model for super-resolution enhancement of CT images or videos.
[0011] Furthermore, the construction and training process of the degradation generative adversarial network model includes constructing a multi-scale degradation kernel generation module. Specifically, a multi-scale degradation kernel generator based on dynamic convolution is used to combine the local gradient distribution and frequency domain characteristics of the CT image to generate degradation kernels related to the image content.
[0012] Furthermore, the multi-scale degradation kernel generator based on dynamic convolution is used to generate a degradation kernel related to the image content by combining the local gradient distribution and frequency domain features of the CT image. The method includes:
[0013] Its spatial gradient information is extracted and weighted convolution is performed on Gaussian kernels at multiple scales to obtain the degradation kernel response map. A three-layer perceptron is used to generate a noise pattern based on the average grayscale value of the entire image, which is weightedly fused with the convolution result to finally output a dynamic degradation kernel related to the image content.
[0014] Furthermore, the construction and training process of the degradation generative adversarial network model includes adopting a degradation-aware residual correction generator, a dynamic upsampling mechanism guided by cross-scale feature fusion and degradation kernel, and embedding degradation kernel information as prior knowledge into the generation process.
[0015] Furthermore, the dynamic upsampling mechanism adopts an adaptive upsampling filter parameterized by a degradation kernel.
[0016] Furthermore, the dynamic upsampling mechanism is expressed as follows:
[0017]
[0018]
[0019] Where, It is the high-resolution feature map after dynamic upsampling; for Position opposite The weight coefficient of each neighboring pixel is dynamically calculated based on the degradation kernel parameter and coordinate encoding; Indicates that the low-resolution image is The pixel value at each position; is a positive integer; is a positive integer; Indicates the The neighborhood pixel index corresponding to the target position in the low-resolution image; express The low-resolution neighborhood corresponding to the target location; For the The coordinate encoding vector of the position; For the The coordinate encoding vector of the position; for The degenerate kernel parameter vector corresponding to the position.
[0020] Furthermore, the construction and training process of the degenerate generative adversarial network model includes adopting a multi-scale gradient discriminator to construct a dual discrimination mechanism in the original image space and the gradient space.
[0021] Furthermore, the multi-scale gradient discriminator includes constructing an image pyramid, obtaining images at multiple resolutions, and performing discrimination and scoring at each scale through an independent Patch Generative Adversarial Network sub-discriminator. It also includes extracting the gradient map of the image and inputting it into an independent gradient domain discriminator, outputting the result for edge discrimination, and proportionally weighting the multi-scale discrimination result and the gradient discrimination result to output the overall discrimination score.
[0022] Furthermore, the construction and training process of the degenerate generative adversarial network model includes a three-stage training process.
[0023] Furthermore, in the first stage of the three-stage training process, the discriminator is frozen to minimize Optimize the generator; the loss function of the generator in stage 2 is , the loss function of the discriminator is ; The optimization objective of the training process is expressed as: ,and ; The third stage enables the multi-scale gradient discriminator, and the generator optimization goal is ;
[0024] in, is the structural similarity loss; is pixel-level adversarial loss; represents the generator parameters when minimizing the loss function; represents the discriminator parameters when maximizing the loss function; Gradient domain adversarial loss
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] 1. The present invention adopts a dynamic degradation kernel generation mechanism that is adaptive to image content and realistically simulates the degradation process of CT images under different devices and tissue structures.
[0027] 2. The present invention adopts a dynamic upsampling and residual correction network guided by the degradation kernel to significantly improve the detail restoration ability and structural fidelity of super-resolution images.
[0028] 3. The present invention constructs a multi-scale gradient discriminator and HU-weighted structural similarity loss function to effectively enhance the image texture authenticity and key tissue density consistency.
[0029] 4. The present invention adopts a strategy based on training dynamics and three-stage optimization to achieve a balance in the training game between the generator and the discriminator, thereby improving the model stability and convergence quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0031] Figure 1 This is a flowchart of an image and video super-resolution enhancement method based on a degenerate generative adversarial network according to an embodiment of the present application;
[0032] Figure 2 is an original CT image according to an embodiment of the present application;
[0033] Figure 3 is a kernel function heat map of a dynamic degradation kernel according to an embodiment of the present application;
[0034] Figure 4 is a comparison chart of super-resolution performance under different noise levels according to an embodiment of the present application;
[0035] Figure 5 These are the degraded image, anatomical attention map, and generated low-resolution image according to an embodiment of the present application.
[0036] Figure 6 The preliminary upsampling map, the residual feature map, the residual correction map, and the skeleton region mask according to the embodiment of the present application;
[0037] Figure 7 This is to compare the changes in structural similarity indicators between this technology and conventional generative adversarial networks during training;
[0038] Figure 8 It is the gradient map of the final enhanced image and the final super-resolution enhanced image according to the embodiment of the present application. DETAILED DESCRIPTION
[0039] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0040] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0041] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention and are not drawn according to the number, shape and size of components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be changed at will, and the component layout form may also be more complicated.
[0042] See also Figure 1 , an image and video super-resolution enhancement method based on degradation generative adversarial network, comprising the following steps:
[0043] S1, image acquisition;
[0044] In order to train the degenerate generative adversarial network model, it is necessary to collect high-quality CT images and video sequences as high-resolution original sample data.
[0045] Image data primarily comes from public or authorized databases such as TCIA and LIDC-IDRI, covering CT images of different locations and acquired using different equipment. Video data is generated by reconstructing continuous CT slices into frame sequences or acquiring multiple frames of continuous real-time CT images using dynamic imaging equipment to create video clips simulating clinical observation scenarios, which are used to expand the model's temporal modeling capabilities.
[0046] During the acquisition process, to ensure training data quality, abnormal samples such as severe artifacts, uneven faults, and out-of-focus images must be removed. If the acquired video frame sequence has inconsistent frame rates or motion blur, temporal interpolation and inter-frame registration should be performed to unify the inter-frame interval and enhance clarity to construct standardized video samples.
[0047] In one embodiment, Figure 2 It is the original CT image, which has certain characteristic morphology and bone structure.
[0048] S2, image and video preprocessing;
[0049] During the acquisition process, original CT images often have problems such as inconsistent image grayscale range, noise interference, and artifact pollution. Therefore, the images need to be standardized and preprocessed before model training.
[0050] This includes cropping or padding all images to a standard size, such as 256×256 or 512×512, to ensure consistent image sizes for the input model.
[0051] In addition, the image grayscale values are normalized according to Hounsfield units (HU), and the grayscale dynamic range acquired by different devices is uniformly mapped to the standard range to reduce data offset.
[0052] Optionally, to enhance the robustness and generalization ability of the model, the training images can be subjected to random rotation, horizontal flipping, noise superposition, contrast perturbation and other operations, so that the model can learn the image distribution patterns under different imaging conditions.
[0053] For video data, smoothing filtering and local motion compensation need to be applied between frames to reduce information loss caused by inter-frame jitter.
[0054] S3. Construction and training of super-resolution enhancement model
[0055] The present invention adopts a degradation generative adversarial network model to perform super-resolution enhancement on CT images. The degradation generative adversarial network model is a generative model that combines image degradation modeling with an adversarial training mechanism. It simulates the low-resolution degradation process of real CT images through a degradation kernel generation module and guides the generator to learn the mapping relationship of restoring high-quality images from degraded images. The generator not only learns to reconstruct images, but also simulates the degradation mechanism in the real imaging process, thereby improving the adaptability of the model in image super-resolution tasks.
[0056] The construction and training process of the degenerate generative adversarial network model include the following:
[0057] S301, constructing a multi-scale degradation kernel generation module;
[0058] CT images have complex noise distribution, low contrast, and diverse degradation patterns caused by acquisition using different equipment;
[0059] Conventional super-resolution methods use a fixed degradation kernel for blur modeling, which cannot adapt to the multi-scale degradation characteristics of CT images in real scenes and device-related noise differences;
[0060] The present invention adopts a multi-scale degradation kernel generator based on dynamic convolution, combines the local gradient distribution and frequency domain characteristics of CT images, and generates degradation kernels related to the image content;
[0061] Specifically, for the input CT image, its spatial gradient information is first extracted and weighted convolution is performed on it with Gaussian kernels at multiple scales to obtain the degradation kernel response map. Then, a three-layer perceptron is used to generate a noise pattern based on the average grayscale value of the entire image. This noise pattern is weightedly fused with the convolution result to finally output a dynamic degradation kernel related to the image content, which can be expressed as:
[0062]
[0063] Where, is the dynamically generated degenerate kernel, Indicates that the input is The dynamically generated degenerate kernel when
[0064] It is a high-resolution CT image; Represents high-resolution images In the Under the scale, the standard deviation is The result after blurring with Gaussian kernel; is a positive integer; is a multi-scale quantity, such as, ;
[0065] is the weight coefficient of the sth scale, which is obtained based on the Softmax weighting of the frequency domain energy and is calculated as follows: ,The frequency domain energy of CT images is concentrated in low-frequency parts such as organs and high-frequency parts such as calcification points, and weights are dynamically allocated to match the degradation intensity of different tissues; is the fast Fourier transform function; is the Softmax function; is the height of the input image; is the width of the input image; is a positive integer; is a positive integer; Indicates that After fast Fourier transform, the The complex amplitude value of the frequency component;
[0066] It is an image space gradient operator used to capture the edge and texture structure of CT images. The implementation method is as follows: , in the vertical dimension ,CT images have low contrast, but the gradients of anatomical structures such as bones and blood vessels are significant, and the gradient operator is used to enhance local structure modeling; is the gradient of the image in the horizontal direction, which is used to capture the edge information of the image in the horizontal direction; is the gradient of the image in the ordinate direction, which is used to capture the edge information of the image in the vertical direction; Indicates that the image is at position The pixel value at , that is, the grayscale value of the adjacent pixel to the right of the current pixel; Indicates that the image is at position The pixel value at , that is, the grayscale value of the adjacent pixel to the left of the current pixel; Indicates that the image is at position The pixel value at , that is, the grayscale value of the adjacent pixel below the current pixel; Indicates that the image is at position The pixel value at , that is, the grayscale value of the adjacent pixel above the current pixel; Represents the convolution operation; is the Gaussian kernel of the sth scale. Multi-scale Gaussian kernels simulate different degrees of blur degradation. The resolution difference of CT equipment leads to multi-scale blur, such as blur of small blood vessels and blur of large organs. The multi-scale design covers the actual degradation mode. For the The Gaussian kernel standard deviation at each scale, such as, ; For the adaptive noise term, the noise pattern is generated from the global average grayscale through a fully connected neural network. generate; It is a 3-layer fully connected neural network used to generate device-related noise patterns based on global image features. It relies on the same loss function as the generator and is updated using error backpropagation. For example, the loss function uses structural similarity loss. ; is the global average pooling operation, The representation is to average all pixel values of the input image to obtain a scalar.
[0067] In one embodiment, the dynamic degradation kernel generated by multi-scale gradient features and frequency domain weighting is used to present the spatial distribution of the kernel function in the form of a heat map. Figure 3 As shown in the figure, the warm areas indicate locations with higher degradation intensity, which is consistent with the degradation characteristics of key anatomical structures in medical images. The cold areas correspond to flat areas of soft tissue with lower degradation intensity. The degradation kernel function can accurately preserve the morphological characteristics of diagnosis-related areas by fusing local gradient information and adaptive noise patterns.
[0068] In this example, by simulating super-resolution reconstruction tasks under different noise levels, the performance differences between traditional bicubic interpolation, deep learning-based super-resolution convolutional networks, and generative adversarial networks are compared with this technology. The super-resolution performance comparison results under different noise levels are shown in Figure 2. Figure 4As shown in the experiment, the performance trends of various methods as noise increases are plotted using the noise level (standard deviation) on the horizontal axis and the peak signal-to-noise ratio (PSNR) of the reconstructed image on the vertical axis. The results show that traditional methods experience a sharp decline in performance as noise increases. In particular, bicubic interpolation, due to the limitations of a fixed interpolation kernel, cannot adapt to complex noise patterns. While deep learning-based super-resolution convolutional networks have some noise immunity, they do not adequately model device-related noise. Generative adversarial network methods, lacking guidance on the degradation process, produce images that deviate from the true distribution. However, this technology, through a dynamic degradation kernel generation module that combines image gradients with frequency domain features to adaptively model the degradation process, maintains a stable PSNR despite varying noise levels, demonstrating its strong adaptability to multi-scale degradation and device noise.
[0069] S302: Generate a degraded image
[0070] Traditional generative adversarial networks directly use random noise input when generating degraded images, which can easily lead to domain shift between the generated low-resolution images and the actual degradation pattern;
[0071] This paper maps CT images to a low-resolution space using a degradation kernel and combines the channel attention mechanism to strengthen the degradation modeling of key anatomical structures. Through the anatomical structure attention mechanism, stronger degradation constraints are imposed on key areas such as bones and blood vessels, preserving the morphological features required for medical diagnosis.
[0072] Specifically, the input CT image and the dynamic degradation kernel are first convolved channel by channel to generate a degraded image feature map. Then, the deep features extracted by VGG-16 are transformed into a channel attention map through 1×1 convolution to enhance the key areas. The attention map is then multiplied channel by channel with the degradation feature map and downsampled at a fixed rate to obtain a low-resolution image output, which is expressed as:
[0073]
[0074] Where, It is a 4x downsampling operation used to reduce the image resolution; is the generated degraded low-resolution image; It is a dynamically generated degradation kernel used to simulate the degradation process of medical CT images;
[0075] Indicates channel-by-channel multiplication; is the channel attention mask, Indicates that the input is The channel attention mask at time , is calculated as ; It is a 1×1 convolution operation; is the Sigmoid activation function; Extract functions for pre-trained VGG-16 networks, Represents input Afterwards, the high-level semantic features extracted by the pre-trained VGG-16 network and the high-level semantic features such as bones and blood vessels extracted by VGG-16 are enhanced to ensure that the degree of degradation of key anatomical structures is controllable and to avoid the loss of diagnostic information.
[0076] Optionally, the channel attention mask can be replaced by an anatomical attention map. Specifically, the nnUNet segmentation network pre-trained on medical images is used to generate the anatomical attention map, which directly generates the bone / organ mask, forcing degradation constraints in key areas to preserve morphological features.
[0077] In one embodiment, the degradation process is visualized, such as Figure 5 As shown in the figure, the original image is processed by the dynamic degradation kernel to produce the degraded image. Directional blurring is visible at bone edges, while noise texture is enhanced in soft tissue areas. In the anatomical attention map, the highlighted areas correspond to bone boundaries, preventing the loss of important details during downsampling. The resulting low-resolution image is the final degradation result after 4x downsampling. While maintaining overall structural integrity, it also introduces controlled blurring of local details.
[0078] S303, using the generator to generate enhanced images
[0079] Traditional super-resolution generators directly perform upsampling through deconvolution or interpolation, ignoring the spatial correlation of multi-scale anatomical structures in CT images, resulting in distortion of details;
[0080] The present invention adopts a degradation-aware residual correction generator, adopts a dynamic upsampling mechanism guided by cross-scale feature fusion and degradation kernel, and embeds the degradation kernel information as prior knowledge into the generation process;
[0081] Specifically, the residual correction module is used to gradually eliminate degradation artifacts, while anatomical structure constraints are used to ensure the geometric topological consistency of key areas such as vascular bifurcations and calcification points.
[0082] The specific process is as follows:
[0083] 1) Perform degenerate kernel guided feature extraction
[0084] The low-resolution image and the dynamic degradation kernel are concatenated into a 6-channel input, and multi-scale features are extracted through cascaded degradation-aware convolution blocks. Each convolution block is defined as:
[0085]
[0086] Where, For the Layer feature map; For the Layer feature map; is a positive integer; To represent the 3×3 convolution operation; For the degenerate core The adaptively scaled version of the layer is used for feature extraction at different scales, i.e. ; Degenerate core Resize to the original size times to meet the requirements of feature extraction at different scales; Indicates channel splicing; is the LeakyReLU activation function with a slope of 0.2.
[0087] 2) Perform dynamic upsampling
[0088] An adaptive upsampling filter parameterized by a degradation kernel is used to avoid edge blurring caused by a fixed interpolation kernel, which can be expressed as:
[0089]
[0090]
[0091] Where, It is the high-resolution feature map after dynamic upsampling; for Position opposite The weight coefficient of each neighboring pixel is dynamically calculated based on the degradation kernel parameter and coordinate encoding to avoid edge blurring such as small blood vessel breakage caused by a fixed interpolation kernel and adaptively retain high-frequency structures. Indicates that the low-resolution image is The pixel value at each position; is a positive integer; is a positive integer; Indicates the The neighborhood pixel index corresponding to the target position in the low-resolution image; express The low-resolution neighborhood corresponding to the target location; For the The coordinate encoding vector of the position is calculated as ; For the The coordinate encoding vector of the position; The low-resolution image The horizontal coordinate of the position; The low-resolution image The vertical coordinate of the position; for The degenerate kernel parameter vector corresponding to the position; is the vector inner product.
[0092] 3) Input the feature map and upsampled features into the residual correction network
[0093] Construct a cascade of residual correction units, each of which contains a degraded residual gating module and an anatomical structure constraint module;
[0094] The degradation residual gating module generates gating weights through the degradation kernel to suppress the residual propagation in the artifact area, which is expressed as:
[0095]
[0096] Where, Indicates the passing strength of the residual signal of each pixel. A high value means that the residual is allowed to pass, and a low value means that the residual update of the area is suppressed. It is an element-by-element multiplication operation; For the The feature map of the layer is used as the input of the degraded residual gating module; is the last layer number of the degenerate residual gating module;
[0097] The anatomical structure constraint module uses a pre-trained U-Net segmentation network to extract the bone mask and imposes an L1 constraint in the residual path. The HU value of the bone region is stable and critical for diagnosis, forcing the generator to accurately reconstruct its geometric topology, which is expressed as:
[0098]
[0099]
[0100] Where, Skeleton masks extracted for the pre-trained U-Net segmentation network; is the anatomical structure constraint loss function; is the L1 norm; It is an element-by-element multiplication operation; Represents the super-resolution reconstructed image, which is the final output of the generator; It is the high-resolution feature map after dynamic upsampling; is a high-resolution target image used to calculate the loss function; For the A residual correction unit to gradually eliminate degradation artifacts and optimize image quality; is the total number of residual correction units, which is used to define the depth of the residual correction module in the network; r is a positive integer.
[0101] Generated Input to the multi-scale gradient discriminator, forcing The texture details and edge continuity approach the real high-resolution CT distribution.
[0102] It should be noted that the method of dynamically upsampling the generator based on the degeneration kernel to generate high-resolution images is not easy for technical personnel in this field to think of. If the degeneration kernel guidance is missing, the generator cannot specifically correct the different blur patterns of medical CT images.
[0103] In one embodiment, upsampling and residual correction are analyzed, such as Figure 6 As shown in the figure, the overall structure of the preliminary upsampling image is complete but the edges are blurred. The red area of the residual feature map represents the positive residual that needs to be enhanced, and the blue area represents the negative noise that needs to be suppressed. The sharpness of the bone edges in the residual correction image is significantly improved, and the block artifacts in some areas are suppressed. The white area of the bone area mask is a prominent structure, which is used to impose anatomical constraints during the residual correction process to ensure topological invariance and avoid geometric distortion caused by over-correction.
[0104] S304, using a discriminator to perform image discrimination;
[0105] The discriminator of conventional generative adversarial networks uses a single scale discrimination, which makes it difficult to capture the cross-scale feature correlations of different tissue structures in CT images;
[0106] The present invention adopts a multi-scale gradient discriminator to construct a dual discrimination mechanism in the original image space and the gradient space, forcing the generator to meet the medical image quality standards in terms of texture details and edge continuity.
[0107] Specifically, for the input image, we first construct an image pyramid to obtain images at multiple resolutions. At each scale, we use an independent Patch Generative Adversarial Network sub-discriminator to perform discrimination and scoring. At the same time, we extract the gradient map of the image and input it into an independent gradient domain discriminator, which outputs the result for edge discrimination. Then, the multi-scale discrimination results and the gradient discrimination results are weighted proportionally to output the overall discrimination score, which is expressed as:
[0108]
[0109] Where, is the comprehensive discrimination score of the multi-scale gradient discriminator, Represents the input image The total score after multi-scale discrimination; is the input image of the discriminator. For each training batch, the discriminator receives and ; is the number of pyramid scales, such as, ; is a positive integer; is the image pyramid at the mth scale; For the The weight coefficient of each scale is calculated as ; is an exponential function with a natural constant as its base; is the calculation function of the structural similarity index, represents the structural similarity index; is the reference image, i.e. the real high-resolution image; Adjust the coefficient for sharpness to control the sharpness of the weight distribution, e.g. ; is the base discriminator based on the Patch-generated adversarial network, Indicates the The scaled image passes through the output of the base discriminator; is the balance coefficient, for example, set to ; is the gradient domain discriminator, The gradient map is output by the gradient domain discriminator; is a gradient map, which strengthens the edge continuity judgment. The calculation method is expressed as ,The gradient information of tissue boundaries in CT images is crucial for diagnosis, ,such as the gradient information of the lung-mediastinum interface, and ,the gradient domain discriminator prevents edge blurring.
[0110] S305, calculate the loss function of the generative adversarial network
[0111] Traditional adversarial loss uses global image-level discrimination, which can easily lead to excessive smoothing of detailed features in small lesion areas in medical images.
[0112] The present invention adopts pixel-level adversarial loss based on local windows, calculates adversarial loss within the local window, and calculates adversarial loss independently for each local window, thereby enhancing the generation quality control of small lesion areas.
[0113] Specifically, the super-resolution image is divided into fixed-size window areas. The discriminator calculates the authenticity score of each area, and then the average of all window scores is taken as the overall pixel-level adversarial loss to enhance the model's ability to model local area details. It is expressed as:
[0114]
[0115] Where, It is pixel-level adversarial loss; The height of the local window, such as 32 pixels; The width of the local window, such as 32 pixels; For the The super-resolution reconstruction result of a local window is used to represent the super-resolution reconstruction result; is a positive integer, and in , it represents the local window index of is a logarithmic function with a default base of 10; represents the discriminant score of the th local window.
[0116] S306. Construct the structural similarity constraint
[0117] Regarding the sensitivity of tissue density values in CT images, a weighted structural similarity loss based on the Hounsfield unit (HU) is adopted to retain the density distribution characteristics of different tissues;
[0118] Specifically, the image is divided into regions according to the HU value, and the bone (HU>300) and soft tissue (-50<HU<300) are respectively extracted as two regional maps. Then, the structural similarity of each region is calculated, and it is weighted according to the proportion of the regional area in the whole image to obtain the overall structural similarity loss, which is used to optimize the tissue density distribution of the generated image, expressed as:
[0119]
[0120] In the formula, is the structural similarity loss; represents two regions divided according to the Hounsfield unit in the image, namely the bone region and the soft tissue region; is the bone region segmented according to the Hounsfield unit;
[0121] is the soft tissue region segmented according to the Hounsfield unit; is the regional weight, which is equal to the number of pixels in the bone or soft tissue region divided by the total number of pixels in the whole image.
[0122] Furthermore, the structural similarity loss is calculated for the bone region (HU>300) and the soft tissue region (-50<HU<300) respectively to retain the density distribution characteristics of different tissues;
[0123] The HU refers to the gray value unit in CT scanning and is used to represent tissue density.
[0124] S307. Optimize the dynamic learning rate strategy
[0125] To balance the training progress of the generator and the discriminator, an adaptive learning rate adjustment mechanism based on gradient statistics is adopted. The learning rate of the generator is dynamically adjusted by monitoring the gradient amplitude ratio of the generator and the discriminator in real time. When the discriminator is too powerful, the learning rate of the generator is automatically reduced;
[0126] Specifically, in each training cycle, the expected gradient norm of the current generator and discriminator is calculated respectively as a sign of dynamic changes in training. If the discriminator gradient is significantly greater than that of the generator, the generator learning rate is reduced to prevent it from falling into the optimization dilemma. Otherwise, the learning rate is maintained or slightly increased to make the generator converge quickly in the early stage of training, which can be expressed as:
[0127]
[0128] Where, For the generator Adaptive learning rate for each iteration; is the initial learning rate, which is calculated as ; A small constant to prevent the denominator from being 0, such as setting .
[0129] S308. Implementing a phased training strategy
[0130] To meet the feature learning requirements of CT images at different resolution levels, a three-stage training process is adopted:
[0131] 1) Stage 1 (iterations 0-1000): Freeze the discriminator and minimize Optimize generators;
[0132] The goal of the first stage is to train the generator G to basically restore high-resolution images with similar structures, ensuring that the generator converges to the initial solution and is not misled by the weak discriminator; the loss function of the first stage adopts the structural similarity loss , encouraging image structure restoration; the optimization strategy of stage one only updates the generator parameters , discriminator Freeze, the optimization objective is expressed as ; is the generator parameter; represents the generator parameters when minimizing the loss function; Indicates that the input is and The structural similarity loss.
[0133] 2) Phase 2 (1000-5000 iterations): Jointly optimize the total loss function ;
[0134] The goal of the second stage is to add adversarial supervision to guide the generator to evolve in a more realistic direction and gradually establish an adversarial game relationship between the generator and the discriminator.
[0135] In the loss function of stage 2, the loss function of the generator is , the loss function of the discriminator is ;
[0136] The optimization objective of the training process in stage 2 is expressed as: ,and ;
[0137] represents the discriminator parameters when maximizing the loss function.
[0138] 3) Phase 3 (5000-10000 iterations): Enable the multi-scale gradient discriminator and optimize ;
[0139] The goal of stage three is to improve the authenticity of texture details and edge structures. In stage three, the discriminator simultaneously captures strong texture + edge (gradient domain) features.
[0140] The optimization goal of the generator in stage three is ;
[0141] in, The gradient domain adversarial loss is calculated as follows: first, the gradient of the super-resolution image output by the generator and the real high-resolution image are calculated respectively, and the gradient components in the horizontal and vertical directions are obtained by the Sobel operator. Then, the squares of the gradient amplitudes in the two directions are summed to obtain the gradient amplitude map. Then, the gradient maps of the real image and the generated image are input into independent gradient domain discriminators respectively. The gradient domain discriminator outputs a authenticity score. Then, the Wasserstein adversarial loss is used to calculate the difference between the expected value of the discriminant score of the real gradient map and the expected value of the discriminant score of the generated gradient map as the gradient domain adversarial loss;
[0142] The independent gradient domain discriminator is composed of a multi-layer convolutional neural network, such as a convolutional neural network with 3 layers of convolution.
[0143] The discriminator optimization goal of stage three is to and The probability of discrimination is approximately 50%, that is, it is almost impossible to distinguish and .
[0144] In one embodiment, to verify the effectiveness of the three-stage training strategy, the experiment compared the changes in the structural similarity index of this technology and the conventional generative adversarial network during the training process, such as Figure 7As shown. The horizontal axis is the number of training iterations, and the vertical axis is the structural similarity index. The convergence process of the two is shown by a line graph. Experiments show that the conventional method stagnates after rapid improvement in the early stage of training, because its single discriminant scale is difficult to capture the multi-level features of medical images; the staged optimization strategy of this technology prioritizes learning structural reconstruction by freezing the discriminator in the first stage, and quickly establishes the basic mapping relationship; after introducing the adversarial loss in the second stage, the details of the generated image gradually approach the true distribution; in the third stage, the multi-scale gradient discriminator is enabled, and the edge continuity is strengthened through gradient space constraints, and finally the structural similarity index converges to a higher level. The stage dividing point marked by the vertical dotted line in the figure further shows that the staged strategy can effectively balance the game relationship between the generator and the discriminator, and avoid falling into the local optimum too early.
[0145] S4, performing super-resolution enhancement of the image;
[0146] The degraded generative adversarial network model after training can be directly applied to super-resolution enhancement of CT images or videos.
[0147] For a single CT image, the user inputs it into the model's generator module. The model dynamically upsamples and compensates for structural details based on its content and the degradation kernel prior learned during training, outputting a clear, well-layered high-resolution image while preserving key anatomical structures such as bones and blood vessels in the original image.
[0148] For enhanced processing of CT video sequences, images can be input frame by frame and the model can be used to generate high-resolution frame sequences, or inter-frame motion estimation and feature fusion modules can be combined to improve the image resolution of each frame while maintaining temporal continuity.
[0149] In one embodiment, Figure 8 As shown, in the gradient map of the final enhanced image, the highlighted areas indicate strong edge responses after reconstruction. The image shows continuous and complete bone contours and the direction of vascular branches, verifying the effectiveness of the multi-scale gradient discriminator in constraining edge continuity. Compared to the original image, the final super-resolution enhanced image significantly improves the sharpness and contrast of key areas while maintaining the consistency of the HU value distribution. The synergistic effect of dynamic upsampling and residual correction eliminates the blur caused by interpolation and avoids the over-sharpening artifacts commonly found in generative adversarial networks.
[0150] In this implementation, a dynamic degradation kernel generation mechanism adaptive to image content is implemented to realistically simulate the degradation process of medical CT images under different devices and tissue structures. A dynamic upsampling and residual correction network guided by the degradation kernel is used to significantly improve the detail restoration and structural fidelity of super-resolution images. A multi-scale gradient discriminator and HU-weighted structural similarity loss function are constructed to effectively enhance image texture realism and key tissue density consistency. A strategy based on training dynamics and three-stage optimization is adopted to achieve a balance in the training game between the generator and discriminator, improving model stability and convergence quality.
[0151] The embodiments described above are merely descriptions of preferred implementations of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should fall within the scope of protection determined by the claims of the present invention.
Claims
1. A method for image and video super-resolution enhancement based on a degenerate generative adversarial network, characterized in that: include: Image acquisition; Collect CT images and video sequences as high-resolution raw sample data; Image and video preprocessing; This includes image standardization, smoothing filtering and local motion compensation between frames of the video; Super-resolution enhancement model construction and training; Super-resolution enhancement of CT images based on the degradation generative adversarial network model; Perform super-resolution enhancement of images; The trained degraded generative adversarial network model is used for super-resolution enhancement of CT images or videos.
2. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 1 is characterized in that: The construction and training process of the degradation generative adversarial network model includes constructing a multi-scale degradation kernel generation module. Specifically, a multi-scale degradation kernel generator based on dynamic convolution is used to combine the local gradient distribution and frequency domain characteristics of the CT image to generate degradation kernels related to the image content.
3. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 2 is characterized in that: The multi-scale degradation kernel generator based on dynamic convolution is used to generate the degradation kernel related to the image content by combining the local gradient distribution and frequency domain characteristics of the CT image. The method includes: Its spatial gradient information is extracted and weighted convolution is performed on Gaussian kernels at multiple scales to obtain the degradation kernel response map. A three-layer perceptron is used to generate a noise pattern based on the average grayscale value of the entire image, which is weightedly fused with the convolution result to finally output a dynamic degradation kernel related to the image content.
4. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 1, characterized in that: The construction and training process of the degradation generative adversarial network model includes adopting a degradation-aware residual correction generator, a dynamic upsampling mechanism guided by cross-scale feature fusion and degradation kernel, and embedding degradation kernel information as prior knowledge into the generation process.
5. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 4 is characterized in that: The dynamic upsampling mechanism uses an adaptive upsampling filter parameterized by a degradation kernel.
6. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 4 is characterized in that: The dynamic upsampling mechanism is expressed as follows: 、 ; Where, It is the high-resolution feature map after dynamic upsampling; for Position opposite The weight coefficient of each neighboring pixel is dynamically calculated based on the degradation kernel parameter and coordinate encoding; Indicates that the low-resolution image is The pixel value at each position; is a positive integer; is a positive integer; Indicates the The neighborhood pixel index corresponding to the target position in the low-resolution image; express The low-resolution neighborhood corresponding to the target location; For the The coordinate encoding vector of the position; For the The coordinate encoding vector of the position; for The degenerate kernel parameter vector corresponding to the position.
7. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 1, characterized in that: The construction and training process of the degenerate generative adversarial network model includes adopting a multi-scale gradient discriminator to construct a dual discrimination mechanism in the original image space and the gradient space.
8. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 7 is characterized in that: The multi-scale gradient discriminator includes constructing an image pyramid, obtaining images at multiple resolutions, and performing discrimination and scoring at each scale through an independent Patch Generative Adversarial Network sub-discriminator. It also includes extracting the gradient map of the image and inputting it into an independent gradient domain discriminator, outputting the result for edge discrimination, and proportionally weighting the multi-scale discrimination result and the gradient discrimination result to output the overall discrimination score.
9. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 1, characterized in that: The construction and training process of the degenerate generative adversarial network model includes a three-stage training process.
10. The image and video super-resolution enhancement method based on the degradation generative adversarial network according to claim 8, characterized in that: In the first stage of the three-stage training process, the discriminator is frozen, minimizing Optimize the generator; the loss function of the generator in stage 2 is , the loss function of the discriminator is ; The optimization objective of the training process is expressed as: ,and ; The third stage enables the multi-scale gradient discriminator, and the generator optimization goal is ; in, is the structural similarity loss; is pixel-level adversarial loss; represents the generator parameters when minimizing the loss function; represents the discriminator parameters when maximizing the loss function; is the gradient domain adversarial loss.
Citation Information
Patent Citations
Liver CT image enhancement processing method and system
CN110136217B
Image enhancement method and system based on brightness correction and fusion channel prior
CN114627009B
A medical image processing method and image processing system
CN119228750B
Terahertz image super-resolution reconstruction method based on generative adversarial network
CN115358922A
Infrared and visible light image fusion method based on multi-discriminator generative adversarial network
CN115601282A
Cited By
Bone evaluation method based on neural network enhancement and finite element analysis
CN121616708A
Multi-modal image segmentation and enhancement processing method
CN121640238A
Vehicle tank body surface texture analysis method and system based on deep learning
CN121724987A
Airborne photoelectric video image super-resolution processing method based on deep learning
CN122312384A
