Image and video super-resolution enhancement method based on degenerate generative adversarial network

Abstract

The application discloses an image video super-resolution enhancement method based on a degradation generative adversarial network, relates to the field of image processing, and comprises the following steps: image acquisition and preprocessing; super-resolution enhancement model building and training; and super-resolution enhancement of images based on a degradation generative adversarial network model. The application realizes an image content self-adaptive dynamic degradation kernel generation mechanism, truly simulates the degradation process of images under different devices and organizational structures, adopts a degradation kernel guided dynamic up-sampling and residual correction network, significantly improves the detail restoration capability and structural fidelity of super-resolution images, effectively enhances the image texture authenticity and key organizational density consistency, realizes the balance of the training game between the generator and the discriminator, and improves the model stability and convergence quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an image and video super-resolution enhancement method based on a degradation generative adversarial network. BACKGROUND

[0002] With the rapid development of imaging technology, the role of image processing technology is becoming increasingly important. However, due to factors such as imaging device resolution, radiation dose control, and scanning speed, the collected images often have problems such as insufficient resolution, blurred details, and noise interference, especially in critical areas. Low-resolution images can lead to unclear tissue structure and decreased judgment accuracy. Therefore, how to perform high-quality super-resolution reconstruction on images without changing the original acquisition conditions has become an important problem that needs to be solved in the current image processing field. Existing super-resolution methods are mostly based on fixed degradation models or direct interpolation strategies, lack the ability to model the real imaging degradation process, and perform poorly in terms of the fidelity of critical anatomical structures and edge sharpness, making it difficult to meet the high requirements of structure consistency and density restoration in clinical practice. Therefore, there is an urgent need to propose an image super-resolution enhancement method that can adaptively simulate the actual degradation mechanism, strengthen the restoration ability of critical structures, and balance image clarity and diagnostic value.

[0003] Existing technologies such as patents with publication numbers CN114627009B, CN119228750B, and CN110136217B have problems such as being unable to accurately simulate the complex degradation patterns of images under different devices and tissues, blurred details and distorted critical anatomical structures, lacking multi-level modeling ability for image edges and texture details, and being prone to local optimization or poor convergence. SUMMARY

[0004] To solve the technical problem of poor image processing effect in the prior art, the present application provides an image and video super-resolution enhancement method based on a degradation generative adversarial network.

[0005] The present application is achieved by the following technical solutions:

[0006] An image and video super-resolution enhancement method based on a degradation generative adversarial network, comprising:

[0007] Image acquisition; collect CT images and video sequences as high-resolution original sample data;

[0008] Image and video preprocessing; including standardizing the image, and applying smoothing filtering and local motion compensation between frames to the video;

[0009] Super-resolution enhancement model building and training; based on a degradation generative adversarial network model to perform super-resolution enhancement on the CT images;

[0010] Perform super-resolution enhancement of images; use the trained degenerate generative adversarial network model for super-resolution enhancement of CT images or videos.

[0011] Furthermore, the construction and training process of the degradation generative adversarial network model includes constructing a multi-scale degradation kernel generation module. Specifically, a multi-scale degradation kernel generator based on dynamic convolution is used to generate degradation kernels related to image content by combining the local gradient distribution and frequency domain features of CT images.

[0012] Furthermore, the method of using a multi-scale degradation kernel generator based on dynamic convolution, combined with the local gradient distribution and frequency domain features of CT images, to generate a degradation kernel related to image content includes:

[0013] The spatial gradient information is extracted and weighted convolution is performed using Gaussian kernels at multiple scales to obtain a degradation kernel response map. A three-layer perceptron is used to generate a noise pattern based on the average gray value of the entire image, and this pattern is weighted and fused with the convolution result to finally output a dynamic degradation kernel related to the image content.

[0014] Furthermore, the construction and training process of the degradation generative adversarial network model includes using a degradation-aware residual correction generator, employing a dynamic upsampling mechanism guided by cross-scale feature fusion and degradation kernel, and embedding degradation kernel information as prior knowledge into the generation process.

[0015] Furthermore, the dynamic upsampling mechanism employs an adaptive upsampling filter with degenerate kernel parameterization.

[0016] Furthermore, the expression for the dynamic upsampling mechanism is as follows:

[0017]

[0018]

[0019] In the formula, This is a high-resolution feature map after dynamic upsampling; for Position at the first The weight coefficients of each neighboring pixel are dynamically calculated based on the degenerate kernel parameters and coordinate encoding. Indicates the low-resolution image in the first... The pixel value at each position; It is a positive integer; It is a positive integer; Indicates the first The neighboring pixel index of the target location in the low-resolution image; express The low-resolution neighborhood corresponding to the target location; For the first The coordinate encoding vector of each position; For the first The coordinate encoding vector of each position; for The degenerate kernel parameter vector corresponding to the position.

[0020] Furthermore, the construction and training process of the degenerate generative adversarial network model includes employing a multi-scale gradient discriminator to construct a dual discrimination mechanism in the original image space and gradient space.

[0021] Furthermore, the multi-scale gradient discriminator includes constructing an image pyramid, acquiring images at multiple resolutions, and performing discrimination and scoring at each scale through an independent Patch Generative Adversarial Network sub-discriminator. It also includes extracting the gradient map of the image and inputting it into an independent gradient domain discriminator, outputting the result for edge discrimination, and weighting the multi-scale discrimination result and the gradient discrimination result proportionally to output the overall discrimination score.

[0022] Furthermore, the construction and training process of the degenerate generative adversarial network model includes a three-stage training process.

[0023] Furthermore, in the first stage of the three-stage training process, the discriminator is frozen and minimized. Optimize the generator; the loss function of the generator in stage two is: The loss function of the discriminator is The optimization objective of the training process is represented as: ,and The third stage employs a multi-scale gradient discriminator, with the generator optimization objective being... ;

[0024] in, For structural similarity loss; For pixel-level adversarial loss; These represent the generator parameters when minimizing the loss function; These represent the discriminator parameters when maximizing the loss function; Adversarial loss in the gradient domain

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] 1. This invention employs an image content adaptive dynamic degradation kernel generation mechanism to realistically simulate the degradation process of CT images under different devices and tissue structures.

[0027] 2. This invention employs a dynamic upsampling and residual correction network guided by a degenerate kernel, which significantly improves the detail restoration capability and structural fidelity of super-resolution images.

[0028] 3、The application constructs a multi-scale gradient discriminator and a HU weighted structural similarity loss function, effectively enhancing the image texture authenticity and key tissue density consistency.

[0029] 4、The application adopts a strategy based on training dynamics and three-stage optimization to balance the training game between the generator and the discriminator, improve the model stability and convergence quality. BRIEF DESCRIPTION OF DRAWINGS

[0030] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0031] Figure 1 is a flow diagram of an image video super-resolution enhancement method based on a degradation generative adversarial network according to an embodiment of the present application;

[0032] Figure 2 is a raw CT image according to an embodiment of the present application;

[0033] Figure 3 is a kernel function thermodynamic diagram of a dynamic degradation kernel according to an embodiment of the present application;

[0034] Figure 4 is a super-resolution performance comparison diagram under different noise levels according to an embodiment of the present application;

[0035] Figure 5 is a degraded image, an anatomical attention map and a generated low-resolution image according to an embodiment of the present application.

[0036] Figure 6 is a preliminary up-sampling diagram, a residual feature map, a residual correction diagram and a bone region mask according to an embodiment of the present application;

[0037] Figure 7 is a comparison of the structural similarity index in the training process of the present technology and the conventional generative adversarial network;

[0038] Figure 8 is a gradient diagram of the final enhanced image and the final super-resolution enhanced image according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0040] Following, the embodiments of the present application are illustrated by specific examples, and other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied by other different specific embodiments, and various modifications or changes can be made to the details in the specification without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0041] It should also be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only show the components related to the present application in the drawings, not drawn according to the number, shape and size of the components in actual implementation. The shape, number and proportion of each component in actual implementation can be arbitrarily changed, and the layout form of the components can also be more complex.

[0042] Referring to Figure 1 A method for image and video super-resolution enhancement based on a degradation generative adversarial network, comprising the following steps:

[0043] S1, image acquisition;

[0044] In order to train the degradation generative adversarial network model, high-quality CT images and video sequences need to be collected as high-resolution original sample data.

[0045] The image data mainly comes from public or authorized databases such as TCIA, LIDC-IDRI, etc., covering CT images under different parts and different device acquisition conditions. The video data is obtained by reconstructing continuous CT slices into frame sequences, or by collecting multiple continuous real-time CT images through dynamic imaging devices to form video clips simulating clinical observation scenes, which are used to expand the time sequence modeling capability of the model.

[0046] During the acquisition process, in order to ensure the quality of the training data, abnormal samples such as severe artifacts, uneven layers, and out-of-focus images need to be removed. If the frame rate of the collected video frame sequence is inconsistent or motion blur exists, time interpolation and inter-frame registration operations should be performed to unify the inter-frame interval and enhance the clarity, and to construct standardized video samples.

[0047] In one embodiment, as Figure 2 The original CT image has certain characteristic morphology and skeletal structure.

[0048] S2, image and video preprocessing;

[0049] The original CT image often has problems such as inconsistent image gray scale range, noise interference, and artifact pollution during acquisition, and therefore needs to be standardized and pretreated before model training.

[0050] For example, all images are uniformly cropped or padded to a standard size, such as 256x256 or 512x512, to ensure that the image size of the input model is consistent;

[0051] In addition, the image gray scale value is normalized according to the Hounsfield unit (HU), and the gray scale dynamic range collected by different devices is uniformly mapped to a standard range to reduce data deviation.

[0052] Optionally, to enhance the robustness and generalization ability of the model, random rotation, horizontal flip, noise superposition, contrast disturbance and other operations can be performed on the training images, so that the model can learn the image distribution pattern under different imaging conditions.

[0053] For video data, smoothing filtering and local motion compensation need to be applied between frames to reduce information loss caused by frame jitter.

[0054] S3, super-resolution enhancement model building and training

[0055] The present application adopts a degradation generative adversarial network model to perform super-resolution enhancement on CT images, wherein the degradation generative adversarial network model is a generative model combining image degradation modeling and adversarial training mechanism, a degradation kernel generation module is used to simulate the low-resolution degradation process of real CT images, and a generator is guided to learn the mapping relationship from degraded images to high-quality images, so that the generator not only learns to reconstruct images, but also needs to simulate the degradation mechanism in the real imaging process, thereby improving the adaptability of the model in the image super-resolution task.

[0056] The construction and training process of the degradation generative adversarial network model includes the following steps:

[0057] S301, constructing a multi-scale degradation kernel generation module;

[0058] The noise distribution in the CT image is complex, the contrast is low, and the degradation modes caused by different devices are diversified;

[0059] The conventional super-resolution method uses a fixed degradation kernel for blur modeling, which cannot adapt to the multi-scale degradation characteristics of the CT image in the actual scene and the device-related noise difference;

[0060] The present application adopts a multi-scale degradation kernel generator based on dynamic convolution, which combines the local gradient distribution and frequency domain features of the CT image to generate a degradation kernel related to the image content;

[0061] Specifically, for the input CT image, first extract its spatial gradient information, and combine multiple scales of Gaussian kernel for weighted convolution, and then get the degradation kernel response map, then use three layers of perception machine to generate noise pattern according to the average gray value of the whole image, and then weighted fusion with the convolution result, finally output the dynamic degradation kernel related to the image content, expressed as:

[0062]

[0063] In the formula, The dynamically generated degradation kernel is The dynamically generated degradation kernel when the input is ;

[0064] The CT high-resolution image is The dynamically generated degradation kernel is The result after blurring under the first Scale using a Gaussian kernel with a standard deviation of ; N is a positive integer; The number of multi-scale is, for example, ;

[0065] The weight coefficient of the s-th scale is obtained based on the Softmax weighting of frequency energy, and the calculation method is represented as The CT image frequency energy is concentrated in the low frequency part of the organ and the high frequency part of the calcification point, and the weight is dynamically allocated to match the degradation intensity of different tissues; The fast Fourier transform function is The Softmax function is The height of the input image is The width of the input image is N is a positive integer; N is a positive integer; The complex amplitude value of the first Frequency component in the frequency domain after the fast Fourier transform of ;

[0066] The image spatial gradient operator is used to capture the edge and texture structure of the CT image, and the implementation is that in the horizontal coordinate dimension In the vertical coordinate dimension The CT image contrast is low, but the gradient of anatomical structures such as bones and blood vessels is significant, and the gradient operator is used to strengthen the local structure modeling; The gradient of the image in the horizontal coordinate direction is used to capture the edge information of the image in the horizontal direction; This is the gradient of the image in the vertical direction, used to capture edge information of the image in the vertical direction; Indicates the image at position The pixel value at that location is the grayscale value of the pixel adjacent to the right of the current pixel. Indicates the image at position The pixel value at that location is the grayscale value of the pixel adjacent to the left of the current pixel. Indicates the image at position The pixel value at that location is the grayscale value of the pixel adjacent to the current pixel. Indicates the image at position The pixel value at that location is the grayscale value of the pixel adjacent to the current pixel. Indicates the convolution operation; The Gaussian kernel is at the s-th scale. Multi-scale Gaussian kernels simulate different degrees of blurring degradation. The resolution difference of CT equipment leads to multi-scale blurring, such as blurring of small blood vessels and blurring of large organs. The multi-scale design covers the actual degradation mode. For the first Gaussian kernel standard deviation at various scales, such as... ; For adaptive noise terms, a noise pattern is generated from the global average grayscale using a fully connected neural network. generate; It is a 3-layer fully connected neural network used to generate device-specific noise patterns based on global image features. It relies on the same loss function as the generator and is updated using backpropagation of errors. For example, the loss function uses structural similarity loss. ; This is a global average pooling operation. The representation is obtained by averaging all pixel values ​​of the input image to obtain a scalar.

[0067] In one embodiment, the spatial distribution of the kernel function is presented in the form of a heatmap by using a dynamically degenerate kernel generated through multi-scale gradient features and frequency domain weighting. For example... Figure 3 As shown in the figure, the warm-colored areas represent locations with higher degradation intensity, which is consistent with the degradation characteristics of key anatomical structures in medical images. The cool-colored areas correspond to flat areas of soft tissue with lower degradation intensity. The degradation kernel function can accurately preserve the morphological features of diagnostic-related areas by fusing local gradient information and adaptive noise patterns.

[0068] In this embodiment, by simulating super-resolution reconstruction tasks under different noise levels, the performance differences between traditional bicubic interpolation, deep learning-based super-resolution convolutional networks, generative adversarial networks, and this technology are compared. The performance comparison results of super-resolution under different noise levels are as follows: Figure 4As shown, the experiment takes the noise level (standard deviation) as the horizontal axis, and the peak signal-to-noise ratio of the reconstructed image as the vertical axis, and shows the performance change trend of each method with the increase of noise through the broken line chart. The results show that the traditional method has a sharp decline in performance when the noise increases, especially the bicubic interpolation method, which cannot adapt to complex noise patterns due to the limitation of fixed interpolation kernel; the super-resolution convolution network based on deep learning has certain anti-noise ability, but it is insufficient to model the device-related noise; the generative adversarial network method has a deviation between the generated image and the true distribution due to the lack of degradation process guidance. However, the present technology generates a dynamic degradation kernel through the dynamic degradation kernel generation module, and combines image gradients and frequency domain features to adaptively model the degradation process, so that the peak signal-to-noise ratio remains stable when the noise level changes, verifying the strong adaptability of the present technology to multi-scale degradation and device noise.

[0069] S302, generating a degraded image

[0070] The traditional generative adversarial network directly uses random noise input when generating a degraded image, which can easily cause domain deviation between the generated low-resolution image and the true degradation mode;

[0071] The present application maps the CT image to the low-resolution space through the degradation kernel, and combines the channel attention mechanism to strengthen the degradation modeling of key anatomical structures. Through the anatomical structure attention mechanism, stronger degradation constraints are applied to key areas such as bones and blood vessels, and the morphological features required for medical diagnosis are preserved.

[0072] Specifically, the input CT image and the dynamic degradation kernel are first generated through a channel-by-channel convolution operation to generate a degraded image feature map, then the deep features extracted by VGG-16 are transformed into a channel attention map through a 1x1 convolution, and the key areas are strengthened, then the attention map and the degraded feature map are multiplied channel by channel, and fixed rate down-sampling is performed to obtain a low-resolution image output, which is represented as:

[0073]

[0074] In the formula, is a 4x down-sampling operation for reducing the image resolution; is the generated degraded low-resolution image; is a dynamically generated degradation kernel for simulating the degradation process of medical CT images;

[0075] represents channel-by-channel multiplication; is a channel attention mask, represents the channel attention mask when the input is , and the calculation method is represented as ; is a 1x1 convolution operation; is a Sigmoid activation function; To extract functions for pre-trained VGG-16 networks, Indicates input Subsequently, the high-level semantic features extracted by the pre-trained VGG-16 network, such as those related to bones and blood vessels, are enhanced to ensure that the degree of degradation of key anatomical structures is controllable and to avoid the loss of diagnostic information.

[0076] Optionally, the channel attention mask can be replaced with an anatomical attention map. Specifically, an anatomical attention map is generated using an nnUNet segmentation network pre-trained on medical images, which directly generates bone / organ masks and forces degradation constraints to be applied to key regions to preserve morphological features.

[0077] In one embodiment, the degradation process is visualized, such as... Figure 5 As shown, the original image is processed by a dynamic degradation kernel to obtain the degraded image. Directional blurring of bone edges and enhanced noise and texture in soft tissue areas are visible. In the anatomical attention map, highlighted areas correspond to bone boundaries, preventing the loss of important details during downsampling. The generated low-resolution image is the final degradation result after 4x downsampling, maintaining overall structural integrity while exhibiting controllable blurring of local details.

[0078] S303. Generate an enhanced image using a generator.

[0079] Traditional super-resolution generators directly upsample through deconvolution or interpolation, ignoring the spatial correlation of multi-scale anatomical structures in CT images, resulting in detail distortion.

[0080] This invention employs a degradation-aware residual correction generator and a dynamic upsampling mechanism guided by cross-scale feature fusion and degradation kernel, embedding degradation kernel information as prior knowledge into the generation process;

[0081] Specifically, the degradation artifacts are gradually eliminated through the residual correction module, while anatomical structure constraints are used to ensure the geometric topological consistency of key areas such as vascular bifurcation and calcification points.

[0082] The specific process is as follows:

[0083] 1) Perform degenerate kernel-guided feature extraction

[0084] The low-resolution image is concatenated with a dynamic degradation kernel to form a 6-channel input. Multi-scale features are extracted through cascaded degradation-aware convolutional blocks, each defined as follows:

[0085]

[0086] In the formula, For the first Layer feature map; For the first Layer feature map; is a positive integer; is a 3x3 convolution operation; is the degenerated kernel in the adaptive scaling version of the layer for feature extraction at different scales, i.e. ; represents adjusting the size of the degenerated kernel to times of the original one to adapt to the requirement of feature extraction at different scales; represents channel concatenation; is a LeakyReLU activation function with a slope of 0.2.

[0087] 2) Dynamic upsampling

[0088] An adaptive upsampling filter parameterized by the degenerated kernel is adopted to avoid edge blur caused by fixed interpolation kernels, which is represented as:

[0089]

[0090]

[0091] wherein, is the high-resolution feature map after dynamic upsampling; is the weight coefficient of the neighbor pixel at the position, which is dynamically calculated based on the degenerated kernel parameter and the coordinate encoding to avoid edge blur such as small blood vessel breakage caused by fixed interpolation kernels and to adaptively preserve high-frequency structures; represents the pixel value of the low-resolution image at the position; is a positive integer; is a positive integer; represents the target position in the low-resolution image corresponding to the neighbor pixel index; represents the low-resolution neighbor corresponding to the target position; is the coordinate encoding vector at the position, which is calculated as ; is the coordinate encoding vector at the position; is the horizontal coordinate of the position in the low-resolution image; is the vertical coordinate of the position in the low-resolution image; is the degenerated kernel parameter vector corresponding to the position; is the vector inner product.

[0092] 3) input the feature map and the up-sampled feature into the residual correction network

[0093] A cascaded residual correction unit is constructed, each unit containing a degenerative residual gate module and an anatomical structure constraint module;

[0094] The degenerative residual gate module generates gating weights through a degenerative kernel to suppress the transmission of residual errors in the artifact area, represented as:

[0095]

[0096] In the formula, represents the passing intensity of each pixel point residual signal, and a high value indicates that the residual is allowed to pass, and a low value indicates that the residual update of the region is suppressed; is an element-wise multiplication operation; is the feature map at the first layer, which is used as the input of the degenerative residual gate module; is the last layer number of the degenerative residual gate module;

[0097] The anatomical structure constraint module extracts a bone mask using a pre-trained U-Net segmentation network, applies an L1 constraint in the residual path, and the HU value of the bone region is stable and critical for diagnosis, forcing the generator to accurately reconstruct its geometric topology, represented as:

[0098]

[0099]

[0100] In the formula, is the bone mask extracted by the pre-trained U-Net segmentation network; is the anatomical structure constraint loss function; is the L1 norm; is an element-wise multiplication operation; represents the image after super-resolution reconstruction, which is the final output of the generator; is the high-resolution feature map after dynamic up-sampling; is the high-resolution target image, which is used to calculate the loss function; is the first residual correction unit, which is used to gradually eliminate degenerative artifacts and optimize image quality; is the total number of residual correction units, which is used to define the depth of the residual correction module in the network; r is a positive integer.

[0101] The generated is input to the multi-scale gradient discriminator, which forces The texture details and edge continuity are approximated to real high-resolution CT distribution.

[0102] It should be noted that the way of generating high-resolution images based on the degenerate kernel for the dynamic up-sampling of the generator is not easily thought of by those skilled in the art, and if the degenerate kernel guide is missing, the generator cannot correct different blur patterns of medical CT images.

[0103] In one embodiment, the up-sampling and residual correction are analyzed, as shown in Figure 6 As shown, the preliminary up-sampled image has a complete overall structure but blurred edges, the red area of the residual feature map represents the positive residual that needs to be enhanced, and the blue area represents the negative noise that needs to be suppressed, the residual correction image has a significantly improved skeleton edge sharpness, and part of the block artifacts in the area are suppressed, the white area of the skeleton area mask is the prominent structure, which is used to impose anatomical constraints in the residual correction process to ensure topological invariance and avoid geometric distortion caused by over-correction.

[0104] S304, image discrimination is performed by using the discriminator;

[0105] The discriminator of the conventional generative adversarial network adopts single-scale discrimination, which is difficult to capture the cross-scale feature correlation of different tissue structures in the CT image;

[0106] The present application adopts a multi-scale gradient discriminator to construct a double discrimination mechanism in the original image space and the gradient space, and forces the generator to simultaneously meet the medical image quality standards in texture details and edge continuity

[0107] Specifically, for the input image, first construct an image pyramid to obtain images at multiple resolutions, each scale is discriminated and scored by an independent Patch generative adversarial network sub-discriminator, at the same time, the gradient image of the image is extracted and input into an independent gradient domain discriminator, and the result for edge discrimination is output, then the multi-scale discrimination result and the gradient discrimination result are weighted in proportion, and the overall discrimination score is output, which is represented as:

[0108]

[0109] In the formula, is the comprehensive discrimination score of the multi-scale gradient discriminator, represents the input image The total score after multi-scale discrimination; is the input image of the discriminator, for each training batch, the discriminator simultaneously receives and ; is the number of pyramid scales, such as ; is a positive integer; is the image pyramid at the mth scale; is the weight coefficient of the first scale, and the calculation manner is represented as ; ; is an exponential function with a natural constant as a base; is a structural similarity index calculation function, represents a structural similarity index; is a reference image, that is, a real high-resolution image; is a sharpness adjustment coefficient, which controls the sharpness of the weight distribution, such as ; is a base discriminator based on a Patch generative adversarial network, represents the output of the first scale image through the base discriminator; is a balance coefficient, such as ; is a gradient domain discriminator, is the output of the gradient image through the gradient domain discriminator; is a gradient image, which enhances the edge continuity discrimination, and the calculation manner is represented as The gradient information of the tissue boundary in the CT image is crucial for diagnosis, such as the gradient information of the lung-mediastinum interface, and the gradient domain discriminator prevents edge blurring.

[0110] S305, loss function calculation of the generative adversarial network is performed

[0111] The traditional adversarial loss adopts global image-level discrimination, which is easy to cause over-smoothing of the detailed features for the micro-lesion region in the medical image;

[0112] The present application adopts a pixel-level adversarial loss based on a local window, calculates the adversarial loss in the local window, and independently calculates the adversarial loss of each local window to enhance the generation quality control of the micro-lesion region;

[0113] Specifically, the super-resolution image is divided into window regions of a fixed size, the authenticity score of each region is calculated by the discriminator, and then the average of all window scores is taken as the overall pixel-level adversarial loss, which enhances the ability of the model in local area detail modeling, and is represented as:

[0114]

[0115] In the formula, is a pixel-level adversarial loss; is the height of the local window, such as 32 pixels; is the width of the local window, such as 32 pixels; is the super-resolution reconstruction result of the first local window, which represents the super-resolution reconstruction result; is a positive integer, and denotes the local window index of ; is a logarithmic function with a default base of 10; denotes the discrimination score of the th local window.

[0116] S306, constructing a structural similarity constraint

[0117] To address the sensitivity of tissue density values in CT images, a weighted structural similarity loss based on Hounsfield units (HU) is used to preserve the density distribution characteristics of different tissues.

[0118] Specifically, the image is regionally divided according to the HU value, and the bones (HU>300) and soft tissues (-50<HU<300) are extracted as two region maps, respectively. Then the structural similarity of each region is calculated, and the overall structural similarity loss is obtained by weighting the area proportion of each region in the whole image, which is used to optimize the tissue density distribution of the generated image, denoted as:

[0119]

[0120] In the formula, is the structural similarity loss; denotes the two regions in the image divided according to the Hounsfield unit, i.e. the bone region and the soft tissue region; is the bone region segmented according to the Hounsfield unit;

[0121] is the soft tissue region segmented according to the Hounsfield unit; is the region weight, which is equal to the number of pixels in the bone or soft tissue region divided by the total number of pixels in the whole image.

[0122] Further, the structural similarity loss is calculated for the bone region (HU>300) and the soft tissue region (-50<HU<300) respectively to preserve the density distribution characteristics of different tissues.

[0123] The HU refers to the gray value unit in CT scanning, which is used to represent tissue density.

[0124] S307, optimizing a dynamic learning rate strategy

[0125] To balance the training progress of the generator and the discriminator, an adaptive learning rate adjustment mechanism based on gradient statistics is used. The generator learning rate is dynamically adjusted by real-time monitoring of the gradient amplitude ratio of the generator and the discriminator. When the discriminator is too powerful, the generator learning rate is automatically reduced.

[0126] Specifically, in each training cycle, the gradient norm expectation of the current generator and discriminator is calculated respectively as a sign of training dynamics, if the discriminator gradient is significantly greater than the generator, the generator learning rate is reduced to prevent it from falling into an optimization dilemma, otherwise the learning rate is maintained or slightly increased to make the generator converge quickly in the early stage of training, denoted as:

[0127]

[0128] wherein, is the adaptive learning rate of the generator in the th iteration; is the initial learning rate, the calculation method is represented as ; is a small constant to prevent the denominator from being 0, for example, set to .

[0129] S308, implement phase type training strategy

[0130] For the feature learning needs of different resolution levels of CT images, a three-stage training process is adopted:

[0131] 1) Stage one (iteration 0-1000 times): freeze the discriminator, minimize the generator;

[0132] The goal of stage one is to train the generator G to basically restore the structure of the high-resolution image similar to the structure, ensuring that the generator converges to the initial solution and is not misled by the weak discriminator; the loss function of stage one adopts structural similarity loss to encourage image structure restoration; the optimization strategy of stage one only updates the generator parameters , the discriminator is frozen, and the optimization objective is represented as ; is the generator parameter; denotes the generator parameter when minimizing the loss function; denotes the structural similarity loss when the input is and .

[0133] 2) Stage two (iteration 1000-5000 times): jointly optimize the total loss function ;

[0134] The goal of stage two is to add adversarial supervision to guide the generator to evolve in a more realistic direction, and gradually establish the adversarial game relationship between the generator and the discriminator;

[0135] In the loss function of stage two, the loss function of the generator is , and the loss function of the discriminator is ;

[0136] The optimization objective of the second stage training process is represented as: , and ;

[0137] represents the discriminator parameter when maximizing the loss function.

[0138] 3) Stage three (iterations 5000-10000): enable multi-scale gradient discriminator, optimize ;

[0139] The goal of the third stage is to improve the authenticity of texture details and edge structures; in the third stage, the discriminator simultaneously captures strong texture + edge (gradient domain) features.

[0140] The optimization objective of the generator in the third stage is ;

[0141] wherein, is the gradient domain adversarial loss, the calculation method of the gradient domain adversarial loss is as follows: first, gradient calculation is performed on the super-resolution image output by the generator and the real high-resolution image respectively, the horizontal and vertical gradient components are obtained through the Sobel operator, then the gradient amplitude of the two directions is squared and summed to obtain the gradient amplitude map, then the gradient maps of the real image and the generated image are input into independent gradient domain discriminators, the gradient domain discriminators output authenticity scores, then the Wasserstein adversarial loss is used to calculate the difference between the expected value of the real gradient map discrimination score and the expected value of the generated gradient map discrimination score as the gradient domain adversarial loss;

[0142] The independent gradient domain discriminator is composed of a multi-layer convolutional neural network, for example, a convolutional neural network with 3 layers of convolution.

[0143] The optimization objective of the discriminator in the third stage is to make the discrimination probability of and be approximately equal to 50%, that is, close to completely unable to distinguish and .

[0144] In one embodiment, to verify the effectiveness of the three-stage training strategy, experiments compare the changes in structural similarity index during the training process of the present technology and the conventional generative adversarial network, such as Figure 7As shown in the figure, the horizontal axis represents the number of training iterations, and the vertical axis represents the structural similarity index. The convergence process of both is illustrated by a line graph. Experiments show that conventional methods improve rapidly in the early stages of training but then stagnate because their single discriminative scale is insufficient to capture the multi-layered features of medical images. In contrast, the phased optimization strategy of this technique first freezes the discriminator to prioritize learning structural reconstruction, quickly establishing a basic mapping relationship. In the second stage, adversarial loss is introduced, and the generated image details gradually approach the real distribution. In the third stage, a multi-scale gradient discriminator is enabled, which strengthens edge continuity through gradient space constraints, ultimately causing the structural similarity index to converge to a higher level. The phase boundaries marked by the vertical dashed lines in the figure further demonstrate that the phased strategy can effectively balance the game relationship between the generator and the discriminator, avoiding premature entrapment in local optima.

[0145] S4. Perform super-resolution enhancement on the image;

[0146] The degenerate generative adversarial network model, once trained, can be directly applied to super-resolution enhancement of CT images or videos.

[0147] For a single CT image, the user inputs it into the model's generator module. The model performs dynamic upsampling and structural detail compensation based on its content and the degenerate kernel prior learned during training, outputting a clear, layered, high-resolution image while maintaining the original image's key anatomical structures such as bones and blood vessels without distortion.

[0148] For the enhancement processing of CT video sequences, images can be input frame by frame and a model can be used to generate a high-resolution frame sequence, or the inter-frame motion estimation and feature fusion modules can be combined to improve the image resolution of each frame while maintaining temporal continuity.

[0149] In one embodiment, such as Figure 8 As shown in the gradient map of the final enhanced image, the highlighted areas represent strong edge responses after reconstruction. Continuous and complete skeletal contours and vascular branching are visible in the image, verifying the effectiveness of the multi-scale gradient discriminator in constraining edge continuity. Compared to the original image, the final super-resolution enhanced image significantly improves the sharpness and contrast of key regions while maintaining the consistency of the HU value distribution. Through the synergistic effect of dynamic upsampling and residual correction, it eliminates blurring caused by interpolation and avoids the over-sharpening artifacts common in generative adversarial networks.

[0150] In the embodiment, a dynamic degradation kernel generation mechanism is realized by using image content self-adaptation, and a real simulation of the degradation process of medical CT images under different devices and tissue structures is realized. By using the degradation kernel guided dynamic up-sampling and residual correction network, the detail restoration ability and structure fidelity of the super-resolution image are significantly improved. A multi-scale gradient discriminator and a HU weighted structure similarity loss function are constructed, which effectively enhances the image texture reality and key tissue density consistency. By using the training dynamic and three-stage optimization strategy, the balance of the training game between the generator and the discriminator is realized, and the model stability and convergence quality are improved.

[0151] The above-described embodiments are only used to describe the preferred embodiments of the present application, and are not used to limit the scope of the present application. Without departing from the design spirit of the present application, various modifications and improvements to the technical solutions of the present application made by those skilled in the art shall fall within the protection scope of the present application defined by the claims.

Claims

1. A method for image and video super-resolution enhancement based on degenerate generative adversarial networks, characterized in that, include: Image acquisition; Acquire CT images and video sequences as high-resolution raw sample data; Image and video preprocessing; This includes image standardization, applying smoothing filters and local motion compensation between video frames; Super-resolution enhancement model construction and training; Super-resolution enhancement of CT images based on a degenerate generative adversarial network model; Perform super-resolution enhancement on images; The trained degenerate generative adversarial network model is used for super-resolution enhancement of CT images or videos; The construction and training process of the degenerate generative adversarial network model includes using a degradation-aware residual correction generator, employing a cross-scale feature fusion and degradation kernel-guided dynamic upsampling mechanism, and embedding degradation kernel information as prior knowledge into the generation process. The expression for the dynamic upsampling mechanism is as follows: ； ； In the formula, This is a high-resolution feature map after dynamic upsampling; for Position at the first The weight coefficients of each neighboring pixel are dynamically calculated based on the degenerate kernel parameters and coordinate encoding. Indicates the low-resolution image in the first... The pixel value at each position; It is a positive integer; It is a positive integer; Indicates the first The neighboring pixel index of the target location in the low-resolution image; express The low-resolution neighborhood corresponding to the target location; For the first The coordinate encoding vector of each position; For the first The coordinate encoding vector of each position; for The degenerate kernel parameter vector corresponding to the position.

2. The image and video super-resolution enhancement method based on degenerate generative adversarial networks according to claim 1, characterized in that, The construction and training process of the degradation generative adversarial network model includes constructing a multi-scale degradation kernel generation module. Specifically, a multi-scale degradation kernel generator based on dynamic convolution is used to generate degradation kernels related to image content by combining the local gradient distribution and frequency domain features of CT images.

3. The image and video super-resolution enhancement method based on degenerate generative adversarial networks according to claim 2, characterized in that, The method employs a multi-scale degradation kernel generator based on dynamic convolution, combining the local gradient distribution and frequency domain features of CT images to generate degradation kernels related to image content, including: The spatial gradient information is extracted and weighted convolution is performed using Gaussian kernels at multiple scales to obtain the degradation kernel response map. A three-layer perceptron is used to generate a noise pattern based on the average gray value of the entire image, and this pattern is weighted and fused with the convolution result to finally output a dynamic degradation kernel related to the image content.

4. The image and video super-resolution enhancement method based on degenerate generative adversarial networks according to claim 1, characterized in that, The dynamic upsampling mechanism employs an adaptive upsampling filter with degenerate kernel parameterization.

5. The image and video super-resolution enhancement method based on degenerate generative adversarial networks according to claim 1, characterized in that, The construction and training process of the degenerate generative adversarial network model includes the use of a multi-scale gradient discriminator to construct a dual discrimination mechanism in the original image space and gradient space.

6. The image and video super-resolution enhancement method based on degenerate generative adversarial networks according to claim 5, characterized in that, The multi-scale gradient discriminator includes constructing an image pyramid to acquire images at multiple resolutions, performing discrimination and scoring at each scale using an independent Patch Generative Adversarial Network (GAN) sub-discriminator, extracting the gradient map of the image and inputting it into an independent gradient domain discriminator, outputting the result for edge detection, and weighting the multi-scale discrimination result and the gradient discrimination result proportionally to output the overall discrimination score.

7. The image and video super-resolution enhancement method based on degenerate generative adversarial networks according to claim 1, characterized in that, The construction and training process of the degenerate generative adversarial network model includes a three-stage training process.

8. The image and video super-resolution enhancement method based on degenerate generative adversarial networks according to claim 7, characterized in that, In the three-stage training process, the first stage freezes the discriminator and minimizes it. Optimize the generator; the loss function of the generator in stage two is: The loss function of the discriminator is ; The optimization objective of the training process is represented as: ,and ; The third stage employs a multi-scale gradient discriminator, with the generator optimization objective being... ; in, For structural similarity loss; For pixel-level adversarial loss; These represent the generator parameters when minimizing the loss function; These represent the discriminator parameters when maximizing the loss function; Gradient domain adversarial loss.

Images