An image enhancement method and system based on semantic constraint degradation modeling

Abstract

This application discloses an image enhancement method and system based on semantic constraint degradation modeling. The method includes: extracting semantic masks and multi-scale degradation features from low-resolution images used for training; deeply fusing the extracted semantic masks and multi-scale degradation features using a dual-stream parallel architecture to generate semantic-structural fusion features; forming multimodal guiding conditions and using the multimodal guiding conditions and semantic-structural fusion features as input to reconstruct a high-resolution predicted image through a diffusion generation model; constructing a structural consistency optimization total loss based on the high-resolution predicted image and its corresponding target image, and optimizing the diffusion generation model based on the structural consistency optimization total loss; inputting the low-resolution image to be predicted into the optimized diffusion generation model to obtain its corresponding high-resolution image. Using the scheme of this application, a comprehensive and refined understanding of low-resolution images is achieved through multi-module collaboration and deep fusion.

Description

Technical Field

[0001] This application generally relates to the fields of computer vision and image processing technology. More specifically, this application relates to an image enhancement method and system based on semantic constraint degradation modeling. Background Technology

[0002] Image super-resolution technology aims to recover high-resolution images from low-resolution images. Traditional methods (such as interpolation and sparse representation) often fail to achieve good reconstruction results when faced with complex real-world application scenarios due to overly simplistic assumptions about the image degradation process.

[0003] In recent years, deep learning technologies, such as SRCNN, ESRGAN, and SwinIR, have achieved significant breakthroughs in the field of image super-resolution. However, most of these methods are trained on idealized degradation models (such as Bicubic downsampling), resulting in insufficient generalization ability and degraded reconstruction results when dealing with real-world images containing unknown blur, noise, and compression artifacts. To address this issue, research has gradually shifted towards real-world image super-resolution, primarily by constructing more realistic degradation models (such as Real-ESRGAN) or collecting real-world paired datasets (such as RealSR) to improve the model's real-world adaptability.

[0004] Against this backdrop, controllable generation techniques such as ControlNet were proposed. By adding additional conditional branches (such as edge maps and depth maps) to the pre-trained diffusion model, it achieves precise control over the structural information of the generated images. While ControlNet enhances the controllability of generation, it also introduces new problems: first, it relies on additional preprocessing to extract conditional data, increasing system complexity; second, it increases the computational cost and memory usage of the model, reducing inference speed; and third, it is highly sensitive to the quality of the conditional data, as incorrect conditional inputs can lead to distortion or artifacts in the generated images.

[0005] In view of this, there is an urgent need to provide an image enhancement scheme based on semantic constraint degradation modeling to solve the above problems and achieve a balance between the detail richness and structural fidelity of the generated image. Summary of the Invention

[0006] To address at least one or more of the technical problems mentioned above, this application proposes an image enhancement scheme based on semantic constraint degradation modeling in several aspects.

[0007] In a first aspect, this application provides an image enhancement method based on semantically constrained degradation modeling, comprising: inputting a low-resolution image used for training into a semantically guided generation module and a multi-level degradation feature modeling module respectively, and extracting semantic masks and multi-scale degradation features respectively; performing deep fusion of the extracted semantic masks and multi-scale degradation features based on a dual-stream parallel architecture to generate semantic-structural fusion features; forming multimodal guiding conditions, and using the multimodal guiding conditions and the semantic-structural fusion features together as input, and reconstructing a high-resolution predicted image through a diffusion generation model; constructing a structural consistency optimization total loss based on the high-resolution predicted image and its corresponding target image, and optimizing the diffusion generation model based on the structural consistency optimization total loss; inputting the low-resolution image to be predicted into the optimized diffusion generation model to obtain its corresponding high-resolution image.

[0008] In some embodiments, the following steps are performed during the extraction of semantic masks and multi-scale degradation features: the semantic guidance generation module analyzes the low-resolution image used for training based on a pre-trained semantic segmentation model and simultaneously generates a pixel-level semantic mask, which is used to mark a preset region in the image; the multi-level degradation feature modeling module, based on a pyramid convolution structure, extracts multi-scale degradation features of the low-resolution image used for training in parallel through degradation-aware convolution at multiple progressively decreasing spatial resolution scales, whereby the multi-scale degradation features include the image's blur intensity, noise distribution, and texture loss pattern.

[0009] In some embodiments, the multi-level degradation feature modeling module employs an intermediate reconstruction supervision mechanism, which performs the following steps: at each scale of the pyramid convolutional structure, the extracted multi-scale degradation features are restored to the RGB space through convolutional mapping to generate an intermediate reconstruction image; the intermediate reconstruction image is constrained and optimized with the high-resolution pyramid target image at the corresponding scale.

[0010] In some embodiments, the dual-stream parallel architecture includes an image branch and a semantic branch. During the deep fusion of the extracted semantic mask and multi-scale degradation features based on the dual-stream parallel architecture, the following steps are performed: The image branch receives low-resolution image features for training and the multi-scale degradation features; the multi-scale degradation features are used to guide the processing of the low-resolution image features for training to remove degradation effects and extract structural features; the semantic branch receives the semantic mask and uses a self-attention mechanism to extract and output discriminative semantic features; the discriminative semantic features are used as guiding information, and a cross-modal attention mechanism is used to weight and modulate the structural features to generate semantic-structural fusion features.

[0011] In some embodiments, the following steps are performed during the formation of multimodal guidance conditions: content analysis is performed on a low-resolution image used for training using a pre-trained visual-language model to generate a text description; the text description is jointly encoded with the semantic mask to form multimodal guidance conditions.

[0012] In some embodiments, during the process of reconstructing a high-resolution predicted image using a diffusion generative model, the following steps are performed: inputting the multimodal guidance conditions and the semantic-structural fusion features into a ControlNet network; the ControlNet network extracts multi-level latent space control features corresponding to the backbone network layers of the diffusion generative model; and fusing the multi-level latent space control features into the feature decoding process of the diffusion generative model to reconstruct the high-resolution image.

[0013] In some embodiments, during the feature decoding process of fusing the multi-level latent space control features into the diffusion generation model, the following steps are performed: the weights of each level of latent space control features in the multi-level latent space control features are adjusted through the zero convolutional layer in the ControlNet network; the adjusted latent space control features at each level are injected layer by layer into the corresponding decoder layer and intermediate layer of the diffusion generation model in the manner of residual addition.

[0014] In some embodiments, during the process of constructing the total loss for structural consistency optimization based on the high-resolution predicted image and its corresponding target image, the following steps are performed: The high-resolution predicted image and its corresponding target image are used as input pairs, and downsampled at multiple decreasing spatial scales to construct a multi-scale image pyramid; at each level of the image pyramid, for each pixel in the high-resolution predicted image and its corresponding target image, the Sobel operator is used to calculate its horizontal and vertical gradient components, thereby forming a multi-scale pyramidal gradient vector representation; for each level of the image pyramid, the gradient components corresponding to the high-resolution predicted image and the target image obtained at that level are used as inputs, and the orientation-sensitive structure-preserving loss at that level is calculated respectively; the orientation-sensitive structure-preserving losses of all levels are accumulated to obtain the total loss for structural consistency optimization.

[0015] In some embodiments, the formula for calculating the orientation-sensitive structure retention loss is: , To maintain loss for direction-sensitive structures, To predict the horizontal gradient vector at pixel i in a high-resolution image, To predict the gradient vector in the vertical direction at pixel i in a high-resolution image, Let be the gradient vector of the target image at pixel i in the horizontal direction. Let be the gradient vector in the vertical direction of the target image at pixel i, and N be the total number of pixels.

[0016] In a second aspect, this application provides an image enhancement system based on semantically constrained degradation modeling, employing the image enhancement method based on semantically constrained degradation modeling as described in any embodiment of the first aspect. The system includes: a semantically guided generation module, a multi-level degradation feature modeling module, a feature fusion module, a generation control module, a model optimization module, and a prediction module. The semantically guided generation module is used to extract semantic masks from low-resolution images used for training. The multi-level degradation feature modeling module is used to extract multi-scale degradation features from low-resolution images used for training. The feature fusion module is used to perform image enhancement based on dual-stream... The parallel architecture performs deep fusion of the extracted semantic mask and multi-scale degradation features to generate semantic-structural fusion features. The generation control module is used to form multimodal guiding conditions and use the multimodal guiding conditions and the semantic-structural fusion features as input to reconstruct the high-resolution prediction image through a diffusion generation model. The model optimization module is used to construct a structural consistency optimization total loss based on the high-resolution prediction image and its corresponding target image, and optimize the diffusion generation model based on the structural consistency optimization total loss. The prediction module is used to input the low-resolution image to be predicted into the optimized diffusion generation model to obtain its corresponding high-resolution image.

[0017] Through the image enhancement scheme based on semantic constraint degradation modeling provided above, this application embodiment achieves a comprehensive and refined understanding of low-resolution images through multi-module collaboration and deep fusion. It not only grasps the image content at the semantic level but also captures degradation features at multiple scales, deeply fusing high-level semantic information with low-level structural details to form extremely rich and instructive semantic-structural fusion features. These features, along with multimodal guiding conditions, serve as input to effectively drive advanced diffusion generation models for high-resolution image reconstruction, thereby fully leveraging the generation capabilities of diffusion models. By introducing structural consistency optimization total loss and orientation-sensitive structure preservation loss, the edges, textures, and geometric shapes of the generated images can be precisely constrained, ensuring that while improving resolution, the perceptual quality, structural fidelity, and detail recovery capabilities of the generated images are significantly enhanced. This effectively avoids the blurring, artifacts, and structural distortion problems common in traditional methods, thereby generating visually more natural and structurally more accurate high-resolution images.

[0018] Furthermore, in some embodiments, the understanding and processing capabilities for low-resolution images are significantly enhanced by introducing a semantically guided generation module and a multi-level degradation feature modeling module. The semantically guided generation module utilizes a pre-trained semantic segmentation model to accurately generate pixel-level semantic masks, providing valuable high-level semantic context and region priority information for high-resolution reconstruction. Simultaneously, the multi-level degradation feature modeling module employs a pyramid convolutional structure to extract multi-dimensional degradation features such as blur intensity, noise distribution, and texture loss patterns in parallel across multiple spatial scales, achieving comprehensive perception and refined modeling of complex degradation information. Moreover, this degradation feature modeling module, through an innovative intermediate reconstruction supervision mechanism, restores the extracted degradation features to intermediate images at each scale of the pyramid and performs constraint optimization with the corresponding targets. This not only ensures the effectiveness and reconstruction guidance of the learned degradation features but also greatly improves the model's generalization ability when handling complex, real-world degradation, as well as the structural fidelity and visual quality of the final generated image.

[0019] Furthermore, in some embodiments, an innovative dual-stream parallel architecture achieves deep decoupling and efficient fusion of image content semantics and structure, significantly improving the accuracy and robustness of super-resolution reconstruction. The image branch intelligently guides the processing of low-resolution image features using received multi-scale degradation features, accurately removing degradation effects and extracting purer, more accurate structural features. This ensures that the reconstructed basic structural information is unaffected by blurring, noise, or other interference. The semantic branch extracts discriminative semantic features from the semantic mask through a self-attention mechanism, effectively capturing the perceptual importance of different objects and regions in the image and providing high-level semantic guidance. In addition, by introducing a cross-modal attention mechanism, discriminative semantic features are used as guiding information to weight and modulate structural features. This ingenious fusion method ensures that semantic information can directly guide and optimize the expression of structural features, enabling the model to not only focus on the accuracy of details during reconstruction but also maintain semantic consistency.

[0020] Furthermore, in some embodiments, by fusing textual descriptions from a visual-language model with semantic masks to form multimodal guidance conditions, the model is endowed with deep semantic understanding capabilities beyond the pixel level, ensuring that the generated content conforms to the semantic context of the image. Through the ControlNet architecture, combined with unique zero-convolution weight adjustment and residual injection strategies, multimodal guidance information and semantic-structural features are integrated into the decoding process of the diffusion model in a non-destructive and highly smooth manner. This not only ensures stability in the early stages of training but also enables fine-grained control of the generation process at each level, effectively avoiding feature conflicts. By constructing multi-scale pyramid-shaped gradient vector representations and calculating orientation-sensitive structure-preserving loss, this method overcomes the limitations of single-scale supervision. It can enforce a high degree of consistency between the generated image and the target image in edge direction and geometric structure at every spatial scale, from macroscopic contours to microscopic textures. Attached Figure Description

[0021] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this application are illustrated by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts, wherein:

[0022] Figure 1 An exemplary flowchart of an image enhancement method based on semantic constraint degradation modeling according to an embodiment of this application is shown;

[0023] Figure 2 An exemplary flowchart of the present application embodiment is shown, which demonstrates the deep fusion of extracted semantic masks and multi-scale degradation features based on a dual-stream parallel architecture.

[0024] Figure 3 An exemplary flowchart illustrating the reconstruction of a high-resolution predicted image using a diffusion generation model according to an embodiment of this application is shown;

[0025] Figure 4 An exemplary flowchart illustrating the optimization of the total loss for structural consistency in an embodiment of this application is shown;

[0026] Figure 5 An exemplary structural block diagram of an image enhancement system based on semantic constraint degradation modeling according to an embodiment of this application is shown. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] It should be understood that the terms "comprising" and "including" used in the specification and claims of this application indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0030] Figure 1 An exemplary flowchart of an image enhancement method 100 based on semantic constraint degradation modeling according to an embodiment of this application is shown.

[0031] like Figure 1 As shown, in step S110, the low-resolution image used for training is input into the semantic guidance generation module and the multi-level degradation feature modeling module respectively, and semantic mask and multi-scale degradation features are extracted respectively.

[0032] In the embodiments of this application, during the extraction of semantic masks and multi-scale degradation features, the semantic-guided generation module analyzes the low-resolution image used for training based on a pre-trained semantic segmentation model and simultaneously generates a pixel-level semantic mask. The multi-level degradation feature modeling module, based on a pyramid convolutional structure, extracts multi-scale degradation features of the low-resolution image used for training in parallel through degradation-aware convolution at multiple progressively decreasing spatial resolution scales.

[0033] Specifically, semantic masks are used to mark predetermined regions in an image.

[0034] In the embodiments of this application, the preset area includes the outline of a person, the edge of a building, and other important visual objects.

[0035] In some embodiments of this application, multi-scale degradation features of the low-resolution image used for training are extracted in parallel through degradation-aware convolution at downsampling scales of 1 / 2, 1 / 4, and 1 / 8 of the original resolution. In other embodiments of this application, the spatial resolution scale can also be set according to actual needs and historical experience.

[0036] Specifically, multi-scale degradation features include the blur intensity, noise distribution, and texture loss patterns of the image.

[0037] In the embodiments of this application, the multi-level degradation feature modeling module adopts an intermediate reconstruction supervision mechanism. During the execution of this mechanism, firstly, at each scale of the pyramid convolutional structure, the extracted multi-scale degradation features are restored to the RGB space through convolutional mapping to generate intermediate reconstructed images. Secondly, the intermediate reconstructed images are constrained and optimized against the corresponding high-resolution pyramid target images, thereby ensuring that the extracted multi-scale degradation features are highly aligned with the real, high-quality, high-resolution pyramid target images.

[0038] Based on step S110, a dual-depth analysis of low-resolution images, from high-level semantic content to low-level physical degradation, is achieved. On one hand, a pre-trained model is used to generate pixel-level semantic masks to accurately locate key visual regions such as people and buildings, providing region-level importance priors for subsequent reconstruction and ensuring the detailed restoration of key objects. On the other hand, multi-scale degradation features are extracted in parallel through a pyramid convolutional structure, which can comprehensively capture blur intensity, noise distribution, and texture loss patterns at different spatial resolutions, significantly enhancing the model's ability to perceive degradation in complex real-world scenes. By introducing an intermediate reconstruction supervision mechanism, it is mandatory that the extracted degradation features can restore intermediate results aligned with real high-resolution images at various scales. This not only verifies the effectiveness of the features but also greatly corrects the biases in the feature extraction process, ensuring that the extracted features possess high physical realism and reconstruction guidance value, laying a solid foundation for the final generation of high-quality images.

[0039] After step S110 is completed, in step S120, the extracted semantic mask and multi-scale degradation features are deeply fused based on a dual-stream parallel architecture to generate semantic-structural fusion features.

[0040] In embodiments of this application, the dual-stream parallel architecture includes an image branch and a semantic branch.

[0041] In the embodiments of this application, the specific process of deep fusion of the extracted semantic mask and multi-scale degradation features based on a dual-stream parallel architecture can be found in [reference needed]. Figure 2 .

[0042] Figure 2 An exemplary flowchart illustrating the deep fusion of extracted semantic masks and multi-scale degradation features based on a dual-stream parallel architecture according to an embodiment of this application is shown.

[0043] like Figure 2As shown, in step S210, low-resolution image features and multi-scale degradation features for training are received through the image branch. The multi-scale degradation features are used to guide the processing of the low-resolution image features for training to remove degradation effects and extract structural features. In step S220, a semantic mask is received through the semantic branch, and a self-attention mechanism is used to extract and output discriminative semantic features. In step S230, the discriminative semantic features are used as guiding information, and the structural features are weighted and modulated through a cross-modal attention mechanism to generate semantic-structural fusion features.

[0044] Specifically, step S230 enables the generated semantic-structural fusion features to possess both structural details and semantic awareness.

[0045] Based on step S120, a dual-stream parallel architecture was constructed to achieve deep decoupling and collaborative optimization of image physical structure information and high-level semantic information. Specifically, the image branch cleverly utilizes multi-scale degradation features as priors to specifically guide the restoration of low-resolution features, effectively removing degradation effects such as noise and blur, thereby extracting pure and accurate structural features. The semantic branch, on the other hand, leverages a self-attention mechanism to deeply mine semantic masks, generating discriminative semantic features to pinpoint the perceptual focus of image content. A cross-modal attention mechanism organically combines the two, using semantic features to finely weight and modulate structural features. This mechanism not only ensures the quality of physical structure restoration but also endows low-level features with semantic perception capabilities, enabling the final generated semantic-structural fusion features to maintain geometric detail accuracy while possessing deep semantic consistency, providing strong support rich in contextual information for subsequent high-fidelity image generation.

[0046] After step S120 is completed, in step S130, multimodal guiding conditions are formed, and the multimodal guiding conditions and semantic-structural fusion features are used as input to reconstruct the high-resolution predicted image through a diffusion generation model.

[0047] In the embodiments of this application, during the formation of multimodal guidance conditions, firstly, a pre-trained visual-language model is used to perform content analysis on the low-resolution image used for training to generate a text description. Secondly, the text description and a semantic mask are jointly encoded to form the multimodal guidance conditions.

[0048] In the embodiments of this application, the specific process of reconstructing a high-resolution predicted image using a diffusion generation model can be found in [reference needed]. Figure 3 .

[0049] Figure 3 An exemplary flowchart illustrating the reconstruction of a high-resolution predicted image using a diffusion generation model according to an embodiment of this application is shown.

[0050] like Figure 3 As shown, in step S310, the multimodal guidance conditions and semantic-structural fusion features are input into the ControlNet network. In step S320, the ControlNet network extracts multi-level latent space control features corresponding to the backbone network layers of the diffusion generation model. In step S330, the multi-level latent space control features are fused into the feature decoding process of the diffusion generation model to reconstruct the high-resolution image.

[0051] In embodiments of this application, the ControlNet network extracts multi-level latent space control features through its internal trainable encoder copies.

[0052] In the embodiments of this application, during the feature decoding process of fusing multi-level latent space control features into the diffusion generative model, firstly, the weights of each level of latent space control features in the multi-level latent space control features are adjusted through zero convolutional layers in the ControlNet network. Secondly, the adjusted latent space control features at each level are injected layer by layer into the corresponding decoder layer and intermediate layer of the diffusion generative model by adding residuals.

[0053] Based on step S130, by introducing multimodal guidance conditions and the ControlNet fine-grained control mechanism, image reconstruction is elevated to a new level of semantic understanding and generative control. First, by combining text descriptions generated by a vision-language model with semantic masks, multimodal guidance conditions rich in deep textual semantic context are constructed, overcoming the limitations of single visual features. Second, multi-level latent space control features are extracted using the ControlNet architecture, and a strategy of zero-convolutional layer weight adjustment and residual layer-by-layer injection is adopted to achieve non-destructive and highly stable implantation into the diffusion model decoding process. The advantage of this mechanism is that it not only uses zero convolution to ensure numerical stability in the early stages of training, but also achieves comprehensive and precise control over the generation process from coarse-grained to fine-grained through multi-level residual connections, ensuring that the final reconstructed high-resolution image is not only rich in detail, but also highly consistent with the input image in terms of semantic logic and structural layout.

[0054] After completing step S130, in step S140, a structural consistency optimization total loss is constructed based on the high-resolution predicted image and its corresponding target image, and the diffusion generation model is optimized based on the structural consistency optimization total loss.

[0055] In the embodiments of this application, the specific process involved in constructing structural consistency optimization of the total loss based on the high-resolution predicted image and its corresponding target image can be found in [reference needed]. Figure 4 .

[0056] Figure 4 An exemplary flowchart illustrating the total loss of structural consistency optimization in an embodiment of this application is shown.

[0057] like Figure 4 As shown, in step S410, the high-resolution prediction image and its corresponding target image are used as input pairs, and downsampling is performed at multiple decreasing spatial scales to construct a multi-scale image pyramid. In step S420, at each level of the image pyramid, for each pixel in the high-resolution prediction image and its corresponding target image, the Sobel operator is used to calculate its horizontal and vertical gradient components, thereby forming a multi-scale pyramid-shaped gradient vector representation. In step S430, for each level of the image pyramid, the gradient components corresponding to the high-resolution prediction image and the target image obtained at that level are used as inputs to calculate the orientation-sensitive structure preservation loss at that level. In step S440, the orientation-sensitive structure preservation losses of all levels are accumulated to obtain the total structure consistency optimization loss.

[0058] In the embodiments of this application, the formula for calculating the orientation-sensitive structure retention loss is as follows: , To maintain loss for direction-sensitive structures, To predict the horizontal gradient vector at pixel i in a high-resolution image, To predict the gradient vector in the vertical direction at pixel i in a high-resolution image, Let be the gradient vector of the target image at pixel i in the horizontal direction. Let be the gradient vector in the vertical direction of the target image at pixel i, and N be the total number of pixels.

[0059] In the embodiments of this application, the formula for calculating the total loss of structural consistency optimization is as follows: , To optimize the total loss for structural consistency, M represents the total number of levels in the image pyramid. Preserve the orientation-sensitive structure loss for the k-th layer of the image pyramid.

[0060] In the embodiments of this application, during the process of optimizing the diffusion generation model based on the total loss of structural consistency optimization, the diffusion generation model is backpropagated through the total loss of structural consistency optimization, and the parameters of the diffusion generation model are updated through the gradient descent algorithm. This enables the diffusion generation model to adaptively reconstruct a high-resolution image that is structurally highly consistent with the target image at all scales, with sharp edges and accurate orientation in the subsequent generation process.

[0061] Based on step S140, a multi-scale, orientation-sensitive structural consistency supervision system is constructed, significantly improving the geometric fidelity and visual clarity of the generated images. Its core advantage lies in overcoming the limitations of traditional pixel-level loss functions. By using an image pyramid, the constraint range covers every spatial scale from macroscopic contours to microscopic textures, ensuring the consistency of reconstruction results across different resolutions. Simultaneously, the Sobel operator is used to construct a pyramid-shaped gradient vector representation and calculate the orientation-sensitive structure-preserving loss, ensuring that the optimization objective is closely focused on the directionality of edges and the accuracy of geometric topology. This not only effectively suppresses edge blurring and artifacts common in super-resolution tasks but also forces the model to strictly adhere to the structural rules of the target image during generation, thereby reconstructing high-quality images with sharp edges and accurate structure.

[0062] After step S140 is completed, in step S150, the low-resolution image to be predicted is input into the optimized diffusion generation model to obtain its corresponding high-resolution image.

[0063] In summary, through the image enhancement scheme based on semantic constraint degradation modeling provided above, the embodiments of this application achieve a comprehensive and refined understanding of low-resolution images through multi-module collaboration and deep fusion. It not only grasps the image content at the semantic level but also captures degradation features at multiple scales, deeply fusing high-level semantic information with low-level structural details to form extremely rich and instructive semantic-structural fusion features. These features, together with multimodal guiding conditions, serve as input to effectively drive advanced diffusion generation models for high-resolution image reconstruction, thereby fully leveraging the generation capabilities of diffusion models. By introducing structural consistency optimization total loss and orientation-sensitive structure preservation loss, the edges, textures, and geometric shapes of the generated images can be precisely constrained, ensuring that while improving resolution, the perceptual quality, structural fidelity, and detail recovery capabilities of the generated images are significantly enhanced. This effectively avoids the blurring, artifacts, and structural distortion problems common in traditional methods, thereby generating visually more natural and structurally more accurate high-resolution images.

[0064] Furthermore, in some embodiments, the understanding and processing capabilities for low-resolution images are significantly enhanced by introducing a semantically guided generation module and a multi-level degradation feature modeling module. The semantically guided generation module utilizes a pre-trained semantic segmentation model to accurately generate pixel-level semantic masks, providing valuable high-level semantic context and region priority information for high-resolution reconstruction. Simultaneously, the multi-level degradation feature modeling module employs a pyramid convolutional structure to extract multi-dimensional degradation features such as blur intensity, noise distribution, and texture loss patterns in parallel across multiple spatial scales, achieving comprehensive perception and refined modeling of complex degradation information. Moreover, this degradation feature modeling module, through an innovative intermediate reconstruction supervision mechanism, restores the extracted degradation features to intermediate images at each scale of the pyramid and performs constraint optimization with the corresponding targets. This not only ensures the effectiveness and reconstruction guidance of the learned degradation features but also greatly improves the model's generalization ability when handling complex, real-world degradation, as well as the structural fidelity and visual quality of the final generated image.

[0065] Furthermore, in some embodiments, an innovative dual-stream parallel architecture achieves deep decoupling and efficient fusion of image content semantics and structure, significantly improving the accuracy and robustness of super-resolution reconstruction. The image branch intelligently guides the processing of low-resolution image features using received multi-scale degradation features, accurately removing degradation effects and extracting purer, more accurate structural features. This ensures that the reconstructed basic structural information is unaffected by blurring, noise, or other interference. The semantic branch extracts discriminative semantic features from the semantic mask through a self-attention mechanism, effectively capturing the perceptual importance of different objects and regions in the image and providing high-level semantic guidance. In addition, by introducing a cross-modal attention mechanism, discriminative semantic features are used as guiding information to weight and modulate structural features. This ingenious fusion method ensures that semantic information can directly guide and optimize the expression of structural features, enabling the model to not only focus on the accuracy of details during reconstruction but also maintain semantic consistency.

[0066] Furthermore, in some embodiments, by fusing textual descriptions from a visual-language model with semantic masks to form multimodal guidance conditions, the model is endowed with deep semantic understanding capabilities beyond the pixel level, ensuring that the generated content conforms to the semantic context of the image. Through the ControlNet architecture, combined with unique zero-convolution weight adjustment and residual injection strategies, multimodal guidance information and semantic-structural features are integrated into the decoding process of the diffusion model in a non-destructive and highly smooth manner. This not only ensures stability in the early stages of training but also enables fine-grained control of the generation process at each level, effectively avoiding feature conflicts. By constructing multi-scale pyramid-shaped gradient vector representations and calculating orientation-sensitive structure-preserving loss, this method overcomes the limitations of single-scale supervision. It can enforce a high degree of consistency between the generated image and the target image in edge direction and geometric structure at every spatial scale, from macroscopic contours to microscopic textures.

[0067] This application also provides an image enhancement system based on semantic constraint degradation modeling. It can use the aforementioned image enhancement method 100 based on semantic constraint degradation modeling to perform image enhancement based on semantic constraint degradation modeling, or it can use other methods to perform image enhancement based on semantic constraint degradation modeling. This application does not limit this.

[0068] Figure 5 An exemplary structural block diagram of an image enhancement system based on semantic constraint degradation modeling according to an embodiment of this application is shown.

[0069] like Figure 5 As shown, the system 500 includes a semantic guidance generation module 510, a multi-level degradation feature modeling module 520, a feature fusion module 530, a generation control module 540, a model optimization module 550, and a prediction module 560.

[0070] Specifically, the semantic guidance generation module 510 is used to extract a semantic mask from the low-resolution image used for training.

[0071] Specifically, the multi-level degradation feature modeling module 520 is used to extract multi-scale degradation features from the low-resolution image used for training.

[0072] Specifically, the feature fusion module 530 is used to perform deep fusion of the extracted semantic mask and multi-scale degradation features based on a dual-stream parallel architecture to generate semantic-structural fusion features.

[0073] Specifically, the generation control module 540 is used to form multimodal guiding conditions and take the multimodal guiding conditions and semantic-structural fusion features as input to reconstruct high-resolution predicted images through a diffusion generation model.

[0074] Specifically, the model optimization module 550 is used to construct a structural consistency optimization total loss based on the high-resolution predicted image and its corresponding target image, and to optimize the diffusion generation model based on the structural consistency optimization total loss.

[0075] Specifically, the prediction module 560 is used to input the low-resolution image to be predicted into the optimized diffusion generation model to obtain its corresponding high-resolution image.

[0076] When system 500 performs image enhancement based on semantic constraint degradation modeling using the aforementioned image enhancement method 100, the aforementioned step S110 is executed through the semantic guidance generation module 510 and the multi-level degradation feature modeling module 520, the aforementioned step S120 is executed through the feature fusion module 530, the aforementioned step S130 is executed through the generation control module 540, the aforementioned step S140 is executed through the model optimization module 550, and the step S150 is executed through the prediction module 560. The specific execution process can be found above and will not be repeated here.

[0077] While numerous embodiments of this application have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will arise for those skilled in the art without departing from the spirit and intent of this application. It should be understood that various alternatives to the embodiments of this application described herein may be employed in the practice of this application. The appended claims are intended to define the scope of protection of this application and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. An image enhancement method based on semantic constraint degradation modeling, characterized in that, The method comprises the following steps: input low-resolution images for training into a semantic-guided generation module and a multi-level degradation feature modeling module respectively to extract semantic masks and multi-scale degradation features respectively; deeply fuse the extracted semantic masks and multi-scale degradation features based on a double-flow parallel architecture to generate semantic-structure fusion features; form a multi-modal guidance condition and use the multi-modal guidance condition and the semantic-structure fusion features as inputs to reconstruct high-resolution predicted images through a diffusion generative model; construct a structure consistency optimization total loss based on the high-resolution predicted images and their corresponding target images, and optimize the diffusion generative model based on the structure consistency optimization total loss; input low-resolution images to be predicted into the optimized diffusion generative model to obtain their corresponding high-resolution images; wherein the double-flow parallel architecture comprises an image branch and a semantic branch; the following steps are performed in the process of deeply fusing the extracted semantic masks and multi-scale degradation features based on the double-flow parallel architecture: receive low-resolution image features for training and the multi-scale degradation features through the image branch, and use the multi-scale degradation features to guide the processing of the low-resolution image features for training to remove the degradation effect and extract structural features; receive the semantic masks through the semantic branch and extract and output discriminative semantic features using a self-attention mechanism; use the discriminative semantic features as guidance information to weight and modulate the structural features through a cross-modal attention mechanism to generate semantic-structure fusion features; the following steps are performed in the process of constructing a structure consistency optimization total loss based on high-resolution predicted images and their corresponding target images: downsample the high-resolution predicted images and their corresponding target images at multiple decreasing spatial scales to construct a multi-scale image pyramid; on each level of the image pyramid, calculate the gradient components in the horizontal and vertical directions of each pixel in the high-resolution predicted images and their corresponding target images using a Sobel operator to form a multi-scale pyramid gradient vector representation; for each level of the image pyramid, use the gradient components of the high-resolution predicted images and target images obtained on the level as inputs to calculate their direction-sensitive structure preservation losses on the level respectively; add all the direction-sensitive structure preservation losses of all levels to obtain the structure consistency optimization total loss; the calculation formula of the direction-sensitive structure preservation loss is: , is the loss for the directionally sensitive structure, is the gradient vector in the horizontal direction at pixel i for the high-resolution predicted image, is the gradient vector in the vertical direction at pixel i for the high-resolution predicted image, is the gradient vector in the horizontal direction at pixel i for the target image, is the gradient vector in the vertical direction at pixel i for the target image, and N is the total number of pixels.

2. The image enhancement method based on semantic constraint degradation modeling according to claim 1, characterized in that, in the process of extracting semantic masks and multi-scale degradation features, the following steps are performed: the semantic-guided generation module analyzes the low-resolution images for training based on a pre-trained semantic segmentation model and synchronously generates pixel-level semantic masks, which are used to mark preset regions in the images; The multi-level degradation feature modeling module extracts multi-scale degradation features of the low-resolution image for training through degradation-aware convolution in parallel at multiple spatial resolution scales gradually decreasing in level based on a pyramid convolution structure, the multi-scale degradation features including blur intensity, noise distribution, and texture loss patterns of the image.

3. The image enhancement method based on semantic constraint degradation modeling according to claim 2, characterized in that, The multi-level degradation feature modeling module adopts an intermediate reconstruction supervision mechanism, which performs the following steps: At each scale of the pyramid convolution structure, the extracted multi-scale degradation features are restored to the RGB space through convolution mapping to generate an intermediate reconstruction image; The intermediate reconstruction image is constrained and optimized with the high-resolution pyramid target image at the corresponding scale.

4. The image enhancement method based on semantic constraint degradation modeling according to claim 1, characterized in that, In the process of forming the multi-modal guidance condition, the following steps are performed: The low-resolution image for training is analyzed for content through a pre-trained visual-language model to generate a text description; The text description is jointly encoded with the semantic mask to form the multi-modal guidance condition.

5. The image enhancement method based on semantic constraint degradation modeling according to claim 1 or 4, characterized in that, In the process of reconstructing the high-resolution predicted image through the diffusion generative model, the following steps are performed: The multi-modal guidance condition and the semantic-structure fusion feature are input into the ControlNet network; The ControlNet network extracts multi-level latent space control features corresponding to the backbone network levels of the diffusion generative model; The multi-level latent space control features are fused into the feature decoding process of the diffusion generative model to reconstruct the high-resolution image.

6. The image enhancement method based on semantic constraint degradation modeling according to claim 5, characterized in that, In the process of fusing the multi-level latent space control features into the feature decoding process of the diffusion generative model, the following steps are performed: The weight of each level of the multi-level latent space control features is adjusted through a zero convolution layer in the ControlNet network; The adjusted multi-level latent space control features are injected into the corresponding decoder layers and intermediate layers of the diffusion generative model layer by layer in a residual addition manner.

7. An image enhancement system based on semantic constraint degradation modeling, characterized in that, An image enhancement method based on semantic constraint degradation modeling is adopted, the system comprising a semantic guidance generation module, a multi-level degradation feature modeling module, a feature fusion module, a generation control module, a model optimization module, and a prediction module; The semantic guidance generation module is used to extract a semantic mask from the low-resolution image for training; The multi-level degradation feature modeling module is used to extract multi-scale degradation features from the low-resolution image for training; The feature fusion module is used to deeply fuse the extracted semantic mask and multi-scale degradation features based on a dual-flow parallel architecture to generate semantic-structure fusion features; The generation control module is used to form a multi-modal guidance condition and input the multi-modal guidance condition and the semantic-structure fusion features into the diffusion generative model to reconstruct a high-resolution predicted image; The model optimization module is used to construct a structure consistency optimization total loss based on the high-resolution predicted image and its corresponding target image, and optimize the diffusion generative model based on the structure consistency optimization total loss. The prediction module is configured to input a low-resolution image to be predicted into the optimized diffusion generation model to obtain a corresponding high-resolution image.

Images