End-to-end image restoration method and system based on image-text feature mapping, and medium

Abstract

The invention discloses an end-to-end image recovery method and system based on image-text feature mapping and a medium, and relates to the technical field of image processing.The method comprises the steps that a sample set comprising degraded and clean images is called, and a four-layer feature space is established through an image-text feature mapping mechanism; analyzing the degradation features by adopting an isometric tight frame, constructing a prompt vector set, and guiding a feature space to carry out adaptive dynamic prompt fusion to form a dynamic prompt vector set; a first model is established through image recovery loss function iterative training, then an AdamW optimizer is used for gradient descent optimization to obtain a second model, and finally image recovery processing is executed. The technical problem that an existing image restoration method is unstable in restoration effect and poor in adaptability in the multi-degradation scene is solved, and the technical effect that the image restoration precision and adaptability in the multi-degradation scene are improved by introducing a dynamic prompt generation and hierarchical fusion mechanism is achieved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an end-to-end image restoration method, system, and medium based on image and text feature mapping. Background Technology

[0002] Existing image restoration techniques typically target single degradation types, such as deraining, dehazing, and denoising, with independently designed models, making it difficult to achieve unified restoration of multiple degradation scenarios within a single network. Some all-in-one methods based on cue learning, such as PromptIR, while introducing cue vector mechanisms, have statically fixed cue parameters that cannot adaptively adjust according to the degradation features of the input image, leading to performance degradation in complex degradation scenarios. On the other hand, pre-training methods based on degradation classifiers, such as MiOIR, rely on a large amount of degradation label data and lack sufficient generalization ability for unknown degradation types. Summary of the Invention

[0003] This application provides an end-to-end image restoration method, system, and medium based on image and text feature mapping, which solves the technical problems of existing image restoration methods where the prompt parameters are fixed and cannot be dynamically adjusted according to the input degradation features, resulting in unstable restoration effects and poor adaptability in multiple degradation scenarios.

[0004] The first aspect of this application provides an end-to-end image restoration method based on image-text feature mapping. The method includes: retrieving an image restoration processing sample set, which includes a degraded image sample set and a clean image sample set; performing hierarchical feature detection on the degraded image sample set according to an image-text feature mapping mechanism to establish a four-layer feature space for the degraded image; performing degradation prototype feature parsing on the four-layer feature space of the degraded image according to an isoangular compact frame to construct a first degradation feature hint vector set; guiding the four-layer feature space of the degraded image to perform adaptive hierarchical dynamic hint fusion based on the first degradation feature hint vector set to establish a dynamic degradation feature hint vector set; iteratively training the degraded image sample set, the dynamic degradation feature hint vector set, and the clean image sample set using an image restoration loss function to establish a first image restoration model; optimizing the first image restoration model using gradient descent according to the AdamW optimizer to obtain a second image restoration model, and performing image restoration processing based on the second image restoration model.

[0005] A second aspect of this application provides an end-to-end image restoration system based on image-text feature mapping. The system includes: an image sample retrieval module for retrieving an image restoration processing sample set, which includes a degraded image sample set and a clean image sample set; a hierarchical feature detection module for performing hierarchical feature detection on the degraded image sample set according to an image-text feature mapping mechanism to establish a four-layer feature space for the degraded image; a degraded prototype feature parsing module for performing degraded prototype feature parsing on the four-layer feature space of the degraded image according to an isoangular tight frame to construct a first degraded feature hint vector set; and dynamic hint fusion. The system comprises a fusion module, used to guide the four-layer feature space of the degraded image to perform adaptive hierarchical dynamic prompt fusion based on the first degraded feature prompt vector set, and to establish a dynamic degraded feature prompt vector set; an iterative training module, used to call the image restoration loss function to iteratively train the degraded image sample set, the dynamic degraded feature prompt vector set, and the clean image sample set to establish an image restoration first model; and an image restoration processing module, used to perform gradient descent optimization on the image restoration first model based on the AdamW optimizer to obtain an image restoration second model, and to perform image restoration processing based on the image restoration second model.

[0006] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of the first aspect.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application provides an end-to-end image restoration method, system, and medium based on image-text feature mapping, which relates to the field of image processing technology. Through image-text feature mapping, isoangular tight frame parsing, and an adaptive hierarchical dynamic prompting mechanism, it performs multi-layer feature modeling and dynamic prompt fusion on degraded image samples. Combined with a custom image restoration loss function and the AdamW optimization algorithm, it achieves end-to-end unified image restoration under multiple degradation types. This solves the technical problems of existing image restoration methods where prompt parameters are fixed and cannot be dynamically adjusted according to input degradation features, resulting in unstable restoration effects and poor adaptability in multi-degradation scenarios. By introducing a dynamic prompt generation and hierarchical fusion mechanism, the prompt vector can adaptively match different degradation features, thereby improving the accuracy and adaptability of image restoration in multi-degradation scenarios. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A schematic diagram of the end-to-end image restoration method based on image and text feature mapping provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of an end-to-end image restoration system based on image and text feature mapping provided in an embodiment of this application.

[0010] Figure labeling: Image sample retrieval module 11, hierarchical feature detection module 12, degenerate prototype feature parsing module 13, dynamic prompt fusion module 14, iterative training module 15, image restoration processing module 16. Detailed Implementation

[0011] This application provides an end-to-end image restoration method, system, and medium based on image and text feature mapping, which solves the technical problems of existing image restoration methods where the prompt parameters are fixed and cannot be dynamically adjusted according to the input degradation features, resulting in unstable restoration effects and poor adaptability in multiple degradation scenarios.

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0013] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0014] Example 1, as Figure 1 As shown, this application provides an end-to-end image restoration method based on image-text feature mapping, the method comprising: P10: Retrieve the image restoration processing sample set, which includes a degraded image sample set and a clean image sample set.

[0015] Specifically, the image restoration processing sample set is the key data set used to train the image restoration model in this application. It consists of two parts: a degraded image sample set and a clean image sample set. The degraded image sample set refers to the set of images affected by various degradation factors, including but not limited to rain, fog, noise, blur, and low light. These factors lead to a decrease in image quality, causing details and structural information in the image to become blurred or lost. The clean image sample set refers to the set of images that have not been affected by any degradation. These images usually have high quality and can clearly display various details and structural information in the image.

[0016] In this application, small batches of data are extracted from the sample set. ,in This represents the degraded image data to be recovered. Here, B represents the corresponding clean image data, and B indicates the batch size, i.e., the number of images fed into the network at one time. This method of extracting small batches of data helps improve the efficiency and stability of model training, while also enabling the model to better learn the restoration patterns of images with different degrees of degradation and types.

[0017] Furthermore, to improve the model's generalization ability and robustness, the degraded image samples are derived from multiple publicly available standard datasets, such as the denoising dataset BSD68, the dehazing dataset SOTS, and the deraining dataset RAIN100L. Through multi-source sample fusion, the model can achieve a consistent restoration effect under different degradation scenarios. By accessing such an image restoration processing sample set, this application enables the model to learn during training how to restore degraded image samples to the state of their corresponding clean image samples, thereby achieving effective restoration of various degraded images.

[0018] P20: Based on the image-text feature mapping mechanism, perform hierarchical feature detection on the degraded image sample set to establish a four-layer feature space for the degraded image.

[0019] Furthermore, step P20 in this embodiment of the application also includes: P21: Activate the image-text feature mapping mechanism, which includes a predetermined convolutional layer, a first-scale visual Mamba module, a second-scale visual Mamba module, a third-scale visual Mamba module, and a fourth-scale visual Mamba module; P22: Perform convolution processing on the degraded image sample set according to the predetermined convolutional layer to obtain a degraded image convolutional feature set; P23: Perform scale-adaptive feature detection on the degraded image convolutional feature set according to the first-scale visual Mamba module to generate a first-layer feature space of the degraded image; P24: Perform scale-adaptive feature detection on the first-layer feature space of the degraded image according to the second-scale visual Mamba module, the third-scale visual Mamba module, and the fourth-scale visual Mamba module respectively to obtain a second-layer feature space, a third-layer feature space, and a fourth-layer feature space of the degraded image; P25: Generate a fourth-layer feature space of the degraded image based on the first-layer feature space, the second-layer feature space, the third-layer feature space, and the fourth-layer feature space of the degraded image.

[0020] It should be understood that, based on the image-text feature mapping mechanism, hierarchical feature detection is performed on the degraded image sample set to establish a four-layer feature space for the degraded image, thereby providing multi-scale semantic support for subsequent degraded prototype analysis and adaptive prompt generation.

[0021] Specifically, the image-text feature mapping mechanism is first activated. This mechanism includes predefined convolutional layers and visual Mamba modules at multiple scales, including a first-scale visual Mamba module, a second-scale visual Mamba module, a third-scale visual Mamba module, and a fourth-scale visual Mamba module. The visual Mamba module is a feature extraction unit based on visual sequence modeling. Its structure is similar to an improved Transformer, but it employs a state-space sequence modeling mechanism for information propagation, enabling global feature capture and context-dependent awareness with lower computational cost.

[0022] Next, the pre-defined convolutional layer uses a 3×3 kernel to perform preliminary processing on the degraded image sample set, extracting basic image features such as edges and textures. This small-sized kernel can effectively capture local feature information of the image while maintaining computational efficiency.

[0023] Subsequently, the first-scale vision Mamba module performs scale-adaptive feature detection on the convolutional feature set of the degraded image after processing through predetermined convolutional layers. Through normalization (Norm) and multilayer perceptron (MLP) processing, the model can identify fine-grained differences in the degraded signal within a smaller receptive field, generating the first-layer feature space of the degraded image. This layer mainly contains shallow features of the image, such as local texture and edge information, providing a foundation for subsequent feature detection. Based on the first-layer feature space, the second-scale vision Mamba module further extracts mid-layer features of the image. These features can characterize a larger range of structural information in the image, helping the model better understand the overall layout and semantic content of the image, thus generating the second-layer feature space of the degraded image.

[0024] Next, the third-scale visual Mamba module continues feature detection at higher scales, generating a third-layer feature space for the degraded image. The features in this layer are more abstract, capturing high-level semantic information in the image, which is crucial for understanding complex scenes and objects within the image. Finally, the fourth-scale visual Mamba module, as the final scale feature detection module, generates a fourth-layer feature space for the degraded image. The features in this layer are the most abstract, providing a deep understanding of the image's global structure and semantics, supporting high-level decision-making during image restoration.

[0025] In other words, the second layer of features focuses on the structural outline and the morphology of the degraded region, the third layer of features focuses on global semantics and spatial relationships, and the fourth layer of features integrates cross-scale semantics to achieve global perception and abstract representation of complex degradation patterns.

[0026] Finally, the first, second, third, and fourth layer feature spaces of the degraded image are integrated to form a complete four-layer feature space. This four-layer feature space contains rich feature information from shallow to deep and from local to global, preserving both the detailed features of the degraded region and possessing macroscopic scene understanding capabilities. It can provide multi-scale input support for the subsequent degradation prototype feature parsing based on the equiangular compact frame (ETF). Through this hierarchical feature detection and integration approach, the model can better understand and process various degraded images, thereby achieving more effective image restoration results.

[0027] Furthermore, step P23 in this embodiment of the application also includes: P23-1: Normalize the degraded image convolutional feature set using a norm layer to obtain a first normalized feature set; P23-2: Input the first normalized feature set into the first-scale visual Mamba module to obtain a second normalized feature set; P23-3: Perform a residual connection between the degraded image convolutional feature set and the second normalized feature set to obtain a third normalized feature set; P23-4: Perform norm layer normalization based on the third normalized feature set to obtain a fourth normalized feature set; P23-5: Perform a nonlinear mapping on the fourth normalized feature set using a two-layer MLP network to obtain a fifth normalized feature set; P23-6: Perform a residual connection between the degraded image convolutional feature set and the fifth normalized feature set to obtain the first-layer feature space of the degraded image.

[0028] Optionally, the process of performing scale-adaptive feature detection on the convolutional feature set of the degraded image can be further refined to establish the first layer of feature space for the degraded image.

[0029] First, convolution feature sets of degraded images Normalization is performed at the norm layer to obtain the first standardized feature set. This normalization operation can eliminate the distribution differences in brightness, contrast, etc. among different images, giving the model a stable learning starting point in the feature space.

[0030] Next, the first standardized feature set The input is fed into the first-scale visual Mamba module. The visual Mamba module is specifically designed for feature extraction, capable of extracting richer and more discriminative features from the input feature set. After processing by the visual Mamba module, the second-scaled features are obtained. The visual Mamba module here handles visual dynamic modeling, enhancing the global correlation of image features through a state-space sequence modeling mechanism, thereby improving the extracted... It can capture the contextual dependencies of degenerate regions in the spatial dimension.

[0031] Then, to maintain feature consistency and prevent information loss, the original degraded image is convolved with the features. With the extracted standardized features Perform residual join, i.e., perform calculations Thus, the third standardized feature set is obtained. This residual design allows the model to retain information from both input and extracted features, which helps mitigate gradient vanishing and improves network training stability.

[0032] Subsequently, The normalization process is performed again at the norm layer to obtain the fourth normalized feature set. This step helps to further strengthen the consistency of feature distribution, preparing for subsequent nonlinear mapping. Then, based on the two-layer MLP network... By performing a nonlinear mapping, the fifth standardized feature set is obtained. MLP networks, through nonlinear transformations of two layers, can learn complex relationships between input features, thereby extracting more abstract and high-level features. These two MLP layers can be viewed as feature decoupling units, enhancing the model's nonlinear expressive power through the combined effect of linear mapping and activation functions.

[0033] Finally, the degraded image is convolved with the feature set. With the fifth standardized feature set Perform residual join, i.e. execute The first feature space of the degraded image is obtained. This layer of features retains the texture foundation of the original degraded image while incorporating contextual semantic information extracted by the visual Mamba module, providing structured input for subsequent higher-level multi-scale feature modeling.

[0034] Through the above steps, this application achieves hierarchical feature detection of degraded image sample sets, establishing the first-layer feature space of the degraded images, laying a solid foundation for subsequent image restoration processes. This process not only extracts shallow features of the images but also delves into high-level semantic information through techniques such as residual connections and nonlinear mapping, thereby improving the overall performance and quality of image restoration.

[0035] P30: Based on the isoangular tight frame, the degradation prototype features of the four-layer feature space of the degradation image are analyzed to construct the first degradation feature hint vector set.

[0036] Furthermore, step P30 in this embodiment of the application also includes: P31: Construct a degraded prototype vector set based on an isoangular compact frame; P32: Perform global normalization and linear layer mapping on the fourth layer feature space of the degraded image to obtain a degraded mapping vector set; P33: Calculate the cosine similarity between the degraded mapping vector set and the degraded prototype vector set to obtain a degraded mapping prototype similarity distribution; P34: Calculate an adaptive prototype feature set based on the degraded mapping prototype similarity distribution and the temperature coefficient; P35: Inject the fourth layer feature space of the degraded image into the adaptive prototype feature set based on learnable weight conditions to obtain the first degraded feature cue vector set. The learnable weight conditions include a first learnable weight parameter, a second learnable weight parameter, a third learnable weight parameter, and a fourth learnable weight parameter.

[0037] It should be understood that, based on the isoangular tight frame, the four-layer feature space of the degraded image is analyzed to perform degraded prototype feature parsing in order to construct the first set of degraded feature cue vectors for dynamic cue generation.

[0038] First, based on the isoangular compact frame, a set of degenerate prototype vectors is constructed. The isoangular compact frame is a mathematical tool used to generate a set of feature vectors with specific properties; here, it is used to construct degenerate prototype features, providing the model with feature representations for different degeneracy types. For example, a random orthogonal matrix U is generated, and then... Where K represents the number of degenerate anchors, which should be greater than the number of degenerate types. It is an identity matrix. Unit norm normalization yields That is, a set of degenerate anchor points that are angularly distributed in a high-dimensional space, used to represent directional prototypes of different types of degeneration. This will serve as the degenerate prototype vector set.

[0039] Next, the fourth layer feature space of the degraded image is analyzed. Perform global normalization and linear layer mapping to obtain the degenerate mapping vector set. Global normalization helps adjust the distribution of features to have a uniform proportion, while linear layer mapping is used to map the normalized features to a new feature space to facilitate subsequent similarity calculations.

[0040] Then, cosine similarity is calculated between the degenerate mapping vector set and the degenerate prototype vector set to obtain the similarity distribution of the degenerate mapping prototypes. Cosine similarity is a commonly used method to measure the similarity between two vectors; by calculating the cosine of the angle between two vectors, the degree of similarity between them can be obtained. For example, calculating... ;here, Represents z with each degenerate prototype vector The cosine similarity between the features helps determine the degree of similarity between the feature vectors and each degenerate prototype.

[0041] Next, based on the similarity distribution of the degenerate mapping prototype, according to the temperature coefficient Calculate the adaptive prototype feature set. Temperature coefficient. This is an important parameter used to control the smoothness of the similarity distribution. By adjusting the temperature coefficient, different degrees of smoothness in the similarity distribution can be obtained, thus affecting the calculation results of the adaptive prototype features. For example, based on similarity... and temperature coefficient Calculate adaptive prototype features Temperature coefficient Depend on The output of the two-layer MLP network is used to control the smoothness of the distribution, thereby affecting the calculation results of the adaptive prototype features.

[0042] Finally, based on the learnable weight conditions, the fourth-layer feature space of the degraded image is injected into the adaptive prototype feature set to obtain the first degraded feature cue vector set. The learnable weight conditions include a first learnable weight parameter, a second learnable weight parameter, a third learnable weight parameter, and a fourth learnable weight parameter. These parameters can be optimized through the training process to obtain the best image restoration effect. For example, the current features... With adaptive prototype features By combining these, we obtain the first set of degenerate feature hint vectors. ,in Indicates the learnable weight parameters. , , , , and This represents the learnable weight parameters.

[0043] Through the above steps, this application can realize the degraded prototype feature parsing of the four-layer feature space of the degraded image and construct the first degraded feature hint vector set, that is, the enhanced feature direction after degraded prototype adaptive modulation on the basis of the original features. This vector set can provide important feature support for the subsequent image restoration process.

[0044] P40: Based on the first degradation feature hint vector set, the four-layer feature space of the degradation image is guided to perform adaptive hierarchical dynamic hint fusion to establish a dynamic degradation feature hint vector set.

[0045] Furthermore, step P40 in this embodiment of the application also includes: P41: Perform visual Mamba fusion of the first degradation feature hint vector set with the third layer feature space of the degradation image to obtain a first fused feature output set; P42: Input the first fused feature output set into the adaptive hierarchical dynamic hint module to obtain a second degradation feature hint vector set; P43: Perform visual Mamba fusion of the second degradation feature hint vector set with the second layer feature space of the degradation image to obtain a second fused feature output set; P44: Input the second fused feature output set into the adaptive hierarchical dynamic hint module to obtain a third degradation feature hint vector set; P45: Perform visual Mamba fusion of the third degradation feature hint vector set with the first layer feature space of the degradation image to generate the dynamic degradation feature hint vector set.

[0046] Optionally, the first degradation feature cue vector set is used to guide the adaptive hierarchical dynamic cue fusion of the four-layer feature space of the degradation image, thereby establishing a dynamic degradation feature cue vector set and realizing semantic guidance and dynamic enhancement between features at different scales. This process aims to enable the model to transmit degradation perception signals at the feature level through layer-by-layer cue injection and multi-scale information interaction, thereby improving the network's adaptability to complex composite degradation.

[0047] First, the first set of degenerate feature hint vectors With the third layer feature space of the degraded image Visual Mamba fusion is performed. The visual Mamba module is a feature processing module that effectively fuses cue vectors with the feature space to enhance feature representation capabilities. In this process, the visual Mamba module uses... As a guiding signal, The local features are dynamically weighted and adjusted to achieve multi-scale feature interaction guided by prompts. The state-space modeling mechanism of the Mamba module can capture global dependencies in long sequence features, allowing prompt information to spread fully in the spatial dimension, thereby enhancing the consistency of degradation perception and obtaining the first fusion feature output set. .

[0048] Next, the first fusion feature output set An adaptive hierarchical dynamic suggestion module is introduced. This module dynamically adjusts the suggestion vector based on the input features to form guidance signals for shallower features, achieving progressive semantic hierarchy. Through this module's processing, a second set of degenerate feature suggestion vectors is obtained. .

[0049] Then, the second degenerate feature hint vector set With the second layer feature space of the degraded image The second fused feature output set is obtained by fusing the features through the visual Mamba module. This stage of fusion primarily targets the structural details and texture features of the image. Through cues and guidance, the network is able to distinguish between degraded and non-degraded regions at a mesoscale level.

[0050] Next, the second fusion feature output set will be... The input is processed by the adaptive hierarchical dynamic suggestion module, which then makes dynamic adjustments to obtain the third degenerate feature suggestion vector set. This step allows the cue features to propagate further into shallower layers and establishes cross-layer semantic consistency within the model.

[0051] Finally, the third degradation feature hint vector set With the first layer feature space of the degraded image Visual Mamba fusion is performed to generate the final dynamic degradation feature cue vector set, i.e., the dynamic degradation feature cue vector set. This result integrates contextual and detailed feature information from multi-layered cue features, preserving the degradation recognition capability of deep semantics while enhancing the guidance role of shallow texture restoration.

[0052] Through this series of processing steps, an adaptive hierarchical dynamic fusion mechanism for degradation prompts is achieved: upper-layer prompts guide deep semantic features, while lower-layer prompts adjust local detail responses, ultimately forming a dynamic degradation feature prompt set spanning multiple scales. This design not only enhances the interactivity between prompt features and the multi-layer space of the image but also provides a unified and dynamically optimized feature guidance foundation for subsequent loss function-based recovery training.

[0053] P50: The image restoration loss function is invoked to iteratively train the degraded image sample set, the dynamic degradation feature cue vector set, and the clean image sample set to establish the first image restoration model.

[0054] The image restoration loss function is: Where loss represents the image restoration loss, N represents the number of samples in the degraded image sample set, and i represents the i-th degraded image sample in the degraded image sample set. This represents the output restored image corresponding to the i-th degraded image sample. This represents the clean image sample corresponding to the i-th degraded image sample. This represents the adjustment coefficient. This represents the number of degraded anchor points, where j represents the j-th degraded image sample in the set of degraded image samples. This represents the degradation prototype feature corresponding to the i-th degraded image sample. This represents the degradation prototype feature of the j-th degraded image sample.

[0055] Specifically, the image restoration loss function is invoked to perform end-to-end iterative training on a degraded image sample set, a dynamic degradation feature cue vector set, and a clean image sample set to establish the first image restoration model. The core of this step lies in jointly optimizing the image reconstruction error and degradation prototype constraints, enabling the network to possess the separability and stability of degradation features while learning the restoration mapping, thereby achieving unified restoration capabilities under multiple degradation types.

[0056] First, the image restoration loss function is called, as shown above. This function measures the difference between the restored image output by the model and the real clean image, while also considering a regularization term between the degraded prototype features. During iterative training, the model parameters are optimized by minimizing the loss function. The first loss term... The first term is the image reconstruction error term, used to constrain the pixel-level differences between the reconstructed image output by the model and the corresponding ground truth image, ensuring the sharpness and detail restoration of the reconstruction result. The second loss term... Degenerate prototype constraint terms, through different degenerate prototype features and The inner product relationship applies an equiangular tight frame constraint to ensure that the prototype features are uniformly distributed and orthogonal to each other in the feature space. It encourages different degenerate prototype features to maintain a certain distance from each other, thereby improving the model's ability to identify different types of degradation.

[0057] During training, the model uses degraded images As input, feature enhancement is performed by combining dynamic degradation feature cue vectors to generate the output restored image. By using backpropagation to calculate the gradient derivative of the loss function and update the parameters, the network gradually learns the mapping relationship from degraded images to clean images. Through iterative training, it learns how to recover high-quality images based on the features of the degraded image and the features of the degradation prototype. This process involves not only the restoration of image content but also the identification and adaptation to the degradation type, thereby achieving more accurate image restoration results. Finally, through continuous iteration and optimization, a first-stage image restoration model is established, providing a foundation for subsequent image restoration processing.

[0058] P60: The first image restoration model is optimized by gradient descent using the AdamW optimizer to obtain the second image restoration model, and image restoration processing is performed based on the second image restoration model.

[0059] Furthermore, step P60 in this embodiment of the application also includes: P61: Calculate the first-order moment estimate and second-order moment estimate of the image restoration first model according to the AdamW optimizer; P62: Perform bias correction on the first-order moment estimate and the second-order moment estimate to obtain the bias-corrected first-order moment and the bias-corrected second-order moment; P63: Calculate the gradient update amount based on the learning rate of the image restoration first model and the bias-corrected first-order moment and the bias-corrected second-order moment; P64: Introduce a weight decay factor to perform decoupled decay optimization on the gradient update amount and establish an adaptive gradient update mechanism; P65: Based on the AdamW optimizer, perform gradient descent optimization on the image restoration first model according to the adaptive gradient update mechanism to generate the image restoration second model.

[0060] It should be understood that the AdamW optimizer is used to perform gradient descent optimization on the first image restoration model to obtain the second image restoration model, and the final image restoration processing is performed based on this model. The core objective of this step is to improve the training stability and restoration accuracy of the model by introducing bias correction and weight decay mechanisms, while maintaining the advantages of adaptive learning rate and suppressing overfitting and parameter drift.

[0061] First, the AdamW optimizer is used to calculate the first-order and second-order moment estimates of each network parameter in the first image restoration model. The first-order moment estimate represents the exponentially weighted average of the gradient, used to characterize the average trend of the current parameters, while the second-order moment estimate represents the exponentially weighted average of the squared gradient, used to reflect the variance information of the gradient, thereby enabling adaptive learning rate adjustment in subsequent calculations.

[0062] Next, bias correction is applied to the calculated first-order and second-order moment estimates to eliminate systematic bias caused by exponential weighted decay in the early stages of training. Bias correction is a key feature of the AdamW optimizer; it provides more accurate gradient information by adjusting the exponential moving average in the initial stage of the optimization process, thereby ensuring the stability and efficiency of the optimization process.

[0063] Then, based on the learning rate of the first image restoration model, the gradient update amount is calculated according to the bias-corrected first and second moments. The formula for calculating the gradient update amount is as follows: ,in, For gradient update amount, For learning rate, The first moment after deviation correction. The second moment after deviation correction. This is to prevent the use of tiny constants with zero denominators, thus improving numerical stability. This formula allows for targeted updates of parameters based on the gradient direction and magnitude.

[0064] To suppress overfitting and weight inflation during gradient update, a weight decay factor is introduced. Decoupled decay is applied to the network parameters. This prevents overfitting and improves generalization performance by penalizing large weight values. The adaptive gradient update mechanism is as follows: ,in, The first value represents the model parameter values ​​at time step t+1, i.e., the updated model parameters; the second value represents the model parameter values ​​at time step t, i.e., the model parameters at the current iteration step. For learning rate, The first moment after deviation correction. The second moment after deviation correction. This is to prevent tiny constants with a denominator of zero. This is the weight decay factor. This is the weight decay term, which achieves decoupled decay of parameters. By adding a weight decay term to the regular gradient descent term, this formula separates the control of gradient direction and weight decay, thus maintaining the adaptive properties of the gradient while effectively constraining the parameter size and preventing overfitting during optimization.

[0065] Finally, based on the AdamW optimizer, gradient descent optimization is performed on the first image restoration model according to the adaptive gradient update mechanism. The parameter weights of the first image restoration model are optimized through multiple iterations until the image restoration loss function converges, resulting in the second image restoration model, which is used to perform the final image restoration processing. The optimized second model exhibits higher stability and generalization ability in multiple degradation scenarios, achieving consistently high-quality restoration of different types of degraded images, such as those with rain, fog, noise, blur, and low light.

[0066] Through the above processing steps, not only was the optimization iteration from the first image restoration model to the second model completed, but also the introduction of bias correction and weight decay strategies ensured that the model achieved a balance between convergence speed, stability and restoration effect, providing the optimal parameter configuration for subsequent actual image restoration processing.

[0067] In summary, the embodiments of this application have at least the following technical effects: This application introduces a dynamic cue generation mechanism based on an isoangular tight frame, enabling the cue vector to adaptively adjust according to the input degradation features. This allows for unified restoration of various degraded images, including those with rain, fog, noise, blur, and low illumination, within a single model. Combined with a hierarchical dynamic cue fusion strategy, it promotes interaction and optimization between features at different scales, significantly enhancing image detail restoration and semantic consistency. Furthermore, the model can be trained efficiently without relying on degradation labels, exhibiting stronger generalization ability. Finally, the joint optimization using a custom image restoration loss function and the AdamW optimizer improves training convergence speed and stability, resulting in an overall improvement in the accuracy and robustness of image restoration.

[0068] This technology achieves the goal of improving the accuracy and adaptability of image restoration in multi-degradation scenarios by introducing dynamic prompt generation and hierarchical fusion mechanisms, enabling prompt vectors to adaptively match different degradation features.

[0069] Example 2, based on the same inventive concept as the end-to-end image restoration method based on image-text feature mapping in the previous examples, such as... Figure 2 As shown, this application provides an end-to-end image restoration system based on image-text feature mapping. The system and method embodiments in this application are based on the same inventive concept. The system includes: The image sample retrieval module 11 is used to retrieve the image restoration processing sample set, which includes a degraded image sample set and a clean image sample set.

[0070] The hierarchical feature detection module 12 is used to perform hierarchical feature detection on the degraded image sample set according to the image-text feature mapping mechanism, and establish a four-layer feature space for the degraded image.

[0071] The degradation prototype feature parsing module 13 is used to perform degradation prototype feature parsing on the four-layer feature space of the degradation image according to the isoangular tight frame, and construct the first degradation feature hint vector set.

[0072] The dynamic prompt fusion module 14 is used to guide the four-layer feature space of the degraded image to perform adaptive hierarchical dynamic prompt fusion based on the first degradation feature prompt vector set, and to establish a dynamic degradation feature prompt vector set.

[0073] The iterative training module 15 is used to call the image restoration loss function to iteratively train the degraded image sample set, the dynamic degradation feature cue vector set, and the clean image sample set to establish the first image restoration model.

[0074] The image restoration processing module 16 is used to perform gradient descent optimization on the first image restoration model according to the AdamW optimizer to obtain a second image restoration model, and to perform image restoration processing according to the second image restoration model.

[0075] Furthermore, the hierarchical feature detection module 12 is also used to perform the following steps: The image-text feature mapping mechanism is activated, which includes a predetermined convolutional layer, a first-scale visual Mamba module, a second-scale visual Mamba module, a third-scale visual Mamba module, and a fourth-scale visual Mamba module. The degraded image sample set is convolved using the predetermined convolutional layer to obtain a degraded image convolutional feature set. Scale-adaptive feature detection is performed on the degraded image convolutional feature set using the first-scale visual Mamba module to generate a first-layer feature space for the degraded image. Scale-adaptive feature detection is then performed on the first-layer feature space of the degraded image using the second-scale, third-scale, and fourth-scale visual Mamba modules respectively to obtain a second-layer, third-layer, and fourth-layer feature space for the degraded image. Finally, a fourth-layer feature space for the degraded image is generated based on the first-layer, second-layer, third-layer, and fourth-layer feature spaces.

[0076] Furthermore, the hierarchical feature detection module 12 is also used to perform the following steps: The degraded image convolutional feature set is normalized using a norm layer to obtain a first normalized feature set; the first normalized feature set is input into a first-scale visual Mamba module to obtain a second normalized feature set; the degraded image convolutional feature set and the second normalized feature set are residually connected to obtain a third normalized feature set; the third normalized feature set is normalized using a norm layer to obtain a fourth normalized feature set; the fourth normalized feature set is nonlinearly mapped using a two-layer MLP network to obtain a fifth normalized feature set; the degraded image convolutional feature set and the fifth normalized feature set are residually connected to obtain the first-layer feature space of the degraded image.

[0077] Furthermore, the degenerate prototype feature parsing module 13 is also used to perform the following steps: Based on an isoangular compact frame, a degraded prototype vector set is constructed. Global normalization and linear layer mapping are performed on the fourth-layer feature space of the degraded image to obtain a degraded mapping vector set. Cosine similarity is calculated between the degraded mapping vector set and the degraded prototype vector set to obtain a degraded mapping prototype similarity distribution. Based on the degraded mapping prototype similarity distribution, an adaptive prototype feature set is calculated according to a temperature coefficient. Based on learnable weight conditions, the fourth-layer feature space of the degraded image is injected into the adaptive prototype feature set to obtain the first degraded feature cue vector set. The learnable weight conditions include a first learnable weight parameter, a second learnable weight parameter, a third learnable weight parameter, and a fourth learnable weight parameter.

[0078] Furthermore, the dynamic prompt fusion module 14 is also used to perform the following steps: The first degradation feature hint vector set is fused with the third layer feature space of the degradation image using visual Mamba to obtain a first fused feature output set. The first fused feature output set is input into the adaptive hierarchical dynamic hint module to obtain a second degradation feature hint vector set. The second degradation feature hint vector set is fused with the second layer feature space of the degradation image using visual Mamba to obtain a second fused feature output set. The second fused feature output set is input into the adaptive hierarchical dynamic hint module to obtain a third degradation feature hint vector set. The third degradation feature hint vector set is fused with the first layer feature space of the degradation image using visual Mamba to generate the dynamic degradation feature hint vector set.

[0079] Furthermore, in the iterative training module 15, the image restoration loss function is: Where loss represents the image restoration loss, N represents the number of samples in the degraded image sample set, and i represents the i-th degraded image sample in the degraded image sample set. Let represent the output restored image corresponding to the i-th degraded image sample, and yi represent the clean image sample corresponding to the i-th degraded image sample. Let K represent the adjustment coefficient, K represent the number of degradation anchors, and j represent the j-th degradation image sample in the degradation image sample set. This represents the degradation prototype feature corresponding to the i-th degraded image sample. This represents the degradation prototype feature of the j-th degraded image sample.

[0080] Furthermore, the image restoration processing module 16 is also used to perform the following steps: Based on the AdamW optimizer, the first-order moment estimates and second-order moment estimates of the first image restoration model are calculated; the first-order moment estimates and second-order moment estimates are biased to obtain bias-corrected first-order moments and bias-corrected second-order moments; based on the learning rate of the first image restoration model, the gradient update amount is calculated according to the bias-corrected first-order moments and bias-corrected second-order moments; a weight decay factor is introduced to perform decoupled decay optimization on the gradient update amount to establish an adaptive gradient update mechanism; based on the AdamW optimizer, the first image restoration model is optimized by gradient descent according to the adaptive gradient update mechanism to generate the second image restoration model.

[0081] In Embodiment 3, based on the same inventive concept as the end-to-end image restoration method based on image and text feature mapping in the foregoing embodiments, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in Embodiment 1.

[0082] Through the foregoing detailed description of the end-to-end image restoration method based on image-text feature mapping, those skilled in the art can clearly understand the end-to-end image restoration method, system, and medium based on image-text feature mapping in this embodiment. Therefore, for the sake of brevity, further details are omitted here. As for the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be found in the method section.

[0083] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0084] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0085] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0086] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. An end-to-end image restoration method based on image-text feature mapping, characterized in that, The method includes: Retrieve an image restoration processing sample set, which includes a degraded image sample set and a clean image sample set; Based on the image-text feature mapping mechanism, hierarchical feature detection is performed on the degraded image sample set to establish a four-layer feature space for the degraded image; Based on the isoangular tight frame, the degradation prototype features of the four-layer feature space of the degradation image are analyzed to construct the first degradation feature cue vector set; Based on the first degradation feature hint vector set, the four-layer feature space of the degradation image is guided to perform adaptive hierarchical dynamic hint fusion to establish a dynamic degradation feature hint vector set; The image restoration loss function is called to iteratively train the degraded image sample set, the dynamic degradation feature cue vector set, and the clean image sample set to establish the first image restoration model; The first image restoration model is optimized by gradient descent using the AdamW optimizer to obtain the second image restoration model, and image restoration processing is performed based on the second image restoration model.

2. The method as described in claim 1, characterized in that, Based on the image-text feature mapping mechanism, hierarchical feature detection is performed on the degraded image sample set to establish a four-layer feature space for the degraded image, including: The image-text feature mapping mechanism is activated, which includes a predetermined convolutional layer, a first-scale visual Mamba module, a second-scale visual Mamba module, a third-scale visual Mamba module, and a fourth-scale visual Mamba module. The degraded image sample set is convolved according to the predetermined convolutional layer to obtain the degraded image convolutional feature set; Based on the first-scale visual Mamba module, scale-adaptive feature detection is performed on the convolutional feature set of the degraded image to generate the first-layer feature space of the degraded image; The second-scale visual Mamba module, the third-scale visual Mamba module, and the fourth-scale visual Mamba module respectively perform scale-adaptive feature detection on the first-layer feature space of the degraded image to obtain the second-layer feature space, the third-layer feature space, and the fourth-layer feature space of the degraded image. The four-layer feature space of the degraded image is generated based on the first layer feature space, the second layer feature space, the third layer feature space, and the fourth layer feature space of the degraded image.

3. The method as described in claim 2, characterized in that, Based on the first-scale visual Mamba module, scale-adaptive feature detection is performed on the convolutional feature set of the degraded image to generate the first-layer feature space of the degraded image, including: The degraded image convolutional feature set is normalized using a norm layer to obtain a first standardized feature set; The first standardized feature set is input into the first-scale visual Mamba module to obtain the second standardized feature set; The degraded image convolutional feature set is residually concatenated with the second normalized feature set to obtain the third normalized feature set; The fourth standardized feature set is obtained by normalizing the feature set using the third standardized feature set at the norm layer. The fifth standardized feature set is obtained by performing a nonlinear mapping on the fourth standardized feature set using a two-layer MLP network. The degraded image convolutional feature set and the fifth normalized feature set are residually concatenated to obtain the first layer feature space of the degraded image.

4. The method as described in claim 1, characterized in that, Based on the isoangular tight frame, degradation prototype features are parsed in the four-layer feature space of the degraded image to construct a first degradation feature cue vector set, including: Based on the isoangular tight frame, construct a degenerate prototype vector set; Global normalization and linear layer mapping are performed on the fourth layer feature space of the degraded image to obtain the degraded mapping vector set; The cosine similarity between the degenerate mapping vector set and the degenerate prototype vector set is calculated to obtain the degenerate mapping prototype similarity distribution; Based on the similarity distribution of the degenerate mapping prototype, an adaptive prototype feature set is calculated according to the temperature coefficient; Based on learnable weight conditions, the fourth layer feature space of the degraded image is injected into the adaptive prototype feature set to obtain the first degraded feature cue vector set.

5. The method as described in claim 1, characterized in that, Based on the first degradation feature cue vector set, the degraded image's four-layer feature space is guided to perform adaptive hierarchical dynamic cue fusion, establishing a dynamic degradation feature cue vector set, including: The first set of degraded feature hint vectors is fused with the third layer feature space of the degraded image using visual Mamba to obtain the first fused feature output set; The first fused feature output set is input into the adaptive hierarchical dynamic prompting module to obtain the second degenerate feature prompting vector set; The second set of degraded feature cue vectors is fused with the second layer feature space of the degraded image using visual Mamba to obtain the second fused feature output set; The second fused feature output set is input into the adaptive hierarchical dynamic prompting module to obtain the third degenerate feature prompting vector set; The third degradation feature cue vector set is fused with the first layer feature space of the degradation image using visual Mamba to generate the dynamic degradation feature cue vector set.

6. The method as described in claim 1, characterized in that, The image restoration loss function is: ； Where loss represents the image restoration loss, N represents the number of samples in the degraded image sample set, and i represents the i-th degraded image sample in the degraded image sample set. Let represent the output restored image corresponding to the i-th degraded image sample, and yi represent the clean image sample corresponding to the i-th degraded image sample. Let K represent the adjustment coefficient, K represent the number of degradation anchors, and j represent the j-th degradation image sample in the degradation image sample set. This represents the degradation prototype feature corresponding to the i-th degraded image sample. This represents the degradation prototype feature of the j-th degraded image sample.

7. The method as described in claim 1, characterized in that, The image restoration first model is optimized using gradient descent based on the AdamW optimizer to obtain the image restoration second model, which includes: Based on the AdamW optimizer, calculate the first-order moment estimate and the second-order moment estimate of the image restoration first model; The first-order moment estimate and the second-order moment estimate are subjected to deviation correction to obtain the deviation-corrected first-order moment and the deviation-corrected second-order moment; Based on the learning rate of the image recovery first model, the gradient update amount is calculated according to the bias-corrected first moment and the bias-corrected second moment; A weight decay factor is introduced to perform decoupled decay optimization on the gradient update amount, and an adaptive gradient update mechanism is established. Based on the AdamW optimizer, the first image restoration model is optimized by gradient descent according to the adaptive gradient update mechanism to generate the second image restoration model.

8. The method as described in claim 4, characterized in that, The learnable weight conditions include a first learnable weight parameter, a second learnable weight parameter, a third learnable weight parameter, and a fourth learnable weight parameter.

9. An end-to-end image restoration system based on image-text feature mapping, characterized in that, The system is used to perform the steps of the method according to any one of claims 1 to 8, the system comprising: The image sample retrieval module is used to retrieve the image restoration processing sample set, which includes a degraded image sample set and a clean image sample set. The hierarchical feature detection module is used to perform hierarchical feature detection on the degraded image sample set according to the image-text feature mapping mechanism, and establish a four-layer feature space for the degraded image. The degradation prototype feature parsing module is used to perform degradation prototype feature parsing on the four-layer feature space of the degradation image based on the isoangular tight frame, and to construct the first degradation feature hint vector set; The dynamic prompting fusion module is used to guide the four-layer feature space of the degraded image to perform adaptive hierarchical dynamic prompting fusion based on the first degradation feature prompting vector set, and to establish a dynamic degradation feature prompting vector set; The iterative training module is used to call the image restoration loss function to iteratively train the degraded image sample set, the dynamic degradation feature cue vector set, and the clean image sample set to establish the first image restoration model; The image restoration processing module is used to perform gradient descent optimization on the first image restoration model according to the AdamW optimizer to obtain a second image restoration model, and to perform image restoration processing according to the second image restoration model.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images