A distortion perception based no-reference rain removal image quality assessment method and system

Abstract

This invention relates to the field of image quality assessment, specifically to a method and system for assessing the quality of derained images without reference based on distortion perception. The method includes: acquiring the derained image to be evaluated and performing unified input processing; extracting multi-level visual features through a convolutional Transformer backbone network; constructing rain pattern residual features using a rain pattern residual encoder; constructing structure distortion-perceived features using a structure distortion encoder combined with edge priors, residual enhancement, and attention convolution; performing distortion-guided interactive enhancement on the rain pattern residual features, structure distortion-perceived features, and multi-level visual features to obtain rain pattern enhancement features and structure enhancement features; and finally outputting the derained image quality assessment result through a feature fusion module and a quality prediction module.

Description

Technical Field

[0001] This invention relates to the field of image quality assessment, and specifically to a method and system for assessing the quality of non-reference deraining images based on distortion perception. Background Technology

[0002] In recent years, with the widespread application of vision systems such as intelligent surveillance, autonomous driving, UAV remote sensing, and mobile terminals, image quality directly affects the performance of subsequent high-level tasks such as object detection, scene understanding, and visual perception. Due to the influence of rainy environments, images often exhibit degradation phenomena such as rain streaks, fogging effects, and detail occlusion. Therefore, single-image deraining techniques have become an important research direction in the field of computer vision. With the development of deep learning technology, various deraining algorithms have emerged, capable of removing rain streaks and restoring image content to a certain extent. However, different deraining algorithms still exhibit significant differences in their ability to remove rain streaks, preserve texture details, and restore structure. Therefore, objective and accurate quality assessment of derained images is of great importance.

[0003] Currently, image quality assessment methods mainly fall into two categories: full-reference (FR) image quality assessment methods and no-reference (NR) image quality assessment methods. Full-reference image quality assessment methods typically require a corresponding distortion-free reference image and evaluate image quality by calculating metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). However, in practical applications, original rain-free images are often difficult to obtain, thus significantly limiting full-reference assessment methods. No-reference image quality assessment methods do not require a reference image and have gradually become a research hotspot in recent years. However, existing general-purpose no-reference image quality assessment methods are mainly designed for common distortion types such as compression distortion, noise, and blur, and are insufficient to accurately describe complex distortion features in de-rained images, such as residual rain streaks, excessive smoothing, texture loss, and structural distortion.

[0004] In addition, existing quality assessment methods for derained images mainly rely on general image quality features or overall semantic information, which do not pay enough attention to the structural degradation and loss of local details that occur during the deraining process. They are difficult to fully model the coupling relationship between rain streak residue and structural distortion, resulting in room for improvement in the consistency between the prediction results and human subjective perception.

[0005] Therefore, there is an urgent need for a no-reference deraining image quality assessment method that can model the distortion characteristics of deraining images and effectively integrate structural and visual perception information, so as to improve the accuracy and reliability of deraining image quality evaluation. Summary of the Invention

[0006] This invention provides a distortion-aware, reference-free deraining image quality assessment method and system, aiming to improve the accuracy and reliability of deraining image quality evaluation.

[0007] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: On the one hand, a distortion-aware, no-reference deraining image quality assessment method is proposed, comprising: The rain-removed image to be evaluated is obtained, and the rain-removed image is subjected to unified input processing. The processed rain-removed image is then input into a backbone feature extraction network that combines local convolutional feature extraction with self-attention feature interaction to obtain a multi-level visual representation containing local texture, structural contour and global semantic information. Based on the de-rained image, rain streak residual features and structural distortion perception features are constructed. The rain streak residual features are obtained by encoding-decoding feature modeling oriented towards rain streak degradation information, and the structural distortion perception features are obtained by introducing edge priors and performing attention encoding on structurally sensitive regions. Rain pattern residual features and structural distortion perception features are used as distortion guidance information and interactively enhanced with multi-level visual representation to obtain rain pattern enhancement features and structural enhancement features. Multi-level visual representations, rain pattern enhancement features, and structural enhancement features are fused to obtain quality-perceived features, and the quality evaluation results of the de-rained image are output based on the quality-perceived features.

[0008] Furthermore, the unified input processing includes at least one of the following: image scale adaptation, pixel value normalization, channel format unification, data type conversion, and boundary completion of the derained image. Edge information or gradient information is generated based on the derained image and used as auxiliary input in the process of constructing structural distortion-aware features.

[0009] Furthermore, the backbone feature extraction network includes an initial feature mapping unit and multiple staged feature extraction units. Each staged feature extraction unit includes a local spatial feature extraction subunit and a global correlation modeling subunit. The local spatial feature extraction subunit performs convolutional encoding on texture, edge, and rain pattern residue, while the global correlation modeling subunit performs self-attention interaction on features at different spatial locations.

[0010] Furthermore, the multi-level visual representation includes shallow visual features, mid-level visual features, and deep visual features output from different stages of the backbone feature extraction network. Among them, shallow visual features participate in the enhancement of rain pattern residue-related features, mid-level visual features participate in the enhancement of texture distribution and local structure-related features, and deep visual features participate in the enhancement of global structure and semantic-related features.

[0011] Furthermore, rain streak residual features are obtained through a rain streak residual encoder, which includes an encoder, a decoder, a skip connection unit, and a multi-scale fusion unit. The encoder extracts layered rain streak degradation features from the de-rained image, the decoder performs spatial restoration on the layered rain streak degradation features, the skip connection unit introduces features from different levels of the encoder into the decoder, and the multi-scale fusion unit performs channel adaptation and splicing fusion on features from different scales of the decoder.

[0012] Furthermore, the rain pattern residual encoder undergoes rain pattern degradation prior learning before being used for quality assessment. The rain pattern degradation prior learning uses rainy day images and corresponding rainless images as training samples. The rainy day images are input into the generative encoder-decoder network to obtain the restored images, and the restored images and rainless images are distinguished by the discriminant network. The training constraints include image restoration constraints and adversarial discrimination constraints.

[0013] Furthermore, the structural distortion-aware features are obtained through a structural distortion encoder, which includes an edge prior extraction unit, a shallow structure encoding unit, a residual enhancement unit, an attention convolution unit, and a feature concatenation unit. The edge prior extraction unit extracts structural contour information from the de-rained image, the shallow structure encoding unit performs convolutional mapping on the structural contour information, the residual enhancement unit performs residual encoding on the structural features after convolutional mapping, the attention convolution unit performs weighted encoding on the structurally sensitive regions, and the feature concatenation unit fuses the structural features from different encoding stages.

[0014] Furthermore, the structure distortion encoder undergoes structure restoration prior learning before being used for quality assessment. The structure restoration prior learning uses degraded images and corresponding clear images as training samples. The structure encoding features are extracted based on the edge information of the degraded images. The attention prediction branch generates a structure attention map and modulates the structure encoding features. Then, the decoding branch reconstructs the clear image. The training constraints include image restoration constraints and structure attention supervision constraints.

[0015] Furthermore, interactive feature enhancement includes: mapping rain pattern residual features and structural distortion perception features to distortion query features respectively, mapping multi-level visual representations to key features and value features, and obtaining rain pattern enhancement features and structural enhancement features respectively through cross-attention; fusion includes attention aggregation of multi-level visual representations, combining the aggregated multi-level visual representations with rain pattern enhancement features and structural enhancement features to form a quality perception feature sequence, and outputting quality evaluation results through feature interaction network and quality regression network.

[0016] On the other hand, a distortion-aware, no-reference deraining image quality assessment system is provided, comprising: The image acquisition module is used to acquire the rain-removed image to be evaluated, perform unified input processing on the rain-removed image, and input the processed rain-removed image into the backbone feature extraction network that combines local convolutional feature extraction and self-attention feature interaction to obtain a multi-level visual representation containing local texture, structural contour and global semantic information. The feature extraction module is used to construct rain streak residual features and structural distortion perception features based on the de-rained image. The rain streak residual features are obtained by encoding-decoding feature modeling oriented towards rain streak degradation information, and the structural distortion perception features are obtained by introducing edge priors and performing attention encoding on structurally sensitive regions. The feature enhancement module is used to interactively enhance the rain pattern residual features and structural distortion perception features as distortion guidance information with the multi-level visual representation to obtain rain pattern enhancement features and structural enhancement features. The quality assessment module is used to fuse multi-level visual representations, rain pattern enhancement features, and structural enhancement features to obtain quality-perceived features, and output the quality assessment results of the de-rained image based on the quality-perceived features.

[0017] Compared to existing solutions, the beneficial effects achieved by this invention are: The present invention proposes a distortion-aware, reference-free deraining image quality assessment method and system. Compared with existing general reference-free image quality assessment methods, it can effectively model the unique distortions in deraining images, such as residual rain streaks, texture loss, edge blurring, and structural degradation, thereby improving the representation ability of quality-related features.

[0018] By fusing hierarchical visual features and distortion perception features, the network achieves synergistic utilization of local texture information and global semantic information, enhancing its ability to perceive quality-sensitive areas and thus improving the consistency between quality prediction results and human subjective visual perception.

[0019] This invention can evaluate the quality of derained images without the need for a reference image. It has good accuracy, robustness and cross-dataset generalization ability, and can be widely used in deraining algorithm performance evaluation, image restoration model optimization and related computer vision fields. Attached Figure Description

[0020] Figure 1 This is a flowchart of the method steps of a distortion-aware, reference-free deraining image quality assessment method according to an embodiment of the present invention; Figure 2 This is a diagram of a convolutional Transformer backbone feature extraction network structure according to an embodiment of the present invention; Figure 3 This is a network structure diagram of a feature fusion module according to an embodiment of the present invention; Figure 4This is a network structure diagram of a quality prediction module according to an embodiment of the present invention; Figure 5 This is a system block diagram of a distortion-aware, reference-free deraining image quality assessment system according to an embodiment of the present invention. Detailed Implementation

[0021] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0022] This embodiment provides a distortion-aware, no-reference deraining image quality assessment method. Figure 1 This is a flowchart illustrating the method steps of a distortion-aware, reference-free deraining image quality assessment method according to an embodiment of the present invention. Figure 1 The method includes the following procedures: Step 1: Obtain the derained image to be evaluated and extract multi-level visual features of the image; In step one, the input de-raining image I is preprocessed, including size normalization, pixel value normalization, channel format unification, tensor format conversion, and image size padding. Specifically, the input de-raining image is adjusted to the input size required by the convolutional Transformer backbone feature extraction network, and pixel values ​​are normalized or standardized to reduce the impact of differences in image size, brightness, and numerical scale on feature extraction. Simultaneously, the image is uniformly converted to RGB three-channel format and floating-point tensor format to meet the input requirements of subsequent convolution operations and Transformer self-attention calculations. Furthermore, to enhance the perception of structural distortion information, edge or gradient information can be extracted from the de-raining image, for example, using the Sobel operator to obtain edge magnitude maps as auxiliary information for subsequent structural distortion encoding. During the training phase, data augmentation techniques such as random cropping and random flipping can be used to improve the model's generalization ability to different types of de-raining image distortion and scene content. The above preprocessing operations do not change the quality properties of the derained image itself, but are used to unify the input format, enhance the expression of structural and texture information, and provide stable input for subsequent multi-scale visual feature extraction, rain streak modeling, and structural distortion modeling.

[0023] Given a derained image I as input, the derained image I is first preprocessed and then input into a convolutional Transformer backbone feature extraction network. Figure 2 This is a diagram of a convolutional Transformer backbone feature extraction network structure according to an embodiment of the present invention, such as... Figure 2As shown, the convolutional Transformer backbone feature extraction network is used to extract visual features with different receptive fields and semantic levels from the input de-rain image. This backbone network combines the advantages of convolutional neural networks and Transformer structures. The convolutional layers are mainly used to capture fine-grained spatial information such as local texture, edge contours, and residual rain streaks, while the Transformer structure is used to model the dependencies between distant regions in the image, enhancing the network's ability to perceive changes in global structure, scene semantics, and overall visual quality. Specifically, the input de-rain image first undergoes initial feature mapping through a convolutional Token embedding layer, converting the original image into a shallow feature representation. Subsequently, the features are sequentially input into multiple convolutional Transformer blocks for feature extraction. Each stage includes a local convolutional feature extraction unit and a self-attention feature interaction unit: the local convolutional unit preserves texture, edge, and residual rain streak features within the image's spatial neighborhood, while the self-attention unit establishes global relationships between different spatial locations, thereby obtaining hierarchical visual features that combine local detail expression and global context modeling capabilities.

[0024] At different stages of the backbone network, the spatial resolution of the feature maps is progressively reduced through downsampling operations, while the number of feature channels is increased. This allows the network to sequentially acquire detailed texture features, structural contour features, and high-level semantic features from shallow to deep layers. (Shallow visual features) It mainly reflects local texture, edge details, and residual rain streaks; mid-level visual features It mainly reflects texture distribution, local structural continuity, and degradation information such as blurring or over-smoothing; deep visual features It primarily reflects the overall image structure, scene semantics, and global quality perception information. Through the aforementioned multi-stage feature extraction process, the convolutional Transformer backbone network can provide effective multi-scale visual representations for subsequent rain ripple persistence encoders, structural distortion encoders, distortion-guided feature enhancement modules, and quality prediction modules. : The multi-scale feature representation is obtained: in, This represents the visual features extracted from layer i. Shallow visual features, output from the early stages of the backbone network, have high spatial resolution and mainly include local texture, fine edges, residual rain streaks, local noise, and detail occlusion information. These features are suitable for characterizing fine-grained distortions that persist after rain removal, such as residual rain streaks, local texture destruction, and loss of edge details. Mid-level visual features, output from the middle stages of the backbone network, possess both spatial detail and semantic expressive power, mainly including texture distribution, region structure, edge continuity, local blurring, over-smoothing, and texture loss. These features can be used to describe local structural degradation and regional quality changes during rain removal. Deep visual features, output from the later stages of the backbone network, have lower spatial resolution but stronger semantic expressive power, mainly including global structure, scene semantics, target contours, overall content consistency, and global visual quality information. These features are suitable for determining whether the overall structure of a rain-removed image is destroyed, and whether the image has obvious structural distortion or semantic inconsistencies. Deep visual features are used to characterize global semantic and structural information.

[0025] Step 2: Construct rain streak residual features and structural distortion perception features for the de-rained image to be evaluated; Given an input derained image I, a rain streak residual encoder (RDE) and a structure distortion encoder (SDE) are used to model quality-related distortion information such as residual rain streaks, texture loss, edge blurring, and structure degradation in the derained image, thus obtaining rain streak residual features. and structural distortion perception features : in, Indicates channel splicing. , They represent downsampling or upsampling, respectively. Second-rate, This represents the convolution operation. This represents attention convolution. Indicates intermediate features, Indicates to Enhanced structural features obtained by residual feature extraction This indicates edge magnitude calculation. This represents a 3×3 convolution with a stride of 2.

[0026] Rain Residual Encoder: The rain residual encoder adopts an encoder-decoder structure, combined with multi-scale skip connections and multi-branch feature fusion mechanisms, to obtain feature representations with strong rain residual distortion perception capabilities. Specifically, the input image I first enters the encoder for feature extraction. The encoder consists of multiple convolutional feature extraction modules. As the network depth gradually increases, the spatial resolution of the feature maps gradually decreases, while the receptive field gradually expands. Shallow-layer encoded features are mainly used to capture fine-grained rain streaks, edge textures, and local noise perturbations; mid-layer encoded features are used to characterize rain streak distribution, local occlusion, and texture destruction; deep-layer encoded features are used to describe the impact of rain residual on a larger range of image structure and overall visual quality. In the middle part of the encoder, the network further compresses and abstracts the high-level rain streak degradation features, enabling it to extract more stable rain streak distortion priors. Subsequently, the features enter the decoder, where spatial resolution is restored through stepwise upsampling. During decoding, features from different levels of the encoder are concatenated with features from the decoder through skip connections. This design can fuse shallow, detailed rain pattern information with deep semantic degradation information, preventing the weakening or loss of subtle residual rain patterns during deep feature extraction. Furthermore, intermediate features at different scales in the decoding process undergo channel transformation and dimensionality unification via 1×1 convolutions. The 1×1 convolutions adjust the number of channels for features at each scale, reducing redundant information and enhancing the fusionability of rain pattern features at different scales. Subsequently, the rain pattern features from multiple scales are concatenated again to obtain the final residual rain pattern features. The rain pattern persistence encoder does not simply rely on a single layer of features for rain pattern detection. Instead, it extracts rain pattern degradation information from local to global levels at the encoding end, restores spatial details at the decoding end, and preserves fine-grained rain pattern responses using skip connections and multi-scale feature fusion. The final output is the rain pattern persistence feature. It can comprehensively reflect the residual rain streak intensity, rain streak spatial distribution, local texture degradation, and rain artifact information in the de-raining image to be evaluated, providing rain streak distortion priors for the subsequent distortion-guided feature enhancement module, thereby improving the ability of the no-reference de-raining image quality evaluation model to perceive rain streak-related distortion.

[0027] In some embodiments, the rain streak residual encoder is pre-trained with an adversarial generative network to learn discriminative rain streak distortion priors. The pre-training phase uses paired rainy and no-rain images as training samples, wherein the rainy images are... Input: Rainless image The labels are used for supervision. The generator network adopts an encoder-decoder structure. The encoder is responsible for extracting rain streak degradation features, and the decoder reconstructs the corresponding rainless image based on the encoded features. The discriminator network is used to distinguish between the generated image and the real rainless image, thereby improving the realism of the reconstruction result. During the training phase, image reconstruction loss and adversarial loss are jointly optimized, and their loss function is expressed as: in, For image reconstruction loss, To combat the losses, These are the weighting coefficients.

[0028] The structural distortion encoder first extracts edge information from the image using the Sobel operator, obtaining an edge response map that reflects structural contours and texture changes. By introducing edge priors, the encoder can focus more on regions in the image closely related to structural quality, such as object boundaries, texture transition regions, and detailed contour regions. Subsequently, the edge response features are sequentially input into multiple StemConv modules for initial structural feature extraction. The StemConv modules are mainly used to perform convolutional mapping and shallow feature encoding on the input edge information, transforming the raw edge response into a format suitable for subsequent network processing. After multi-level StemConv processing, the network obtains a basic structural representation including edge intensity, texture distribution, and local structural changes. Based on this, the basic structural features are further input into the ResBlock residual module. ResBlock enhances the local structural expressiveness while preserving the original structural information, mitigating information decay during deep feature extraction through residual connections, enabling the network to more stably extract edge continuity, texture integrity, and structural degradation information. The features processed by ResBlock can be represented as follows: Subsequently, the residual enhancement features are sequentially processed through two AttnConv attention convolutional modules. AttnConv adaptively weights structurally sensitive regions, making the network pay more attention to areas with blurred edges, missing textures, structural damage, and significant local distortion. The first AttnConv enhances the residual structural features with attention, resulting in primary attention-enhanced features; the second AttnConv further strengthens the response in structurally degraded regions, resulting in secondary attention-enhanced features. Through multi-level attention convolution processing, the structural distortion encoder can extract structural quality-related information from different levels of enhancement. Finally, the structural distortion encoder fuses the features from different stages through channel concatenation, including basic structural features, residual enhancement features, primary attention-enhanced features, and secondary attention-enhanced features, to obtain the final structural distortion-perceived features. .

[0029] In some embodiments, the structural distortion encoder is pre-trained using a recovery-driven supervised learning approach to learn structural distortion priors. The input is a blurred image. The supervision label is the corresponding clear image. To enhance the network's ability to perceive structural information, the Sobel operator is first used to extract edge information from the input image, which is then input into an encoder consisting of StemConv layers and residual blocks for structural feature extraction. The encoder output features are further processed through an attention prediction branch to generate a structural attention map A, and the features are then modulated. Where F represents the encoder output feature, and ⊙ denotes element-wise multiplication. The modulated features are then used by the decoder to reconstruct the corresponding sharp image.

[0030] During the training phase, image reconstruction loss and structural attention supervision loss are jointly optimized, and the overall loss function is: in, To reconstruct the image and the real, clear image loss; To predict the mean squared error loss between the attention map and the structural supervision labels. The structural supervision labels are obtained from a clear image through Sobel edge detection, thresholding, normalization, and downsampling, and are used to guide the network to focus on structurally sensitive regions.

[0031] Step 3: Enhance features by combining rain streak residual features, structural distortion perception features, and multi-scale visual representation; The rain ripple distortion features obtained in step two Structural distortion perception features and multi-scale visual representation The input distortion-guided feature enhancement module enhances the hierarchical visual features according to different types of distortion features, resulting in rain pattern enhancement features. and structural enhancement features This is to improve the network's ability to characterize regions of quality-related distortion.

[0032] First, linear mapping is performed on the rain ripple distortion features and structural distortion features respectively to generate the corresponding query features: in, Rain streak distortion characteristics Corresponding query features Perceiving features of structural distortion Corresponding query features and These respectively represent the characteristics of rain streak distortion. and structural distortion perception features The corresponding mapping matrix.

[0033] Subsequently, corresponding keys and values ​​are generated based on different levels of visual features: Among them, shallow visual features Used to characterize local texture and rain pattern information, deep visual features Used to represent global semantic and structural information.

[0034] Furthermore, information interaction between distorted features and multi-scale visual representations is achieved through a cross-attention mechanism, the computation of which is expressed as follows: Where d represents the feature dimension, and Q, K, and V represent the query feature, key feature, and value feature, respectively. When the input is ( When rain streak enhancement features are obtained. When the input is ( When obtaining structural enhancement features, Finally, the enhanced features are output to step four for fusion to obtain more accurate quality-perceived features.

[0035] Step 4: Input the multi-scale visual representation, rain pattern enhancement features, and structure enhancement features into the feature fusion module and the quality prediction module to fuse quality-related features from different sources, establish the mapping relationship between visual features and subjective quality scores, and output the quality evaluation results of the rain-removed image.

[0036] Figure 3 This is a network structure diagram of a feature fusion module according to an embodiment of the present invention. Figure 4 This is a network structure diagram of a quality prediction module according to an embodiment of the present invention, referred to... Figure 3 and Figure 4 Step four includes the following: First, the multi-scale visual representations are aggregated using attention pooling, and then the rain pattern enhancement features are added. and structural enhancement features By splicing the sequences together, a unified quality-perceived feature sequence is constructed: Furthermore, a lightweight Transformer is used to perform feature interaction and fusion on the quality-aware feature sequence, establishing the correlation between different features to obtain the fused quality representation. in, The feature fusion function, for example, concatenates multi-scale visual representations, rain pattern enhancement features, and structural enhancement features through channel concatenation, and then fuses them through a 1×1 convolution or linear layer to obtain the i-th layer fused feature. Global pooling is then performed on the fused features from each layer, and the concatenation yields the final quality representation used for quality prediction. This indicates the quality characteristics after fusion.

[0037] Finally, the fused quality features are input into the quality prediction module to establish a mapping relationship between the quality features and subjective visual scores, thereby obtaining the quality prediction result of the rain-removed image. Among them, R This represents the quality regression function. This represents the predicted image quality score; , This represents the weight parameters of the fully connected layer; , Indicates the bias parameter; This represents a non-linear activation function, such as ReLU, GELU, or LeakyReLU.

[0038] When quality scores need to be limited to a fixed range, such as the range of subjective quality scores... , The Sigmoid function can be used for further normalization mapping. in, This indicates the final image quality evaluation result. and These represent the minimum and maximum quality scores, respectively. If the training labels have been normalized to... Then it can be used directly. As the final quality score; if the training label is an unnormalized MOS or DMOS score, it can be linearly scaled according to the label value range.

[0039] Through the aforementioned feature fusion function and quality regression function, the model can jointly model the multi-scale visual content information extracted by the backbone network with rain pattern residue features and structural distortion perception features, so that the final quality prediction result can simultaneously reflect the intensity of residual rain patterns, the degree of texture loss, the degree of edge blurring, the degree of structural degradation, and the overall visual perception quality of the image.

[0040] This invention also provides a distortion-aware, reference-free deraining image quality assessment system, applicable to the aforementioned distortion-aware, reference-free deraining image quality assessment method. Figure 5 This is a system block diagram of a distortion-aware, reference-free deraining image quality assessment system according to an embodiment of the present invention, such as... Figure 5 As shown, the system includes: The image acquisition module is used to acquire the rain-removed image to be evaluated, perform unified input processing on the rain-removed image, and input the processed rain-removed image into the backbone feature extraction network that combines local convolutional feature extraction and self-attention feature interaction to obtain a multi-level visual representation containing local texture, structural contour and global semantic information. The feature extraction module is used to construct rain streak residual features and structural distortion perception features based on the de-rained image. The rain streak residual features are obtained by encoding-decoding feature modeling oriented towards rain streak degradation information, and the structural distortion perception features are obtained by introducing edge priors and performing attention encoding on structurally sensitive regions. The feature enhancement module is used to interactively enhance the rain pattern residual features and structural distortion perception features as distortion guidance information with the multi-level visual representation to obtain rain pattern enhancement features and structural enhancement features. The quality assessment module is used to fuse multi-level visual representations, rain pattern enhancement features, and structural enhancement features to obtain quality-perceived features, and output the quality assessment results of the de-rained image based on the quality-perceived features.

[0041] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0042] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0043] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0044] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0045] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0046] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A distortion-aware, reference-free deraining image quality assessment method, characterized in that, The methods include: The rain-removed image to be evaluated is obtained, and the rain-removed image is subjected to unified input processing. The processed rain-removed image is then input into a backbone feature extraction network that combines local convolutional feature extraction with self-attention feature interaction to obtain a multi-level visual representation containing local texture, structural contour and global semantic information. Based on the de-rained image, rain streak residual features and structural distortion perception features are constructed. The rain streak residual features are obtained by encoding-decoding feature modeling oriented towards rain streak degradation information, and the structural distortion perception features are obtained by introducing edge priors and performing attention encoding on structurally sensitive regions. Rain pattern residual features and structural distortion perception features are used as distortion guidance information and interactively enhanced with multi-level visual representation to obtain rain pattern enhancement features and structural enhancement features. Multi-level visual representations, rain pattern enhancement features, and structural enhancement features are fused to obtain quality-perceived features, and the quality evaluation results of the de-rained image are output based on the quality-perceived features.

2. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 1, characterized in that, The unified input processing includes at least one of the following: image scale adaptation, pixel value normalization, channel format unification, data type conversion, and boundary completion of the derained image. Edge information or gradient information is generated based on the derained image and used as auxiliary input in the process of constructing structural distortion-aware features.

3. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 1, characterized in that, The backbone feature extraction network includes an initial feature mapping unit and multiple staged feature extraction units. Each staged feature extraction unit includes a local spatial feature extraction subunit and a global correlation modeling subunit. The local spatial feature extraction subunit performs convolutional encoding on texture, edge, and rain pattern residue, while the global correlation modeling subunit performs self-attention interaction on features at different spatial locations.

4. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 3, characterized in that, Multi-level visual representation includes shallow visual features, mid-level visual features, and deep visual features output from different stages of the backbone feature extraction network. Among them, shallow visual features participate in the enhancement of rain pattern residue-related features, mid-level visual features participate in the enhancement of texture distribution and local structure-related features, and deep visual features participate in the enhancement of global structure and semantic-related features.

5. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 1, characterized in that, Rain streak residual features are obtained through a rain streak residual encoder, which includes an encoder, a decoder, a skip connection unit, and a multi-scale fusion unit. The encoder extracts layered rain streak degradation features from the de-rained image, the decoder performs spatial restoration on the layered rain streak degradation features, the skip connection unit introduces features from different levels of the encoder into the decoder, and the multi-scale fusion unit performs channel adaptation and splicing fusion on features from different scales of the decoder.

6. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 5, characterized in that, Before being used for quality assessment, the rain pattern residual encoder undergoes rain pattern degradation prior learning. The rain pattern degradation prior learning uses rainy day images and corresponding rainless images as training samples. The rainy day images are input into the generative encoder-decoder network to obtain the restored images, and the restored images and rainless images are distinguished by the discriminant network. The training constraints include image restoration constraints and adversarial discrimination constraints.

7. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 1, characterized in that, The structural distortion-aware features are obtained through a structural distortion encoder, which includes an edge prior extraction unit, a shallow structure encoding unit, a residual enhancement unit, an attention convolution unit, and a feature concatenation unit. The edge prior extraction unit extracts structural contour information from the de-rained image, the shallow structure encoding unit performs convolutional mapping on the structural contour information, the residual enhancement unit performs residual encoding on the structural features after convolutional mapping, the attention convolution unit performs weighted encoding on the structurally sensitive regions, and the feature concatenation unit fuses the structural features from different encoding stages.

8. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 7, characterized in that, Before being used for quality assessment, the structure-distortion encoder undergoes structure restoration prior learning. The structure restoration prior learning uses degraded images and corresponding clear images as training samples. The structure encoding features are extracted based on the edge information of the degraded images. The attention prediction branch generates a structure attention map and modulates the structure encoding features. Then, the decoding branch reconstructs the clear image. The training constraints include image restoration constraints and structure attention supervision constraints.

9. The method for evaluating the quality of a no-reference deraining image based on distortion perception according to claim 1, characterized in that, Interactive feature enhancement includes mapping rain pattern residual features and structural distortion perception features to distortion query features, mapping multi-level visual representations to key features and value features, and obtaining rain pattern enhancement features and structural enhancement features through cross-attention. Fusion includes attention aggregation of multi-level visual representations, combining the aggregated multi-level visual representations with rain pattern enhancement features and structural enhancement features to form a quality perception feature sequence, and outputting quality evaluation results through feature interaction network and quality regression network.

10. A distortion-aware, reference-free deraining image quality assessment system, characterized in that, A distortion-aware, no-reference deraining image quality assessment method applicable to any one of claims 1 to 9, the system comprising: The image acquisition module is used to acquire the rain-removed image to be evaluated, perform unified input processing on the rain-removed image, and input the processed rain-removed image into the backbone feature extraction network that combines local convolutional feature extraction and self-attention feature interaction to obtain a multi-level visual representation containing local texture, structural contour and global semantic information. The feature extraction module is used to construct rain streak residual features and structural distortion perception features based on the de-rained image. The rain streak residual features are obtained by encoding-decoding feature modeling oriented towards rain streak degradation information, and the structural distortion perception features are obtained by introducing edge priors and performing attention encoding on structurally sensitive regions. The feature enhancement module is used to interactively enhance the rain pattern residual features and structural distortion perception features as distortion guidance information with the multi-level visual representation to obtain rain pattern enhancement features and structural enhancement features. The quality assessment module is used to fuse multi-level visual representations, rain pattern enhancement features, and structural enhancement features to obtain quality-perceived features, and output the quality assessment results of the de-rained image based on the quality-perceived features.

Images