A method for recovering a severe weather image based on self-adaption during continuous testing

Abstract

This invention relates to an adaptive severe weather image restoration method based on continuous testing, belonging to the field of computer vision and image processing technology. It includes the following steps: constructing a severe weather dataset and dividing it into training and testing sets; constructing a severe weather image restoration framework, including a pre-trained model DA-CLIP, a student model, and a teacher model; inputting the publicly available dataset into the pre-trained model DA-CLIP for training, and initializing the student and teacher models using the parameters of the trained DA-CLIP model; inputting the training set of the severe weather dataset into the student and teacher models respectively for model training and optimization, obtaining a trained severe weather image restoration framework; inputting the test set images of the severe weather dataset into the trained severe weather image restoration framework to obtain the image restoration result. This invention can improve the model's image restoration performance under complex and realistic weather conditions.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to an adaptive method for restoring severe weather images during continuous testing. Background Technology

[0002] With the widespread application of vision systems in complex environments, such as autonomous driving, intelligent security, and remote sensing monitoring, image degradation caused by severe weather has become a core bottleneck restricting the performance of downstream tasks. Meteorological factors such as rain, snow, and fog, through scattering, absorption, and occlusion mechanisms, severely damage image contrast, color fidelity, and structural integrity, leading to significant performance degradation of algorithms such as object detection and semantic segmentation in real-world deployments. Current mainstream image restoration methods mostly focus on modeling single degradation types, heavily relying on large amounts of labeled data to train dedicated models. Their generalization ability is limited by the coverage of preset degradation patterns, making them vulnerable to complex meteorological interference or unknown environmental disturbances, and failing to meet the dynamic and ever-changing visual perception needs of open worlds.

[0003] Existing research often employs unified restoration methods for multiple degradation scenarios. By constructing multi-task learning frameworks, introducing conditional control mechanisms, and combining attention mechanisms with conditional normalization strategies, these methods have improved the generalization ability and deployment efficiency of the models to some extent. However, the adaptability of these all-in-one methods is still limited by the degradation patterns learned during the training phase, and they cannot accurately capture degradation types, intensity combinations, and complex degradation phenomena that are not fully covered in real-world scenarios. Traditional conditional control relies on coarse-grained degradation classification, making it difficult to achieve fine-grained modeling of degradation degree, spatial distribution, and interactions. Even with the introduction of continuous testing feedback, the "rigidity" of the model's internal feature learning and decision-making mechanisms cannot be transformed into fine-grained processing strategies, making it difficult to ensure the continuous and stable operation of the system in complex environments. Summary of the Invention

[0004] To solve the above problems, the present invention provides the following technical solution: This invention provides a method for reconstructing severe weather images based on continuous testing and adaptation, comprising the following steps: S1. Construct a severe weather dataset and divide it into a training set and a test set; Furthermore, using a fog effect simulation method based on linear weighted mixing, original images from the publicly available snow100k dataset are selected and processed to generate a snow fog dataset. An image rain effect synthesis method based on random rain line modeling and Gaussian blur fusion is then used to process the original images from the snow100k dataset to generate a rain and snow dataset. The fog effect simulation method and the image rain effect synthesis method are then used to sequentially process the original images from the snow100k dataset to generate a rain, fog, and snow coexistence dataset. Finally, the snow fog dataset, rain and snow dataset, and rain, fog, and snow coexistence dataset are merged to obtain a severe weather dataset.

[0005] Furthermore, the steps of the fog effect simulation method include: S111. Perform range constraint processing on the preset fog intensity parameter ρ, normalize it to the closed interval [0,1], and obtain the normalized fog intensity parameter. ; S112. Create a full-channel fog base layer A with the same spatial dimensions as the input image, including height H and width W. The color intensity of each pixel in the full-channel fog base layer is precisely controlled by a configurable parameter. S113. Using a weighted mixing function, the original input image is... Linear blending is performed with the full-channel fog effect base layer A to obtain a composite image with a fog effect. .

[0006] Furthermore, the steps of the image rain effect synthesis method include: S121. Based on the spatial dimension of the input image, create a single-channel initial rain layer mask, wherein all pixel values ​​of the initial rain layer mask are set to zero. S122. On the initial rain layer mask, generate N raindrop lines with specific lengths and directions. The generation process of each raindrop line includes: randomly generating starting coordinates. and Randomly generate the pixel length of the raindrop lines. Based on the starting coordinates and length, and combined with the preset raindrop falling angle, calculate the endpoint coordinates of the raindrop line. and In the current rain layer mask, draw a straight line segment defined by the start and end points using white pixel values; S123. Apply Gaussian blur filtering to the generated initial rain layer mask to obtain a single-channel Gaussian blurred rain layer mask. S124. The single-channel Gaussian blur rain layer mask is copied into a three-channel version to match the number of channels in the input image. The resulting three-channel rain layer is obtained after channel conversion. S125. Linearly blend the input image with the three-channel rain layer after channel conversion to obtain the final composite image of the rain effect. .

[0007] S2. Construct a severe weather image restoration framework, including a pre-trained model DA-CLIP, a student model based on DA-CLIP, and a teacher model based on DA-CLIP; input the publicly available dataset into the pre-trained model DA-CLIP for training, and initialize the student model and the teacher model using the model parameters of the trained pre-trained model DA-CLIP; Furthermore, the DA-CLIP-based student model includes a DA-CLIP encoder, a fully connected linear layer, a ReLU activation function, an unflatten operation, several transposed convolutional modules (ConvTranspose2d), a batch normalization layer (BatchNorm2d), and a sigmoid activation function; after each transposed convolutional module (ConvTranspose2d), a batch normalization layer (BatchNorm2d) and a ReLU activation function are sequentially connected. The input image is encoded by a DA-CLIP encoder to obtain student model image features. These student model image features undergo dimensionality transformation through a fully connected linear layer, mapping the encoded high-dimensional feature vectors to a dimensional space suitable for subsequent image reconstruction processing, resulting in dimensional transformation features. These dimensional transformation features are then subjected to a ReLU activation function to introduce a non-linear transformation, enhancing the model's expressive power. An Unflatten operation then reshapes the one-dimensional feature vectors into a two-dimensional feature map with spatial structure, thereby restoring its spatial topology in the image, resulting in reshaped features. These reshaped features undergo upsampling through a first transposed convolutional module (ConvTranspose2d), followed by batch normalization (BatchNorm2d) and ReLU activation, before being input into a second transposed convolutional module (ConvTranspose2d) for further upsampling. This process continues until the feature map, after five upsampling layers, is processed by batch normalization (BatchNorm2d) and ReLU activation, and then input into a Sigmoid activation function to obtain the restored image of the student model.

[0008] Furthermore, the initialization parameters of the DA-CLIP-based teacher model are set by deeply replicating the parameters of the DA-CLIP-based student model. During training, the parameters are dynamically updated through an exponential moving average mechanism to gradually absorb the effective knowledge learned by the student model. The DA-CLIP-based teacher model also includes a DA-CLIP encoder, a fully connected linear layer, a ReLU activation function, an unflatten operation, several transposed convolutional modules (ConvTranspose2d), a batch normalization layer (BatchNorm2d), and a sigmoid activation function. Each transposed convolutional module (ConvTranspose2d) is followed by a batch normalization layer (BatchNorm2d) and a ReLU activation function.

[0009] S3. Input the training set of the severe weather dataset into the student model and the teacher model respectively for model training. Optimize the severe weather image restoration framework by calculating the difference between the restoration results of the teacher model and the restoration results of the student model, update the student model and the teacher model, and obtain the trained severe weather image restoration framework. Furthermore, the model is trained through a multi-loss function collaborative optimization mechanism, including a contrastive loss function and a consistency loss function, and a total loss function is constructed based on the contrastive loss function and the consistency loss function.

[0010] Furthermore, the construction process of the loss function is compared as follows: The student model reconstructs images using a pre-trained VGG-16 network as a feature encoder. Teacher model restoration image and input image Feature extraction is performed, followed by L2 norm normalization to obtain the normalized anchor feature vector. Positive sample feature vector and negative sample feature vectors ; Calculate the cosine similarity between the anchor feature vector and the positive sample feature vector to obtain the first cosine similarity. Calculate the cosine similarity and second cosine similarity between the anchor feature vector and the negative sample feature vector. ; first cosine similarity Divide by a temperature parameter T to obtain the first scaled similarity. Similarly, the second scaled similarity is obtained. Based on the first scaled similarity Second scaling similarity The contrastive loss function is constructed as follows: , in, This represents the contrastive loss function.

[0011] Furthermore, the consistency loss function measures the image restored by the student model using the L1 loss function. and the restored image of the teacher model Differences at the pixel level.

[0012] Furthermore, the parameters of the student and teacher models are updated: The student model uses an online learning strategy for parameter updates. The update trigger mechanism is controlled by the following two conditions: when the total loss is greater than a preset threshold, or when the number of samples processed is a multiple of 3, the parameter update process of the student model is triggered. During the parameter update process, the system clears all accumulated gradients to zero, calculates the gradient of the total loss with respect to all trainable parameters of the student model through the backpropagation algorithm, clips the calculated gradients to limit the L2 norm of the gradients to within 1.0, and finally uses the Adam optimizer to update the parameters of the student model. The teacher model uses an exponential moving average strategy for parameter updates. Whenever the student model completes a parameter update, the system automatically triggers the teacher model's update process. The teacher model's update does not require gradient calculation; it updates parameters using the exponential moving average formula.

[0013] S4. Input the test set images of the severe weather dataset into the trained severe weather image restoration framework to obtain the image restoration results.

[0014] The advantages of this invention are: This invention constructs a more realistic scene using multimodal severe weather data. Compared to existing methods that typically model only a single degradation type (such as defogging or deraining only), this invention systematically synthesizes image data of snow, fog, rain, and their combinations through methods such as linear weighted mixed fog effect simulation, random rain line modeling, and Gaussian blur fusion. This constructs a unified severe weather dataset covering multiple degradation modes, significantly improving the model's generalization ability to complex real-world weather conditions. By introducing a degradation-aware pre-training mechanism, the invention achieves decoupling of degradation type and content features. The pre-trained model has an independent degradation-aware control branch, and through feature decoupling design and control signal guidance mechanisms, the model can simultaneously learn the high-level semantic content and degradation attributes of the image, thereby achieving accurate identification and targeted repair of different severe weather types during the restoration process. To enhance the online robustness of the model, a student-teacher architecture that adapts during continuous testing is adopted. A student-teacher collaborative update mechanism is proposed to introduce a student-teacher collaborative update mechanism during inference. The student model performs online gradient updates through multiple loss functions, while the teacher model maintains output stability through exponential moving average. The collaboration between the two achieves adaptive optimization during the testing phase, effectively addressing out-of-distribution or unseen composite degradation scenarios. The high-fidelity consistency of the restoration results is ensured through a multi-loss collaborative optimization strategy. By combining contrastive loss (a similarity measure based on the pre-trained feature space) and pixel-level consistency loss, the outputs of the student and teacher models are constrained at both the feature level and the pixel level, ensuring that the restored image is not only visually clear and natural, but also highly consistent with the original clean image in terms of high-level semantics. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0016] Figure 1 This is a flowchart of the steps of the method of the present invention; Figure 2 The figure shows the experimental results of the method of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1 In this embodiment, as Figure 1 As shown, this invention provides a method for image restoration of severe weather based on continuous testing and adaptation, the specific steps of which include: The first step is to construct a severe weather dataset. This involves generating a snow and fog dataset, achieved through a fog effect simulation method based on linear weighted mixing. During implementation, the system first applies strict range constraints to the preset fog intensity parameters, precisely normalizing them to the closed interval [0,1] to ensure the physical validity and controllability of the parameters. This normalization operation follows the mathematical formula: Where ρ represents the original, unprocessed input fog intensity parameter, and This is the normalized fog intensity parameter after precise processing.

[0019] Subsequently, the system creates a full-channel fog base layer with the exact same spatial dimensions (height H and width W) as the original image to be processed. The color intensity of each pixel in this base layer is precisely controlled by a configurable parameter; this invention sets its default value to 255, which represents a pure white fog effect, aiming to simulate the uniform distribution of scattered light in an image in a natural environment. Then, the system uses the weighted blending function provided by the OpenCV image processing library to linearly blend the original input image with the aforementioned full-channel fog base layer, thereby generating a composite image with a realistic fog effect. This blending process is based on the following mathematical expression: in, A represents the pixel values ​​of the original clean image; A represents the pixel values ​​of the full-channel haze base layer, which is set to 255 in this embodiment; B represents a constant offset, usually set to 0; the parameter τ represents the transmittance weight of the original input image in the final blending result, and its value is strictly limited to [0,1]. This transmittance weight τ is determined by... The calculation shows that, among which These are the fog intensity parameters that have been previously normalized.

[0020] Furthermore, to construct a rain and snow dataset, this method employs an image rain effect synthesis approach based on random rain line modeling and Gaussian blur fusion, simulating the visual effects of rain through a series of refined operations. First, rain layer initialization is performed. Based on the specific spatial dimensions of the input image, the system creates an initial rain layer mask. The dimensions of this mask are set to H×W×1, where H and W represent the height and width of the input image, respectively, and "1" indicates that it is a single-channel image. All pixel values ​​in this initial mask are set to zero, serving as a blank canvas for drawing raindrop lines. Next, the raindrop line generation stage begins. On this initial rain layer mask, the system generates N raindrop lines with preset lengths and directions.

[0021] The generation of each raindrop line includes the following steps: randomly determining the starting coordinates, where the starting coordinates in the horizontal direction are... By using the closed interval [0, W] 1] is obtained by uniformly and randomly sampling an integer value, i.e. Starting coordinates in the vertical direction Similarly, by using the closed interval [0, H] 1] is obtained by uniformly and randomly sampling an integer value, i.e. Where i represents the index of the i-th raindrop line currently being generated. This represents the operation of randomly sampling an integer.

[0022] Randomly generate the pixel length of each raindrop line. Its value is within the preset minimum length. and maximum length The interval is determined by uniformly random sampling of integer values, i.e. Based on the determined starting coordinates and length, and combined with the preset raindrop falling angle (set to 45° in this embodiment), the endpoint coordinates of the raindrop line are calculated. The specific calculation formula is as follows: and .

[0023] In the current rain layer mask, the system uses an image drawing function to accurately draw a straight line segment defined by the start and end points with a white pixel value (255). The line thickness is usually set to 1 to 2 pixels, thus clearly constructing raindrop lines on the rain layer mask. The above series of operations are performed sequentially on all N raindrop lines, and the rain layer mask is updated in real time for each operation. After all raindrop lines are generated, a Gaussian blur filter is applied to the generated rain layer mask. The filtering operation uses a preset convolution kernel size (e.g., 5x5) and standard deviation (e.g., ...). To simulate the visual blurring effect and motion trajectory of raindrops, a single-channel Gaussian blur rain layer mask is obtained. Next, channel conversion is performed to duplicate the single-channel Gaussian blur rain layer mask into a three-channel version, ensuring it matches the number of channels in the input image, thus preparing for subsequent image blending operations. Finally, weighted compositing is performed, linearly blending the input image with the channel-converted three-channel rain layer to obtain the final composite image with the rain effect. The compositing calculation formula is: in, R(p) is the pixel value vector of the original clean image at pixel position p; R(p) is the pixel value vector of the synthesized three-channel rain layer at pixel position p. This is the transparency parameter of the rain layer, and its value is limited to [0,1], which is used to precisely control the prominence of the rain effect in the composite image.

[0024] Furthermore, similarly, to construct a rain, fog, and snow coexistence dataset, this invention employs the aforementioned fog effect simulation method and rain effect synthesis method to sequentially process the original images in the snow100k dataset. In practice, the rain effect synthesis method is first applied to the original snow scene image to obtain a rain and snow image. Then, the fog effect simulation method is applied to this rain and snow image, ultimately yielding a rain, fog, and snow coexistence image. Additionally, this invention can also select other processing sequences based on actual needs and simulation effects, such as generating snow and fog images first and then superimposing rain effects, or generating rain images first and then superimposing fog effects. Both methods effectively obtain datasets simulating complex severe weather. Finally, the snow and fog dataset, rain and snow dataset, and rain, fog, and snow coexistence dataset generated by the above three methods are merged to construct a comprehensive severe weather dataset (mixdataset) for model training and testing.

[0025] The second step involves constructing a comprehensive severe weather dataset. This method further develops a continuously test-time adaptive severe weather image restoration framework (CT-DA-CLIP) based on the image restoration model DA-CLIP. This framework enables robust restoration of severe weather images. Its core components include the pre-trained model DA-CLIP, a student model based on DA-CLIP, and a teacher model based on DA-CLIP.

[0026] The pre-trained model DA-CLIP has the following structural and functional characteristics: Firstly, the basic vision-language matching module. This module mainly includes a visual feature extraction unit, a text feature extraction unit, and feature alignment and scaling parameters. The visual feature extraction unit adopts the ViT-B-32 network structure of Vision Transformer (ViT). This network structure consists of a series of deeply stacked Transformer encoder layers. Each encoder layer contains a multi-head self-attention mechanism and a feed-forward network. These components work together to take a 224x224 pixel RGB image as input and encode it into a series of visual feature tags with rich semantic information. Then, through global pooling or extraction of specific category tokens (CLS tokens), it is converted into a fixed-dimensional visual feature vector. This vector accurately captures the global and local semantic content of the image. The text feature extraction unit adopts a text encoder based on the Transformer architecture to encode the input natural language text (text descriptions of image degradation types, such as "hazy fog" or "dense rain lines") into a fixed-dimensional semantic feature vector. The feature alignment scaling parameter is a learnable scalar whose function is to dynamically adjust the scale of visual feature vectors and text feature vectors when calculating cosine similarity, thereby optimizing the similarity calculation accuracy in cross-modal feature space. This parameter is usually used as a temperature parameter in contrastive learning loss function.

[0027] Secondly, the degradation-aware control branch. This branch is specifically designed to deeply perceive and encode degradation features of the input image. Its main components include a degradation feature encoder and a degradation pattern extractor, employing a parameter isolation mechanism to ensure functional independence. The degradation feature encoder uses the exact same network structure as the basic visual feature extraction unit (ViT-B-32), but all its weights and bias parameters are isolated from and independently trainable. This allows the encoder to focus on capturing visual features highly correlated with severe weather degradation from the input image, thus avoiding interference from image content feature extraction tasks during its learning process. The degradation pattern extractor is closely connected to the back end of the degradation feature encoder. Its structure consists of one or more fully connected layers (Multi-Layer Perceptron, MLP). This extractor is specifically used to further abstract and process the features output by the degradation feature encoder, ultimately generating a high-dimensional degradation feature vector that accurately represents the type (e.g., fog, rain, snow), intensity, and spatial distribution of image degradation. The parameter isolation mechanism means that all weights and bias parameters of the degradation perception control branch are independent of the basic vision-language matching module during model initialization and subsequent training and update, so as to completely decouple the two branches in feature learning and effectively avoid feature confusion.

[0028] Third, the dual-path feature output structure. This pre-trained model, through collaborative processing, can generate feature vectors of two different modalities in parallel. First, the content feature vector is generated by the basic visual feature extraction unit. This vector mainly represents the high-level semantic content of the input image, such as the categories of objects present in the image, the layout structure of the scene, and other information, and has robustness to changes in the image content itself. Second, the degradation feature vector is generated by the degradation-aware control branch. This vector accurately represents the severe weather degradation attributes suffered by the input image, including but not limited to micro-degradation details such as fog concentration, rain line density and direction, and snowflake size and distribution.

[0029] Fourth, the degradation perception collaborative mechanism. This mechanism finely regulates the content feature generation process through degradation features to achieve efficient and targeted image restoration. First, there is a control signal guidance mechanism. The hidden states generated during processing by the visual degradation perception subsystem (i.e., the aforementioned degradation perception control branch) (such as activation values ​​of intermediate layers or the final generated degradation feature vector) are cleverly used as control signals to dynamically regulate the feature generation process of the basic visual feature extraction unit. This regulation can be achieved through advanced mechanisms such as conditional normalization, allowing the extraction of content features to be adjusted and optimized in a targeted manner based on the real-time detected degradation type. Second, there is a feature decoupling design. This invention introduces specific regularization terms or loss functions during model design and training to ensure that the content feature vector and the degradation feature vector are mutually orthogonal or approximately orthogonal in the feature space. This minimizes information redundancy and cross-coupling between the two types of features, enabling them to independently carry the semantic content information and degradation attribute information of the image. This is crucial for the downstream restoration task to accurately decouple degradation. Furthermore, during the training process, this invention can selectively freeze the parameters of the basic vision-language matching module according to specific task requirements and optimization goals, and train only the visual degradation perception subsystem.

[0030] Furthermore, within the CT-DA-CLIP framework, the student and teacher models based on DA-CLIP employ the exact same network architecture as DA-CLIP, but perform different functions within the overall framework. The student model, as the core learning entity of the entire framework, operates as follows: When a low-quality input image is received, it is first encoded by the DA-CLIP encoder (referring to the basic visual feature extraction unit of DA-CLIP) to obtain the student model's image features. These student model image features are first transformed through a fully connected linear layer to map the encoded high-dimensional feature vector to a dimensional space suitable for subsequent image reconstruction processing. Subsequently, the linearly transformed features undergo a non-linear transformation using the ReLU (Rectified Linear Unit) activation function to enhance the model's expressive power. Through an Unflatten operation, the one-dimensional feature vector is cleverly reshaped into a two-dimensional feature map with spatial structure, thereby restoring its spatial topology in the image. Then, the spatial feature map is sequentially upsampled through five ConvTranspose2d (transposed convolution) modules. Each upsampling module is closely followed by a BatchNorm2d (batch normalization) layer to stabilize the training process, and a ReLU activation function to maintain the network's non-linear expressive power. The ConvTranspose2d module recovers the image details and resolution through learning, using a 2x2 stride and corresponding padding to efficiently achieve a 2x upsampling. Finally, the feature map after five layers of fine upsampling and feature refinement is output as the restored image of the student model through a Sigmoid activation function. The Sigmoid function precisely maps the pixel value range to [0,1], which is fully compatible with the standard image pixel value range.

[0031] The teacher model serves as a carrier providing stable knowledge and guiding the student model's learning. Its initial parameters are obtained by deeply replicating the student model's parameters, ensuring consistency of the initial state. However, during subsequent training, the teacher model's parameters are not updated through direct gradient backpropagation, but rather dynamically through an exponential moving average (EMA) mechanism to guarantee the stability and robustness of its parameters. The teacher model's workflow is as follows: The low-quality input image first undergoes preprocessing through a data augmentation module. In this module, random-intensity Gaussian noise is added to the input image. The mean of this Gaussian noise is set to 0, and the standard deviation is randomly selected within a preset range (e.g., 0.01~0.1) to simulate different degrees of test-time domain changes, thereby effectively enhancing the model's robustness and generalization ability. The Gaussian noise-enhanced image is then input into the DA-CLIP encoder (referring to the DA-CLIP basic visual feature extraction unit) to obtain the teacher model's image features. The subsequent processing flow of the teacher model image features is exactly the same as that of the student model image features. That is, the features are transformed into a spatial feature map by passing through a linear layer, ReLU activation function, and Unflatten operation in sequence. Then, upsampling is performed through five ConvTranspose2d modules. Each upsampling module is followed by a BatchNorm2d normalization layer and ReLU activation function. Finally, the restored image of the teacher model is output through Sigmoid activation function.

[0032] The third step, to effectively utilize pre-trained knowledge and optimize model initialization, involves inputting the publicly available snow100k dataset into the pre-trained model DA-CLIP for training, and then using the parameters of the pre-trained DA-CLIP model to initialize both the student and teacher models. The specific implementation details of this step demonstrate the strategic nature and phased optimization of model training.

[0033] The DA-CLIP model training consists of two key stages. The first stage aims to enable the model to accurately learn and distinguish different types of image degradation, outputting image features with strong degradation-awareness capabilities. This stage utilizes a large-scale image-text pair dataset (such as a dataset containing millions of images and their corresponding text descriptions) and trains through contrastive learning to maximize the similarity between matching image-text pairs while minimizing the similarity between mismatched pairs, thereby learning robust, cross-modal features in a shared embedding space. In this stage, the degradation-aware control branch is also co-trained, enabling it to learn degradation features decoupled from image content features, thus achieving accurate capture of image degradation attributes.

[0034] In the second stage, the system utilizes the pre-trained DA-CLIP model from the first stage to extract degradation-perceived features from the input degraded image. These extracted features serve as conditional information to further train a specialized image inpainting model. This model incorporates stochastic differential equations (SDEs) for diffusion modeling to achieve high-quality image restoration. The SDE diffusion model defines a forward diffusion process that progressively adds noise to the data and learns an inverse denoising process to recover the original data from the noise. During training, the model solves the inverse SDE process to progressively denoise the noisy image to a clean, noise-free image. The L1 loss function (i.e., pixel-level mean absolute error) is used to supervise the quality of image reconstruction, ensuring that the model can generate high-fidelity restored images with textures and details highly consistent with the original image.

[0035] After completing the two training phases described above, the weights of the trained image restoration model DA-CLIP are assigned to the image restoration student model and the image restoration teacher model, respectively, as their initial states. In practice, the weights of the DA-CLIP encoder (basic visual feature extraction unit) are precisely used to initialize the corresponding encoder parts in the student and teacher models. Furthermore, if the second phase of DA-CLIP training includes an image inpainting decoder (such as a decoder based on a U-Net architecture), the weights of this decoder are also used to initialize the corresponding parts of the newly constructed image restoration decoder (composed of linear layers, ReLU, Unflatten, and ConvTranspose2d module sequences) in the student and teacher models, thus ensuring that the model possesses powerful image reconstruction capabilities from initialization.

[0036] Step 4: After model initialization, the images from the constructed severe weather dataset (mixdataset) are input into the student model and teacher model respectively for image restoration, thus obtaining the restoration results of the student model and the teacher model. Specific steps include: The first step is obtaining the student model reconstruction results. As the core learning entity in the framework, the student model directly receives a low-quality input image y. This image y is first processed by the DA-CLIP encoder (whose weights are initialized by the pre-trained DA-CLIP model) to extract high-dimensional student model image features. These features then undergo a series of sophisticated decoding and reconstruction operations: First, a fully connected linear layer performs a dimensionality transformation to map the features from their encoding space to a dimension more suitable for image reconstruction. Second, a ReLU activation function introduces non-linearity to enhance the model's expressive power and feature complexity. Third, an Unflatten operation reshapes the original one-dimensional feature vector into a two-dimensional feature map with spatial structure, thus restoring its topological structure in the image. Finally, this spatial feature map is sequentially upsampled through five ConvTranspose2d modules. Each ConvTranspose2d module is followed by a BatchNorm2d layer to stabilize the training process and a ReLU activation function to maintain the non-linearity of the features. The ConvTranspose2d module typically uses a 2x2 kernel size and a 2-step size to improve the resolution of the feature map layer by layer. Finally, after multi-layer upsampling and refined feature processing, the feature map is normalized to the range of pixel values ​​between [0,1] by a Sigmoid activation function, thereby outputting the restored image of the student model.

[0037] Secondly, the acquisition of the teacher model's reconstruction results is crucial. The teacher model, as a knowledge carrier providing stable and high-quality output, is constructed by deeply replicating the initial parameters of the student model, ensuring both possess the same capabilities from the outset. Unlike the student model, the teacher model's parameters are frozen during normal training, not participating in direct backpropagation updates. Instead, they are dynamically updated using an exponential moving average (EMA) mechanism to maintain model stability and gradually absorb the effective knowledge learned by the student model. Regarding input processing, the low-quality input image y received by the teacher model is first preprocessed using Gaussian noise enhancement. Specifically, random-intensity Gaussian noise is added to the input image y, with a mean of 0 and a standard deviation uniformly selected within a preset range (e.g., 0.01~0.1, with a uniform random selection centered at 0.05). This generates an image with additional noise interference, thereby enhancing the model's robustness in the testing domain. This enhanced image is then input into the DA-CLIP encoder to obtain the teacher model's image features. The subsequent processing flow of the teacher model's image features is exactly the same as that of the student model's image features. Specifically, it sequentially performs dimensionality transformation through a linear layer, ReLU activation, unflattening to reshape the features into a spatial feature map, and upsampling using five ConvTranspose2d modules. Each upsampling module is followed by BatchNorm2d normalization and ReLU activation. Finally, the restored teacher model image is output through a Sigmoid activation function. The formula for calculating the teacher model's weight update is: , in, The weights of the teacher model updated at the current time step; α represents the weights of the teacher model at the previous time step; α is a smoothing factor, which is usually close to 1 and is used to control the stability of the teacher model weight updates. The weights of the student model are updated at the current time step.

[0038] The EMA mechanism allows the parameters of the teacher model to change slowly and stably, enabling it to gradually absorb the effective knowledge learned by the student model. At the same time, it effectively smooths out the performance fluctuations and noise that may occur during the training of the student model, thereby maintaining the continuous stability and high quality of the teacher model's output.

[0039] To further optimize the performance of the CT-DA-CLIP framework and achieve fine-grained control over the restored images, the framework is optimized by calculating the difference between the restoration results of the teacher model and the student model. Based on this difference, the reconstructed images of both the teacher and student models are updated, and the updated output of the teacher model is ultimately used as the final restoration result. This optimization process is achieved by introducing a multi-loss function collaborative optimization mechanism, and the specific steps include: First, a contrastive loss function is constructed. This invention employs a pre-trained VGG-16 network as a feature encoder to extract perceptual features from the student model's restored image, the teacher model's restored image, and the low-quality input image, and then performs contrastive learning based on these features. Specifically, the system uses VGG-16 to process the restored image (I...) output by the student model... student The restored image output by the teacher model (I) teacher ) and low-quality input images (i.e., images in the severe weather dataset mixdataset constructed in this invention, I degraded Each feature vector is extracted separately to obtain its original feature vector. Then, these original feature vectors are normalized using the L2 norm to eliminate scale differences and make them more suitable for calculating cosine similarity, thereby promoting effective comparison in the feature space. The normalization formula is as follows: The formula for normalizing the features of the student model-reconstructed image is: , in, This is the original feature vector extracted from the output image of the student model; The L2 norm of the corresponding feature. This is the normalized anchor point feature vector.

[0040] The normalization formula for the features of the teacher model-reconstructed image is: , in, This is the original feature vector extracted from the output image of the teacher model. The L2 norm of the corresponding feature. This is the normalized feature vector of the positive sample.

[0041] The formula for normalizing features of low-quality input images is: , in, This is the original feature vector extracted from a low-quality input image. The L2 norm of the corresponding feature. This is the normalized negative sample feature vector.

[0042] In the contrastive learning paradigm, the normalized features of the images reconstructed by the student model are... Explicitly defined as anchor points, the normalized features of the teacher model's reconstructed image. Normalized features of low-quality input images, used as positive samples. These are then treated as negative samples. Subsequently, the cosine similarity between the anchor point and the positive sample, as well as between the anchor point and the negative sample, is calculated to quantify their proximity in the feature space. The formula for calculating the similarity between the anchor point and the positive sample (teacher model features) is: The formula for calculating the similarity between the anchor point and the negative sample (low-quality image features) is: , where · represents the vector inner product operation.

[0043] In order to effectively adjust the scaling of similarity and thus optimize the gradient flow and convergence of contrastive learning, the similarity calculated above is divided by a temperature parameter T.

[0044] The formula for scaling the similarity between anchor points and positive samples is: , The formula for scaling the similarity between anchor points and negative samples is: , The final formula for calculating contrastive loss is: , in, The loss function is an exponential function. This loss function aims to make the features of the student model's reconstructed image as close as possible to the features of the teacher model's reconstructed image in the perceptual space, while significantly distancing them from the features of the low-quality input image. This, in turn, guides the student model to generate more semantically consistent and visually higher-quality reconstruction results at the feature space level.

[0045] Secondly, a consistency loss function is constructed. The L1 loss function (mean absolute error) is used to measure the pixel-level difference between the images restored by the student model and the teacher model. This loss function ensures the precision and fidelity of the restoration results by forcing the student model's output image to maintain a high degree of consistency with the teacher model's output image at the pixel level. The specific formula is as follows: , in, The restored image output for the student model. The restored image output by the teacher model, and It is the L1 norm, which is the sum of the absolute differences of all corresponding pixels in an image.

[0046] Next, we construct the overall loss function. To comprehensively optimize the performance of the student and teacher models, we weight and combine the contrastive loss function and the consistency loss function to form a comprehensive overall loss function. This ensures both consistency and quality of the image restoration results at the high-level semantic feature level, as well as high fidelity at the low-level pixel level. The formula for calculating the overall loss function is: , in, The losses are as described above. For the aforementioned loss of consistency; and This is the weighting coefficient of the contrast loss, used to adjust the contribution of the contrast loss to the total loss.

[0047] Furthermore, multi-loss collaborative optimization is performed. By applying this loss function, the output results of the image restoration student network and teacher network are optimized, aiming to ensure that the image restoration effect is more accurate and reasonable in key indicators such as perceptual features, semantic content, and degradation decoupling, thus enabling the student model to learn deeper image representations. In addition, by introducing a weighted consistency loss function, the high-fidelity consistency between the output results of the image restoration student network and teacher network at the pixel level is accurately measured, thereby ensuring the visual quality and detail preservation of the restored image. Finally, the above two loss functions are combined with preset weight coefficients (such as... =0.6) are combined and applied to the image restoration student network and the image restoration teacher network to synergistically improve image restoration performance and optimize the model's adaptive and generalization capabilities when facing different complex degradation scenarios, so that it can better handle severe weather conditions not fully covered in the training data.

[0048] Step 5: Adaptive Model Update. The student model employs an online learning strategy for parameter updates, enabling it to quickly adapt to new degradation patterns encountered during continuous testing. The update trigger mechanism is controlled by two conditions to balance responsiveness and periodicity: when the total loss... When the loss exceeds a preset threshold of 0.01, the system immediately triggers a parameter update process for the student model to address a significant performance degradation. Similarly, when the number of samples processed is a multiple of 3, the system also triggers a parameter update for the student model, ensuring periodic exploratory updates and enabling learning even if the loss does not reach the threshold. This combined triggering mechanism ensures a timely response when model performance significantly degrades, while also allowing for periodic exploratory updates to prevent the model from getting trapped in local optima.

[0049] During parameter updates, the system first clears all accumulated gradients to zero to ensure the independence and accuracy of each update. Then, it calculates the gradient of the total loss with respect to all trainable parameters of the student model using the backpropagation algorithm. To prevent gradient explosion during training, the system prunes the calculated gradients, limiting their L2 norm to within 1.0, thereby enhancing training stability. Finally, the system uses the Adam optimizer to update the student model's parameters. By adaptively adjusting the learning rate of each parameter, the Adam optimizer converges efficiently and stably, enabling the student model to quickly adapt to new domain data and achieve real-time online learning.

[0050] The teacher model employs an exponential moving average (EMA) strategy for parameter updates, aiming to maintain model stability and general robustness. The system automatically triggers the teacher model's update process whenever the student model completes a parameter update. The teacher model's update does not require gradient calculation; instead, it directly updates parameters using the aforementioned exponential moving average formula. Specifically, for each parameter of the teacher model, the system updates it to a weighted average of the current teacher and student parameters. The weight of the teacher parameter is set to 0.99, and the weight of the student parameter is set to 0.01. This update strategy ensures that the teacher model's parameters change slowly and stably, gradually absorbing the latest knowledge learned by the student model, while effectively smoothing out performance fluctuations and noise that may occur during the student model's training, thus maintaining the continuous stability and high quality of the teacher model's output.

[0051] Example 2 In this embodiment, to comprehensively evaluate the performance of the proposed method, a test set containing various severe weather images was constructed. These images were generated using the severe weather dataset construction method described in detail above. The fog intensity parameter ρ was randomly selected between 0.3 and 0.7, and the minimum length L of the raindrop lines... min Set to 15 pixels, maximum length L max Set to 45 pixels, rain layer transparency parameter Set to 0.4.

[0052] The DA-CLIP pre-trained model was trained in two stages on a private dataset containing 1 million image-text pairs. The first stage used the InfoNCE loss function, with a temperature parameter T set to 0.07 and a learning rate of 5e-5, for 50 epochs. The second stage used a conditional SDE diffusion model, optimized with L1 loss, with a learning rate of 1e-4, for 100 epochs. Both the student and teacher models were initialized from the pre-trained DA-CLIP model. During continuous test-time adaptation, the Adam optimizer had a learning rate of 1e-4, compared to the loss weights. The L2 norm threshold for gradient clipping is set to 0.6, and the EMA smoothing factor α for the teacher model is set to 0.99. Total loss. The update threshold is 0.01, and an update is forcibly triggered every 3 samples processed.

[0053] The results show that, Figure 2As shown, this method demonstrates significant image restoration capabilities under various adverse weather conditions. For test images originally covered by dense fog, moderate rain lines, and snowflakes, the visibility of the images is greatly improved after processing by this method. The fog is significantly dissipated, the rain line traces are effectively removed, and the occlusion effect of snowflakes on details is also alleviated. The restored images are visually closer to the original clean images, with clear details and textures, and accurate color reproduction.

[0054] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for image restoration of severe weather based on continuous testing and adaptation, characterized in that, Includes the following steps: S1. Construct a severe weather dataset and divide it into a training set and a test set; S2. Construct a severe weather image restoration framework, including a pre-trained model DA-CLIP, a student model based on DA-CLIP, and a teacher model based on DA-CLIP; input the publicly available dataset into the pre-trained model DA-CLIP for training, and initialize the student model and the teacher model using the model parameters of the trained pre-trained model DA-CLIP; S3. Input the training set of the severe weather dataset into the student model and the teacher model respectively for model training. Optimize the severe weather image restoration framework by calculating the difference between the restoration results of the teacher model and the restoration results of the student model, update the student model and the teacher model, and obtain the trained severe weather image restoration framework. S4. Input the test set images of the severe weather dataset into the trained severe weather image restoration framework to obtain the image restoration results.

2. The method for image restoration of severe weather based on continuous testing adaptation according to claim 1, characterized in that, Step S1 specifically includes: A fog effect simulation method based on linear weighted mixing is used to process original images from the publicly available snow100k dataset to generate a snow fog dataset. An image rain effect synthesis method based on random rain line modeling and Gaussian blur fusion is then used to process the original images from the snow100k dataset to generate a rain and snow dataset. The fog effect simulation method and the image rain effect synthesis method are then used to sequentially process the original images from the snow100k dataset to generate a rain, fog, and snow coexistence dataset. Finally, the snow fog dataset, rain and snow dataset, and rain, fog, and snow coexistence dataset are merged to obtain a severe weather dataset.

3. The method for image restoration of severe weather based on continuous testing adaptation according to claim 2, characterized in that, The DA-CLIP-based student model includes a DA-CLIP encoder, a fully connected linear layer, a ReLU activation function, an unflatten operation, several transposed convolutional modules (ConvTranspose2d), a batch normalization layer (BatchNorm2d), and a Sigmoid activation function; after each transposed convolutional module (ConvTranspose2d), a batch normalization layer (BatchNorm2d) and a ReLU activation function are sequentially connected. The input image is encoded by a DA-CLIP encoder to obtain student model image features. These student model image features undergo dimensionality transformation through a fully connected linear layer to obtain dimensionality-transformed features. The dimensionality-transformed features are then subjected to a ReLU activation function to introduce a non-linear transformation, and then an Unflatten operation is used to reshape the one-dimensional feature vector into a two-dimensional feature map with spatial structure, resulting in reshaped features. The reshaped features are then upsampled by the first transposed convolutional module ConvTranspose2d, processed sequentially by a batch normalization layer BatchNorm2d and a ReLU activation function, and then input into the second transposed convolutional module ConvTranspose2d for further upsampling. This process is repeated, and the feature map after five upsampling layers is processed by a batch normalization layer BatchNorm2d and a ReLU activation function before being input into a Sigmoid activation function to obtain the restored image of the student model.

4. The method for image restoration of severe weather based on continuous testing adaptation according to claim 3, characterized in that, The initialization parameters of the DA-CLIP-based teacher model are set by deeply replicating the parameters of the DA-CLIP-based student model. During training, the parameters are dynamically updated through an exponential moving average mechanism to gradually absorb the effective knowledge learned by the student model. The DA-CLIP-based teacher model also includes a DA-CLIP encoder, a fully connected linear layer, a ReLU activation function, an unflatten operation, several transposed convolutional modules (ConvTranspose2d), a batch normalization layer (BatchNorm2d), and a sigmoid activation function. Each transposed convolutional module (ConvTranspose2d) is followed by a batch normalization layer (BatchNorm2d) and a ReLU activation function.

5. The method for image restoration of severe weather based on continuous testing and adaptation as described in claim 4, characterized in that, Model training is performed through a multi-loss function collaborative optimization mechanism, including a contrastive loss function and a consistency loss function, and a total loss function is constructed based on the contrastive loss function and the consistency loss function.

6. The method for image restoration of severe weather based on continuous testing adaptation according to claim 5, characterized in that, The process of constructing the comparison loss function is as follows: The student model reconstructs images using a pre-trained VGG-16 network as a feature encoder. Teacher model restoration image and input image Feature extraction is performed to obtain the restored features. The three extracted features are then normalized using the L2 norm to obtain the normalized anchor feature vector. Positive sample feature vector and negative sample feature vectors ; Calculate the cosine similarity between the anchor feature vector and the positive sample feature vector to obtain the first cosine similarity. Calculate the cosine similarity and second cosine similarity between the anchor feature vector and the negative sample feature vector. ; first cosine similarity Divide by a temperature parameter T to obtain the first scaled similarity. ; Similarly, the second scaled similarity is obtained. Based on the first scaled similarity Second scaling similarity The contrastive loss function is constructed as follows: ， in, This represents the contrastive loss function.

7. The method for image restoration of severe weather based on continuous testing adaptation according to claim 6, characterized in that, The consistency loss function measures the image restored by the student model using the L1 loss function. and the restored image of the teacher model Differences at the pixel level.

8. The method for image restoration of severe weather based on continuous testing adaptation according to claim 7, characterized in that, Parameter updates for student and teacher models: The student model uses an online learning strategy for parameter updates. The update trigger mechanism is controlled by the following two conditions: when the total loss is greater than a preset threshold, or when the number of samples processed is a multiple of 3, the parameter update process of the student model is triggered. During the parameter update process, the system clears all accumulated gradients to zero, calculates the gradient of the total loss with respect to all trainable parameters of the student model through the backpropagation algorithm, clips the calculated gradients to limit the L2 norm of the gradients to within 1.0, and finally uses the Adam optimizer to update the parameters of the student model. The teacher model uses an exponential moving average strategy for parameter updates. Whenever the student model completes a parameter update, the system automatically triggers the teacher model's update process. The teacher model's update does not require gradient calculation; it updates parameters using the exponential moving average formula.

9. A method for reconstructing severe weather images based on continuous testing adaptation as described in claim 8, characterized in that, The steps of the fog effect simulation method include: S111. Perform range constraint processing on the preset fog intensity parameter ρ, normalize it to the closed interval [0,1], and obtain the normalized fog intensity parameter. ; S112. Create a full-channel fog base layer A with the same spatial dimensions as the input image, with height H and width W. The color intensity of each pixel in the full-channel fog base layer is precisely controlled by a configurable parameter. S113. Using a weighted mixing function, the original input image is... Linear blending is performed with the full-channel fog effect base layer A to obtain a composite image with a fog effect. .

10. A method for reconstructing severe weather images based on continuous testing adaptation as described in claim 9, characterized in that, The steps of the image rain effect synthesis method include: S121. Based on the spatial dimension of the input image, create a single-channel initial rain layer mask, wherein all pixel values ​​of the initial rain layer mask are set to zero. S122. On the initial rain layer mask, generate N raindrop lines with specific lengths and directions. The generation process of each raindrop line includes: randomly generating starting coordinates. and Randomly generate the pixel length of the raindrop lines. Based on the starting coordinates and length, and combined with the preset raindrop falling angle, calculate the endpoint coordinates of the raindrop line. and In the current rain layer mask, draw a straight line segment defined by the start and end points using white pixel values; S123. Apply Gaussian blur filter to the generated initial rain layer mask to obtain a single-channel Gaussian blurred rain layer mask. S124. The single-channel Gaussian blur rain layer mask is copied into a three-channel version to match the number of channels in the input image, thus obtaining a three-channel rain layer after channel conversion. S125. Linearly blend the input image with the three-channel rain layer after channel conversion to obtain the final composite image of the rain effect. .

Images