Integrative image inpainting model based on distillation method and system of dynamic data flow

Abstract

The application relates to the field of big data models and discloses a distillation method and system for an integrated image restoration model based on dynamic data flow, which can adaptively consider knowledge transfer and historical knowledge retention in a new task learning process. The method comprises the following steps: a training set containing playback samples is constructed by adopting a space perception representative playback strategy. New samples and playback samples are cross-used during training. When the playback samples are identified according to sample identification, the following steps are performed: the frozen teacher model and the dynamically updated student model are used to quantify feature drift, the channel weights of the playback samples are dynamically adjusted, and differentiated knowledge retention is realized; then, the teacher model and the student model are jointly optimized to generate response level distillation loss caused by prediction difference and feature level distillation loss caused by projection feature difference based on the channel weight weighting. The continuous learning of the integrated image restoration is realized through dynamic data flow distillation.

Description

Technical Field

[0001] This invention relates to the field of big data models, and in particular to a distillation method and system for an integrated image restoration model based on dynamic data flow. Background Technology

[0002] From a physical imaging model perspective, haze particles scatter light, leading to a significant decrease in image contrast, color cast, and severe detail blurring. Rainfall creates streaks of varying directions and densities, physically obscuring local textures. Snowfall manifests as complex snowflakes and streaks; their irregular shapes not only obscure background details but also introduce high-frequency noise. This degradation not only severely damages the visual realism of images but also poses a serious challenge to downstream advanced vision tasks. For example, in object detection, blurred edges make it difficult for region extraction networks to accurately locate objects; in semantic segmentation, weather-related interference is often misidentified as valid texture, leading to serious misjudgments and missed detections. Therefore, image inpainting under severe weather conditions has become a crucial prerequisite for improving the robustness of vision systems.

[0003] To improve the versatility of image restoration models, integrated severe weather restoration models have gradually become a research hotspot, aiming to handle multiple degradation types through a unified network. However, most existing integrated models rely on a static joint training paradigm, assuming that all types of degradation data have been collected before training. This is severely inconsistent with the streaming nature of data arriving dynamically over time in real-world scenarios. In a dynamic data streaming environment, if the model is simply fine-tuned using new task data, the network is highly susceptible to catastrophic forgetting, meaning that after learning a new weather restoration task, the restoration performance for older tasks will deteriorate sharply. Existing continuous learning attempts to introduce regularization or simple replay mechanisms still face the following significant limitations:

[0004] (1) Loss of feature details: Existing feature-level constraints mostly rely on global dimensionality reduction or static compression, ignoring the semantic differences between channels, resulting in the loss of key high-frequency detail information during the distillation process.

[0005] (2) No difference in sample focus: The traditional distillation paradigm usually applies a fixed supervisory weight to all replay samples, ignoring the heterogeneity of forgetting difficulty among different samples, making it difficult to carry out targeted consolidation of difficult samples that are easy to forget.

[0006] (3) Low data playback efficiency: Traditional random sampling strategies are difficult to effectively cover the feature distribution boundary of old tasks, resulting in serious semantic redundancy in the memory buffer and wasting limited storage resources.

[0007] Therefore, there is an urgent need for a continuous, integrated image restoration method that can effectively overcome the above-mentioned shortcomings and achieve efficient knowledge transfer and historical knowledge retention in dynamic data streams. Summary of the Invention

[0008] The purpose of this invention is to disclose an integrated image restoration model based on a distillation method and system of dynamic data flow, which adaptively balances knowledge transfer and historical knowledge retention during the learning process of new tasks.

[0009] To achieve the above objectives, the integrated image restoration model disclosed in this invention, based on a distillation method for dynamic data streams, includes: Step S1: Construct a dataset for the current learning task. The dataset consists of replay samples that have been filtered using a spatially aware representative replay strategy and have been marked with historical representative sample identifiers, and new samples for the current learning task. Step S2: Train the image restoration model by mixing replay samples with new samples until the model meets the convergence metric for the new learning task. The process of training the new task by identifying any replay sample based on historical representative sample identifiers includes at least the following sub-steps: Step S21: By quantifying the degree of feature drift of the replay samples between the teacher model and the student model, the weights of each channel of the replay samples are dynamically adjusted to achieve differentiated knowledge retention of different replay samples; the teacher model is the old model that was frozen after the previous task converged and does not participate in the training of new samples; the student model has the same network structure as the teacher model and is retrained with the parameter copy of the model that converged in the previous task as the initial value. Step S22: Backpropagation is performed based on the joint optimization objective function to iterate the learnable parameters of the integrated image inpainting model. The joint optimization objective function integrates response-level distillation loss and feature-level distillation loss. The feature-level distillation loss is calculated based on the weights of each channel and the difference between the projected features of the teacher model and the student model in the corresponding channel. The response-level distillation loss is calculated based on the difference between the images predicted by the teacher model and the student model respectively.

[0010] Preferably, the image restoration model is applied to restore images under at least two types of severe weather, and the models learn from each type of severe weather sequentially.

[0011] Preferably, the process of processing the feature drift degree of the quantized replay samples between the teacher model and the student model includes: Step S211: Calculate the statistical vectors of each channel of the feature map extracted by the teacher model and student model at the same node of the feature extraction network using two channel adaptive projection modules respectively. Then, generate adaptive channel weights by combining the vector matrix composed of the statistical vectors of each channel with the channel compression mapping matrix and the channel restoration mapping matrix; the specific calculation formula is as follows: ;in, It is a vector matrix composed of statistical vectors of each channel. This represents the generated adaptive channel weight matrix. This represents the learnable channel compression mapping matrix. This represents the learnable channel recovery mapping matrix. Represents the ReLU activation function. This represents the Sigmoid activation function; Step S212: Perform channel-level recalibration on the input features using the adaptive weights of each channel to obtain the projected features corresponding to the teacher model and the student model in the consistent channel subspace. and ; Step S213: Calculate the playback samples Characteristic drift degree : ;in, This represents the number of channels in the feature map. Indicates the height of the feature map. Indicates the width of the feature map. express Norm; Step S214: Map the feature drift level to the dynamic adaptive weights of the samples using the Sigmoid function. To characterize the risk of forgetting samples during continuous learning: ;in, The drift reference threshold represents the level of risk of forgetting. The weight sensitivity coefficient represents the sensitivity to changes in weights.

[0012] Preferably, after the image inpainting model achieves the convergence metric for the current learning task, a spatially aware representative replay strategy is used to select replay samples from the new samples of the current task, including: Step S11: First, randomly select a candidate set from the new samples of the current new task, and then select the samples in the candidate set. The currently trained student model is input into the same node of the feature extraction network to obtain deep feature maps. Then, adaptive region pooling is performed on each deep feature map, and the pooling results are flattened into feature vectors. ; Step S12: Analyze the feature vectors of each sample in the candidate set. Normalization is performed to obtain a sample representation of the spatial distribution of image features. ; Step S13: Randomly select a sample from the candidate set to form the initial selected set, and then add the sample that is farthest from the current selected set in the image feature space distribution to the selected set to achieve the gradual addition of samples to the selected set until the sample size of the selected set meets the termination number.

[0013] Preferably, the method of the present invention further includes: Step S14: Set up a sample feature repository to store the replay samples that are continuously accumulated in the order of tasks; and determine whether the total number of accumulated samples exceeds the preset storage limit; if so, perform the same filtering process as steps S11 to S13 on each sample in the sample feature repository based on the currently trained student model, and then prune the unfiltered samples.

[0014] Preferably, the joint optimization objective function for the replay samples Specifically: ;in, ; ; Loss function for new samples Specifically: ; in, express Norm; Indicates the playback image sample The predicted image, This indicates that the teacher model is working on replay samples. The predicted image, This indicates that the student model responds to replay samples. The predicted image; This represents the student model's prediction image for a new sample. Represents a true, clear image; This represents the weight coefficient of the contrastive regularization term during new task learning. This indicates a contrastive regularization term; This represents the weighting coefficient of the contrastive regularization term in response-level distillation; This represents the sample set of the current playback batch. Indicates the total number of samples in the batch; This represents the weighting coefficient of the response-level distillation term. This represents the weighting coefficient of the characteristic distillation term.

[0015] To achieve the above objectives, this invention discloses an integrated image restoration model based on a dynamic data stream distillation system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method.

[0016] This invention dynamically adjusts the weights of each channel of the replay samples by quantifying the feature drift of the replay samples between the teacher and student models, and has the following beneficial effects: 1. It can adaptively determine the importance of each channel based on the feature content of the input image. For example, it assigns higher weights to effective channels containing edge, texture, and structural information, and lower weights to channels that mainly contain interfering or redundant background information such as rain patterns, haze, and snowflakes. Therefore, it can preserve the high-frequency structure and key texture details of the image as much as possible during knowledge distillation.

[0017] 2. It breaks through the limitation of fixed sample weights in traditional distillation methods and achieves differentiated knowledge retention for different playback samples. Essentially, it accurately quantifies and assesses the forgetting risk of each sample. For example, by assigning greater distillation weights to difficult samples with high feature drift, fine-grained and differentiated knowledge consolidation is achieved, which greatly enhances its anti-forgetting stability while ensuring the model's plasticity.

[0018] 3. By training with a mix of replay samples and new samples, the feature distribution of the old and new tasks is dynamically balanced during the iterative update of learnable parameters. This allows for adaptive balance between knowledge transfer and retention of historical knowledge during the learning process of new tasks, effectively mitigating catastrophic forgetting and significantly improving the accuracy of continuous integrated image restoration.

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a structural diagram of the integrated image restoration model based on a distillation network of dynamic data flow disclosed in the embodiments of the present invention.

[0021] Figure 2 This is a functional structure diagram of the Channel Adaptive Projection (CAP) module disclosed in an embodiment of the present invention.

[0022] Figure 3 These are examples of qualitative restoration effects on defogging, deraining, and desnowing tasks disclosed in embodiments of the present invention. The top row shows weather degradation images randomly selected from test data, and the bottom row shows examples of the final restoration results obtained using the embodiments of the present invention. Detailed Implementation

[0023] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0024] Example 1 This embodiment describes an image inpainting model applied to inpaint images under at least two types of severe weather, with each severe weather condition learned sequentially. It discloses a distillation method for an integrated image inpainting model based on dynamic data flow, referring to... Figure 1 and Figure 2 This includes the following steps: Step S1: Construct the dataset for the current learning task. The dataset consists of replay samples that have been filtered using a spatially aware representative replay strategy and have been marked with historical representative sample identifiers, and new samples for the current learning task.

[0025] In this step, each type of severe weather corresponds to a degraded task. Severe weather types include, but are not limited to, fog, rain, and snow. The new samples used for supervised learning differ for each task. During the initial learning process of the first task, the number of replay samples (also known as "old samples") is zero. After each image restoration model is trained to meet the convergence criteria for the current learning task, a spatially aware representative replay strategy is used to select representative replay samples from the numerous new samples and add them to the sample feature repository, along with historical representative sample identifiers. This allows the image restoration model to accurately and effectively identify new and old samples during the training of the next task.

[0026] In this embodiment, the spatially-aware representative replay strategy for selecting replay samples from new samples of the current new task includes: Step S11: First, randomly select a candidate set from the new samples of the current new task, and then select the samples in the candidate set. The currently trained student model is input into the same node of the feature extraction network (the node is only used to represent the specific spatiotemporal location of the extracted target feature map, which will not be elaborated on later) to obtain deep feature maps. Then, adaptive region pooling is performed on each deep feature map, and the pooling result is flattened into feature vectors. .

[0027] Optionally, specific methods may be adopted. Adaptive region pooling. If the number of channels in the deep feature map is... The result obtained after pooling is The feature tensor; after flattening, we obtain a length of The feature vector contains both deep semantic information and preserves the approximate spatial layout of the image, and can be used for sample similarity calculation in subsequent farthest point sampling.

[0028] Step S12: Analyze the feature vectors of each sample in the candidate set. Normalization is performed to obtain a sample representation of the spatial distribution of image features. The calculation formula can be specifically as follows: ;in, express Norm, To prevent division by zero of constants.

[0029] Step S13: Randomly select a sample from the candidate set to form the initial selected set, and then add the sample that is farthest from the current selected set in the image feature space distribution to the selected set to achieve the gradual addition of samples to the selected set until the sample size of the selected set meets the termination number.

[0030] Step S14: Determine whether the total number of samples accumulated in the sample feature repository exceeds the preset storage limit; if so, perform the same filtering process as steps S11 to S13 on each sample in the sample feature repository based on the currently trained student model, and then prune the unfiltered samples.

[0031] In summary, if an episode has already been selected... This represents the set of samples that have been selected into the sample feature repository in the current task. Iterative selection of the candidate set is based on the distance from the already selected set. The sample with the furthest Euclidean distance The specific calculation formula for storing it in the sample feature repository is as follows: ;in, This represents a sample in the candidate set. This represents the samples in the selected set. and These represent the minimum and maximum values, respectively. It is an abbreviation for the independent variable (argument).

[0032] In this embodiment, the network structure of the image restoration model typically consists of a preprocessing module, multiple cascaded groups forming a backbone feature extraction network, and then connected to a postprocessing module to form a complete network baseline framework. Each group can consist of multiple basic blocks, and each basic block can sequentially include a convolutional layer, a ReLU activation function, a channel attention layer (CALayer), and a pixel attention layer (PALayer).

[0033] Step S2: Train the image restoration model by mixing replay samples with new samples until the model meets the convergence metric for the new learning task. The process of training the new task by identifying any replay sample based on historical representative sample identifiers includes at least the following sub-steps: Step S21: By quantifying the feature drift of the replay samples between the teacher model and the student model, the weights of each channel of the replay samples are dynamically adjusted to achieve differentiated knowledge retention of different replay samples. The teacher model is the old model that was frozen after the previous task converged and does not participate in the training of new samples. The student model has the same network structure as the teacher model and is retrained using the parameter copy of the model that converged in the previous task as the initial value.

[0034] Furthermore, the process of quantifying the degree of feature drift between the teacher model and the student model in replay samples includes: Step S211: Calculate the statistical vectors of each channel of the feature map extracted by the teacher model and student model at the same node in the feature extraction network (this node may or may not be the same as the node in step S11) using two Channel Adaptive Projection (CAP) modules respectively. Then, generate adaptive channel weights by combining the vector matrix composed of the statistical vectors of each channel with the channel compression mapping matrix and the channel restoration mapping matrix; the specific calculation formula is as follows: ;in, It is a vector matrix composed of statistical vectors of each channel. This represents the generated adaptive channel weight matrix. This represents the learnable channel compression mapping matrix. This represents the learnable channel recovery mapping matrix. Represents the ReLU activation function. This represents the Sigmoid activation function. Figure 2 In this context, GAP stands for Global Average Pooling, which is used to average the input feature map across each channel in the spatial dimension, compressing the feature map into corresponding channel statistical vectors for subsequent channel compression mapping matrices and channel restoration mapping matrices to generate adaptive channel weights.

[0035] Step S212: Perform channel-level recalibration on the input features using the adaptive weights of each channel to obtain the projected features corresponding to the teacher model and the student model in the consistent channel subspace. and .

[0036] like Figure 1 and Figure 2 As shown, let the input image be... The feature map of the student model is obtained by inputting it into the backbone feature extraction network of the student model in step S211. ,in, , These represent the number of channels, height, and width of the feature map, respectively.

[0037] First of all Global mean pooling is performed in the spatial dimension to obtain the channel statistical vector. The first corresponding to the channel distribution element The calculation formula is: ;in, In the feature map The Middle The spatial location of each channel The characteristic response value. The numerical values ​​reflect the strength of the student model's response to local edges, textures, or degradation patterns at that channel and location.

[0038] For example, if a certain feature extraction layer The input is The convolution kernel parameters are , bias is Then the layer is in the th order. The first channel, the first Convolutional response at each position It can be represented as: ;in, , This is the spatial location index within the convolution kernel. This is the input channel index.

[0039] If this layer is followed by a nonlinear activation function Then the output feature value at that position It can be represented as: When the first is selected The output of a layer or a feature extraction module is used as At that time, there were: Therefore, it can be concluded that These are not manually set values, but intermediate feature values ​​obtained by layer-by-layer calculation of the input image through the student model's backbone feature extraction network.

[0040] Subsequently, the channel weight matrix of the student model is generated through a bottleneck structure that includes channel compression and channel recovery mapping. : ;in, , ; and These are the learnable parameters in the CAP module, which are adaptively updated during model training using the backpropagation algorithm. Statistical vectors for each channel The vector matrix formed.

[0041] In this embodiment, channel compression and channel restoration only occur within the channel weight generation branch of the CAP module. Specifically, Used to obtain global mean pooling 3D channel statistical vector Temporarily compressed to Dimensions are used to extract more compact channel relationships; Used to restore it to The dimension is adjusted to generate channel weights consistent with the original feature map's channel number. This process does not change the backbone network structure of the teacher and student models, nor does it alter the channel number or spatial resolution of the feature maps. This represents the channel reduction rate within the CAP module, used to control the temporary compression degree of the channel statistics vector, rather than representing the difference in the number of channels between the teacher and student models. The channel reduction rate... The preferred value is a natural number between 8 and 32, and the most preferred value is 16.

[0042] For ease of understanding, the following will be based on the number of channels. Taking a deep feature map as an example, this illustrates the process of channel compression and channel restoration in the CAP module. The size of the input feature map can be represented as... First, global mean pooling is performed on each channel in the spatial dimension, which means averaging the feature response values ​​of all spatial locations in each channel to obtain 64 numbers. These 64 numbers constitute the channel statistical vector. Then, the CAP module internally uses a channel compression mapping matrix. The 64-dimensional channel statistical vector is temporarily compressed to 4 dimensions, where the channel reduction rate is... ,Right now This is used to extract the main channel relationships; then the mapping matrix is ​​recovered through the channels. The 4-dimensional channel representation is restored to 64 dimensions, and then 64 channel weights with values ​​between 0 and 1 are obtained by applying the sigmoid function. Finally, these 64 channel weights are applied to the 64 channels of the original feature map. After this processing, the feature map size remains the same. In other words, the CAP module does not change the backbone network structure, the number of feature map channels, or the spatial resolution, but rather adaptively adjusts the importance of different channels.

[0043] Next, the obtained channel weights are used to perform channel-level recalibration on the input features to map the features of the frozen old model (i.e., the teacher model and the student model) to a consistent channel subspace for feature-level distillation constraints, thus obtaining the projected features of the student model. , its first The formula for calculating the feature map of each channel is: ;in, Represents the projective features of the student model. This indicates that the projection feature is in the first... individual channels, spatial locations eigenvalues ​​at that location Represents the channel weight matrix The Each element.

[0044] The processing mechanism of the first CAP, which processes the feature maps, statistical vectors, adaptive channel weights, and channel-level recalibration extracted by the student model, is the same as the mechanism of the second CAP, which processes the feature maps, statistical vectors, adaptive channel weights, and channel-level recalibration extracted by the teacher model. Therefore, it will not be elaborated further.

[0045] It should be noted that this applies both in the frozen teacher model and in the student model where the relevant learning parameters are continuously iterated during training. and All are known quantities that change according to the changes in relevant learning parameters in the backbone extraction network, and the weights of each channel adaptively derived from the input image are consistent with the weights of each channel in the backbone extraction network; however, for the same replay sample, as long as the replay sample does not belong to the first sample of the new task learning, then in the first CAP... and The value of is the same as in the second CAP. and The values ​​of are usually inconsistent.

[0046] Step S213: Calculate the playback samples Characteristic drift degree : ;in, This represents the number of channels in the feature map. Indicates the height of the feature map. Indicates the width of the feature map. express Norm.

[0047] Step S214: Map the feature drift level to the dynamic adaptive weights of the samples using the Sigmoid function. To characterize the risk of forgetting samples during continuous learning: ;in, The drift reference threshold represents the level of risk of forgetting. The weight sensitivity coefficient represents the sensitivity to changes in weights.

[0048] In this step, the feature drift reference threshold Preferably, it is a constant between 0.5 and 0.9; the sensitivity to control weight changes The preferred constant is between 1.0 and 2.0.

[0049] Step S22: Backpropagation is performed based on the joint optimization objective function to iterate the learnable parameters of the integrated image inpainting model. The joint optimization objective function integrates response-level distillation loss and feature-level distillation loss. The feature-level distillation loss is calculated based on the weights of each channel and the difference between the projected features of the teacher model and the student model in the corresponding channel. The response-level distillation loss is calculated based on the difference between the images predicted by the teacher model and the student model respectively.

[0050] It should be noted that during the current task training process, new task samples and historical replay samples can be used interchangeably in the same mini-batch, but their corresponding losses have different functions. New task samples are mainly used to calculate the new task learning loss to learn new knowledge for the current task; historical replay samples are mainly used to calculate the response-level distillation loss and feature-level distillation loss to retain old task knowledge. During training, the losses corresponding to both types of samples are aggregated and backpropagated to update the student model parameters, while the teacher model remains frozen.

[0051] In this step, if the replay samples are used in the training of the image inpainting model for the new task using mini-batch gradient descent, the joint optimization objective function for the replay samples... Specifically: ;in, ; .

[0052] in, express Norm; Indicates the playback image sample The predicted image, This indicates that the teacher model is working on replay samples. The predicted image, This indicates that the student model responds to replay samples. The predicted image; This represents the weighting coefficient of the contrastive regularization term in response-level distillation; This represents the sample set of the current playback batch. Indicates the total number of samples in the batch; This represents the weighting coefficient of the response-level distillation term. This represents the weighting coefficient of the characteristic distillation term.

[0053] Ideally, the weighting coefficients of the response-level distillation term... The preferred constant is between 0.5 and 2.0, which is the weighting coefficient of the characteristic distillation term. The preferred constant is between 1.0 and 5.0.

[0054] In contrast, the loss function for new samples Specifically: ;in, This represents the student model's prediction image for a new sample. Represents a true, clear image; This represents the weight coefficient of the contrastive regularization term during new task learning. This indicates a contrastive regularization term.

[0055] Based on the steps described in this embodiment, referring to... Figure 3 The final image restoration model, through actual comparisons of images collected under different severe weather conditions before (top row) and after (bottom row), fully demonstrates that this embodiment has at least the following beneficial effects: (1) More accurate feature preservation: By introducing the Channel Adaptive Projection (CAP) module, this invention can adaptively determine the importance of each channel based on the feature content of the input image. CAP assigns higher weights to effective channels containing edge, texture, and structural information; and lower weights to channels mainly containing interference or redundant background information such as rain patterns, haze, and snowflakes. Thus, the deep features of the teacher model and the student model can be mapped to a more consistent semantic subspace for alignment at the channel level. Simultaneously, since CAP does not change the spatial resolution of the feature map, it can preserve as much high-frequency structure and key texture details as possible during knowledge distillation.

[0056] (2) More Fine-Grained Knowledge Transfer: Breaking through the limitation of fixed sample weights in traditional distillation methods, the Sample Adaptive Dynamic Weighting (SADW) mechanism proposed in this invention can accurately quantify and assess the forgetting risk of each sample. By assigning greater distillation weights to difficult samples with high feature drift, fine-grained and differentiated knowledge consolidation is achieved, which greatly enhances its anti-forgetting stability while ensuring the plasticity of the model.

[0057] (3) More efficient playback storage: The Spatially Aware Representative Replay (SARR) strategy designed in this invention accurately captures the data distribution boundaries of each historical task with extremely low storage budget through hybrid sampling and dynamic feature pruning. Compared with traditional random storage strategies, it effectively removes semantic redundancy and greatly improves the efficiency of experience playback and the reliability of deployment.

[0058] In addition, to facilitate a thorough understanding of the main points and technical details of this embodiment by those skilled in the art, the following supplementary explanations are provided: A. Throughout the continuous learning process, the modules do not operate simultaneously or in any particular order. Instead, they follow a cyclical pattern: "Training for the current task → Sample selection after the task ends → Updating the sample library → Freezing the model for the next task." Specifically, when the first task arrives, since there are no historical tasks, no frozen old models as teacher models, and no replay sample library, the teacher model branch, replay sample playback, CAP feature distillation, and Sample Adaptive Dynamic Weighting (SADW) in Figure 1 do not actually play the role of "preserving old knowledge." The model mainly uses the data from the first task for normal supervised training.

[0059] After the first task is completed, the currently trained student model has the ability to extract the features of the task. At this time, the sample selection mechanism is started: the samples from the first task are input into the trained student model, deep feature maps are extracted, and after 2×2 adaptive region pooling and flattening, the sample representation is obtained. Then, representative samples are selected by sampling the farthest point and stored in the sample feature storage repository. At the same time, the trained student model is frozen and used as the teacher model for the next task.

[0060] It wasn't until the second task began that it truly took shape. Figure 1 The teacher-student dual-model structure is shown: the frozen model from the previous stage serves as the teacher model, and the model currently being trained serves as the student model. The training data consists of new samples from the second task and historical replay samples from the previous stage's sample library. For new task samples, the student model primarily learns new knowledge through the new task learning loss. For historical replay samples, both the teacher and student models extract features and perform channel adaptive projection using the CAP module to align their features semantically. Then, SADW assigns distillation weights to different replay samples based on the drift between the projected features of the teacher and student models. Finally, the student model parameters are updated using a joint loss function. Specifically, new task samples and historical replay samples can be in the same training mini-batch, but corresponding losses are calculated separately based on the sample identifiers. New task samples calculate the new task learning loss, while historical replay samples calculate response-level distillation loss and feature-level distillation loss. The losses corresponding to both types of samples are then combined and used for backpropagation to update the student model parameters.

[0061] After the second task is trained, the trained student model is used to extract features from the second task samples, select representative samples for the second task, update the sample database, and freeze the student model as the teacher model for the third task. This process is repeated for subsequent tasks.

[0062] Therefore, the sample selection mechanism solves the problem of "which samples are saved for later use after the task is completed", while CAP and SADW solve the problem of "how to use the saved replay samples to constrain the student model and avoid forgetting during the training process starting from the second task". The joint loss function uses new task samples and historical replay samples to update the student model together during the training phase.

[0063] Furthermore, this embodiment mentions two models in the training description: a "teacher model" and a "student model." However, these are not two models that are ultimately deployed and used simultaneously. Rather, they represent two roles in the continuous learning and training process. Specifically, during the training of the first task, there is only one model learning. After the first task is completed, this model is copied and frozen, serving as the teacher model for the next training stage, providing knowledge constraints for the old task. Meanwhile, the model trained in the second task is called the student model. During training, the teacher model is only responsible for generating reference outputs or reference features for the old task and does not participate in parameter updates; the student model is the object being optimized and updated. After the second task is completed, the original teacher model is no longer used as the final model. Instead, the trained student model is frozen and serves as the teacher model for the third task. Subsequent tasks follow the same pattern. Therefore, the "teacher-student" relationship is merely a knowledge distillation relationship between two adjacent training stages: the model trained in the previous stage acts as the teacher, and the model being trained in the current stage acts as the student. After each stage, the model that is ultimately retained is always the currently trained student model. After all tasks are trained, the final student model is obtained and deployed, not the simultaneous existence of both the teacher and student models. In other words, the teacher model is only a temporary reference during the training process to prevent forgetting; the one that is truly updated and ultimately outputs is the same continuous learning model.

[0064] When the second task begins, the model trained on the previous task is copied twice. One copy is frozen as the teacher model, and the other is used as the initial parameters for the student model to continue training the new task. In other words, the student model does inherit the parameters learned from the old task at the beginning, and is not trained from scratch; the teacher model remains unchanged, serving as a reference standard for old knowledge.

[0065] In this embodiment, the network structure of the image restoration model adopts the following feature fusion attention network, which can be used after adopting the existing conventional single-task image restoration network.

[0066] Suppose the input is a three-channel image of severe weather degradation. ,in, and These represent the height and width of the input image, respectively. The network structure of the image inpainting model outputs a three-channel inpainted image with the same spatial dimensions as the input image.

[0067] The image restoration model's network structure preprocessing module includes a convolutional kernel with a size of [missing information]. A two-dimensional convolutional layer with a stride of 1 and padding of 1, having 3 input channels and 64 output channels, is used to map the input image to a format of size 1. Shallow features.

[0068] The image restoration model's network structure's backbone feature extraction module comprises three cascaded feature groups, each with 64 input and output channels, and includes 19 sequentially connected basic blocks and a tail convolutional layer. The kernel size of the tail convolutional layer in the image restoration model's network structure is... The step size is 1, the padding is 1, and the number of input channels and output channels are both 64. The output of the tail convolutional layer of the network structure group of the image restoration model is added element-wise to the input of the corresponding feature group to obtain the output feature of that feature group.

[0069] Each basic block consists of a first convolutional layer, a ReLU non-linear activation layer, a second convolutional layer, a channel attention layer, and a pixel attention layer. The kernel size of the first and second convolutional layers is [missing information]. The stride is 1, the padding is 1, and the number of input and output channels is 64. The input features of the basic block pass through the first convolutional layer and the ReLU nonlinear activation layer in sequence, and are then added to the input features of the basic block for the first time element-wise. The added features then pass through the second convolutional layer, the channel attention layer, and the pixel attention layer in sequence, and are added to the input features of the basic block for the second time element-wise, to obtain the output features of the basic block.

[0070] The network structure of the image inpainting model's channel attention layer first uses global average pooling to filter channels of size 1000. The input features are compressed to a size of The channel description features are then sequentially processed through a first set of input channels with 64 channels and output channels with 8 channels. Convolutional layer, ReLU nonlinear activation layer, second layer with 8 input channels and 64 output channels Convolutional layers and sigmoid activation layers generate channel attention weights, and the network structure of the image inpainting model multiplies the channel attention weights with the input features channel by channel.

[0071] The network structure of the image inpainting model consists of a pixel attention layer, which includes a first layer with 64 input channels and 8 output channels. Convolutional layer, ReLU nonlinear activation layer, second layer with 8 input channels and 1 output channel Convolutional layers and sigmoid activation layers are used to generate a size of The pixel attention weight map of the image restoration model's network structure is generated and multiplied pixel by pixel with the input features.

[0072] The features output by the three feature groups are denoted as follows: , and Their dimensions are all The network structure feature fusion module of the image restoration model will... , and By splicing along the channel dimension, we obtain a size of The splicing features are then processed; subsequently, global average pooling is performed on the splicing features of the image inpainting model's network structure, and the network structure is sequentially processed through a first layer with 192 input channels and 4 output channels. Convolutional layer, ReLU nonlinear activation layer, second layer with 4 input channels and 192 output channels Convolutional layers and sigmoid activation layers generate fusion weights corresponding to the three feature groups.

[0073] The network structure fusion weights of the image inpainting model are divided into three groups of 64-channel weights along the channel dimension, and then respectively... , and After multiplying each channel, we add each element to obtain a size of The fusion features of the image restoration model's network structure are then further weighted by a pixel attention layer to obtain the deep features output by the backbone feature extraction module. .

[0074] The image inpainting model's network structure includes a post-processing convolutional layer and a second post-processing convolutional layer connected in sequence. The kernel size of the first post-processing convolutional layer is... The stride is 1, padding is 1, and both the input and output channels are 64; the kernel size of the second post-processing convolutional layer is... The image has a stride of 1, padding of 1, 64 input channels, and 3 output channels. The image residual output from the second post-processing convolutional layer is added element-wise to the input severe weather degradation image to obtain the final restored image.

[0075] Preferably, each convolutional layer in the network structure of the image restoration model uses a stride of 1, and no downsampling or upsampling layers are set, so the spatial size of each feature map remains unchanged during network processing.

[0076] Example 2 This embodiment discloses an integrated image restoration model based on a dynamic data stream distillation system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a series of steps of the method disclosed in Embodiment 1 above.

[0077] Similarly, the technical effects achieved by this system are the same as those in Example 1, and will not be repeated here.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A unified image restoration model based on a distillation method of dynamic data flow, characterized in that, include: Step S1: Construct a dataset for the current learning task. The dataset consists of replay samples that have been filtered using a spatially aware representative replay strategy and have been marked with historical representative sample identifiers, and new samples for the current learning task. Step S2: Train the image restoration model by mixing replay samples with new samples until the model meets the convergence metric for the new learning task. The process of training the new task by identifying any replay sample based on historical representative sample identifiers includes at least the following sub-steps: Step S21: By quantifying the degree of feature drift of the replay samples between the teacher model and the student model, the weights of each channel of the replay samples are dynamically adjusted to achieve differentiated knowledge retention of different replay samples; the teacher model is the old model that was frozen after the previous task converged and does not participate in the training of new samples; the student model has the same network structure as the teacher model and is retrained with the parameter copy of the model that converged in the previous task as the initial value. Step S22: Backpropagation is performed based on the joint optimization objective function to iterate the learnable parameters of the integrated image inpainting model. The joint optimization objective function integrates response-level distillation loss and feature-level distillation loss. The feature-level distillation loss is calculated based on the weights of each channel and the difference between the projected features of the teacher model and the student model in the corresponding channel. The response-level distillation loss is calculated based on the difference between the images predicted by the teacher model and the student model respectively.

2. The integrated image restoration model based on the distillation method of dynamic data flow according to claim 1, characterized in that, The image restoration model is applied to restore images under at least two types of severe weather, and the models learn from each type of severe weather sequentially.

3. The integrated image restoration model based on the distillation method of dynamic data flow according to claim 2, characterized in that, The process of handling the feature drift between the teacher model and the student model in the quantization replay samples includes: Step S211: Calculate the statistical vectors of each channel of the feature map extracted by the teacher model and student model at the same node of the feature extraction network using two channel adaptive projection modules respectively. Then, generate adaptive channel weights by combining the vector matrix composed of the statistical vectors of each channel with the channel compression mapping matrix and the channel restoration mapping matrix; the specific calculation formula is as follows: ;in, It is a vector matrix composed of statistical vectors of each channel. This represents the generated adaptive channel weight matrix. This represents the learnable channel compression mapping matrix. This represents the channel recovery mapping matrix that can be learned. Represents the ReLU activation function. This represents the Sigmoid activation function; Step S212: Perform channel-level recalibration on the input features using the adaptive weights of each channel to obtain the projected features corresponding to the teacher model and the student model in the consistent channel subspace. and ; Step S213: Calculate the playback samples Characteristic drift degree : ;in, This represents the number of channels in the feature map. Indicates the height of the feature map. Indicates the width of the feature map. express Norm; Step S214: Map the feature drift level to the dynamic adaptive weights of the samples using the Sigmoid function. To characterize the risk of forgetting samples during continuous learning: ;in, The drift reference threshold represents the level of risk of forgetting. The weight sensitivity coefficient represents the sensitivity to changes in weights.

4. The integrated image restoration model based on the distillation method of dynamic data flow according to claim 3, characterized in that, After the image inpainting model achieves convergence for the current learning task, a spatially aware representative replay strategy is used to select replay samples from the new samples of the current task, including: Step S11: First, randomly select a candidate set from the new samples of the current new task, and then select the samples in the candidate set. The currently trained student model is input into the same node of the feature extraction network to obtain deep feature maps. Then, adaptive region pooling is performed on each deep feature map, and the pooling results are flattened into feature vectors. ; Step S12: Analyze the feature vectors of each sample in the candidate set. Normalization is performed to obtain a sample representation of the spatial distribution of image features. ; Step S13: Randomly select a sample from the candidate set to form the initial selected set, and then add the sample that is farthest from the current selected set in the image feature space distribution to the selected set to achieve the gradual addition of samples to the selected set until the sample size of the selected set meets the termination number.

5. The integrated image restoration model based on the distillation method of dynamic data flow according to claim 4, characterized in that, Also includes: Step S14: Set up a sample feature repository to store the replay samples that are continuously accumulated in the order of tasks; and determine whether the total number of accumulated samples exceeds the preset storage limit; if so, perform the same filtering process as steps S11 to S13 on each sample in the sample feature repository based on the currently trained student model, and then prune the unfiltered samples.

6. The distillation method based on dynamic data flow for the integrated image restoration model according to any one of claims 1 to 5, characterized in that, The joint optimization objective function for the replay samples Specifically: ;in, ; ; Loss function for new samples Specifically: ; in, express Norm; Indicates the playback image sample The predicted image, This indicates that the teacher model is working on replay samples. The predicted image, This indicates that the student model responds to replay samples. The predicted image; This represents the student model's prediction image for a new sample. Represents a true, clear image; This represents the weight coefficient of the contrastive regularization term during new task learning. This indicates a contrastive regularization term; This represents the weighting coefficient of the contrastive regularization term in response-level distillation; This represents the sample set of the current playback batch. Indicates the total number of samples in the batch; This represents the weighting coefficient of the response-level distillation term. This represents the weighting coefficient of the characteristic distillation term.

7. A distillation system based on dynamic data flow for an integrated image restoration model, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 6.

Images