A multi-dimensional adaptive distillation water quality prediction model for time series data

Abstract

The application discloses a multi-dimension adaptive distillation water quality prediction model for time series data, and relates to the technical field of machine learning model compression and optimization. The model migrates the knowledge of a high-performance teacher model to a lightweight student model based on a knowledge distillation technology, adapts to water quality time series data prediction of an edge computing device, and cooperatively realizes water quality data awareness and environmental change awareness of a distillation process through an adaptive time series importance weighting mechanism, a multi-scale feature decomposition alignment mechanism and a drift adaptive correction distillation mechanism. The teacher model is a large-scale high-precision water quality prediction time series model, and the student model is a lightweight model deployed on an edge water quality sensor. The application enables the distillation process to have "water quality data awareness" and "environmental change awareness", and dynamically, selectively and structurally transfer the knowledge of the teacher model.

Description

Technical Field

[0001] This invention relates to the field of machine learning model compression and optimization technology, and more specifically to a multi-dimensional adaptive distilled water quality prediction model for time-series data. Background Technology

[0002] Water quality monitoring is a crucial link in national environmental protection and public health security. With the development of IoT and AI technologies, large-scale time-series models based on deep learning (such as Transformer and LSTM) have demonstrated superior performance in water quality prediction and pollution early warning. However, these models have a huge number of parameters and are computationally complex, making them unsuitable for direct deployment on edge water quality sensors where computing, storage, and energy resources are severely limited. Knowledge distillation, as an effective model compression method, aims to transfer knowledge from a high-performance "teacher model" to a lightweight "student model." However, traditional distillation techniques have significant shortcomings when applied to water quality time-series data, mainly in the following three aspects: First, they ignore the inherent characteristics of time-series data, treating every data point in the time series equally. Water quality data exhibits strong periodicity (such as daily water consumption peaks), abrupt changes (such as sudden pollution events), and trends (such as long-term eutrophication). A uniform distillation strategy cannot guide the student model to focus on learning knowledge from moments with key monitoring value (such as pollution occurrences and regular peaks), leading to a decline in the student model's performance on key events and wasting limited learning capacity on stable, information-free periods. Secondly, there is the issue of coarse alignment of information across multiple time scales. When processing water quality data, the teacher model extracts and fuses features from multiple time scales, including long-term trends (low frequency), medium-term cycles (medium frequency), and short-term anomalies (high frequency). Traditional distillation typically only aligns the model's final output or features of a single layer. This coarse-grained alignment fails to allow the student model to accurately inherit the teacher's analytical and reasoning abilities at different scales, especially losing the ability to finely model "slow-changing" trends and "rapid-changing" anomalies in water quality, thus affecting the model's comprehensiveness. Thirdly, there is the vulnerability to environmental concept drift. The distribution of water quality data changes significantly with seasons, climate, and surrounding human activities (i.e., concept drift). For example, the baseline level of dissolved oxygen in water bodies is drastically different in winter than in summer. The knowledge of the student model generated by the traditional static distillation process is fixed on the data distribution at the time of training. When the deployment environment changes, the model's performance degrades severely, lacking the ability to adapt to new environments online, leading to false alarms or false negatives. Frequent retraining is necessary, making it unsuitable for long-term unattended sensor nodes.

[0003] Therefore, there is an urgent need for a distillation technique specifically designed for the characteristics of water quality time series data and capable of overcoming the above three major defects, in order to train a lightweight, accurate, robust, and adaptive edge-side water quality prediction model. Summary of the Invention

[0004] In view of this, the present invention provides a multi-dimensional adaptive distilled water quality prediction model for time-series data, which enables the distillation process to have "water quality data awareness" and "environmental change awareness", and dynamically, selectively and structurally transfers the knowledge of the teacher model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A multi-dimensional adaptive distillation water quality prediction model for time-series data is proposed. The model is based on knowledge distillation technology, which transfers knowledge from a high-performance teacher model to a lightweight student model. It is adapted to water quality time-series data prediction on edge computing devices. Through an adaptive time-series importance weighting mechanism, a multi-scale feature decomposition alignment mechanism, and a drift adaptive correction distillation mechanism, the model achieves water quality data awareness and environmental change awareness during the distillation process. The teacher model is a large-scale, high-precision water quality prediction time-series model, and the student model is a lightweight model deployed on edge water quality sensors.

[0006] Optionally, the adaptive time-series importance weighting mechanism introduces a time sensitivity function: the time series data is... The teacher model output is Student model output ; in, Represents the adaptive temporal knowledge distillation loss. Represents the total length of the time series. The index representing the current time step. The weights representing the importance of time steps The representative teacher model in the input Output at time The student model represents the input The output at that time.

[0007] Optional, time step importance weights The expression is as follows: ; in, Represents the trend consistency weight. Represents the weight of periodic similarity. This represents the mutation detection weight, used to measure whether a mutation has occurred at a given time point.

[0008] Optional multi-scale eigenvalue decomposition alignment mechanism, as follows: The network features of the teacher model are decomposed into three frequency bands: Teacher Model layer Decomposed by wavelet packet transform: ; in, The representative teacher model in the Features learned by layers It refers to the number of layers in a neural network. Representative Teacher Model The layer in the first The coefficients on each frequency component This represents the basis function corresponding to the i-th frequency; Multi-scale distillation loss is used to examine the learning progress at each layer, checking the understanding of long-term trends, periodic patterns, and short-term anomalies for each frequency. The expression is as follows: ; in, It represents the combined magnitude of all coefficients of the i-th frequency component in the l-th layer, used to characterize the overall strength of this frequency component in the features of this layer; The adaptive weighting coefficients are calculated using the following formula: Represents the distance function. Representing frequency projection No. The layer in the first The coefficients at each frequency.

[0009] Optional, adaptive weighting coefficients The expression is as follows: ; in, Representing the Layer The importance weight of each frequency Representative coefficient The Frobenius norm, Representative coefficient The entropy.

[0010] Optionally, the drift adaptive correction distillation mechanism defines the drift detection function as follows: ; in, This represents the degree of drift detected at time point t. This represents the probability distribution of various outcomes that the model expects given the current input x. This represents the KL divergence.

[0011] Optionally, a drift correction loss is defined to balance the learning representation of the teacher model and historical data, expressed as follows: ; in, This represents the learning coefficient at time t. The loss from distillation of traditional knowledge is represented by the following expression: ; in, Representing the student model, Representing the teacher model; The loss represents the historical sample replay loss, expressed as follows: ; in, Represents the number of historical samples. Representing the A historical sample, Represents the true labels corresponding to historical samples. This represents the drift adaptive correction loss.

[0012] Optional, the learning coefficient at time t The expression is as follows:

[0013] in, Represents the basic learning number, This represents the drift sensitivity coefficient.

[0014] As can be seen from the above technical solution, compared with the prior art, the present invention provides a multi-dimensional adaptive distilled water quality prediction model for time-series data, which has the following beneficial effects: 1. By defining an adaptive temporal importance weighting mechanism, this addresses the problem of traditional distillation methods treating all time points equally, leading to insufficient learning of key water quality events by the student model. It guides the student model to invest more "attention" and "learning energy" in pollution outbreaks, regular peaks, and stages of continuous deterioration, thereby achieving better key event monitoring performance with a smaller model size, improving learning efficiency and model practicality. Specifically, it defines abrupt change detection weight, periodic similarity weight, and trend consistency weight.

[0015] 2. By defining a multi-scale feature decomposition alignment mechanism, the student model can emulate the teacher model's hierarchical and structured learning of water quality patterns across different time scales. Specifically, using techniques such as wavelet packet transform, the features of the teacher model's intermediate layers are decomposed into three components corresponding to low frequency (trend), mid frequency (period), and high frequency (anomaly). During distillation, instead of directly aligning the original features, the student model's projected features are required to align with each frequency component of the teacher model. An adaptive weight is assigned to each component, taking into account both the component's energy intensity (importance) and information entropy (complexity).

[0016] 3. By defining a drift-adaptive correction distillation mechanism, the problem of static distillation models being unable to adapt to seasonal / environmental drift in water quality data is addressed, giving the student model the ability to continuously adapt to environmental changes after deployment. Specifically, a learning coefficient is defined and dynamically adjusted based on the degree of drift. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Water quality data exhibits strong periodicity; for example, 6-8 AM is the peak water consumption period, while 2 AM is a period of stable data. Based on these characteristics, an adaptive time-series importance weighting system is introduced to focus on moments of abrupt change, periodic events, and trend shifts, while reducing the weight given to stable periods.

[0021] This invention discloses a multi-dimensional adaptive distilled water quality prediction model for time-series data, such as... Figure 1 As shown, the model uses knowledge distillation technology to transfer knowledge from a high-performance teacher model to a lightweight student model, adapting to water quality time-series data prediction on edge computing devices. It achieves water quality data awareness and environmental change awareness during the distillation process through an adaptive time-series importance weighting mechanism, a multi-scale feature decomposition alignment mechanism, and a drift adaptive correction distillation mechanism. The teacher model is a large-scale, high-precision water quality prediction time-series model, and the student model is a lightweight model deployed on edge water quality sensors.

[0022] Traditional distillation treats all points in the time series equally; this embodiment introduces a time-sensitivity function: Let the time series data be The teacher model output is Student model output ; in, Represents the adaptive temporal knowledge distillation loss. Represents the total length of the time series (total number of time steps). Index representing the current time step (from 1 to ...) ), The weights representing the importance of time steps The representative teacher model (large model) is in the input Output at time The student model (small model) represents the input. The output at that time.

[0023] The weight function It consists of three parts: Mutation detection weights: ; in, This represents the mutation detection weight, measuring whether a mutation has occurred at a given time point. This represents a hyperparameter that controls the sensitivity to mutations; it is typically set to 0.5-1.5. Representing a point in time Raw sensor data (such as pH value, dissolved oxygen, etc.).

[0024] Represented by time point Centered on, front and back The average value of data at each time point.

[0025] This represents the current value minus the surrounding average value; the larger the difference, the more significant the mutation.

[0026] Periodic similarity weight: ; in, Represents the weight of periodic similarity. This represents the number of periods considered, for example... =3 indicates looking at the past 3 cycles. Represents a periodic index. =1 indicates the previous cycle. =2 represents the previous cycle, and so on. Represents the current time point Data, Representing the past The data at the position corresponding to the current time point within each cycle. For example: if =24, =25 (1 AM the next day), then: =1: (1 a.m. yesterday) =2: (1 a.m. the day before yesterday).

[0027] The similarity function calculates the degree of similarity between two data points; cosine similarity is commonly used. value range 1 indicates that they are exactly the same, and -1 indicates that they are completely opposite.

[0028] Trend consistency weight: ; in, Represents the trend consistency weight. Represents the radius of the trend judgment window ( =2~5), Represents the offset, from arrive This indicates looking at both the front and back. One point, The teacher model at a specific point in time The predicted value, This represents a sign function, focusing only on the trend direction. If the difference is greater than 0, it indicates an upward trend. If the difference is less than 0, it indicates a downward trend. If the difference equals 0, it indicates a stable trend.

[0029] This represents calculating the average value; the total number of values ​​within the window is... One point, Final weights: ; During training, large time series models typically learn from three aspects: long-term trends, medium-term cycles, and short-term anomalies. Therefore, the distillation process also needs to follow this learning principle, dividing the model into low-frequency, medium-frequency, and high-frequency layers for hierarchical learning.

[0030] The teacher network characteristics are decomposed into three frequency bands: Let the teacher model be... layer Decomposed by wavelet packet transform: ; in, The representative teacher model in the Features learned by layers It refers to the number of layers in a neural network. Teacher Model No. The layer in the first The coefficients on each frequency component This represents the basis function corresponding to the i-th frequency. Corresponding to low-frequency trend components, Corresponding to the mid-frequency periodic component, Corresponding to high-frequency anomalous components.

[0031] Multi-scale distillation loss is used to examine the learning progress at each layer, and to check the understanding of long-term trends, periodic patterns, and short-term anomalies for each frequency separately: ; in, The adaptive weighting coefficients are calculated using the following formula. Distance function Represents frequency projection. Representing the The layer in the first Coefficients at each frequency: ; in, Representing the Layer The importance weight of each frequency.

[0032] Representative coefficient The Frobenius norm represents the intensity or energy of this frequency component; the larger the value, the more pronounced the frequency component.

[0033] Representative coefficient The greater the entropy, the more complex and information-rich the frequency component.

[0034] The first part can be understood as the intensity ratio; if the low-frequency trend is strong (water quality continues to deteriorate), this part is important. The second part can also be understood as the complexity ratio; if the periodic pattern is very complex (the daily change pattern is different), the entropy is high. Whoever has more information needs to learn more.

[0035] Considering that water quality data can drift, different seasons and climates can lead to the same values, which may be normal in summer but abnormal in winter.

[0036] To detect different data patterns across two time periods, KL divergence is used to detect drift.

[0037] Define the drift detection function: ; in, This represents the degree of drift detected at time point t. This represents the probability distribution of various outcomes that the model expects given the current input x. This represents the KL divergence.

[0038] Adaptive distillation intensity, this value measures the extent to which the teacher model's behavior is learned at each time step: ; in, This represents the learning coefficient at time t. This represents the basic learning number, ranging from 0.8 to 1.0. This represents the drift sensitivity coefficient, ranging from 0.1 to 0.5.

[0039] Define a drift correction loss to balance the learning from the teacher model and historical data: ; in, ; Represents the loss of traditional knowledge through distillation. , Represents the historical sample playback loss, Represents the number of historical samples. Representing the A historical sample, Represents the true labels corresponding to historical samples. This represents the drift adaptive correction loss.

[0040] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-dimensional adaptive distilled water quality prediction model for time-series data, characterized in that, The model is based on knowledge distillation technology, which transfers knowledge from a high-performance teacher model to a lightweight student model. It is adapted to water quality time series data prediction on edge computing devices. Through an adaptive time series importance weighting mechanism, a multi-scale feature decomposition alignment mechanism, and a drift adaptive correction distillation mechanism, it achieves water quality data awareness and environmental change awareness during the distillation process. The teacher model is a large-scale, high-precision water quality prediction time series model, and the student model is a lightweight model deployed on edge water quality sensors. The adaptive time-series importance weighting mechanism introduces a time sensitivity function: the time series data is... The teacher model output is Student model output ； in, Represents the adaptive temporal knowledge distillation loss. Represents the total length of the time series. The index representing the current time step. The weights representing the importance of time steps The representative teacher model in the input Output at time The student model represents the input Output at time; Importance weight of time step The expression is as follows: ； in, Represents the trend consistency weight. Represents the weight of periodic similarity. This represents the mutation detection weight, used to measure whether a mutation has occurred at a given time point; The multi-scale eigenvalue decomposition alignment mechanism is as follows: The network features of the teacher model are decomposed into three frequency bands: Teacher Model layer Decomposed by wavelet packet transform: ； in, The representative teacher model in the Features learned by layers It refers to the number of layers in a neural network. Representative Teacher Model The layer in the first The coefficients on each frequency component This represents the basis function corresponding to the i-th frequency; Multi-scale distillation loss is used to examine the learning progress at each layer, checking the understanding of long-term trends, periodic patterns, and short-term anomalies for each frequency. The expression is as follows: ； in, The adaptive weighting coefficients are calculated using the following formula: Represents the distance function. Representing frequency projection No. The layer in the first Coefficients at each frequency; Adaptive weight coefficients The expression is as follows: ； in, It represents the combined magnitude of all coefficients of the i-th frequency component in the l-th layer, used to characterize the overall strength of this frequency component in the features of this layer; Representing the Layer The importance weight of each frequency Representative coefficient The Frobenius norm, Representative coefficient Entropy; The drift adaptive correction distillation mechanism defines the drift detection function as follows: ； in, This represents the degree of drift detected at time point t. This represents the probability distribution of various outcomes that the model expects given the current input x. Represents the KL divergence; The drift correction loss is defined to balance the learning representation of the teacher model and historical data, and is expressed as follows: ； in, This represents the learning coefficient at time t. The loss from distillation of traditional knowledge is represented by the following expression: ； in, Representing the student model, Representing the teacher model; The loss represents the historical sample replay loss, expressed as follows: ； in, Represents the number of historical samples. Representing the A historical sample, Represents the true labels corresponding to historical samples. This represents the drift adaptive correction loss; The learning coefficient at time t The expression is as follows: in, Represents the basic learning number, This represents the drift sensitivity coefficient.

Images