Time sequence database data compression method, system and device and storage medium

Abstract

The invention relates to the technical field of wind power generation equipment data processing. The invention provides a time sequence database data compression method, system and device and a storage medium. The method comprises the steps of dynamically partitioning time sequence data based on a preset time window; hybrid coding compression is performed on each data block, and a composite strategy of difference coding and run length coding is adopted for numerical data; a metadata index of the compression blocks is established, the time range, the data feature statistics and the compression parameters of each data block are recorded, and the metadata index comprises extreme value distribution, variance features and data fluctuation frequency indexes; and dynamically adjusting a compression strategy according to a historical data feature analysis result, predicting data fluctuation modes of different equipment sensors through a machine learning model, and automatically selecting an optimal coding combination and compression granularity for subsequent data blocks. The problems that when wind power plant time sequence data are processed through an existing method, time sequence characteristics are difficult to adapt, the storage cost is high and the query efficiency is low are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing for wind power generation equipment, and more specifically, to a method, system, device, and storage medium for data compression in a time series database. Background Art

[0002] With the continuous growth of the global demand for clean energy, wind power generation, as an important form of renewable energy, has put forward higher requirements for the efficient monitoring and operation and maintenance management of equipment in its large-scale operation. During the operation of wind farm equipment, a large amount of data with strong time sequence is generated, which covers key information such as equipment status, environmental parameters, and power output, and needs to meet diverse requirements such as real-time monitoring, trend analysis, and fault prediction.

[0003] From the perspective of data characteristics, wind farm data has significant time series characteristics: on the one hand, the equipment needs to provide real-time feedback on the operation status through high-frequency data collection, requiring the database to have the ability to efficiently process high-speed data writing; on the other hand, to support long-term operation and maintenance decisions, historical data needs to be stored for a long time, and the data scale grows exponentially over time. Traditional data storage and management solutions face severe challenges.

[0004] Traditional relational databases have obvious limitations in processing time series data: their data models are difficult to adapt to the regular characteristics of time series, the indexing mechanism is prone to performance bottlenecks when facing high-frequency writing, and the storage cost for a large amount of historical data is high and the query efficiency is low, which cannot meet the requirements of low-latency data processing in real-time monitoring. Although some enterprises have tried to implement data storage through distributed file systems or customized data platforms, such solutions generally lack optimization for time series data and have functional deficiencies in data compression, time dimension query, real-time analysis, etc., and it is difficult to balance storage efficiency and data queryability and analyzability. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, system, device, and storage medium for data compression in a time series database, aiming to solve the problems that existing methods are difficult to adapt to time series characteristics, have high storage costs, and low query efficiency when processing wind farm time series data.

[0006] The present invention is achieved through the following technical solutions:

[0007] A method for data compression in a time series database includes the following steps:

[0008] Dynamically divide the time series data based on a preset time window, where each data block contains a monitoring data sequence with continuous timestamps, and the time window is adaptively adjusted according to the data collection frequency and the equipment status change threshold;

[0009] Perform hybrid coding compression on each data block, and adopt a composite strategy of differential coding and run-length coding for numerical data; among them, for the steady-state data segment, run-length coding based on the time baseline is adopted, and for the fluctuating data segment, differential coding based on dynamic precision truncation is adopted;

[0010] Establish a compressed block metadata index, record the time range, data feature statistics, and compression parameters of each data block, and the metadata index includes extreme value distribution, variance characteristics, and data fluctuation frequency indicators;

[0011] Dynamically adjust the compression strategy according to the analysis results of historical data characteristics, predict the data fluctuation patterns of different device sensors through a machine learning model, and automatically select the optimal coding combination and compression granularity for subsequent data blocks.

[0012] Optionally, the process of dynamically dividing the time-series data based on a preset time window, where each data block contains a monitoring data sequence with continuous timestamps, and the specific process of adaptively adjusting the time window according to the data acquisition frequency and device status change threshold is as follows:

[0013] Obtain the current data acquisition frequency of the device, and determine the initial time window length based on the acquisition frequency;

[0014] Real-time monitor the change value of the device status parameter, calculate the change rate of the status parameter at adjacent time points, and compare the change rate of the status parameter at adjacent time points with the preset device status change threshold;

[0015] If the change rate of the status parameter at the adjacent time points exceeds the device status change threshold, it is determined that the device is in a dynamic fluctuation state, and the time window length is reduced according to a preset ratio;

[0016] If the change rate of the status parameter at the adjacent time points does not exceed the device status change threshold, it is determined that the device is in a steady-state operation state, and the time window length is increased according to a preset ratio;

[0017] Set the minimum length and maximum length of the time window as constraints, and through real-time dynamic monitoring and comparison, make the time window length within the range of the minimum length and the maximum length, and adaptively adjust according to the current data acquisition frequency and status change of the device to form a data block containing a monitoring data sequence with continuous timestamps.

[0018] Optionally, the specific process of performing hybrid coding compression on each data block and adopting a composite strategy of differential coding and run-length coding for numerical data is as follows:

[0019] Detect the fluctuation characteristics of the numerical data sequence in the data block, calculate the variance of the data segment or the change amplitude of adjacent data points, and divide the data block into a steady-state data segment and a fluctuating data segment according to the preset fluctuation threshold;

[0020] For the steady-state data segment, using the monitoring value at the first time point as the time baseline, run-length encoding is adopted to record the data points with consecutive same or changed values within the baseline error range, and the encoding content includes the baseline value, the number of consecutive points, and the error range;

[0021] For the fluctuating data segment, differential encoding is used to calculate the difference between the data at adjacent time points and the previous valid data point, and the difference is truncated in combination with the dynamic precision index in the data block metadata, and the significant digits that meet the precision requirements are retained;

[0022] During the encoding process, the boundaries of the two encoding strategies are marked to form a hybrid encoding result including the encoding type identifier, the baseline value, the difference truncation precision, and the data point sequence. At the same time, the compression parameters generated during the encoding process are recorded for subsequent decoding.

[0023] Optionally, the specific process of establishing the compressed block metadata index and recording the time range, data characteristic statistics, and compression parameters of each data block is as follows:

[0024] Obtain the start timestamp and end timestamp of the time-series data within the data block to determine the time range of the data block;

[0025] Calculate the maximum value, minimum value, and variance of the numerical data within the data block, and count the number of times the change amplitude between adjacent data points exceeds the preset fluctuation threshold to form data characteristic statistics including extreme value distribution, variance characteristics, and data fluctuation frequency indicators;

[0026] Record the compression parameters during the data block compression process, and organize the time range, data characteristic statistics, and compression parameters in an associated manner to form a metadata index entry corresponding to each data block, which is used to support the subsequent rapid query, decoding, and dynamic adjustment of the compression strategy of the data.

[0027] Optionally, the specific process of dynamically adjusting the compression strategy according to the historical data characteristic analysis results and predicting the data fluctuation patterns of different device sensors through a machine learning model is as follows:

[0028] Collect the historical time-series data of the target device sensor, and extract the statistical feature vectors of the data within each time window. The statistical feature vectors include the mean value, variance, range, autocorrelation coefficient, and energy distribution parameters after frequency-domain wavelet transform;

[0029] Construct a supervised learning data set based on a time sliding window, and associate the statistical feature vectors with the corresponding data fluctuation pattern labels, and the labels are dynamically generated according to the compression efficiency index of the actual encoding strategy adopted within the historical data segment;

[0030] A gated recurrent unit neural network is used to train a fluctuation pattern prediction model shared by multiple sensors. The input of the model is the feature vector of the current time window, and the output is the fluctuation intensity level and the expected data change trend parameters of the next time window. In real-time data stream processing, the metadata index features of the current data block are input into the fluctuation pattern prediction model to obtain the probability distribution of the data fluctuation pattern in the future time window. According to the maximum probability fluctuation pattern, the optimal coding combination is matched from the preset strategy library. The coding combination includes the switching threshold between run-length coding and differential coding, the dynamic precision truncation bit width, and the time window scaling factor.

[0031] A compression strategy feedback optimization mechanism is established to regularly count the deviation degree between the actual compression rate and the prediction result. When the deviation degree exceeds the preset threshold, model incremental learning is triggered, and the weight parameters of the fluctuation pattern prediction model are updated to adapt to the drift characteristics of the device operating state.

[0032] Optionally, the specific process of using the gated recurrent unit neural network to train the fluctuation pattern prediction model shared by multiple sensors is as follows:

[0033] Let the statistical feature vector input at time t be X t ;

[0034] Define the hidden state update equation of the gated recurrent unit as shown in the following formula (1):

[0035]

[0036] Where, represents the update gate; r t represents the reset gate; represents the candidate hidden state; h t represents the current hidden state; h t-1 represents the previous hidden state; W z 、W r and W h represent the weight matrices of the update gate, reset gate, and candidate hidden state respectively; b z 、b r and b h represent the corresponding bias terms; σ represents the Sigmoid function; tanh represents the hyperbolic tangent function; ⊙ represents the Hadamard product;

[0037] The output layer mapping function is as shown in the following formula (2):

[0038]

[0039] Where, represents the predicted probability distribution of the data fluctuation pattern of the next time window by the model; W o represents the output layer weight matrix; b odenotes the output layer bias term; Softmax denotes the normalized exponential function;

[0040] Construct a multi-task loss function as shown in the following equation (3):

[0041]

[0042] where, denotes the total loss function; α denotes the weight coefficient of the classification task loss; β denotes the weight coefficient of the regression task loss; λ denotes the weight coefficient of the L2 regularization term; denotes the cross-entropy loss; denotes the true fluctuation pattern class label of the i-th sample; denotes the probability distribution of the fluctuation pattern predicted by the model; denotes the mean squared error loss; denotes the true trend parameter vector of the i-th sample; denotes the trend parameter vector predicted by the model; Θ denotes the set of all trainable parameters of the model; θ denotes a single parameter in Θ;

[0043] Define a policy matching function as shown in the following equation (4):

[0044]

[0045] where, s * denotes the optimal coding policy; S denotes a preset policy library containing several candidate coding policies; s denotes a single policy in the preset policy library; k denotes the total number of categories of the fluctuation pattern; P(c j ∣X t ) denotes the conditional probability that given the feature vector X t of the current time window, the data fluctuation pattern belongs to the category c j ; c j denotes the j-th data fluctuation pattern category; η(s, c j ) denotes the fitness index of the policy s for the fluctuation pattern category c j .

[0046] Optionally, the specific process of establishing a compression policy feedback optimization mechanism to regularly count the deviation degree between the actual compression rate and the prediction result and trigger model incremental learning when the deviation degree exceeds the preset threshold is as follows:

[0047] Establish a compression policy feedback optimization mechanism as shown in the following equation (5):

[0048]

[0049] where, δ t denotes the deviation degree at time t; T denotes the time window length; τ denotes the historical time index; represents the predicted compression ratio at the \(t\)-th time step; represents the actual compression ratio at the \(t\)-th time step;

[0050] When \(\delta\) t > \(\theta\), incremental learning is triggered to update the parameter \(\theta\) as shown in the following formula (6):

[0051]

[0052] where, \(\leftarrow\) represents the parameter update symbol; \(\gamma\) represents the learning rate; represents the gradient of the new loss function with respect to \(\Theta\); represents the loss function of the new sample.

[0053] Based on the same inventive concept, the present invention also provides a blade appearance state monitoring and recognition analysis system for implementing the above-mentioned time series database data compression method, including a data chunking module, a hybrid coding module, a metadata indexing module, and a strategy optimization module connected in sequence;

[0054] wherein, the data chunking module is used to dynamically chunk the time series data based on a preset time window, obtain the current data acquisition frequency of the device to determine the initial time window length, monitor the change rate of the device state parameters in real time, adaptively adjust the time window length according to the comparison result between the change rate and a preset threshold, and the time window is restricted by a minimum length and a maximum length to form data chunks containing continuous time stamp monitoring data sequences;

[0055] The hybrid coding module is used to perform hybrid coding compression on each data chunk, divide the data chunk into a steady-state data segment and a fluctuating data segment through fluctuation feature detection, perform run-length coding based on a time baseline on the steady-state data segment, perform differential coding based on dynamic precision truncation on the fluctuating data segment, mark the coding strategy boundary and record the compression parameters for subsequent decoding;

[0056] The metadata indexing module is used to establish a metadata index for the compressed chunks, record the time range, data feature statistics, and compression parameters of each data chunk, and the data feature statistics include extreme value distribution, variance feature, and data fluctuation frequency index to support fast query, decoding, and dynamic adjustment of the compression strategy of the data;

[0057] The strategy optimization module is used to dynamically adjust the compression strategy according to the analysis result of historical data features, train a fluctuation pattern prediction model shared by multiple sensors through a gated recurrent unit neural network, output the probability distribution of the data fluctuation pattern within the future time window based on the statistical feature vector, match the optimal coding combination from a preset strategy library, and trigger model incremental learning according to the deviation degree between the actual compression rate and the prediction result through a compression strategy feedback optimization mechanism to adapt to the drift characteristics of the device operation state.

[0058] Based on the same inventive concept, the present invention also provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the above-mentioned time-series database data compression method.

[0059] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned time-series database data compression method.

[0060] The technical solution of the present invention has at least the following advantages and beneficial effects:

[0061] Adaptive dynamic block division optimizes data processing efficiency: By dynamically adjusting the time window based on the data acquisition frequency and the device state change threshold, the data block division is more in line with the actual working conditions; when the device state is stable, a larger time window is adopted to reduce the number of blocks and lower the management overhead; when the state fluctuates, the window is narrowed to accurately capture data changes, ensuring the timeliness of real-time monitoring, effectively balancing the high-speed data writing performance and the long-term storage requirements, and avoiding the processing bottleneck of the traditional fixed block division scheme when the data changes frequently.

[0062] The hybrid coding strategy improves the compression ratio and data availability: Different coding methods are adopted for the steady-state section and the fluctuating section of numerical data; for steady-state data, run-length coding based on the time baseline is used to greatly reduce redundant storage by utilizing data continuity; for fluctuating data, differential coding with dynamic precision truncation is used to reduce the data volume while retaining key change features; on the premise of ensuring data accuracy, the compression efficiency is significantly improved, especially suitable for the massive time-series data accumulated in wind farms for a long time, which can effectively reduce the storage cost and at the same time provide high-quality data support for upper-layer applications such as fault prediction and trend analysis.

[0063] The metadata index enhances the data query and analysis capabilities: A metadata index including the time range, statistical features, and compression parameters is established, enabling the system to quickly locate the target data block without complete decompression, greatly improving the query efficiency in the time dimension; in real-time monitoring, invalid data blocks can be quickly filtered through the metadata to meet the low-latency processing requirements; in long-term operation and maintenance analysis, based on the statistical features, the data trend can be directly evaluated, avoiding full-scale data scanning, and significantly optimizing the performance of complex analysis scenarios.

[0064] Machine Learning-Driven Dynamic Policy Optimization: By analyzing historical data features and predicting fluctuation patterns, it automatically selects the optimal encoding combination and compression granularity for different device sensors, can adapt to changes in device operating states and environmental factors, and realizes the intelligent iteration of compression strategies; compared with traditional fixed compression schemes, it can continuously optimize the balance between storage efficiency and data analyzability, and is especially suitable for complex scenarios of multiple types of devices and multi-dimensional data in wind farms, providing efficient and flexible technical support for large-scale operation and maintenance.

[0065] Systematically Solving the Pain Points of Traditional Time-Series Data Management: Aiming at the defects of traditional relational databases and customized platforms in time-series data processing, through full-link optimization, a complete solution covering data writing, storage, query, and analysis is constructed from data chunking, encoding compression to metadata management and dynamic policy adjustment; it not only meets the low-latency requirements of real-time monitoring in wind farms, but also provides an efficient data foundation for long-term data-driven operation and maintenance decisions, promoting the intelligent operation upgrade in the field of renewable energy. Brief Description of the Drawings

[0066] Figure 1 It is a schematic flowchart of the data compression method for a time-series database according to an embodiment of the present invention;

[0067] Figure 2 It is a schematic structural diagram of the data compression system for a time-series database according to an embodiment of the present invention. Detailed Embodiments

[0068] The following are the detailed embodiments in combination with the drawings.

[0069] Referring to Figure 1 , a data compression method for a time-series database includes the following steps:

[0070] Step 1: Dynamically chunk the time-series data based on a preset time window. Each data chunk contains a monitoring data sequence with continuous timestamps, and the time window is adaptively adjusted according to the data acquisition frequency and the device state change threshold.

[0071] In some embodiments, the specific process of dynamically chunking the time-series data based on a preset time window, where each data chunk contains a monitoring data sequence with continuous timestamps and the time window is adaptively adjusted according to the data acquisition frequency and the device state change threshold is as follows:

[0072] Obtain the current data acquisition frequency of the device and determine the initial time window length based on the acquisition frequency;

[0073] Real-time monitor the change values of the device state parameters, calculate the change rate of the state parameters at adjacent time points, and compare the change rate of the state parameters at adjacent time points with the preset device state change threshold;

[0074] If the change rate of the status parameters at adjacent time points exceeds the device status change threshold, it is determined that the device is in a dynamic fluctuation state, and the time window length is reduced by a preset ratio;

[0075] If the change rate of the status parameters at adjacent time points does not exceed the device status change threshold, it is determined that the device is in a steady-state operation state, and the time window length is increased by a preset ratio;

[0076] Set the minimum and maximum lengths of the time window as constraints. Through real-time dynamic monitoring and comparison, the time window length is within the range of the minimum and maximum lengths, and it is adaptively adjusted according to the current data acquisition frequency and status change of the device, forming a data block containing a continuous time-stamp monitoring data sequence.

[0077] Step 2: Perform hybrid coding compression on each data block. For numerical data, a composite strategy of differential coding and run-length coding is adopted; among them, for the steady-state data segment, run-length coding based on the time baseline is used, and for the fluctuating data segment, differential coding based on dynamic precision truncation is used.

[0078] In some embodiments, the specific process of performing hybrid coding compression on each data block and adopting a composite strategy of differential coding and run-length coding for numerical data is as follows:

[0079] Perform fluctuation feature detection on the numerical data sequence in the data block, calculate the variance of the data segment or the change amplitude of adjacent data points, and divide the data block into a steady-state data segment and a fluctuating data segment according to a preset fluctuation threshold;

[0080] For the steady-state data segment, using the monitoring value at the first time point as the time baseline, run-length coding is used to record consecutive identical or data points whose changes are within the baseline error range. The coding content includes the baseline value, the number of consecutive points, and the error range;

[0081] For the fluctuating data segment, differential coding is used to calculate the difference between the data at adjacent time points and the previous valid data point, and the difference is truncated in combination with the dynamic precision index in the data block metadata, retaining the significant digits that meet the precision requirements;

[0082] During the coding process, the boundaries of the two coding strategies are marked to form a hybrid coding result including the coding type identifier, the baseline value, the difference truncation precision, and the data point sequence. At the same time, the compression parameters generated during the coding process are recorded for subsequent decoding.

[0083] Step 3: Establish a compressed block metadata index, record the time range, data feature statistics, and compression parameters of each data block. The metadata index includes extreme value distribution, variance characteristics, and data fluctuation frequency indicators.

[0084] In some embodiments, the specific process of establishing a compressed block metadata index and recording the time range, data feature statistics, and compression parameters of each data block is as follows:

[0085] Obtain the start timestamp and end timestamp of the time-series data within the data block to determine the time range of the data block;

[0086] Calculate the maximum value, minimum value, and variance of the numerical data within the data block, and count the number of times the change amplitude between adjacent data points exceeds a preset fluctuation threshold, forming data feature statistics including extreme value distribution, variance characteristics, and data fluctuation frequency indicators;

[0087] Record the compression parameters during the data block compression process, and associate and organize the time range, data feature statistics, and compression parameters to form a metadata index entry corresponding to each data block, which is used to support the subsequent rapid query, decoding, and dynamic adjustment of the compression strategy of the data.

[0088] Step 4: Dynamically adjust the compression strategy according to the historical data feature analysis results. Predict the data fluctuation patterns of different device sensors through a machine learning model, and automatically select the optimal coding combination and compression granularity for subsequent data blocks.

[0089] In some embodiments, the specific process of dynamically adjusting the compression strategy according to the historical data feature analysis results and predicting the data fluctuation patterns of different device sensors through a machine learning model is as follows:

[0090] Collect the historical time-series data of the target device sensor, extract the statistical feature vectors of the data within each time window. The statistical feature vectors include the mean, variance, range, autocorrelation coefficient, and energy distribution parameters after frequency-domain wavelet transform;

[0091] Construct a supervised learning dataset based on a time sliding window, associate the statistical feature vectors with the corresponding data fluctuation pattern labels, and the labels are dynamically generated according to the compression efficiency indicators of the actual coding strategy within the historical data segment;

[0092] Use a gated recurrent unit neural network to train a fluctuation pattern prediction model shared by multiple sensors. The input of the model is the feature vector of the current time window, and the output is the fluctuation intensity level and the expected data change trend parameters of the next time window; in real-time data stream processing, input the metadata index features of the current data block into the fluctuation pattern prediction model to obtain the probability distribution of the data fluctuation pattern in the future time window, and match the optimal coding combination from the preset strategy library according to the maximum probability fluctuation pattern. The coding combination includes the switching threshold between run-length coding and differential coding, the dynamic precision truncation bit width, and the time window scaling factor;

[0093] Establish a compression strategy feedback optimization mechanism, regularly count the deviation degree between the actual compression rate and the prediction result, and trigger model incremental learning when the deviation degree exceeds the preset threshold, and update the weight parameters of the fluctuation pattern prediction model to adapt to the drift characteristics of the device operation state.

[0094] In some embodiments, the specific process of training the fluctuation pattern prediction model shared by multiple sensors using a gated recurrent unit neural network is as follows:

[0095] Let the statistical feature vector input at time t be X t ;

[0096] Define the hidden state update equation of the gated recurrent unit, as shown in the following formula (1):

[0097]

[0098] Among them, z t represents the update gate; r t represents the reset gate; represents the candidate hidden state; h t represents the current hidden state; h t-1 represents the previous hidden state; W z 、W r and W h respectively represent the weight matrices of the update gate, reset gate and candidate hidden state; b z 、b r and b h represent the corresponding bias terms; σ represents the Sigmoid function; tanh represents the hyperbolic tangent function; ⊙ represents the Hadamard product;

[0099] The output layer mapping function is as shown in the following formula (2):

[0100]

[0101] Among them, represents the predicted probability distribution of the model for the data fluctuation pattern in the next time window; W o represents the output layer weight matrix; b o represents the output layer bias term; Softmax represents the normalized exponential function;

[0102] Construct a multi-task loss function, as shown in the following formula (3):

[0103]

[0104] Among them, represents the total loss function; α represents the weight coefficient of the classification task loss; β represents the weight coefficient of the regression task loss; λ represents the weight coefficient of the L2 regularization term; denotes the cross - entropy loss; y i (c) denotes the true fluctuation pattern class label of the i - th sample; denotes the probability distribution of the fluctuation pattern predicted by the model; denotes the mean squared error loss; denotes the true trend parameter vector of the i - th sample; denotes the trend parameter vector predicted by the model; Θ denotes the set of all trainable parameters of the model; θ denotes a single parameter in Θ;

[0105] Define the policy matching function as shown in Equation (4) below:

[0106]

[0107] where s * denotes the optimal coding policy; S denotes a preset policy library containing several candidate coding policies; s denotes a single policy in the preset policy library; k denotes the total number of categories of the fluctuation pattern; P(c j ∣X t ) denotes the conditional probability that given the feature vector X t of the current time window, the data fluctuation pattern belongs to category c j ; c j denotes the j - th data fluctuation pattern category; η(s, c j ) denotes the fitness index of policy s for the fluctuation pattern category c j .

[0108] In some embodiments, establish a compression policy feedback optimization mechanism, regularly count the deviation degree between the actual compression rate and the prediction result, and when the deviation degree exceeds the preset threshold, the specific process of triggering the incremental learning of the model is as follows:

[0109] Establish a compression policy feedback optimization mechanism as shown in Equation (5) below:

[0110]

[0111] where δ t denotes the deviation degree at time t; T denotes the time window length; τ denotes the historical time index; denotes the predicted compression rate at the τ - th time step; denotes the actual compression rate at the τ - th time step;

[0112] When δ t > θ, trigger incremental learning and update the parameter θ as shown in Equation (6) below:

[0113]

[0114] where ← denotes the parameter update symbol; γ denotes the learning rate; Denotes the gradient of the new loss function with respect to Θ; Denotes the loss function of the new sample.

[0115] Based on the same inventive concept, corresponding to any of the above embodiments, with reference to Figure 2 , the present invention provides a time series database data compression system for implementing the foregoing time series database data compression method, including a data chunking module, a hybrid coding module, a metadata indexing module, and a strategy optimization module connected in sequence;

[0116] Among them, the data chunking module is used to dynamically chunk time series data based on a preset time window, obtain the current data acquisition frequency of the device to determine the initial time window length, monitor the change rate of the device status parameters in real time, and adaptively adjust the time window length according to the comparison result between the change rate and the preset threshold. The time window is constrained by a minimum length and a maximum length to form data blocks containing continuous timestamp monitoring data sequences;

[0117] The hybrid coding module is used to perform hybrid coding compression on each data block. By detecting the fluctuation characteristics, the data block is divided into a steady-state data segment and a fluctuation data segment. The run-length coding based on the time baseline is used for the steady-state data segment, and the differential coding based on the dynamic precision truncation is used for the fluctuation data segment. The coding strategy boundary is marked and the compression parameters are recorded for subsequent decoding;

[0118] The metadata indexing module is used to establish a metadata index for the compressed blocks, record the time range, data feature statistics, and compression parameters of each data block. The data feature statistics include extreme value distribution, variance characteristics, and data fluctuation frequency indicators to support fast query, decoding, and dynamic adjustment of the compression strategy of the data;

[0119] The strategy optimization module is used to dynamically adjust the compression strategy according to the analysis result of the historical data characteristics. By training a fluctuation pattern prediction model shared by multiple sensors through a gated recurrent unit neural network, the probability distribution of the data fluctuation pattern within the future time window is output based on the statistical feature vector, the optimal coding combination is matched from the preset strategy library, and the model incremental learning is triggered through the compression strategy feedback optimization mechanism according to the deviation degree between the actual compression rate and the prediction result to adapt to the drift characteristics of the device operating state.

[0120] Based on the same inventive concept, corresponding to any of the above embodiments, the present invention provides an electronic device, including a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the time series database data compression method of the embodiment.

[0121] Optionally, the above electronic device may be a server.

[0122] In addition, this embodiment also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the data compression method of the time series database in the embodiment is implemented.

[0123] It can be understood that the processor in the embodiments of the present invention may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0124] The method steps in the embodiments of the present invention may be implemented in a hardware manner or by a processor executing software instructions. The software instructions may be composed of corresponding software modules. The software modules may be stored in a random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable hard disks, CD-ROMs, or any other form of storage medium well-known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may be located in an ASIC.

[0125] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a storage medium or transmitted through the storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A method for compressing data in a time-series database, characterized in that, It includes the following steps: Dynamically partition the time series data based on a preset time window. Each data block contains a monitoring data sequence with continuous timestamps, and the time window is adaptively adjusted according to the data acquisition frequency and the device status change threshold; Perform hybrid coding compression on each data block. For numerical data, a composite strategy of differential coding and run-length coding is adopted; among them, for the steady-state data segment, run-length coding based on the time baseline is used, and for the fluctuating data segment, differential coding based on dynamic precision truncation is used; Establish a compressed block metadata index, record the time range, data characteristic statistics, and compression parameters of each data block. The metadata index includes extreme value distribution, variance characteristics, and data fluctuation frequency indicators; Dynamically adjust the compression strategy according to the analysis results of historical data characteristics. Predict the data fluctuation patterns of different device sensors through a machine learning model, and automatically select the optimal coding combination and compression granularity for subsequent data blocks.

2. The method for compressing time-series database data according to claim 1, wherein, The specific process of dynamically partitioning the time series data based on a preset time window, where each data block contains a monitoring data sequence with continuous timestamps, and the time window is adaptively adjusted according to the data acquisition frequency and the device status change threshold is as follows: Obtain the current data acquisition frequency of the device, and determine the initial time window length based on the acquisition frequency; Monitor the change value of the device status parameters in real time, calculate the change rate of the status parameters at adjacent time points, and compare the change rate of the status parameters at adjacent time points with the preset device status change threshold; If the change rate of the status parameters at adjacent time points exceeds the device status change threshold, it is determined that the device is in a dynamic fluctuation state, and the time window length is reduced by a preset ratio; If the change rate of the status parameters at adjacent time points does not exceed the device status change threshold, it is determined that the device is in a steady-state operation state, and the time window length is increased by a preset ratio; Set the minimum length and maximum length of the time window as constraints. Through real-time dynamic monitoring and comparison, make the time window length within the range of the minimum length and the maximum length, and adaptively adjust according to the current data acquisition frequency and status change of the device, forming data blocks containing continuous timestamp monitoring data sequences.

3. The method for compressing time-series database data according to claim 1, wherein The specific process of performing hybrid coding compression on each data block and adopting a composite strategy of differential coding and run-length coding for numerical data is as follows: Detect the fluctuation characteristics of the numerical data sequence in the data block, calculate the variance of the data segment or the change amplitude of adjacent data points, and divide the data block into a steady-state data segment and a fluctuating data segment according to the preset fluctuation threshold; For the steady-state data segment, use the monitoring value at the first time point as the time baseline, and adopt run-length coding to record data points that are continuously the same or change within the baseline error range. The coding content includes the baseline value, the number of consecutive points, and the error range; For the fluctuating data segment, use differential coding to calculate the difference between adjacent time point data and the previous valid data point, and truncate the difference in combination with the dynamic precision index in the data block metadata, retaining the significant digits that meet the precision requirements; During the encoding process, mark the boundaries of the two encoding strategies to form a hybrid encoding result including the encoding type identifier, baseline value, difference truncation precision, and data point sequence. At the same time, record the compression parameters generated during the encoding process for subsequent decoding.

4. The method for compressing time series database data according to claim 1, wherein, The specific process of establishing the compressed block metadata index and recording the time range, data feature statistics, and compression parameters of each data block is as follows: Obtain the start timestamp and end timestamp of the time series data within the data block to determine the time range of the data block; Calculate the maximum value, minimum value, and variance of the numerical data within the data block, and count the number of times the change amplitude between adjacent data points exceeds the preset fluctuation threshold to form data feature statistics including extreme value distribution, variance characteristics, and data fluctuation frequency indicators; Record the compression parameters during the data block compression process, and organize the time range, data feature statistics, and compression parameters in an associated manner to form a metadata index entry for each data block, which is used to support the rapid query, decoding, and dynamic adjustment of compression strategies for subsequent data.

5. The method for compressing time-series database data according to claim 1, wherein, The specific process of dynamically adjusting the compression strategy according to the historical data feature analysis results and predicting the data fluctuation patterns of different device sensors through a machine learning model is as follows: Collect the historical time series data of the target device sensor, and extract the statistical feature vectors of the data within each time window. The statistical feature vectors include mean, variance, range, autocorrelation coefficient, and energy distribution parameters after frequency domain wavelet transform; Construct a supervised learning dataset based on a time sliding window, and associate the statistical feature vectors with the corresponding data fluctuation pattern labels. The labels are dynamically generated according to the compression efficiency indicators of the actual encoding strategies adopted within the historical data segments; Use a gated recurrent unit neural network to train a fluctuation pattern prediction model shared by multiple sensors. The input of the model is the feature vector of the current time window, and the output is the fluctuation intensity level and expected data change trend parameters of the next time window. In real-time data stream processing, input the metadata index features of the current data block into the fluctuation pattern prediction model to obtain the probability distribution of the data fluctuation patterns within the future time window, and match the optimal encoding combination from the preset strategy library according to the maximum probability fluctuation pattern. The encoding combination includes the switching threshold between run-length encoding and differential encoding, the dynamic precision truncation bit width, and the time window scaling factor; Establish a compression strategy feedback optimization mechanism, regularly count the deviation degree between the actual compression rate and the prediction result. When the deviation degree exceeds the preset threshold, trigger model incremental learning to update the weight parameters of the fluctuation pattern prediction model to adapt to the drift characteristics of the device operating state.

6. The method for compressing time series database data according to claim 5, wherein, The specific process of using a gated recurrent unit neural network to train a fluctuation pattern prediction model shared by multiple sensors is as follows: Let the statistical feature vector input at time t be X t ; Define the hidden state update equation of the gated recurrent unit as shown in the following formula (1): Among them, z t represents the update gate; r t represents the reset gate; represents the candidate hidden state; h t represents the current hidden state; h t-1 represents the previous hidden state; W z , W r and W h represent the weight matrices of the update gate, the reset gate, and the candidate hidden state respectively; b z , b r and b h represent the corresponding bias terms; σ represents the Sigmoid function; tanh represents the hyperbolic tangent function; ⊙ represents the Hadamard product; The output layer mapping function is as shown in the following formula (2): Among them, represents the predicted probability distribution of the model for the data fluctuation pattern in the next time window; W o represents the output layer weight matrix; b o represents the output layer bias term; Softmax represents the normalized exponential function; Construct a multi-task loss function as shown in the following formula (3): Among them, represents the total loss function; α represents the weight coefficient of the classification task loss; β represents the weight coefficient of the regression task loss; λ represents the weight coefficient of the L2 regularization term; represents the cross-entropy loss; represents the true fluctuation pattern category label of the i-th sample; represents the probability distribution of the fluctuation pattern predicted by the model; represents the mean squared error loss; represents the true trend parameter vector of the i-th sample; represents the trend parameter vector predicted by the model; Θ represents the set of all trainable parameters of the model; θ represents a single parameter in Θ; Define the strategy matching function as shown in the following formula (4): Among them, s * represents the optimal coding strategy; S represents a preset strategy library, which contains several candidate coding strategies; s represents a single strategy in the preset strategy library; k represents the total number of categories of fluctuation patterns; P(c j ∣X t ) represents the conditional probability that, given the feature vector X t of the current time window, the data fluctuation pattern belongs to category c j ; c j represents the j-th data fluctuation pattern category; η(s, c j ) represents the fitness index of strategy s for fluctuation pattern category c j .

7. The method for compressing time-series database data according to claim 5, wherein The specific process of establishing the compression strategy feedback optimization mechanism, regularly counting the deviation degree between the actual compression rate and the prediction result, and triggering model incremental learning when the deviation degree exceeds the preset threshold is as follows: A compression strategy feedback optimization mechanism is established as shown in the following formula (5): Among them, δ t represents the deviation degree at time t; T represents the time window length; τ represents the historical time index; represents the predicted compression ratio at the time step; represents the actual compression ratio at the time step; When δ t > θ, incremental learning is triggered to update the parameter θ as shown in the following equation (6): Among them, ← represents the parameter update symbol; γ represents the learning rate; represents the gradient of the new loss function with respect to Θ; represents the loss function of the new sample.

8. A time series database data compression system for implementing the time series database data compression method described in any one of claims 1-7, characterized in that, It includes a data chunking module, a hybrid coding module, a metadata indexing module, and a strategy optimization module that are connected in sequence; Among them, the data chunking module is used to dynamically chunk the time-series data based on a preset time window, obtain the current data acquisition frequency of the device to determine the initial time window length, monitor the change rate of the device status parameters in real time, adaptively adjust the time window length according to the comparison result between the change rate and the preset threshold, and the time window is restricted by the minimum length and the maximum length to form data chunks containing continuous timestamp monitoring data sequences; The hybrid coding module is used to perform hybrid coding compression on each data chunk, divide the data chunk into a steady-state data segment and a fluctuating data segment through fluctuation feature detection, use run-length coding based on the time baseline for the steady-state data segment, use differential coding based on dynamic precision truncation for the fluctuating data segment, mark the coding strategy boundary and record the compression parameters for subsequent decoding; The metadata indexing module is used to establish a metadata index for the compressed chunks, record the time range, data feature statistics, and compression parameters of each data chunk, and the data feature statistics include extreme value distribution, variance feature, and data fluctuation frequency index to support fast query, decoding, and dynamic adjustment of the compression strategy of the data; The strategy optimization module is used to dynamically adjust the compression strategy according to the analysis result of historical data features, train a fluctuation pattern prediction model shared by multiple sensors through a gated recurrent unit neural network, output the probability distribution of the data fluctuation pattern in the future time window based on the statistical feature vector, match the optimal coding combination from the preset strategy library, and trigger model incremental learning according to the deviation degree between the actual compression rate and the prediction result through the compression strategy feedback optimization mechanism to adapt to the drift characteristics of the device operating state.

9. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the time-series database data compression method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the time-series database data compression method described in any one of claims 1-7.

Images