An industrial data full life cycle management method and system

Abstract

The application relates to the technical field of industrial big data storage and management, and discloses an industrial data full-life-cycle management method and system, which comprises the following steps: acquiring original time sequence data of industrial equipment, and performing Shannon entropy calculation to obtain a time sequence importance score sequence; performing threshold comparison according to the time sequence importance score sequence to determine a key transient window and a stable normal window; performing lossless difference coding on the key transient window to obtain a high-fidelity data segment; performing feature extraction and sparsification processing on the stable normal window, and performing time domain splicing and index construction to obtain a primary mixed storage structure; continuously monitoring a query access path to generate a heat distribution map, and obtaining a hierarchical archiving data set. The method can realize value perception layered storage and dynamic full-life-cycle management of massive industrial data.

Description

Technical Field

[0001] This invention relates to the field of industrial big data storage and management technology, and in particular to a method and system for managing the entire lifecycle of industrial data. Background Technology

[0002] Currently, with the deepening of Industry 4.0 and smart manufacturing strategies, various high-frequency sensors and monitoring terminals are widely deployed in production sites, generating massive amounts of continuous time-series data. How to build an efficient industrial data processing system to efficiently store and extract high-value value from this massive amount of data throughout its entire lifecycle has become a key aspect of the digital transformation of manufacturing enterprises.

[0003] In existing technologies, a uniform sampling strategy or a fixed compression algorithm is typically used to process all acquired streaming data. For example, the system often does not distinguish whether the device is currently in a stable "normal period" or a "transient period" with abnormal fluctuations, and samples and writes data to disk at a preset fixed frequency (e.g., 10Hz); or, a general lossless compression algorithm (e.g., Lempel-Ziv algorithm) or a fixed lossy compression algorithm (e.g., the rotating door compression algorithm SDT) is used to apply the same compression parameters and storage strategy to data in all time periods. How to eliminate redundant information in normal operation to the greatest extent and optimize the hierarchical storage layout while ensuring the integrity of critical transient event data is a technical challenge that urgently needs to be solved.

[0004] Existing technologies lack the ability to dynamically perceive the value density of data. Summary of the Invention

[0005] This invention provides a method and system for managing the entire lifecycle of industrial data, in order to solve the problem of the lack of dynamic perception capability of data value density in the prior art.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides a method for industrial data lifecycle management, comprising:

[0007] Obtain the raw time series data of industrial equipment, perform time window segmentation of preset length, calculate the Shannon entropy within each time window, and obtain the time series importance score sequence;

[0008] The time-series importance score sequence is compared with a preset diversion threshold. Time windows with scores higher than the diversion threshold are marked as critical transient windows, and time windows with scores lower than or equal to the diversion threshold are marked as stationary normal windows.

[0009] A difference sequence is calculated from the original data sequence within the critical transient window, and the difference sequence is matched and encapsulated based on a preset global mapping dictionary to generate a high-fidelity data segment.

[0010] The data within the stationary normal window is processed to obtain a sparse transform coefficient matrix, and the sparse transform coefficient matrix is ​​interval mapped and encoded based on a non-uniform quantizer to generate a simplified background data segment.

[0011] Based on the preset frame structure, the high-fidelity data segment and the background simplified data segment are spliced ​​in the time domain, and a metadata index containing timestamps and importance tags is established to construct a primary hybrid storage structure and write it into the preset first-level storage medium.

[0012] During the residency period of the primary hybrid storage structure, query operations initiated through the metadata index are continuously recorded, access frequency is statistically analyzed, and a heat distribution map is generated.

[0013] If the heat map shows that the access frequency of the background simplified data segment is lower than the preset cold archiving threshold, then the background simplified data segment is stripped from the primary hybrid storage structure and migrated to the preset second-level storage medium to obtain a hierarchical archived dataset.

[0014] Secondly, this invention provides an industrial data lifecycle management system, comprising:

[0015] The streaming slicing and evaluation module is used to acquire the raw time series data of industrial equipment, perform time window segmentation of preset length, calculate the Shannon entropy within each time window, and obtain the time series importance score sequence.

[0016] The diversion decision module is used to compare the time-series importance score sequence with a preset diversion threshold, mark time windows with scores higher than the diversion threshold as critical transient windows, and mark time windows with scores lower than or equal to the diversion threshold as stationary normal windows.

[0017] The high-fidelity processing module is used to calculate the difference sequence for the original data sequence within the key transient window, and to match and encapsulate the difference sequence based on a preset global mapping dictionary to generate a high-fidelity data segment.

[0018] The background simplification processing module is used to process the data within the stationary normal window to obtain a sparse transform coefficient matrix, and to perform interval mapping and encoding on the sparse transform coefficient matrix based on a non-uniform quantizer to generate a background simplification data segment.

[0019] The hybrid storage construction module is used to perform time-domain splicing of the high-fidelity data segment and the background simplified data segment based on a preset frame structure, and to establish a metadata index containing timestamps and importance tags, thereby constructing a primary hybrid storage structure and writing it into a preset first-level storage medium.

[0020] The lifecycle monitoring module is used to continuously record query operations initiated through the metadata index during the residency of the primary hybrid storage structure, count the access frequency, and generate a heat distribution map.

[0021] The dynamic archive migration module is used to separate the background simplified data segment from the primary hybrid storage structure and migrate it to a preset second-level storage medium if the heat distribution map shows that the access frequency of the background simplified data segment is lower than a preset cold archive threshold, thereby obtaining a hierarchical archive dataset.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) This invention achieves dynamic perception and precise segmentation of industrial data value density by performing time-series importance scoring and diversion decisions based on Shannon entropy on the original time-series data. Compared with the "blind" processing method of fixed-frequency sampling in the prior art, this invention uses the Shannon entropy formula to quantify waveform complexity, which can adaptively identify key transient windows containing fault symptoms and normal windows of stable operation. This mechanism solves the core contradiction of "high-value data being submerged and low-value data occupying resources" from the source, ensuring that under limited bandwidth and computing power conditions, the system can always focus on the data segments with the most diagnostic value, significantly improving the sensitivity and anomaly detection rate of equipment status monitoring.

[0024] (2) This invention constructs a hybrid storage structure that integrates high fidelity and high compression ratio by implementing differentiated compression coding and time-domain splicing strategies for data windows of different importance levels. For critical transient windows, lossless differential coding is used to completely preserve the original waveform features, ensuring the absolute accuracy of fault tracing analysis; for stable normal windows, transform domain sparsity and non-uniform quantization are used to significantly reduce the amount of redundant data, retaining only low-frequency trends as background references. This "dual-modal" processing method increases the average compression ratio of the overall data by several times while ensuring zero loss of key event information, effectively solving the technical problem of balancing "fidelity" and "storage space" in massive industrial data storage.

[0025] (3) This invention achieves global optimization of storage resource allocation by establishing a dynamic archiving mechanism based on access heat distribution maps throughout the entire lifecycle. This invention does not stop at the initial write operation, but continuously monitors the query access path of mixed data frames and quantifies and generates retrieval efficiency indicators. Once the background simplified data segment enters a "cold data" state, the system automatically strips it from the first-level solid-state storage medium and migrates it to the second-level mechanical storage medium, retaining only the critical high-fidelity data and indexes in the high-speed layer. This refined cold and hot data separation strategy enables enterprises to significantly extend the online retention period of high-value data without increasing hardware investment. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a method for managing the entire lifecycle of industrial data provided in the first embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of an industrial data lifecycle management system provided in the second embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Reference Figure 1 The first embodiment of the present invention provides a method for industrial data lifecycle management, including the following steps:

[0030] S11: Obtain the raw time series data of industrial equipment, perform time window segmentation of preset length, calculate the Shannon entropy within each time window, and obtain the time series importance score sequence;

[0031] S12, compare the time-series importance score sequence with a preset diversion threshold, mark the time window with a score higher than the diversion threshold as a critical transient window, and mark the time window with a score lower than or equal to the diversion threshold as a stable normal window;

[0032] S13, calculate the difference sequence for the original data sequence within the key transient window, and match and encapsulate the difference sequence based on a preset global mapping dictionary to generate a high-fidelity data segment;

[0033] S14, the data within the stationary normal window is processed to obtain a sparse transform coefficient matrix, and the sparse transform coefficient matrix is ​​interval mapped and encoded based on a non-uniform quantizer to generate a simplified background data segment.

[0034] S15, based on the preset frame structure, the high-fidelity data segment and the background simplified data segment are spliced ​​in the time domain, and a metadata index containing timestamps and importance tags is established to construct a primary hybrid storage structure and write it into the preset first-level storage medium.

[0035] S16, During the residency period of the primary hybrid storage structure, continuously record the query operations initiated through the metadata index, count the access frequency and generate a heat distribution map;

[0036] S17. If the heat distribution map shows that the access frequency of the background simplified data segment is lower than the preset cold archiving threshold, then the background simplified data segment is stripped from the primary hybrid storage structure and migrated to the preset second-level storage medium to obtain a hierarchical archived dataset.

[0037] In step S11, the raw time series data of the industrial equipment is acquired, and time windows of a preset length are segmented. The Shannon entropy within each time window is calculated to obtain the time series importance score sequence, including:

[0038] The raw time series data of industrial equipment is acquired, and the raw time series data is numerically discretized to map continuous analog signal values ​​into discrete symbols in a finite set of states.

[0039] Statistically analyze the probability distribution of each discrete symbol appearing within each time window, and calculate the information entropy value of the probability distribution based on the Shannon entropy formula;

[0040] The calculated information entropy value is used as an element of the temporal importance score sequence.

[0041] It should be noted that the acquisition of raw time-series data from industrial equipment is achieved through high-frequency data acquisition cards (DAQ cards) deployed on key components of the equipment (such as motor bearing housings, gearbox housings, and hydraulic pump outlets). These cards are equipped with high-precision piezoelectric or Hall effect sensors to capture continuous analog voltage signals (such as vibration acceleration, instantaneous current, and acoustic emission signals) in the physical field in real time at a preset sampling rate (e.g., 10kHz). The system converts these analog signals into digitized floating-point numerical sequences using a built-in analog-to-digital converter (ADC) and buffers them in a ring buffer, awaiting subsequent window segmentation processing.

[0042] It is worth noting that the preset sampling rate is determined based on the Nyquist-Shannon sampling theorem combined with spectral analysis of historical fault signals. The system performs Fast Fourier Transform (FFT) on the vibration signals from historical fault cases to identify the highest effective frequency band of the fault characteristic frequencies. (e.g., 3kHz), and select The minimum sampling frequency is then rounded up to a standard level (e.g., 10kHz) to prevent aliasing and ensure waveform fidelity.

[0043] It should be noted that the numerical discretization of the original time series data is achieved using the Symbolic Aggregation Approximation (SAX) method. This process aims to eliminate the interference of small high-frequency noise (white noise) on the overall trend judgment and map the infinite continuous floating-point numerical space to a finite symbolic space, thereby reducing computational complexity. Specifically, the system first determines the dynamic range of data changes within the current time window. And according to the preset discretization granularity Divide the range into Each data point is a non-overlapping range of values ​​(state slots). Then, the system iterates through each raw data point within the window. Determine the interval index into which the value falls, and replace it with the discrete symbol corresponding to that interval. (For example This transforms the original time series into a symbolic sequence.

[0044] It should be noted that calculating the probability distribution of each discrete symbol within each time window is the process by which the system constructs an empirical probability distribution vector after completing the discretization mapping. Specifically, the system initializes a vector of length... The system uses a counting array, where each element corresponds to the frequency of occurrence of a discrete symbol. Then, it iterates through all discrete symbols within the current time window, counting the frequency of each symbol. After the traversal is complete, the frequencies will be... Divide by the total window length This yields the normalized probability distribution vector. ,in And satisfy This vector objectively reflects the distribution pattern of data states within the window.

[0045] It should be noted that calculating the information entropy value of the probability distribution based on the Shannon entropy formula, and using the calculated information entropy value as an element of the temporal importance scoring sequence, is a process of quantifying waveform complexity using information theory principles. The calculation formula is as follows:

[0046]

[0047] in, This is the calculated information entropy value. This directly characterizes the uncertainty or disorder of the signal within the current window. When the waveform is stable and uniform (e.g., under no-load operation), data points are concentrated in a few symbol intervals, resulting in low entropy. When the waveform fluctuates violently and has diverse patterns (e.g., due to fault impacts or sudden load changes), data points are evenly distributed across various intervals, resulting in high entropy. The system follows the natural timing sequence of the time window ( ), calculate the entropy value of each window in sequence. These scalar values ​​are then stored sequentially into a one-dimensional array, ultimately forming a time-series importance score sequence. .

[0048] It is worth noting that the determination of the preset time window length and the size of the finite state set is based on autocorrelation analysis and signal-to-noise ratio statistics of historical operating data of industrial equipment. The preset length (Window Size) is determined by calculating the average duration of various typical transient impact events (such as fracture signals and surge signals) in the historical data, and selecting 1.2 times the duration distribution as the window length (e.g., 1024 sampling points) to ensure that a single window can completely encompass an independent transient event. The size of the finite state set (i.e., the discretization granularity mentioned above)... The determination of the bin width is based on Scott's Rule, which optimizes the histogram distribution of historical data. The system iterates through the historical dataset to calculate the ideal bin width that minimizes the mean square error of the histogram integration. This leads to the determination of the optimal set size. It is usually set to a small integer (e.g., 8) to maximize the noise reduction effect while preserving key trend features.

[0049] For example, suppose a bearing vibration sensor acquires a raw time series data containing 100 sampling points (window length...). According to statistics, the numerical range of this data segment is as follows: The system sets the finite state set to a size of 4, that is, it is divided into 4 equal-width intervals. Mapped to the symbol "A", Mapped to the symbol "B", Mapped to the symbol "C", The symbol is mapped to "D". After discretization, "A" appears 70 times, "B" appears 10 times, "C" appears 10 times, and "D" appears 10 times within the window. The probabilities of each symbol are then given. Substituting into the Shannon entropy formula, we calculate:

[0050]

[0051]

[0052]

[0053] This value This is the Shannon entropy for that time window, and it is stored in the temporal importance score sequence as the first... element This is used for subsequent flow determination. In contrast, a stationary signal consisting entirely of "A"s has an entropy of 0.

[0054] In step S12, the temporal importance score sequence is compared with a preset triage threshold. Time windows with scores higher than the triage threshold are marked as critical transient windows, and time windows with scores lower than or equal to the triage threshold are marked as stationary normal windows. This includes:

[0055] The time-series importance score sequences are sorted and score distribution curves are constructed;

[0056] Identify the step change points or density peak points in the scoring distribution curve, and extract the corresponding scoring values ​​as the diversion threshold;

[0057] Traverse all time windows, mark windows with score values ​​greater than the diversion threshold as critical transient windows, and mark windows with score values ​​less than or equal to the diversion threshold as stable normal windows.

[0058] It should be noted that sorting the temporal importance score sequence and constructing the score distribution curve is achieved by using an efficient sorting algorithm (such as quicksort or mergesort) to sort the unordered temporal importance score sequence output in step S11 in descending order. Let the original score sequence be... ,in This represents the total number of time windows. The system first generates an auxiliary array containing the original indices, then sorts this array according to the score values ​​to obtain an ordered score sequence. ,satisfy Subsequently, the system uses the sorted rank (Rank, () is used as the x-axis, with the corresponding rating values. Using the vertical axis as the ordinate, a two-dimensional rating distribution curve data structure is constructed in memory. This process aims to eliminate interference from the time dimension and transform the fluctuating characteristics of the data into long-tailed distribution features in the numerical dimension, thereby intuitively presenting the proportional relationship between high-value information and low-value background.

[0059] It should be noted that identifying the step change points or density peaks in the rating distribution curve and extracting the corresponding rating values ​​as the triage threshold is achieved using the Kneedle Algorithm. This step aims to automatically pinpoint the statistically natural boundary between the complex signal region with high entropy (high importance) and the stable background region with low entropy (low importance). Specifically, the system first... and the finish line Construct a straight line vector between the curve and the data points; then, iterate through each data point on the curve. Calculate the perpendicular distance from the point to the line vector. Select the option that makes the vertical distance... Point where the maximum value is reached This serves as the elbow point (i.e., the point of step change). This elbow point represents the location where the rate of decrease in the score value undergoes the largest abrupt change, and its corresponding ordinate score value is the shunting threshold. .

[0060] It should be noted that traversing all time windows, marking windows with scores greater than the diversion threshold as critical transient windows, and marking windows with scores less than or equal to the diversion threshold as stationary normal windows, thus completing the classification and labeling of time windows, is a process of establishing a status bitmap that corresponds one-to-one with the original time windows. The system then rescans the unsorted sequence according to the original chronological order. The rating of each window With the calculated threshold Perform a comparison one by one. If Then the first one will be in the state bitmap A position of "1" indicates that the window contains incompressible transient features; if Then the first one will be in the state bitmap A position of "0" indicates that the window has a highly compressible, stable background.

[0061] It is worth noting that the reason why the method for determining the diversion threshold adopts adaptive identification based on curve shape rather than a preset fixed value is based on statistical analysis of the non-stationary characteristics of industrial data. Since the entropy baseline differs significantly under different operating conditions (such as heavy-load operation and idle standby), a fixed threshold is highly likely to lead to missed or incorrect judgments. By identifying the geometric features (maximum curvature point) of the distribution curve, it can be mathematically guaranteed that the separated key transient windows do indeed belong to the sparse outliers in the long-tailed distribution, while the stationary normal windows belong to the dominant background terms. This ensures that the subsequent compression strategy achieves Pareto optimality between information retention rate and compression ratio.

[0062] For example, suppose step S11 outputs a scoring sequence for 5 time windows. The first step is sorting and curve construction. The system sorts the sequence in descending order to obtain an ordered sequence. The corresponding coordinates are The second step is to identify the threshold. (System connection starting point) and the finish line Calculate the distance from each point to the line, and find the second point. up to point 3 A sharp drop in value occurred, with the curvature being greatest at point 2 or the midpoint between point 3. Using the Kneedle algorithm, the optimal segmentation threshold was determined to be either a weighted average of these two values ​​or the smaller, higher value. The resulting splitting threshold was calculated here. The third step is classification and labeling. Traverse the original sequence. Window 1 (0.15) Marked as a stationary normal window; Window 2 (2.85) > 1.5 Marked as a critical transient window; Window 3 (0.20) Marked as a stationary normal window; Window 4 (2.90) > 1.5 Marked as a critical transient window; Window 5 (0.10) 1.5 The windows were marked as stable and normal. Ultimately, the system automatically identified windows 2 and 4 as key data, and windows 1, 3, and 5 as background data.

[0063] In step S13, a difference sequence is calculated for the original data sequence within the critical transient window, and the difference sequence is matched and encapsulated based on a preset global mapping dictionary to generate a high-fidelity data segment, including:

[0064] Extract the original data sequence within the key transient window, calculate the difference values ​​between adjacent data points, and obtain the difference sequence;

[0065] The differential sequence is matched based on a preset global mapping dictionary, and the high-frequency differential patterns are mapped to short codewords.

[0066] The differential sequence is entropy-encoded and encapsulated using the global mapping dictionary to generate a binary bit stream as the high-fidelity data segment.

[0067] It should be noted that extracting the original data sequence within the critical transient window and calculating the difference between adjacent data points is achieved using first-order differential pulse code modulation (DPCM) technology. This step utilizes the physical characteristic of industrial time series data having high autocorrelation over short time spans, meaning that the numerical jumps between adjacent sampling points are typically much smaller than the absolute amplitude of the signal. Specifically, for a length of... Key transient window raw data sequence The system first extracts the first data point. This value is retained as the full anchor value; subsequently, starting from the second data point, the formula is used... (in The difference between the current value and the previous value is calculated point by point. This operation transforms the original high-bit-width absolute value sequence into a low dynamic range difference sequence where the numerical distribution is concentrated near zero. .

[0068] It should be noted that the matching of the difference sequences based on a pre-set global mapping dictionary, and the mapping of high-frequency difference patterns to short codewords, is implemented using the Static Huffman Coding algorithm. This global mapping dictionary is not built in real-time for a single window, but is pre-generated based on statistical learning of massive amounts of historical key transient data. The system statistically analyzes the probability density function of historical difference values ​​(usually approximating a Laplace distribution) and constructs a binary coding tree based on the frequency of each difference symbol. Within the tree structure, extremely frequent small difference values ​​(such as...) are assigned... Shorter binary codeword lengths (e.g., 1-3 bits), and assigned smaller, less frequent large transition values ​​(e.g., ... (A relatively long codeword length.) This variable-length encoding mechanism ensures that the storage cost is inversely proportional to the probability of the information appearing.

[0069] It should be noted that entropy encoding and encapsulation of the difference sequence using the global mapping dictionary to generate a binary bitstream as the high-fidelity data segment is a process of table lookup and replacement and bitstream compaction. The system traverses the difference sequence. For each element in the dataset, its corresponding binary codeword is searched in the global mapping dictionary, and these variable-length codewords are concatenated sequentially into a continuous binary bitstream. For the very few out-of-range differential values ​​that exceed the dictionary's coverage, the system automatically inserts a preset escape code, and then directly appends the original fixed-length binary code of the out-of-range value to ensure that data can be recorded and restored without loss under any extreme conditions. The resulting bitstream is a high-fidelity data segment that achieves extreme compression at the physical level but contains complete waveform information at the logical level.

[0070] It is worth noting that the determination of the data source and update cycle for constructing the global mapping dictionary is based on the Kullback-Leibler divergence analysis of historical data distribution drift. The determination of the preset dictionary calibration cycle (e.g., 30 days) is based on the statistical analysis of the probability distribution stability of historical full lifecycle data. Specifically, the system backtracks the historical differential data sequence of the device throughout its lifecycle, uses a sliding window to calculate the cumulative rate of KL divergence between the differential probability distributions of two adjacent time periods (e.g., every day); it calculates the average time span required for the KL divergence value to accumulate from 0 to a critical point that significantly affects compression efficiency (e.g., 0.05), and sets this average time span as the preset dictionary calibration cycle. Every preset dictionary calibration cycle, the system selects the latest transient dataset and calculates the KL divergence value between its differential distribution and the current dictionary baseline distribution. If the divergence value exceeds the preset update threshold (e.g., 0.05), it indicates that the aging of the device or changes in operating conditions have caused a significant shift in the signal characteristics. The system will then re-trigger the Huffman tree construction process, generate a new global mapping dictionary, and send it to the encoding engine to maintain optimal compression efficiency.

[0071] For example, suppose a sequence of current data (in amperes) at the instant of motor startup is captured within a critical transient window. The first step is to perform first-order difference calculations. The baseline value is retained at 100. Subsequent difference calculations are as follows: , , , The resulting difference sequence The second step is to consult a pre-defined global mapping dictionary. Assuming historical statistics show that the difference value "0" has the highest probability of occurrence, it is encoded as 1; "+2" and "-2" have the next highest probabilities, encoded as 010 and 011 respectively; "+3" has a low probability, encoded as 00100. The base value "100" uses an 8-bit fixed-length code 01100100. The third step is to perform entropy encoding encapsulation. The bitstream is concatenated as follows: 01100100 (base) + 010 (+2) + 00100 (+3) + 011 (-2) + 1 (0). The final high-fidelity data segment bitstream is 01100100010001000111. Compared to the original storage (5 8-bit integers, totaling 40 bits), this bitstream only occupies 20 bits, achieving a 50% storage space saving while ensuring 100% lossless data restoration.

[0072] In step S14, the data within the stationary normal window is processed to obtain a sparse transform coefficient matrix, and the sparse transform coefficient matrix is ​​interval-mapped and encoded based on a non-uniform quantizer to generate a simplified background data segment, including:

[0073] The data within the stationary normal window are converted into a transformation coefficient matrix using the discrete cosine transform.

[0074] A high-frequency component mask matrix is ​​generated based on a preset compression force, and the coefficients of the corresponding high-frequency region in the transformation coefficient matrix are set to zero to obtain a sparse transformation coefficient matrix.

[0075] A non-uniform quantizer is used to perform interval mapping and encoding on the non-zero elements in the sparse transform coefficient matrix to obtain the background simplified data segment that retains only low-frequency trend information.

[0076] It should be noted that converting the data within the stationary normal window into a transform coefficient matrix using the Discrete Cosine Transform (DCT) leverages the excellent energy compression properties of DCT to transform the signal from a dispersed distribution in the time domain to a concentrated distribution in the frequency domain. Specifically, for a length of... The original data sequence of the steady normal window The system uses the one-dimensional DCT-II transform formula to calculate the first... Transformation coefficients :

[0077]

[0078] in, ;when Time coefficient ,when Time coefficient The calculated coefficient sequence This constitutes the transformation coefficient matrix (represented as a vector in the one-dimensional case). In this matrix, (The DC component) directly characterizes the mean of the data within the window, while to The AC component represents the fluctuations in the signal from low frequency to high frequency details. Since the data characteristics of the stationary normal window are slowly varying or constant, most of the signal energy will be concentrated in... And a few low-frequency coefficients.

[0079] It should be further explained that the extraction of the mean feature as a benchmark in this embodiment is naturally achieved through Discrete Cosine Transform (DCT); according to the definition of DCT transform, its first transform coefficient, i.e., the DC component... It exists with the arithmetic mean of all sample values ​​in the input sequence. The proportional relationship is such that, therefore, after performing the DCT transform, the mean feature is fully and accurately encoded in the first element of the transform coefficient matrix. In this process, subsequent masking and quantization processing are all based on this. The coefficients serve as essential baseline information, ensuring that the overall trend level of the window remains undistorted during the data simplification process.

[0080] It should be noted that generating a high-frequency component mask matrix based on a preset compression level, and setting the coefficients of the corresponding high-frequency regions in the transform coefficient matrix to zero to obtain a sparse transform coefficient matrix, is a process of performing frequency domain filtering and data sparsification. The system first determines the cutoff index of the retained coefficients based on the preset compression level. (in ), and then construct a length of binary mask vector In this mask vector, indices 0 to... The element is set to 1 (reserved area), index to The elements are forced to be 0 (cutoff region). The system will transform the coefficient sequence. With mask vector Perform the element-wise Hadamard product operation, i.e. After this operation, all frequencies above the cutoff frequency... The high-frequency noise figure is completely filtered out, and the resulting sparse transform coefficient matrix is... Only include the front A number of non-zero low-frequency effective coefficients are used to achieve a very high proportion of data dimensionality reduction.

[0081] It should be noted that the use of a non-uniform quantizer to perform interval mapping and encoding on the non-zero elements in the sparse transform coefficient matrix to obtain the simplified background data segment that retains only low-frequency trend information is specifically for the retained low-frequency coefficients. The process involves lossy compression. Given that DCT coefficients typically follow a Laplace distribution or a generalized Gaussian distribution (meaning the probability of a value being closer to 0 is higher), the system employs a non-uniform scalar quantizer. This quantizer sets a denser quantization step size in regions with smaller absolute values ​​to ensure resolution for subtle trend changes; and a sparser quantization step size in regions with larger absolute values ​​to save encoding space. The system then packages the quantized discrete index values ​​using run-length encoding or compact fixed-length encoding, resulting in a binary stream that serves as the simplified background data segment. While this segment loses high-frequency details, it fully preserves the overall mean and evolution trend of the data, meeting the background backtracking needs in industrial scenarios.

[0082] It is worth noting that the preset compression force (i.e., the cutoff index) The determination of the energy retention rate (DCT) is based on statistical analysis of the cumulative energy spectrum of historical stationary normal data. The system iterates through the historical database, traversing the window samples marked as stationary normal, and calculates the energy distribution of their DCT coefficients. A target energy retention rate threshold is then set. (For example, 95%), find the minimum value that satisfies the condition using the following formula. value:

[0083]

[0084] The system selects historical samples to calculate The 95th percentile of the value is used as the preset cutoff index. This strategy ensures that data volume is compressed to the maximum extent possible (usually...). Only (1 / 8 to 1 / 16 of the original value), it can restore more than 95% of the energy characteristics of the signal, effectively preventing trend distortion caused by over-compression.

[0085] For example, suppose a stationary normal window contains 8 sampling points ( ), data sequence is It exhibits a linear upward trend. The first step is to perform a DCT transform. This yields the coefficient sequence. It can be seen that the energy is highly concentrated on the first two coefficients. The second step is to apply a mask. Assuming the preset compression force determines that the first two coefficients are retained ( Mask vector The dot product operation yields a sparse matrix. The third step is quantization encoding. Assuming... Quantized to an 8-bit integer. The data is quantized to a 4-bit integer. Ultimately, only these 12 bits of data need to be stored. At the decoding end, the data reconstructed through the inverse IDCT transform is... The maximum error is only 0.01, but the amount of data is reduced from the original 256 bits (8 32-bit floating points) to 12 bits, with a compression ratio of up to 21 times.

[0086] In step S15, based on a preset frame structure, the high-fidelity data segment and the background simplified data segment are concatenated in the time domain, and a metadata index containing timestamps and importance tags is established to construct a primary hybrid storage structure and write it to a preset first-level storage medium, including:

[0087] Obtain the byte lengths of the high-fidelity data segment and the background simplified data segment respectively, and construct storage header information containing length information and version number;

[0088] The data is physically spliced ​​in the order of "storage header information - simplified background data segment - high-fidelity data segment" to generate a mixed data frame, and the checksum of the mixed data frame is calculated.

[0089] Generate a metadata index that records the starting physical address, time span, and importance classification label of the mixed data frame;

[0090] The metadata index is associated with the hybrid data frame and written into the preset first-level storage medium, wherein the first-level storage medium is a solid-state drive or a high-speed flash memory array.

[0091] It should be noted that obtaining the byte lengths of the high-fidelity data segment and the background simplified data segment, and constructing the storage header information containing length information and version number, is implemented based on bit field operations. Since the length of the compressed data stream generated in steps S13 and S14 may not be an integer multiple of 8, the system first performs zero-padding on these two data segments to align them to byte boundaries, and then calculates the number of bytes after alignment. and Subsequently, the system constructs a fixed-length (e.g., 4 bytes, or 32 bits) binary header structure. In this structure, a fixed high-order region (e.g., bits 28-31) is used to store the protocol version number to support future compression algorithm iterations; the middle region (e.g., bits 14-27) is used to store the length of the background compressed data segment. The lower-order regions (e.g., bits 0-13) are used to store the length of the high-fidelity data segment. This compact header design allows the decoder to directly parse the frame structure without decompressing the payload.

[0092] It should be noted that the physical concatenation of the mixed data frame in the order of "storage header information - background simplified data segment - high-fidelity data segment" and the calculation of the checksum of the mixed data frame are serialized packet assembly processes performed in a memory buffer. The system allocates a contiguous memory space and uses efficient memory copy instructions (such as memcpy) to sequentially write the generated storage header, background simplified data segment, and high-fidelity data segment into the corresponding offset positions of this space, forming a physically contiguous payload. Subsequently, the system uses CRC-32 (32-bit cyclic redundancy check) to perform polynomial division on the full binary data of the payload to generate a checksum. This checksum is appended to the end of the data frame and used in subsequent read operations to verify whether bit flips or corruption have occurred during data transmission or storage.

[0093] It should be noted that generating a metadata index that records the starting physical address, time span, and importance classification labels of the mixed data frames is a process of constructing a key-value data structure that supports fast random access. The system extracts the start timestamp of the current time window. Used as the index key; extract the end timestamp The pre-allocated physical start address of this frame in the storage medium The importance markers (such as Hybrid-Critical) generated in step S12 are used as index values. These index entries are designed as fixed-length structures (e.g., 32 bytes) to facilitate efficient insertion into the B+ tree index file, enabling millisecond-level retrieval and location of massive amounts of time-series data.

[0094] It should be noted that associating the metadata index with the hybrid data frame and writing it into the preset first-level storage medium is achieved through direct I / O of the underlying file system or DMA (direct memory access) channels of the NVMe protocol. The system first writes the assembled hybrid data frame to the data area of ​​the first-level storage medium in units of pages; after confirming successful writing and returning the physical address, the corresponding metadata index entry is written to the metadata log area of ​​the medium. The first-level storage medium specifically refers to an industrial-grade solid-state drive (SSD) or a high-speed flash memory array based on SLC / eMLC particles, characterized by high IOPS (read / write speeds per second) and low latency, to ensure that the storage system can withstand the high-concurrency write pressure of the raw data stream.

[0095] It is worth noting that the preset write bandwidth and capacity planning of the first-level storage medium, as well as the physical layout strategy of the hybrid data frames, are determined based on extreme value statistical analysis of historical data throughput peaks. The continuous write speed of the first-level medium... Must meet ,in To prevent buffer overflows caused by write blocking, the instantaneous data output rate is set at the 99.9th percentile of historical monitoring. Simultaneously, the physical layout, placing the simplified background data segment before the high-fidelity data segment, is designed based on a preview-first access mode. Statistics show that over 80% of historical query requests only need to load a low-frequency trend overview for plot preview. This layout allows the system to utilize scatter-gather I / O technology during reading, responding quickly by loading only the frame header and the preceding background segment, without needing to read the longer high-fidelity data at the back, thus significantly reducing I / O bus occupancy.

[0096] For example, assuming the current time window has been processed, a mixed data frame needs to be encapsulated. The first step is to obtain the length and construct the header. The background abbreviated data segment generated in step S14, after alignment, is 2 bytes (i.e., The high-fidelity data segment generated in step S13 is 3 bytes after alignment (i.e., Protocol version number (Binary 0001). The header information (32 bits) is constructed as follows: 0001(Version)+000000000000010( )+00000000000011( = 0x10002003 (4 bytes) in hexadecimal. The second step is physical concatenation. The contents of the memory buffer are: [Header:0x10002003] + [BG:2 bytes] + [HiFi:3 bytes]. The current payload length is... Bytes. Calculate the CRC-32 checksum (4 bytes), assuming the result is 0xAABBCCDD, and append it to the end. The final total length of the hybrid data frame is 13 bytes. The third step is to generate and write the index. The index entry content is {Key:2023-10-27 12:00:01.000,Value:[Span:100ms,Addr:0x00A0,Tag:Hybrid-Critical]}. The system writes this 13-byte data frame to the SSD physical address 0x00A0 and synchronizes the index entry to the metadata area. Compared to the original uncompressed storage (assuming the original data is 100 bytes), this hybrid structure not only saves 87% of space but also supports a fast retrieval mode with a read-only background.

[0097] In step S16, during the residency of the primary hybrid storage structure, query operations initiated through the metadata index are continuously recorded, access frequency is statistically analyzed, and a heat map is generated, including:

[0098] Parse the query request for the mixed data frame and identify the data segment types accessed in the query path;

[0099] The number of times the high-fidelity data segment and the simplified background data segment are accessed within a preset period are counted respectively, and the access latency overhead is calculated.

[0100] Based on the number of accesses and the access latency overhead, the retrieval efficiency index of each data segment is quantified and mapped to the visualized popularity distribution map.

[0101] It should be noted that parsing the query request for the mixed data frame and identifying the data segment type accessed in the query path is achieved through an I / O filter (FilterDriver) deployed in the Storage Controller Layer. This filter can intercept read commands initiated through the metadata index in real time and parse the read start offset and read length in the command. The system compares the extracted read range with the segment length parameter recorded in the storage header information constructed in step S15. and The system performs spatial overlap comparison. If the read range only covers the address range where the background simplified data segment is located, the query is marked as a preview access (Preview Mode); if the read range extends to cover the address range where the high-fidelity data segment is located, the query is marked as an analysis access (Analysis Mode). This identification mechanism can accurately distinguish whether the user is quickly browsing trends or performing precise fault diagnosis.

[0102] It should be noted that the method of counting the number of accesses to the high-fidelity data segment and the simplified background data segment within a preset period, and calculating the access latency overhead, is implemented using a memory-based time sliding window counter. The system maintains two state counters for each mixed data frame. (Number of previews) and (Number of deep accesses). Calculate access latency overhead. This aims to convert the number of accesses into physical I / O resource occupancy time, and its calculation formula is as follows:

[0103]

[0104] in, The average random access latency of the first-level storage medium. This refers to the sequential read throughput of the medium. and These represent the byte lengths of the two data segments. This formula accurately reflects the actual load pressure on the storage bus and read / write heads (or controller) under different access modes.

[0105] It should be noted that the quantification of retrieval efficiency metrics for each data segment based on the number of accesses and the access latency overhead, and mapping them to the visualized popularity distribution map, is achieved using a multi-dimensional normalized weighted algorithm. The retrieval efficiency metric (i.e., popularity value) The formula for calculating ) is:

[0106]

[0107] in, This represents the maximum single-frame access frequency recorded by the system within the current period. This represents the maximum I / O time capacity of a single storage node. The preset weighting coefficients (and This indicator The range of values ​​is This comprehensively represents the business popularity and system cost of the data frame. Finally, the system uses a pre-built color lookup table (LUT) to... Values ​​are linearly mapped to RGB color level values ​​(e.g.) Mapped to a cool blue, The color is mapped to a warm yellow. The color is mapped to a warm red color, thus generating an intuitive heat map. The preset color lookup table is based on statistical experiments on the cognitive response latency of maintenance personnel.

[0108] It is worth noting that the determination of the preset period and weighting coefficients is based on statistical analysis of user behavior patterns in historical query logs. The preset period is determined by analyzing historical access traffic sequences using the autocorrelation function (ACF) to extract the most significant lag time, typically chosen as 24 hours or 7 days to match shift or weekly inspection cycles in industrial production. The weighting coefficients... and The determination of the eigenvalues ​​is based on dimensionality reduction of the historical load feature matrix using Principal Component Analysis (PCA). The system constructs a covariance matrix containing the access frequency vector and the I / O latency vector, calculates the eigenvector of the first principal component, and normalizes the absolute values ​​of the corresponding components in this vector as weight values ​​(e.g., ...). This aims to objectively balance the contribution of access frequency and resource consumption in the popularity evaluation system.

[0109] For example, suppose that within a preset 24-hour period, the system monitors access to a certain mixed data frame (ID: Frame_007). The first step is to calculate access and computational overhead. The system detects preview-type access. Second, deep access Next. Known SSD media parameters. Read rate Make reading time consuming Read the complete frame ( )time consuming Then the total cost The second step is to calculate the popularity value. Let's assume the system's maximum access frequency... I / O time capacity Weight Substitute into the formula:

[0110]

[0111]

[0112] The third step is map mapping. In the heat distribution map, this value... Falling into the low heat range (<0.2), the system renders it as dark blue (RGB:0,0,139). This intuitively tells the administrator that although the data frame has a certain amount of preview, its overall resource consumption is extremely low, belonging to warm or cold data, and is a potential archiving and migration target when storage space is tight.

[0113] In step S17, if the heat map shows that the access frequency of the background simplified data segment is lower than a preset cold archiving threshold, then the background simplified data segment is stripped from the primary hybrid storage structure and migrated to a preset second-level storage medium to obtain a hierarchical archived dataset, including:

[0114] Identify cold data blocks in the heat distribution map whose access frequency is lower than the cold archive threshold. The cold data blocks correspond to background simplified data segments that have not been accessed for a long time.

[0115] The cold data block is read from the first-level storage medium and transferred to the preset second-level storage medium, wherein the second-level storage medium is a high-capacity hard disk drive or a tape library;

[0116] The physical address mapping in the metadata index is updated to point to a new address in the second-level storage medium, while the high-fidelity data segment and the metadata index are retained in the first-level storage medium, forming the hierarchical archive dataset with cold and hot separation.

[0117] It should be noted that identifying cold data blocks and performing stripping migration is implemented using object-based storage tiering technology. The system iterates through the heat map generated in step S16 and filters out the heat values. The data frame ID is less than the preset cold archiving threshold. For each target frame, the physical offset and length of the background simplified data segment in the first-level medium (SSD) are located using the storage header information generated in step S15. The data segment is read into memory via the DMA (Direct Memory Access) channel and encapsulated as a Sequential WriteCommand, which is then appended to the preset free sector of the second-level storage medium (HDD or tape library). This sequential write mode maximizes the throughput performance of the mechanical medium and avoids frequent head seeks. Updating the physical address mapping relationship in the metadata index uses a multi-level pointer redirection mechanism. The system modifies the Value field of the corresponding timestamp in the metadata index, splitting the original single address pointer pointing to consecutive physical blocks into a composite pointer structure. The main pointer pointing to the storage header and high-fidelity data segment in the SSD is retained, and a new address pointer pointing to the background simplified data segment in the HDD is added. At the same time, a TRIM or Deallocate instruction is sent to the file system of the first-level medium to mark the physical pages occupied by the original background data segment as invalid, so that the garbage collection mechanism (GC) can release space.

[0118] It is worth noting that the preset cold archiving threshold and the selection strategy for the second-level storage media are determined based on a full lifecycle analysis. The cold archiving threshold is determined by statistically analyzing the access frequency decay curve of historical data from its generation date, calculating the time span required for the access popularity to drop to 10% of the initial peak, and selecting the popularity value corresponding to that time point as the threshold (e.g., 0.2). This setting ensures that only data that has truly entered a dormant period will be migrated, avoiding frequent hot / cold jitter. The second-level storage media typically uses high-capacity enterprise-grade SATA hard drives or LTO tapes, whose unit capacity consumption must be less than 20% of that of the first-level media to achieve significant optimization.

[0119] An exemplary case, continuing from the previous steps. The heat value of the current mixed data frame. The first step is threshold determination. A preset cold archiving threshold is set. .because The system determines that the background simplified data segment in the frame is a cold data block and triggers a migration operation. The second step is data stripping and migration. Based on the frame structure of S15 [Header:4B][BG:2B][HiFi:3B][CRC:4B], the system reads 2 bytes of background data from the SSD's physical address 0x00A0+4 and writes it to the secondary media (HDD) at physical address 0xB000. The third step is index update. The system updates the metadata index, modifying the address mapping of the frame to {Header_HiFi:SSD:0x00A0,BG:HDD:0xB000}. Simultaneously, the SSD controller marks the physical units at addresses 0x00A4 to 0x00A5 as erasable. Ultimately, the data in this time window forms a "hot and cold separation" pattern: "index and key waveforms respond in seconds on the SSD, background trends are archived on the HDD," perfectly balancing retrieval performance and storage consumption.

[0120] In summary, this invention overcomes the blindness of traditional one-size-fits-all data processing by introducing a temporal importance scoring mechanism based on Shannon entropy, achieving precise perception and dynamic allocation of industrial data value density. By constructing a dual-modal compression system integrating lossless differential coding and transform domain sparsity, it significantly reduces redundant usage of stable, normal data while ensuring zero loss of critical fault symptoms. Furthermore, by establishing a closed-loop, lifecycle-based hierarchical archiving system based on access heat maps, it achieves a fundamental shift in storage resources from static accumulation to on-demand allocation. Ultimately, against the backdrop of explosive growth in massive sensor data, this invention successfully achieves an optimal balance between data fidelity and retrieval efficiency, providing solid technical support for the in-depth mining and asset management of industrial big data.

[0121] Reference Figure 2 The second embodiment of the present invention provides an industrial data lifecycle management system, comprising:

[0122] The streaming slicing and evaluation module is used to acquire the raw time series data of industrial equipment, perform time window segmentation of preset length, calculate the Shannon entropy within each time window, and obtain the time series importance score sequence.

[0123] The diversion decision module is used to compare the time-series importance score sequence with a preset diversion threshold, mark time windows with scores higher than the diversion threshold as critical transient windows, and mark time windows with scores lower than or equal to the diversion threshold as stationary normal windows.

[0124] The high-fidelity processing module is used to calculate the difference sequence for the original data sequence within the key transient window, and to match and encapsulate the difference sequence based on a preset global mapping dictionary to generate a high-fidelity data segment.

[0125] The background simplification processing module is used to process the data within the stationary normal window to obtain a sparse transform coefficient matrix, and to perform interval mapping and encoding on the sparse transform coefficient matrix based on a non-uniform quantizer to generate a background simplification data segment.

[0126] The hybrid storage construction module is used to perform time-domain splicing of the high-fidelity data segment and the background simplified data segment based on a preset frame structure, and to establish a metadata index containing timestamps and importance tags, thereby constructing a primary hybrid storage structure and writing it into a preset first-level storage medium.

[0127] The lifecycle monitoring module is used to continuously record query operations initiated through the metadata index during the residency of the primary hybrid storage structure, count the access frequency, and generate a heat distribution map.

[0128] The dynamic archive migration module is used to separate the background simplified data segment from the primary hybrid storage structure and migrate it to a preset second-level storage medium if the heat distribution map shows that the access frequency of the background simplified data segment is lower than a preset cold archive threshold, thereby obtaining a hierarchical archive dataset.

[0129] It should be noted that the industrial data lifecycle management system provided in this embodiment of the invention is used to execute all process steps of the industrial data lifecycle management method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0130] This invention also provides an electronic device. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as an industrial data lifecycle management program. When the processor executes the computer program, it implements the steps described in the various embodiments of the industrial data lifecycle management method above, for example... Figure 1 The step S11 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above system embodiments, such as the streaming slicing and evaluation module.

[0131] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0132] The electronic device may be a desktop computer, laptop, handheld computer, or smart tablet, etc. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0133] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting all parts of the electronic device via various interfaces and lines.

[0134] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0135] If the modules / units integrated into the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or system capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0136] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0137] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for managing the entire lifecycle of industrial data, characterized in that, include: The raw time series data of industrial equipment is acquired, and time windows of a preset length are divided. The Shannon entropy within each time window is calculated based on frequency to obtain the time series importance score sequence. The time-series importance score sequence is compared with a preset diversion threshold. Time windows with scores higher than the diversion threshold are marked as critical transient windows, and time windows with scores lower than or equal to the diversion threshold are marked as stable normal windows. The diversion threshold is set based on the recognition of curve shape. A difference sequence is calculated for the original data sequence within the critical transient window, and the difference sequence is matched and encapsulated based on a preset global mapping dictionary to generate a high-fidelity data segment. The high-fidelity data segment includes the first data point extracted as the full reference value and a binary bit stream generated by entropy encoding and encapsulation of the difference sequence. The data within the stationary normal window is processed to obtain a sparse transform coefficient matrix. The sparse transform coefficient matrix is ​​then interval-mapped and encoded based on a non-uniform quantizer to generate a simplified background data segment, wherein the simplified background data segment includes low-frequency transform coefficients and a binary data stream. Based on the preset frame structure, the high-fidelity data segment and the background simplified data segment are spliced ​​in the time domain, and a metadata index containing timestamps and importance tags is established to construct a primary hybrid storage structure and write it into the preset first-level storage medium. During the residency period of the primary hybrid storage structure, query operations initiated through the metadata index are continuously recorded, access frequency is statistically analyzed, and a heat distribution map is generated. If the heat map shows that the access frequency of the background simplified data segment is lower than the preset cold archiving threshold, then the background simplified data segment is stripped from the primary hybrid storage structure and migrated to the preset second-level storage medium to obtain a hierarchical archived dataset.

2. The industrial data lifecycle management method according to claim 1, characterized in that, The process involves acquiring the raw time-series data of industrial equipment, performing time window segmentation of a preset length, calculating the Shannon entropy within each time window, and obtaining a time-series importance score sequence, including: The raw time series data of industrial equipment is acquired, and the raw time series data is numerically discretized to map continuous analog signal values ​​into discrete symbols in a finite set of states. Statistically analyze the probability distribution of each discrete symbol appearing within each time window, and calculate the information entropy value of the probability distribution based on the Shannon entropy formula; The calculated information entropy value is used as an element of the temporal importance score sequence, wherein the information entropy value is used to quantify the waveform complexity and information content within the corresponding time window.

3. The industrial data lifecycle management method according to claim 1, characterized in that, The step of comparing the temporal importance score sequence with a preset triage threshold, marking time windows with scores higher than the triage threshold as critical transient windows, and marking time windows with scores lower than or equal to the triage threshold as stationary normal windows includes: The time-series importance score sequences are sorted and score distribution curves are constructed; Identify the step change points or density peak points in the scoring distribution curve, and extract the corresponding scoring values ​​as the diversion threshold; Traverse all time windows, mark windows with score values ​​greater than the diversion threshold as critical transient windows, and mark windows with score values ​​less than or equal to the diversion threshold as stable normal windows.

4. The industrial data lifecycle management method according to claim 1, characterized in that, The process involves calculating a difference sequence from the original data sequence within the critical transient window, and then matching and encapsulating the difference sequence based on a preset global mapping dictionary to generate a high-fidelity data segment, including: Extract the original data sequence within the key transient window, calculate the difference values ​​between adjacent data points, and obtain the difference sequence; The differential sequence is matched based on a preset global mapping dictionary, and the high-frequency differential patterns are mapped to short codewords. The differential sequence is entropy-encoded and encapsulated using the global mapping dictionary to generate a binary bit stream as the high-fidelity data segment.

5. The industrial data lifecycle management method according to claim 1, characterized in that, The process involves processing the data within the stationary normal window to obtain a sparse transform coefficient matrix, and then performing interval mapping and encoding on the sparse transform coefficient matrix based on a non-uniform quantizer to generate a simplified background data segment, including: The data within the stationary normal window are converted into a transformation coefficient matrix using the discrete cosine transform. A high-frequency component mask matrix is ​​generated based on a preset compression force, and the coefficients of the corresponding high-frequency region in the transformation coefficient matrix are set to zero to obtain a sparse transformation coefficient matrix. A non-uniform quantizer is used to perform interval mapping and encoding on the non-zero elements in the sparse transform coefficient matrix to obtain the background simplified data segment that retains only low-frequency trend information.

6. The industrial data lifecycle management method according to claim 1, characterized in that, Based on a preset frame structure, the high-fidelity data segment and the background simplified data segment are concatenated in the time domain, and a metadata index containing timestamps and importance tags is established to construct a primary hybrid storage structure, which is then written to a preset first-level storage medium, including: Obtain the byte lengths of the high-fidelity data segment and the background simplified data segment respectively, and construct storage header information containing length information and version number; The data is physically spliced ​​in the order of "storage header information - simplified background data segment - high-fidelity data segment" to generate a mixed data frame, and the checksum of the mixed data frame is calculated. Generate a metadata index that records the starting physical address, time span, and importance classification label of the mixed data frame; The metadata index is associated with the hybrid data frame and written into the preset first-level storage medium, wherein the first-level storage medium is a solid-state drive or a high-speed flash memory array.

7. The industrial data lifecycle management method according to claim 6, characterized in that, During the residency period in the primary hybrid storage structure, query operations initiated through the metadata index are continuously recorded, access frequency is statistically analyzed, and a heat map is generated, including: Parse the query request for the mixed data frame and identify the data segment types accessed in the query path; The number of times the high-fidelity data segment and the simplified background data segment are accessed within a preset period are counted respectively, and the access latency overhead is calculated. Based on the number of accesses and the access latency overhead, the retrieval efficiency index of each data segment is quantified and mapped to the visualized popularity distribution map.

8. The industrial data lifecycle management method according to claim 1, characterized in that, If the heat map shows that the access frequency of the background simplified data segment is lower than a preset cold archiving threshold, then the background simplified data segment is stripped from the primary hybrid storage structure and migrated to a preset second-level storage medium to obtain a hierarchical archived dataset, including: Identify cold data blocks in the heat distribution map whose access frequency is lower than the cold archive threshold. The cold data blocks correspond to background simplified data segments that have not been accessed for a long time. The cold data block is read from the first-level storage medium and transferred to the preset second-level storage medium, wherein the second-level storage medium is a high-capacity hard disk drive or a tape library; The physical address mapping in the metadata index is updated to point to a new address in the second-level storage medium, while the high-fidelity data segment and the metadata index are retained in the first-level storage medium, forming the hierarchical archive dataset with cold and hot separation.

9. An industrial data lifecycle management system, characterized in that, include: The streaming slicing and evaluation module is used to acquire the raw time series data of industrial equipment, perform time window segmentation of preset length, calculate the Shannon entropy within each time window, and obtain the time series importance score sequence. The diversion decision module is used to compare the time-series importance score sequence with a preset diversion threshold, mark time windows with scores higher than the diversion threshold as critical transient windows, and mark time windows with scores lower than or equal to the diversion threshold as stationary normal windows. The high-fidelity processing module is used to calculate the difference sequence for the original data sequence within the key transient window, and to match and encapsulate the difference sequence based on a preset global mapping dictionary to generate a high-fidelity data segment. The background simplification processing module is used to process the data within the stationary normal window to obtain a sparse transform coefficient matrix, and to perform interval mapping and encoding on the sparse transform coefficient matrix based on a non-uniform quantizer to generate a background simplification data segment. The hybrid storage construction module is used to perform time-domain splicing of the high-fidelity data segment and the background simplified data segment based on a preset frame structure, and to establish a metadata index containing timestamps and importance tags, thereby constructing a primary hybrid storage structure and writing it into a preset first-level storage medium. The lifecycle monitoring module is used to continuously record query operations initiated through the metadata index during the residency of the primary hybrid storage structure, count the access frequency, and generate a heat distribution map. The dynamic archive migration module is used to separate the background simplified data segment from the primary hybrid storage structure and migrate it to a preset second-level storage medium if the heat distribution map shows that the access frequency of the background simplified data segment is lower than a preset cold archive threshold, thereby obtaining a hierarchical archive dataset.

Images