Time window based time series data compression tiered storage method and system

Abstract

The application provides a time window-based time series data compression hierarchical storage method and system, relates to the technical field of time windows, and the method comprises the following steps: firstly, collecting the running time series data of an industrial device, dividing the time series data into data blocks according to fixed time windows, and extracting statistical features of each data block to generate working condition feature data; then, analyzing the energy distribution features of the data blocks, generating optical pulse coding sequences through photoelectric pulse conversion and pulse width modulation technology; determining the compression level through analysis of the pulse density distribution, and combining the working condition feature data to perform compression processing by using a corresponding coding algorithm; finally, distributing the data to different storage areas according to the compression level to complete hierarchical storage. The application improves the storage efficiency of industrial monitoring data and reduces the storage cost.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of time window, in particular to a time window-based time series data compression hierarchical storage method and system. BACKGROUND

[0002] In the field of industrial IoT device monitoring, with the continuous improvement of sensor sampling frequency and deployment scale, device runtime time series data presents an explosive growth, which has the characteristics of continuous collection, periodic change and long-term preservation, thereby putting forward higher requirements on the capacity planning, access efficiency and cost control of the storage system, and thus an intelligent hierarchical storage scheme is needed, which can automatically distinguish the storage priority according to the data value, reduce the overall storage overhead while ensuring the integrity of key data.

[0003] At present, an existing scheme for this demand adopts a segmented compression storage strategy based on a fixed threshold. The scheme first segments the time series data into equal-length segments, then calculates the peak value and mean value features of each segment of data, and divides the data into high fluctuation segments and low fluctuation segments according to a pre-set fixed threshold; then, the high fluctuation segments are saved with lossless compression mode to retain detailed features, while the low fluctuation segments are compressed with lossy compression mode to reduce data volume; finally, the processed data is stored in different performance storage media according to the category.

[0004] However, this scheme has certain limitations in actual application, for example, the fixed threshold is difficult to adapt to the dynamic changes of device operating conditions, which may lead to important operating condition features being lossy compressed or redundant data occupying too much storage resources; at the same time, relying solely on peak value and mean value features for judgment is not sufficient to represent the energy distribution characteristics of the data; in addition, the matching degree of the segmented compression strategy and the actual value characteristics of the data needs to be improved, which affects the rationality of storage resource allocation. SUMMARY

[0005] The present application provides a time window-based time series data compression hierarchical storage method and system to solve the problems of low storage efficiency and high storage cost of industrial monitoring data in the prior art.

[0006] To solve the above technical problems, in a first aspect, the present application provides a time window-based time series data compression hierarchical storage method, comprising:

[0007] Collecting runtime time series data of an industrial device;

[0008] Dividing the runtime time series data of the industrial device into a plurality of data blocks according to a fixed time window, and performing feature extraction processing on each data block to generate operating condition feature data;

[0009] The energy distribution characteristics of each data block are obtained by analyzing and processing each data block, and the energy distribution characteristics are pulse conversion processed, and based on the processing result, an optical pulse coding sequence is generated by pulse width modulation technology;

[0010] The density distribution map is generated by density analysis processing on the optical pulse coding sequence, and the compression level corresponding to the density distribution map is determined;

[0011] In combination with the working condition characteristic data, each data block is compressed by using the coding algorithm corresponding to the compression level, and the compressed data block is distributed to different storage areas for storage according to the compression level, so as to complete the hierarchical storage.

[0012] Optionally, the density distribution map is generated by density analysis processing on the optical pulse coding sequence, and the compression level corresponding to the density distribution map is determined, comprising:

[0013] The optical pulse coding sequence is time segmented and divided into multiple equal length time periods;

[0014] The target pulse number of each time period is counted by statistical processing;

[0015] The density distribution map is generated based on the target pulse number of each time period;

[0016] The density distribution map is divided to determine multiple density intervals;

[0017] Based on the preset mapping relationship, the data blocks corresponding to each density interval are marked as the corresponding compression level.

[0018] Optionally, the density distribution map is generated based on the target pulse number of each time period, comprising:

[0019] The target pulse number of each time period is arranged to form a pulse time sequence;

[0020] The average pulse density of each sliding window is obtained by averaging the pulse time sequence;

[0021] The average pulse density of each sliding window is arranged in time sequence to form a density value sequence;

[0022] The density value sequence is interpolated to generate a density distribution curve using a cubic spline interpolation algorithm;

[0023] The density distribution curve is divided into multiple density intervals according to a preset density threshold;

[0024] Each density interval is color coded to generate a density distribution map.

[0025] Optionally, the analyzing and processing each data block to obtain an energy distribution feature of each data block, and performing pulse conversion processing on the energy distribution feature, generating an optical pulse code sequence based on the processing result through pulse width modulation technology, comprising:

[0026] integrating each data block to obtain the sum of squares of amplitudes of all sampling points in the data block, and obtaining a total energy value of the data block based on the sum of squares of amplitudes;

[0027] based on the total energy value, normalizing the sampling points in the data block to obtain the energy contribution degree of each sampling point;

[0028] generating an energy distribution feature according to the energy contribution degree, and mapping the energy distribution feature of each sampling point to generate an analog electrical signal;

[0029] converting each analog electrical signal into an optical pulse signal with a corresponding pulse width through pulse width modulation technology, the optical pulse signal being a processing result;

[0030] serializing all optical pulse signals in time sequence to generate an optical pulse code sequence.

[0031] Optionally, the converting each analog electrical signal into an optical pulse signal with a corresponding pulse width through pulse width modulation technology, comprising:

[0032] voltage sampling processing the analog electrical signal to obtain a corresponding voltage amplitude;

[0033] mapping the voltage amplitude to a corresponding pulse width value according to a preset conversion curve;

[0034] optimizing the pulse width value based on a convolutional neural network, and generating a square wave pulse signal based on the optimized pulse width value;

[0035] inputting the square wave pulse signal into an optical emitter device to drive the optical emitter device to generate an optical pulse signal.

[0036] Optionally, the feature extraction processing each data block to generate working condition feature data, comprising:

[0037] first processing each data block to obtain the arithmetic mean of all sampling points in the data block, and obtaining mean feature data based on the arithmetic mean;

[0038] performing second processing on each data block to obtain an average value of a sum of squares of deviations of each sampling point in the data block from the mean value feature data, and obtaining variance feature data based on the average value of the sum of squares of deviations;

[0039] combining the mean value feature data and the variance feature data in chronological order to form a preliminary feature sequence;

[0040] performing standardization processing on the preliminary feature sequence to generate working condition feature data.

[0041] Optionally, in combination with the working condition feature data, each data block is compressed by using an encoding algorithm corresponding to the compression level, including:

[0042] When the compression level is the no-compression level, the data block is encapsulated, and header information containing a timestamp and a data block identifier is added in the encapsulation process to generate and store a complete data packet;

[0043] When the compression level is the first-level compression level, based on the working condition feature data, the most similar historical working condition feature data is searched in a preset historical database, based on a historical data block corresponding to the historical working condition feature data, a difference value data between the current data block and the historical data block is calculated, and a first compression processing is performed on the difference value data by using an adaptive encoding algorithm to obtain a corresponding compressed data block;

[0044] When the compression level is the second-level compression level, a second compression processing is performed on the data block by using a lossy compression algorithm to obtain a corresponding compressed data block.

[0045] In a second aspect, the present application provides a time window-based time series data compression hierarchical storage system, including:

[0046] An acquisition module is configured to acquire runtime time series data of an industrial device;

[0047] A division module is configured to divide the runtime time series data of the industrial device into a plurality of data blocks according to a fixed time window, and perform feature extraction processing on each data block to generate working condition feature data;

[0048] An analysis module is configured to perform analysis processing on each data block to obtain an energy distribution feature of each data block, and perform pulse conversion processing on the energy distribution feature, and based on the processing result, generate an optical pulse code sequence by using a pulse width modulation technology;

[0049] A generation module is configured to perform density analysis processing on the optical pulse code sequence to generate a density distribution map, and determine a compression level corresponding to the density distribution map;

[0050] The compression module is configured to combine the working condition characteristic data, compress each data block by using an encoding algorithm corresponding to the compression level, and store the compressed data block in different storage areas according to the compression level, so as to complete the hierarchical storage.

[0051] In a third aspect, the present application provides an electronic device, comprising:

[0052] a memory configured to store a computer program;

[0053] a processor configured to execute the computer program to implement the steps of the time window-based time series data compression and hierarchical storage method according to the first aspect.

[0054] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executable by a processor to implement the steps of the time window-based time series data compression and hierarchical storage method according to the first aspect.

[0055] The technical scheme provided by the present application has the following beneficial effects:

[0056] First, the running time series data of the industrial equipment is collected to obtain the original monitoring data of the equipment running, thereby providing a data basis for subsequent processing. Then, the data blocks are divided according to a fixed time window and statistical features are extracted, so that continuous data is converted into discrete features, thereby forming standardized working condition data representing the running state of the equipment. Then, energy distribution feature analysis is performed on the data blocks and a light pulse coding sequence is generated, so that the data energy features are converted into optical signals, thereby providing a pulse density analysis basis for compression level judgment. Then, pulse density analysis is performed on the light pulse coding sequence and the compression level is determined, so that the best compression strategy is automatically selected according to the data activity level, thereby realizing intelligent allocation of storage resources. Then, the corresponding encoding algorithm is used to compress the working condition characteristic data, and different compression methods are used for different data characteristics, so as to ensure the integrity of important data and improve the compression efficiency. Finally, the compressed data is stored according to the level, thereby realizing hierarchical storage management of the data, optimizing the utilization rate of the storage resources, and reducing the storage cost.

[0057] Further, first, time segmentation processing is performed on the light pulse coding sequence, the number of pulses in each time period is counted, and a density distribution map is generated. Then, according to the division result of the density interval and the preset mapping relationship, the compression level corresponding to each data block is automatically determined. This process realizes intelligent compression level judgment based on the data activity features, and makes the compression strategy match the data importance, thereby improving the rationality and efficiency of the storage resource allocation.

[0058] These and other aspects of the present application will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings described below are only some embodiments of the present application, and all other embodiments obtained by a person of ordinary skill in the art without creative work based on these embodiments are within the protection scope of the present application.

[0060] Figure 1 A flow chart of a time window-based time series data compression hierarchical storage method provided by an embodiment of the present application;

[0061] Figure 2 A specific implementation schematic diagram of a time window-based time series data compression hierarchical storage method provided by an embodiment of the present application;

[0062] Figure 3 A structural schematic diagram of a time window-based time series data compression hierarchical storage system provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] In view of the problems in the prior art, the present application provides a time window-based time series data compression hierarchical storage method. The method realizes intelligent compression decision through multi-dimensional feature collaborative analysis. Specifically, first, the time series data is divided into fixed time windows, and statistical features are extracted to generate standardized working condition data. Then, the photoelectric pulse conversion technology is innovatively introduced to convert the data energy distribution features into optical pulse coding sequences. Subsequently, the optimal compression level is dynamically determined through analysis of the pulse density distribution, and the corresponding coding strategy is selected in combination with the working condition feature data. Therefore, the method breaks through the limitation of fixed threshold, and through the dual judgment of optical pulse density and working condition features, the compression level is accurately matched with the data importance, which not only guarantees the integrity of the key data, but also improves the storage resource utilization efficiency, effectively solving the problem that the compression strategy does not match the data value in the prior art.

[0064] In order to enable the person skilled in the art to better understand the present application, the present application will be further described in detail below in combination with the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the protection scope of the present application.

[0065] The core of the present application is to provide a time window-based time series data compression hierarchical storage method. A flow schematic diagram of a specific embodiment of the method is shown in Figure 1 The method comprises:

[0066] Step 101: collecting runtime time series data of the industrial equipment.

[0067] In step 101, the runtime time series data of the industrial equipment refers to the time series data continuously collected by sensors during the operation of the industrial equipment, such as vibration, temperature, pressure, etc. These data are recorded in time order and reflect the running state and change trend of the equipment, where the vibration parameter can refer to vibration data such as vibration acceleration, with unit g.

[0068] For example, taking vibration monitoring of a certain wind turbine as an example, a vibration sensor is installed at the bearing of the generator to collect vibration acceleration data at a frequency of 100 times per second. After 10 minutes of continuous collection, 60000 vibration data points arranged in time order are obtained, which form the runtime time series data of the industrial equipment.

[0069] Step 102: dividing the runtime time series data of the industrial equipment into multiple data blocks according to a fixed time window, and performing feature extraction processing on each data block to generate working condition feature data.

[0070] In step 102, the fixed time window refers to a pre-set interval of equal time length for dividing continuous time series data into multiple data blocks; the data block refers to the collection of all sampling point data within the fixed time window; and the working condition feature data refers to the standardized statistical feature data for unified representation of the equipment running state.

[0071] In the embodiments of the present application, the continuous time series data is first divided into multiple data blocks according to a fixed time length, then the arithmetic mean value of all sampling points of each data block is calculated to obtain the mean feature data, and the average of the square deviation of each sampling point from the mean value is calculated to obtain the variance feature data, then the mean feature data and the variance feature data are combined in time order to form a preliminary feature sequence, and finally the sequence is standardized to generate the working condition feature data.

[0072] For example, the 10-minute vibration data is first divided into 60 data blocks according to 10-second intervals, each containing 1000 sampling points; then the arithmetic mean value of 1000 sampling points of the first data block is calculated to obtain the mean feature data 0.25g, and the average of the square deviation of each sampling point from 0.25g is calculated to obtain the variance feature data 0.08g², then the two feature values are combined and standardized to generate the working condition feature data of the data block; all 60 data blocks are processed in this way to finally generate a complete working condition feature data sequence.

[0073] Step 103: Analyzing each data block to obtain energy distribution characteristics of each data block, and performing pulse conversion processing on the energy distribution characteristics, and generating an optical pulse coding sequence based on the processing result by using pulse width modulation technology.

[0074] In step 103, the energy distribution characteristics refer to the distribution of the energy values of the sampling points in the data block, reflecting the energy concentration degree of the data; and the optical pulse coding sequence refers to a sequence composed of a series of optical pulses with different widths, and each pulse width corresponds to a specific energy value.

[0075] In the embodiments of the present application, first, the sum of the squares of the amplitudes of all sampling points in each data block is calculated to obtain a total energy value, then the sampling points are normalized based on the total energy value to obtain the energy contribution degree of each sampling point, then the energy distribution characteristics sequence is generated according to the energy contribution degree, and the energy distribution characteristics are input into the photoelectric conversion unit to generate an analog electrical signal, then the analog electrical signal is pulse width modulated to generate an optical pulse signal with a corresponding width, and finally the optical pulse signals are arranged in time sequence to generate an optical pulse coding sequence.

[0076] For example, first, the sum of the squares of the amplitudes of 1000 sampling points in the first data block is calculated to obtain a total energy value of 1250g²·s, then the energy contribution degree of each sampling point is calculated, wherein the energy contribution degree of the 500th sampling point is 0.0024, then the energy contribution degree is converted into a 2.4-volt analog electrical signal, and then the analog electrical signal is converted into a 24-microsecond-width optical pulse signal by pulse width modulation; after processing all the sampling points, the optical pulse coding sequence is finally generated by arranging 1000 optical pulses in time sequence.

[0077] Step 104: Performing density analysis processing on the optical pulse coding sequence to generate a density distribution map, and determining a compression level corresponding to the density distribution map.

[0078] In step 104, the density distribution map refers to a graphical representation that directly displays the changes in pulse density in different time periods; and the compression level refers to different compression degree levels divided according to the importance of the data, and the compression level includes no compression, first-level compression, and second-level compression.

[0079] In the embodiments of the present application, first, the optical pulse coding sequence is divided into a plurality of time periods according to a fixed time interval, then the number of pulses in each time period is counted to generate a pulse number sequence, and the pulse number sequence is subjected to sliding average processing to obtain an average pulse density, then the density distribution map is generated based on the average pulse density, and the density interval is divided to determine the high, medium, and low density intervals, and finally the compression level corresponding to each data block is determined according to the mapping relationship between the density interval and the compression level.

[0080] For example, the light pulse coding sequence of the first data block is first divided into 10 time periods at 1 second intervals, then the number of pulses in each time period is counted to obtain the sequence [25, 28, 32, 45, 52, 48, 38, 30, 26, 24], and a sliding average processing is performed to obtain the average pulse density sequence [28.3, 35, 41.7, 48.3, 46, 38.7, 31.3, 26.7], then a density distribution map is generated, and a density interval division is performed, and finally the first compression level corresponding to the data block is determined.

[0081] Step 105: In combination with the working condition characteristic data, the data blocks are compressed by using the encoding algorithm corresponding to the compression level, and the compressed data blocks are distributed to different storage areas according to the compression level for storage, so as to complete the hierarchical storage.

[0082] In step 105, the hierarchical storage refers to a storage strategy of storing data of different compression levels to different performance storage media.

[0083] In the embodiments of the present application, the original data is directly stored without compression level, the difference data is losslessly compressed based on the similar historical data block searched in the historical database according to the working condition characteristic data for the first compression level, and the high-ratio compression is performed by using the lossy compression algorithm for the second compression level; finally, the compressed data is stored in the high-speed, medium-speed, and large-capacity storage areas according to the level, and the storage index is established to record the storage location information.

[0084] For example, the first data block belongs to the first compression level, first based on its working condition characteristic data, the working condition characteristic data includes the mean value 0.25 and the variance 0.08, the similar historical data block is searched in the historical database, then the difference data between the current data block and the historical data block is calculated, and the difference data is losslessly compressed, finally the compressed data is stored in the medium-speed storage area, and the storage location and compression characteristic information are recorded.

[0085] The method realizes the adaptive hierarchical storage of the time series data of the industrial equipment through multi-dimensional feature analysis and intelligent compression decision.

[0086] In order to solve the problem of storage efficiency optimization of the monitoring data of the industrial equipment, in some embodiments, step 104: the density analysis processing is performed on the light pulse coding sequence to generate a density distribution map, and a compression level corresponding to the density distribution map is determined, as shown in Figure 2 As shown in the figure, it includes:

[0087] Step 201: Time segmentation processing is performed on the light pulse coding sequence to divide it into multiple equal-length time periods.

[0088] In the embodiment of the present application, the complete optical pulse coding sequence is segmented according to a preset time unit to generate a series of equal-length time interval segments, wherein each segment contains a number of optical pulse signals, providing an analysis basis for subsequent pulse quantity statistics.

[0089] Step 202: Statistical processing is performed on each time period to count the target pulse quantity.

[0090] In the embodiment of the present application, the number of optical pulse signals is counted for each divided time period one by one, and the number of optical pulses contained in each time period is recorded, thereby forming a pulse quantity sequence arranged in time sequence.

[0091] Step 203: A density distribution map is generated based on the target pulse quantity of each time period.

[0092] In the embodiment of the present application, based on the pulse quantity statistical results of each time period, a density distribution map is generated using data visualization technology, wherein the map can intuitively reflect the density distribution characteristics of the optical pulse coding sequence in different time periods.

[0093] Step 204: The density distribution map is divided to determine a plurality of density intervals.

[0094] In the embodiment of the present application, the map is divided into a high-density interval, a medium-density interval, and a low-density interval according to a preset density threshold, wherein each interval represents a different data activity level.

[0095] Step 205: Based on a preset mapping relationship, the data blocks corresponding to each density interval are marked as corresponding compression levels.

[0096] In step 205, the mapping relationship is a pre-set corresponding rule that associates different density intervals with corresponding compression levels, such as a high-density interval corresponding to a low compression level and a low-density interval corresponding to a high compression level.

[0097] In the embodiment of the present application, according to the pre-established mapping relationship table, the data blocks corresponding to each density interval are marked as corresponding compression levels, wherein the high-density interval is marked as a no-compression level, the medium-density interval is marked as a first-level compression level, and the low-density interval is marked as a second-level compression level.

[0098] The following is a specific example:

[0099] In the wind turbine vibration monitoring scenario, the density analysis is performed on the light pulse coding sequence generated by the first data block, which contains 1000 light pulse signals and has a total duration of 10 seconds. First, the light pulse coding sequence is time segmented, and 10 equal-length time segments are divided according to 1-second intervals, each of which contains a theoretical capacity of 100 light pulse signals.

[0100] Then, statistical processing is performed on each time segment, and the actual number of light pulses in each time segment is obtained as 25, 28, 32, 45, 52, 48, 38, 30, 26, and 24, thereby forming a target pulse number sequence [25, 28, 32, 45, 52, 48, 38, 30, 26, 24].

[0101] Subsequently, a density distribution map is generated based on the sequence, in which the horizontal axis represents the 10 time segment numbers, and the vertical axis represents the pulse density value in units of pulses per second. The density distribution map is divided, and the average pulse density is calculated using a sliding window average algorithm, with a window length of 3 time segments. The first window calculates , the second window calculates , and the average pulse density sequence [28.3, 35, 41.7, 48.3, 46, 38.7, 31.3, 26.7] is obtained by sequential calculation.

[0102] Subsequently, the map is divided into three density intervals according to the preset density threshold. The density value greater than 40 is the high-density interval, which includes the 3rd to 5th time segments. The density value between 30 and 40 is the medium-density interval, which includes the 2nd and 6th time segments. The density value less than 30 is the low-density interval, which includes the 1st, 7th to 10th time segments.

[0103] Finally, based on the preset mapping relationship, the data segment corresponding to the high-density interval is marked as the non-compression level, the medium-density interval is marked as the first compression level, and the low-density interval is marked as the second compression level. It is determined that the main part of the data block belongs to the first compression level and needs to be processed using the difference compression algorithm based on similar working conditions.

[0104] In the embodiments of the present application, intelligent determination of data compression levels is achieved through density analysis processing, which accurately matches the compression strategy with the data activity level, ensuring the integrity of important data and improving the utilization efficiency of storage resources.

[0105] To solve the accuracy and visualization problems of the density distribution map generation, in some embodiments, step 203: generating a density distribution map based on the target pulse number of each time segment, includes:

[0106] Step 301: arrange the target pulse numbers of each time period to form a pulse time sequence.

[0107] In step 301, the pulse time sequence refers to a sequence formed by arranging the target pulse numbers of each time period in time sequence, wherein each value in the sequence represents the number of light pulses in the corresponding time period, reflecting the trend and law of the number of light pulses changing with time.

[0108] In the embodiment of the present application, the target pulse numbers of each time period obtained by statistics are sorted in ascending order of time period number, generating a pulse time sequence that fully reflects the change of pulse number with time.

[0109] Step 302: average processing is performed on the pulse time sequence to obtain the average pulse density of each sliding window.

[0110] In step 302, the average pulse density refers to the average value of the pulse density obtained by average processing, and each value represents the average value of the pulse number of multiple time periods in a sliding window, which is used to smooth the fluctuations of a single time period and more stably reflects the change trend of the pulse density.

[0111] In the embodiment of the present application, the average pulse density values obtained by average processing are arranged in corresponding time sequence to generate a continuously changing density value sequence.

[0112] Step 303: arrange the average pulse density of each sliding window in time sequence to form a density value sequence.

[0113] In step 303, the density value sequence refers to a sequence formed by arranging the average pulse density values calculated for each sliding window in time sequence, reflecting the trend of the pulse density changing with time.

[0114] In the embodiment of the present application, the average pulse density values obtained by average processing are arranged in corresponding time sequence to generate a continuously changing density value sequence.

[0115] Step 304: interpolate the density value sequence to generate a density distribution curve using a cubic spline interpolation algorithm.

[0116] In step 304, the density distribution curve refers to a continuous and smooth curve generated by interpolating the density value sequence using a cubic spline interpolation algorithm, wherein the curve takes time as the horizontal axis and pulse density as the vertical axis, and intuitively displays the continuous distribution characteristics of the pulse density changing with time.

[0117] Step 305: divide the density distribution curve into multiple density intervals according to a preset density threshold.

[0118] In step 305, the density threshold refers to a preset critical value for dividing the density interval, and the plurality of density intervals refer to different density level regions divided according to the numerical range of the density distribution curve.

[0119] In the embodiment of the present application, the curve is divided into different density level regions such as a high-density interval, a medium-density interval and a low-density interval according to the preset density threshold.

[0120] Step 306: Color coding processing is performed on each density interval to generate a density distribution map.

[0121] In the embodiment of the present application, a specific color identifier is assigned to each density interval, a dark color identifier is used for the high-density interval, and a light color identifier is used for the low-density interval, to generate a density distribution map that intuitively displays the density distribution.

[0122] In the embodiment of the present application, the accuracy of data processing and the visual effect are improved through the data processing and visualization technology of the system.

[0123] In order to solve the demand for feature extraction and signal conversion of industrial equipment monitoring data, in some embodiments, step 103: each data block is analyzed and processed to obtain the energy distribution characteristics of each data block, and the energy distribution characteristics are pulse conversion processed, based on the processing result, an optical pulse coding sequence is generated through pulse width modulation technology, including:

[0124] Step 401: Each data block is integrated to obtain the sum of the amplitude squares of all sampling points in the data block, and based on the sum of the amplitude squares, the total energy value of the data block is obtained.

[0125] In step 401, the total energy value of the data block is a value obtained by accumulating the amplitude squares of all sampling points, which reflects the overall energy level of the data block.

[0126] In the embodiment of the present application, the square of the amplitude of each sampling point in the data block is first taken, and then the amplitude square values of all sampling points are added to obtain a value representing the total energy of the data block, which represents the overall energy characteristics of the data block.

[0127] Step 402: Based on the total energy value, the sampling points in the data block are normalized to obtain the energy contribution degree of each sampling point.

[0128] In step 402, the energy contribution degree is the proportion of the energy value of a single sampling point in the total energy, which reflects the contribution degree of the sampling point to the overall energy.

[0129] In the embodiment of the present application, the energy contribution degree of each sampling point is obtained by dividing the square of the amplitude of each sampling point by the total energy value of the data block, and finally the energy contribution degrees of all sampling points are combined to form the energy distribution feature sequence.

[0130] Step 403: generating an energy distribution feature according to the energy contribution degree, and performing mapping processing on the energy distribution feature of each sampling point to generate an analog electrical signal.

[0131] In step 403, the analog electrical signal is an electrical signal whose voltage value is proportional to the energy contribution degree.

[0132] In the embodiment of the present application, the energy contribution degree of each sampling point is multiplied by a fixed coefficient to convert into an analog electrical signal of a corresponding voltage value, and this conversion process maintains the relative relationship of the energy distribution feature.

[0133] Step 404: converting each analog electrical signal into an optical pulse signal with a corresponding pulse width by pulse width modulation technology, and the optical pulse signal is a processing result.

[0134] In step 404, the optical pulse signal refers to an optical domain signal converted from the analog electrical signal by pulse width modulation technology, and the pulse width of the signal is proportional to the voltage value of the input analog electrical signal, that is, the higher the voltage value, the wider the pulse width, and the lower the voltage value, the narrower the pulse width.

[0135] In the embodiment of the present application, the analog electrical signal is first input into an optical pulse modulation device, and then an optical pulse signal with a corresponding width is generated according to the voltage value, wherein the voltage value is proportional to the optical pulse width.

[0136] Step 405: performing serialization processing on all optical pulse signals in time sequence to generate an optical pulse coding sequence.

[0137] In the embodiment of the present application, by energy feature analysis and photoelectric signal conversion, the energy distribution feature of the data block is converted into an optical pulse coding sequence, which provides an accurate data basis for subsequent pulse density analysis, so as to realize effective extraction of data features and optimized conversion of signal form.

[0138] In order to solve the problem of photoelectric signal conversion precision and stability, in some embodiments, step 404: converting each analog electrical signal into an optical pulse signal with a corresponding pulse width by pulse width modulation technology, comprises:

[0139] Step 501: performing voltage sampling processing on the analog electrical signal to obtain a corresponding voltage amplitude.

[0140] In step 501, the voltage amplitude refers to the specific voltage value of the analog electrical signal at the sampling time.

[0141] In the embodiment of the present application, the input analog electrical signal is timing sampled by an analog-to-digital converter to obtain the voltage value corresponding to each sampling time, thereby obtaining accurate voltage amplitude data.

[0142] Step 502: mapping the voltage amplitude to a corresponding pulse width value according to a preset conversion curve.

[0143] In step 502, the conversion curve is a pre-calibrated correspondence curve of voltage value and pulse width.

[0144] Step 503: performing optimization processing on the pulse width value based on a convolutional neural network, and generating a square wave pulse signal based on the optimized pulse width value.

[0145] In step 503, the square wave pulse signal refers to a rectangular pulse signal with a specific width and amplitude.

[0146] In the embodiment of the present application, the convolutional neural network is used to smooth the pulse width value to eliminate abnormal fluctuations, and a regular square wave pulse signal is generated based on the optimized width value.

[0147] Step 504: inputting the square wave pulse signal to a light emitting device to drive the light emitting device to generate a light pulse signal.

[0148] In step 504, the light emitting device refers to a light emitting device that converts electrical signals into optical signals.

[0149] In the embodiment of the present application, the generated square wave pulse signal is input to a laser driver to drive the light emitting device to generate a light pulse signal synchronized with the electrical pulse signal, thereby completing the electro-optical conversion.

[0150] In the embodiment of the present application, the combination of deep learning and signal processing technology realizes high-precision conversion of analog electrical signals to optical pulse signals, improves the accuracy and stability of pulse width control, and thereby provides a reliable signal basis for subsequent data processing.

[0151] In order to solve the problem of standardizing the characteristics of industrial equipment monitoring data, in some embodiments, step 102: performing feature extraction processing on each data block to generate working condition feature data, comprising:

[0152] Step 601: performing first processing on each data block to obtain the arithmetic mean value of all sampling points in the data block, and obtaining mean value feature data based on the arithmetic mean value.

[0153] In step 601, the arithmetic mean value is a value obtained by dividing the sum of the values of all sampling points in the data block by the number of sampling points, and the mean feature data is data representing the average level of the data block obtained by arithmetic mean calculation.

[0154] In the embodiment of the present application, the sum of the values of all sampling points in the data block is first calculated, and then the sum is divided by the total number of sampling points to obtain the mean feature data reflecting the overall level of the data block.

[0155] In step 602, the average of the sum of squares of deviations is a value obtained by dividing the sum of squares of deviations of each sampling point from the mean value by the number of sampling points, and the variance feature data is data representing the fluctuation degree of the data obtained by calculating the average of the sum of squares of deviations.

[0156] In step 602, the average of the sum of squares of deviations is a value obtained by dividing the sum of squares of deviations of each sampling point from the mean value by the number of sampling points, and the variance feature data is data representing the fluctuation degree of the data obtained by calculating the average of the sum of squares of deviations.

[0157] In the embodiment of the present application, the difference between each sampling point and the mean feature data is first calculated, then the sum of squares of the difference is calculated, and then the sum is divided by the total number of sampling points to obtain the variance feature data reflecting the data fluctuation feature.

[0158] In step 603, the mean feature data and the variance feature data are combined in time sequence to form a preliminary feature sequence.

[0159] In step 603, the preliminary feature sequence is a feature data sequence formed by arranging the mean feature data and the variance feature data in time sequence, and the preliminary feature sequence can contain feature information of multiple data blocks.

[0160] In step 604, the preliminary feature sequence is standardized to generate working condition feature data.

[0161] In the embodiment of the present application, the preliminary feature sequence is subjected to numerical format uniform processing, and finally the working condition feature data with a standard format is generated, wherein the numerical format uniform processing includes data precision adjustment, unit unification and format standardization.

[0162] In the embodiment of the present application, the uniform and standardized working condition feature data is generated through statistical feature extraction and standardization processing of the system, which provides a reliable feature basis for subsequent data analysis and processing, thereby ensuring the consistency and comparability of the data.

[0163] In order to solve the hierarchical compression storage problem of industrial equipment monitoring data, in some embodiments, step 105: the combination of the working condition characteristic data, the encoding algorithm corresponding to the compression level is used to compress each data block, including:

[0164] Step 701: When the compression level is the non-compression level, the data block is encapsulated, and the header information containing the timestamp and the data block identifier is added in the encapsulation process to generate and store the complete data packet.

[0165] In step 701, the timestamp and the data block identifier are information for identifying data time and identity, and the complete data packet is a complete data unit containing original data and management information.

[0166] In the embodiments of the present application, the time information and the identifier information are added to the data block of the non-compression level to form a data packet containing complete data content and management information, and the data packet is directly stored.

[0167] Step 702: When the compression level is the first compression level, based on the working condition characteristic data, the highest similarity historical working condition characteristic data is found in the preset historical database, based on the historical data block corresponding to the historical working condition characteristic data, the difference value data of the current data block and the historical data block is calculated, and the adaptive encoding algorithm is used to perform first compression processing on the difference value data to obtain the corresponding compressed data block.

[0168] In step 702, the historical database is a database for storing historical working condition characteristic data, the highest similarity means the closest historical data to the current working condition characteristic data, the difference value data is the numerical difference of the corresponding sampling points of the current data block and the historical data block, and the first compression processing can mean lossless compression.

[0169] Step 703: When the compression level is the second compression level, the lossy compression algorithm is used to perform second compression processing on the data block to obtain the corresponding compressed data block.

[0170] In step 703, the second compression processing means that the data block is not compressed, the original data is directly retained and stored after adding the necessary header information; the first compression level means that after finding the similar historical data block based on the working condition characteristic data, only the difference part of the current data block and the historical data block is losslessly compressed, and the compression is realized on the premise of ensuring the data integrity; the second compression level means that the data block is directly lossy compressed, and higher compression ratio is realized by discarding part of the data details, but the data accuracy is lost.

[0171] In the embodiment of the present application, the data blocks of the secondary compression level are processed by using a lossy compression algorithm to reduce the data amount as much as possible under the premise of guaranteeing the main features, thereby realizing high-ratio compression.

[0172] In the embodiment of the present application, the intelligent management of data storage is realized by using the hierarchical compression strategy, the integrity of important data is guaranteed, the storage efficiency is improved, and an economic and efficient storage solution for long-term monitoring of industrial equipment is provided.

[0173] Figure 3 A structure diagram of a time window-based time series data compression hierarchical storage system provided in the embodiment of the present application is shown in the figure, and the system comprises:

[0174] The acquisition module 31 is configured to acquire the runtime series data of the industrial equipment.

[0175] The division module 32 is configured to divide the runtime series data of the industrial equipment into a plurality of data blocks according to a fixed time window, perform feature extraction processing on each data block to generate working condition feature data.

[0176] The analysis module 33 is configured to perform analysis processing on each data block to obtain the energy distribution features of each data block, perform pulse conversion processing on the energy distribution features, and generate an optical pulse code sequence based on the processing result by using a pulse width modulation technology.

[0177] The generation module 34 is configured to perform density analysis processing on the optical pulse code sequence to generate a density distribution map, and determine a compression level corresponding to the density distribution map.

[0178] The compression module 35 is configured to combine the working condition feature data, perform compression processing on each data block by using an encoding algorithm corresponding to the compression level, and store the compressed data blocks in different storage areas according to the compression level to complete hierarchical storage.

[0179] The time window-based time series data compression hierarchical storage system in the embodiment of the present application is used to realize the time window-based time series data compression hierarchical storage method described above, and therefore the specific embodiments of the time window-based time series data compression hierarchical storage system can be seen from the embodiment part of the time window-based time series data compression hierarchical storage method described above. The specific embodiments can be referred to the description of the corresponding embodiment part, and will not be described here.

[0180] The present application also provides an electronic device comprising a memory for storing a computer program and a processor for executing the computer program to realize the steps of the time window-based time series data compression hierarchical storage method described above.

[0181] The application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the time window based time series data compression hierarchical storage method.

[0182] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a read-only memory, a random access memory, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0183] The embodiments of the application further provide a computer program product, and the computer program product includes a computer program, and the computer program is executed by a processor to implement the steps in the time window based time series data compression hierarchical storage method embodiments.

[0184] The skilled person can further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0185] The above describes in detail the time window based time series data compression hierarchical storage method and system provided by the application. The principles and implementation manners of the application are described by using specific examples in this paper, and the above example description is only used to help understand the method of the application and its core idea. It should be pointed out that for ordinary skilled person in the art, some improvements and modifications can be made to the application without departing from the principles of the application, and these improvements and modifications also fall within the protection scope of the application.

Claims

1. A time-window-based method for compressed and hierarchical storage of time-series data, characterized in that, include: Collect runtime sequence data of industrial equipment; The running sequence data of the industrial equipment is divided into multiple data blocks according to a fixed time window, and feature extraction processing is performed on each data block to generate operating condition feature data. Each data block is analyzed and processed to obtain the energy distribution characteristics of each data block, and the energy distribution characteristics are subjected to pulse conversion processing. Based on the processing results, an optical pulse coded sequence is generated using pulse width modulation technology. The optical pulse coding sequence is subjected to density analysis to generate a density distribution map, and the compression level corresponding to the density distribution map is determined. Based on the aforementioned operating condition characteristic data, the encoding algorithm corresponding to the compression level is used to compress each data block, and the compressed data blocks are allocated to different storage areas according to the compression level to complete hierarchical storage. The process involves analyzing and processing each data block to obtain its energy distribution characteristics, performing pulse conversion on these characteristics, and generating an optical pulse-coded sequence based on the processing results using pulse width modulation (PWM) technology. This includes: The data block is integrated to obtain the sum of squared amplitudes of all sampling points within the data block. Based on the sum of squared amplitudes, the total energy value of the data block is obtained. Based on the total energy value, the sampling points within the data block are normalized to obtain the energy contribution of each sampling point; Based on the energy contribution, an energy distribution feature is generated, and the energy distribution feature of each sampling point is mapped to generate an analog electrical signal. Each analog electrical signal is converted into an optical pulse signal with a corresponding pulse width using pulse width modulation technology; the optical pulse signal is the processing result. All optical pulse signals are serialized in chronological order to generate optical pulse coded sequences; The process of converting each analog electrical signal into an optical pulse signal with a corresponding pulse width using pulse width modulation technology includes: The analog electrical signal is subjected to voltage sampling processing to obtain the corresponding voltage amplitude; According to the preset conversion curve, the voltage amplitude is mapped to the corresponding pulse width value; The pulse width value is optimized using a convolutional neural network, and a square wave pulse signal is generated based on the optimized pulse width value. The square wave pulse signal is input to the optical emitting device to drive the optical emitting device to generate an optical pulse signal.

2. The method according to claim 1, characterized in that, The step of performing density analysis on the optical pulse coded sequence to generate a density distribution map and determining the compression level corresponding to the density distribution map includes: The optical pulse coded sequence is time-segmented into multiple equal-length time segments; Perform statistical processing on each time period to determine the number of target pulses; A density distribution map is generated based on the number of target pulses in each time period. The density distribution map is divided into multiple density intervals. Based on a preset mapping relationship, the data blocks corresponding to each density interval are marked with the corresponding compression level.

3. The method according to claim 2, characterized in that, The generation of a density distribution map based on the number of target pulses in each time period includes: The number of target pulses in each time period is sorted to form a pulse time series; The pulse time series is averaged to obtain the average pulse density of each sliding window; The average pulse density of each sliding window is arranged in chronological order to form a density value sequence; The density value sequence is interpolated, and a density distribution curve is generated using a cubic spline interpolation algorithm; The density distribution curve is divided into multiple density intervals according to a preset density threshold. Each density interval is color-coded to generate a density distribution map.

4. The method according to claim 1, characterized in that, The step of performing feature extraction processing on each data block to generate operating condition feature data includes: Each data block undergoes a first processing step to obtain the arithmetic mean of all sampling points within the data block, and the mean feature data is obtained based on the arithmetic mean. A second processing is performed on each data block to obtain the average of the sum of squared deviations between each sampling point in the data block and the mean feature data. Based on the average of the sum of squared deviations, variance feature data is obtained. The mean feature data and the variance feature data are combined and processed in chronological order to form a preliminary feature sequence; The preliminary feature sequence is standardized to generate operating condition feature data.

5. The method according to claim 1, characterized in that, The step of combining the operating condition characteristic data and using the encoding algorithm corresponding to the compression level to compress each data block includes: When the compression level is no compression level, the data block is encapsulated. During the encapsulation process, header information containing timestamps and data block identifiers is added to generate and store a complete data packet. When the compression level is Level 1, based on the operating condition feature data, the historical operating condition feature data with the highest similarity is searched in the preset historical database. Based on the historical data block corresponding to the historical operating condition feature data, the difference data between the current data block and the historical data block is calculated, and the difference data is subjected to first compression processing using an adaptive encoding algorithm to obtain the corresponding compressed data block. When the compression level is level two, a lossy compression algorithm is used to perform a second compression process on the data block to obtain the corresponding compressed data block.

6. A time-window-based time-series data compression and hierarchical storage system, characterized in that, include: The acquisition module is used to collect runtime sequence data of industrial equipment; The segmentation module is used to divide the runtime sequence data of the industrial equipment into multiple data blocks according to a fixed time window, and to perform feature extraction processing on each data block to generate operating condition feature data. The analysis module is used to analyze and process each data block to obtain the energy distribution characteristics of each data block, and to perform pulse conversion processing on the energy distribution characteristics. Based on the processing results, an optical pulse coding sequence is generated using pulse width modulation technology. The generation module is used to perform density analysis processing on the optical pulse coding sequence, generate a density distribution map, and determine the compression level corresponding to the density distribution map. The compression module is used to combine the operating condition feature data and use the encoding algorithm corresponding to the compression level to compress each data block, and then allocate the compressed data blocks to different storage areas according to the compression level to complete hierarchical storage. The process involves analyzing and processing each data block to obtain its energy distribution characteristics, performing pulse conversion on these characteristics, and generating an optical pulse-coded sequence based on the processing results using pulse width modulation (PWM) technology. This includes: The data block is integrated to obtain the sum of squared amplitudes of all sampling points within the data block. Based on the sum of squared amplitudes, the total energy value of the data block is obtained. Based on the total energy value, the sampling points within the data block are normalized to obtain the energy contribution of each sampling point; Based on the energy contribution, an energy distribution feature is generated, and the energy distribution feature of each sampling point is mapped to generate an analog electrical signal. Each analog electrical signal is converted into an optical pulse signal with a corresponding pulse width using pulse width modulation technology; the optical pulse signal is the processing result. All optical pulse signals are serialized in chronological order to generate optical pulse coded sequences; The process of converting each analog electrical signal into an optical pulse signal with a corresponding pulse width using pulse width modulation technology includes: The analog electrical signal is subjected to voltage sampling processing to obtain the corresponding voltage amplitude; According to the preset conversion curve, the voltage amplitude is mapped to the corresponding pulse width value; The pulse width value is optimized using a convolutional neural network, and a square wave pulse signal is generated based on the optimized pulse width value. The square wave pulse signal is input to the optical emitting device to drive the optical emitting device to generate an optical pulse signal.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the time-window-based time-series data compression hierarchical storage method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the time-window-based time-series data compression and hierarchical storage method as described in any one of claims 1 to 5.

Images