Data compression method and device and computer equipment

Abstract

The embodiment of the invention is suitable for the technical field of data processing, and provides a data compression method and device and computer equipment. By applying the data compression method provided by the embodiment of the invention, the data can be compressed in a targeted manner according to different data modes by fully utilizing the characteristics of the data, so that the data compression efficiency is improved, the compression cost and the storage space required for storing the data are reduced, and the validity of the compressed data is ensured. Accordingly, the embodiment of the invention further provides a data compression quality assessment method, by designing a reasonable data quality assessment standard, the similarity between the data recovered after compression and the original data can be effectively assessed, and the data quality can be accurately judged.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and in particular to a data compression method, apparatus, and computer equipment. Background Art

[0002] Massive amounts of data contain a wealth of information, and storing all of it requires enormous amounts of storage space. Data compression can reduce this storage space. Data compression is a common data processing technique that reorganizes data without losing key information, reducing the amount of data and thus reducing storage space.

[0003] Based on the distortion of the compressed data, data compression techniques can be divided into lossy and lossless compression. Lossless compression technology can completely restore the data to its original state without losing any information. However, lossless compression has a high computational and storage cost. Lossy compression is more common in the existing data compression field. It reduces storage space by discarding redundant parts of the data. Although lossy compression loses some original data information, it is acceptable for overall data analysis and prediction. Compared with lossless compression, lossy compression has lower computational complexity and higher compression rate, achieving a better balance between reducing storage space and preserving data information.

[0004] In existing technologies, general compression algorithms for common text, program data, and other data types typically process these data indiscriminately using bytes as processing units. However, massive amounts of data encompass a wide variety of data types, and existing technologies are unable to fully utilize the data's characteristics for compression. For example, using existing algorithms to compress time series data, which exhibits characteristics such as high repetition, high mutation rates, and real-time data integration, is difficult to achieve effective compression. Summary of the Invention

[0005] The embodiments of the present application provide a data compression method, apparatus, and computer device that can fully utilize the characteristics of data, compress data in a targeted manner according to different data patterns, improve the efficiency of data compression, reduce compression costs and the storage space required to store data, and ensure the validity of the compressed data.

[0006] To achieve the above objectives, this application adopts the following technical solutions:

[0007] A first aspect of the present application provides a data compression method, comprising:

[0008] Collecting original data to be compressed, where the original data to be compressed is time series data, which is obtained by processing data values ​​of a target type in chronological order;

[0009] Based on the data value of the target type, identifying the pattern category of the time series data;

[0010] determining key data points corresponding to the pattern category, the pattern category having corresponding data storage rules;

[0011] The key data points and their corresponding data values ​​are stored according to the data storage rules to complete the compression of the time series data.

[0012] It should be understood that the data compression method provided by this application can be used to perform low-loss compression on time series data. Since time series data contains multiple features, this application can identify the pattern category of the collected time series data by analyzing and utilizing the pattern information contained in the time series data. In this way, key data points can be specifically determined based on different pattern categories. When performing data compression, only key data points and their corresponding data values ​​can be stored, thereby reducing data redundancy, reducing the storage space required to store data, and improving the efficiency of data compression and storage.

[0013] In a possible implementation of the first aspect of the present application, identifying the pattern category of the time series data based on the data value of the target type includes: dividing the time series data into multiple subsequences; and determining the pattern category of the time series data based on the change of the data value of the target type in each of the subsequences obtained by the division over time.

[0014] In another possible implementation of the first aspect of the present application, identifying the pattern category of the time series data based on the data value of the target type includes: using a trained neural network to identify the data value of the target type and outputting the pattern category of the time series data.

[0015] The neural network can be obtained by training a sample data set, wherein the sample data set includes labeled supervised data and unlabeled unsupervised data. The unsupervised data can be used to pre-train the encoder. During pre-training, a deep neural network with dilated causal convolution can be used as an encoder, and the encoder obtained by pre-training can be used to output the feature value of each subsequence obtained by dividing the time series data in the form of a sliding window. The supervised data in the sample data set can be used to train a classifier, and by using the feature value of each subsequence as the input data of the classifier, the pattern category of the time series data can be directly output by the neural network.

[0016] In a possible implementation of the first aspect of the present application, the data storage rules may include the number of the key data points to be stored and the location information of each key data point, and the key data points and their corresponding data values ​​are stored according to the data storage rules to complete the compression of the time series data, including: obtaining the data value of the key data point at the corresponding position according to the location information of each key data point; writing the data value of the key data point into a data structure with a preset data format; and storing the data structure with the data value written.

[0017] It should be understood that key data points are data that need to be retained during data compression, while non-key data points are redundant parts of the data. By discarding non-key data points during data compression, data storage space can be reduced.

[0018] In a possible implementation of the first aspect of the present application, the data format may include the following information:

[0019] Pattern category, start time, duration, start data value, end data value, average value, maximum data value, and minimum data value.

[0020] In a possible implementation of the first aspect of the present application, the timing data may include network data, the data value of the target type may include the signal strength and MAC address at different times in the network data, and the mode category may include a roaming mode, a frequency switching mode, a mobile mode or a stable mode determined based on the signal strength and the MAC address.

[0021] The roaming mode may include an early roaming mode or a late roaming mode, and the stable mode may include a strong signal stable mode, a weak signal stable mode or a leaving mode.

[0022] In a possible implementation of the first aspect of the present application, the method further includes: obtaining the stored key data points and their corresponding data values; and performing interpolation based on the key data points and their corresponding data values ​​to restore the original data.

[0023] A second aspect of the present application provides a method for evaluating data compression quality, comprising:

[0024] Obtaining an original data sequence and a data sequence to be evaluated, wherein the data sequence to be evaluated is a data sequence restored after data compression of the original data sequence, and the original data sequence and the data sequence to be evaluated have the same length;

[0025] Dividing the original data sequence and the data sequence to be evaluated into a plurality of data segments respectively;

[0026] Determine the difference between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated;

[0027] The data compression quality of the original data sequence is evaluated based on the difference value.

[0028] The data compression quality evaluation method provided in this application is based on the following main idea: first, two data sequences are segmented according to the increase or decrease trend of the data, and then a plurality of evaluation factors are designed to evaluate the difference between the two data sequences. For example, the evaluation factors may include various factors for evaluating the overall distribution interval, the degree of fluctuation, and the difference in the overall trend. On this basis, a weighted sum is taken based on the ratio of the length of each segmented interval to the total length of the sequence. Finally, the number of key data points that were not successfully restored after the data was restored is counted, and the total score is deducted as appropriate to calculate the loss value used to evaluate the data compression quality.

[0029] The data compression quality assessment method provided in this application, through the design of a reasonable time series data quality assessment standard, can accurately evaluate the similarity between compressed and restored time series data and the original data. This method can effectively examine the similarity between signal trends and key data points, and is robust to noise.

[0030] In a possible implementation of the second aspect of the present application, dividing the original data sequence and the data sequence to be evaluated into multiple data segments respectively includes: determining the data increase or decrease trends of the original data sequence and the data sequence to be evaluated; segmenting the original data sequence and the data sequence to be evaluated according to the data increase or decrease trends to obtain multiple data segments; wherein the data increase or decrease trend in any of the data segments remains unchanged.

[0031] In a possible implementation of the second aspect of the present application, determining the difference value between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated includes: determining multiple evaluation factors, the multiple evaluation factors including an overall distribution interval difference factor, a fluctuation severity difference factor, and an overall trend difference factor; and calculating the difference value between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated based on the multiple evaluation factors.

[0032]

[0033] Here, x represents the original data sequence, y represents the data sequence to be evaluated,

[0034]

[0035] In a possible implementation of the second aspect of the present application, the data compression quality of the original data sequence is evaluated based on the difference value, including: calculating a data sequence score value based on the difference value corresponding to each of the data segments; counting the number of key data points in the data sequence to be evaluated that have not been successfully recovered; subtracting the data sequence score value based on the number of key data points that have not been successfully recovered to obtain a total score for the data sequence; and calculating a loss value for evaluating the data compression quality based on the total score of the data sequence.

[0036] Generally, the smaller the loss value is, the smaller the difference between the data obtained after the compressed data is restored and the original data is, the less data information is lost during the compression process, and the quality of the corresponding data compression method is higher.

[0037] In a possible implementation of the second aspect of the present application, the data sequence score value is calculated based on the difference values ​​corresponding to each of the data segments, including: performing weighted summation on the difference values ​​corresponding to each of the data segments to obtain the data sequence score value; wherein the weight value of each of the data segments is the ratio of the length of the corresponding data segment to the total length of the data sequence.

[0038] In a possible implementation of the second aspect of the present application, the data sequence score value is subtracted based on the number of the key data points that have not been successfully recovered to obtain a total score for the data sequence, including: calculating the ratio between the number of the key data points that have not been successfully recovered and the total length of the data sequence; and calculating the data sequence score value minus the product of the ratio and a constant value to obtain the total score for the data sequence.

[0039] Because the number of key data points that were not successfully recovered affects the evaluation of data compression quality, after calculating the number of key data points that were not successfully recovered, the data sequence score can be subtracted based on this number to obtain the data sequence's overall score. Generally, the more key data points that were not successfully recovered, the more points need to be subtracted, and the lower the final data sequence score. Conversely, the fewer key data points that were not successfully recovered, the fewer points need to be subtracted, and the higher the final data sequence score.

[0040] For example, the formula for calculating the total score of a data sequence can be expressed as:

[0041]

[0042] Where x represents the original data sequence and y represents the data sequence to be evaluated. mis the total number of intervals into which the two data sequences are divided, N is the number of key data points that have not been successfully recovered, L is the total length of the data sequence, and w i is the weight value of the i-th segment of data, which is equal to the ratio of the length of the i-th segment of data to the total length of the data sequence. α, β, γ and λ are all constants.

[0043] As an example of the present application, α=1, β=1, γ=8, and λ=0.2 may be taken.

[0044] In a possible implementation of the second aspect of the present application, the counting of the number of key data points that were not successfully recovered after data recovery of the data sequence to be evaluated includes: counting the number of data points in the original data sequence whose data values ​​are less than a key threshold but whose corresponding data values ​​in the data sequence to be evaluated are greater than or equal to the key threshold; or, counting the number of data points in the original data sequence whose data values ​​are greater than or equal to the key threshold but whose corresponding data values ​​in the data sequence to be evaluated are less than the key threshold.

[0045] A third aspect of the present application provides a data compression device, comprising:

[0046] A time series data acquisition module is used to acquire the original data to be compressed, where the original data to be compressed is time series data, which is obtained by processing data values ​​of the target type in chronological order;

[0047] a pattern category identification module, configured to identify the pattern category of the time series data based on the data value of the target type;

[0048] A key data point determination module, configured to determine key data points corresponding to the pattern category, wherein the pattern category has corresponding data storage rules;

[0049] The data compression module is used to store the key data points and their corresponding data values ​​according to the data storage rules to complete the compression of the time series data.

[0050] A fourth aspect of the present application provides a device for evaluating data compression quality, comprising:

[0051] A data sequence acquisition module is configured to acquire an original data sequence and a data sequence to be evaluated, wherein the data sequence to be evaluated is a data sequence recovered after data compression of the original data sequence, and the length of the original data sequence is equal to that of the data sequence to be evaluated;

[0052] A data segment division module, configured to divide the original data sequence and the data sequence to be evaluated into a plurality of data segments respectively;

[0053] a difference value determination module, configured to determine a difference value between each data segment in the original data sequence and a corresponding data segment in the data sequence to be evaluated;

[0054] A data compression quality evaluation module is used to evaluate the data compression quality of the original data sequence based on the difference value.

[0055] A fifth aspect of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, can implement the method described in any one of the first and second aspects above.

[0056] A sixth aspect of the present application provides a computer-readable storage medium, the computer-readable storage medium comprising a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the method described in any one of the first and second aspects above.

[0057] A seventh aspect of the present application provides a chip comprising a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the data compression method described in any one of the first and second aspects above. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a schematic diagram of a data compression process provided by an embodiment of the present application;

[0059] Figure 2 This is a schematic diagram of an application scenario to which the data compression method provided in an embodiment of the present application is applicable;

[0060] Figure 3 This is a schematic diagram comparing real data and fitted data provided in an embodiment of the present application;

[0061] Figure 4 This is another schematic diagram comparing real data and fitted data provided in an embodiment of the present application;

[0062] Figure 5 is a schematic diagram of a data compression method provided in an embodiment of the present application;

[0063] Figure 6 is a schematic diagram of a method for evaluating data compression quality provided in an embodiment of the present application;

[0064] Figure 7 This is a structural block diagram of a data compression device provided in an embodiment of the present application;

[0065] Figure 8This is a structural block diagram of a data compression quality evaluation device provided in an embodiment of the present application;

[0066] Figure 9 This is a structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] To facilitate a clear description of the technical solutions of the embodiments of the present application, terms such as "first" and "second" are used in the embodiments of the present application to distinguish between identical or similar items having substantially the same functions and effects. For example, the terms "first key data point," "second key data point," and so on are merely used to distinguish different data points in a data sequence and do not constitute any other limitation.

[0068] It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0069] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field will know that with the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0070] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0071] The steps involved in a data compression method or data compression quality evaluation provided in the embodiments of the present application are merely examples. Not all steps are required, nor are all the contents of each step optional. These steps may be added or removed as needed during use. The same step or steps or contents with the same function in different embodiments of the present application may be used as a reference for reference.

[0072] Storing massive amounts of data requires significant storage space, and processing this massive amount of data is also cumbersome. For example, when a terminal or device fails, accessing all network data for analysis is difficult, making fault detection time-consuming and costly. Data compression allows data to be reorganized without losing key information, reducing the data volume and thus reducing storage space.

[0073] Generally speaking, data compression needs to meet the following requirements: first, reduce data storage space and processing costs; second, the compression operation does not lose or loses less valuable information in the data, retains the local and global characteristics of the data as much as possible, and can effectively restore or approximately restore the original data after compression; third, reduce the computational complexity of the compression algorithm and increase the speed of data compression and decompression.

[0074] As mentioned above, data compression techniques can be categorized as lossy or lossless based on the distortion of the compressed data. Lossless compression techniques completely restore the original data without losing any information and are commonly used for archiving or other high-fidelity purposes. They ensure complete data recovery while reducing data storage space. Lossless compression techniques primarily include run-length encoding, dictionary encoding, and entropy encoding, and are well-suited for data with distinct patterns, such as text files, database tables, and medical images. However, lossless compression carries significant computational and storage costs. Lossy compression is more common in the current data compression field. It reduces storage space by discarding redundant portions of the data. While some original data information is lost, this is acceptable for overall data analysis and prediction. Common lossy compression techniques include transform-based methods, prediction-based methods, vector quantization, fractal compression, compressed autoencoder-based methods, and deep learning-based methods. Compared to lossless compression, lossy compression offers lower computational complexity and higher compression ratios, achieving a better balance between reducing storage space and preserving data information.

[0075] In the existing technology, for some ordinary text, program and other data, general compression algorithms can ignore the characteristics of the data and directly process it indiscriminately using bytes as processing units during compression. However, some types of data contain their own unique characteristics, which can express rich intrinsic information. Taking network data as an example, massive network data contains multiple features, and these features can represent unique pattern information in the time series. For example, severe jitter in network data often means that the original data is abnormal and may contain information such as pattern conversion and sampling errors. The existing technology does not fully utilize the unique pattern information of the time series when compressing network data, and the compression effect is poor.

[0076] On the other hand, in order to judge the quality of data compression technology, it is also necessary to evaluate the quality of the compressed and restored data. In the existing technology, the indicators used to evaluate the quality of data compression include mean square error (MSE), normalized mean absolute error (NMAE) and cosine error (COS). i Represents the original data, Represents the data recovered after compression, and the calculation formula of the mean square error is:

[0077]

[0078] The smaller the MSE value, the smaller the error.

[0079] The formula for calculating the standard mean absolute error is:

[0080]

[0081] Similarly, a smaller NMAE value indicates a smaller error.

[0082] The formula for calculating cosine error is:

[0083]

[0084] For cosine error, the closer the COS value is to 1, the smaller the error is.

[0085] However, the above-mentioned evaluation indicators are not able to effectively evaluate the data compression quality in all fields. Taking network data as an example, there are three obvious shortcomings in using mean square error to evaluate data compression quality. First, when calculating the difference in signal values ​​at each position, the result is only related to the signal value at the current position and has nothing to do with the signal value at any other position. This evaluation method only regards the data as isolated signal values ​​and ignores the changing trend of the signal. Second, the mean square error is too sensitive to data points with large gaps between the original data and the data recovered after compression. The large gap at a certain point has an obvious impact on the evaluation result, causing the MSE value to rise rapidly. Third, the result obtained by the mean square error is only relative, not absolute, and its value cannot intuitively express the quality of the data. Therefore, the mean square error is not suitable for quality evaluation of network data compression.

[0086] Compared to mean square error (MSE), the standard mean absolute error (SAE) and cosine error (SME) take into account the original signal value in the network data. Specifically, the SMEE considers not only the difference between the original signal and the signal to be evaluated, but also the ratio of this difference to the original signal value. However, the SMEE and cosine error still suffer from two issues: ignoring the changing trend of the signal and causing a sudden change in the signal value at a single point to cause a sudden change in the overall evaluation result. Therefore, the SMEE and cosine error are also unsuitable for evaluating the compression quality of network data.

[0087] Furthermore, in some business scenarios, applying existing methods to assess data compression quality is prone to false negatives. For example, if a terminal or device has already failed, applying existing methods to assess network data quality may result in the conclusion that the terminal or device is operating normally, adversely affecting subsequent troubleshooting and other operations.

[0088] In response to the above problems, the embodiments of the present application provide a data compression method and a method for evaluating the quality of data compression. Among them, the data compression method provided by the embodiments of the present application can perform pattern mining based on the multiple features contained in the data, identify the pattern category to which the data belongs, and perform targeted compression based on the data of different pattern categories. The data compression method of the embodiment of the present application can be applied to time series data, such as network data. Among them, the network data can include the signal strength of the terminal or device at each time point, and the waveform formed by the signal strength of multiple consecutive time points is a type of time series data. The data compression method of the embodiment of the present application can perform pattern mining on such time series data, identify key data points, and achieve low-loss data compression. After testing, the data compression method of the embodiment of the present application can achieve a compression rate of more than 3 times.

[0089] On the other hand, by designing reasonable data quality assessment criteria, the data compression quality assessment method provided by the embodiments of the present application can evaluate the degree of similarity between the compressed and recovered data and the original data, accurately judging the quality of the data. Specifically, the data compression quality assessment method provided by the present application can be applied to the quality assessment of compressed and recovered time series data, for example, the network data in the above example.

[0090] Below, the data compression method and the data compression quality evaluation method provided in the embodiments of the present application are respectively introduced in detail. Among them, the data mentioned in the embodiments of the present application can be time series data. For ease of understanding, the following text uses network data as an example to introduce the method provided in the embodiments of the present application. For example, network data such as the signal strength perceived by the terminal or device that changes over time. It should be understood that the method provided in the embodiments of the present application can also be applied to other types of time series data. For example, data types such as the power consumption and latency of the central processing unit (CPU) that change over time.

[0091] The data compression method and data compression quality evaluation method provided in the embodiments of the present application can be executed by a computer device, that is, the above method is implemented by a computer device. In one example, the computer device can be a desktop computer, a cloud server, or other device. The embodiments of the present application do not limit the specific type of computer device.

[0092] (1) Data compression method

[0093] After collecting raw data, embodiments of the present application can perform pattern mining on the raw data based on a certain type of data value in the raw data to identify the pattern category of the raw data. For example, the raw data can be collected network data, which can include the signal strength of the terminal at each moment. By analyzing the changes in signal strength, the pattern category of the network data can be determined.

[0094] In one possible implementation of an embodiment of the present application, the mode category of network data may include roaming mode, frequency switching mode, mobile mode, or stable mode. For other types of time series data, the mode category can be determined based on the specific service. Each mode category can have corresponding data storage rules. When performing data compression, the data can be stored according to the above data storage rules to complete the data compression.

[0095] Specifically, after determining the pattern category of the network data, the key data points corresponding to the pattern category can be further determined. The above-mentioned key data points can be some data points that reflect the characteristics of the corresponding pattern category. For example, the first key data point and the second key data point. Among them, the first key data point can be the data point corresponding to the maximum value in the continuous interval, the second key data point can be the data point corresponding to the minimum value, and so on. Then, these key data points and their corresponding data values ​​are stored according to the storage rules of the current pattern category. Accordingly, the data of non-key data points is discarded. When the data is needed later, the original time series data can be restored based on the key data points.

[0096] In one possible implementation of the present application, an unsupervised method for learning universal embeddings of time series can be used to learn the pattern features contained in time series data. Specifically, a neural network can be trained in combination with network features to perform pattern matching on high-frequency collected network data, identify key pattern data, and upload it. On the other hand, the pattern category information obtained by pattern recognition and the saved key data points can be used to decode the compressed data using compressed data recovery rules to restore the original network data.

[0097] like Figure 1 The above is a schematic diagram of a data compression process provided by an embodiment of the present application. Figure 1 In the data compression process shown, sample data is first used to train a deep neural network that can classify the patterns in time series data. The deep neural network then performs pattern recognition on the collected time series data. Key data points are then identified based on the recognized pattern categories and their values ​​are stored. The compressed time series data can be restored using the compressed data recovery rules to obtain the original time series data. Figure 1 The data compression process shown in mainly includes data labeling, pattern recognition and data recovery stages, which are introduced below.

[0098] (1) Data labeling

[0099] Before data labeling, time series data can be collected first. The collected time series data can be used as sample data for subsequent training of deep neural networks. Figure 1The timing data in may be network data. For network data, the corresponding test tool may be used to monitor the wireless local area network in the environment where the terminal is located to obtain the signal strength value at each moment. The timing data in the embodiment of the present application may include a waveform data, which may be a signal strength value arranged in chronological order by processing the signal strength value at each of the above moments. In one example, the above signal strength value may be a dynamic signal strength value or an RSSI signal strength value. In addition, when monitoring the wireless local area network, the test tool may also obtain the current MAC address. The above signal strength value and MAC address both belong to timing data.

[0100] like Figure 1 As shown, after the time series data is collected, the time series data can be preprocessed. Exemplarily, the preprocessing may include operations such as data cleaning, timestamp conversion, and sliding window division, which are not limited in the present embodiment.

[0101] In the embodiments of this application, Figure 1 The pattern division and labeling shown in the figure can be configured based on business needs. The pattern categories of time series data can vary in different business scenarios. Taking network data as an example, by observing changes in MAC addresses and trends in signal strength, network data can be assigned pattern categories such as roaming, frequency switching, mobile, stable, weak signal, and leaving using a sliding window. Table 1 shows the multiple pattern categories obtained by dividing network data patterns and the classification criteria for each pattern category.

[0102] Table 1. Examples of network data model categories and model standards:

[0103]

[0104] By classifying the collected network data according to the aforementioned pattern criteria, we can determine the pattern categories of this data. These pattern categories can then be used as labels for this data during subsequent training. This results in a dataset containing both labeled supervised data and unlabeled unsupervised data. Together, these labeled supervised data and unlabeled unsupervised data constitute the sample dataset for training deep neural networks.

[0105] (2) Pattern recognition

[0106] In the embodiments of this application, Figure 1 The pattern recognition phase shown in Figure 1 can be further divided into a training phase and an application phase. The training phase is the phase where the deep neural network is trained using the sample dataset obtained above, and the application phase is the phase where the trained deep neural network is used to perform pattern recognition on the collected unlabeled time series data.

[0107] (2.1) Training phase

[0108] In an embodiment of the present application, a deep neural network with dilated causal convolution can be used as an encoder, and a large amount of unlabeled data can be used to perform representation learning based on unsupervised thinking to pre-train the encoder and extract the feature vector of the time series data as the input information of the classifier. Among them, the deep neural network with dilated causal convolution is a type of neural network. In the deep neural network with dilated causal convolution, the information of each neuron comes from the neurons in the previous period of time. The use of network information grows exponentially, and it can contain more global and higher semantic features.

[0109] On the other hand, labeled data can be used to train an SVM classifier. The feature vector obtained by the encoder can be used as the input information of the classifier, and then the SVM is trained using the label information to perform classification, and the final pattern recognition classification result can be obtained.

[0110] (2.2) Application stage

[0111] After completing the learning and training of a deep neural network, it can be applied to actual business scenarios to perform pattern recognition on collected unlabeled time series data. This process is also the application stage of the deep neural network.

[0112] In a possible implementation of an embodiment of the present application, during the application stage, the trained deep neural network can divide the collected time series data without label information into multiple time series in the form of a sliding window, and then identify the pattern categories of the time series data, save key data points according to the characteristics of different patterns, and report them.

[0113] As shown in Table 2, the pattern characteristics of each pattern category in Table 1 and the storage rules of the corresponding key data points are listed. After identifying the pattern category of the network data, the key data points can be determined and the corresponding data values ​​can be stored according to the storage shown in Table 2. Among them, the roaming mode in Table 1 can be further divided into the early stage and the late stage. The early stage of roaming can represent the stage where the terminal starts roaming from the current MAC address to another MAC address; the late stage of roaming can represent the stage where the terminal roams from another MAC address to the current MAC address. After accessing the current MAC address, the terminal may end the roaming mode.

[0114] Table 2. Characteristics of each network data model category and examples of key data point storage rules:

[0115]

[0116] like Figure 1As shown, after using a deep neural network to identify the pattern category of time series data, the key data points that need to be stored can be determined according to the storage rules shown in Table 2, and the key data points and their corresponding data values ​​can be stored according to the rules, that is, the data is reported to the analyzer. For example, in the early stage of roaming, the storage rules for key data points can include a label and two key data points: the maximum value and the minimum value. Among them, the label can be used to record the pattern category of the current time series data.

[0117] In one possible implementation of an embodiment of the present application, the data format of the key data points reported to the analyzer may include the following information in Table 3, namely: pattern category, start time, duration, start data value, end data value, average value, maximum data value and minimum data value.

[0118] Table 3, Example of the data format for reporting data:

[0119]

[0120] The above application phase can also be considered as the phase of compressing time series data. For the compressed time series data, data recovery can be performed according to the corresponding rules when needed.

[0121] (2.3) Data recovery

[0122] In an embodiment of the present application, the pattern category information of the time series data obtained by pattern recognition and the saved key data points can be used to decode the compressed data using the compressed data recovery rules to restore the original time series data.

[0123] As a specific application of the embodiment of this application, Figure 2 , which is a schematic diagram of an application scenario applicable to the data compression method provided in an embodiment of the present application. Figure 2 The application scenario shown can be a data signal analysis scenario of a terminal device. Figure 2 In the scenario shown, this method can be applied to compress the data of the access point and report it to the analyzer, and the communication fault can be analyzed based on the corresponding compressed data.

[0124] In the existing technology, the fault analysis of network data is based on the data reported in minutes. Due to the limitation of data size and processing method, the existing technology cannot achieve faster and more accurate fault perception. Figure 2 As shown, when data from the network element is collected in seconds, it can not only be aggregated and reported in minutes, but also accurately perceive faults in seconds when they occur, thereby improving data transmission and storage capabilities.

[0125] (2) Evaluation Methods for Data Compression Quality

[0126] The main idea of ​​the data compression quality evaluation provided in the embodiment of the present application is to first segment the two data sequences according to the increase or decrease trend of the data, and then design multiple evaluation factors to evaluate the overall distribution range, fluctuation severity, and overall trend difference of the two data sequences respectively. Then, a weighted sum is taken according to the ratio of the length of each segmented interval to the total length of the sequence. Finally, the number of key data points that were not successfully restored after the data was recovered is counted, and the total score is deducted as appropriate to calculate the loss value used to evaluate the data compression quality.

[0127] In a possible implementation of the embodiment of the present application, the formula for calculating the total score of the data sequence can be expressed as:

[0128]

[0129] Where x represents the original data sequence and y represents the data sequence to be evaluated. m is the total number of intervals into which the two data sequences are divided, N is the number of key data points that have not been successfully recovered, L is the total length of the data sequence, and w i is the weight value of the i-th data segment, which is equal to the ratio of the length of the i-th data segment to the total length of the data sequence. α, β, γ, and λ are all constants. As an example of an embodiment of the present application, α=1, β=1, γ=8, and λ=0.2 can be taken.

[0130] also, Indicates the overall distribution interval difference, Indicates the difference in the intensity of fluctuations. Indicates the overall trend difference (k x and k y are the slopes of the two data series in this series respectively). Here,

[0131] In the embodiment of the present application, the process of evaluating the compression quality of a time series data sequence in the above manner and calculating the loss value of the fitted data may mainly include the following steps:

[0132] (1) Segment two data sequences of the same length.

[0133] Among them, the two data sequences are the original data sequence and the data sequence to be evaluated. The original data sequence is a piece of collected original data. After the original data is compressed according to the data compression method provided in the aforementioned embodiment, the data sequence obtained by restoring the compressed data is the data sequence to be evaluated.

[0134] When segmenting an original data sequence and a data sequence to be evaluated that are of the same length, the criterion for maintaining the data increment or decrement within each segmented interval can be used. That is, the data within each segmented interval obtained by segmenting the original data sequence and the data sequence to be evaluated is monotonically increasing or monotonically decreasing.

[0135] (2) Score the two signals in each interval.

[0136] The following formula can be used to calculate the scores of the two signals in each interval:

[0137] score i (x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ ……(5)

[0138] The meaning of each parameter in the above formula (5) can be found in the above introduction to formula (4).

[0139] (3) Calculate the ratio of each interval to the total length of the sequence as the weight of the score corresponding to each interval, and perform weighted summation of the scores of each interval.

[0140] (4) Count the number of key data points that were not successfully restored after the data was restored.

[0141] The unrecovered key data points mentioned above may refer to data points in the original data sequence whose data value is less than a key threshold, but whose corresponding data value in the data sequence to be evaluated is not less than the key threshold; or data points in the original data sequence whose data value is not less than the key threshold, but whose corresponding data value in the data sequence to be evaluated is less than the key threshold. By counting these unrecovered key data points, the number N of unrecovered key data points can be obtained.

[0142] The above critical threshold can be determined according to specific services. For example, for the signal strength in network data, -65dBm can be set as the critical threshold, which can be used to indicate the quality of the signal strength.

[0143] (5) According to the number N of key data points that have not been restored, the score value is corrected.

[0144] In an embodiment of the present application, the sum of the weighted scores of all intervals after two data sequences are divided into multiple intervals minus the product of a constant and the ratio of the number N of key data points that have not been restored to the total length of the data sequence can be used as the total score of the data sequence. That is, the above formula (4) can be used to calculate the total score of the data sequence.

[0145] (6) Calculate the final loss value.

[0146] In the embodiment of the present application, the loss value of the data sequence can be calculated using the following formula:

[0147] Loss=1-socre……(6)

[0148] In this way, after the above steps, the loss value of the data sequence to be evaluated can be calculated. Generally, the smaller the loss value, the smaller the difference between the recovered data and the original data, the less data information is lost during the compression process, and the higher the quality of the corresponding data compression method.

[0149] As a verification of the embodiment of this application, Figure 3 , which shows a schematic diagram comparing real data and fitted data provided in an embodiment of the present application. Figure 3 The results of the evaluation of the data increase and decrease trends are shown. Figure 3 The horizontal axes shown in (a) and (b) represent different moments, and the vertical axes represent data values ​​at each moment, that is, the signal strength values ​​in the network data corresponding to each moment. Figure 3 In (a) and (b), the solid straight lines are fitting data, which are used to represent the data obtained after the compressed data is restored; the dotted lines are real data, which are used to represent the original data. Figure 3 Specifically, the loss values ​​calculated by applying two different compressed data quality assessment methods are shown, where Figure 3 (a) is the loss value obtained by applying the compressed data quality assessment method provided in the embodiment of the present application, and Figure 3 (b) in the figure is the loss value obtained by applying the NMAE evaluation method. Figure 3 It can be seen from (a) and (b) that the loss value obtained by the evaluation method provided in the embodiment of the present application is considered to be Figure 3 The quality of the fitted data in (a) is better, and the NMAE evaluation shows that Figure 3 The quality of the fitted data in (b) is higher. Figure 3 visible, Figure 3 The trend of the fitted data in (a) is more consistent with the real data, which also shows that the evaluation method provided in the embodiment of the present application is more accurate in grasping the trend of data increase and decrease.

[0150] As another verification of the embodiment of the present application, Figure 4 , which shows another schematic diagram of comparison between real data and fitting data provided in an embodiment of the present application. Figure 4 The results of the assessment of the grasp of key data points are shown. Figure 4 In the example shown, the critical threshold is -65dBm. Figure 4 As can be seen from (a) and (b), the first half of the real data is less than -65dbm. Figure 4 The fitting data of (a) in the horizontal axis range 0 to 12 is also less than -65dbm, and Figure 4 The fitted data of (b) is greater than -65dbm in the range of 0 to 12 on the horizontal axis. Figure 4 The quality of the fitting data in (a) and (b) is the same, but the evaluation method provided in the embodiment of the present application believes that Figure 4 The quality of the data fit in (a) is better. Figure 4 visible, Figure 4 The fitting data and real data in (a) have a more accurate grasp of the key data points, which also shows that the evaluation method provided in the embodiment of the present application can be used to detect key data points.

[0151] Combined with the introduction of the above embodiments, Figure 5 As shown, a schematic diagram of a data compression method provided by an embodiment of the present application is shown. The method can be applied to a computer device, that is, the execution subject of the method can be a computer device or a corresponding module in the computer device. By executing this method, the computer device can achieve fast and efficient compression of time series data. Figure 5 As shown, the method may include the following steps S501-S504:

[0152] S501 : Collect original data to be compressed, where the original data to be compressed is time series data, which is obtained by processing data values ​​of a target type in chronological order.

[0153] In the embodiment of the present application, the original data to be compressed collected by the computer device may be time series data to be compressed. Different business scenarios have different time series data.

[0154] Time series data, also known as time series data, refers to a data sequence recorded in sequence at different time points for the same indicator. Each data point is associated with a specific time point, which can be a period number or a time point number, and is required to be comparable. The time series data in the embodiment of the present application can be waveform data obtained by processing the data value of a certain target type in chronological order. Exemplarily, in the field of communication or network technology, the time series data can be network data, and the data value of the target type in the network data can be a signal strength value. There are corresponding signal strength values ​​at different moments or time points, and the signal strength values ​​at each moment recorded in chronological order can form waveform-like time series data.

[0155] For ease of understanding, the embodiments of the present application are introduced using network data as an example of time series data.

[0156] S502. Identify the pattern category of the time series data based on the data value of the target type.

[0157] The data value of the target type in the network data can be the signal strength value in the above example, or it can be a MAC address and / or other data values, depending on the actual needs of data collection in the business scenario.

[0158] In one possible implementation of the embodiments of the present application, the mode category of the network data may be the mode category described in the aforementioned embodiments. For example, the roaming mode, frequency switching mode, mobile mode, or stable mode described in Table 1 of the aforementioned embodiments may be used. These mode categories may be determined based on the signal strength and MAC address at different times in the network data.

[0159] In combination with Table 1 and Table 2, the above roaming mode may include the early roaming mode or the late roaming mode; the stable mode in Table 1 may refer to the strong signal stable mode, and accordingly, the weak signal or leaving mode in Table 1 also belongs to the stable mode.

[0160] In a possible implementation of an embodiment of the present application, when a computer device identifies the pattern category of time series data based on the data value of the target type, the computer device can divide the time series data into multiple subsequences, and determine the pattern category of the time series data according to the change of the data value of the target type in each subsequence obtained by the division over time. The above division process can be performed by the computer device in the form of a sliding window to obtain multiple subsequences by sliding a window of a certain length from the starting moment. For example, assuming that each subsequence contains data values ​​of 5 moments, the time series data is divided starting from the starting moment T0, and the multiple subsequences obtained are a subsequence containing moments T0-T4, a subsequence containing moments T1-T5, a subsequence containing moments T2-T6, and so on.

[0161] In another possible implementation of the embodiment of the present application, the computer device identifies the pattern category of the time series data, which may be to use a trained neural network to identify the data value of the target type and output the pattern category of the time series data. The above-mentioned neural network may be obtained by training the deep neural network with dilated causal convolution in the aforementioned embodiments. That is, the deep neural network is obtained by training a sample data set, and the sample data set includes labeled supervised data and unlabeled unsupervised data. Among them, the unsupervised data is used to pre-train the encoder, and the supervised data is used to train the classifier. The encoder obtained by pre-training is used to output the feature value of each subsequence obtained by dividing the time series data in the form of a sliding window. By using the feature value of each subsequence as the input data of the classifier, the pattern category of the time series data can be output.

[0162] Therefore, for the collected network data, which is time series data, the pattern category can be directly identified using the trained deep neural network, and the deep neural network can directly output the pattern category of the collected network data. This process is the application stage of pattern recognition in the above embodiment.

[0163] S503: Determine key data points corresponding to the pattern category, where the pattern category has corresponding data storage rules.

[0164] The key data points of time series data of different mode categories may be different. For each mode category in Table 1, the corresponding key data points can be as shown in Table 2. For example, referring to Table 2, in the early stage of roaming mode, the key data points may include the maximum value data point and the minimum value data point.

[0165] The data storage rules corresponding to different mode categories may include the number of key data points to be stored and the location information of each key data point. The data storage rules of each mode category can be pre-configured according to the needs of specific business.

[0166] S504: Store the key data points and their corresponding data values ​​according to the data storage rule to complete the compression of the time series data.

[0167] In the embodiment of the present application, the key data points are the data that need to be retained when compressing the data, and the non-key data points are the redundant parts of the data. When compressing the data, the storage space of the data can be reduced by discarding the non-key data points.

[0168] In a possible implementation method of an embodiment of the present application, the data value of the key data point at the corresponding position can be obtained based on the position information of each key data point. By writing the data value of the key data point into a data structure with a preset data format and storing the data structure with the data value written, compression of the time series data can be achieved.

[0169] The above data format may include the following information in Table 3 of the above embodiment: mode category, start time, duration, start data value, end data value, average value, maximum data value and minimum data value.

[0170] After data compression is completed, the original data can be restored by obtaining the stored key data points and their corresponding data values ​​and performing interpolation based on the key data points and their corresponding data values.

[0171] By analyzing and utilizing the pattern information contained in the time series data, the embodiments of the present application can store key data points corresponding to different pattern categories, thereby reducing data redundancy and storage space and improving the efficiency of data compression and storage.

[0172] like Figure 6 As shown, a schematic diagram of a method for evaluating data compression quality provided by an embodiment of the present application is shown. The method can be applied to a computer device, that is, the execution subject of the method can be a computer device or a corresponding module in the computer device. The computer device can evaluate the quality of compressed data by executing the method. Figure 6 As shown, the method may include the following steps S601-S604:

[0173] S601 : Acquire an original data sequence and a data sequence to be evaluated, where the data sequence to be evaluated is obtained by compressing the original data sequence, and the original data sequence and the data sequence to be evaluated have the same length.

[0174] In the embodiments of the present application, the original data sequence and the data sequence to be evaluated may be two data sequences of equal length as in the aforementioned embodiments, wherein the data sequence to be evaluated is a data sequence recovered after data compression of the original data sequence. In one example, the data sequence to be evaluated may be processed using the aforementioned data compression method during the data compression phase.

[0175] S602: Divide the original data sequence and the data sequence to be evaluated into multiple data segments respectively.

[0176] In the embodiments of the present application, the original data sequence and the data sequence to be evaluated can be segmented based on the criterion that the data within each segmented interval remains unchanged. That is, the data within each interval obtained by segmenting the original data sequence and the data sequence to be evaluated is monotonically increasing or monotonically decreasing. Each interval is also referred to as a data segment.

[0177] Therefore, in one possible implementation of the present invention, the data increase or decrease trends of the original data sequence and the data sequence to be evaluated can be determined. The two data sequences are then segmented based on the data increase or decrease trends to obtain multiple data segments, or multiple intervals. The data increase or decrease trends within any data segment or interval remain unchanged. Within an interval, the data maintains a monotonically increasing or monotonically decreasing trend.

[0178] S603: Determine a difference value between each data segment in the original data sequence and a corresponding data segment in the data sequence to be evaluated.

[0179] In the embodiments of the present application, the difference between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated can be used to indicate the degree of similarity between the recovered data and the original data. Generally, the smaller the difference between the recovered data and the original data, the higher the compression quality and the less information lost during the data compression and recovery process. Conversely, the larger the difference between the recovered data and the original data, the worse the compression quality and the more information lost during the data compression and recovery process.

[0180] In an embodiment of the present application, the difference between each data segment in the original data and the corresponding data segment in the data sequence to be evaluated can be evaluated using multiple different evaluation factors. These multiple evaluation factors can include the overall distribution interval difference factor, the fluctuation intensity difference factor, the overall trend difference factor, and the like in the aforementioned embodiment. Thus, after determining the multiple evaluation factors, the difference value between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated can be calculated based on the multiple evaluation factors.

[0181] Specifically, you can use Indicates the overall distribution interval difference, using Indicates the difference in the intensity of fluctuations, using The meanings of the parameters in the above formulas can be found in the introduction of formula (4) in the above embodiment, and will not be repeated here.

[0182] S604: Evaluate the data compression quality of the original data sequence based on the difference value.

[0183] In the embodiments of the present application, the difference between each data segment in the original data and the corresponding data segment in the data sequence to be evaluated can be used to determine the difference between the data segment and the corresponding data segment. For example, the difference between the data segment and the corresponding data segment can be determined based on the maximum value of the difference between the data segments. Alternatively, the difference between the data segments can be processed, and the processed result can be used to represent the difference between the data sequence as a whole.

[0184] In a possible implementation of the embodiment of the present application, a data sequence score value may be calculated based on the difference values ​​corresponding to each data segment.

[0185] Specifically, the difference values ​​corresponding to each data segment can be weighted and summed to obtain the data sequence score value; wherein the weight value of each data segment is the ratio of the length of the corresponding data segment to the total length of the data sequence.

[0186] Then, the number of key data points in the data sequence to be evaluated that were not successfully recovered can be counted. The key data points that were not successfully recovered can refer to data points in the original data sequence whose data values ​​are less than a key threshold, and whose data values ​​at the corresponding points in the data sequence to be evaluated are not less than the key threshold; or data points in the original data sequence whose data values ​​are not less than the key threshold, and whose data values ​​at the corresponding points in the data sequence to be evaluated are less than the key threshold.

[0187] Therefore, we can count the number of data points in the original data sequence whose data values ​​are less than the critical threshold but whose corresponding data values ​​in the data sequence to be evaluated are greater than or equal to the critical threshold; or we can count the number of data points in the original data sequence whose data values ​​are greater than or equal to the critical threshold but whose corresponding data values ​​in the data sequence to be evaluated are less than the critical threshold. By counting these key data points, we can obtain the number N of key data points that were not successfully recovered.

[0188] Because the number of key data points that were not successfully recovered affects the evaluation of data compression quality, after calculating the number of key data points that were not successfully recovered, the data sequence score can be subtracted based on this number to obtain the data sequence's overall score. Generally, the more key data points that were not successfully recovered, the more points need to be subtracted, and the lower the final data sequence score. Conversely, the fewer key data points that were not successfully recovered, the fewer points need to be subtracted, and the higher the final data sequence score.

[0189] In one possible implementation of an embodiment of the present application, the ratio between the number of key data points that have not been successfully recovered and the total length of the data sequence can be calculated, and the total score of the data sequence can be obtained by calculating the data sequence score value minus the product of the ratio and a constant value.

[0190] Specifically, the total score of the data sequence is also the total score in the aforementioned embodiment. The calculation formula thereof can be referred to the aforementioned formula (4), which will not be repeated here.

[0191] In this way, a loss value, used to evaluate data compression quality, can be calculated based on the total score of the data sequence. This loss value is Loss = 1 - socre. Generally, a smaller loss value indicates a smaller difference between the recovered data and the original data, less data information is lost during the compression process, and a higher quality data compression method.

[0192] The data compression quality assessment method provided in this application embodiment accurately evaluates the similarity between compressed and restored time series data and the original data by designing appropriate quality assessment criteria for time series data. This method effectively examines signal trends and the similarity of key data points, and is robust to noise.

[0193] In the embodiment of the present application, the computer device can be divided into functional modules according to the above-mentioned method example. For example, one or more functions can be integrated into a functional module. For example, the various functions of the data compression method in the aforementioned embodiments can be implemented by a certain module in the computer device. Alternatively, each functional module can be divided corresponding to each function. For example, the various functions of the data compression method in the aforementioned embodiments can be implemented by multiple modules in the computer device. The division of modules in the embodiment of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The following is an example of dividing each functional module corresponding to each function.

[0194] Corresponding to the above embodiments, refer to Figure 7 , shows a structural block diagram of a data compression device provided in an embodiment of the present application. The device may include a time series data acquisition module 701, a pattern category recognition module 702, a key data point determination module 703, and a data compression module 704, wherein:

[0195] The time series data acquisition module 701 is used to acquire the original data to be compressed, where the original data to be compressed is time series data, which is obtained by processing data values ​​of the target type in chronological order;

[0196] A pattern category identification module 702 is configured to identify the pattern category of the time series data based on the data value of the target type;

[0197] Key data point determination module 703, for determining key data points corresponding to the pattern category, the pattern category having corresponding data storage rules;

[0198] The data compression module 704 is configured to store the key data points and their corresponding data values ​​according to the data storage rules, thereby compressing the time series data.

[0199] In a possible implementation of the embodiment of the present application, the pattern category identification module 702 may be specifically configured to:

[0200] Dividing the time series data into multiple subsequences;

[0201] The pattern category of the time series data is determined according to how the data value of the target type in each of the subsequences obtained by the division changes over time.

[0202] In another possible implementation of the embodiment of the present application, the pattern category identification module 702 may also be used to:

[0203] The trained neural network is used to identify the data value of the target type and output the pattern category of the time series data.

[0204] In an embodiment of the present application, the neural network is obtained by training a sample data set, wherein the sample data set includes labeled supervised data and unlabeled unsupervised data, the unsupervised data is used to pre-train the encoder, and the supervised data is used to train the classifier. The encoder obtained by pre-training is used to output the feature value of each subsequence obtained by dividing the time series data in the form of a sliding window, and the pattern category of the time series data is output by using the feature value of each subsequence as the input data of the classifier.

[0205] In a possible implementation of the embodiment of the present application, the data storage rule may include the number of key data points to be stored and the location information of each key data point. The data compression module 704 may be specifically configured to:

[0206] According to the position information of each key data point, obtaining the data value of the key data point at the corresponding position;

[0207] Writing the data value of the key data point into a data structure having a preset data format;

[0208] The data structure having the data value written thereto is stored.

[0209] In a possible implementation of the embodiment of the present application, the data format may include the following information:

[0210] Pattern category, start time, duration, start data value, end data value, average value, maximum data value, and minimum data value.

[0211] In a possible implementation of an embodiment of the present application, the time series data may include network data, the target type data value may include signal strength and MAC address at different times in the network data, and the mode category may include roaming mode, frequency switching mode, mobile mode, or stable mode determined based on the signal strength and MAC address. The roaming mode may include early roaming mode or late roaming mode, and the stable mode may include strong signal stable mode, weak signal stable mode, or leaving mode.

[0212] In a possible implementation of the embodiment of the present application, the apparatus may further include a data recovery module, which may be specifically configured to:

[0213] Obtaining the stored key data points and their corresponding data values;

[0214] Interpolation is performed based on the key data points and their corresponding data values ​​to restore the original data.

[0215] An embodiment of the present application also provides a data compression device, which can be the computer device in the aforementioned embodiment or a related module in the computer device. The application of this device can implement the relevant functions in the aforementioned method embodiments to achieve compression of time series data.

[0216] Reference Figure 8 , shows a structural block diagram of a data compression quality evaluation device provided in an embodiment of the present application. The device may include a data sequence acquisition module 801, a data segment division module 802, a difference value determination module 803, and a data compression quality evaluation module 804, wherein:

[0217] A data sequence acquisition module 801 is configured to acquire an original data sequence and a data sequence to be evaluated, wherein the data sequence to be evaluated is a data sequence recovered after data compression of the original data sequence, and the length of the original data sequence is equal to that of the data sequence to be evaluated;

[0218] A data segment division module 802 is configured to divide the original data sequence and the data sequence to be evaluated into a plurality of data segments respectively;

[0219] a difference value determination module 803, configured to determine a difference value between each data segment in the original data sequence and a corresponding data segment in the data sequence to be evaluated;

[0220] The data compression quality evaluation module 804 is configured to evaluate the data compression quality of the original data sequence based on the difference value.

[0221] In a possible implementation of the embodiment of the present application, the data segment division module 802 may be specifically configured to:

[0222] Determining the data increase and decrease trends of the original data sequence and the data sequence to be evaluated;

[0223] The original data sequence and the data sequence to be evaluated are segmented according to the data increase or decrease trend to obtain a plurality of data segments; wherein the data increase or decrease trend in any of the data segments remains unchanged.

[0224] In a possible implementation of the embodiment of the present application, the difference value determination module 803 may be specifically configured to:

[0225] Determining a plurality of evaluation factors, wherein the plurality of evaluation factors include an overall distribution interval difference factor, a fluctuation severity difference factor, and an overall trend difference factor;

[0226] The difference value between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated is calculated based on the multiple evaluation factors.

[0227] In a possible implementation of the embodiment of the present application, the data compression quality evaluation module 804 may be specifically configured to:

[0228] Calculating a data sequence score value according to the difference value corresponding to each of the data segments;

[0229] Counting the number of key data points that have not been successfully recovered in the data sequence to be evaluated;

[0230] Deducting the data sequence score based on the number of key data points that were not successfully recovered to obtain a total data sequence score;

[0231] A loss value for evaluating data compression quality is calculated according to the total score of the data sequence.

[0232] In another possible implementation of the embodiment of the present application, the data compression quality evaluation module 804 may also be used to:

[0233] The difference values ​​corresponding to each of the data segments are weighted and summed to obtain the data sequence score value; wherein the weight value of each of the data segments is the ratio of the length of the corresponding data segment to the total length of the data sequence.

[0234] In another possible implementation of the embodiment of the present application, the data compression quality evaluation module 804 may also be used to:

[0235] Calculating the ratio between the number of the key data points that were not successfully recovered and the total length of the data sequence;

[0236] The total score of the data sequence is calculated by subtracting the product of the ratio and the constant value from the data sequence score value.

[0237] In another possible implementation of the embodiment of the present application, the data compression quality evaluation module 804 may also be used to: count the number of data points in the original data sequence whose data values ​​are less than a critical threshold but whose corresponding data values ​​in the data sequence to be evaluated are greater than or equal to the critical threshold; or,

[0238] The number of data points in the original data sequence whose data values ​​are greater than or equal to the critical threshold but whose corresponding data values ​​in the data sequence to be evaluated are less than the critical threshold is counted.

[0239] An embodiment of the present application also provides a device for evaluating data compression quality. The data compression device can be the computer device in the aforementioned embodiment or a related module in the computer device. The application of this device can implement the relevant functions in the aforementioned method embodiments to achieve the evaluation of data compression quality.

[0240] Since the functions implemented by each module in the device embodiment are similar to the contents described in the aforementioned embodiments, the device embodiment is described briefly. For relevant details, please refer to the description of the aforementioned embodiments and will not be repeated here.

[0241] Reference Figure 9 , shows a schematic diagram of a computer device provided by an embodiment of the present application. Figure 9 As shown, the computer device 900 in the embodiment of the present application includes: a processor 910, a memory 920, and a computer program 921 stored in the memory 920 and executable on the processor 910. When the processor 910 executes the computer program 921, the steps of each embodiment of the above-mentioned data compression method or data compression quality evaluation method are implemented, such as Figure 5 Alternatively, when the processor 910 executes the computer program 921, the functions of the modules in the above-mentioned device embodiments are realized, for example, Figure 7 Functions of modules 701 to 704 are shown.

[0242] Exemplarily, the computer program 921 may be divided into one or more modules / units, which are stored in the memory 920 and executed by the processor 910 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 921 in the computer device 900. For example, the computer program 921 may be divided into a time series data acquisition module, a pattern category recognition module, a key data point determination module, and a data compression module to implement the data compression function in the aforementioned embodiments, wherein the specific functions of each module are as follows:

[0243] A time series data acquisition module is used to acquire the original data to be compressed, where the original data to be compressed is time series data, which is obtained by processing data values ​​of the target type in chronological order;

[0244] a pattern category identification module, configured to identify the pattern category of the time series data based on the data value of the target type;

[0245] A key data point determination module, configured to determine key data points corresponding to the pattern category, wherein the pattern category has corresponding data storage rules;

[0246] The data compression module is used to store the key data points and their corresponding data values ​​according to the data storage rules to complete the compression of the time series data.

[0247] Alternatively, the computer program 921 may be divided into a data sequence acquisition module, a data segmentation module, a difference value determination module, and a data compression quality assessment module to implement the data compression quality assessment function in the aforementioned embodiments. Specific functions of each module are as follows:

[0248] A data sequence acquisition module is configured to acquire an original data sequence and a data sequence to be evaluated, wherein the data sequence to be evaluated is a data sequence recovered after data compression of the original data sequence, and the length of the original data sequence is equal to that of the data sequence to be evaluated;

[0249] A data segment division module, configured to divide the original data sequence and the data sequence to be evaluated into a plurality of data segments respectively;

[0250] a difference value determination module, configured to determine a difference value between each data segment in the original data sequence and a corresponding data segment in the data sequence to be evaluated;

[0251] The data compression quality evaluation module is used to evaluate the data compression quality of the original data sequence based on the difference value.

[0252] The computer device 900 may be a device for implementing the steps in the above-mentioned various method embodiments. The computer device 900 may be a desktop computer, a cloud server, or other devices. The computer device 900 may include, but is not limited to, a processor 910 and a memory 920. It will be understood by those skilled in the art that Figure 9 This is merely an example of the computer device 900 and does not constitute a limitation of the computer device 900 . The computer device 900 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device 900 may also include input and output devices, network access devices, buses, etc.

[0253] The processor 910 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0254] The memory 920 may be an internal storage unit of the computer device 900, such as a hard disk or memory of the computer device 900. The memory 920 may also be an external storage device of the computer device 900, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the computer device 900. Furthermore, the memory 920 may include both an internal storage unit of the computer device 900 and an external storage device. The memory 920 is used to store the computer program 921 and other programs and data required by the computer device 900. The memory 920 may also be used to temporarily store data that has been output or is about to be output.

[0255] An embodiment of the present application further discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the methods described in the aforementioned embodiments are implemented.

[0256] An embodiment of the present application further discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the above embodiments is implemented.

[0257] The embodiments of the present application further disclose a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the methods described in the aforementioned embodiments.

[0258] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed in the present application should be covered by the protection scope of the present application.

Claims

1. A data compression method, characterized in that: include: Collecting original data to be compressed, where the original data to be compressed is time series data, which is obtained by processing data values ​​of a target type in chronological order; Based on the data value of the target type, identifying the pattern category of the time series data; determining key data points corresponding to the pattern category, the pattern category having corresponding data storage rules; The key data points and their corresponding data values ​​are stored according to the data storage rules to complete the compression of the time series data.

2. The method according to claim 1, characterized in that The identifying the pattern category of the time series data based on the data value of the target type includes: Dividing the time series data into multiple subsequences; The pattern category of the time series data is determined according to how the data value of the target type in each of the subsequences obtained by the division changes over time.

3. The method according to claim 1, characterized in that The identifying the pattern category of the time series data based on the data value of the target type includes: The trained neural network is used to identify the data value of the target type and output the pattern category of the time series data.

4. The method according to claim 3, characterized in that The neural network is obtained by training a sample data set, which includes labeled supervised data and unlabeled unsupervised data. The unsupervised data is used to pre-train the encoder, and the supervised data is used to train the classifier. The encoder obtained by pre-training is used to output the feature value of each subsequence obtained by dividing the time series data in the form of a sliding window, and the pattern category of the time series data is output by using the feature value of each subsequence as the input data of the classifier.

5. The method according to any one of claims 1 to 4, characterized in that The data storage rule includes the number of the key data points to be stored and the location information of each key data point. The key data points and their corresponding data values ​​are stored according to the data storage rule to complete the compression of the time series data, including: According to the position information of each key data point, obtaining the data value of the key data point at the corresponding position; Writing the data value of the key data point into a data structure having a preset data format; The data structure having the data value written thereto is stored.

6. The method according to claim 5, characterized in that The data format includes the following information: Pattern category, start time, duration, start data value, end data value, average value, maximum data value, and minimum data value.

7. The method according to any one of claims 1 to 6, characterized in that The time series data includes network data, the data value of the target type includes the signal strength and MAC address at different times in the network data, and the mode category includes roaming mode, frequency switching mode, mobile mode or stable mode determined based on the signal strength and the MAC address.

8. The method according to claim 7, characterized in that The roaming mode includes an early roaming mode or a late roaming mode, and the stable mode includes a strong signal stable mode, a weak signal stable mode or a leaving mode.

9. The method according to any one of claims 1 to 8, characterized in that Also includes: Obtaining the stored key data points and their corresponding data values; Interpolation is performed based on the key data points and their corresponding data values ​​to restore the original data.

10. A method for evaluating data compression quality, characterized in that: include: Obtaining an original data sequence and a data sequence to be evaluated, wherein the data sequence to be evaluated is obtained by compressing the original data sequence, and the original data sequence and the data sequence to be evaluated have the same length; Dividing the original data sequence and the data sequence to be evaluated into a plurality of data segments respectively; Determine the difference between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated; The data compression quality of the original data sequence is evaluated based on the difference value.

11. The method according to claim 10, characterized in that The step of dividing the original data sequence and the to-be-evaluated data sequence into a plurality of data segments comprises: Determining the data increase and decrease trends of the original data sequence and the data sequence to be evaluated; The original data sequence and the data sequence to be evaluated are segmented according to the data increase or decrease trend to obtain a plurality of data segments; wherein the data increase or decrease trend in any of the data segments remains unchanged.

12. The method according to claim 10 or 11, characterized in that Determining the difference between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated includes: Determining a plurality of evaluation factors, wherein the plurality of evaluation factors include an overall distribution interval difference factor, a fluctuation severity difference factor, and an overall trend difference factor; The difference value between each data segment in the original data sequence and the corresponding data segment in the data sequence to be evaluated is calculated based on the multiple evaluation factors.

13. The method according to any one of claims 10 to 12, characterized in that The evaluating the data compression quality of the original data sequence based on the difference value includes: Calculating a data sequence score value according to the difference value corresponding to each of the data segments; Counting the number of key data points that are not successfully recovered after data recovery is performed on the data sequence to be evaluated; Deducting the score of the data sequence based on the number of the key data points that were not successfully recovered to obtain a total score of the data sequence; A loss value for evaluating data compression quality is calculated according to the total score of the data sequence.

14. The method according to claim 13, characterized in that Calculating a data sequence score value according to the difference value corresponding to each of the data segments includes: The difference values ​​corresponding to each of the data segments are weighted and summed to obtain the data sequence score value; wherein the weight value of each of the data segments is the ratio of the length of the corresponding data segment to the total length of the data sequence.

15. The method according to claim 13 or 14, characterized in that The step of subtracting the data sequence score based on the number of the key data points that were not successfully recovered to obtain a total data sequence score includes: Calculating the ratio between the number of the key data points that were not successfully recovered and the total length of the data sequence; The total score of the data sequence is calculated by subtracting the product of the ratio and the constant value from the data sequence score value.

16. The method according to any one of claims 13 to 15, characterized in that The counting of the number of key data points that are not successfully recovered after data recovery is performed on the data sequence to be evaluated includes: Counting the number of data points in the original data sequence whose data values ​​are less than a critical threshold but whose corresponding data values ​​in the data sequence to be evaluated are greater than or equal to the critical threshold; or The number of data points in the original data sequence whose data values ​​are greater than or equal to the critical threshold but whose corresponding data values ​​in the data sequence to be evaluated are less than the critical threshold is counted.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 9 or 10 to 16 is implemented.

18. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 or 10 to 16 is implemented.

Images