Data processing method, device, electronic device and computer-readable storage medium

Abstract

This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium; the method includes: determining a deviation based on a first data sequence; determining a feature value based on the first data sequence; determining a compression threshold boundary based on the deviation and the feature value, the compression threshold boundary being used to determine a compression strategy for compressing second data; performing lossy compression on data in the second data that falls within the compression threshold boundary; wherein the deviation includes: a pre-set static deviation, or a dynamic deviation generated based on the first data sequence.

Description

Technical Field

[0001] This application relates to big data technology, and more particularly to a data processing method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In related technologies, to effectively compress data and save on data transmission and persistence costs, different compression methods can be applied to different regions of a data sequence; for example, lossy compression can be performed on a portion of the data sequence, while lossless compression or no compression can be performed on another portion. Therefore, it is necessary to determine the compression threshold boundaries for different compression methods applied to the data sequence.

[0003] The compression threshold boundary for lossy compression is typically determined through manual, empirical annotation. However, this method requires a high level of data processing skills and experience from the staff. Staff with weak data processing skills or limited experience may determine the compression threshold boundary with significant errors. In scenarios with large variations in data sequences, the determined compression threshold boundary may also be inaccurate. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and computer-readable storage medium that can improve the accuracy of determining compression threshold boundaries.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a data processing method, including:

[0007] The deviation is determined based on the first data sequence;

[0008] Determine feature values ​​based on the first data sequence;

[0009] The compression threshold boundary is determined based on the deviation and the feature value, and the compression threshold boundary is used to determine the compression strategy for compressing the second data;

[0010] Lossy compression is performed on the data in the second data that falls within the compression threshold boundary;

[0011] The deviation includes: a pre-set static deviation, or a dynamic deviation generated based on the first data sequence.

[0012] In some embodiments, determining the feature value based on the first data sequence includes:

[0013] Determine the maximum and minimum values ​​of the data in the first data sequence.

[0014] The characteristic value is determined to include the maximum value and the minimum value of the data.

[0015] In some embodiments, determining the compression threshold boundary based on the deviation and the feature value includes:

[0016] The maximum value of the compression threshold boundary is determined to be the sum of the maximum value of the data in the first data sequence and the deviation;

[0017] The minimum value of the compression threshold boundary is determined to be the difference between the minimum value of the data in the first data sequence and the deviation.

[0018] In some embodiments, determining the feature value based on the first data sequence includes:

[0019] Determine the average value of the first data sequence;

[0020] The characteristic value is determined to be the average value of the first data sequence.

[0021] In some embodiments, determining the compression threshold boundary based on the deviation and the feature value includes:

[0022] If the feature value is the average value of the first data sequence, then the maximum value of the compression threshold boundary is determined to be the sum of the product of the average value and the deviation and the average value;

[0023] The minimum value of the compression threshold boundary is determined to be the difference between the average value and the product of the average value and the deviation.

[0024] In some embodiments, determining the feature value based on the first data sequence includes:

[0025] A predicted data sequence is determined based on the first data sequence; the first data sequence is related to historical data.

[0026] The feature value is determined by each predicted data point in the predicted data sequence.

[0027] In some embodiments, determining the compression threshold boundary based on the deviation and the feature value includes:

[0028] The feature value includes each predicted data in the predicted data sequence, then the maximum value of the compression threshold boundary is determined to be the sum of the product of the predicted data and the deviation and the predicted data;

[0029] The minimum value of the compression threshold boundary is determined to be the difference between the predicted data and the product of the predicted data and the deviation.

[0030] In some embodiments, determining the compression threshold boundary based on the deviation and the feature value includes:

[0031] If the feature value includes each predicted data in the predicted data sequence, then a residual is generated based on the predicted data sequence, and the residual is used as the deviation;

[0032] The maximum value of the compression threshold boundary is determined to be the maximum value in the predicted data sequence minus the product of the maximum value in the predicted data sequence and the residual;

[0033] The minimum value of the compression threshold boundary is determined to be the minimum value in the predicted data sequence minus the product of the minimum value in the predicted data sequence and the residual.

[0034] Secondly, embodiments of this application provide a data processing apparatus, including:

[0035] The first determining module is used to determine the first data deviation and characteristic value;

[0036] The second determining module is used to determine a compression threshold boundary based on the deviation and the feature value, wherein the compression threshold boundary is used to determine a compression strategy for compressing the second data.

[0037] The compression module is used to perform lossy compression on the data in the second data that is within the compression threshold boundary;

[0038] The deviation includes: a pre-set static deviation, or a dynamic deviation generated based on the first data sequence.

[0039] In some embodiments, the first determining module is configured to determine the maximum and minimum data values ​​in the first data sequence, and to determine that the feature value includes the maximum and minimum data values.

[0040] In some embodiments, the second determining module is used to determine that the maximum value of the compression threshold boundary is the sum of the maximum value of the data in the first data sequence and the deviation;

[0041] The minimum value of the compression threshold boundary is determined to be the difference between the minimum value of the data in the first data sequence and the deviation.

[0042] In some embodiments, the first determining module is configured to determine the average value of the first data sequence; and to determine the feature value as the average value of the first data sequence.

[0043] In some embodiments, the second determining module is configured to determine, if the feature value is the average value of the first data sequence, the maximum value of the compression threshold boundary is the sum of the product of the average value and the deviation and the average value;

[0044] The minimum value of the compression threshold boundary is determined to be the difference between the average value and the product of the average value and the deviation.

[0045] In some embodiments, the first determining module is configured to determine a predicted data sequence based on the first data sequence; the first data sequence is related to historical data.

[0046] The feature value is determined by each predicted data point in the predicted data sequence.

[0047] In some embodiments, the second determining module is configured to determine, when the feature value includes each predicted data in the predicted data sequence, that the maximum value of the compression threshold boundary is the sum of the product of the predicted data and the deviation and the predicted data;

[0048] The minimum value of the compression threshold boundary is determined to be the difference between the predicted data and the product of the predicted data and the deviation.

[0049] In some embodiments, the second determining module is configured to generate a residual based on the prediction data sequence when the feature value includes each prediction data in the prediction data sequence, and use the residual as the deviation.

[0050] The maximum value of the compression threshold boundary is determined to be the maximum value in the predicted data sequence minus the product of the maximum value in the predicted data sequence and the residual;

[0051] The minimum value of the compression threshold boundary is determined to be the minimum value in the predicted data sequence minus the product of the minimum value in the predicted data sequence and the residual.

[0052] Thirdly, embodiments of this application provide an electronic device, including:

[0053] Memory, used to store executable instructions;

[0054] The processor, when executing executable instructions stored in the memory, implements the data processing method provided in the embodiments of this application.

[0055] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions for implementing the data processing method provided in embodiments of this application when executed by a processor.

[0056] The data processing method provided in this application embodiment determines a deviation and a feature value based on a first data sequence; determines a compression threshold boundary based on the deviation and the feature value, the compression threshold boundary being used to determine a compression strategy for compressing second data; and performs lossy compression on data in the second data that falls within the compression threshold boundary; wherein the deviation includes: a pre-set static deviation, or a dynamic deviation generated based on the first data sequence. Thus, the process of determining the compression threshold boundary is implemented by an electronic device, eliminating the need for manual annotation of the compression threshold boundary and improving the accuracy of determining the compression threshold boundary. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of an optional flow of the data processing method provided in the embodiments of this application;

[0058] Figure 2 This is a detailed flowchart illustrating a data processing method provided in an embodiment of this application;

[0059] Figure 3 This is a schematic diagram of the architecture of the data processing system provided in the embodiments of this application;

[0060] Figure 4 This is a schematic diagram of the architecture of the terminal device provided in the embodiments of this application. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0063] In the following description, the terms "first," "second," and "third" are used merely to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first," "second," and "third" may be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. In the following description, the term "multiple" refers to at least two.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0065] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0066] 1) Lossy compression: It effectively reduces the repetition and redundancy in data information through a certain encoding method, thereby achieving the purpose of data compression; after the data is restored by lossy compression, it cannot be exactly the same as the data before lossy compression, and a certain loss will occur.

[0067] 2) Lossless compression: By using a certain encoding method, the redundancy and repetition in the data information are effectively reduced, thereby achieving the purpose of data compression; after the data is restored by lossless compression, it is exactly the same as the data before lossless compression, and no loss occurs.

[0068] 3) Compression threshold boundary: The boundary line used for lossy compression of data. Lossy compression can be performed on data within this boundary line.

[0069] 4) Historical data sequence: The set of data up to the current moment.

[0070] See Figure 1 , Figure 1 This is a schematic diagram of an optional flow of the data processing method provided in the embodiments of this application, which will be combined with Figure 1 The steps shown are explained.

[0071] Step S101: Determine the deviation based on the first data sequence.

[0072] In some embodiments, the deviation can be a pre-set static deviation or a dynamic deviation generated based on the first data sequence.

[0073] In some embodiments, if the first data sequence is data generated from a business scenario represented by server temperature, the deviation can be determined to be zero. If the first data sequence is data generated from business scenarios such as server CPU load, memory utilization, or the number of active database connections, the deviation can be preset to a fixed value. If the first data sequence has specific attributes, the business scenario corresponding to the first data sequence is sufficiently stable, or the first data sequence is data with periodic characteristics and suitable for processing by a time series decomposition algorithm (Seasonal Trend decomposition procedure based on Loess, STL), the deviation can be calculated based on the first data sequence; in this scenario, the deviation is determined based on the first data sequence, and if the first data sequence is historical data, the deviation is dynamically changing.

[0074] Step S102: Determine feature values ​​based on the first data sequence.

[0075] In some embodiments, the feature value is a factor used to determine the compression threshold boundary.

[0076] Step S103: Determine the compression threshold boundary based on the deviation and eigenvalue.

[0077] In some embodiments, the compression threshold boundary is used to determine the compression strategy for the second data compression.

[0078] Step S104: Perform lossy compression on the data in the second data that is within the compression threshold boundary.

[0079] In some embodiments, the second data is the data to be compressed. Lossy compression is performed on the data in the second data that lies between the maximum value and the minimum value of the compression threshold boundary; lossless compression or no compression is performed on the data in the second data that is greater than the maximum value of the compression threshold boundary and the data in the second data that is less than the minimum value of the compression threshold boundary.

[0080] In some embodiments, the second data can be lossily compressed based on a lossy compression step size; the lossy compression step size range records lossy compression merging rules of different granularities. For example, if the CPU workload is between 0% and 25%, the lossy compression step size range is 5%, with each 5% divided into an interval. Second data falling within an interval is normalized to its nearest value; for example, 2.7%, 3.1%, and 4.6% are recorded as 5%, while 14.1% and 15.1% are recorded as 15%. Between 25% and 50%, the lossy compression step size range is 3%, with each 3% recorded as an interval. Lossy compression is no longer performed above 50%.

[0081] Regarding step S102, the method for determining feature values ​​differs depending on the scenario.

[0082] Scenario 1: A business scenario represented by server temperature. The characteristic values ​​are determined as the maximum and minimum values ​​in a first data sequence. This first data sequence is related to the server's historical temperatures; as an example, the first data sequence can be a dataset of server temperature data within a historical timeframe.

[0083] Scenario 2: For business scenarios involving server CPU load, memory usage, or database active connection count, the average value of the first data sequence can be determined first, and the feature value can be determined as the average value of the first data sequence.

[0084] Scenario 3: If the business scenario corresponding to the first data sequence is stable enough, or if the first data sequence is periodic and suitable for STL processing, the predicted data sequence can be determined first based on the first data sequence, and then the feature values ​​can be determined to include each predicted data in the predicted data sequence.

[0085] In some embodiments, a first data sequence can be input into a pre-trained neural network model, and the data output by the neural network model constitutes a predicted data sequence.

[0086] The first data sequence can be historical data corresponding to the second data to be processed.

[0087] Regarding step S103, for scenario one above, the feature values ​​are the maximum and minimum values ​​of the data in the first data sequence. Therefore, the maximum value of the compression threshold boundary is determined to be the sum of the maximum value of the data in the first data sequence and the deviation; the minimum value of the compression threshold boundary is determined to be the difference between the minimum value of the data in the first data sequence and the first deviation.

[0088] For scenario two above, an Isolation Forest (iForest) can be used to analyze the first data sequence to obtain the average value of the first data sequence. The average value is multiplied by the deviation to obtain an offset. The sum of the average value of the first data sequence and the first offset is the maximum value of the compression threshold boundary, and the difference between the average value of the first data sequence and the first offset is the minimum value of the compression threshold boundary.

[0089] In practical implementation, percentages, box plots, standard deviations, and other proportional relationships can also be used to determine the compression threshold boundaries. As an example, the first data sequence is arranged in descending order, and the data located at the 25th and 75th percentiles in the first data sequence are taken as the maximum and minimum values ​​of the compression threshold boundaries; where 25% and 75% are the percentiles of the first data sequence.

[0090] For scenario three above, if the first data sequence has periodic characteristics and is suitable for STL processing, then the feature value includes each predicted data in the predicted data sequence; the maximum value for determining the compression threshold boundary is the sum of the product of the predicted data and the bias, and the minimum value for determining the compression threshold boundary is the difference between the predicted data and the product of the predicted data and the bias. Specifically, the first data sequence can be split into trend components, periodic components, and residual components using STL, with the residual components used as the bias.

[0091] For scenario three above, if the business scenario corresponding to the first data sequence is stable enough and has sufficient historical data, the feature value includes each predicted data in the predicted data sequence; firstly, a residual is generated based on the predicted data sequence, and the residual is used as the deviation; the maximum value of the compression threshold boundary is determined to be the maximum value in the predicted data sequence minus the product of the maximum value in the predicted data sequence and the residual; the minimum value of the compression threshold boundary is determined to be the minimum value in the predicted data sequence minus the product of the minimum value in the predicted data sequence and the residual.

[0092] The data processing methods provided in the embodiments of this application will be described in detail below for different scenarios.

[0093] Example 1

[0094] Because different servers have different hardware configurations, different rack locations, and different operating loads, their temperatures vary greatly. Therefore, the compression threshold boundaries for lossy compression set for different servers will also differ.

[0095] In some embodiments, for a particular server, historical temperature data (also known as a first data sequence) of the server can be obtained, and the maximum and minimum historical temperatures in the historical data can be obtained.

[0096] 1) If the sum of the historical maximum temperature (also known as the characteristic value) and the deviation is greater than or equal to the maximum value of the preset static threshold, then the maximum value of the compression threshold boundary is determined to be equal to the maximum value of the static threshold.

[0097] 2) If the sum of the historical maximum temperature (also known as the characteristic value) and the deviation is less than the maximum value of the preset static threshold, then the maximum value of the compression threshold boundary is determined to be equal to the sum of the historical maximum temperature and the deviation. The deviation can be a preset value.

[0098] 3) If the difference between the historical minimum temperature (also known as the characteristic value) and the deviation is less than or equal to the minimum value of the preset static threshold, then the minimum value of the compression threshold boundary is determined to be equal to the minimum value of the static threshold.

[0099] 4) If the difference between the historical minimum temperature (also known as the characteristic value) and the deviation is greater than the minimum value of the preset static threshold, then the minimum value of the compression threshold boundary is determined to be equal to the difference between the historical minimum temperature and the deviation.

[0100] In some embodiments, if Threshold is used hd Threshold represents the maximum value of the compression threshold boundary. ld Threshold represents the minimum value of the compression threshold boundary. h Threshold represents the maximum value of the static threshold. l The maximum value represents the minimum static threshold, Bias represents the deviation, and Max represents the maximum value. h Max represents the maximum value in historical data (also known as the maximum historical temperature). d Let represent the minimum historical data value (also known as the minimum historical temperature value); then the maximum value of the compression threshold boundary is shown in the following formula (1):

[0101]

[0102] The minimum value of the compression threshold boundary is shown in the following formula (2):

[0103]

[0104] In some embodiments, the time range, deviation, and compression threshold boundaries of the server's temperature data corresponding to the historical data can be pre-marked by the server's maintenance personnel, using the methods determined in 1) to 4). The static threshold is the boundary where the server's temperature is considered normal. For example, if the server's temperature is between the maximum and minimum values ​​of the static threshold, then the server's temperature is considered normal; if the server's temperature is greater than the maximum value of the static threshold, then the server's temperature is considered abnormal, indicating that the server has been running for too long, has too many running threads, may be experiencing a failure, or has already failed.

[0105] In this embodiment, the compression threshold boundary is determined by the maximum and minimum values ​​of historical data and a preset deviation. This requires less computing power and reduces the workload of setting the compression threshold. The process of determining the compression threshold boundary is implemented by electronic equipment, eliminating the need for manual annotation of the compression threshold boundary and improving the accuracy of the determination.

[0106] Example 2

[0107] For business scenarios involving server CPU load, memory utilization, or database active connection count, in some embodiments, a preset deviation can be set, such as a preset deviation of 25%. The first data sequence (also known as the historical data sequence) is arranged in descending order, and the data located at the 25th and 75th percentiles in the first data sequence are used as the maximum and minimum values ​​of the compression threshold boundary, respectively. Here, 25% and 75% are the percentiles of the first data sequence.

[0108] In other embodiments, iForest can be used to analyze the first data sequence to obtain data intervals above and / or below the average value of the first data sequence (also known as historical data sequence). The maximum value of this data interval is taken as the maximum value of the compression threshold boundary, and the minimum value of this data interval is taken as the minimum value of the compression threshold boundary. The histogram corresponding to the first data sequence conforms to or approximates a normal distribution.

[0109] In practical implementation, after calculating the maximum and minimum values ​​of the compression threshold boundaries, a method similar to that in Scenario 1 can be used to further determine the final compression threshold boundaries by combining them with the static threshold. For example, in a CPU-loaded scenario, if the maximum value of the compression threshold boundaries obtained using the isolated forest method is greater than the static threshold, then the maximum value of the compression threshold boundaries ultimately used for the second data compression can be determined to be the static threshold; conversely, if the minimum value of the compression threshold boundaries obtained using the isolated forest method is less than the static threshold, then the minimum value of the compression threshold boundaries ultimately used for the second data compression can be determined to be the minimum value of the compression threshold boundaries obtained using the isolated forest method.

[0110] In this embodiment, the compression threshold boundary is determined using statistical methods (such as sorting the first data sequence and counting the data at a preset percentile). This requires less computational power, reducing the workload of setting the compression threshold. The process of determining the compression threshold boundary is implemented by electronic devices, eliminating the need for manual annotation of the compression threshold boundary and improving the accuracy of determining the compression threshold boundary.

[0111] Example 3

[0112] For scenarios where the first data sequence exhibits periodicity and is suitable for STL processing, the first data sequence can be decomposed into trend components, periodic components, and residual components using STL. A histogram of the residual components is calculated to obtain the main distribution area of ​​the residual components, thus determining the bias. If the residual components are distributed within two standard deviations, the bias can be determined to be two standard deviations. The first data sequence (also known as the historical data sequence) is input into a pre-trained neural network model to obtain the predicted data sequence output by the network model. The maximum value in the predicted data sequence is multiplied by the bias and then added to the maximum value in the predicted data sequence to obtain the maximum value of the compression threshold boundary. The minimum value in the predicted data sequence is multiplied by the bias and then subtracted from the minimum value in the predicted data sequence to obtain the minimum value of the compression threshold boundary. Specifically, when using the neural network model to obtain the predicted data sequence, the trend component and periodic component obtained from the decomposition of the first data sequence can be input into the neural network model separately to obtain the corresponding predicted data sequences.

[0113] In this embodiment, the compression threshold boundary is determined based on deep learning technology, enabling finer-grained setting of the compression threshold boundary and improving the accuracy of its determination. The process of determining the compression threshold boundary is implemented by an electronic device, eliminating the need for manual annotation and further enhancing accuracy.

[0114] Example 4

[0115] For scenarios where the business scenario corresponding to the first data sequence is sufficiently stable and has enough historical data, deep learning methods, such as RNN or LSTM, can be used to determine the predicted data sequence based on the first data sequence. The difference between the first data sequence and the predicted data sequence is calculated to obtain the residual; this residual is used as the deviation. If the deviation is 3 standard deviations, the maximum value in the predicted data sequence minus the product of the maximum value in the predicted data sequence and 0.15% is the maximum value of the compression threshold boundary, and the minimum value in the predicted data sequence minus the product of the minimum value in the predicted data sequence and 0.15% is the minimum value of the compression threshold boundary. In this embodiment, an envelope of the upper and lower intervals can be generated based on the residual and the predicted data sequence; this envelope is the compression threshold boundary. Determining the compression threshold boundary based on the residual yields a more accurate compression threshold boundary.

[0116] In this embodiment, the compression threshold boundary is determined based on deep learning technology, enabling finer-grained setting of the compression threshold boundary and improving the accuracy of its determination. The process of determining the compression threshold boundary is implemented by an electronic device, eliminating the need for manual annotation and further enhancing accuracy.

[0117] In some embodiments provided in this application, a detailed processing flow of the data processing method can be as follows: Figure 2 As shown, the strategies used to determine the compression threshold boundaries differ for the four scenarios described above. Therefore, a pre-defined correspondence between the four scenarios and the strategies used to determine the compression threshold boundaries can be established. These four scenarios can be determined based on business scenarios or data attributes. Business scenarios could be CPU temperature-related or CPU load-related; data attributes could be periodic or have sufficient historical data. Staff determine this correspondence and store it, along with the parameters involved in the strategy used to determine the compression threshold boundaries, such as deviation and static thresholds, in a database or storage. Deviation and static thresholds can also be imported in batches through integration with external monitoring systems, network technology service management system databases, or web application programming interfaces (Web APIs); for example, batch importing static thresholds from a Zabbix database or obtaining static thresholds by integrating with the Prometheus Web API. After receiving second data from an external monitoring device or IoT system, the data processing device searches a pre-stored correspondence for the strategy used to determine the compression threshold boundary. It then uses this strategy to determine the compression threshold boundary and performs lossy compression on the input second data based on this boundary. Second data outside the compression threshold boundary can be left uncompressed or subjected to lossless compression. The lossily compressed second data can then be saved to a target database or transmitted to a target system.

[0118] The operations of finding the strategy used to determine the compression threshold boundary, calculating the compression threshold boundary based on the found strategy, and compressing the second data can all be implemented by the time data series filter.

[0119] In this application embodiment, different strategies for determining compression threshold boundaries are configured for different business scenarios or data with different attributes; therefore, different strategies can be adapted for different business scenarios or data with different attributes to improve the accuracy of determining compression threshold boundaries; the data processing method provided in this application embodiment can be widely applied in intelligent operation and maintenance systems or the Internet of Things field.

[0120] The following describes an exemplary application of the electronic device for the data processing method provided in the embodiments of this application. The electronic device provided in the embodiments of this application can be implemented as various types of terminal devices or as a server.

[0121] See Figure 3 , Figure 3This is a schematic diagram of the architecture of the data processing system 100 provided in the embodiments of this application. The terminal device 400 is connected to the server 200 through the network 300, and the server 200 is connected to the database 500. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0122] In some embodiments, taking an electronic device as a terminal device as an example, the data processing method provided in this application embodiment can be implemented collaboratively by the terminal device. For example, server 200 obtains a first data sequence from database 500 and sends the first data sequence to terminal device 400; terminal device 400 runs client 410, which can be a client for performing data processing. Client 410 determines a deviation based on the first data sequence; determines a feature value based on the first data sequence; determines a compression threshold boundary based on the deviation and the feature value, the compression threshold boundary being used to determine a compression strategy for compressing second data; and performs lossy compression on the data in the second data that falls within the compression threshold boundary.

[0123] In some embodiments, taking the electronic device as a server as an example, the data processing method provided in this application embodiment can be implemented by the server. For example, the server 200 obtains a first data sequence from the database 500, determines a deviation based on the first data sequence, determines a feature value based on the first data sequence, determines a compression threshold boundary based on the deviation and the feature value, the compression threshold boundary is used to determine a compression strategy for compressing second data, and performs lossy compression on the data in the second data that is within the compression threshold boundary.

[0124] In some embodiments, the terminal device 400 or server 200 can implement the data processing method provided in this application embodiment by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that only needs to be downloaded to a browser environment to run; or it can be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0125] In some embodiments, server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network or local area network to achieve data computation, storage, processing, and sharing. Terminal device 400 can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0126] Taking the example of a terminal device provided in this application embodiment, it can be understood that in the case where the electronic device is a server, Figure 4 Some parts of the structure shown (such as the user interface, presentation module, and input processing module) can be omitted. See also Figure 4 , Figure 4 This is a schematic diagram of the structure of the terminal device 400 provided in the embodiments of this application. Figure 4 The terminal device 400 shown includes at least one processor 460, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 4 The general labeled all buses as Bus System 440.

[0127] Processor 460 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor.

[0128] User interface 430 includes one or more output devices that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0129] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 460.

[0130] Memory 450 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0131] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0132] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0133] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0134] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0135] Input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices.

[0136] In some embodiments, the data processing apparatus provided in this application can be implemented in software. Figure 4 A data processing device 455 stored in memory 450 is shown, which may be software in the form of programs and plug-ins, and may include the following software modules:

[0137] The first determining module 4551 is used to determine the deviation and feature value of the first data; the second determining module 4552 is used to determine a compression threshold boundary based on the deviation and the feature value, the compression threshold boundary being used to determine a compression strategy for compressing the second data; the compression module 4553 is used to perform lossy compression on the data in the second data that falls within the compression threshold boundary. These modules are logically related, and therefore can be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0138] In some embodiments, the first determining module 4551 is configured to determine the maximum and minimum values ​​of the data in the first data sequence.

[0139] The characteristic value is determined to include the maximum value and the minimum value of the data.

[0140] In some embodiments, the second determining module 4552 is used to determine that the maximum value of the compression threshold boundary is the sum of the maximum value of the data in the first data sequence and the deviation;

[0141] The minimum value of the compression threshold boundary is determined to be the difference between the minimum value of the data in the first data sequence and the deviation.

[0142] In some embodiments, the first determining module 4551 is configured to determine the average value of the first data sequence; and to determine the feature value as the average value of the first data sequence.

[0143] In some embodiments, the second determining module 4552 is configured to determine the maximum value of the compression threshold boundary as the sum of the product of the average value and the deviation and the average value if the feature value is the average value of the first data sequence;

[0144] The minimum value of the compression threshold boundary is determined to be the difference between the average value and the product of the average value and the deviation.

[0145] In some embodiments, the first determining module 4551 is configured to determine a predicted data sequence based on the first data sequence; the first data sequence is related to historical data.

[0146] The feature value is determined by each predicted data point in the predicted data sequence.

[0147] In some embodiments, the second determining module 4552 is configured to determine, when the feature value includes each predicted data in the predicted data sequence, that the maximum value of the compression threshold boundary is the sum of the product of the predicted data and the deviation and the predicted data;

[0148] The minimum value of the compression threshold boundary is determined to be the difference between the predicted data and the product of the predicted data and the deviation.

[0149] In some embodiments, the second determining module 4552 is configured to generate a residual based on the prediction data sequence when the feature value includes each prediction data in the prediction data sequence, and use the residual as the deviation.

[0150] The maximum value of the compression threshold boundary is determined to be the maximum value in the predicted data sequence minus the product of the maximum value in the predicted data sequence and the residual;

[0151] The minimum value of the compression threshold boundary is determined to be the minimum value in the predicted data sequence minus the product of the minimum value in the predicted data sequence and the residual.

[0152] This application also provides an electronic device, which includes a processor and a memory. The memory is used to store executable instructions. The processor is used to execute the executable instructions stored in the memory to implement the data processing method provided in this application.

[0153] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described in this application.

[0154] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example... Figure 1 and Figure 2 The data processing method is shown.

[0155] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0156] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0157] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0158] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0159] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: The deviation is determined based on the first data sequence; Feature values ​​are determined based on the first data sequence, and the feature values ​​are factors used to determine the compression threshold boundary; The compression threshold boundary is determined based on the deviation and the feature value, and the compression threshold boundary is used to determine the compression strategy for compressing the second data; Lossy compression is performed on the data in the second data that falls within the compression threshold boundary; The deviation includes: a pre-set static deviation, or a dynamic deviation generated based on the first data sequence; The step of determining feature values ​​based on the first data sequence includes: Determine the maximum and minimum values ​​of the data in the first data sequence, and determine that the feature value includes the maximum and minimum values ​​of the data; Alternatively, determine the average value of the first data sequence, and determine the feature value as the average value of the first data sequence; Alternatively, a predicted data sequence can be determined based on the first data sequence; the first data sequence is related to historical data; and the feature value is determined to include each predicted data in the predicted data sequence.

2. The method according to claim 1, characterized in that, Determining the compression threshold boundary based on the deviation and the feature value includes: The maximum value of the compression threshold boundary is determined to be the sum of the maximum value of the data in the first data sequence and the deviation; The minimum value of the compression threshold boundary is determined to be the difference between the minimum value of the data in the first data sequence and the deviation.

3. The method according to claim 1, characterized in that, Determining the compression threshold boundary based on the deviation and the feature value includes: If the feature value is the average value of the first data sequence, then the maximum value of the compression threshold boundary is determined to be the sum of the product of the average value and the deviation and the average value; The minimum value of the compression threshold boundary is determined to be the difference between the average value and the product of the average value and the deviation.

4. The method according to claim 1, characterized in that, Determining the compression threshold boundary based on the deviation and the feature value includes: The feature value includes each predicted data in the predicted data sequence, then the maximum value of the compression threshold boundary is determined to be the sum of the product of the predicted data and the deviation and the predicted data; The minimum value of the compression threshold boundary is determined to be the difference between the predicted data and the product of the predicted data and the deviation.

5. The method according to claim 1, characterized in that, Determining the compression threshold boundary based on the deviation and the feature value includes: If the feature value includes each predicted data in the predicted data sequence, then a residual is generated based on the predicted data sequence, and the residual is used as the deviation; The maximum value of the compression threshold boundary is determined to be the maximum value in the predicted data sequence minus the product of the maximum value in the predicted data sequence and the residual; The minimum value of the compression threshold boundary is determined to be the minimum value in the predicted data sequence minus the product of the minimum value in the predicted data sequence and the residual.

6. A data processing apparatus, characterized in that, The device includes: A first determining module is used to determine a deviation and a feature value based on a first data sequence, wherein the feature value is a factor used to determine the compression threshold boundary; The second determining module is used to determine the compression threshold boundary based on the deviation and the feature value, wherein the compression threshold boundary is used to determine the compression strategy for compressing the second data; The compression module is used to perform lossy compression on the data in the second data that is within the compression threshold boundary; The deviation includes: a pre-set static deviation, or a dynamic deviation generated based on the first data sequence; The first determining module is further configured to determine the maximum and minimum values ​​of data in the first data sequence, and to determine that the feature value includes the maximum and minimum values ​​of data; Alternatively, determine the average value of the first data sequence, and determine the feature value as the average value of the first data sequence; Alternatively, a predicted data sequence can be determined based on the first data sequence; the first data sequence is related to historical data; and the feature value is determined to include each predicted data in the predicted data sequence.

Images