Offline data processing method and device, terminal equipment and storage medium

Abstract

The application provides an offline data processing method and device, terminal equipment and a storage medium, and belongs to the technical field of data processing. The method comprises the following steps: acquiring offline SQL data corresponding to the current fault; compressing the offline SQL data to obtain compressed SQL data; acquiring the type of the current fault, and predicting the duration period of the current fault according to the type of the current fault; predicting the offline SQL volume of the current fault according to the duration period of the current fault; if the duration period of the current fault is less than a preset period and the offline SQL volume of the current fault is less than a preset volume, the compressed SQL data is written into the process memory, otherwise, the compressed SQL data is written into the local disk; wherein the preset volume is a preset percentage of the available volume of the current process memory. The application can reduce the offline SQL data occupation space, and can also take into account the application scenarios of long fault period and short fault period.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an offline data processing method, apparatus, terminal device, and storage medium. Background Technology

[0002] Monitoring products typically prefer time-series databases for periodically storing monitoring data, such as InfluxDB. In scenarios where applications and data are stored separately or deployed in clusters, network fluctuations, node offline failures, and other faults can all cause data write failures to the database.

[0003] To address this situation, relevant technologies typically use incremental backups to store failed SQL data on the local disk, reloading and executing this offline SQL data after the fault is recovered. However, because monitoring products can connect to millions of measurement points and require writing hundreds of millions of data entries per day, and the number of SQL statements and data entries is often linearly related, the size of the local offline SQL will grow rapidly in cases of long fault cycles, resulting in significant disk space consumption. Furthermore, SQL backups only support a single mode of local disk file I / O (read and write), which cannot accommodate application scenarios with both long and short fault cycles. Summary of the Invention

[0004] This invention provides an offline data processing method, apparatus, terminal device, and storage medium to solve the problems of existing technologies that write offline SQL data to local disks, which occupy a large amount of disk space and cannot accommodate application scenarios with both long and short failure cycles.

[0005] In a first aspect, embodiments of the present invention provide an offline data processing method, comprising: Retrieve the offline SQL data corresponding to this failure; Compress the offline SQL data to obtain compressed SQL data; Obtain the fault type and predict the fault duration based on the fault type; Based on the duration of this failure, predict the offline SQL volume of this failure; If the duration of this fault is less than the preset period and the offline SQL volume of this fault is less than the preset volume, then the compressed SQL data will be written to the process memory; otherwise, the compressed SQL data will be written to the local disk. The preset volume is a preset percentage of the current available memory volume of the process.

[0006] In one possible implementation, the duration of the current fault is predicted based on the fault type, including: Obtain the fault duration period of multiple historical faults with the same fault type as the current fault; Calculate the average and standard deviation of the failure duration for multiple historical failures; Based on the mean and standard deviation, predict the duration of this fault.

[0007] In one possible implementation, the offline SQL volume for this failure is predicted based on the duration of the failure, including: Multiply the duration of this failure by a preset scaling factor to obtain the offline SQL volume of this failure; The preset ratio coefficient is determined based on the offline SQL volume and fault duration of multiple historical faults of the same type as the current fault.

[0008] In one possible implementation, the offline SQL data is compressed to obtain compressed SQL data, including: Parse offline SQL data to obtain various feature information and data information from the offline SQL data; Determine whether each feature information has been stored in the preset dictionary; For feature information not stored in the preset dictionary, generate the code for the feature information and store the correspondence between the feature information and the code for the feature information in the preset dictionary; For feature information that has been stored in a preset dictionary, retrieve the corresponding code for that feature information from the preset dictionary; A compression header is generated based on the encoding of each feature information, a compressed body is generated based on each data information, and the compression header and compressed body are combined to generate compressed SQL data.

[0009] In one possible implementation, parsing offline SQL data includes: Parse offline SQL data using a lazy loading method.

[0010] In one possible implementation, the feature information includes at least one of the following: database name, table name, label name, and field name; The data information includes at least one of the tag values ​​corresponding to the tag name and the field values ​​corresponding to the field name.

[0011] In one possible implementation, the predefined dictionary includes a first-level dictionary, a second-level dictionary, and a third-level dictionary; The first-level dictionary includes the correspondence between database names and their corresponding codes; The second-level dictionary includes the correspondence between data table names and their corresponding codes; The first level of the three-level dictionary includes the encoding of database names; the second level includes the encoding of data table names; and the third level includes at least one of the following: the mapping between tag names and their corresponding encodings, and the mapping between field names and their corresponding encodings.

[0012] Secondly, embodiments of the present invention provide an offline data processing apparatus, comprising: The acquisition module is used to obtain the offline SQL data corresponding to this fault. The compression module is used to compress offline SQL data to obtain compressed SQL data; The cycle prediction module is used to obtain the fault type and predict the duration of the fault based on the fault type. The SQL volume prediction module is used to predict the offline SQL volume of this fault based on the duration of this fault. The dynamic persistence module is used to write compressed SQL data into the process memory if the duration of the current fault is less than a preset period and the offline SQL volume of the current fault is less than a preset volume; otherwise, it writes compressed SQL data into the local disk. The preset volume is a preset percentage of the available memory volume of the current process.

[0013] Thirdly, embodiments of the present invention provide a terminal device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the offline data processing method as described in the first aspect or any possible implementation thereof.

[0014] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the offline data processing method as described in the first aspect or any possible implementation thereof.

[0015] This invention provides an offline data processing method, apparatus, terminal device, and storage medium. The method first compresses offline SQL data to obtain compressed SQL data, reducing the space occupied by the offline SQL data and avoiding excessive space usage. Then, based on the type of the current fault, the duration of the current fault is predicted, and based on the duration of the current fault, the volume of the offline SQL data for the current fault is predicted. Based on the duration of the current fault and the volume of the offline SQL data for the current fault, it is determined whether to write the compressed SQL data to the process memory or the local disk. The optimal storage medium can be automatically selected according to the fault scenario, achieving a good balance between space usage and time efficiency, thus improving the applicability of the method. Specifically, when the duration of the current fault is short and the volume of the offline SQL data for the current fault is small, the compressed SQL data is written to the process memory; otherwise, the compressed SQL data is written to the local disk. This avoids disk I / O operations in small data volume scenarios, significantly improving data read and write speed without consuming excessive memory, avoiding memory resource waste. In large data volume scenarios, it fully utilizes the advantages of disk capacity while ensuring data security and preventing data loss. Therefore, it can accommodate application scenarios with both long and short fault durations. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the implementation of an offline data processing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a three-level dictionary provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of two first-level dictionaries provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of compressed SQL data provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an offline data processing device provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0020] As mentioned earlier, related technologies typically use incremental backups to store failed SQL data on the local disk, and then reload and execute this offline SQL data after the fault is recovered. However, because monitoring products can connect to millions of test points and require writing hundreds of millions of data entries per day, and the number of SQL statements and data entries is often linearly related, the size of the local offline SQL will grow rapidly in cases of long fault cycles, resulting in significant disk space consumption.

[0021] In addition, SQL backup only supports a single method of local disk file I / O (read and write). This method is one of the best solutions in scenarios with long failure cycles (meaning massive amounts of offline SQL data), but in scenarios with short failure cycles (meaning less SQL data), the performance loss caused by file I / O is significant, making it impossible to take into account both long and short failure cycles.

[0022] To address the aforementioned issues, this application provides an offline data processing method for processing offline SQL data during a fault, thereby resolving the technical problems described above.

[0023] See Figure 1 The diagram illustrates the implementation flowchart of the offline data processing method provided in the embodiment of the present invention. The execution subject of the offline data processing method is a terminal device, which can be a computer or a server or other devices.

[0024] See Figure 1 The method is described in detail below: In S101, retrieve the offline SQL data corresponding to this fault.

[0025] As mentioned earlier, network fluctuations or node offline failures can cause data write failures to the database. In such cases, incremental backups are needed to store the failed SQL data. This failed SQL data during the failure period is called offline SQL data. Once the failure is resolved, the offline SQL data is reloaded and executed. Therefore, when a failure occurs, it is necessary to retrieve the offline SQL data corresponding to that failure. Since offline SQL data may be continuously generated during the failure period, every time new offline SQL data is generated, it should be retrieved.

[0026] The method provided in this application can be applied to any database, such as time-series databases (InfluxDB, Prometheus, etc.), relational databases (MySQL, PostgreSQL, etc.), or NoSQL databases (MongoDB, Redis, etc.), etc.

[0027] For different types of databases, the offline SQL data mentioned above corresponds to the offline SQL data specific to that database type. For example, for InfluxDB, the offline SQL data includes offline InfluxQL data, and so on.

[0028] In S102, the offline SQL data is compressed to obtain compressed SQL data.

[0029] In order to reduce the size of offline SQL data and its space occupation, this application embodiment compresses the offline SQL data to obtain compressed SQL data.

[0030] Since new offline SQL data may be continuously generated during this failure period, the compression process in this embodiment may continue indefinitely during this failure. For example, during this failure period, offline SQL data can be acquired periodically, and the acquired offline SQL data can be compressed each time it is acquired.

[0031] In S103, the fault type is obtained, and the duration of the fault is predicted based on the fault type.

[0032] Each failure has a corresponding failure type, such as the aforementioned network fluctuation failure, node offline failure, etc. Different failure types correspond to different failure durations. Therefore, the embodiments of this application can predict the duration of a failure based on the current failure type.

[0033] The duration of this fault can be understood as the duration of this fault, that is, the length of time this fault lasts.

[0034] In S104, the offline SQL volume of this failure is predicted based on the duration of this failure.

[0035] Since offline SQL data will continue to be generated during the duration of this failure, the embodiments of this application can predict the volume of offline SQL data during this failure based on the duration of the failure.

[0036] The offline SQL volume in this failure can be understood as the space occupied by storing all offline SQL data related to this failure. Therefore, the offline SQL volume in this failure can also be referred to as the space occupied by the offline SQL data in this failure.

[0037] In some possible implementations, the offline SQL volume of this failure is positively correlated with the duration of this failure. That is, the longer the duration of this failure, the larger the offline SQL volume; the shorter the duration of this failure, the smaller the offline SQL volume.

[0038] In S105, if the duration of the current fault is less than the preset period and the offline SQL volume of the current fault is less than the preset volume, the compressed SQL data will be written to the process memory; otherwise, the compressed SQL data will be written to the local disk. The preset volume is a preset percentage of the available volume of the current process memory.

[0039] Compressed SQL data can be stored on the local disk or in process memory. The former saves memory but has low read efficiency and is suitable for scenarios with long fault durations and large offline SQL volumes; the latter consumes some memory but has very high read efficiency and is suitable for scenarios with short fault durations and small offline SQL volumes.

[0040] Therefore, this application embodiment sets two thresholds: a preset period and a preset volume. When the duration of the current fault is less than the preset period and the offline SQL volume of the current fault is less than the preset volume, it indicates that the duration of the current fault is short and the offline SQL volume of the current fault is small. In this case, the compressed SQL data can be written to the process memory, i.e., memory persistence is used. Otherwise, the compressed SQL data is written to the local disk. That is, when the duration of the current fault is greater than or equal to the preset period, or the offline SQL volume of the current fault is greater than or equal to the preset volume, it indicates that the duration of the current fault is long or the offline SQL volume of the current fault is large. In this case, the compressed SQL data can be written to the local disk, i.e., disk persistence is used.

[0041] Process memory refers to the memory allocated to the current process. However, it's impossible for all process memory to be used to store compressed SQL data, as this would affect the normal operation of other functions of the process. Therefore, the preset volume is a preset percentage of the available memory volume of the current process. The preset percentage is greater than 0 and less than 100%. Furthermore, to avoid affecting the normal use of other functions of the process, and to reserve buffer space so that even if the offline SQL volume suddenly increases, it won't crash the entire process, the preset percentage is usually not too large; for example, it could be 20% or 30%, etc.

[0042] The specific value of the preset period can be set according to actual needs. It is necessary to ensure that when the duration of the current fault is less than the preset period, the compressed SQL data will be written into the process memory without affecting the normal operation of various functions in the process.

[0043] The judgment process in this application embodiment can achieve a good balance between space usage and time efficiency in SQL data backup.

[0044] This application first compresses the offline SQL data to obtain compressed SQL data, reducing the space occupied by the offline SQL data. Whether stored in memory or on the local disk, it avoids excessive space consumption. Then, based on the type of the current fault, the duration of the current fault is predicted, and based on the duration of the current fault, the volume of the offline SQL data for the current fault is predicted. Based on the duration of the current fault and the volume of the offline SQL data for the current fault, it is determined whether to write the compressed SQL data to the process memory or the local disk. The optimal storage medium can be automatically selected according to the fault scenario, achieving a good balance between space consumption and time efficiency, improving the applicability of the method, and ensuring that the SQL backup mechanism can dynamically adapt to the fault scenario and resource configuration. Specifically, when the duration of the current fault is short and the volume of the offline SQL data for the current fault is small, the compressed SQL data is written to the process memory; otherwise, the compressed SQL data is written to the local disk. This avoids disk I / O operations in small data volume scenarios, significantly improving data read and write speed, without consuming too much memory and avoiding memory resource waste. In large data volume scenarios, it fully utilizes the advantages of disk capacity and ensures data security and no data loss, thus accommodating both application scenarios with long and short fault periods.

[0045] In some embodiments, the above-mentioned prediction of the duration of the current fault based on the current fault type may include: Obtain the fault duration period of multiple historical faults with the same fault type as the current fault; Calculate the average and standard deviation of the failure duration for multiple historical failures; Based on the mean and standard deviation, predict the duration of this fault.

[0046] In this embodiment, the prediction of the duration of the current fault, based on the fault type, is made using historical fault data. Historical fault data can include all fault data prior to this fault or fault data within a preset time period prior to this fault, but the number of fault data entries must be greater than or equal to a first preset quantity; that is, a certain scale of historical fault data is required for prediction. The first preset quantity can be set according to actual needs and is not specifically limited here.

[0047] Historical failure data can include at least the failure type, failure duration, and offline SQL volume for each failure. Each time a failure occurs, the failure data is recorded as historical failure data for subsequent predictions.

[0048] In this embodiment, the failure duration periods of multiple historical faults of the same type as the current fault are first obtained from historical fault data. These multiple historical faults can be all historical faults of the same type as the current fault in the historical fault data, or they can be a second preset number of historical faults of the same type as the current fault in the historical fault data. The second preset number is greater than or equal to 3 and less than or equal to a first preset number.

[0049] Based on the calculation formulas in related technologies, the average and standard deviation of the failure duration of the above-mentioned historical failures are calculated, and then the duration of the current failure is predicted based on the calculated average and standard deviation.

[0050] For example, the duration of this fault can be predicted using a conservative normal distribution strategy. That is, the duration of this fault is T = μ + 1.3σ; where μ is the mean and σ is the standard deviation.

[0051] In one possible implementation, if the number of fault data entries in the historical fault data does not reach the first preset number, the fault duration period can be predicted based on expert experience and the current fault type.

[0052] The embodiments of this application predict the duration of the current fault based on historical fault data, which can ensure the accuracy of the prediction. By predicting the duration of the current fault using the average value and standard deviation, both typical cases and fluctuation ranges can be taken into account, further improving the accuracy of the prediction.

[0053] In some embodiments, predicting the offline SQL volume for the current fault based on the duration of the current fault includes: Multiply the duration of this failure by a preset scaling factor to obtain the offline SQL volume of this failure; The preset ratio coefficient is determined based on the offline SQL volume and fault duration of multiple historical faults of the same type as the current fault.

[0054] The relationship between the offline SQL volume V of this fault, the number of measurement points n, the fault duration T, and the archiving period t is as follows: V = k * n (T / t). Here, k is a preset coefficient that can be set according to actual needs. A measurement point can be understood as a data acquisition point or a data acquisition quantity. For example, a device may have many physical quantities or state quantities that need to be monitored, acquired, or controlled; each of these quantities can be called a measurement point.

[0055] In a stable operating system, the number of measurement points n and the archiving period t can be considered constants. Therefore, the above relationship is equivalent to: V = k' * T. Here, k' is the converted preset coefficient, which is the preset proportional coefficient mentioned above. Its value can be approximated based on historical fault data.

[0056] For example, the ratio of the offline SQL volume to the fault duration of multiple historical faults of the same type as the current fault can be calculated to obtain the coefficient values ​​corresponding to the multiple historical faults, and the average value of the coefficient values ​​corresponding to the multiple historical faults can be calculated as a preset proportional coefficient; or, a first average value of the offline SQL volume of the multiple historical faults can be calculated, a second average value of the fault duration of the multiple historical faults can be calculated, and the ratio of the first average value and the second average value can be used as a preset proportional coefficient.

[0057] After determining the value of the preset ratio coefficient, multiplying the duration of this fault by the preset ratio coefficient will give the offline SQL volume of this fault.

[0058] This application's embodiments combine similar fault data from historical fault data to predict the offline SQL volume of the current fault, which can avoid the blindness of setting a fixed ratio coefficient and continuously optimize the prediction accuracy based on the accumulation of historical fault data.

[0059] In related technologies, raw SQL statements often contain many common, structured, redundant statements or keywords, resulting in low information density (or an inability to express complete data information within a limited character length). This leads to additional network bandwidth and system resource (CPU and memory resources, etc.) consumption during SQL backup and restore. To address this issue, embodiments of this application improve the information density of SQL data through compression.

[0060] In some embodiments, the above-described compression of offline SQL data to obtain compressed SQL data includes: Parse offline SQL data to obtain various feature information and data information from the offline SQL data; Determine whether each feature information has been stored in the preset dictionary; For feature information not stored in the preset dictionary, generate the code for the feature information and store the correspondence between the feature information and the code for the feature information in the preset dictionary; For feature information that has been stored in a preset dictionary, retrieve the corresponding code for that feature information from the preset dictionary; A compression header is generated based on the encoding of each feature information, a compressed body is generated based on each data information, and the compression header and compressed body are combined to generate compressed SQL data.

[0061] In some embodiments, the feature information includes at least one of database name, table name, tag name, and field name; The data information includes at least one of the tag values ​​corresponding to the tag name and the field values ​​corresponding to the field name.

[0062] In some embodiments, the preset dictionary includes a first-level dictionary, a second-level dictionary, and a third-level dictionary; The first-level dictionary includes the correspondence between database names and their corresponding codes; The second-level dictionary includes the correspondence between data table names and their corresponding codes; The first level of the three-level dictionary includes the encoding of database names; the second level includes the encoding of data table names; and the third level includes at least one of the following: the mapping between tag names and their corresponding encodings, and the mapping between field names and their corresponding encodings.

[0063] Offline SQL data is a structured language that, in addition to valid data information (variable parts), also includes fixed characteristic information such as database name, table name (measurement), tag name (tag), and field name (invariant or common parts).

[0064] This application embodiment can parse offline SQL data, identify and extract feature information from the offline SQL data, and construct a three-level dictionary. The structure of the three-level dictionary is as follows: Figure 2 As shown in the diagram, the keys of the first-level dictionary in the three-level dictionary represent the codes corresponding to the database names; the keys of the second-level dictionary represent the codes corresponding to the table names; and the keys of the third-level dictionary represent at least one of the codes for tag names and field names. The values ​​of the third-level dictionary represent at least one of the codes for tag names and field names. Therefore, the third-level dictionary includes at least one of the mappings between tag names and their corresponding codes, as well as the mappings between field names and their corresponding codes, and is sorted in order. Furthermore, the three-level dictionary also indicates which table a tag name or field name belongs to, and which database that table belongs to.

[0065] For example, the encoding of the database name can be a natural number starting from 1 and incrementing by 1; the encoding of the data table name can be a natural number starting from 1 and incrementing by 1, prefixed with the encoding of the database to which it belongs; the encoding of the tag name can be a natural number starting from 1 and incrementing by 1, prefixed with 0; and the encoding of the field name can be a natural number starting from 1 and incrementing by 1, prefixed with 1.

[0066] Different database names have different encodings; different table names have different encodings; different tag names have different encodings; different field names have different encodings.

[0067] This application embodiment also maintains two first-level dictionaries (a first-level dictionary and a second-level dictionary), which are used to manage the correspondence (mapping relationship) between database names and codes and the correspondence (mapping relationship) between data table names and codes in the third-level dictionaries, respectively. For example... Figure 3 As shown, Figure 3 The database name in the first-level dictionary is encoded as the key of the first-level dictionary in the third-level dictionary, which also stores the database name corresponding to that encoding. Figure 3 The encoding corresponding to the data table name in the second-level dictionary is the key of the second-level dictionary in the third-level dictionary, which also stores the data table name corresponding to that encoding.

[0068] This embodiment of the application parses offline SQL data to obtain various feature information and data information from the offline SQL data; it then determines whether each feature information has been stored in a preset dictionary. If it has been stored, the corresponding code for that feature information is directly obtained from the preset dictionary; if it has not been stored, the code for that feature information is generated according to the aforementioned rules, and the correspondence between the feature information and its code is stored in the preset dictionary. After obtaining the codes for each feature information, a compression header can be generated based on the codes for each feature information, and a compressed body can be generated based on each data information. The compression header and compressed body are then combined to generate compressed SQL data. The compressed body can be generated by concatenating the data information according to the key order of the third-level dictionary in a three-level dictionary, using the concatenation operator "|".

[0069] For example, the compressed SQL data is as follows Figure 4 As shown, it includes a compression header and a compression body. The compression header includes database encoding, data table encoding, and tag and / or field encoding, while the compression body includes concatenated data information.

[0070] In this case, the length of the encoding of the feature information is less than the length of the feature information itself.

[0071] In some embodiments, parsing offline SQL data as described above includes: Parse offline SQL data using a lazy loading method.

[0072] Lazy loading means loading data or creating objects only when needed, rather than loading them in advance during initialization. This application embodiment uses lazy loading to parse offline SQL data, storing new feature information and its corresponding codes into a preset dictionary only when new feature information is encountered, instead of storing all potentially useful feature information and their corresponding codes into the preset dictionary at the beginning.

[0073] This application's embodiments, through feature information extraction and encoding, can represent the common parts of tens or even hundreds of millions of SQL data entries using only a fixed number of data dictionaries. Furthermore, the dictionary size remains largely unchanged with the increase in the number of SQL data entries, ultimately achieving significant compression of offline SQL data size. This drastically reduces space usage, saving over 60% and ensuring that even low-spec server resources can handle the synchronization of massive amounts of SQL data. By compressing the SQL data, the information density of the compressed SQL data is increased, reducing network bandwidth and system resource consumption during SQL backup and restore processes.

[0074] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0075] Figure 5 A schematic diagram of the offline data processing device provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below: like Figure 5 As shown, the offline data processing device 30 includes: an acquisition module 31, a compression module 32, a period prediction module 33, an SQL volume prediction module 34, and a dynamic persistence module 35.

[0076] Module 31 is used to obtain the offline SQL data corresponding to this fault; Compression module 32 is used to compress offline SQL data to obtain compressed SQL data; The cycle prediction module 33 is used to obtain the current fault type and predict the duration of the current fault based on the current fault type. The SQL volume prediction module 34 is used to predict the offline SQL volume of this fault based on the duration of this fault. The dynamic persistence module 35 is used to write the compressed SQL data into the process memory if the duration of the current fault is less than the preset period and the offline SQL volume of the current fault is less than the preset volume; otherwise, it writes the compressed SQL data into the local disk. The preset volume is a preset percentage of the available volume of the current process memory.

[0077] In one possible implementation, the period prediction module 33 predicts the duration of the current fault based on the current fault type, including: Obtain the fault duration period of multiple historical faults with the same fault type as the current fault; Calculate the average and standard deviation of the failure duration for multiple historical failures; Based on the mean and standard deviation, predict the duration of this fault.

[0078] In one possible implementation, the SQL volume prediction module 34 is specifically used for: Multiply the duration of this failure by a preset scaling factor to obtain the offline SQL volume of this failure; The preset ratio coefficient is determined based on the offline SQL volume and fault duration of multiple historical faults of the same type as the current fault.

[0079] In one possible implementation, compression module 32 is specifically used for: Parse offline SQL data to obtain various feature information and data information from the offline SQL data; Determine whether each feature information has been stored in the preset dictionary; For feature information not stored in the preset dictionary, generate the code for the feature information and store the correspondence between the feature information and the code for the feature information in the preset dictionary; For feature information that has been stored in a preset dictionary, retrieve the corresponding code for that feature information from the preset dictionary; A compression header is generated based on the encoding of each feature information, a compressed body is generated based on each data information, and the compression header and compressed body are combined to generate compressed SQL data.

[0080] In one possible implementation, the compression module 32 parses the offline SQL data, including: Parse offline SQL data using a lazy loading method.

[0081] In one possible implementation, the feature information includes at least one of the following: database name, table name, label name, and field name; The data information includes at least one of the tag values ​​corresponding to the tag name and the field values ​​corresponding to the field name.

[0082] In one possible implementation, the predefined dictionary includes a first-level dictionary, a second-level dictionary, and a third-level dictionary; The first-level dictionary includes the correspondence between database names and their corresponding codes; The second-level dictionary includes the correspondence between data table names and their corresponding codes; The first level of the three-level dictionary includes the encoding of database names; the second level includes the encoding of data table names; and the third level includes at least one of the following: the mapping between tag names and their corresponding encodings, and the mapping between field names and their corresponding encodings.

[0083] Figure 6 This is a schematic diagram of a terminal device provided in an embodiment of the present invention. Figure 6 As shown, the terminal device 4 in this embodiment includes a processor 40 and a memory 41. The memory 41 is used to store a computer program 42, and the processor 40 is used to call and run the computer program 42 stored in the memory 41 to execute the steps in the above-described offline data processing method embodiments. Alternatively, the processor 40 is used to call and run the computer program 42 stored in the memory 41 to implement the functions of each module / unit in the above-described device embodiments.

[0084] For example, the computer program 42 may be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 42 in the terminal device 4.

[0085] The terminal device 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device 4 and does not constitute a limitation on terminal device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0086] The processor 40 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0087] The memory 41 can be an internal storage unit of the terminal device 4, such as a hard disk or memory of the terminal device 4. The memory 41 can also be an external storage device of the terminal device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 4. Furthermore, the memory 41 can include both internal and external storage units of the terminal device 4. The memory 41 is used to store the computer program and other programs and data required by the terminal device. The memory 41 can also be used to temporarily store data that has been output or will be output.

[0088] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the offline data processing methods described above.

[0089] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the offline data processing methods described above.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0091] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0092] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0093] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various offline data processing method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0097] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An offline data processing method, characterized in that, include: Retrieve the offline SQL data corresponding to this failure; The offline SQL data is compressed to obtain compressed SQL data; Obtain the current fault type and predict the duration of the current fault based on the current fault type; Based on the duration of this failure, predict the offline SQL volume of this failure; If the duration of the current fault is less than a preset period, and the offline SQL volume of the current fault is less than a preset volume, then the compressed SQL data is written to the process memory; otherwise, the compressed SQL data is written to the local disk. The preset volume is a preset percentage of the current available memory volume of the process.

2. The offline data processing method according to claim 1, characterized in that, The step of predicting the duration of the current fault based on the current fault type includes: Obtain the fault duration period of multiple historical faults with the same fault type as the current fault; Calculate the average and standard deviation of the fault duration periods for the multiple historical faults; Based on the average value and the standard deviation, the duration of this fault is predicted.

3. The offline data processing method according to claim 1, characterized in that, The prediction of the offline SQL volume for this fault based on its duration includes: Multiply the duration of this failure by a preset scaling factor to obtain the offline SQL volume of this failure; The preset ratio coefficient is determined based on the offline SQL volume and fault duration of multiple historical faults of the same type as the current fault.

4. The offline data processing method according to any one of claims 1 to 3, characterized in that, The step of compressing the offline SQL data to obtain compressed SQL data includes: Parse the offline SQL data to obtain various feature information and data information from the offline SQL data; Determine whether each of the aforementioned feature information has been stored in a preset dictionary; For feature information not stored in the preset dictionary, generate an encoding for the feature information and store the correspondence between the feature information and its encoding in the preset dictionary; For feature information already stored in the preset dictionary, the corresponding code for the feature information is retrieved from the preset dictionary; A compression header is generated based on the encoding of each feature information, a compressed body is generated based on each data information, and the compression header and the compressed body are combined to generate compressed SQL data.

5. The offline data processing method according to claim 4, characterized in that, The parsing of the offline SQL data includes: The offline SQL data is parsed using a lazy loading method.

6. The offline data processing method according to claim 4, characterized in that, The feature information includes at least one of the following: database name, table name, tag name, and field name; The data information includes at least one of the tag values ​​corresponding to the tag name and the field values ​​corresponding to the field name.

7. The offline data processing method according to claim 4, characterized in that, The preset dictionary includes a first-level dictionary, a second-level dictionary, and a third-level dictionary; The first-level dictionary includes the correspondence between database names and their corresponding codes; The second-level dictionary includes the correspondence between data table names and their corresponding codes; The first level of the three-level dictionary includes the encoding of database names, the second level of the three-level dictionary includes the encoding of data table names, and the third level of the three-level dictionary includes at least one of the following: the correspondence between tag names and corresponding encodings, and the correspondence between field names and corresponding encodings.

8. An offline data processing device, characterized in that, include: The acquisition module is used to obtain the offline SQL data corresponding to this fault. A compression module is used to compress the offline SQL data to obtain compressed SQL data; The cycle prediction module is used to obtain the current fault type and predict the duration of the current fault based on the current fault type. The SQL volume prediction module is used to predict the offline SQL volume of this fault based on the duration of this fault. The dynamic persistence module is used to write the compressed SQL data into the process memory if the duration of the current fault is less than a preset period and the offline SQL volume of the current fault is less than a preset volume; otherwise, it writes the compressed SQL data into the local disk. The preset volume is a preset percentage of the available memory volume of the current process.

9. A terminal device, characterized in that, It includes a memory and a processor, the memory being used to store a computer program, and the processor being used to call and run the computer program stored in the memory to perform the offline data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the offline data processing method as described in any one of claims 1 to 7.

Images