Power intelligent terminal-oriented database high-performance storage system

Abstract

The invention relates to a database high-performance storage system for an electric power intelligent terminal, which is characterized by comprising a database function unit, a data cleaning unit based on a lightweight rule engine and an asymmetric data compression unit based on bit back coding, the database function unit comprises a data access engine and an OpenAPI interface; the data cleaning unit based on the lightweight rule engine comprises a data deduplication module, a rule configuration module, a rule loading module, a regularization verification module and a mode matching module adopting a mode matching algorithm mainly based on a regular expression. The asymmetric data compression unit based on bit-back coding introduces an asymmetric data compression technology based on bit-back coding to perform data compression on the basis of Huffman coding and arithmetic coding. According to the invention, the storage and access performance of the data is improved in an intelligent terminal environment with limited resources.

Description

Technical Field

[0001] The present invention relates to the field of database high-performance storage, and in particular to a database high-performance storage system for power intelligent terminals. Background Art

[0002] Smart terminals are highly integrated devices with multiple functions such as data processing, communication, and control. They are an important part of the new power system and undertake most of the data generation and collection tasks. These terminals usually use embedded system technology to combine software and hardware to achieve various power applications. The widespread use of smart terminals has greatly improved the real-time and reliability of data processing in IoT environments such as new power systems. However, this also puts higher requirements on the database system deployed on smart terminals.

[0003] With the development of database technology, especially the widespread application of embedded databases, databases have become an indispensable part of smart terminals. Such databases need to have efficient data storage and processing capabilities to meet the challenges of smart terminals in resource-constrained environments. Specifically, databases deployed on smart terminals must have high concurrency processing, low-latency response, multi-language support, powerful data compression and storage optimization capabilities to meet the complex needs of power smart terminals.

[0004] However, in practical applications, power smart terminals often face a series of problems, such as large data volume, diverse data types, limited storage space, and limited processing performance. These problems make it difficult for traditional database technology to operate efficiently in the smart terminal environment. Therefore, the requirements of power smart terminals on databases have become more stringent. Not only do they need to support efficient storage and processing of massive data, but they also need to have fast data access and powerful data compression capabilities to maximize the use of limited system resources. Summary of the invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide a high-performance database storage system for power smart terminals, so as to improve the storage and access performance of data in a resource-constrained smart terminal environment, and provide strong support for the application of power smart terminals in new power systems.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A high-performance database storage system for electric power intelligent terminals, characterized in that it includes a database functional unit, a data cleaning unit based on a lightweight rule engine, and an asymmetric data compression unit based on bit-back coding; the database functional unit includes a data access engine and an OpenAPI interface, the data access engine implements support for C, C++, Go, and Java programming languages ​​through SQLAPI extension, and the OpenAPI interface provides a unified SQL data operation service to allow users to set the data transmission protocol type between the client and the server according to requirements; the data cleaning unit based on the lightweight rule engine includes a data deduplication module, a rule configuration module, a rule loading module, a regularization verification module, and a pattern matching module using a pattern matching algorithm based on regular expressions; the asymmetric data compression unit based on bit-back coding introduces an asymmetric data compression technology based on bit-back coding for data compression on the basis of Huffman coding and arithmetic coding.

[0008] Furthermore, the data deduplication module uses Bloom filter as a deduplication rule, and replaces it with Roaring Bitmap, hash function, fingerprint recognition or similarity measurement according to different application scenarios of smart terminals.

[0009] Furthermore, the rule configuration module allows the operator to customize the complexity and application scenarios of the rules according to actual needs, ensuring that the system can run under different resource conditions; the rule loading module is used to preload or precompile complex pattern matching rules; the rule verification module checks each piece of data one by one according to the pre-set rule set to ensure that it meets the specified format requirements. The verification content includes data type check (such as integer, floating point number, string, etc.), data range verification (such as numerical range, date range, etc.), and data dependency verification (such as timestamp order, consistency of associated fields, etc.).

[0010] Furthermore, the working mode of the data cleaning unit based on the lightweight rule engine is as follows:

[0011] First, the system uses the data deduplication module to deduplicate the collected data, remove duplicate data, reduce storage space usage, and improve data quality;

[0012] Subsequently, the rule engine reads and loads the predefined rule configuration information, including obtaining the user-defined rule set from the rule configuration module and precompiling or preloading it through the rule loading module to improve the efficiency of subsequent data processing;

[0013] Finally, the system applies a regularized verification module and a pattern matching module to further process the data. The regularized verification module ensures that the data conforms to the predetermined format and dependencies, while the pattern matching module identifies and processes anomalies and specific patterns in the data to ensure the accuracy and consistency of the data.

[0014] Further, the asymmetric data compression technology based on bit-back encoding is as follows:

[0015] S1: Use the inference model to compress data through bit-back encoding, let X = (X1, X2, ..., X N ) is the data in the database, and each data X2 is taken from the finite dictionary table. The minimum description length of the original data X is subject to the entropy H(X) = E[-logp data (X)]; using the parameterized probability function pθ(X) for modeling, the original data X is modeled as a marginal distribution with latent variables p(z), that is:

[0016]

[0017] S2: Introducing the approximate posterior distribution Encode the original data X using latent variables, with a coding length of E[-logp θ (X|Z)-logp(z)], and the bit-back coding principle shows that using Encode the original data X, so the expected increase in code length when encoding data X is expressed as:

[0018]

[0019] S3: By encoding the message in a LIFO (last in, first out) manner, compared to arithmetic coding in a FIFO (first in, first out) manner, asymmetric data systems can implement bit-back coding by using latent variables to link bits;

[0020] S4: Based on the probabilistic modeling method, the posterior distribution and the likelihood function are jointly learned given the latent variable z; when the true posterior distribution p θ When (Z|X) is difficult to solve, we can optimize the approximate posterior The objective function of the encoding is Optimizing the approximate posterior It can be understood as minimizing the true posterior p θ (Z|X) and approximate posterior The Kullback–Leibler (KL) divergence between them; by minimizing the KL distance, the description length of the original data X is reduced to close to the entropy of the source message;

[0021] S5: Data encoding and decoding based on BBC-ANDC.

[0022] Further, the data encoding and decoding based on BBC-ANDC are as follows:

[0023] coding:

[0024] a Using the approximate posterior from step 3.2 Decode Z0;

[0025] b uses the likelihood function p θ (X|Z0) encodes X0;

[0026] c Encode Z0 using the prior distribution p(Z);

[0027] decoding:

[0028] a Decode Z0 using the prior distribution p(Z);

[0029] b uses the likelihood function p θ (X|Z0) decodes X0;

[0030] cUsing approximate posterior Code Z0.

[0031] The present invention has the following beneficial effects:

[0032] 1. The present invention improves the data storage and access performance in a resource-constrained intelligent terminal environment, and provides strong support for the application of power intelligent terminals in new power systems;

[0033] 2. The present invention can effectively encode data by customizing the probability distribution of learning data, and decode potential variables from streaming codes using approximate posterior distribution through bit-backoff coding, thereby shortening the encoding length of the data. Compared with the original data encoding compression method, BBC-ANDC significantly improves the data compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a system architecture diagram of the present invention;

[0035] Figure 2 A rule engine for a smart terminal database in one embodiment of the present invention;

[0036] Figure 3 This is the working mode of the lightweight rule engine in one embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0038] refer to Figure 1 In this embodiment, a database high-performance storage system for electric power intelligent terminals is provided, characterized in that it includes a database functional unit, a data cleaning unit based on a lightweight rule engine, and an asymmetric data compression unit based on bit-back coding; the database functional unit includes a data access engine and an OpenAPI interface, the data access engine implements support for C, C++, Go, and Java programming languages ​​through SQLAPI extension, and the OpenAPI interface provides a unified SQL data operation service to allow users to set the data transmission protocol type between the client and the server according to requirements; the data cleaning unit based on the lightweight rule engine includes a data deduplication module, a rule configuration module, a rule loading module, a regularization verification module, and a pattern matching module using a pattern matching algorithm based on regular expressions; the asymmetric data compression unit based on bit-back coding introduces an asymmetric data compression technology based on bit-back coding on the basis of Huffman coding and arithmetic coding.

[0039] In this embodiment, the data deduplication module uses Bloom filter as the deduplication rule, and replaces it with Roaring Bitmap, hash function, fingerprint recognition or similarity measurement according to different application scenarios of the smart terminal.

[0040] In this embodiment, the rule configuration module allows the operator to customize the complexity and application scenarios of the rules according to actual needs, ensuring that the system can run under different resource conditions; the rule loading module is used to preload or precompile complex pattern matching rules; the rule verification module checks each piece of data one by one according to a pre-set rule set to ensure that it meets the specified format requirements. The verification content includes data type check (such as integer, floating point number, string, etc.), data range verification (such as numerical range, date range, etc.), and data dependency verification (such as timestamp order, consistency of associated fields, etc.).

[0041] refer to Figure 2 and 3 In this embodiment, the working mode of the data cleaning unit based on the lightweight rule engine is as follows:

[0042] First, the system uses the data deduplication module to deduplicate the collected data, remove duplicate data, reduce storage space usage, and improve data quality;

[0043] Subsequently, the rule engine reads and loads the predefined rule configuration information, including obtaining the user-defined rule set from the rule configuration module and precompiling or preloading it through the rule loading module to improve the efficiency of subsequent data processing;

[0044] Finally, the system applies a regularized verification module and a pattern matching module to further process the data. The regularized verification module ensures that the data conforms to the predetermined format and dependencies, while the pattern matching module identifies and processes anomalies and specific patterns in the data to ensure the accuracy and consistency of the data.

[0045] In this embodiment, the asymmetric data compression technology based on bit-back encoding is as follows:

[0046] S1: Use the inference model to compress data through bit-back encoding, let X = (X1, X2, ..., X N ) is the data in the database, and each data X2 is taken from the finite dictionary table. The minimum description length of the original data X is subject to the entropy H(X) = E[-logp data (X)]; using the parameterized probability function p θ (X) is used for modeling, then the original data X is modeled as a marginal distribution with latent variables p(z), that is:

[0047] p θ (X) = ∫p θ (X,Z)dz=∫p θ (X|Z)pzdz

[0048] S2: Introducing the approximate posterior distribution Encode the original data X using latent variables, with a coding length of E[-logp θ (X|Z)-logp(z)], and the bit-back coding principle shows that using Encode the original data X, so the expected increase in code length when encoding data X is expressed as:

[0049]

[0050] S3: By encoding the message in a LIFO (last in, first out) manner, compared to arithmetic coding in a FIFO (first in, first out) manner, asymmetric data systems can implement bit-back coding by using latent variables to link bits;

[0051] S4: Based on the probabilistic modeling method, the posterior distribution and the likelihood function are jointly learned given the latent variable z; when the true posterior distribution p θ When (Z|X) is difficult to solve, we can optimize the approximate posterior The objective function of the encoding is Optimizing the approximate posterior It can be understood as minimizing the true posterior p θ (Z|X) and approximate posterior The Kullback–Leibler (KL) divergence between them; by minimizing the KL distance, the description length of the original data X is reduced to close to the entropy of the source message;

[0052] S5: Data encoding and decoding based on BBC-ANDC.

[0053] In this embodiment, the data encoding and decoding based on BBC-ANDC is as follows:

[0054] coding:

[0055] a Using the approximate posterior from step 3.2 Decode Z0;

[0056] b uses the likelihood function p θ (X|Z0) encodes X0;

[0057] c Encode Z0 using the prior distribution p(Z);

[0058] decoding:

[0059] a Decode Z0 using the prior distribution p(Z);

[0060] b uses the likelihood function p θ (X|Z0) decodes X0;

[0061] cUsing approximate posterior Code Z0.

[0062] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0063] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0064] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0065] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0066] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A high-performance database storage system for power intelligent terminals, characterized in that: It includes a database function unit, a data cleaning unit based on a lightweight rule engine, and an asymmetric data compression unit based on bit-back coding; the database function unit includes a data access engine and an OpenAPI interface, the data access engine implements support for C, C++, Go, and Java programming languages ​​through SQLAPI extension, and the OpenAPI interface provides a unified SQL data operation service to allow users to set the data transmission protocol type between the client and the server according to requirements; the data cleaning unit based on the lightweight rule engine includes a data deduplication module, a rule configuration module, a rule loading module, a regularization verification module, and a pattern matching module using a pattern matching algorithm based on regular expressions; the asymmetric data compression unit based on bit-back coding introduces an asymmetric data compression technology based on bit-back coding for data compression on the basis of Huffman coding and arithmetic coding.

2. According to claim 1, a database high-performance storage system for electric power intelligent terminals is characterized in that: The data deduplication module adopts Bloom filter as the deduplication rule, and replaces it with Roaring Bitmap, hash function, fingerprint recognition or similarity measurement according to different application scenarios of the smart terminal.

3. According to claim 2, a high-performance database storage system for power intelligent terminals is characterized in that: The rule configuration module allows the operator to customize the complexity and application scenarios of the rules according to actual needs, ensuring that the system can run under different resource conditions; the rule loading module is used to preload or precompile complex pattern matching rules; the rule verification module checks each piece of data one by one according to the pre-set rule set to ensure that it meets the specified format requirements. The verification content includes data type check, data range verification, and data dependency verification.

4. According to the high-performance database storage system for electric power intelligent terminals according to claim 3, it is characterized in that: The working mode of the data cleaning unit based on the lightweight rule engine is as follows: First, the system uses the data deduplication module to deduplicate the collected data, remove duplicate data, reduce storage space usage, and improve data quality; Subsequently, the rule engine reads and loads the predefined rule configuration information, including obtaining the user-defined rule set from the rule configuration module and precompiling or preloading it through the rule loading module to improve the efficiency of subsequent data processing; Finally, the system applies a regularized verification module and a pattern matching module to further process the data. The regularized verification module ensures that the data conforms to the predetermined format and dependencies, while the pattern matching module identifies and processes anomalies and specific patterns in the data to ensure the accuracy and consistency of the data.

5. According to the high-performance database storage system for electric power intelligent terminals according to claim 1, it is characterized in that: The asymmetric data compression technology based on bit-back encoding is specifically as follows: S1: Use the inference model to compress data through bit-back encoding, let X = (X1, X2, ..., X N ) is the data in the database, and each data X2 is taken from the finite dictionary table. The minimum description length of the original data X is subject to the entropy H(X) = E[-logp data (X)]; using the parameterized probability function p θ (X) is used for modeling, then the original data X is modeled as a marginal distribution with latent variables p(z), that is: p θ (X)=∫p θ (X,Z)dz=∫p θ (X|Z)pzdz S2: Introducing the approximate posterior distribution Encode the original data X using latent variables, with a coding length of E[-logp θ (X|Z)-logp(z)], and the bit-back coding principle shows that using E[- Encode the original data X, so the expected increase in code length when encoding data X is expressed as: S3: By encoding the message in a LIFO (last in, first out) manner, compared to arithmetic coding in a FIFO (first in, first out) manner, asymmetric data systems can implement bit-back coding by using latent variables to link bits; S4: Based on the probabilistic modeling method, the posterior distribution and the likelihood function are jointly learned given the latent variable z; when the true posterior distribution p θ When (Z|X) is difficult to solve, we can optimize the approximate posterior The objective function of the encoding is Optimizing the approximate posterior It can be understood as minimizing the true posterior p θ (Z|X) and approximate posterior The Kullback–Leibler (KL) divergence between them; by minimizing the KL distance, the description length of the original data X is reduced to close to the entropy of the source message; S5: Data encoding and decoding based on BBC-ANDC.

6. A database high-performance storage system for electric power intelligent terminals according to claim 5, characterized in that: The data encoding and decoding based on BBC-ANDC are as follows: coding: a Using the approximate posterior from step 3.2 Decode Z0; b uses the likelihood function p θ (X|Z0) encodes X0; c Encode Z0 using the prior distribution p(Z); decoding: a Decode Z0 using the prior distribution p(Z); b uses the likelihood function p θ (X|Z0) decodes X0; cUsing approximate posterior Code Z0.

Images