A storage method for multi-storage engine database for big data

Abstract

The present invention discloses a multi-storage engine database system for big data and a storage method thereof. The system includes: an SQL compiler; a stored procedure compiler; a transaction management unit; a distributed memory column storage; a distributed execution engine; a data source connector; a multi-tenant management component; and a middleware management unit. The method includes: unified data storage; unified metadata management; a unified data model; a unified SQL parsing engine; and unified security management and control. The present invention ensures support for at least two or more domestic processors. The multi-storage engine database has a 10x-100x performance improvement compared to the open source Hadoop version, meeting the low latency requirements of online storage and online business analysis systems. The SQL support completeness and performance are significantly ahead of Cloudera Impala. It also outperforms other Hadoop and MPP products in the TPC-DS and TPC-H benchmark tests, and can achieve deep integration of military databases, typical domestic CPUs, and military application scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a storage method for a multi-storage engine database for big data. Background Art

[0002] With the explosive growth of battlefield information, the extensive use of sensors, and the continuous improvement of information systems, traditional database storage methods can no longer meet the needs of capturing and understanding battlefield information and a large amount of military knowledge. In military applications, a large number of data types such as pictures, texts, and numerical values ​​have become increasingly complex and large. It is almost impossible to perform statistics, screening, and machine learning in massive amounts of disorganized data. This requires data analysts and model trainers to initialize the data before analysis and training, that is, to organize and classify the data, filter out dirty and erroneous data, and fill in missing data. However, due to the continuous expansion of military information data, data analysts and model trainers need to spend a lot of energy on data organization, need to learn a lot of additional data processing knowledge, and require certain engineering capabilities, which increases the threshold for analysts and algorithm personnel and greatly increases the labor costs of enterprises. In order to solve the above problems, a highly available multi-storage engine database based on big data architecture is urgently needed to automatically organize and classify data to reduce more labor costs, so that data analysts and model trainers can spend more time and energy on analysis and algorithms. In order to fill this technical gap, multi-storage engine database technology for big data has shown great advantages in solving knowledge representation, knowledge storage, knowledge query, etc., and can meet the needs of military information retrieval, intelligence analysis, and situational awareness. As the physical starting point of data analysis, storage is facing the challenge of big data applications.

[0003] Against the backdrop of typical combat and support data needs, and based on domestically produced basic software and hardware structures and technical features, we strive to achieve breakthroughs in key technologies such as unified management technology for multiple storage engines for massive military data, unified SQL parsing and model mapping technology, multi-engine distributed elastic expansion technology, database real-time computing engine technology, and graph databases, to form a typical domestically produced, independently controllable multi-storage engine fusion database, and develop a prototype system for the database core engine that can support unified management and processing of complex data in typical military application scenarios. Summary of the invention

[0004] The purpose of the present invention is to provide a system and storage method for a multi-storage engine database for big data, so as to solve the problems existing in the prior art.

[0005] To achieve the above object, the present invention provides a multi-storage engine database system for big data, comprising:

[0006] SQL compiler supports ANSI SQL 92 and SQL 99 standards, and ANSI SQL 2003 OLAP core extensions, meeting the SQL requirements of data warehouse services and facilitating smooth application migration;

[0007] The stored procedure compiler supports complete data types, process control, Package, cursor, exception handling, and dynamic SQL execution. It also supports high-speed statistics, addition, deletion, modification, and distributed transaction operations in stored procedures, meeting the needs of data application migration from relational databases to the Inceptor platform.

[0008] The transaction management unit uses the two-phase locking protocol and MVCC to achieve consistency and isolation control, and supports the Serializable Snapshot Isolation isolation level, thus ensuring transaction consistency in concurrent situations;

[0009] Distributed memory column storage, Holodesk, a column storage engine based on memory or SSD. Holodesk stores data in memory or SSD as columns, supplemented by a memory-based execution engine, completely avoiding the delay caused by IO;

[0010] The distributed execution engine independently builds a distributed data layer, which separates the computing data from the computing engine JVM memory space, effectively reducing the impact of JVM GC on system performance and stability;

[0011] Data source connector connects the execution engine and various data sources, and connects data from multiple different data sources to the engine for real-time statistical analysis without having to import the data into HDFS in advance, making it more convenient for users to build diversified business needs;

[0012] The multi-tenant management component provides complete multi-tenant management functions, including tenant resource management, tenant permission management, and security control modules, which facilitates the management and allocation of multiple tenants on a unified big data platform; it allows the configuration and management of CPU and memory resources for multiple tenants, and different tenants use different CPU and memory resource pools, so they will not interfere with each other;

[0013] The middleware management unit supports JDBC 4.0 and ODBC 3.5 standards, so it can support Hibernate / Spring middleware, is fully compatible with Tableau / QlikView / Cognos reporting tools, and can be fully connected with the company's current data application layer.

[0014] Furthermore, the stored procedure compiler includes a complete optimizer, including CFG Optimizer, Parallel Optimizer, and DAG Optimizer. The CFG Optimizer optimizes the code in the stored procedure and completes major optimizations such as loop unrolling, redundant code elimination, and function inlining.

[0015] Furthermore, the transaction management unit supports starting a transaction with Begin Transaction and ending a transaction with commit or rollback, implements consistency and isolation control through a two-phase locking protocol and MVCC, and supports the Serializable Snapshot Isolation isolation level, thereby ensuring transaction consistency in concurrent situations.

[0016] Furthermore, the Holodesk supports building distributed indexes for data fields, allowing users to build OLAP-Cube for multi-field combinations and store the cube directly in memory or on SSD, without the need for additional BI tools to build the cube.

[0017] Furthermore, the distributed execution engine has a cost-based optimizer and a rule-based optimizer, supplemented by 100 optimization rules, to ensure that SQL applications can achieve maximum performance without manual modification; the distributed execution engine includes two execution modes: low latency mode and high throughput mode.

[0018] Furthermore, after the execution plan starts, the data source connector extracts required data from other data sources through a pre-established connection, enters the execution engine layer to participate in SQL calculation, and releases the relevant database connection and corresponding resources after the calculation is completed.

[0019] The present invention also provides a storage method for a multi-storage engine database oriented to big data using the multi-storage engine database system oriented to big data, comprising:

[0020] Step 1: Unify data storage and integrate physically independent and management-separate database systems into a complete and unified data storage;

[0021] Step 2: Unified metadata management: deploy a metadata warehouse in the logical data warehouse to store and manage the RDBMS cluster of metadata; create a new table or a new index in each database system, and the logical data warehouse will simultaneously create a metadata two-dimensional mapping table in the metadata warehouse, and record the relationship between the metadata two-dimensional mapping table and the original table and / or index, laying the foundation for SQL unified access; the metadata management system stores the information of all database objects and provides a query interface for other systems to retrieve. The queryable information includes but is not limited to the physical distribution of data, data distribution characteristics, maximum and minimum values, and permission information;

[0022] Step 3: Unify the data model, map the data model of the multi-engine database system to the relational data two-dimensional mapping table of the relational database, and lay a theoretical foundation for centralized and unified management of metadata and unified SQL access;

[0023] Step 4: Unify the SQL parsing engine to provide unified SQL parsing, optimization, and execution services for all data systems in the logical data warehouse. The SQL statements submitted by the user are first received by the SQL engine, converted into Spark code after parsing and optimization, and then handed over to the high-performance distributed computing cluster for execution. During the execution process, the native APIs of each system will be called to implement data access.

[0024] Step 5: Unify security management and control, establish a unified authentication system based on users and roles, comply with the account and / or role RBAC model, implement permission management through roles, and perform batch authorization management for users; support the security protocol Kerberos, use LDAP as the account management system, and perform unified security authentication of account information through Kerberos.

[0025] Furthermore, the unified metadata management in step 2 further includes providing metadata access and collection for Hive, HDFS, and HBase; providing a unified metadata management UI interface and a RESTful API interface for related services, providing various types of microservice docking methods, and the data storage of the UI front-end page data of the unified metadata management adopts a MySQL database table; providing a Messaging message queue (currently using Kafka) and a metadata operation interface and a data message bus method for the API interface; providing a unified metadata TypeSystem type system, a Graph graph computing storage query engine layer, an intelligent label algorithm, and a knowledge graph model; providing a public storage encapsulation layer for the graph computing query engine, and supporting the JanusGraph open source graph computing storage query engine.

[0026] Furthermore, the unified metadata management in step 2 further includes real-time and efficient indexing of heterogeneous data such as graphs, key values, documents, and relationships, and supports mixed indexing BRPQ of numerical point attributes, numerical segment attributes, spatial locations, and texts, specifically including the construction of PSPQ indexes, the combination of PSPQ and keyword indexes, spatial keyword distributed queries with relational attributes, and index construction based on Lucene.

[0027] Furthermore, the unified security management architecture in step 5 is divided into four layers: the system layer uses an improved ApacheDS to increase the read and write efficiency by more than 10 times, uses the same set of users and a unified LDAP / Kerberos authentication method, avoids OpenLDAP from using Kerberos authentication, and accelerates LDAP authentication efficiency; the business layer implements complete ARBAC model support, provides RESTAPI, and user-friendly WebUI and password policy functional support; the plug-in layer uses plug-ins to provide authentication, authorization, group mapping, and quota management for each component, so that each component can use a unified user, group, and permission management model; the various platform PaaS and SaaS services connected to the service layer are protected by unified security management.

[0028] The method of the present invention has the following advantages:

[0029] The present invention ensures that it can support at least two or more domestic processors. Through the verification of the prototype system, the multi-storage engine database has a 10x-100x performance improvement compared to the open source Hadoop version, meeting the low latency requirements of online storage and online business analysis systems. Combined with the optimization of the execution engine and data storage layer, the SQL support completeness and performance are significantly ahead of Cloudera Impala. It also outperforms other Hadoop and MPP products in the TPC-DS and TPC-H benchmark tests, and can achieve a deep integration of military databases, typical domestic CPUs, and military application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The overall architecture diagram of key technology implementation is shown;

[0031] Figure 2 A unified metadata management architecture is shown;

[0032] Figure 3 shows a Thrift communication protocol diagram;

[0033] Figure 4 A unified security management and control architecture diagram is shown;

[0034] Figure 5 A diagram of the Kerbeors authentication process is shown;

[0035] Figure 6 The PSPQ index structure is shown;

[0036] Figure 7 A schematic diagram based on B-tree query is shown;

[0037] Figure 8 The BRPQ index structure is shown. DETAILED DESCRIPTION

[0038] The technical scheme of the present invention will be clearly and completely described below in conjunction with specific embodiments, but those skilled in the art should understand that the embodiments described below are only used to illustrate the present invention and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Based on the domestic processor platform, the present invention innovatively proposes to build a unified data storage engine, establish a unified metadata management mechanism, and a unified data model through multi-storage engine database technology for big data, and create a unified and efficient distributed SQL engine to provide powerful platform support for military business scenarios.

[0040] The present invention provides a multi-storage engine database system for big data. Figure 1 As shown, including:

[0041] 1. SQL Compiler SQL 2003 Compiler

[0042] Enterprise-level data warehouses, data marts and other applications are mostly developed based on SQL, but most Hadoop products in the industry have poor SQL compatibility or do not support SQL modular expansion, so the cost of application migration is very high or even unfeasible. In order to reduce the cost of application migration, Transwarp Inceptor has developed a complete SQL compiler that supports ANSI SQL 92 and SQL 99 standards, and supports ANSI SQL 2003 OLAP core extensions, which can meet the SQL requirements of most existing data warehouse businesses and facilitate smooth application migration.

[0043] In addition to a better SQL semantic analysis layer, Inceptor includes a powerful optimizer to ensure the best performance of SQL on the engine. Inceptor includes three levels of optimizers: the first is a rule-based optimizer that applies static optimization rules and generates a logical execution plan; the second is a cost-based optimizer that measures the CPU, IO, and network costs of multiple different execution plans to select a more reasonable plan and generate a physical execution plan; the last is a code generator that generates more efficient execution code or Java Byte Code for some core execution logic, thereby ensuring the best performance of SQL services on distributed platforms.

[0044] 2. PL / SQL Compiler

[0045] Most of the existing data warehouse applications in China are based on SQL 2003, and a large number of stored procedures are used to build complex applications. Therefore, in addition to the SQL compiler, Transwarp Inceptor also includes a stored procedure compiler for compiling and executing stored procedures.

[0046] Inceptor supports Oracle PL / SQL and DB2 SQL PL, two major SQL standards, including complete data types, process control, package, cursor, exception handling and dynamic SQL execution, and supports high-speed statistics, addition, deletion, modification and distributed transaction operations in stored procedures. Therefore, with the addition of stored procedure compiler, Inceptor can meet the migration of most data applications from relational databases to the Inceptor platform.

[0047] In addition to SQL syntax support, the stored procedure compiler includes a complete optimizer, including CFGOptimizer, Parallel Optimizer, and DAG Optimizer. CFG Optimizer optimizes the code in the stored procedure, completing loop unrolling, redundant code elimination, function inlining and other major optimizations. Parallel Optimizer parallelizes some originally serial logic, using the computing power of the cluster to improve the overall execution speed, and significantly improves the performance of some key functions such as cursors. DAG Optimizer will generate a DAG based on the Figure 2 The optimization generates a more reasonable physical execution plan, focusing on reducing the overhead of tasks such as shuffle. In order to effectively be compatible with other databases, Inceptor supports isolating the differences between different SQL standards through different dialect settings, thereby avoiding ambiguity in data calculation and processing standards, and thus ensuring the correctness of data processing.

[0048] 3. Transaction Manager

[0049] In order to better meet the needs of data warehouse business scenarios, Inceptor provides complete SQL support for adding, deleting and modifying data, allowing data to be processed from multiple data sources. At the same time, in order to effectively ensure the accuracy of data processing, Inceptor provides support for distributed transactions, ensuring the ACID of data during processing, that is, atomicity, consistency, isolation and persistence.

[0050] Inceptor supports starting a transaction with Begin Transaction and ending a transaction with commit or rollback. The transaction management unit implements consistency and isolation control through the two-phase locking protocol and MVCC, and supports the Serializable Snapshot Isolation isolation level, thus ensuring transaction consistency in concurrent situations.

[0051] Inceptor supports the semantic specifications of SQL2003 regarding addition, deletion, modification and query, supports Insert, Update, Delete, Truncate and Merge Into primitives, supports updating data tables individually or from other data tables and nested queries, and has a built-in consistency check function to prevent illegal changes.

[0052] Through the optimization of the SQL compiler, the SQL execution plan for adding, deleting and modifying data is executed concurrently in the cluster through the distributed engine. The overall throughput of the system can reach several times that of the relational database, which can meet the high throughput requirements of batch processing business. In addition, through reasonable resource planning, Inceptor allows tenants to perform high-speed statistical analysis on data while adding, deleting and modifying data.

[0053] 4. Distributed memory column storage Holodesk

[0054] In order to speed up interactive analysis, Inceptor launched Holodesk, a column storage engine based on memory or SSD. Holodesk stores data in memory or SSD in column format, and with the help of a memory-based execution engine, it can completely avoid the delay caused by IO and greatly improve the data scanning speed.

[0055] In addition to columnar storage to speed up statistical analysis, Holodesk supports building distributed indexes for data fields. By using intelligent indexing technology to build the best query solution for queries, Inceptor can reduce SQL query latency to milliseconds.

[0056] Holodesk allows users to build OLAP-Cube for multiple field combinations and store the cube directly in memory or SSD, without the need for additional BI tools to build the cube. Therefore, for some complex statistical analysis and report interactive queries, Holodesk can achieve a response in seconds.

[0057] In addition to performance advantages, Holodesk also excels in availability. Holodesk's metadata and storage both natively support high availability, and support exception handling and disaster recovery through consistency protocols and multiple versions. In abnormal situations, Holodesk can automatically restore and rebuild all table information and data without manual recovery, thereby reducing development and operation costs and ensuring system availability.

[0058] Stability. Inceptor focused on optimizing the performance of Holodesk based on SSD, making the performance based on PCIE SSD reach more than 80% of the full memory solution. Therefore, combined with the use of low-cost memory / flash hybrid storage solutions, the analysis performance can be close to that of full memory storage, ensuring the high cost-effectiveness of the solution.

[0059] 5. Distributed Execution Engine

[0060] Inceptor has developed a dedicated distributed computing engine based on Apache Spark, which not only greatly improves computing performance, but also effectively solves many problems of Spark in terms of stability, ensuring that the computing engine can run uninterruptedly 24 hours a day, 7 days a week. In addition, the Inceptor engine independently builds a distributed data layer, which separates the computing data from the computing engine JVM memory space, thus effectively reducing the impact of JVM GC on system performance and stability.

[0061] In terms of SQL execution plan optimization, Inceptor implements cost-based optimizers and rule-based optimizers, supplemented by more than 100 optimization rules, which can ensure that SQL applications can achieve maximum performance without manual changes. The execution engine can also automatically identify and optimize common data processing problems such as data skew, which can solve most computing scenarios with data skew and prevent the impact of data skew on system stability.

[0062] In order to better adapt to various data scenarios, Inceptor's execution engine includes two execution modes: low latency mode and high throughput mode. Low latency mode is mainly used in scenarios with relatively small data volumes. The execution engine will generate a physical execution plan with low execution latency. By reducing or avoiding some high-latency tasks (such as IO, network, etc.), it ensures that the SQL execution time is very short, reaching or approaching the performance of relational databases in these scenarios. High throughput mode is mainly used in big data scenarios, and improves the performance of complex statistical analysis on very large data volumes through reasonable distributed execution. Therefore, Inceptor's execution engine can meet data business needs on various data volumes from GB to PB.

[0063] 6. Data Source Connector Stargate

[0064] Enterprise data may be scattered in multiple systems, which cannot share data or conduct relevant analysis, resulting in data silos. By building a unified big data platform, the data silos in most scenarios can be effectively solved. However, there are still some data that cannot be migrated to a unified platform due to various relationships. In order to solve such problems, Inceptor launched the data source connector Stargate. Stargate is a connector that connects the execution engine and various data sources. It can connect data from multiple different data sources to the engine for real-time statistical analysis without having to import the data into HDFS in advance, making it more convenient for users to build diversified business needs. At the grammatical level, Inceptor is compatible with the OracleDB-Link specification. By creating a database link, a connection pool with other data sources can be pre-established. Then, in SQL, the data of the data source can be accessed in real time in Inceptor through the table_name@database link method without other operations. After the execution plan starts, Stargate extracts the required data from other data sources through the pre-established connection and inputs it into the execution engine layer to participate in SQL calculation. After the calculation is completed, the relevant database connection and corresponding resources are released.

[0065] Currently, Stargate supports relational databases including Oracle, DB2, Mysql, Teradata and PostgreSQL. In addition, Stargate can currently connect to Holodesk, HDFS, Hyperbase, Elastic Search, etc., and will be able to connect to Redis as a data source in the future.

[0066] 7. Multi-tenant management component Guardian

[0067] Guardian provides complete multi-tenant management functions, including tenant resource management, tenant permission management, and security control modules, which can facilitate the management and allocation of multiple tenants on a unified big data platform.

[0068] Guardian allows the configuration and management of CPU and memory resources for multiple tenants. Different tenants use different CPU and memory resource pools, so they will not interfere with each other. In addition, different priorities can be set for different users to achieve service quality control (QoS). Guardian supports the configuration and management of user disk space through SQL, including quotas, changes and management of data space and temporary space, to facilitate the platform's reasonable allocation, control and billing of storage resources.

[0069] Guardian supports the use of LDAP protocol for user access control and Kerberos protocol as the underlying access control, thus ensuring data security and isolation. Guardian supports a complete set of SQL-based database / table permission control. Administrators can set user permissions for querying, modifying, deleting, etc. on tables, and includes a complete set of role settings. User permission control can be easily implemented through the setting of role groups.

[0070] In addition, Guardian supports Row Level Security, which allows precise row-level permission control of table data. In a multi-tenant scenario, it can ensure that different tenants can only see the data in the table that they have permissions for, and not the data belonging to other tenants, thus achieving precise data isolation.

[0071] 8. Middleware Management Unit Connector

[0072] Inceptor fully supports JDBC 4.0 and ODBC 3.5 standards, so it can support middleware such as Hibernate / Spring, is fully compatible with reporting tools such as Tableau / QlikView / Cognos, and can be fully connected with the company's current data application layer.

[0073] In addition, Inceptor also supports docking with other data synchronization tools, has completed mutual authentication and integration with IBM CDC, and can support tools such as Oracle Golden Gate and SAPData Service. Therefore, enterprise users can synchronize transaction data to Inceptor in real time for interactive statistical analysis.

[0074] The technical solution of the present invention includes:

[0075] 1. Unified data storage

[0076] Integrate physically independent and management-separate database systems into a complete and unified data storage.

[0077] 2. Unified metadata management mechanism

[0078] In order to achieve centralized and unified management of metadata, a metadata warehouse is deployed in the logical data warehouse, that is, an RDBMS cluster used to store and manage metadata. If a new table or new index is created in each database system, the logical data warehouse will simultaneously create a metadata two-dimensional mapping table in the metadata warehouse and record the relationship between the mapping table and the original table / index, thus laying the foundation for unified SQL access.

[0079] The metadata management system is responsible for storing the information of all database objects, including schema, table, view, column, trigger, sequence, and stored procedures, and provides query interfaces for other systems to retrieve. The queryable information includes but is not limited to the physical distribution of data, distribution characteristics of data, maximum and minimum values, permission information, etc. Because metadata information is very critical data, its reliability determines whether the entire database system can provide services, so the prototype system needs to be fully designed and consider the reliability of the metadata storage system. The present invention plans to use a consistent and highly reliable storage service to record metadata information, and the optional systems include Apache Zookeeper or Apache Etcd systems. Zookeeper and Etcd both store the same data in multiple instances, and use a consistent algorithm (such as the Raft algorithm) to ensure that the data in each instance is completely consistent. In this way, even if some instances cannot provide services due to software and hardware problems, as long as there is an instance that is still working normally, the metadata system can still ensure that the service does not go offline.

[0080] 3. Unified Data Model

[0081] In order to make all data systems in multiple engines support SQL access, it is necessary to achieve unification in the data model. It is found through research that although the technical principles of each system are different, most of them adopt a data model similar to a two-dimensional table. For example, HBase, although it adopts a four-dimensional table model, it also organizes data by row, and introduces column families to relax the restrictions on columns, so the four-dimensional table can be mapped to a two-dimensional table under specific constraints. Through this idea, the present invention can uniformly map the data model of the multi-engine database system to a two-dimensional table of a relational database, also known as a two-dimensional mapping table of relational data, which lays a theoretical foundation for centralized and unified management of metadata and unified SQL access.

[0082] 4. Unified SQL parsing engine

[0083] Based on the high-performance computing framework Spark, we build a general distributed SQL engine. This engine is responsible for providing unified SQL parsing, optimization, and execution services for all data systems in the logical data warehouse. The SQL statements submitted by users are first received by the SQL engine, converted into Spark code after parsing and optimization, and then handed over to the high-performance distributed computing cluster for execution. During the execution process, the native APIs of each system will be called to achieve data access.

[0084] 5. Unified security control

[0085] The unified authentication system based on users and roles complies with the account / role RBAC (Role-Based Access Control) model to implement permission management through roles and batch authorization management of users. Support the security protocol Kerberos, use LDAP as the account management system, and use Kerberos to perform unified security authentication of account information. In a complex battlefield confrontation environment, unified security management and control can prevent the enemy's information confrontation system from eavesdropping on the information transmission content of the party, interfering with the signal of the party's system, and injecting false deception information into the information system. The information security requirements for data applications in information warfare are:

[0086] (1) Data encryption: Provides encryption protection for data during transmission and static storage, and can effectively protect sensitive data when it is accessed without authorization. In terms of data encryption and decryption, high-performance, low-latency end-to-end and storage-layer encryption and decryption can be achieved through efficient encryption and decryption solutions (non-sensitive data can be unencrypted without affecting performance). At the same time, the effective use of encryption requires secure and flexible key management. Open source solutions in this regard are still relatively weak and require the use of commercial key management products. In addition, encryption and decryption are transparent to upper-layer businesses. Upper-layer businesses only need to specify sensitive data, and the business is completely unaware of the encryption and decryption process.

[0087] (2) User privacy data desensitization: Provide data desensitization and personal information de-identification functions, and provide user data encryption services that meet international cryptographic algorithms.

[0088] (3) Multi-tenant isolation: Implement multi-tenant access isolation measures, implement data security level classification, support tag-based mandatory access control, provide an ACL-based data access authorization model, provide global data views and private data views, and provide access control for data views.

[0089] (4) Data lifecycle management: Understanding the source of data in the big data platform, as well as knowing how the data is used, who destroys it and where, is critical to monitoring whether there is illegal data access in the big data system. This needs to be achieved through security auditing. The purpose of security auditing is to capture a complete record of activities within the system that cannot be changed. The big data platform must be able to conduct all-round security management and control of data, so that it can be "managed before, controlled during, and checked after."

[0090] (5) Log audit: Log audit is an indispensable measure for data management, data tracing, and attack detection. The big data platform has log management and analysis capabilities.

[0091] Specific embodiments of the present invention:

[0092] 1) Unified metadata management implementation method

[0093] (1) Metadata Management Architecture

[0094] In order to achieve centralized and unified management of metadata, a metadata warehouse is deployed in the logical data warehouse, that is, an RDBMS cluster used to store and manage metadata. If a new table or new index is created in each database system, the logical data warehouse will simultaneously create a metadata two-dimensional mapping table in the metadata warehouse and record the relationship between the metadata two-dimensional mapping table and the original table / index, thus laying the foundation for SQL unified access. We have conducted in-depth research on metadata management and designed the architecture as follows: Figure 2 As shown:

[0095] Metadata Sources Access

[0096] Provides metadata access and collection functions for Hive, HDFS, HBase, etc.

[0097] Spring Framework UI&Restful API

[0098] Provides a unified metadata management UI interface and a RESTful API interface for related services, and provides various types of microservice docking methods. The data storage of the UI front-end page data of the unified metadata management uses a MySQL database table. The page data is entered through front-end page operations, and obtained by querying the back-end platform service in real time or offline.

[0099] Metadata Integration&Notification API

[0100] Provides a metadata operation interface and data message bus for the Messaging message queue (currently using Kafka) and API interface (HTTP or REST)

[0101] Core Platform

[0102] Provides a unified metadata type system, graph computing storage query engine layer, intelligent labeling algorithm, knowledge graph model, etc.

[0103] Graph Database

[0104] Provides a public storage encapsulation layer for graph computing query engines and supports the Janus Graph open source graph computing storage query engine.

[0105] (2) Metadata Storage Principle

[0106] Metadata store uses HBase to store entity information and uses ElasticSearch to store index information. Metadata store clients connect to Metadata store services, and Metadata store connects to MySQL database to access metadata. With metastore service, multiple clients can connect at the same time, and these clients do not need to know the username and password of MySQL database, they only need to connect to metastore service. Unified metadata starts metadata as a separate service. Start a MetaStoreServer on the server side, and the client uses Thrift protocol to access the metadata database through MetaStoreServer. Thrift communication protocol is as follows: Figure 3 As shown:

[0107] The underlying IO module is responsible for the actual data transmission, including Socket, file, or compressed data stream.

[0108] TTransport is responsible for sending and receiving messages in byte stream mode. It is the implementation of the underlying IO module in the Thrift framework. Each underlying IO module will have a corresponding TTransport to be responsible for the transmission of Thrift's byte stream (ByteStream) data on the IO module. For example, TSocket corresponds to Socket transmission, and TFileTransport corresponds to file transmission.

[0109] TProtocol is mainly responsible for assembling structured data into Message, or reading structured data from Message structure. TProtocol converts a typed data into a byte stream to be transmitted to TTransport, or reads a certain length of byte data from TTransport and converts it into a specific type of data. For example, int32 will be Encoded into a four-byte byte data by TBinaryProtocol, or TBinaryProtocol takes out four bytes of data from TTransport and decodes them into int32.

[0110] TServer is responsible for receiving the client's request and forwarding it to the Processor for processing. The main task of TServer is to efficiently accept the client's request, especially to complete the request quickly in the case of high concurrent requests.

[0111] Processor (or TProcessor) is responsible for responding to the Client's request, including RPC request forwarding, call parameter parsing and user logic calling, return value writing and other processing steps. Processor is the key process for the server to transfer from the Thrift framework to the user logic. Processor is also responsible for writing data to or reading data from the Message structure.

[0112] 2) Unify the implementation methods of security control

[0113] In a complex battlefield confrontation environment, unified security control can prevent the enemy's information confrontation system from eavesdropping on the information transmission content of the enemy, interfering with the signal of the enemy's system, and injecting false deception information into the information system. Therefore, the research on unified security control technology based on multi-storage engine database is very important. This section studies the unified security control architecture, unified service encryption and authentication services, and unified data permission access control to ensure the security and unified control of the entire multi-storage engine database platform.

[0114] like Figure 4 As shown in the figure, the unified security management and control architecture is divided into four layers:

[0115] The system layer uses an improved ApacheDS to increase the read and write efficiency by more than 10 times. It uses the same set of users and a unified LDAP / Kerberos authentication method to avoid OpenLDAP using Kerberos authentication and speed up LDAP authentication efficiency.

[0116] The business layer implements complete ARBAC model support, provides RESTAPI, user-friendly WebUI and password policy support. At the same time, the JWTToken mechanism is adopted to prepare for the implementation of SSO. The user authentication and authorization are unified, and the same set of users and authorization mechanisms are used from Web services to the bottom layer of Hadoop; at the same time, the LDAP interface, RESTAPI and LoginService are opened for third-party applications to integrate authentication and authorization.

[0117] The plug-in layer uses plug-ins to provide authentication, authorization, group mapping, and quota management for each component, so that each component can use a unified user, group, and permission management model.

[0118] The various platform PaaS and SaaS services connected to the service layer are protected by unified security management and control.

[0119] The Kerberos protocol provides unified service encryption and authentication services. The specific process of Kerbeors authentication is as follows: the client sends a request to the authentication server (AS) to obtain the certificate of a certain server, and then the response of the AS contains these certificates encrypted with the client key.

[0120] like Figure 5 As shown in Figure 1, the certificate consists of: 1) the server TGT; 2) a temporary encryption key (also called a session key "sessionkey"). The client sends the TGT (including the client identity encrypted with the server key and a copy of the session key) to the server. The session key can be used to authenticate the client or the server (now shared by the client and the server), and can also be used to provide encryption services for future communications between the two parties, or to provide further communication encryption services for the two parties by exchanging independent sub-session keys.

[0121] 3) Efficient real-time indexing for heterogeneous data such as graphs, key values, documents, and relationships

[0122] In view of the difficulty of traditional relational databases in meeting the problem of complex data types in military big data processing, including heterogeneous data such as documents, relations, and spatiotemporal locations, the present invention proposes to process the PSPQ (Point and Segment Preferences Query) index based on numerical point and numerical segment preference attribute information, and then combines it with the keyword index to propose a hybrid index BRPQ (Boolean Range with Preferences Query index) that supports numerical point attributes, numerical segment attributes, spatial locations, and text. Specifically, it includes the construction of PSPQ index, the combination of PSPQ and keyword index, spatial keyword distributed query with relational attributes, and index construction based on Lucene.

[0123] PSPQ Index Structure

[0124] The PSPQ index structure consists of two parts: a hash table and an inverted file. Figure 6 shown.

[0125] (1) Hash table. It is used to index the numeric point relationship attribute information and contains key-value pairs (key, pIF). Among them, key is the identifier of the numeric point relationship attribute information and pIF is the pointer to the inverted file.

[0126] Hash table key value design. There are multiple numerical point attributes, and the queries for each attribute are range queries. It is a major challenge to perform joint range queries on multiple attributes while ensuring the pruning rate. To address this challenge, each attribute value interval is divided into multiple small intervals, and the key value of each small interval is assigned with the minimum value of the interval. Then, the Cartesian product of the keys of the small intervals of each attribute is used to obtain the key value of the hash table. This has two major advantages. On the one hand, data can be quickly accessed based on the key value, and joint range queries can be performed on multiple attributes. On the other hand, when pruning, hash table key values ​​that do not meet the query range can be quickly filtered out based on preference queries. The more attribute value intervals are divided, the more hash table key values ​​are filtered out, and the stronger the pruning ability.

[0127] Preprocess and split each attribute value interval to obtain the hash table key value. For each numerical point attribute, the attribute values ​​of all objects are divided into equal parts to obtain each interval. The purpose of equal division is to make the number of objects in each corresponding interval roughly equal, and the data is evenly associated with each hash table key value, which is more stable during pruning, and the number of objects associated with the pruned hash table key value will not be too many or too few.

[0128] Of course, since the numeric point attribute is a numeric attribute and different attributes are independent, the numeric point preference attribute can also be indexed using a B-tree. However, the cost of the B-tree index is much higher than that of the BRPQ index, so this project does not use the B-tree index. The following is a detailed description of this. Figure 7 shown.

[0129] The query cost increases after B-tree is used. When B-tree is used to index multiple numerical point preference attributes, a B-tree index needs to be established for each attribute. Multiple B-tree indexes cannot be combined with the index for processing spatial keywords and the index for processing numerical segment user preference attributes in this article. As shown in the figure above, after using B-tree, it is necessary to use the spatial keyword index to obtain the candidate result set that meets the spatial and keyword queries, and use the numerical segment user preference attribute index to obtain the candidate result set that meets the numerical segment preference. The candidate result set that meets this attribute preference is obtained on each B-tree, and then the intersection of all candidate result sets is calculated. When querying based on the BRPQ index, the result set of the data set after the spatial keyword query is queried again for preference query. Assuming that the number of numerical point user preference attributes is p, the B-tree needs to query the entire data set p+2 times, while the BRPQ index only needs to query once. In the query process, the spatial keyword query process costs of the two are the same. For preference query, BRPQ performs preference query based on the results obtained by the spatial keyword query, while the B-tree performs p preference queries on the data set to find objects that meet the preferences of each attribute. Moreover, BRPQ uses hashing technology for numerical point preference queries, with a time complexity of O(1), while the time complexity of B-tree is O(log2N), where N is the height of the B-tree and the time complexity of p B-trees is O(plog2N). In summary, the query cost based on B-tree is significantly higher than that of BRPQ.

[0130] The index space increases after B-tree is used. When indexing multiple numeric point preference attributes based on B-tree, a B-tree index needs to be established for each attribute. Assuming that the number of numeric point user preference attributes is p and the number of data set objects is n, not only (p+2)*n pointers to the data need to be saved, but also the space occupied by p B-tree indexes is large. However, BRPQ only needs to increase the key value of the hash table.

[0131] The insertion cost increases after B-tree is used. After indexing the numerical point user preference attribute based on B-tree, the entire data set needs to be inserted p+2 times. However, BRPQ only needs to be inserted once. During the insertion process, the cost of inserting the spatial keyword index is the same for both, but when inserting the preference attribute, BRPQ uses hash technology, and the insertion cost is O(1). The worst insertion cost based on B-tree index is O(logn), and the time complexity of inserting p B-trees is O(plogn). Therefore, the insertion cost after B-tree is significantly higher than that of BRPQ.

[0132] (2) Inverted file. It is used to index the user preference attribute information of the numerical segment, and contains the key-value pair (key, pL). Among them, key is the identifier of the user preference attribute information of the numerical segment, and pL is the pointer of the inverted linked list corresponding to key. The inverted linked list stores the object identifier or pointer pO, which uniquely identifies an object.

[0133] When searching based on inverted file index, the DAAT algorithm provides high efficiency for searching for a certain data that appears in multiple inverted lists at the same time. When searching based on numerical segment preference, the key value is used for fast and large-scale pruning, and the DAAT algorithm is used for efficient query.

[0134] PSPQ combined with keyword index

[0135] The PSPQ index supports collaborative pruning of numerical point and numerical segment user preference attributes. In order to solve spatial keyword range queries constrained by user preferences, it is necessary to combine PSPQ with spatial keyword index.

[0136] There have been a lot of research on spatial keyword query. The index of spatial location generally uses quadtree or R-tree, such as Figure 8 As shown, BRPQ uses a quadtree, but an R-tree can also be used. Considering the small number of intermediate results and the fact that an ordered keyword tree (ordered keyword trie) provides better query performance when retrieving ordered keywords, an ordered keyword tree is used to index text information, and the paths pruned from the ordered keyword tree will clearly satisfy the keyword query. The combined index BRPQ structure is shown in the figure. The quadtree leaf node contains a pointer to the ordered keyword tree, and a threshold β is set for the object associated with the quadtree leaf node. When the threshold is reached, the quadtree leaf node splits. In order to ensure a small number of intermediate results, the leaf nodes (in the following text, only the nodes associated with the ordered keyword tree are referred to as leaf nodes) will continue to be divided until the number of objects contained in each quadtree node is less than or equal to k. Each path in the ordered keyword tree contains a pointer to the PSPQ.

[0137] Spatial keyword distributed query algorithm with relational attributes

[0138] In recent years, the amount of data has increased from GB to TB or even PB. Therefore, large amounts of data management play a very important role in data analysis. However, how to efficiently index and query data is a bottleneck in data analysis. When a search engine obtains data, it will only try to search for data that is closest in text, closest in geographical location, and satisfies relational attributes. However, due to the sudden increase in data volume, existing search engines cannot efficiently respond to queries containing these three types of information because there is no corresponding efficient index that can process these types of data at the same time. Even existing spatial keyword-oriented queries do not take relational attributes into account, thereby reducing query efficiency. In this era of large-scale data, traditional index structures and algorithms are all based on single-machine implementation, and the processing speed can no longer meet user needs. In order to solve the above problems, a Baseline Algorithm for Keywords and Location-aware with relational attributes Distributed Query index (KLPDQ) is proposed, which supports keywords, geographical location information, and relational attributes. On this basis, a more efficient spatial keyword query processing algorithm with relational attributes is proposed. This algorithm can not only index these three types of data at the same time, but also use distributed indexing and query processing to significantly reduce the speed of indexing and querying.

[0139] Spatial keyword object with relational attributes (Object) = {K, G, P}. OK is the keyword set of the object, OG is the latitude and longitude coordinates of the object, and OP is the set of relational attributes. OP = {P', S}. OPP' is the value point set of the relational attribute and each unit element represents the minimum value of the attribute, and OPS is the set of the value segment attributes of the relational attribute. In the present invention, the value segment attribute only considers the business hours attribute of the merchant.

[0140] Spatial keyword range query with relational attributes (Query) = {K, R, P}. QK is the set of query keywords, and QR is the set of query geographic location ranges. QR == {T, L, B, R}, QRT is the upper left corner of the geographic location range; QRL is the lower left corner of the geographic location range; QRB is the lower right corner of the geographic location range; QRR is the upper right corner of the geographic location range. QP is the query set of the query's relational attributes. Similar to the construction of an object, QP is also composed of queries for numerical points and numerical segments respectively.

[0141] The necessary and sufficient conditions for an object to satisfy a query are: Specifically, the object must contain the keywords in the query statement, must be within the spatial scope of the query, and its numerical point attributes must be greater than the numerical point attributes of the query, and its numerical segment attributes must be within the numerical segment attributes of the query.

[0142] However, because the query and indexing process are based on the big data environment, the mechanism is based on the distributed indexing and query mechanism of Lucence. Spatial, textual and relational attributes are indexed simultaneously, multi-dimensional information is converted into one-dimensional information, and Lucence and Hadoop are used to store and query spatial keyword objects under big data.

[0143] Lucene index construction

[0144] Some basic concepts of Lucene include index, document, domain and word. An index represents a series of documents. A document represents a series of domains. A domain represents a series of words. A word is an array of strings. If the same string belongs to different domains, it is a different word. Therefore, a word represents a pair of strings, one of which is the domain name and the other is a string in the domain.

[0145] (1) Index: In the computer field, an inverted index is a data structure that can be used to store text mapping relationships, such as mapping text and numbers to their locations in database files.

[0146] (2) Segments: A Lucene index that is divided into smaller blocks is called a segment. Each segment is an index. During a Lucene query, all segments are accessed sequentially.

[0147] (3) Document: A document is the smallest unit in the Lucene indexing and querying process. A document represents a series of domains. Each domain has a domain name and a series of values. Because each domain is stored in a document, the document needs to give each domain a unique domain name to prevent conflicts.

[0148] (4) Domain: Each domain consists of two parts, domain name and value. The value can be any string or number. These values ​​are used to represent data. The value can be optionally stored in the index in case the document is selected and the various information in the domain needs to be returned.

[0149] (5) Term: A term represents a string of characters, which consists of two elements: the text information of the string and the domain name to which it belongs.

[0150] Lucene index creation process

[0151] For example, the following are three text messages:

[0152] T0=I was a student, and I came from Nanjing.

[0153] T1=My father is a teacher, and I am a student.

[0154] T2=Nanjing is my hometown, but my father's hometown is Chengdu.

[0155] First, use the tokenizer to get keywords from the three texts. The processing procedure is as follows.

[0156] 1) In the preprocessing stage, Lucene uses a tokenizer to divide the text content into segments. For Chinese, because it involves semantic issues, you can choose to use a Chinese grammar analyzer.

[0157] 2) During the analysis phase, Lucene needs to filter out any punctuation and connecting words, and then convert each word to uppercase. The indexer then calls the function addDocument(Doc) and passes the input value to Lucene for indexing. The processed data is stored in the index file and stored on disk as inverted file data.

[0158] After the preprocessing and analysis phase, the keyword information of these three text data becomes:

[0159] key0=I am student I come from Nanjing.

[0160] key1=My father is teacher I am student.

[0161] key2=Nanjing is my hometown my father hometown is Chengdu.

[0162] Now you can create an inverted index based on keyword information. After the index is created, you can update or delete it through IndexReader.

[0163] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A multi-storage engine database system for big data, comprising: SQL compiler supports ANSI SQL 92 and SQL 99 standards, and ANSI SQL 2003 OLAP core extensions, meeting the SQL requirements of data warehouse services and facilitating smooth application migration; The stored procedure compiler supports complete data types, process control, Package, cursor, exception handling, and dynamic SQL execution. It also supports high-speed statistics, addition, deletion, modification, and distributed transaction operations in stored procedures, meeting the needs of data application migration from relational databases to the Inceptor platform. The transaction management unit uses the two-phase locking protocol and MVCC to achieve consistency and isolation control, supports the Serializable Snapshot Isolation isolation level, and ensures transaction consistency in concurrent situations; Distributed memory column storage, Holodesk, a column storage engine based on memory or SSD. Holodesk stores data in memory or SSD as columns, supplemented by a memory-based execution engine, completely avoiding the delay caused by IO; The distributed execution engine independently builds a distributed data layer, which separates the computing data from the computing engine JVM memory space, reducing the impact of JVM GC on system performance and stability; Data source connector connects the execution engine and various data sources, and performs real-time statistical analysis on the data access engine of various data sources. It does not need to import data into HDFS in advance, which facilitates users' business construction to meet diversified needs. The multi-tenant management component provides complete multi-tenant management functions, including tenant resource management, tenant permission management, and security control modules, which facilitates the management and allocation of multiple tenants on a unified big data platform; it allows the configuration and management of CPU and memory resources for multiple tenants, and different tenants use different CPU and memory resource pools without interfering with each other; The middleware management unit supports JDBC 4.0 and ODBC 3.5 standards, can support Hibernate / Spring middleware, is fully compatible with Tableau / QlikView / Cognos reporting tools, and can be fully connected with the company's current data application layer.

2. The multi-storage engine database system for big data according to claim 1, characterized in that: The stored procedure compiler includes a complete optimizer, including CFG Optimizer, Parallel Optimizer and DAGOptimizer. The CFG Optimizer optimizes the code in the stored procedure and completes loop unrolling, redundant code elimination and function inlining optimization.

3. The multi-storage engine database system for big data according to claim 1, characterized in that: The transaction management unit supports starting a transaction with Begin Transaction and ending a transaction with commit or rollback. It implements consistency and isolation control through a two-phase locking protocol and MVCC, supports the Serializable SnapshotIsolation isolation level, and ensures transaction consistency in concurrent situations.

4. The multi-storage engine database system for big data according to claim 1, characterized in that: The Holodesk supports building distributed indexes for data fields, allowing users to build OLAP-Cube for multi-field combinations and store the Cube directly in memory or SSD, without the need for additional BI tools to build the Cube.

5. The multi-storage engine database system for big data according to claim 1, characterized in that: The distributed execution engine is a cost-based optimizer and a rule-based optimizer, supplemented by 100 optimization rules, to ensure that SQL applications can achieve maximum performance without manual modification; the distributed execution engine includes two execution modes: low latency mode and high throughput mode.

6. The multi-storage engine database system for big data according to claim 1, characterized in that: After the execution plan starts, the data source connector extracts the required data from other data sources through the pre-established connection, enters the execution engine layer to participate in SQL calculation, and releases the relevant database connection and corresponding resources after the calculation is completed.

7. A storage method for a multi-storage engine database for big data using the multi-storage engine database system for big data as claimed in any one of claims 1 to 6, comprising: Step 1: Unify data storage and integrate physically independent and management-separate database systems into a complete and unified data storage; Step 2: Unify metadata management and deploy a metadata warehouse in the logical data warehouse to store and manage the RDBMS cluster dedicated to metadata. A new table or a new index is created in each database system, and the logical data warehouse synchronously creates a metadata two-dimensional mapping table in the metadata warehouse, and records the relationship between the metadata two-dimensional mapping table and the original table and / or index; The metadata management system stores information about all database objects and provides a query interface for other systems to retrieve information, including but not limited to the physical distribution of data, data distribution characteristics, maximum and minimum values, and permission information; Step 3: Unify the data model, and uniformly map the data model of the multi-engine database system into a two-dimensional mapping table of relational data of the relational database; Step 4: Unify the SQL parsing engine to provide unified SQL parsing, optimization, and execution services for all data systems in the logical data warehouse. The SQL statements submitted by the user are first received by the SQL engine, converted into Spark code after parsing and optimization, and then handed over to the high-performance distributed computing cluster for execution. During the execution process, the native APIs of each system are called to implement data access. Step 5: Unify security management and control, establish a unified authentication system based on users and roles, comply with the account and / or role RBAC model, manage permissions through roles, and perform batch authorization management for users; Supports the security protocol Kerberos, uses LDAP as the account management system, and performs unified security authentication of account information through Kerberos.

8. The storage method for a multi-storage engine database for big data according to claim 7, characterized in that: The unified metadata management in step 2 further includes providing metadata access and collection for Hive, HDFS, and HBase; providing a UI interface for unified metadata management and a RESTful API interface for related services, providing various types of microservice docking methods, and the data storage of the UI front-end page data of the unified metadata management adopts a MySQL database table; It provides the metadata operation interface and data message bus mode of the Messaging message queue and API interface; it provides a unified metadata Type System, a Graph computing storage query engine layer, an intelligent label algorithm, and a knowledge graph model; it provides a public storage encapsulation layer for the graph computing query engine and supports the Janus Graph open source graph computing storage query engine.

9. The storage method for a multi-storage engine database for big data according to claim 8, characterized in that: The unified metadata management in step 2 further includes real-time and efficient indexing of heterogeneous data such as graphs, key values, documents, and relationships, supporting mixed indexes BRPQ of numerical point attributes, numerical segment attributes, spatial locations, and texts, including the construction of PSPQ indexes, the combination of PSPQ and keyword indexes, spatial keyword distributed queries with relationship attributes, and index construction based on Lucene.

10. The storage method for a multi-storage engine database for big data according to claim 7, characterized in that: The unified security management and control architecture in step 5 is divided into four layers: the system layer uses the improved ApacheDS, uses the same set of users and a unified LDAP / Kerberos authentication method, avoids OpenLDAP using Kerberos authentication, and accelerates LDAP authentication efficiency; the business layer implements complete ARBAC model support, provides REST API and user-friendly Web UI and password policy function support; The plug-in layer uses plug-ins to provide authentication, authorization, group mapping, and quota management for each component, so that each component uses a unified user, group, and permission management model; the various platform PaaS and SaaS services connected to the service layer are protected by unified security management.

Images