A real-time query method and device for PB-level massive data

Abstract

The present invention discloses a real-time query method and device for PB-level massive data, which is applied to a logical database, including: determining the target index information corresponding to the data query request in response to the data query request input by the user through the logical database; determining the target data file corresponding to the index information according to the target index information; the target index information includes an extraction start position and an extraction end position; extracting data records from the extraction start position in the target data file according to a preset extraction sequence until the extraction end position; and performing JSON serialization on the data records to generate sequence data and output it. This solves the technical problem that the real-time query in the prior art relies on memory resources and has high implementation cost, thereby effectively reducing the cost of real-time query and improving query efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data query technology, and in particular to a real-time query method and device for PB-level massive data. Background Art

[0002] In big data applications, in addition to big data computing and analysis functions, real-time data query capabilities are also required to meet various data retrieval and search functions. There are query requirements for raw data, intermediate layer data, statistical layer data, etc. Different types of data have different data volumes. Usually, the amount of raw data is very large, up to PB level, such as the operator's mobile user traffic track details (users access Internet content through mobile networks track records).

[0003] Providing real-time query scenarios for PB-level data requires the storage engine to have ultra-large data capacity and fast storage mechanism, and to provide second-level response performance for large concurrent query scenarios. At the same time, as a big data application, it should support access to mainstream big data computing frameworks, including MapReduce, Hive, and Spark.

[0004] At present, some open source products in the industry can provide real-time query functions, such as open source Hive, which can provide massive data storage but does not have real-time query functions; open source HBase, although it can perform real-time queries, has a complex storage mechanism for massive data, unstable performance, and query reliance on memory resources, with high implementation costs; relational databases (open source Mysql / commercial Oracle) have real-time query capabilities but cannot meet the storage scenarios of PB-level massive data; open source search engines, queries are heavily dependent on memory resources, host storage resource utilization is low, implementation costs are high, and storage performance is low, which cannot meet PB-level data query scenarios. It can be seen that the above products all have certain limitations and cannot meet the production scenario requirements of implementing queries on PB-level massive data. Summary of the invention

[0005] The present invention provides a real-time query method and device for PB-level massive data, which solves the technical problem in the prior art that real-time query depends on memory resources and has high implementation cost.

[0006] The present invention provides a real-time query method for PB-level massive data, which is applied to a logical database, wherein the logical database includes a Hive database, including:

[0007] In response to a data query request input by a user, determining target index information corresponding to the data query request;

[0008] Determine a target data file corresponding to the target index information from the Hive database; the target index information includes an extraction start position and an extraction end position;

[0009] Extracting data records in the target data file starting from the extraction start position in a preset extraction order until the extraction end position;

[0010] The data record is serialized into JSON to generate and output sequence data.

[0011] Optionally, the logical database includes an HBase index table, and the step of determining target index information corresponding to the data query request in response to the data query request input by the user includes:

[0012] receiving a data query request input by a user, wherein the data query request includes a keyword;

[0013] Determine target index information matching the keyword in the HBase index table.

[0014] Optionally, the Hive database includes multiple Hive data tables, each of the Hive data tables includes multiple partitions, each of the partitions includes multiple data files, the target index information also includes a target Hive table name, a target partition sequence number, and a target file sequence number, and the step of determining the target data file corresponding to the target index information from the Hive database includes:

[0015] Select a target Hive data table from the Hive database according to the target Hive table name;

[0016] Determine a target partition from multiple partitions in the target Hive data table according to the target partition sequence number; wherein the partition is in a usable state;

[0017] A target data file is determined from the target partition according to the target file sequence number.

[0018] Optionally, the method further comprises:

[0019] When a user inputs a data write request, a data write operation is performed in response to the data write request.

[0020] Optionally, the data write request carries a write file specification parameter, and the step of performing a data write operation in response to the data write request includes:

[0021] In response to the data writing request, receiving a plurality of input data files input by a user;

[0022] Sorting the plurality of input data files according to the order of preset keywords to obtain a plurality of sorted data files, wherein the sizes of the sorted data files are set by the write file specification parameter;

[0023] Traversing the plurality of sorted data files, and generating write index information equal to the number of preset keywords corresponding to the sorted data files;

[0024] Add the write index information in the HBase index table;

[0025] Create a new partition in the Hive data table, and write the sorted data file to the new partition;

[0026] The state of the new partition is updated to be usable.

[0027] The present invention also provides a real-time query device for PB-level massive data, which is applied to a logical database, wherein the logical database includes a Hive database, including:

[0028] An index information determination module, configured to determine target index information corresponding to a data query request input by a user;

[0029] A target data file determination module, used to determine the target data file corresponding to the index information according to the target index information; the target index information includes an extraction start position and an extraction end position;

[0030] A data record extraction module, used to extract data records in the target data file starting from the extraction start position in a preset extraction order until the extraction end position;

[0031] The sequence data output module is used to perform JSON serialization on the data records, generate sequence data and output it.

[0032] Optionally, the logical database includes an HBase index table, and the index information determination module includes:

[0033] A data query request receiving submodule, used to receive a data query request input by a user, wherein the data query request includes a keyword;

[0034] The target index information determination submodule is used to determine the target index information matching the keyword in the HBase index table.

[0035] Optionally, the Hive database includes multiple Hive data tables, each of the Hive data tables includes multiple partitions, each of the partitions includes multiple data files, the target index information also includes a target Hive table name, a target partition serial number and a target file serial number, and the target data file determination module includes:

[0036] A target Hive data table selection submodule is used to select a target Hive data table from the Hive database according to the target Hive table name;

[0037] A target partition determination submodule, used to determine a target partition from multiple partitions in the target Hive data table according to the target partition sequence number; wherein the partition is in a usable state;

[0038] The target data file determination submodule is used to determine the target data file from the target partition according to the target file sequence number.

[0039] Optionally, the device further comprises:

[0040] The data writing operation execution module is used for executing the data writing operation in response to the data writing request when the user inputs a data writing request.

[0041] Optionally, the data writing request carries a writing file specification parameter, and the data writing operation execution module includes:

[0042] An input data file receiving submodule, used for receiving a plurality of input data files input by a user in response to the data writing request;

[0043] A sorting submodule, used for sorting the plurality of input data files according to the order of preset keywords to obtain a plurality of sorted data files, wherein the sizes of the sorted data files are set by the write file specification parameter;

[0044] A write index information generation submodule, used to traverse the plurality of sorted data files and generate write index information having a number equal to the number of preset keywords corresponding to the sorted data files;

[0045] A write index information adding submodule, used to add the write index information to the HBase index table;

[0046] A partition writing submodule, used to create a new partition in the Hive data table and write the sorted data file into the new partition;

[0047] The status update submodule is used to update the status of the new partition to a usable status.

[0048] It can be seen from the above technical solutions that the present invention has the following advantages:

[0049] The logical database responds to the data query request input by the user, determines the corresponding target index information, and then determines the corresponding target data file and the extraction start position and extraction end position of the data records in the file according to the target index information, selects the data file and extracts the data records from the extraction start position to the extraction end position, and finally serializes the data records into JSON to generate and output sequence data, thereby solving the technical problems of the existing technology that real-time query depends on memory resources and has high implementation cost, thereby effectively reducing the cost of real-time query and improving query efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A flowchart of a method for real-time query of PB-level massive data provided by an embodiment of the present invention;

[0052] Figure 2 A flowchart of a method for real-time query of PB-level massive data provided by an optional embodiment of the present invention;

[0053] Figure 3 A flowchart of the steps of the data updating process provided by an embodiment of the present invention;

[0054] Figure 4 A data structure diagram of a logical database provided by an embodiment of the present invention;

[0055] Figure 5 A technical architecture diagram of a real-time query device for PB-level massive data provided by an embodiment of the present invention;

[0056] Figure 6 A structural block diagram of a real-time query device for PB-level massive data provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The embodiment of the present invention provides a real-time query method and device for PB-level massive data, which is used to solve the technical problem that real-time query in the prior art depends on memory resources and has high implementation cost.

[0058] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] See also Figure 1 , Figure 1 A flowchart of the steps of a real-time query method for PB-level massive data provided by an embodiment of the present invention.

[0060] The present invention provides a real-time query method for PB-level massive data, which is applied to a logical database, wherein the logical database includes a Hive database, including:

[0061] Step 101, in response to a data query request input by a user, determining target index information corresponding to the data query request;

[0062] Step 102, determining a target data file corresponding to the target index information from the Hive database;

[0063] In an embodiment of the present invention, when a user needs to perform a data query, a data query request input by the user is received, target index information corresponding to the data query request is determined, and the location of the target data file is determined based on the target index information, thereby determining the data file corresponding to the index information.

[0064] It is worth mentioning that the target index information includes an extraction start position and an extraction end position.

[0065] Step 103, extracting data records from the target data file starting from the extraction start position in a preset extraction order until the extraction end position;

[0066] After the target data file is determined, data records are extracted from the extraction start position in the target index information until the extraction end position. The extraction can be performed in sequence, for example, reading can be performed according to keyword sequence, time sequence, etc.

[0067] It is worth mentioning that the keyword order can be set by the user according to the usage scenario, and the embodiment of the present invention does not limit this. For example, in the telecommunications call record query scenario, all call records are sorted according to the number keyword, and the call time is used as the sorting keyword, so the call records of the same number are stored continuously, and the call records of the same number are sorted in a secondary order according to the call time; or they are read in alphabetical order.

[0068] Step 104: JSON serialize the data record to generate and output sequence data.

[0069] In an embodiment of the present application, a logical database responds to a data query request input by a user to determine the corresponding target index information, and then determines the corresponding target data file and the extraction start position and extraction end position of the data records in the file based on the target index information, selects the data file and extracts data records from the extraction start position to the extraction end position, and finally performs JSON serialization on the data records to generate and output sequence data, thereby solving the technical problem of real-time query relying on memory resources and high implementation cost in the prior art, thereby effectively reducing the cost of real-time query and improving query efficiency.

[0070] See also Figure 2 , Figure 2 A flowchart of the steps of a real-time query method for PB-level massive data provided by an embodiment of the present invention.

[0071] The present invention provides a real-time query method for PB-level massive data, which is applied to a logical database, wherein the logical database includes a Hive database, including:

[0072] Step 201, in response to a data query request input by a user, determining target index information corresponding to the data query request;

[0073] Optionally, the logical database includes an HBase index table, and step 201 includes:

[0074] receiving a data query request input by a user, wherein the data query request includes a keyword;

[0075] Determine target index information matching the keyword in the HBase index table.

[0076] In the embodiment of the present invention, a data query request input by a user including a keyword can be received, and matching search of index information can be performed in the HBase index table according to the keyword; when target index information matching the keyword is retrieved, the next step is performed.

[0077] Optionally, if the target index information matching the keyword is not retrieved, empty information is returned.

[0078] It is worth mentioning that before receiving the data query request input by the user, the identity of the user corresponding to the data query request can be authenticated based on the user's registration information in the registration center. If the authentication is successful, the data query operation is executed.

[0079] HBase is a distributed, column-oriented open source database. Just as Bigtable uses the distributed data storage provided by Google File System, HBase provides similar capabilities to Bigtable on top of Hadoop. HBase is a subproject of Apache's Hadoop project. HBase is different from general relational databases. It is a database suitable for unstructured data storage. Another difference is that HBase is based on columns rather than rows.

[0080] Hive is a data warehouse analysis system built on Hadoop. It provides a rich set of SQL query methods to analyze data stored in the Hadoop distributed file system: it can map structured data files to a database table and provide complete SQL query functions; it can convert SQL statements into MapReduce tasks to run, and analyze the required content through its own SQL queries. This set of SQL is referred to as Hive SQL, so that users who are not familiar with mapreduce can easily use SQL language to query, summarize and analyze data. Mapreduce developers can use their own mappers and reducers as plug-ins to support hive for more complex data analysis. It is slightly different from the SQL of relational databases, but supports most statements such as DDL, DML, and common aggregate functions, connection queries, and conditional queries. It also provides a series of functions: it can extract, transform and load data to store, query and analyze large-scale data sets stored in Hadoop, and supports UDF (User-Defined Function), UDAF (User-Defnes Aggregate Function) and UDTF (User-Defined Table-Generating Function). It can also customize map and reduce functions, providing good scalability and extensibility for data operations.

[0081] Furthermore, the Hive database includes multiple Hive data tables, each of the Hive data tables includes multiple partitions, each of the partitions includes multiple data files, and the target index information also includes a target Hive table name, a target partition serial number, and a target file serial number. The above step 102 can be replaced by the following steps 202-204:

[0082] Step 202, selecting a target Hive data table from the Hive database according to the target Hive table name;

[0083] In a specific implementation, the index information may include a Hive data table name, a target partition serial number, and a target file serial number. Since a logical database usually includes multiple data warehouse tool Hive data tables as storage support, after receiving the index information, hierarchical retrieval may be performed to increase retrieval speed, wherein the first step is to determine the target Hive data table according to the target Hive table name in the index information.

[0084] Step 203, determining a target partition from multiple partitions of the target Hive data table according to the target partition sequence number;

[0085] After determining the target Hive data table, the target partition can be determined from multiple partitions in Hive according to the target partition sequence number. To facilitate data update and expansion, the partition can be set to an unusable state and an usable state. When the partition is in the usable state, it means that the partition has not been updated or expanded, and the target partition can be selected from it, thereby preventing data errors caused by simultaneous data update and data retrieval, wherein the partition is in the usable state.

[0086] Step 204: determine the target data file from the target partition according to the target file sequence number.

[0087] In an example of the present invention, the file serial numbers of multiple data files of the target partition may be checked, and when the file serial number is the same as the target file serial number, the data file corresponding to the target file serial number is determined to be the target data file.

[0088] Optionally, data files can be stored in HDFS as SequenceFile files.

[0089] The storage structure of SequenceFile mainly consists of a Header followed by multiple Records. The Header mainly contains information such as Key classname, value classname, storage compression algorithm, user-defined metadata, etc. In addition, it also contains some synchronization identifiers for quickly locating the boundaries of records. Each Record is stored in the form of a key-value pair. The character array used to represent it can be parsed into: the length of the record, the length of the Key, the Key value and the value value at one time, and the structure of the Value value depends on whether the record is compressed.

[0090] HDFS (Hadoop Distributed File System) is a distributed file system designed to run on commodity hardware. HDFS uses a master / slave structure model. An HDFS cluster consists of a NameNode and several DataNodes. The NameNode acts as the master server, managing the file system's namespace and client access operations to files; the DataNodes in the cluster manage stored data.

[0091] Step 205, extracting data records from the target data file starting from the extraction start position in a preset extraction order until the extraction end position;

[0092] In the specific implementation, each partition can have multiple data files, each data file can include multiple keyword records, and the number of records in a single file can reach hundreds of millions. The records of a single keyword are stored sequentially in the same data file, so that the target records can be read sequentially with the least number of IOs. The number of data files is specified by the table parameters (the number of data files is calculated based on the total size of the input file through the maximum number of bytes of each SequceFile parameter).

[0093] Optionally, to further improve retrieval efficiency, the index information may also include information such as the location of the data record in the data file and the number of data records in the partition. Thus, the data record may be quickly determined based on the location of the data record in the data file, and the number of data records in the partition may be displayed for users to perform statistics.

[0094] Step 206: JSON serialize the data record to generate and output sequence data.

[0095] In the specific implementation, the process of JSON serialization of data records can be as follows:

[0096] {

[0097] "resultCode":0,

[0098] "message":"ok",

[0099] "data":[{

[0100] "field1":"value1-1",

[0101] "field2":"value1-2",

[0102] "field3":"value1-3",

[0103] "field4":"value1-4",

[0104] "field5":"value1-5",

[0105] "field6":"value1-6",

[0106] },{

[0107] "field1":"value2-1",

[0108] "field2":"value2-2",

[0109] "field3":"value2-3",

[0110] "field4":"value2-4",

[0111] "field5":"value2-5",

[0112] "field6":"value2-6",

[0113] },{

[0114] "field1":"value3-1",

[0115] "field2":"value3-2",

[0116] "field3":"value3-3",

[0117] "field4":"value3-4",

[0118] "field5":"value3-5",

[0119] "field6":"value3-6",

[0120] }]

[0121] }

[0122] Through the above process, the data records can be serialized into JSON to generate sequence data, which is then output to the user for further use.

[0123] In one example of the present invention, the method further comprises the following steps:

[0124] When a user inputs a data write request, a data write operation is performed in response to the data write request.

[0125] See also Figure 3 , the above step of “performing a data write operation in response to the data write request” may include the following sub-steps S11-S16:

[0126] S11, in response to the data writing request, receiving a plurality of input data files input by a user;

[0127] S12, sorting the multiple input data files according to the order of preset keywords to obtain multiple sorted data files, wherein the sizes of the sorted data files are set by the write file specification parameter;

[0128] S13, traversing the plurality of sorted data files, and generating index information to be added equal to the number of the preset keywords;

[0129] S14, adding the index information to be added to the HBase index table;

[0130] S15, creating a new partition in the Hive data table, and writing the sorted data file to the new partition;

[0131] S16: Update the status of the new partition to be usable.

[0132] In an embodiment of the present invention, in response to the data write request, multiple data files input by the user are received, and the input data records can be sorted in the order of keywords by a sorting algorithm developed based on MapReduce, and encapsulated into a target data file, for example, it can be encapsulated into a SequenceFile file format, sorted according to the keyword (KEY) specified by the table attribute, so that the data records with the same keyword are sorted according to SORT_KEY (SORT_KEY is defined by the table attribute parameter, and generally SORT_KEY uses a timestamp field); then traverse multiple data update files to generate index information equal to the number of keywords in the data file, and the number of data files is specified by a parameter (it is recommended that the parameter setting of a single target data file does not exceed 10G, if the partition size is 10T, the number of target files generated is 1000), wherein each keyword corresponds to an index information, and each index information includes a Hive sequence number, a target partition sequence number, a keyword, a target file sequence number and a record position.

[0133] Optionally, the update index information can be loaded into the HBase index table through the Bulkload interface, and the update index information is added to the HBase index table to generate a new HBase index table. New partitions equal to the number of update files are created in the Hive; the new partitions are mapped to the Hive, and the status of the new partitions is updated to be usable.

[0134] Among them, MapReduce is a distributed computing model and one of the main components of Hadoop. It is responsible for the computing function of large batches of data. MapReduce is divided into two stages: Map and Reduce. A Map function is specified to map a set of key-value pairs into a new set of key-value pairs, and a concurrent Reduce function is specified to ensure that each of the mapped key-value pairs shares the same key group.

[0135] In an embodiment of the present invention, a logical database responds to a data query request input by a user, determines the corresponding target index information, and then determines the corresponding target data file and the extraction start position and extraction end position of the data records in the file according to the target index information, selects the data file and extracts the data records from the extraction start position to the extraction end position, and finally performs JSON serialization on the data records to generate and output sequence data, thereby solving the technical problem that the real-time query in the prior art relies on memory resources and has high implementation cost, thereby effectively reducing the cost of real-time query and improving query efficiency.

[0136] See also Figure 4 , Figure 4 A data structure diagram of a logical database in an embodiment of the present invention is shown, which includes a logical database, data information and index information.

[0137] The logical database is similar to the relational database in function, with concepts such as table name, field, keyword, sort keyword, etc., including multiple partitions Chunk-1 to Chunk-n;

[0138] The data information includes the following two parts: physical table and data file:

[0139] Physical table: The fields and partition information of the logical database are carried by Hive. Each logical database will create a corresponding physical table in Hive. The partitions of the logical database will create corresponding partitions in the Hive physical table. The data files (SequenceFile) will eventually belong to the corresponding partitions.

[0140] Data file: Data records are ultimately stored in data files (SequenceFile). Each partition can have multiple data files, and each file can contain multiple keyword records. The number of records in a single file can reach hundreds of millions. Records of a single keyword are stored sequentially in the same data file, so that the target records can be read sequentially with the least number of IO operations. The number of data files is specified by the table parameters (the number of data files is calculated based on the total size of the input file through the maximum number of bytes of each SequceFile parameter).

[0141] Index information is stored in the HBase index table. Each keyword corresponds to an index information in each partition. The index information includes: physical table; partition number; file number; location of the record in the data file; number of records in the partition Chunk;

[0142] See also Figure 5 , Figure 5 The technical architecture block diagram of a real-time query device for PB-level massive data according to an embodiment of the present invention is shown, which includes three parts: query service, data management and data storage.

[0143] The query service, based on the SpringCloud service framework, provides distributed data query services. The data query function is implemented by querying the logical database. The functions of each service module are as follows: Registration Center: Provides distributed query service instance registration function. The service gateway automatically synchronizes the service instances registered in the registration center to distribute service requests; Configuration Center: Provides centralized configuration services for distributed service instances. When the service instance starts, it obtains the latest configuration from the configuration center; Service Gateway: External centralized service entrance. All external requests are distributed to the service instance through the gateway; Authentication Service: Authentication of query requests. The service that successfully authenticates continues to execute the query.

[0144] Data management, including MapReduce: big data computing framework. All steps of the warehousing algorithm are developed based on the MapReduce framework. The warehousing algorithm realizes the process of generating target partitions from original unordered files (generating data files and indexes). The steps of the warehousing algorithm are: regularization calculation: input unordered data files, generate ordered SequenceFiles sorted by keywords through MapReduce calculation, and unify multiple records of keywords for secondary sorting according to SORT_KEY; index generation: read SequenceFile, extract the sequence number and offset (Offset) of the data file where the data record of each keyword is located, and write index data to the intermediate file; index loading: load the content of the intermediate index file into the corresponding HBase index table through the Bulkload mechanism of HBase; data loading: create the target partition, add the partition Partition corresponding to the partition Chunk in the corresponding table of the logical database Hive, and set the status of the partition Chunk to be available.

[0145] Data storage includes two parts: logical database and physical data. The logical database provides functions to applications in the form of two-dimensional tables. The concept of logical database is similar to that of relational database, and has information such as table name, field, keyword (KEY), sort keyword (SORT_KEY), etc. When using this solution, the application must first define the table structure and create a logical database. When creating a logical database, the table name, field list, keyword field, and sorting keyword field must be specified; physical data, physical data is implemented by integrating multiple technologies such as HDFS, HIVE, HBase, and ZooKeeer, among which: a. HDFS: provides file storage capabilities for logical databases, and data is stored in HDFS in the form of SequenceFile files; b. Hive: provides field, partition and other management functions for logical databases, which correspond to logical databases one by one. In addition, for each Chunk of the logical database, a corresponding Parition will be created in the Hive corresponding table, and multiple SequenceFile data files of the corresponding Chunk will be loaded; c. HBase: provides indexing function for logical databases, and index data is loaded through the Bulkload mechanism during the warehousing process; when querying data, first query the index information through HBase, and then read the data file to obtain records according to the index information; d. ZooKeeper: provides status Data storage such as data and table attributes. Status data includes: Chunk status: records the status of each partition (Chunk) of the table, controls the storage status and life cycle of the Chunk, including: initial status (INIT); regularized (COMPACTED); index created (INDEX_CREATED); index loaded (INDEX_LOADED); data loaded (DATA_LOADED); effective (ENABLED); deleted (DELETED); table attributes, configure table Chunk capacity, primary key and sort keyword and other parameters. The main attributes include: single Chunk maximum capacity (CHUNK_MAX_BYTES): a single partition is recommended not to exceed 10T; including the maximum number of partitions (Chunk) (MAX_CHUNK_COUNT): control the maximum number of partitions for each table, up to 10,000 Chunks; keyword field (KEY); sort field (SORT_KEY): the same keyword is sorted according to the SORT_KEY field.

[0146] See also Figure 6 , Figure 6 A structural block diagram of a real-time query device for PB-level massive data according to an embodiment of the present invention is shown.

[0147] The present invention also provides a real-time query device for PB-level massive data, which is applied to a logical database, wherein the logical database includes a Hive database, including:

[0148] The index information determination module 601 is used to determine target index information corresponding to the data query request in response to the data query request input by the user;

[0149] A target data file determination module 602 is used to determine a target data file corresponding to the target index information from the Hive database; the target index information includes an extraction start position and an extraction end position;

[0150] A data record extraction module 603, configured to extract data records from the target data file starting from the extraction start position in a preset extraction order until the extraction end position;

[0151] The sequence data output module 604 is used to perform JSON serialization on the data record, generate sequence data, and output it.

[0152] Optionally, the logical database includes an HBase index table, and the index information determination module 601 includes:

[0153] A data query request receiving submodule, used to receive a data query request input by a user, wherein the data query request includes a keyword;

[0154] The target index information determination submodule is used to determine the target index information matching the keyword in the HBase index table.

[0155] Optionally, the Hive database includes multiple Hive data tables, each of the Hive data tables includes multiple partitions, each of the partitions includes multiple data files, the target index information also includes a target Hive table name, a target partition serial number and a target file serial number, and the target data file determination module 602 includes:

[0156] A target Hive data table selection submodule is used to select a target Hive data table from the Hive database according to the target Hive table name;

[0157] A target partition determination submodule, used to determine a target partition from multiple partitions in the target Hive data table according to the target partition sequence number; wherein the partition is in a usable state;

[0158] The target data file determination submodule is used to determine the target data file from the target partition according to the target file sequence number.

[0159] Optionally, the device further comprises:

[0160] The data writing operation execution module is used for executing the data writing operation in response to the data writing request when the user inputs a data writing request.

[0161] Optionally, the data writing request carries a writing file specification parameter, and the data writing operation execution module includes:

[0162] An input data file receiving submodule, used for receiving a plurality of input data files input by a user in response to the data writing request;

[0163] A sorting submodule, used for sorting the plurality of input data files according to the order of preset keywords to obtain a plurality of sorted data files, wherein the sizes of the sorted data files are set by the write file specification parameter;

[0164] A write index information generation submodule is used to traverse the plurality of sorted data files and generate index information to be added equal to the number of the preset keywords;

[0165] Write index information adding submodule, used to add the index information to be added to the HBase index table;

[0166] A partition writing submodule, used to create a new partition in the Hive data table and write the sorted data file into the new partition;

[0167] The status update submodule is used to update the status of the new partition to a usable status.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0169] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time query method for PB-level massive data, characterized in that: Applied to a logical database, the logical database includes a Hive database, the method includes: In response to a data query request input by a user, determine target index information corresponding to the data query request; the target index information includes an extraction start position, an extraction end position, a target Hive table name, a target partition serial number, and a target file serial number; Determine a target data file corresponding to the target index information from the Hive database; Extracting data records in the target data file from the extraction start position in a preset extraction order until the extraction end position; the preset extraction order includes a keyword order and a time order; Serialize the data record into JSON to generate and output sequence data; The logical database includes an HBase index table, and the step of determining target index information corresponding to the data query request in response to the data query request input by the user includes: receiving a data query request input by a user, wherein the data query request includes a keyword; Determine target index information matching the keyword in the HBase index table.

2. The method according to claim 1, characterized in that The Hive database includes multiple Hive data tables, each of the Hive data tables includes multiple partitions, each of the partitions includes multiple data files, and the step of determining the target data file corresponding to the target index information from the Hive database includes: Select a target Hive data table from the Hive database according to the target Hive table name; Determine a target partition from multiple partitions in the target Hive data table according to the target partition sequence number; wherein the partition is in a usable state; A target data file is determined from the target partition according to the target file sequence number.

3. The method according to claim 2, characterized in that The method further comprises: When a user inputs a data write request, a data write operation is performed in response to the data write request.

4. The method according to claim 3, characterized in that The data writing request carries a write file specification parameter, and the step of performing a data writing operation in response to the data writing request includes: In response to the data writing request, receiving a plurality of input data files input by a user; Sorting the plurality of input data files according to the order of preset keywords to obtain a plurality of sorted data files, wherein the sizes of the sorted data files are set by the write file specification parameter; Traversing the plurality of sorted data files, generating index information to be added equal to the number of the preset keywords; Add the index information to be added to the HBase index table; Create a new partition in the Hive data table, and write the sorted data file to the new partition; The state of the new partition is updated to be usable.

5. A real-time query device for PB-level massive data, characterized in that: Applied to a logical database, the logical database includes a Hive database, including: An index information determination module is used to determine target index information corresponding to a data query request input by a user in response to the data query request; the target index information includes an extraction start position, an extraction end position, a target Hive table name, a target partition serial number, and a target file serial number; A target data file determination module, used to determine a target data file corresponding to the target index information from the Hive database; A data record extraction module, used to extract data records in the target data file starting from the extraction start position in a preset extraction order until the extraction end position; A sequence data output module, used to perform JSON serialization on the data records, generate sequence data and output it; The logical database includes an HBase index table, and the index information determination module includes: A data query request receiving submodule, used to receive a data query request input by a user, wherein the data query request includes a keyword; The target index information determination submodule is used to determine the target index information matching the keyword in the HBase index table.

6. The device according to claim 5, characterized in that The Hive database includes multiple Hive data tables, each of the Hive data tables includes multiple partitions, each of the partitions includes multiple data files, and the target data file determination module includes: A target Hive data table selection submodule is used to select a target Hive data table from the Hive database according to the target Hive table name; A target partition determination submodule, used to determine a target partition from multiple partitions in the target Hive data table according to the target partition sequence number; wherein the partition is in a usable state; The target data file determination submodule is used to determine the target data file from the target partition according to the target file sequence number.

7. The device according to claim 6, characterized in that The device also includes: A data writing request receiving module, used for receiving the data writing request when the user inputs the data writing request; The data writing operation execution module is used to execute the data writing operation in response to the data writing request.

8. The device according to claim 7, characterized in that The data writing request carries a writing file specification parameter, and the data writing operation execution module includes: An input data file receiving submodule, used for receiving a plurality of input data files input by a user in response to the data writing request; A sorting submodule, used for sorting the plurality of input data files according to the order of preset keywords to obtain a plurality of sorted data files, wherein the sizes of the sorted data files are set by the write file specification parameter; A write index information generation submodule is used to traverse the plurality of sorted data files and generate index information to be added equal to the number of the preset keywords; Write index information adding submodule, used to add the index information to be added to the HBase index table; A partition writing submodule, used to create a new partition in the Hive data table and write the sorted data file into the new partition; The status update submodule is used to update the status of the new partition to a usable status.

Images