Data query method and device and electronic equipment

Abstract

The invention discloses a data query method and device and electronic equipment. The method comprises the steps that a data query request is received, and the data query request carries target query data; in response to the data query request, determining a data query type of the target query data, the data query type including a transaction processing type query and a transaction analysis type query; according to the data query type, determining a predetermined type of file in the distributed file system, the predetermined type of file including a row type storage data file and a column type storage data file; and determining a query result corresponding to the target query data from the predetermined type of file. According to the data query method and device, the technical problem that the data query speed and the data query accuracy are difficult to balance during data query in related technologies is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data query method, device and electronic equipment. Background Art

[0002] With the advent of the big data era, the amount of data is growing. In order to effectively manage and analyze these massive amounts of data, data lakes and data warehouses have become common solutions. Traditional data lakes often face problems such as data quality, data consistency, and data availability, while traditional data warehouses may face performance bottlenecks and storage cost limitations when processing large-scale data, making it difficult to ensure data query accuracy and efficiency. Therefore, in related technologies, when performing data queries, there is a technical problem of difficulty in balancing data query speed and data query accuracy.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a data query method, device and electronic device to at least solve the technical problem in the related art that it is difficult to balance the data query speed and the data query accuracy when performing data query.

[0005] According to one aspect of an embodiment of the present invention, there is provided a data query method, comprising: receiving a data query request, wherein the data query request carries target query data; in response to the data query request, determining a data query type of the target query data, wherein the data query type comprises: transaction processing type query, transaction analysis type query; based on the data query type, determining a predetermined type file in a distributed file system, wherein the predetermined type file comprises a row-based storage data file and a column-based storage data file, the row-based storage data file is used to execute a query operation corresponding to the transaction processing type query, and the column-based storage data file is used to execute a query operation corresponding to the transaction analysis type query; and determining a query result corresponding to the target query data from the predetermined type file.

[0006] Optionally, determining the query result corresponding to the target query data from the predetermined type file includes: when the data query type is a transaction processing type query, determining the target key data based on the target query data; and determining the query result corresponding to the target key data from the row storage data file.

[0007] Optionally, determining a query result corresponding to the target query data from the predetermined type file includes: when the data query type is a transaction analysis query, determining a target query operation corresponding to the target query data, wherein the target query operation includes: a filtering and aggregation operation; performing the filtering and aggregation operation in the columnar storage data file to obtain a filtering and aggregation result; and determining a query result corresponding to the target query data based on the filtering and aggregation result.

[0008] Optionally, before determining the predetermined type of file in the distributed file system based on the data query type, it also includes: obtaining initial data; converting the format of the initial data into a row storage data format to obtain first data, and storing the first data in the row storage data file; converting the format of the first data into a column storage data format to obtain second data, and storing the second data in the column storage data file.

[0009] Optionally, converting the format of the first data into a columnar storage data format to obtain the second data includes: parsing the first data to obtain key-value pair data; and reorganizing the key-value pair data to obtain the second data.

[0010] Optionally, converting the format of the first data into a columnar storage data format to obtain the second data includes: determining an update condition for the second data; and when the current condition meets the update condition, converting the format of the first data into a columnar storage data format to obtain the second data.

[0011] Optionally, the row-based storage data file is a data file stored in an Hfile format, and the column-based storage data file is a data file stored in a Parquet format.

[0012] According to one aspect of an embodiment of the present invention, there is provided a data query device, comprising: a receiving module, for receiving a data query request, wherein the data query request carries target query data; a first determining module, for determining a data query type of the target query data in response to the data query request, wherein the data query type comprises: a transaction processing type query, a transaction analysis type query; a second determining module, for determining a predetermined type file in a distributed file system according to the data query type, wherein the predetermined type file comprises a row-based storage data file and a column-based storage data file, the row-based storage data file is used to execute a query operation corresponding to the transaction processing type query, and the column-based storage data file is used to execute a query operation corresponding to the transaction analysis type query; and a third determining module, for determining a query result corresponding to the target query data from the predetermined type file.

[0013] According to one aspect of an embodiment of the present invention, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement any of the above-mentioned data query methods.

[0014] According to one aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the above-mentioned data query methods.

[0015] In an embodiment of the present invention, a data query request is received, wherein the data query request carries target query data; in response to the data query request, a data query type of the target query data is determined, wherein the data query types include: transaction processing type query, transaction analysis type query; based on the data query type, a predetermined type file in the distributed file system is determined, wherein the predetermined type file includes a row-based storage data file and a column-based storage data file, wherein the row-based storage data file is used to execute query operations corresponding to transaction processing type queries, and the column-based storage data file is used to execute query operations corresponding to transaction analysis type queries; and a query result corresponding to the target query data is determined from the predetermined type file. By determining the data query type of the target query data, it is helpful to dynamically adjust resource usage according to the query type in the future, avoid excessive consumption of resources, and improve resource utilization. By selecting the predetermined type of files in the distributed file system that match the data query type in a targeted manner according to the data query type, it is possible to perform query operations in a targeted manner according to different file types, reducing unnecessary resource consumption for data reading and processing. By determining the predetermined type of files that match the data query type, full utilization of the characteristics of the row-based storage data file or the column-based storage data file format is achieved, which not only allows accurate data query, but also avoids unnecessary data reading and processing, speeds up the query speed, and thus solves the technical problem in the related technology that it is difficult to balance the data query speed and the data query accuracy when performing data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0017] Figure 1 is a flow chart of a data query method according to an embodiment of the present invention;

[0018] Figure 2 is a flow chart of a data query method provided by an optional embodiment of the present invention;

[0019] Figure 3 It is a module diagram of a data query system provided by an optional embodiment of the present invention;

[0020] Figure 4 is a structural block diagram of a data query device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0023] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0024] HBase: HBase is the abbreviation of HBase database (HBase database). It is a distributed, versioned, non-relational database based on Hadoop and is often used to process large-scale structured data.

[0025] Apache Spark: Apache Spark (Apache Spark, or Spark for short) is a memory-based distributed computing framework, mainly used for large-scale data processing, providing an efficient data processing and analysis method. Spark supports multiple data processing modes, including batch processing, stream processing, machine learning, and graph processing.

[0026] Presto: Presto is an open source distributed SQL query engine used to query various data sources such as Hadoop Distributed File System (HDFS), relational databases, and data warehouses.

[0027] Lakehouse: Lakehouse is a data architecture and analysis paradigm that aims to combine the advantages of data lake and data warehouse. It proposes a unified data storage and analysis solution that can handle large-scale and diverse raw data while supporting high-performance analysis and query operations.

[0028] ETL Tools: ETL tools (data extraction, transformation and loading tools) are commonly used processes in data warehouses to extract data from various data sources, clean, transform and integrate it, and load it into the data warehouse.

[0029] Flink: Flink (Apache Flink, a stream processing and batch processing framework) is a distributed computing framework for stream processing and batch processing, providing low-latency data processing capabilities and suitable for real-time data analysis.

[0030] Distributed file system: A distributed file system is a file system that stores files on multiple computers or nodes. It stores file fragments on different nodes through the network, providing high-performance, high-availability, and fault-tolerant data storage and management. In the field of big data, HDFS is a typical example of a distributed file system.

[0031] Apache Hadoop: Apache Hadoop (Apache Open Source Framework, referred to as Hadoop) is an open source software framework for distributed storage and processing of large-scale data sets. Hadoop mainly consists of two parts: HDFS (Hadoop Distributed File System) and MapReduce. Hadoop is designed to run on large clusters composed of ordinary commercial servers. These servers may be unstable, but Hadoop's design ensures high fault tolerance of data.

[0032] HDFS: HDFS (Hadoop Distributed File System) is Hadoop's distributed file system, which can store and manage large-scale data sets. HDFS is designed to run on cheap hardware and provide high-throughput data access, making it suitable for large-scale data processing applications. HDFS divides files into blocks (the default size is 128MB) and distributes these blocks across nodes in the cluster. Each block has multiple copies to improve data reliability and fault tolerance.

[0033] Row-based storage data files: In row-based storage, data is stored in rows, and each row of data contains all the fields or columns of the record. This storage method is similar to the traditional relational database storage method and is suitable for scenarios where complete row data needs to be read frequently, because it can quickly locate and read the entire row of data without having to read data from other irrelevant columns.

[0034] Columnar storage data files: In columnar storage, data is organized and stored by columns, and each column of data is stored together continuously, even if the data comes from different rows. This storage method is very suitable for data analysis and report generation scenarios, because these queries usually only need to access certain columns of the data set, rather than the complete data of each row. Columnar storage can reduce I / O operations and data processing time, improve query efficiency, and also use the similarity of columns for better data compression.

[0035] HFile: The Chinese name of HFile is usually translated as "HBase file" or "HFile file". HFile is the underlying file format used in Apache HBase to store and manage data in HBase tables. HFile is a row-based storage file.

[0036] Parquet file: Parquet file is usually translated into Chinese as "Parquet file" or simply "Parquet file". Parquet is a columnar storage file format used to efficiently store and process large-scale structured and semi-structured data. In the field of big data and data analysis, the Parquet file format is widely used due to its high-performance reading and writing, data compression, and columnar storage features. This file format can support various data processing frameworks, such as Apache Spark and Presto, for efficient analysis and query operations.

[0037] Example 1

[0038] According to an embodiment of the present invention, an embodiment of a data query method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Figure 1 is a flow chart of a data query method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0040] S102, receiving a data query request, wherein the data query request carries target query data;

[0041] In step S102 provided in the present application, a data query request is received.

[0042] Among them, a data query request is involved, which is a request initiated by the target object for querying data. The data query request can be a request initiated by a client (such as an application, a user interface, or an external system) to a database or data storage system, with the purpose of retrieving or querying data stored in the system. For example, a data query request can be issued through an HBase (i.e., a column-based distributed database) client application programming interface (i.e., API) to retrieve data in HBase.

[0043] Among them, target query data is involved, and the target query data refers to data carried in the data query request and related to the specific data or data range to be queried. The target query data includes the target of the query operation.

[0044] By receiving a data query request about query data initiated by a target object, the received data query request carries target query data related to the specific data or data range to be queried, which provides an accurate data basis for the subsequent data query, so that the subsequent data query can be performed on these specific information, thereby helping to ensure that the correct data query results are matched efficiently and accurately according to the data query request, and further helping to solve the technical problem in related technologies that it is difficult to balance data query speed and data query accuracy when performing data query.

[0045] It should be noted that there is no specific limitation on the method of receiving data query requests, and the method can be customized according to actual applications and scenarios.

[0046] S104, in response to the data query request, determining a data query type of the target query data, wherein the data query type includes: transaction processing type query and transaction analysis type query;

[0047] In step S104 provided in the present application, a data query request is responded to, and a data query type of target query data is determined.

[0048] Among them, the data query type is involved, and the data query type is used to represent the access mode or operation type of the target query data in the data query request. For example, the data query type can be divided into transaction processing type query and transaction analysis type query.

[0049] Among them, transaction processing queries are involved. OLTP (Online Transaction Processing) queries are used to indicate queries that require real-time response, frequent read and write operations, and emphasize data consistency and transaction integrity. The characteristics of transaction processing queries are short response time, many concurrent operations, frequent data updates, and usually involve a single record or a group of closely related records.

[0050] Among them, transaction analysis queries are involved. The transaction analysis queries (OLAP, Online Analytical Processing) are used to represent queries that require complex operations and aggregations on large amounts of data (such as statistical analysis, data mining, report generation, etc.). Compared with OLTP, OLAP has lower real-time requirements.

[0051] By responding to the data query request and determining the data query type of the target query data, the data query type that the data query request needs to query is determined, which helps to select the most suitable query path in a targeted manner in the future, thereby helping to improve query efficiency and reduce resource waste. In addition, the data query types include transaction processing queries and transaction analysis queries. When it is determined to be a transaction processing query, the database (such as HBase) can be directly accessed, thereby reducing latency and providing instant response. When it is determined to be a transaction analysis query, unnecessary data reading can be avoided and query time can be reduced, so that resource usage can be dynamically adjusted according to the query type, avoiding excessive resource consumption and improving resource utilization, thereby realizing a unified query entry for transaction processing queries and transaction analysis queries, improving data access and analysis capabilities, simplifying user operations, and thus helping to improve query efficiency in the future while ensuring query accuracy.

[0052] It should be noted that the method for determining the data query type of the target query data in response to the data query request is not specifically limited here, and can be customized according to actual applications and scenarios.

[0053] S106, determining a predetermined type of file in the distributed file system according to the data query type, wherein the predetermined type of file includes a row-based storage data file and a column-based storage data file, the row-based storage data file is used to execute a query operation corresponding to a transaction processing query, and the column-based storage data file is used to execute a query operation corresponding to a transaction analysis query;

[0054] In step S106 provided in the present application, a predetermined type of file in the distributed file system is determined.

[0055] Among them, a distributed file system is involved. The distributed file system is a file system architecture that shares files between multiple computers (nodes). These nodes are connected together through a network to jointly manage a large amount of data. For example, HDFS (Hadoop Distributed File System) of Apache Hadoop (i.e., Apache open source framework) is a distributed file system that can store and manage massive amounts of data and provide high fault tolerance and high-performance data access capabilities. HDFS achieves data reliability and access efficiency by dividing data into blocks and distributing storage on multiple nodes.

[0056] The predetermined type file is a file type predefined in a distributed file system according to a specific design or configuration. For example, the predetermined type file may include a row-type storage data file and a column-type storage data file.

[0057] Among them, a row-based storage data file is involved, which is a file format that stores data in rows. In the row-based storage data file, each row of data is stored as a complete record, including all column data. The row-based storage data file can be applied to transaction processing queries, because queries usually need to obtain data from one or more complete records. The row-based storage data file can reduce I / O (Input / Output) operations during queries and provide faster query responses. For example, the HFile file (i.e., HBase file) used in HBase is a row-based storage data file.

[0058] Among them, a column-based storage data file is involved, which is a file format that stores data by column. In the column-based storage data file, data in the same column is aggregated and stored together. The column-based storage data file can be suitable for transaction analysis queries because the column-based storage data file can only read the column data required for the query, avoiding unnecessary data reading, and improving query efficiency and data compression ratio. For example, Parquet file (Parquet file) is a column-based storage data file.

[0059] By selecting predetermined type files in the distributed file system that match the data query type in a targeted manner according to the data query type, wherein the predetermined type files include row-based storage data files and column-based storage data files, query operations can be performed in a targeted manner according to different file types, thereby reducing unnecessary resource consumption for data reading and processing, thereby effectively reducing data reading and processing time and improving query response time while ensuring query accuracy.

[0060] It should be noted that the method for determining the predetermined type of files in the distributed file system based on the data query type is not specifically limited here, and can be customized according to actual applications and scenarios.

[0061] S108, determining a query result corresponding to the target query data from the predetermined type of files.

[0062] In step S108 provided in the present application, a query result corresponding to the target query data is determined.

[0063] The query result is a specific data result obtained by querying from a predetermined type of file that matches the data query type after processing the data query request.

[0064] It should be noted that the method for determining the query result corresponding to the target query data from the predetermined type of file is not specifically limited here, and can be customized according to the actual application and scenario.

[0065] By determining a predetermined type of file that matches the data query type, it is ensured that in the case of a row-based storage data file, the complete row data can be quickly retrieved in a targeted manner, and unnecessary column data reading can be avoided. In the case of a column-based storage data file, the data of a specific column can be efficiently processed and analyzed, reducing I / O operations and data processing time, thereby achieving full utilization of the characteristics of the row-based storage data file or the column-based storage data file format, not only accurate data query can be performed, but also unnecessary data reading and processing can be avoided, and the query speed is accelerated, thereby effectively solving the technical problem of the related technology in that it is difficult to balance the data query speed and the data query accuracy when performing data query.

[0066] Through the above steps S102-S108, a data query request is received, wherein the data query request carries target query data; in response to the data query request, a data query type of the target query data is determined, wherein the data query types include: transaction processing type query, transaction analysis type query; based on the data query type, a predetermined type file in the distributed file system is determined, wherein the predetermined type file includes a row-based storage data file and a column-based storage data file, the row-based storage data file is used to execute query operations corresponding to transaction processing type queries, and the column-based storage data file is used to execute query operations corresponding to transaction analysis type queries; from the predetermined type file, a query result corresponding to the target query data is determined. By determining the data query type of the target query data, it is helpful to dynamically adjust resource usage according to the query type in the future, avoid excessive consumption of resources, and improve resource utilization. By selecting the predetermined type of files in the distributed file system that match the data query type in a targeted manner according to the data query type, it is possible to perform query operations in a targeted manner according to different file types, reducing unnecessary resource consumption for data reading and processing. By determining the predetermined type of files that match the data query type, full utilization of the characteristics of the row-based storage data file or the column-based storage data file format is achieved, which not only allows accurate data query, but also avoids unnecessary data reading and processing, speeds up the query speed, and thus solves the technical problem in the related technology that it is difficult to balance the data query speed and the data query accuracy when performing data query.

[0067] As an optional embodiment, determining the query results corresponding to the target query data from a predetermined type file includes: when the data query type is a transaction processing type query, determining the target key data based on the target query data; and determining the query results corresponding to the target key data from a row-based storage data file.

[0068] In this embodiment, specific steps of determining a query result corresponding to target query data from a file of a predetermined type are described.

[0069] Among them, the target key data is involved. The target key data is the key value used to locate a specific data row determined based on the target query data when the data query type is a transaction processing query. For example, in HBase, data is stored in the form of key-value pairs. The key (i.e., RowKey) is the value used to uniquely identify a row of data in HBase. Therefore, the target key data is the main basis for data query.

[0070] In the steps involved in this embodiment, first, according to the target query data in the query request, the target key data for locating a specific data row, that is, the specific RowKey to be queried, is parsed and determined. Then, the target key data is searched in the row-type storage data file to locate the entire row of data corresponding to the target key data, thereby obtaining the query result related to the target query data.

[0071] By parsing and determining the target key data used to locate specific data rows based on the target query data in the query request, unnecessary data scanning and reading can be reduced, significantly improving the execution efficiency of transaction processing queries. And by searching in the row-based storage data file through the target key data, not only can the data rows required for the query operation be found quickly, that is, the entire row of data corresponding to the target key data can be located, thereby obtaining query results related to the target query data, but it can also ensure that the read data is consistent and will not be affected by other row data, thereby ensuring the accuracy of data query and improving the efficiency of data query, thereby achieving fast and accurate response to transaction processing queries, while ensuring data consistency and transaction integrity.

[0072] As an optional embodiment, determining a query result corresponding to the target query data from a predetermined type file includes: when the data query type is a transaction analysis query, determining a target query operation corresponding to the target query data, wherein the target query operation includes: filtering and aggregation operations; performing filtering and aggregation operations in a columnar storage data file to obtain filtering and aggregation results; and determining a query result corresponding to the target query data based on the filtering and aggregation results.

[0073] In this embodiment, specific steps of determining a query result corresponding to target query data from a file of a predetermined type are described.

[0074] Among them, it involves the target query operation, which is to perform data screening, aggregation, statistics, etc. to achieve data retrieval and analysis according to the target query data when the data query type is a transaction analysis query. The target query operation includes screening and aggregation operations.

[0075] Among them, the filtering and aggregation operations are involved, which are data operations such as data filtering (filtering data according to specific conditions), data aggregation (summarizing the filtered data or calculating statistical indicators), data sorting, and data grouping. For example, in the target object behavior analysis, you can first filter out all the target object behavior data within a certain time period, then group them by the target object ID, and finally calculate the average number of behaviors of each target object.

[0076] Among them, it involves filtering and aggregation results, which are the operation results obtained after performing filtering and aggregation operations on the target query data when the data query type is a transaction analysis query.

[0077] Determine the query result corresponding to the target query data from a file of a predetermined type, including: when the data query type is a transaction analysis query, determine the target query operation corresponding to the target query data, wherein the target query operation includes: filtering and aggregation operations; performing filtering and aggregation operations in a columnar storage data file to obtain filtering and aggregation results; and determining the query result corresponding to the target query data based on the filtering and aggregation results.

[0078] In the steps involved in this embodiment, first, according to the target query data, it is clear that the target query operation including the screening and aggregation operation needs to be performed. Then, in the column storage data file, according to the characteristics of the column storage data file, only the columns related to the target query data are read and processed, and screening (such as filtering out data that does not meet the conditions) and aggregation (such as calculating the sum, average, etc.) operations are performed to obtain the screening and aggregation results. Finally, according to the screening and aggregation results, a detailed analysis report or data overview is generated as the query result corresponding to the target query data.

[0079] When the data query type is a transaction analysis query, the target query operation corresponding to the target query data, including the filtering and aggregation operation, is determined, which can support multi-condition filtering and multi-dimensional aggregation to meet the complex needs of transaction analysis queries. Transaction analysis queries usually involve the filtering, aggregation and statistics of a large amount of data. The filtering and aggregation operations are performed in the columnar storage data file, so that only the columns related to the operation can be read and processed, thereby greatly reducing I / O operations and resource consumption, and improving resource utilization efficiency. It is possible to extract key information directly related to the target query data from the massive data, significantly improving the execution efficiency of such queries, so that the query results corresponding to the target query data can be accurately and quickly generated, which helps to solve the technical problem of balancing data query speed and data query accuracy when performing data queries.

[0080] As an optional embodiment, before determining a predetermined type of file in a distributed file system based on a data query type, it also includes: obtaining initial data; converting the format of the initial data into a row storage data format to obtain first data, and storing the first data in a row storage data file; converting the format of the first data into a column storage data format to obtain second data, and storing the second data in a column storage data file.

[0081] In this embodiment, the specific steps before determining the predetermined type of files in the distributed file system according to the data query type are described.

[0082] Among them, initial data is involved, which is data collected from a data source through specific technical means and needs to be stored in a database after specific processing.

[0083] Among them, the row-based storage data format is involved, which is a format in which data is stored in the form of rows, that is, each data row contains all related fields. For example, in HBase, data is usually stored in a row-based data format to facilitate row-based query operations.

[0084] Among them, first data is involved, and the first data is data obtained after initial data is converted into a row storage data format.

[0085] Among them, the columnar storage data format is involved, which is a format that stores data in the form of columns, and each column of data contains all the values ​​of a specific field. For example, the Parquet file format is a columnar storage data format that can be used in big data processing tools such as Apache Spark and Presto.

[0086] Among them, second data is involved, and the second data is data obtained by converting the first data into a column storage data format.

[0087] In the steps involved in this embodiment, initial data is obtained and converted into a row storage data format to obtain first data, and then the first data is stored in a row storage data file (such as an HFile file). Then, the first data is converted into a column storage data format to obtain second data, and then the second data is stored in a column storage data file (such as a Parquet file).

[0088] By converting the initial data into a row storage data format to obtain the first data, and then converting the first data into a column storage data format to obtain the second data, the first data is stored in a row storage data file, and the second data is stored in a column storage data file. This dual storage method can respond quickly to various query requests, enhance the stability and reliability of data processing, contribute to the rational allocation and use of resources, reduce unnecessary data processing and storage costs, and achieve flexible adaptation to different types of query requirements, improve data processing efficiency and response speed, so that the appropriate storage format can be selected according to the query type for data retrieval in a targeted manner, reduce data processing delays, improve the real-time nature of decision support, and combine the flexibility of the data lake with the high-performance query capabilities of the data warehouse, providing strong support for building a lake-warehouse integrated data management architecture, so that its massive data can be managed and analyzed more efficiently.

[0089] As an optional embodiment, converting the format of the first data into a columnar storage data format to obtain the second data includes: parsing the first data to obtain key-value pair data; and reorganizing the key-value pair data to obtain the second data.

[0090] In this embodiment, specific steps of converting the format of first data into a column storage data format to obtain second data are described.

[0091] Among them, key-value pair data is involved, which is a kind of data stored in a pairing of "key" and "value". In this key-value pair data, each key corresponds to a value, and the key is usually unique and is used to quickly locate and access data. For example, in HBase, the basic unit of data storage is the key-value pair. Each key-value pair consists of RowKey, Column Family (table structure), Column Qualifier (data column) and Timestamp, where RowKey is the primary key of the data, ColumnFamily and Column Qualifier together constitute the column identifier, and Timestamp records the data version information.

[0092] In the steps involved in this embodiment, the first data in the column storage data format is parsed to obtain key-value pair data, and then the key-value pair data is reorganized. The data values ​​​​with the same column identifier (Column Family and ColumnQualifier combination) can be reorganized into columns to form a column data structure, thereby obtaining the second data in the column storage data format.

[0093] The second data obtained after the key-value pair data is reorganized is presented in the form of column storage, which can greatly reduce the time of data query and analysis, especially when processing large amounts of data and complex queries, it can significantly improve the speed and response time of data analysis. By parsing the first data in the column storage data format, the key-value pair data is obtained, and then the key-value pair data is reorganized to form a column data structure, thereby obtaining the second data in the column storage data format, which helps to flexibly respond to different types of query requirements. Whether it is transaction processing or transaction analysis, it can provide efficient data services, thereby solving the problem that it is difficult to simultaneously meet the needs of transaction processing and transaction analysis under a single data storage format, improving the efficiency and complexity support of data analysis, and thus helping to achieve integrated management of data lakes and data warehouses.

[0094] As an optional embodiment, converting the format of the first data into a columnar storage data format to obtain the second data includes: determining an update condition for the second data; and when the current condition meets the update condition, converting the format of the first data into a columnar storage data format to obtain the second data.

[0095] In this embodiment, specific steps of converting the format of first data into a column storage data format to obtain second data are described.

[0096] Among them, the update condition is involved, which is a specific rule or condition that needs to be met when converting the first data into the second data. The update condition can include a specific time interval (such as every hour or every day), the amount of data reaching a certain threshold, or a request for data format update. For example, the data in the Parquet file can be generated from the data in the HFile file through a regular batch job, or automatically converted and generated when the data in the HFile file meets certain conditions.

[0097] Among them, the current condition is involved, and the current condition is the actual condition corresponding to the current state when the first data needs to be converted into the second data. Only when the current condition meets the update condition will it trigger the conversion of the format of the first data into the column storage data format to obtain the second data.

[0098] In the steps involved in this embodiment, the update condition of the second data is determined, and then it is determined whether the current condition meets the update condition. When the current condition meets the update condition, the data format conversion is performed to convert the first data from the row storage data format to the column storage data format to obtain the second data.

[0099] By setting the update condition, it can be ensured that the first data is converted only when it meets the specific standard, and unnecessary data format conversion can be avoided when the data volume of the first data is small or inactive. By determining the update condition of the second data and converting the data format when the current condition meets the update condition, unnecessary resource consumption and data redundancy can be avoided, the efficiency of data processing and storage can be improved, and the purpose of on-demand conversion, resource optimization and improved processing performance can be achieved, achieving the beneficial effect of reducing resource waste and improving query performance.

[0100] As an optional embodiment, the row-based storage data file is a data file stored in the Hfile format, and the column-based storage data file is a data file stored in the Parquet format.

[0101] In this embodiment, a specific row-based storage data file and a column-based storage data file are described.

[0102] Among them, the Hfile format data file is involved. The Hfile format data file (HBase File) is the underlying file format for storing data in the Apache HBase database. The Hfile format data file stores data in the form of key-value pairs in HBase, and each key-value pair consists of RowKey, Column Family, ColumnQualifier and Timestamp.

[0103] Among them, data files stored in Parquet format are involved. The Parquet format is a columnar storage format that stores data in columns and can be applied to big data processing and analysis frameworks such as Apache Spark and Presto.

[0104] In the steps involved in this embodiment, it is explained that the row-based storage data file can be a data file stored in the Hfile format, and the column-based storage data file can be a data file stored in the Parquet format.

[0105] The row-based data file can be a data file stored in the Hfile format, which can realize fast read and write operations. The data file stored in the Hfile format can store these key-value pairs and realize fast data search and range query through the internal data structure and index mechanism. The data file stored in the Hfile format also supports data compression and encoding, as well as the storage of multi-version data, which enables HBase to handle high-concurrency write operations and provide time travel query capabilities. The column-based data file can be a data file stored in the Parquet format. The data file stored in the Parquet format will store the data of each column independently, that is, if a data set contains multiple columns, this can significantly reduce I / O operations and improve query efficiency when only specific columns are involved in data query. The data file stored in the Parquet format also supports data compression, which can effectively reduce the storage space occupied while maintaining fast data reading. The combination of the Hfile format storage data file and the Parquet format storage data file can meet the needs of transaction processing queries and transaction analysis queries at the same time, thereby helping to achieve accurate and efficient data query, and can effectively solve the technical problem in the related technology that it is difficult to balance data query speed and data query accuracy when performing data query.

[0106] Based on the above embodiments and optional embodiments, an optional implementation is provided, which is described in detail below.

[0107] In the related art, when performing data query, there is a technical problem that it is difficult to balance the data query speed and the data query accuracy.

[0108] In view of this, an optional implementation of the present invention provides a data query method and system, which can also be called a data query method and system with Parquet as the underlying storage, which can effectively solve the technical problem in the related technology that it is difficult to balance the data query speed and data query accuracy when performing data query.

[0109] Figure 2 is a flow chart of a data query method provided by an optional embodiment of the present invention, Figure 3 is a data query system module diagram provided by an optional embodiment of the present invention, such as Figure 2 ,as well as Figure 3 As shown, the details will be introduced below.

[0110] (I) Data query method

[0111] This method combines HBase and Parquet in an integrated lake-warehouse environment. By using HBase as the storage engine of the data lake and taking advantage of its distributed storage and high-performance read and write features, the capacity expansion of the data lake and the efficiency of data storage are achieved; at the same time, Parquet is used as the underlying storage format to organize and compress data in a columnar storage manner, which improves the query performance and storage efficiency of the data. That is, this method first adds an adapter to the Region Server of HBase, generates Parquet files from files when HBase data is flushed to the disk, uses appropriate query tools or programming languages, reads data in Parquet format from HBase, and performs data query and analysis, thereby realizing the integrated management of the data lake and the data warehouse, including data integration, conversion, and processing operations.

[0112] The following are the specific implementation steps of this method:

[0113] S1. Receive a data query request, wherein the data query request carries target query data;

[0114] S2. In response to the data query request, determine the data query type of the target query data, wherein the data query type includes: transaction processing type query and transaction analysis type query;

[0115] For example, transaction processing queries usually involve a single or a group of related data records, and they focus on the real-time and consistency of data. For example, queries in banking transaction systems, checking account balances, performing transfer operations, etc., are usually based on fixed keys (such as account IDs). They need to return results immediately and ensure that the accuracy of the data is not affected by other concurrent operations. This type of query is designed with ACID (atomicity, consistency, isolation, and persistence) properties in mind to ensure that each operation is accurate, consistent, and can be executed independently of other operations. In HBase, transaction processing queries are usually performed through the HBase API, and the underlying query is data in HFile format, because HBase provides efficient key-based query capabilities through HFile.

[0116] Transactional analysis queries involve a large number of data records, and their goal is to analyze the data and discover patterns, trends, or insights. For example, analyze the target behavior of all target objects in the past year to determine the future trend of the target behavior. This type of query may require operations such as aggregation, filtering, and sorting of data. Usually, they do not care about the real-time performance of a single operation, but focus on overall query efficiency and the ability to process large amounts of data. In the scenario of lake warehouse integration, transactional analysis queries can be performed directly from Parquet format data files through tools such as Apache Spark and Presto, because the columnar storage and compression features of the Parquet format are very suitable for analytical queries on large-scale data.

[0117] S3. Determine a predetermined type of file in the distributed file system according to the data query type, wherein the predetermined type of file includes a row-based storage data file and a column-based storage data file. The row-based storage data file is used to execute a query operation corresponding to a transaction processing query, and the column-based storage data file is used to execute a query operation corresponding to a transaction analysis query.

[0118] Specifically, the row-based storage data files in S3 are stored in the Hfile format, and the column-based storage data files are stored in the Parquet format.

[0119] For example, HBase is designed to focus on real-time data reading and writing. Its HFile format stores data blocks and index information, which can quickly respond to row-key-based queries. Although Parquet files perform well in data batch processing and analysis, they are not designed for random access and low-latency queries based on row keys. Parquet files usually traverse the entire file or the corresponding part of the file when querying, which may not be suitable for low-latency, row-key query scenarios common in HBase.

[0120] Therefore, when performing transaction processing queries, HBase will use its internal HFile storage format to provide fast, key-based query services. When performing transaction analysis queries, that is, when a batch of data needs to be queried for analysis and processing, these queries are usually handed over to data analysis tools such as Apache Spark and Presto, which can directly read and process data in Parquet format, because the columnar storage and compression characteristics of the Parquet file format are very suitable for batch analysis and large-scale data queries. In other words, transaction processing queries rely on the real-time performance of HBase and the efficient indexing of the HFile format, while transaction analysis queries use data in Parquet format for batch processing and complex analysis. The query and analysis of the two are complementary, and together constitute the data access and analysis capabilities in the lake warehouse integration solution.

[0121] Specifically, before S3, it also includes:

[0122] A1. Obtain initial data;

[0123] A2. Convert the format of the initial data into a row-type storage data format to obtain first data, and store the first data in a row-type storage data file;

[0124] A3. Convert the format of the first data into a column storage data format to obtain second data, and store the second data in a column storage data file.

[0125] Specifically, A3 also includes:

[0126] A31. Determine an update condition for the second data;

[0127] A32. If the current condition satisfies the update condition, parse the first data to obtain key-value pair data;

[0128] The key-value pair data is reorganized to obtain second data.

[0129] For example, when generating Parquet files, when data is written to HBase, the corresponding HFile files and Parquet files are generated at the same time.

[0130] HBase and Parquet files coexist, that is, when data is written to the HBase database, the data is stored in HFile format and Parquet format respectively, and the coexistence of HBase and Parquet files is ensured, that is, HBase tables and Parquet files are maintained at the same time; for write operations, the corresponding HFile files and Parquet files are generated while writing data to HBase; for backtracking and query operations, you can choose to read data from the HBase table or from the Parquet file to meet different needs.

[0131] S4. Determine the query result corresponding to the target query data from the predetermined type of files.

[0132] Specifically, S4 also includes:

[0133] S41. When the data query type is a transaction processing query, the target key data is determined based on the target query data, and the query result corresponding to the target key data is determined from the row-type storage data file.

[0134] S42. When the data query type is a transaction analysis query, determine the target query operation corresponding to the target query data, wherein the target query operation includes: filtering and aggregation operations; performing filtering and aggregation operations in a columnar storage data file to obtain filtering and aggregation results; and determining the query results corresponding to the target query data based on the filtering and aggregation results.

[0135] For example, integrated management of query and analysis as well as data lake and data warehouse, that is, using query tools or programming languages ​​that support HBase and Parquet formats to perform query and analysis operations from HBase tables or Parquet files; according to needs and performance requirements, select appropriate data sources for query and analysis to meet the integrated management of data lake and data warehouse.

[0136] (II) Data Query System

[0137] The data query system includes an HBase integration module, a Parquet integration module, a data reading and writing module, a data query and analysis module, and a data integration management module.

[0138] Among them, the HBase integration module is used to configure and deploy the HBase cluster to ensure its availability and scalability; it provides an API or client for connecting and interacting with the HBase cluster. The Parquet integration module is used to configure and deploy the Parquet component to ensure its integration with HBase; it provides an API or tool for reading and writing Parquet files. The data reading and writing module is a module used to read data from HBase, which may include operations such as data conversion, processing, and cleaning; it is a module used to write data to HBase in Parquet format. The data query and analysis module is a module that provides query tools or programming languages ​​for reading data from HBase or Parquet files for query and analysis. The data integration management module is a module used to integrate, transform, and process data to achieve integrated management of the data lake and data warehouse; it may include ETL tools or custom development programs for transferring data from the data lake to the data warehouse.

[0139] Optionally, the HBase integration module is implemented by the following steps:

[0140] B1. Build and deploy HBase related environment.

[0141] B2. Configure HBase-related environment, data writing and WAL (Write-Ahead Log) records. When data is written to HBase, ensure that the WAL record contains complete data change information.

[0142] B3. Include sufficient metadata in the WAL record to facilitate subsequent read and write operations.

[0143] Optionally, the Parquet data module integration steps include:

[0144] C1. Each RegionServer integrates a Parquet data processing module.

[0145] C2, Parquet data processing module performs regular scanning, analysis and filtering on HBase's Hlog (HBase log), namely WAL log.

[0146] C3. Write the analyzed and filtered data in the form of Parquet to the HDFS (Hadoop Distributed File System) for data management storage. Each different table needs to be split into different Parquet files for storage, so that further query and analysis can be performed efficiently later.

[0147] Optionally, the data query and analysis module can implement the following steps:

[0148] D1. Spark (Apache Spark, a memory-based distributed computing framework) is used to query and analyze the stored Parquet file data.

[0149] D2. Submit the calculated and analyzed data to data integration management for processing.

[0150] Optionally, the data integration management module can implement the following steps:

[0151] E1. Implement integrated management of data lake and data warehouse, including Spark and Flink (Apache Flink, a stream processing and batch processing framework) or custom developed programs to transfer data from the data lake to the data warehouse.

[0152] E2. The data can be rewritten into HBase, which can provide real-time query scenarios later.

[0153] E3. The data can be written into the data report storage system, and the report data can be generated later.

[0154] Through the above optional implementation, at least the following beneficial effects can be achieved:

[0155] (1) Compared with the related art, the present invention determines the data query type of the target query data, thereby facilitating the subsequent dynamic adjustment of resource usage according to the query type, avoiding excessive resource consumption and improving resource utilization. Moreover, by selectively selecting a predetermined type of file in a distributed file system that matches the data query type according to the data query type, query operations can be performed in a targeted manner according to different file types, thereby reducing unnecessary resource consumption for data reading and processing. Thus, by selecting a predetermined type of file that matches the data query type, full utilization of the characteristics of the row-based storage data file or the column-based storage data file format is achieved, which not only enables accurate data query, but also avoids unnecessary data reading and processing, thereby accelerating the query speed, thereby solving the technical problem in the related art that it is difficult to balance the data query speed and the data query accuracy when performing data query.

[0156] (2) Compared with the related art, the present invention selectively selects predetermined type files in a distributed file system that match the data query type according to the data query type, wherein the predetermined type files include row-based storage data files and column-based storage data files. Thus, query operations can be performed in a targeted manner according to different file types, thereby reducing unnecessary resource consumption for data reading and processing. Thus, on the basis of ensuring the accuracy of the query, the time for data reading and processing can be effectively reduced, thereby improving the query response time.

[0157] (3) Compared with the related art, the present invention can reduce unnecessary data scanning and reading by parsing and determining the target key data used to locate the specific data row according to the target query data in the query request, and significantly improve the execution efficiency of transaction processing queries. And by searching the target key data in the row-type storage data file, it is not only possible to quickly find the data row required for the query operation, that is, locate the entire row of data corresponding to the target key data, so as to obtain the query results related to the target query data, but also to ensure that the read data is consistent and will not be affected by other row data, thereby ensuring the accuracy of data query and improving the efficiency of data query, thereby achieving a fast and accurate response to transaction processing queries, while ensuring the consistency of data and the integrity of transactions.

[0158] (4) Compared with the related art, the present invention obtains the first data by converting the initial data into a row storage data format, and then converts the first data into a column storage data format to obtain the second data, and stores the obtained first data in a row storage data file, and stores the second data in a column storage data file. This dual storage method can respond quickly to various query requests, enhance the stability and reliability of data processing, contribute to the rational allocation and use of resources, reduce unnecessary data processing and storage costs, and achieve flexible adaptation to different types of query requirements, improve the efficiency and response speed of data processing, so that the appropriate storage format can be selected according to the query type for data retrieval in a targeted manner, reduce the delay of data processing, improve the real-time nature of decision support, and combine the flexibility of the data lake with the high-performance query capability of the data warehouse, providing strong support for building a lake-warehouse integrated data management architecture, so that its massive data can be managed and analyzed more efficiently.

[0159] (5) Compared with the related art, the present invention can ensure that the first data is converted only when it meets the specific standard through the setting of the update condition, and can avoid unnecessary data format conversion when the data volume of the first data is small or inactive. By determining the update condition of the second data and converting the data format when the current condition meets the update condition, unnecessary resource consumption and data redundancy can be avoided, the efficiency of data processing and storage can be improved, and the purpose of on-demand conversion, resource optimization and improved processing performance can be achieved, achieving the beneficial effect of reducing resource waste and improving query performance.

[0160] (6) Compared with the related art, the present invention may realize the integrated management of data lake and data warehouse by allowing the coexistence of HFile and Parquet files, and provide more flexible data access and data analysis capabilities. By converting the underlying storage in HBase from the original closed HFile to the open and widely supported Parquet, it is possible to maintain HBase's high-performance data batch, real-time writing and query capabilities, while seamlessly integrating Apache Spark, Presto and other computing and analysis capabilities, thus serving as a new lake-warehouse integration solution. In particular, it allows current HBase stock users to smoothly migrate to the lake-warehouse integration solution. Converting HFile storage to Parquet format may provide higher query performance. Parquet files use columnar storage and compression technology, which can effectively reduce I / O operations and improve query efficiency. The compression and columnar storage characteristics of Parquet files can significantly reduce storage space occupancy. Compared with the HFile storage format, Parquet files can more effectively utilize storage resources.

[0161] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0162] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0163] Example 2

[0164] According to an embodiment of the present invention, a device for implementing the above data query method is also provided. Figure 4 is a structural block diagram of a data query device according to an embodiment of the present invention. Figure 4As shown, the device includes: a receiving module 402, a first determining module 404, a second determining module 406 and a third determining module 408. The device is described in detail below.

[0165] A receiving module 402 is used to receive a data query request, wherein the data query request carries target query data; a first determining module 404 is connected to the above-mentioned receiving module 402, and is used to determine the data query type of the target query data in response to the data query request, wherein the data query types include: transaction processing type query, transaction analysis type query; a second determining module 406 is connected to the above-mentioned first determining module 404, and is used to determine a predetermined type file in the distributed file system based on the data query type, wherein the predetermined type file includes a row-based storage data file and a column-based storage data file, the row-based storage data file is used to execute query operations corresponding to transaction processing type queries, and the column-based storage data file is used to execute query operations corresponding to transaction analysis type queries; a third determining module 408 is connected to the above-mentioned second determining module 406, and is used to determine the query result corresponding to the target query data from the predetermined type file.

[0166] It should be noted here that the above-mentioned receiving module 402, the first determination module 404, the second determination module 406 and the third determination module 408 correspond to steps S102 to S108 in implementing the data query method, and the instances and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned embodiment 1.

[0167] Example 3

[0168] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to execute the instructions to implement any of the above data query methods.

[0169] Example 4

[0170] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute any of the above data query methods.

[0171] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0172] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0175] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0177] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data query method, characterized in that: include: receiving a data query request, wherein the data query request carries target query data; In response to the data query request, determining the data query type of the target query data, wherein the data query type includes: transaction processing type query and transaction analysis type query; Determine a predetermined type of file in the distributed file system according to the data query type, wherein the predetermined type of file includes a row-based storage data file and a column-based storage data file, the row-based storage data file is used to execute a query operation corresponding to the transaction processing query, and the column-based storage data file is used to execute a query operation corresponding to the transaction analysis query; A query result corresponding to the target query data is determined from the predetermined type of files.

2. The method according to claim 1, characterized in that Determining a query result corresponding to the target query data from the predetermined type file includes: In the case where the data query type is a transaction processing query, determining target key data according to the target query data; A query result corresponding to the target key data is determined from the row-type storage data file.

3. The method according to claim 1, characterized in that Determining a query result corresponding to the target query data from the predetermined type file includes: In the case where the data query type is a transaction analysis query, determining a target query operation corresponding to the target query data, wherein the target query operation includes: a screening aggregation operation; Execute the screening and aggregation operation in the column-type storage data file to obtain a screening and aggregation result; According to the screening and aggregation results, a query result corresponding to the target query data is determined.

4. The method according to claim 1, characterized in that: Before determining the predetermined type of files in the distributed file system according to the data query type, the method further includes: Get initial data; Convert the format of the initial data into a line-type storage data format to obtain first data, and store the first data in the line-type storage data file; The format of the first data is converted into a column-based storage data format to obtain second data, and the second data is stored in the column-based storage data file.

5. The method according to claim 4, characterized in that The converting the format of the first data into a columnar storage data format to obtain the second data includes: Parsing the first data to obtain key-value pair data; The key-value pair data is reorganized to obtain the second data.

6. The method according to claim 4, characterized in that Converting the format of the first data into a columnar storage data format to obtain second data includes: determining an update condition for the second data; When the current condition satisfies the update condition, the format of the first data is converted into a column storage data format to obtain the second data.

7. The method according to any one of claims 1 to 6, characterized in that: The row-based storage data file is a data file stored in Hfile format, and the column-based storage data file is a data file stored in Parquet format.

8. A data query device, characterized in that: include: A receiving module, configured to receive a data query request, wherein the data query request carries target query data; A first determination module is used to determine the data query type of the target query data in response to the data query request, wherein the data query type includes: transaction processing type query and transaction analysis type query; A second determination module is used to determine a predetermined type of file in the distributed file system according to the data query type, wherein the predetermined type of file includes a row-type storage data file and a column-type storage data file, the row-type storage data file is used to execute the query operation corresponding to the transaction processing query, and the column-type storage data file is used to execute the query operation corresponding to the transaction analysis query; The third determination module is used to determine the query result corresponding to the target query data from the predetermined type of file.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data query method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the data query method as claimed in any one of claims 1 to 7.

Images