A data query method and device, computer equipment and a storage medium

Abstract

The application provides a data query method and device, computer equipment and a storage medium, which can be applied to the field of cloud computing or intelligent transportation, and is used for solving the problem of low data query efficiency. The method comprises the following steps: obtaining a target query statement, determining at least one target data source and at least one target data unit indicated by the target query statement, respectively performing sub-condition division on the query conditions to obtain a plurality of sub-conditions, respectively generating a sub-query statement corresponding to each of the plurality of sub-conditions, respectively acquiring sub-data indicated by each of the plurality of sub-query statements through a data transmission channel corresponding to each of the plurality of sub-query statements, performing merging processing on the obtained plurality of sub-data to generate target data indicated by the target query statement, and obtaining the target data indicated by the target query statement by simultaneously executing the plurality of sub-query statements and merging the plurality of sub-data, thereby improving the data query efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data query method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the continuous development of technology, the amount of data that devices query from the same data source is increasing day by day, and the query rules for querying data from different data sources are also becoming more and more diverse.

[0003] Under related technologies, when a device queries data from a data source, the amount of data that can be transmitted through the data transmission channel is limited. Therefore, when the amount of data to be queried is large, the data transmission channel is prone to congestion during the process of the data source transmitting the queried data to the device. This results in a longer time for the device to obtain the complete queried data, leading to low data query efficiency.

[0004] On the other hand, since the query rules for querying data differ across different data sources, when a device queries data from multiple data sources, it needs to perform multiple query processes based on the respective query rules of each data source. This makes the process of obtaining the complete queried data quite complex and results in low data query efficiency.

[0005] It is evident that, under the relevant technologies, data query efficiency is low, whether querying data from the same data source or from multiple data sources. Summary of the Invention

[0006] This application provides a data query method, apparatus, computer equipment, and storage medium to solve the problem of low data query efficiency.

[0007] Firstly, a data query method is provided, including:

[0008] Obtain the target query statement, parse the target query statement, determine at least one target data source indicated by the target query statement, and at least one target data unit in each target data source that meets the query conditions contained in the target query statement;

[0009] Based on the data storage characteristics of at least one target data unit, the query conditions are divided into sub-conditions to obtain multiple sub-conditions, and sub-query statements corresponding to each of the multiple sub-conditions are generated respectively.

[0010] Through the data transmission channels corresponding to each of the multiple sub-query statements, the sub-data indicated by each of the multiple sub-query statements is obtained from the at least one target data unit;

[0011] The obtained sub-data is merged to generate the target query statement indicating the target data to be queried.

[0012] Secondly, a data query device is provided, comprising:

[0013] Acquisition module: used to acquire the target query statement, parse the target query statement, determine at least one target data source indicated by the target query statement, and at least one target data unit in each target data source that meets the query conditions contained in the target query statement;

[0014] Processing module: used to divide the query conditions into sub-conditions based on the data storage characteristics of at least one target data unit, obtain multiple sub-conditions, and generate sub-query statements corresponding to each of the multiple sub-conditions respectively;

[0015] The processing module is further configured to: obtain the sub-data indicated by each of the multiple sub-query statements from the at least one target data unit through the data transmission channels corresponding to each of the multiple sub-query statements;

[0016] The processing module is also used to: merge the obtained multiple sub-data to generate the target query statement indicating the target data to be queried.

[0017] Optionally, each piece of data stored in the target data unit has its own index information, and each index information is used to uniquely represent a piece of data or a group of data in the target data unit;

[0018] The processing module is specifically used for:

[0019] For each of the at least one target data unit, the following operations are performed:

[0020] Based on the data storage characteristics of a target data unit, determine the index information of each of the multiple data that meet the query conditions for the target data unit;

[0021] Based on the index information, the multiple data are grouped to generate multiple data shards, wherein each data shard contains at least one piece of data;

[0022] Based on the multiple data shards, the query conditions are divided into sub-conditions to obtain the sub-conditions corresponding to each of the multiple data shards.

[0023] Optionally, when the information type of the index information is a numeric type or a time type, the processing module is specifically used for:

[0024] Based on the value distribution of each index information, determine the value range containing each index information;

[0025] The value range is divided into a specified number of sub-ranges;

[0026] The data whose corresponding index information belongs to the same sub-interval are grouped together to obtain the multiple data fragments.

[0027] Optionally, the processing module is specifically used for:

[0028] For each of the multiple data shards, perform the following operations respectively:

[0029] Based on at least one piece of data involved in a data shard, the query condition is divided into multiple alternative sub-conditions, wherein each alternative sub-condition is used to indicate querying one piece of data in the data shard, or to indicate querying data other than the at least one piece of data;

[0030] Based on the instruction query, the union of each candidate sub-condition of a data in a data shard is used to obtain the sub-condition corresponding to the data shard.

[0031] Optionally, the processing module is specifically used for:

[0032] For each of the aforementioned sub-conditions, perform the following operations:

[0033] Replace the query condition in the target query statement with a sub-condition to generate a sub-query statement;

[0034] Based on a preset data query verification strategy, the sub-query statement is verified to obtain the verification result of the sub-query statement. The verification result is used to characterize whether the sub-data indicated by the sub-condition can be obtained from the corresponding target data unit based on the sub-query statement.

[0035] When the verification result characterizes the sub-data indicated by the sub-condition based on the sub-query statement, the sub-query statement corresponding to the sub-condition is obtained from the corresponding target data unit.

[0036] Optionally, the processing module is specifically used for:

[0037] For each of the multiple subquery statements, perform the following operations:

[0038] Based on a subquery statement indicating the target data unit to be queried, the subquery statement is mapped to a data view, wherein the data view is used to represent the data storage address of the sub-data indicated by the subquery statement;

[0039] Based on the aforementioned subquery statement, a data transmission channel is established;

[0040] Based on the data storage address represented by the data view, the sub-data queried by the sub-query statement is obtained through the data transmission channel.

[0041] Thirdly, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0042] Fourthly, a computer device is provided, comprising:

[0043] Memory, used to store program instructions;

[0044] A processor is configured to invoke program instructions stored in the memory and execute the method described in the first aspect according to the obtained program instructions.

[0045] Fifthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer-executable instructions for causing a computer to perform the method as described in the first aspect.

[0046] In this embodiment, a single target query statement is used to query target data from at least one target data unit in each of at least one target data source. Based on the data storage characteristics of each target data unit, the query conditions included in the target query statement are divided into sub-conditions to obtain multiple sub-conditions, and sub-query statements corresponding to each of the multiple sub-conditions are generated. The target query statement is transformed into multiple sub-query statements, allowing concurrent acquisition of the sub-data indicated by each sub-query statement through its corresponding data transmission channel. By merging the multiple sub-data obtained, the target data indicated by the target query statement can be obtained. This avoids the problem of data congestion on the data transmission channel when the target data volume is large, due to the limited data transmission capacity of the data transmission channel, resulting in a long time for the device to acquire the complete target data and low data query efficiency. Furthermore, when querying target data from multiple data sources or multiple target data units, it is not necessary to loop through the query process multiple times. Different sub-query statements can concurrently acquire corresponding sub-data from different data sources or different target data units, simplifying the process of obtaining the complete target data and improving the data query efficiency. Attached Figure Description

[0047] Figure 1a A schematic diagram illustrating the principle of a data query method based on related technologies;

[0048] Figure 1bA schematic diagram of the principle of a data query method under related technologies. Figure 2 ;

[0049] Figure 1c A schematic diagram of the principle of a data query method under related technologies. Figure 2 ;

[0050] Figure 1d This is one application scenario of the data query method provided in the embodiments of this application;

[0051] Figure 2 A flowchart illustrating a data query method provided in an embodiment of this application;

[0052] Figure 3a A schematic diagram illustrating the principle of the data query method provided in this application embodiment;

[0053] Figure 3b A schematic diagram of the data query method provided in the embodiments of this application. Figure 2 ;

[0054] Figure 4a A schematic diagram three illustrating the principle of the data query method provided in this application embodiment;

[0055] Figure 4b A schematic diagram four illustrating the principle of a data query method provided in this application embodiment;

[0056] Figure 4c A schematic diagram five illustrating the principle of the data query method provided in this application embodiment;

[0057] Figure 5a A schematic diagram six illustrating the principle of a data query method provided in an embodiment of this application;

[0058] Figure 5b A schematic diagram of the data query method provided in the embodiments of this application. Figure 7 ;

[0059] Figure 6a A schematic diagram of the data query method provided in the embodiments of this application. Figure 8 ;

[0060] Figure 6b A schematic diagram nine illustrating the principle of a data query method provided in an embodiment of this application;

[0061] Figure 6c A schematic diagram ten illustrating the principle of a data query method provided in this application embodiment;

[0062] Figure 6d 11. A schematic diagram illustrating the principle of the data query method provided in this application embodiment;

[0063] Figure 6e A schematic diagram twelve illustrating the principle of a data query method provided in this application embodiment;

[0064] Figure 7 A schematic diagram of the structure of a data query device provided in the embodiments of this application;

[0065] Figure 8 A schematic diagram of the structure of the data query device provided in the embodiments of this application. Figure 2 . Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0067] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0068] (1) Data source:

[0069] A data source is a data storage system, including traditional relational databases such as Oracle and MySQL, as well as big data storage systems such as Hive and HBase.

[0070] (2) Java database connection:

[0071] Java Database Connectivity (JDBC) is an application programming interface in the Java language that specifies how client applications access databases, providing methods for querying and updating data in the database.

[0072] (3) Structured Query Language:

[0073] Structured Query Language (SQL) is a database query and programming language used to access, query, update, and manage relational databases. SQL is a high-level, non-procedural programming language that allows working on high-level data structures. SQL does not require specifying how data is stored or understanding the specific data storage method; it is a database system with a completely different underlying structure and can be nested.

[0074] (4) Data warehouse:

[0075] A data warehouse (Hive) is built on the Hadoop framework, a big data software system operating system, and is used to implement processes such as data extraction, transformation, and loading. A data warehouse is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. Hive can map structured data files to a database table and provide query functionality using the Structured Query Language (SQL).

[0076] (5) Cost-based query optimizer:

[0077] Cost-Based Optimization (CBO) calculates the cost of each execution plan by applying a uniform metric to all execution plans for a structured query. For example, it calculates the cost of each execution plan based on statistics of the data objects within the plan and a cost algorithm. Statistics represent attributes of the data objects that affect the efficiency of the execution plan, such as the number of rows or unique key values. The cost algorithm processes the statistics into standardized numerical values, which are then used as the basis for selecting the execution plan.

[0078] (6) Data View:

[0079] A visualized virtual data structure for the result set obtained from a query statement based on a structured query language. The data view in tabular format contains rows and columns, similar to a real data table.

[0080] The embodiments of this application relate to cloud computing technology, which is designed based on cloud storage technology and can be applied to fields such as artificial intelligence, intelligent transportation, or mapping.

[0081] Cloud computing is a computing model that distributes computing tasks across a resource pool composed of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network providing these resources is called the "cloud." From the user's perspective, the resources in the "cloud" are infinitely scalable, readily available, on-demand, expandable, and pay-as-you-go.

[0082] As a provider of fundamental cloud computing capabilities, a cloud resource pool (referred to as a cloud platform, generally called an Infrastructure as a Service (IaaS) platform) is established. Various types of virtual resources are deployed in the resource pool for external customers to choose from. The cloud resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, and network devices.

[0083] Based on logical function, a Platform as a Service (PaaS) layer can be deployed on top of the IaaS layer, followed by a Software as a Service (SaaS) layer. Alternatively, SaaS can be deployed directly on top of IaaS. PaaS is the platform for running software, such as databases and web containers. SaaS comprises various types of business software, such as web portals and bulk SMS senders. Generally, SaaS and PaaS are upper layers compared to IaaS.

[0084] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology and distributed storage file systems to bring together a large number of storage devices of various types in the network (storage devices are also called storage nodes) to work together through application software or application interfaces to provide data storage and business access functions to the outside world.

[0085] Currently, the storage method of storage systems is as follows: Logical volumes are created. During the creation of a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID entity, ID). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0086] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0087] It should be noted that in the embodiments of this application, if the content involves user data, when the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0088] The application areas of the data query method provided in the embodiments of this application will be briefly introduced below.

[0089] With the continuous development of technology, the amount of data that devices query from the same data source is increasing daily, and the query rules for querying data from different data sources are also becoming increasingly sophisticated. When querying data from a data source, the device can, after receiving the query statement, retrieve the corresponding data from the appropriate data source based on the query statement.

[0090] Under related technologies, when a device queries data from a data source, the amount of data that can be transmitted through the data transmission channel is limited. Therefore, when the amount of data to be queried is large, the data transmission channel is prone to congestion during the process of the data source transmitting the queried data to the device. This results in a longer time for the device to obtain the complete queried data, leading to low data query efficiency.

[0091] For related technologies, please refer to Figure 1a Hive, a new type of data warehouse based on the Hadoop big data software system operating framework, is used to query data between traditional relational databases. The relational database can send query statements to the data warehouse, and after receiving the query statements, the data warehouse can obtain the data to be queried as indicated by the query statements. The data warehouse can also send the data to be queried to the relational database, so that the relational database can obtain the data to be queried through the query statements.

[0092] When the amount of data being queried is large, assuming the data is stored in a table and each record contains a primary key, device name, device type, and device uptime, please refer to... Figure 1b Based on the primary key column, the queried data can be divided into multiple groups. For example, if the queried data contains five data records with primary key columns of 1, 2, 3, 4 and 5, the data with primary key column 1 can be divided into the first sub-data group, the data with primary key columns 2 and 3 can be divided into the second sub-data group, and the data with primary key columns 4 and 5 can be divided into the third sub-data group.

[0093] Therefore, relational databases can sequentially query each set of data from the data warehouse. For example, please refer to... Figure 1c The relational database first queries the first sub-data group through the data transmission channel. After obtaining the first sub-data group, the relational database queries the second sub-data group through the same data transmission channel. After obtaining the second sub-data group, the relational database queries the third sub-data group through the same data transmission channel. By using the data transmission channel multiple times in a loop to transmit each group of data, the goal of avoiding congestion on the data transmission channel is achieved.

[0094] In related technologies, methods for solving the data transmission channel congestion problem are actually similar to methods for querying data from multiple data sources. Because the query rules differ across data sources, when a device queries data from multiple sources, it needs to perform multiple query processes based on the respective query rules of each data source. During the process of repeatedly using the data transmission channel to transmit each set of data, and during the repeated query processes, the process of obtaining the complete queried data is quite complex, resulting in low data query efficiency.

[0095] It is evident that, under the relevant technologies, data query efficiency is low, whether querying data from the same data source or from multiple data sources.

[0096] To address the issue of low data query efficiency, this application proposes a data query method. In this method, after obtaining the target query statement, the target query statement is parsed to determine at least one target data source indicated by the target query statement, and at least one target data unit within each target data source that meets the query conditions contained in the target query statement. Based on the data storage characteristics of the obtained at least one target data unit, the query conditions are divided into sub-conditions to obtain multiple sub-conditions, and sub-query statements corresponding to each of the multiple sub-conditions are generated. Through the data transmission channels corresponding to each of the multiple sub-query statements, the sub-data indicated by each of the multiple sub-query statements is obtained from the at least one target data unit. The obtained multiple sub-data are merged to generate the target data indicated by the target query statement.

[0097] In this embodiment, a single target query statement is used to query target data from at least one target data unit in each of at least one target data source. Based on the data storage characteristics of each target data unit, the query conditions included in the target query statement are divided into sub-conditions to obtain multiple sub-conditions, and sub-query statements corresponding to each of the multiple sub-conditions are generated. The target query statement is transformed into multiple sub-query statements, allowing concurrent acquisition of the sub-data indicated by each sub-query statement through its corresponding data transmission channel. By merging the multiple sub-data obtained, the target data indicated by the target query statement can be obtained. This avoids the problem of data congestion on the data transmission channel when the target data volume is large, due to the limited data transmission capacity of the data transmission channel, resulting in a long time for the device to acquire the complete target data and low data query efficiency. Furthermore, when querying target data from multiple data sources or multiple target data units, it is not necessary to loop through the query process multiple times. Different sub-query statements can concurrently acquire corresponding sub-data from different data sources or different target data units, simplifying the process of obtaining the complete target data and improving the data query efficiency.

[0098] The application scenarios of the data query method provided in this application are described below.

[0099] Please refer to Figure 1d This is a schematic diagram illustrating an application scenario of the data query method provided in this application. The application scenario includes a client 101 and a server 102. The client 101 and the server 102 can communicate with each other. The communication method can be wired, such as through a network cable or serial cable; or wireless, such as through Bluetooth or Wi-Fi. No specific limitation is imposed.

[0100] Client 101 generally refers to a device that can provide a target query statement to server 102, such as a terminal device, a third-party application accessible to the terminal device, or a webpage accessible to the terminal device. Terminal devices include, but are not limited to, mobile phones, computers, intelligent transportation equipment, and smart appliances. Server 102 generally refers to a device that can query target data based on the target query statement, such as a terminal device or a server. Servers include, but are not limited to, cloud servers, local servers, or associated third-party servers. Both client 101 and server 102 can use cloud computing to reduce the consumption of local computing resources; similarly, they can also use cloud storage to reduce the consumption of local storage resources.

[0101] As one embodiment, the client 101 and the server 102 can be the same device, and there is no specific limitation. In this embodiment, the client 101 and the server 102 are described as different devices.

[0102] The following is based on Figure 1d Using server 102 as the main component, this paper provides a detailed description of the data query method provided in the embodiments of this application. Please refer to [link / reference]. Figure 2 This is a flowchart illustrating a data query method provided in an embodiment of this application.

[0103] S201, obtain the target query statement, parse the target query statement, determine at least one target data source indicated by the target query statement, and at least one target data unit in each target data source that meets the query conditions contained in the target query statement.

[0104] The server can obtain the target query statement in several ways. It can generate the target query statement in response to the input operation triggered by the target object on the client side, and then send the target query statement to the server. The server can then receive the target query statement sent from the client. Alternatively, the server can generate the target query statement based on the query requirements and pre-stored syntax rules. There are no specific restrictions on the specific method.

[0105] A target query statement can be used to indicate that target data is retrieved from a single target data source, or it can be used to indicate that target data is retrieved from multiple target data sources. For example, please refer to... Figure 3a The target query statement is an SQL statement, `select col1,col2 from userA.mysql.db1.table1 where col2>0`, which indicates that the target data should be retrieved from a target data source `mysql`. For another example, please refer to... Figure 3b The target query statement `select * from (select col1,col2 from userA.mysql.db1.table1 where col2>0) a join (select col3,col4 from userA.oracle.db2.table2 where col4<0) b on a.col1=b.col3` can instruct the query to retrieve target data from two target data sources, MySQL and Oracle.

[0106] After receiving the target query statement, the server can parse it to determine at least one target data source indicated by the query, and at least one target data unit within each target data source that matches the query conditions contained in the target query statement. Each target data source can contain one or more target data units that match the query conditions, with no specific limitation. The query conditions contained in the target query statement can be used to filter target data within the data stored in the corresponding target data source.

[0107] For example, the target query statement `select col1,col2 from userA.mysql.db1.table1 where col2>0` contains the query condition that can be used to query the target data in the `userA.mysql.db1.table1` data table of the MySQL data source, specifically the `col1` and `col2` columns, where the value of `col2` is greater than 0.

[0108] The server parses the target query statement, which can be based on the syntax structure of the target query statement. There are various methods for parsing the target query statement. For example, after the server obtains the target query statement "select col1, col2 from userA.mysql.db1.table1 where col2>0", it can parse the syntax structure of the target query statement to be "select XXX from XXX where XXX". The server can then match userA.mysql.db1.table1 with the various pre-stored alternative data sources to determine that the target data source indicated by the target query statement is mysql. After determining the target data source, the server can determine the target data unit that meets the query conditions based on the query conditions contained in the target query statement, the col1 and col2 columns, and the fact that the value of the col2 column is greater than 0, and determine that the target data unit that meets the query conditions is the target object userA. In the target data source mysql, a database named db1 is created containing table1.

[0109] For example, the server can parse the syntax of the target query statement to obtain an abstract syntax tree (Abstract Syntax Tree). After obtaining the Abstract Syntax Tree, the server can extract the target data units that match the query conditions contained in the target query statement. For instance, after obtaining the target query statement "select col1, col2 from userA.mysql.db1.table1 where col2>0", the server can determine the corresponding Abstract Syntax Tree, where userA is the root node, mysql is a first child node of the root node, db1 is a second child node of the first child node, table1 is a third child node of the second child node, and col1 and col2 are two fourth child nodes of the third child node. Thus, the server can extract from the Abstract Syntax Tree that the target data source indicated by the target query statement is the first child node mysql, the query conditions contained in the target query statement are the fourth child node col1 and col2, and the value of col2 is greater than 0. Based on the query conditions, the server can extract the target data unit as the target object userA corresponding to the root node, and the database corresponding to the second child node named db1 in the target data source mysql, which contains the third child node table1.

[0110] S202, based on the data storage characteristics of at least one target data unit, the query conditions are divided into sub-conditions to obtain multiple sub-conditions, and sub-query statements corresponding to each of the multiple sub-conditions are generated respectively.

[0111] After obtaining the target query statement, at least one target data source, and at least one target data unit within each target data source, the server can, based on the data storage characteristics of the at least one target data unit, divide the query conditions into sub-conditions, obtaining multiple sub-conditions, and generating sub-query statements corresponding to each of the multiple sub-conditions. Thus, the server divides the target query statement into multiple sub-query statements, and the union of the sub-conditions contained in each of the multiple sub-query statements constitutes the original query conditions.

[0112] Data storage characteristics are used to characterize the statistical properties of the data stored in the target data unit. For example, when the target data unit is in tabular form, the data storage characteristics may include the total number of rows in the target data unit, the total number of columns in the target data unit, the data type of each column in the target data unit, the maximum value contained in each column in the target data unit, the minimum value contained in each column in the target data unit, the total number of null values ​​contained in each column in the target data unit, and the total number of non-null values ​​contained in each column in the target data unit.

[0113] The server can divide the query conditions into sub-conditions based on the data storage characteristics corresponding to the target data unit. Taking the form of a table as an example, the server can divide the query conditions based on the maximum and minimum values in a column; or divide the query conditions based on the null and non-null values included in a column, etc. There is no specific limitation.

[0114] Taking the target data unit in a target data source mysql as an example, for example, please refer to Figure 4a , the target query statement is "select col1,col2 from userA.mysql.db1.table1 where col2>0". The data storage characteristics of the target data unit indicate that the total number of rows of the target data unit is 5, the total number of columns is 4, the data type of each column is a numerical type, the maximum values included in each column are 5, 66, 21, and 7 in sequence, the minimum values included in each column are 1, -15, 3, and 2 in sequence, the total number of null values included in each column is 0, and the total number of non-null values included in each column is 5.

[0115] Based on the data storage characteristics of the target data unit, the query condition "where col2>0" can be divided into sub-conditions to obtain three sub-conditions: "where 0<col2<10", "where 10<col2<20", and "where 20<col2<30", and the sub-query statements corresponding to each of the three sub-conditions are generated in sequence: "select col1,col2 from userA.mysql.db1.table1 where 0<col2<10", "select col1,col2 from userA.mysql.db1.table1 where 10<col2<20", and "select col1,col2 from userA.mysql.db1.table1 where 20<col2<30".

[0116] As an embodiment, before dividing the query conditions into sub-conditions based on the data storage characteristics, the server can pre-employ the cost-based query optimization technique CBO to collect the data storage characteristics of each data unit included in each data source. For example, when the server uses the cost-based query optimization method to make an execution plan, it needs to make an execution plan according to the data storage characteristics of each data unit. The server can store the data storage characteristics of each data unit, and when querying data through the target query statement, it can read the corresponding stored data storage characteristics.

[0117] Based on the data storage characteristics of at least one target data unit, the server divides the query condition into sub-conditions respectively, obtaining multiple sub-conditions. When there is one target data unit among the at least one target data unit, the server can divide the query condition into multiple sub-conditions based on the data storage characteristics of this one target data unit. For example, for the target query statement "select col1, col2 from userA.mysql.db1.table1 where col2>0", a target data unit "userA.mysql.db1.table1" that meets the query condition "where col2>0" is obtained. Then, based on the data storage characteristics of this one target data unit, the query condition can be divided into the first sub-condition "where 0<col2<20" and the second sub-condition "where col2>20".

[0118] When there are multiple target data units among the at least one target data unit, the server can respectively divide the query condition into multiple sub-conditions based on the data storage characteristics of each target data unit among the multiple target data units. For example, for the target query statement "select * from (select col1, col2 from userA.mysql.db1.table1 where col2>0) a join (select col3, col4 from userA.oracle.db2.table2 where col4<0) b on a.col1 = b.col3", the query conditions include "where col2>0" and "where col4<0". The server obtains a first target data unit "userA.mysql.db1.table1" that meets the query condition "where col2>0", and a second target data unit "userA.oracle.db2.table2" that meets the query condition "where col4<0". Therefore, for the first target data unit, the query condition "where col2>0" can be divided into the first sub-condition "where 0<col2<10" and the second sub-condition "where col2>10". For the second target data unit, the query condition "where col4<0" can be divided into the third sub-condition "where -20<col4<0" and the fourth sub-condition "where col4<-20".

[0119] As one embodiment, the number of sub-conditions can be determined based on the number of data transmission channels that can be occupied. For example, when there are fewer data transmission tasks, there are more data transmission channels that can be occupied, so the query conditions can be divided into more sub-conditions. In this case, the amount of data in the sub-data indicated by each sub-condition is smaller, thereby improving the efficiency of obtaining the corresponding sub-data based on the sub-conditions and achieving the goal of improving the efficiency of data query.

[0120] The number of sub-conditions can also be determined based on the upper limit of the data transmission channel. For example, if the upper limit of the data transmission channel is 100, the query condition can be divided into multiple sub-conditions, each indicating a sub-data item with a data volume of 100. In this way, each data transmission channel can be used reasonably, avoiding the situation where the data transmission channel transmits a small amount of data and requires more data transmission channels to transmit data, thus improving the flexibility of data query.

[0121] As one example, if each piece of data stored in the target data unit has its own index information, and each index information is used to uniquely represent a piece of data or a group of data in the target data unit, assuming the target data unit is in tabular form, the index information can be data ID, data name, partition, etc. A data ID can represent a piece of data indicated by a data ID in a data table, and a partition can represent a group of data contained in a partition in a data warehouse. Then, the server can determine the index information contained in the target data unit based on the data storage characteristics, and then, based on the index information, divide the query conditions into sub-conditions to obtain multiple sub-conditions. This avoids duplicates among multiple sub-conditions and eliminates the need for filtering for overlapping relationships among multiple sub-conditions, thus improving the efficiency of data querying.

[0122] The following section uses a single target data unit as an example to illustrate the process of obtaining multiple sub-conditions. Other target data units are similar and will not be described in detail here.

[0123] The server can determine the index information for each of the multiple data items in the target data unit that meet the query conditions based on the data storage characteristics of the target data unit. Taking data stored in tabular form as an example, the data storage characteristics can record the type of each column in the target data unit, such as numeric, time, or character types. The server can determine which columns are character types based on the data storage characteristics and use the content stored in those columns as the index information for each data item. The data storage characteristics can also record whether each column in the target data unit contains a specified marker; the server can use the content stored in columns containing the specified marker as the index information for each data item. Furthermore, the data storage characteristics can record whether there are duplicate contents in each column of the target data unit; the server can use the content in columns without duplicate contents as the index information for each data item, and so on. The specific method for determining the index information is not specifically limited.

[0124] After obtaining the index information, the server can group the data based on the index information to generate multiple data shards, each containing at least one piece of data. There must be a correlation between the at least one piece of data involved in a data shard. For example, the index information of the at least one piece of data in a data shard may be the same or similar. Alternatively, if there is a sequential order to the index information, the index information of the at least one piece of data in a data shard may be ordered adjacently, etc. Specific restrictions are not imposed.

[0125] As one example, when the information type of the index information is a numeric type or a time type, the server can generate multiple data shards based on the value size of the index information. The server can determine the value range containing each index information based on the value distribution of each index information. For example, if the value distribution of the index information is between 0 and 50, with a maximum value of 50 and a minimum value of 0, then the value range containing each index information can be determined as [0, 50].

[0126] After determining the value range, the server can divide the value range into a specified number of sub-ranges. The specified number can be related to the data transmission channel, as detailed in the previous section; it can also be set in advance based on empirical values, etc., and there are no specific restrictions. For example, after obtaining the value range [0, 50], the server can divide the value range into three sub-ranges, namely [0, 10], [10, 40], and [40, 50].

[0127] As one embodiment, the threshold value of each sub-interval can be determined based on the value distribution of each index information. For example, [0, 10] contains 10 data, [10, 20] contains 2 data, [20, 30] contains 3 data, [30, 40] contains 5 data, and [40, 50] contains 10 data. Then, based on the fact that each sub-interval contains 10 data, three sub-intervals are divided, namely [0, 10], [10, 40], and [40, 50].

[0128] After obtaining multiple sub-intervals, the server can group the data whose corresponding index information belongs to the same sub-interval into multiple data shards. Continuing with the previous example, after obtaining the three sub-intervals [0, 10], [10, 40], and [40, 50], the server can group the data contained in [0, 10] into one data shard, group the data contained in [10, 40] into another data shard, and group the data contained in [40, 50] into yet another data shard, obtaining a total of three data shards.

[0129] After obtaining multiple data shards, the server can further divide the query conditions into sub-conditions based on each shard, thus obtaining the corresponding sub-conditions for each data shard. The following example illustrates the sub-condition division process using a single data shard; other data shards are similar and will not be elaborated upon here.

[0130] Based on at least one piece of data involved in a data shard, the query condition is divided into multiple alternative sub-conditions. Each alternative sub-condition indicates whether to query one piece of data in a data shard or to query all data except for at least one piece of data. For example, if the query condition indicates that the col2 column in the target data unit has a value between 0 and 10, and the at least one piece of data involved in a data shard consists of three pieces of data with col2 column values ​​of 2, 3, and 8 respectively, then the query condition can be divided into four alternative sub-conditions: the first alternative sub-condition indicates whether to query data with a col2 column value of 2, the second alternative sub-condition indicates whether to query data with a col2 column value of 3, the third alternative sub-condition indicates whether to query data with a col2 column value of 8, and the fourth alternative sub-condition indicates whether to query data with col2 column values ​​of 1, 4-7, and 9-10.

[0131] After obtaining multiple alternative sub - conditions, the server can obtain the sub - condition corresponding to the data shard based on the union of the alternative sub - conditions that indicate querying each piece of data in a data shard. For example, after obtaining the first alternative sub - condition for indicating querying data where the value of column col2 is 2, the second alternative sub - condition for indicating querying data where the value of column col2 is 3, the third alternative sub - condition for indicating querying data where the value of column col2 is 8, and the fourth alternative sub - condition for indicating querying data where the value of column col2 is 1, 4 - 7, and 9 - 10, the server can take the union of the first alternative sub - condition, the second alternative sub - condition, and the third alternative sub - condition, that is, the condition for indicating querying data where the value of column col2 is 2 - 3 and 8 as the sub - condition corresponding to the data shard.

[0132] After obtaining multiple sub - conditions, the server can generate corresponding sub - query statements for the multiple sub - conditions respectively. Taking one sub - condition as an example, the server can replace the query condition in the target query statement with a sub - condition to generate a sub - query statement. For example, for the target query statement "select col1, col2 from userA.mysql.db1.table1 where col2>0", where the query condition is "where col2>0" and a sub - condition is "where 0<col2<10", then the server can replace the query condition in the target query statement with a sub - condition to obtain "select col1, col2 from userA.mysql.db1.table1 where 0<col2<10", which is the corresponding sub - query statement.

[0133] The server can also first select the partial query statement corresponding to the target data source of the one sub-condition in the target query statement, replace the query condition in the partial query statement with the one sub-condition, and generate a sub-query statement. For example, for the target query statement "select * from (select col1, col2 from userA.mysql.db1.table1 where col2>0) a join (select col3, col4 from userA.oracle.db2.table2 where col4<0) b on a.col1 = b.col3", one sub-condition is "where 0<col2<10", and the target data source corresponding to the one sub-condition is mysql. Then the server selects the partial query statement corresponding to the one sub-condition in the target query statement as "select col1, col2 from userA.mysql.db1.table1 where col2>0". Then the server can replace the query condition in the partial query statement with the one sub-condition to obtain "select col1, col2 from userA.mysql.db1.table1 where 0<col2<10", which is the corresponding sub-query statement.

[0134] As an embodiment, after generating the sub-query statement corresponding to the sub-condition, the server can use a preset data query verification policy to verify the sub-query statement. After the statement verification, the server can obtain the verification result of the sub-query statement. The verification result is used to indicate whether the sub-data indicated by the sub-condition included in the sub-query statement can be obtained from the corresponding target data unit based on the sub-query statement. For example, the verification result can indicate whether there is a syntax error in the sub-query statement, and the verification result can also indicate whether the sub-query statement can be applied to query the data in the target data unit. The syntax error in the sub-query statement can be a spelling error of words, an incorrect association of phrases, etc., which is not specifically limited; being unable to be applied to query the data in the target data unit can be that the sub-condition included in the sub-query statement does not correspond to the name of the column included in the target data unit, or the sub-condition does not correspond to the type of the column included in the target data unit, etc., which is not specifically limited.

[0135] If the server determines that the validation result representation based on the subquery statement can retrieve the sub-data indicated by the sub-conditions contained in the subquery statement from the corresponding target data unit, it means that the subquery statement validation has passed, and the subquery statement corresponding to the sub-conditions is obtained. If the server determines that the validation result representation based on the subquery statement cannot retrieve the sub-data indicated by the sub-conditions contained in the subquery statement from the corresponding target data unit, it means that the subquery statement validation has failed. In this case, the server can revise the subquery statement based on the reason for the validation failure, obtaining a revised subquery statement. If the revised subquery statement passes validation, it will be used as the subquery statement corresponding to the sub-conditions.

[0136] As one implementation, correcting a subquery based on the statement validation failure reason can involve the server checking the spelling and punctuation of each string according to a data query validation strategy when the validation failure reason indicates a syntax error. If errors are found, the server can correct the incorrect string based on the string most similar to the incorrect string, or correct the incorrect punctuation based on the correct punctuation.

[0137] Based on the reasons for statement validation failure, the subquery statement can be corrected. Alternatively, when the statement validation failure indicates a logical error, the server, based on the data query validation strategy, checks whether the target object, data source, data table, column, etc., represented by each string exists in the server. If an error is found, the erroneous string can be corrected based on the string most similar to the erroneous string, etc., without any specific restrictions.

[0138] As one embodiment, after obtaining the target query statement, the server can first determine whether the target query statement can be divided into sub-conditions. If it is determined that the target query statement can be divided into sub-conditions, then the methods S202 to S204 are executed. If it is determined that the target query statement cannot be divided into sub-conditions, then the methods S202 to S204 can be discontinued.

[0139] There are several ways to determine whether a target query statement can be divided into sub-conditions. For example, when the data storage characteristics of the target data unit indicate that the target data source contains index information that uniquely represents each data, such as indexes or partitions, it can be determined that the target query statement can be divided into sub-conditions.

[0140] For another example, when it is determined that the target query statement only contains a simple select query and does not contain query operations such as join, union, intersect, and minus, it is determined that the target query statement can be divided into sub - conditions. For another example, when it is determined that the target query statement does not contain an order query operation, or the order query operation does not involve operations such as limit and offset, it is determined that the target query statement can be divided into sub - conditions. For another example, when it is determined that the target query statement does not contain an over window, it is determined that the target query statement can be divided into sub - conditions. For another example, when it is determined that the target query statement does not contain an aggregate function, it is determined that the target query statement can be divided into sub - conditions. For another example, when it is determined that the where operation in the target query statement is composed of and, or involves logical greater than or equal to, less than or equal to, greater than, less than, equal to, and between and, it is determined that the target query statement can be divided into sub - conditions.

[0141] S203, through the data transmission channels corresponding to multiple sub - query statements, respectively obtain the sub - data indicated by each of the multiple sub - query statements from at least one target data unit.

[0142] After obtaining multiple sub - query statements, the server can establish corresponding data transmission channels for the multiple sub - query statements. For example, establish a JDBC connection. The sub - data indicated by each sub - query statement can be transmitted through the corresponding data transmission channel. Thus, the server can concurrently obtain the sub - data indicated by each of the multiple sub - query statements from at least one target data unit through the data transmission channels corresponding to the multiple sub - query statements, avoiding the process of multiple loop queries and the process of a single data query channel being occupied by loops. Multiple sub - data can be obtained simultaneously, improving the efficiency of data query.

[0143] For example, please refer to Figure 4b , the server obtains three sub - query statements, which are "select col1,col2 from userA.mysql.db1.table1 where 0 < col2 < 10", "select col1,col2 from userA.mysql.db1.table1 where 10 < col2 < 20", and "select col1,col2 from userA.mysql.db1.table1 where 20 < col2 < 30" in sequence. The server can query the corresponding sub - data from the data warehouse to the relational database through the three sub - query statements.

[0144] The server establishes a data transmission channel for each sub-query statement, namely the first data transmission channel, the second data transmission channel, and the third data transmission channel in sequence.

[0145] The server obtains the sub-data corresponding to the sub-query statement from the data warehouse to the relational database through the data transmission channels respectively corresponding to the three sub-query statements. Through the sub-query statement "select col1,col2 from userA.mysql.db1.table1 where 0<col2<10", the corresponding first data transmission channel, the server obtains the sub-data with IDs 3, 4, and 5 from the data warehouse to the relational database; through the sub-query statement "select col1,col2 from userA.mysql.db1.table1 where 10<col2<20", the corresponding second data transmission channel, the server obtains the sub-data with ID 1 from the data warehouse to the relational database; through the sub-query statement "select col1,col2 from userA.mysql.db1.table1 where 20<col2<30", the corresponding first data transmission channel, the server obtains the sub-data with ID 2 from the data warehouse to the relational database.

[0146] As an embodiment, the process of obtaining sub-data through a sub-query statement can be implemented through a data view, and the data view can visually display the sub-data indicated by the sub-query statement. The server can map a sub-query statement to a data view based on the target data unit indicated by the sub-query statement, where a data view is used to represent the data storage address of the sub-data indicated by a sub-query statement. After establishing a data transmission channel based on a sub-query statement, the server can obtain the sub-data indicated by a sub-query statement through the data transmission channel based on the data storage address represented by a data view.

[0147] S204, perform a merging process on the obtained multiple sub-data to generate the target data indicated by the target query statement.

[0148] After obtaining multiple sub-data, the server can perform a merging process on the obtained multiple sub-data to generate the target data indicated by the target query statement. When the server directly sends multiple sub-query statements, such as sub-query statement A, sub-query statement B,..., sub-query statement N, to the target data source and obtains the corresponding sub-data, such as sub-data A, sub-data B,..., sub-data N, through the corresponding data transmission channels at the same time, the server can merge the obtained multiple sub-data into the target data after obtaining the multiple sub-data, such as merging sub-data A, sub-data B,..., sub-data N into the target data.

[0149] Please refer to Figure 4c The server can use its computing engine to push down multiple subqueries, such as subquery A, subquery B, ..., subquery N, to their respective target data sources. The server can map each subquery to a data view, such as subquery A to data view A, subquery B to data view B, ..., subquery N to data view N. The data views corresponding to these subqueries can form a query instruction equivalent to the target query. The computing engine can use this equivalent query instruction to query the target data source for the target data, and simultaneously obtain the corresponding sub-data, such as sub-data A, sub-data B, ..., sub-data N, through the corresponding data transmission channels. The server can then merge these multiple sub-data into the target data.

[0150] As one embodiment, the data query method provided in this application can be executed by default by setting the corresponding parameter, such as the supersql.subquery.concurrent.enable parameter, to a specified value, such as true. If the corresponding parameter is not set to a specified value, the server will not execute the data query method by default.

[0151] The following is an example of the data query method provided in the embodiments of this application.

[0152] Please refer to Figure 5a The server includes a verification module, a sub-condition partitioning module, a concurrent transmission module, and a merging module. The verification module verifies whether the obtained target query statement can be queried using the data query method provided in this application. The sub-condition partitioning module divides the query conditions contained in the target query statement into multiple sub-conditions and generates sub-query statements corresponding to each sub-condition. The concurrent transmission module queries the corresponding sub-data based on the sub-query statements corresponding to each sub-condition. The merging module merges the obtained sub-data to obtain the target data.

[0153] Please refer to Figure 5b The sub-condition partitioning module includes a main partitioning sub-module and a secondary partitioning sub-module. The secondary partitioning sub-module can parse the target query statement to determine at least one target data source indicated by the target query statement, and at least one target data unit in each target data source that meets the query conditions. The secondary partitioning sub-module can also obtain the data storage characteristics of the corresponding target data unit, such as the table size, column type, column maximum value, column minimum value, number of null values, number of distinct values, and other basic column information; and also, for example, the index information of each piece of data stored in the target data unit.

[0154] The secondary partitioning module can send the obtained information to the primary partitioning module. Based on the obtained information, the primary partitioning module can further divide the query conditions into sub-conditions, obtaining multiple sub-conditions. The primary partitioning module can then send these multiple sub-conditions to the secondary partitioning module, which will then generate corresponding sub-query statements based on these sub-conditions.

[0155] The following is a detailed example of the data query method provided in the embodiments of this application. Please refer to [link / reference]. Figure 6a .

[0156] S601, in response to the input operation of the target object, the client obtains the target query statement "select * from (select col1, col2 from userA.mysql.db1.table1 where col2>0) a join (select col3, col4 from userA.oracle.db2.table2 where col4<0) b on a.col1=b.col3". The client sends the target query statement to the server, and the server receives the target query statement sent by the client. The target query statement is used to retrieve target data from the relational database.

[0157] S602, after obtaining the target query statement, the server parses the target query statement. Please refer to [reference needed]. Figure 6b The server determines that the target data sources include MySQL and Oracle. For the MySQL target data source, please refer to Table 1 for the target data units that meet the query conditions; for the Oracle target data source, please refer to Table 2 for the target data units that meet the query conditions.

[0158] Table 1

[0159] ID col1 col2 remark 1 5 15 5 2 12 21 3 3 66 6 7 4 -2 8 3 5 -15 3 2 …… …… …… ……

[0160] Table 2

[0161]

[0162]

[0163] S603, after obtaining two target data units, the server retrieves the data storage characteristics of each pre-stored target data unit. The target data units corresponding to Table 1 contain 5 rows and 4 columns, with each column being a numeric type. The first column has a label, with a maximum value of 5 and a minimum value of 1; the second column has a maximum value of 66 and a minimum value of -15; the third column has a maximum value of 21 and a minimum value of 3; and the fourth column has a maximum value of 7 and a minimum value of 2. The target data units corresponding to Table 2 also contain 5 rows and 4 columns, with each column being a numeric type. The first column has a label, with a maximum value of 5 and a minimum value of 1; the second column has a maximum value of 62 and a minimum value of -9; the third column has a maximum value of 9 and a minimum value of -20; and the fourth column has a maximum value of 8 and a minimum value of 3.

[0164] S604, after obtaining the data storage characteristics of each of the two target data units, the server determines, based on the data storage characteristics, that the first column has a label. Then, the index information of each piece of data stored in the two target data units is the ID of each piece of data. Based on the IDs of each piece of data contained in each of the two target data units, the server groups the data in each target data unit that meets the query conditions, obtaining multiple corresponding data fragments.

[0165] Please refer to Figure 6c For each piece of data in the target data unit corresponding to Table 1 that meets the query conditions, the maximum value of the first column is 5 and the minimum value is 1. The server can determine that the ID values ​​are distributed between 1 and 5, thus obtaining the value range [1, 5]. The server divides the value range into two sub-ranges, namely [1, 2] and [3, 5]. Therefore, the server groups the two pieces of data with ID 1 and ID 2 together to obtain one data shard; and groups the three pieces of data with ID 3, ID 4, and ID 5 together to obtain another data shard. Similarly, for each piece of data in the target data unit corresponding to Table 2 that meets the query conditions, the server groups the two pieces of data with ID 1 and ID 2 together to obtain one data shard; and groups the three pieces of data with ID 3, ID 4, and ID 5 together to obtain another data shard.

[0166] S605: After obtaining multiple data shards, the server can divide the query conditions into multiple alternative sub-conditions based on at least one piece of data involved in each data shard. For example, using two data entries with ID 1 and ID 2, please refer to [reference needed]. Figure 6d, the query condition is "where col2>0", and the values of col2 for the two pieces of data with ID 1 and ID 2 are 15 and 21 respectively. Then, the multiple alternative sub - conditions divided by the server can include "where 0<col2<10", "where 10<col2<20", "where 20<col2<30", and "where col2>30". The alternative sub - condition "where 10<col2<20" is used to indicate querying one piece of data with ID 1, the alternative sub - condition "where 20<col2<30" is used to indicate querying one piece of data with ID 2, and the alternative sub - conditions "where 0<col2<10" and "where col2>30" are used to indicate querying other data except the two pieces of data with ID 1 and ID 2.

[0167] The server can determine the sub - condition corresponding to this data shard based on the union of the two alternative sub - conditions "where 10<col2<20" and "where 20<col2<30", that is, "where 10<col2≤30". Similarly, obtain the sub - conditions corresponding to each of the other data shards.

[0168] S606. After obtaining multiple sub - conditions, the server successively processes each sub - condition. Based on the partial query statement in the target query statement for the corresponding target data source, it replaces the included query condition with the sub - condition to generate the corresponding sub - query statement. For example, please refer to Figure 6e , for the sub - condition "where10<col2≤30", the query statement in the target query statement for the target data source Mysql is "select col1,col2 from userA.mysql.db1.table1 where col2>0". The server replaces the query condition "where col2>0" in "select col1,col2 from userA.mysql.db1.table1 where col2>0" with the sub - condition, and obtains the sub - query statement "select col1,col2 fromuserA.mysql.db1.table1 where10<col2≤30". Similarly, obtain the other sub - query statements.

[0169] S607, after obtaining the four subquery statements, the server retrieves the sub-data indicated by each subquery statement from the corresponding target data unit in the data warehouse through the data transmission channel corresponding to each of the four subquery statements. The server then sends the four sub-data from the data warehouse to the relational database through their respective data transmission channels. The server merges the four sub-data sent to the relational database into target data, so that the relational database obtains the target data indicated by the target query statement.

[0170] Based on the same inventive concept, embodiments of this application provide a data query device capable of realizing the functions corresponding to the aforementioned data query method. Please refer to... Figure 7 The device includes an acquisition module 701 and a processing module 702, wherein:

[0171] Acquisition module 701: used to acquire the target query statement, parse the target query statement, determine at least one target data source indicated by the target query statement, and at least one target data unit in each target data source that meets the query conditions contained in the target query statement;

[0172] Processing module 702: is used to divide the query conditions into sub-conditions based on the data storage characteristics of at least one target data unit, obtain multiple sub-conditions, and generate sub-query statements corresponding to each of the multiple sub-conditions respectively;

[0173] The processing module 702 is also used to: obtain the sub-data indicated by each of the multiple sub-query statements from at least one target data unit through the data transmission channels corresponding to each of the multiple sub-query statements;

[0174] The processing module 702 is also used to: merge the obtained multiple sub-data and generate a target query statement to indicate the target data to be queried.

[0175] In one possible embodiment, each piece of data stored in the target data unit has its own index information, and each index information is used to uniquely represent a piece of data or a group of data in the target data unit.

[0176] Processing module 702 is specifically used for:

[0177] For at least one target data unit, perform the following operations respectively:

[0178] Based on the data storage characteristics of a target data unit, determine the index information of each of the multiple data that meet the query conditions for a target data unit;

[0179] Based on the index information, the multiple data are grouped to generate multiple data shards, wherein each data shard contains at least one piece of data;

[0180] Based on multiple data shards, the query conditions are divided into sub-conditions to obtain the sub-conditions corresponding to each of the multiple data shards.

[0181] In one possible embodiment, when the information type of the index information is a numeric type or a time type, the processing module 702 is specifically used for:

[0182] Based on the value distribution of each index information, determine the value range containing each index information;

[0183] Divide the range of values ​​into a specified number of sub-ranges;

[0184] Data that belong to the same sub-interval in the corresponding index information are grouped together to obtain multiple data shards.

[0185] In one possible embodiment, the processing module 702 is specifically used for:

[0186] For each data shard, perform the following operations:

[0187] Based on at least one piece of data involved in a data shard, the query conditions are divided into multiple alternative sub-conditions, wherein each alternative sub-condition is used to indicate querying one piece of data in a data shard, or to indicate querying data other than at least one piece of data;

[0188] Based on the indication, query the union of all alternative sub-conditions for a data in a data shard to obtain the sub-conditions corresponding to the data shard.

[0189] In one possible embodiment, the processing module 702 is specifically used for:

[0190] For each of the multiple sub-conditions, perform the following operations:

[0191] Replace the query condition in the target query statement with a subcondition to generate a subquery statement;

[0192] Based on a preset data query verification strategy, the subquery statement is verified to obtain the verification result of the subquery statement. The verification result is used to characterize whether a sub-condition indicating the query can be obtained from the corresponding target data unit based on the subquery statement.

[0193] The verification result representation is based on the subquery statement. When a sub-condition indicating the sub-data can be obtained from the corresponding target data unit, the subquery statement corresponding to the sub-condition is obtained.

[0194] In one possible embodiment, the processing module 702 is specifically used for:

[0195] For multiple subqueries, perform the following operations respectively:

[0196] Based on a subquery statement indicating the target data unit to be queried, a subquery statement is mapped to a data view, where a data view is used to represent the data storage address of the sub-data indicated by the subquery statement.

[0197] Establish a data transmission channel based on a subquery statement;

[0198] Based on the data storage address represented by a data view, a subquery statement is used to retrieve the sub-data to be queried through a data transmission channel.

[0199] Please refer to Figure 8 The aforementioned data query device can run on a computer device 800. The current and historical versions of the data storage program, as well as the application software corresponding to the data storage program, can be installed on the computer device 800, which includes a processor 880 and a memory 820. In some embodiments, the computer device 800 may include a display unit 840, which includes a display panel 841 for displaying a user-interactive interface, etc.

[0200] In one possible embodiment, the display panel 841 may be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0201] The processor 880 is used to read a computer program and then execute the methods defined by the computer program. For example, the processor 880 reads a data storage program or file, thereby running the data storage program on the computer device 800 and displaying the corresponding interface on the display unit 840. The processor 880 may include one or more general-purpose processors, and may also include one or more DSPs (Digital Signal Processors) for performing related operations to implement the technical solutions provided in the embodiments of this application.

[0202] The memory 820 generally includes main memory and secondary storage. Main memory can be random access memory (RAM), read-only memory (ROM), and cache, etc. Secondary storage can be a hard disk, optical disk, USB flash drive, floppy disk, or tape drive, etc. The memory 820 is used to store computer programs and other data. The computer programs include applications corresponding to each client, and other data may include data generated after the operating system or applications are run, including system data (e.g., operating system configuration parameters) and user data. In this embodiment, program instructions are stored in the memory 820, and the processor 880 executes the program instructions in the memory 820 to implement any of the methods described in the preceding figures.

[0203] The aforementioned display unit 840 is used to receive input digital information, character information, or contact touch operations / non-contact gestures, and to generate signal inputs related to user settings and function control of the computer device 800. Specifically, in this embodiment, the display unit 840 may include a display panel 841. The display panel 841, for example, is a touch screen, which can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or on the display panel 841), and drive corresponding connection devices according to a pre-set program.

[0204] In one possible embodiment, the display panel 841 may include two parts: a touch detection device and a touch controller. The touch detection device detects the player's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 880. It can also receive and execute commands from the processor 880.

[0205] The display panel 841 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the display unit 840, in some embodiments, the computer device 800 may also include an input unit 830. The input unit 830 may include an image input device 831 and other input devices 832, wherein the other input devices may include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0206] In addition to the above, the computer device 800 may also include a power supply 890 for powering other modules, an audio circuit 860, a near-field communication module 870, and an RF circuit 810. The computer device 800 may also include one or more sensors 850, such as an accelerometer, a light sensor, and a pressure sensor. The audio circuit 860 specifically includes a speaker 861 and a microphone 862, for example, the computer device 800 can use the microphone 862 to collect the user's voice and perform corresponding operations.

[0207] As one embodiment, the number of processors 880 can be one or more, and the processors 880 and the memory 820 can be coupled together or relatively independent.

[0208] As one example, Figure 8 The processor 880 in the middle can be used to implement, for example Figure 7 The functions of the acquisition module 701 and the processing module 702 in the process.

[0209] As one example, Figure 8 The processor 880 in the document can be used to implement the functions of the server or terminal devices discussed above.

[0210] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0211] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of software products, for example, through a computer program product. This computer program product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0212] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data query method, characterized in that, include: Obtain the target query statement, parse the target query statement, determine at least one target data source indicated by the target query statement, and at least one target data unit in each target data source that meets the query conditions contained in the target query statement; Each piece of data stored in the target data unit has its own index information, and each index information is used to uniquely represent a piece of data or a group of data in the target data unit. For at least one target data unit obtained, perform the following operations respectively: based on the data storage characteristics of a target data unit, determine the index information of each of the multiple data that meet the query conditions for the target data unit; Based on the index information, the multiple data are grouped to generate multiple data shards, each of which contains at least one piece of data; Based on the multiple data shards, the query conditions are divided into sub-conditions to obtain the sub-conditions corresponding to each of the multiple data shards. And generate the subquery statements corresponding to each of the multiple sub-conditions obtained; Through the data transmission channels corresponding to each of the multiple sub-query statements, the sub-data indicated by each of the multiple sub-query statements is obtained from the at least one target data unit; The obtained sub-data is merged to generate the target query statement indicating the target data to be queried.

2. The method according to claim 1, characterized in that, When the information type of the index information is a numerical type or a time type, the process of grouping the multiple data based on each index information to generate multiple data fragments includes: Based on the value distribution of each index information, determine the value range containing each index information; The value range is divided into a specified number of sub-ranges; The data whose corresponding index information belongs to the same sub-interval are grouped together to obtain the multiple data fragments.

3. The method according to claim 1, characterized in that, The step of dividing the query conditions into sub-conditions based on the multiple data shards to obtain the sub-conditions corresponding to each of the multiple data shards includes: For each of the multiple data shards, perform the following operations respectively: Based on at least one piece of data involved in a data shard, the query condition is divided into multiple alternative sub-conditions, wherein each alternative sub-condition is used to indicate querying one piece of data in the data shard, or to indicate querying data other than the at least one piece of data; Based on the instruction query, the union of each candidate sub-condition of a data in a data shard is used to obtain the sub-condition corresponding to the data shard.

4. The method according to any one of claims 1 to 3, characterized in that, Generate subquery statements corresponding to each of the multiple sub-conditions obtained, including: For each of the aforementioned sub-conditions, perform the following operations: Replace the query condition in the target query statement with a sub-condition to generate a sub-query statement; Based on a preset data query verification strategy, the sub-query statement is verified to obtain the verification result of the sub-query statement. The verification result is used to characterize whether the sub-data indicated by the sub-condition can be obtained from the corresponding target data unit based on the sub-query statement. When the verification result characterizes the sub-data indicated by the sub-condition based on the sub-query statement, the sub-query statement corresponding to the sub-condition is obtained from the corresponding target data unit.

5. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the sub-data indicated by each of the multiple sub-query statements from the at least one target data unit through the data transmission channels corresponding to each of the multiple sub-query statements includes: For each of the multiple subquery statements, perform the following operations: Based on a subquery statement indicating the target data unit to be queried, the subquery statement is mapped to a data view, wherein the data view is used to represent the data storage address of the sub-data indicated by the subquery statement; Based on the aforementioned subquery statement, a data transmission channel is established; Based on the data storage address represented by the data view, the sub-data queried by the sub-query statement is obtained through the data transmission channel.

6. A data query device, characterized in that, include: Acquisition module: used to acquire the target query statement, parse the target query statement, determine at least one target data source indicated by the target query statement, and at least one target data unit in each target data source that meets the query conditions contained in the target query statement; each data stored in the target data unit has its own index information, and each index information is used to uniquely represent a data or a group of data in the target data unit; Processing module: for each of the at least one target data unit obtained, perform the following operations: based on the data storage characteristics of a target data unit, determine the index information of each of the multiple data that meet the query conditions of the target data unit; Based on the index information, the multiple data are grouped to generate multiple data shards, each of which contains at least one piece of data; Based on the multiple data shards, the query conditions are divided into sub-conditions to obtain the sub-conditions corresponding to each of the multiple data shards; and sub-query statements corresponding to each of the obtained sub-conditions are generated respectively. The processing module is further configured to: obtain the sub-data indicated by each of the multiple sub-query statements from the at least one target data unit through the data transmission channels corresponding to each of the multiple sub-query statements; The processing module is also used to: merge the obtained multiple sub-data to generate the target query statement indicating the target data to be queried.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.

8. A computer device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method as described in any one of claims 1 to 5 according to the obtained program instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the method as described in any one of claims 1 to 5.

Images