A data query method and device
By obtaining the metadata and location attribute information of the target data source, generating a query plan and allocating operators to the matching node, the problem of low data query efficiency across cloud service platforms and cross-data areas is solved, and efficient joint data query is achieved.
Patent Information
- Application Number
- CN201910842291.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2019-09-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2039-09-06
AI Technical Summary
The prior art is inefficient in data query across cloud service platforms or across data areas, and requires a lot of work to migrate data to the same data area before querying.
By obtaining the metadata of the target data source, including location attribute information, an operator tree of the query plan is generated, and the operator is allocated to a cloud computing node that matches the location attributes, so as to realize joint data query across platforms and across data areas.
It realizes efficient data query across platforms and across data areas, meets the data joint query needs in most application scenarios in the cloud platform, and improves data query efficiency.
Smart Images

Figure CN112463814B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of cloud data query, and in particular to a data query method and device. Background Art
[0002] With the rapid development of Internet technology, big data and cloud computing have gradually transformed from the concept promotion stage to the actual development stage. Cloud data, as the data foundation supporting cloud computing, is a combination of technologies and platforms for data integration, data analysis, data integration, data distribution, and data early warning based on the cloud computing business model. As part of cloud data, cloud database refers to a database that is optimized or deployed in a cloud environment, which can realize functions such as pay-as-you-go, on-demand expansion, high availability, and storage integration. At present, there are cloud service platforms provided by various cloud service providers on the market, and various types of cloud databases are deployed on each platform. Each cloud database is also divided into multiple data regions.
[0003] Cloud data query is a common means of data analysis, typically using Structured Query Language (SQL) to query data. In related technologies, it is often possible to query data on databases in the same data area. However, when it comes to cross-cloud service platforms or cross-data area data queries, the data implementation method in related technologies is to migrate business data from different platforms or different data areas to the same data area before querying the data. It is found that the data query method in related technologies requires a large workload and has low data query efficiency.
[0004] Therefore, an efficient data query method across cloud service platforms or across data regions is urgently needed in the relevant technology. Summary of the invention
[0005] In order to overcome the problems existing in the related art, the present application provides a data query method and device.
[0006] A data query method, the method comprising:
[0007] Obtain data query instructions for the target data source;
[0008] Parsing the data query instruction to obtain metadata corresponding to the target data source, wherein the metadata includes location attribute information of the target data source;
[0009] Generate an operator tree of the query plan, and mark the location attributes of operators in the operator tree according to the location attribute information of the target data source;
[0010] Based on the operator tree, the operators are allocated to cloud computing nodes in corresponding areas matching the location attributes.
[0011] A data query device comprises a processor and a memory for storing processor executable instructions, wherein when the processor executes the instructions:
[0012] Obtain data query instructions for the target data source;
[0013] Parsing the data query instruction to obtain metadata corresponding to the target data source, wherein the metadata includes location attribute information of the target data source;
[0014] Generate an operator tree of the query plan, and mark the location attributes of operators in the operator tree according to the location attribute information of the target data source;
[0015] Based on the operator tree, the operators are allocated to cloud computing nodes in corresponding areas matching the location attributes.
[0016] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor, enables the processor to execute the above-mentioned data query method.
[0017] The data query method and device provided in the embodiments of the present application can realize cross-platform, cross-data region, and cross-cloud joint data query. In each embodiment of the present application, by using the location attribute information set in the metadata of the target data source, the data query module can determine which data sources are located across cloud platforms, across data regions, and across clouds, thereby setting the operators corresponding to these target data sources in computing nodes with external network access capabilities. Through the above-mentioned joint query of multiple data sources across cloud platforms and data regions on the cloud, the needs of data joint query in most application scenarios in the cloud platform can be met.
[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0020] Figure 1 The figure is a flow chart of a data query method according to an exemplary embodiment.
[0021] Figure 2 It is a schematic diagram of an application scenario according to an exemplary embodiment.
[0022] Figure 3It is a schematic diagram of an application scenario according to an exemplary embodiment.
[0023] Figure 4 It is a schematic diagram of an application scenario according to an exemplary embodiment.
[0024] Figure 5 The figure is a flowchart of a data query method according to an exemplary embodiment.
[0025] Figure 6 It is a schematic diagram of an application scenario according to an exemplary embodiment.
[0026] Figure 7 It is a schematic diagram of an application scenario according to an exemplary embodiment.
[0027] Figure 8 It is a block diagram of a data query device according to an exemplary embodiment. DETAILED DESCRIPTION
[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0029] The data query method described in this application is described in detail below with reference to the accompanying drawings. Figure 1 It is a method flow diagram of an embodiment of the data query method provided by the present application. Although the present application provides the method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiment of the present application. When the method is executed in the actual data query process or device, it can be executed in the order of the method shown in the embodiment or drawings or in parallel (for example, in the environment of parallel processor or multi-thread processing).
[0030] like Figure 1 As shown, the data query method provided in the embodiment of the present application may include:
[0031] S101: Obtain a data query instruction for a target data source.
[0032] In an embodiment of the present application, during the process of performing data query, a data query instruction for a target data source may be obtained. The target data source may include a data source involved in the data query. For example, the target data source may include a data table, a log, etc. storing user data information. The present application does not limit the storage form of the target data source. The target data source may be set in any data area on any cloud platform. In one embodiment, when the target data source involves multiple data sources, the multiple target data sources may be data sources of the same user in different cloud platforms or in different data areas on the same cloud platform. In one example, Enterprise A has purchased cloud computing resources on the first cloud platform, the second cloud platform, the third cloud platform, and the fourth cloud platform, and Enterprise A also has a private cloud platform. Therefore, Enterprise A has data sources stored in these five cloud platforms. In actual application scenarios, Enterprise A has a need to perform data queries across these five cloud platforms, such as searching for relevant data of senior members among registered users of Enterprise A at home and abroad, etc. As Figure 2 As shown in FIG. 1 , the enterprise A can use the data query module set up in the first cloud platform in the Hangzhou data area to jointly query the data sources of the five platforms. Figure 3 As shown, in another embodiment, the same cloud platform may also include multiple data areas. For example, the first cloud platform may include the Hangzhou data area, the Shanghai data area, the Beijing data area, the European data area, etc. Through the data query module set in the Hangzhou data area, a joint query can be performed on the data sources of multiple other data areas of the first cloud platform and the private cloud platform.
[0033] Based on this, in an embodiment of the present application, data query can be performed on a target data source across platforms or data regions based on the data query instruction. In one embodiment, the data instruction may include an SQL statement, for example, the data instruction may include query statements such as select from, inner join, group by, etc. The following is an example 1 of an SQL data query instruction:
[0034] SELECET a.dept_name as department_name,SUM(a.size)as total_size
[0035] FROM first_cloud_oss.t1_a_1 a
[0036] INNER JOIN first_cloud_oss.t2_a_1 b ON a.id=b.id
[0037] INNER JOIN first_cloud_rds_mysql.t3_a_1 c ON a.name=c.name
[0038] INNER JOIN first_cloud_london_rds_mysql.t4_a_r d ON a.order_id=d.order_id
[0039] INNER JOIN second_s3.t5_second e ON a.area_count=e.area_count
[0040] INNER JOIN third_mysql.t6_third f ON a.item_id=f.item_id
[0041] INNER JOIN fourth_mysql.t7_fourth g ON a.supp_id=g.supp_id
[0042] INNER JOIN onpremise_mysql.t8_onpremise h ON a.part_count=h.count
[0043] WHERE a.resource_number>f.resource_count
[0044] AND c.age>30
[0045] AND d.region = 'asia'
[0046] AND h.size>1000
[0047] GROUP BY department_name
[0048] ORDER BY total_size DESC
[0049] According to the above example 1, first_cloud_oss.t1_a_1, first_cloud_oss.t2_a_1, first_cloud_rds_mysql.t3_a_1, second_s3.t5_second, etc. in the example 1 are all names of target data sources (ie, data tables), and id, name, area_count, item_id, etc. are queried fields.
[0050] S103: Parse the data query instruction to obtain metadata corresponding to the target data source, where the metadata includes location attribute information of the target data source.
[0051] In an embodiment of the present application, the data query instruction also includes metadata corresponding to the target data source. The metadata of the target data source is used to define the source of the target data source. The metadata may include some information describing the source of the target data source. In an embodiment of the present application, the metadata may be described using a database schema definition language (Data Definition Language, DDL). By parsing the DDL, the metadata of the target data source may be obtained. In one example, the metadata (DDL statement) of the target data source first_cloud_oss.t1_a_1 in the above example 1 may include:
[0052] CREAT SCHEMA first_cloud_oss WITH DBPROPERTIES(
[0053] CATALOG = 'oss'
[0054] LOCATION='oss: / / first-cloud-bucket / data-1' )
[0056] Among them, first_cloud_oss is the database used to store the data table first_cloud_oss.t1_a_1, the data storage service type of the database first_cloud_oss is oss type, and the connection address is oss: / / First-cloud-bucket / data-1. It should be noted that first_cloud_oss is of SCHEMA type, and SCHEMA is defined as a set of interrelated database objects, such as tables, columns, data types, indexes, foreign keys, etc. Therefore, first_cloud_oss includes but is not limited to databases, and can also include any other objects associated with the database.
[0057] Of course, for the other data tables in the above example 1, each has its own corresponding database. The metadata of all the data tables in example 1 are shown in the form of tables, where Table 1 is the association table between the data table and its database, and Table 2 is the information table of each database.
[0058] Table 1 The relationship between the target data source and its database
[0059] Target data source (data table) Database (schema) t1_a_1 first_cloud_oss t2_a_1 first_cloud_oss t3_a_1 first_cloud_rds_mysql t4_a_r first_cloud_london_rds_mysql t5_second second_s3 t6_third third_mysql t7_fourth fourth_mysql t8_onpremise onpremise_mysql
[0060] Table 2 Database information table
[0061]
[0062]
[0063] In an actual application scenario, if the address of the target data source is different from the address of the local data area, it is necessary to allocate the operator used when accessing the target data source to a cloud computing node with external network access capability. Therefore, it is necessary to determine the address relationship between the address of the target data source and the local data area in the local cloud platform. Based on this, as shown in Table 2, the database corresponding to the target data source has location attribute information, and the location attribute information is used to identify the relationship between the database and the local cloud platform and the local data area in the local cloud platform. In an embodiment of the present application, the local cloud platform may include the platform where the implementation subject of each embodiment of the present application is located, and the local data area is the data area where the implementation subject is located in the local cloud platform. The local cloud platform may include a cloud platform provided by any cloud provider, typically, such as Amazon's AWS, Google's Google Cloud (including GCP and G suite), Microsoft's Azure, Alibaba's Alibaba Cloud, Tencent's Tencent Cloud, Baidu's Baidu Cloud, and the like. The cloud platform can also be divided into multiple data areas, and the data areas can be divided according to geographical areas. In one example, Alibaba Cloud can also be divided into Hangzhou data area, Beijing data area, Shanghai data area, US data area, European data area, Southeast Asia data area, etc. Of course, in other embodiments, the cloud platform can also be divided according to parameters such as IP addresses and user density, and this application does not limit the division method of data areas in the cloud platform.
[0064] The location attribute information may include a long-distance identifier, which is used to determine that the target data source is set in a cloud platform other than the local cloud platform, or is set in a data area other than the local data area in the local cloud platform. In one example, for the above example 1 and the metadata of Tables 1 and 2, the data query module is set in the data area of the first cloud platform located in Hangzhou, China. Then, for data tables that are not in the data area of the first cloud platform located in Hangzhou, China, a long-distance identifier can be set. As shown in Table 2, for data tables with long-distance identifiers, their distance identifiers can be set to "1", otherwise to a short-distance identifier "0". For data tables with a distance identifier of "0", it can be determined that the data table is set in the local data area. For example, for data table t2_a_1, according to the information of the database first_cloud_oss where the data table is located, it can be determined that the database is set in the data area of the first cloud platform located in Hangzhou, China, so its distance identifier is set to a short-distance identifier "0".
[0065] As shown in Table 1 and Table 2, the metadata of the target data source may include not only the database information where the target data source is located, such as the data storage service type, the data storage service address, the cloud platform where the database is located, and the data area. When the database has a remote identifier, the metadata may also include database access account information, such as a user login account and password, so that the implementation subject of the embodiment of the present application has the authority to access the target data source. In other business scenarios, users may also have private cloud accounts in the cloud platform, then the metadata may also include private cloud account information in the corresponding cloud platform, such as a private cloud ID, etc. The metadata may also include microservice account information, etc. The present application does not limit the information contained in the metadata. In an example, the DDL statement of the target data source first_cloud_london_rds_mysql.t4_a_r may include:
[0066] CREAT SCHEMA first_cloud_london_rds_mysql WITH DBPROPERTIES(
[0067] CATALOG = 'mysql',
[0068] LOCATION='firstcloud:eu-weat-1:jdbc:mysql: / / fghj.mysql.rds.aliyuncs.com / orders',
[0069] USER = 'hello',
[0070] PASSWORD = 'world',
[0071] INSTANCE_ID = 'xxxx',
[0072] VPC_ID = 'xxxxx', )
[0074] Among them, CATALOG is the data storage service classification, LOCATION is the data storage service address, USER and PASSWORD are the database access account information (user name and password), INSTANCE_ID is the microservice account, and VPC_ID is the private cloud account.
[0075] In one embodiment of the present application, in the process of determining the location attribute information in the metadata, it can be determined based on the data storage service address LOCATION. In one embodiment of the present application, the statement format of the data storage service address can be set to the form of '[cloud platform:][data area:] connection URL address'. By comparing LOCATION = 'oss: / / first-cloud-bucket / data-1' corresponding to the data table t1_a_1 and LOCATION = 'firstcloud:eu-weat-1:jdbc:mysql: / / fghj.mysql.rds.aliyuncs.com / orders' corresponding to the data table t4_a_r, it is found that the data storage service address corresponding to the data table t1_a_1 does not include the field information of the cloud platform and the data area, while the data storage service address corresponding to the data table t4_a_r includes the field information of the cloud platform and the data area. Based on this, the data query module can determine whether the data storage service address includes cloud platform information and data area information in the process of determining the long-distance identification. If the data storage service address includes cloud platform information or data area information, the location attribute information of the target data source is set as a long-distance identifier; otherwise, it can be set as a short-distance identifier.
[0076] S105: Generate an operator tree of the query plan, and mark the location attributes of the operators in the operator tree according to the location attribute information of the target data source.
[0077] In an embodiment of the present application, after obtaining the data query instruction for the at least one target data source and the metadata corresponding to the at least one target data source, an operator tree of the query plan can be generated. In one embodiment, the operator tree can include a physical query plan tree, and the operators in the operator tree can correspond to the physical operators in the physical query plan tree. The physical query plan tree can be generated based on the steps of generating a syntax tree, generating a logical plan tree, and generating a physical query plan tree.
[0078] Specifically, in one example, query analysis can be first performed on the SQL statement in Example 1 to generate a syntax tree. Query analysis can include lexical analysis and grammatical analysis, wherein lexical analysis refers to identifying meaningful logical units in SQL statements, such as operators (such as SELECT, FROM, INNER JOIN, WHERE, GROUP, etc.), numbers, function names, etc. in Example 1, and the main analysis tools may include LEX. Syntactic analysis combines the identified words into meaningful statements according to grammatical rules, and the main analysis tools may include Yacc. After query analysis is performed on the SQL statement, a syntax tree can be generated, and the leaf nodes of the syntax tree may include numbers, strings, attributes, etc., and other nodes are internal nodes.
[0079] After the syntax tree is generated, a logical plan can be determined based on the syntax tree to generate a logical plan tree. In the logical plan, the execution order of the clauses involved in the syntax tree can be determined, and the logical plan tree can be constructed according to the execution order. Taking the select statement as an example, the order of execution of each clause is: FROM>WHERE>GROUP BY>HAVING>SELECT>DISTINCT>UNION>ORDER BY>LIMIT. Of course, the logical plan can be further optimized, such as reducing the size of the connection operation object. The internal nodes in the generated logical plan tree correspond to the logical operators corresponding to the above clauses, and the leaf nodes correspond to each other. For example, the logical operator corresponding to the SELECT clause is selection (σ).
[0080] After the logical plan tree is generated, the physical query plan can be determined based on the logical plan tree to generate a physical query plan tree. The physical query plan tree can be the instructions actually executed by the physical node, that is, the execution instructions described in the embodiment of the present application, and its purpose is to select suitable physical operators for calculations based on different strategies. The physical query plan can set corresponding physical operators for the logical operators in the logical plan, and a logical operator can correspond to one or more physical operators, such as the physical operator corresponding to the logical operator selection (σ) is Filter. For example, the leaf nodes of the physical query plan tree generated by the logical operator are relational scan operators, and the internal nodes are other operators. Among them, the relational scan operator corresponds to the target data source, such as TableScan(t1_a_1) represents scanning the data table t1_a_1.
[0081] Figure 4 It is the operator tree generated for the SQL data query instruction in Example 1, that is, the execution instruction of the physical node. Among them, the operators Scan&Filter (scan filter operator) on each leaf node correspond to each target data source in Example 1. Figure 4 As shown, the location attribute corresponding to the operator is marked next to each node of the operator tree, and the location attribute is determined according to the location attribute information of the target data source corresponding to the operator. For example, the location attribute may include two identifiers, "REMOTE" and "LOCAL", where "REMOTE" is a long-distance identifier and "LOCAL" is a short-distance identifier. Similarly, the long-distance identifier indicates that the corresponding operator needs to run across cloud platforms or across data regions, and the short-distance identifier indicates that the corresponding operator runs in the local data region.
[0082] S107: Based on the operator tree, the operator is allocated to a cloud computing node in a corresponding area matching the location attribute.
[0083] In an embodiment of the present application, the operator can be allocated to the cloud computing node in the corresponding area that matches the location attribute based on the operator tree. In an embodiment of the present application, when it is determined that the location attribute of the operator is a long-distance identifier, a node resource pool with external network access capability in the cloud computing node in the local data area can be obtained, and the node resource pool includes multiple cloud computing nodes. Then, the operator is allocated to the cloud computing node in the node resource pool.
[0084] In the embodiment of the present application, in the process of allocating the operator to the cloud computing node, if Figure 5 As shown, it may include:
[0085] S501: Obtaining the network access speed of the target data source corresponding to the operator;
[0086] S503: Determine whether the network bandwidth of the node resource pool meets the network access speed;
[0087] S505: If it is determined that the network bandwidth does not meet the network access speed, a new cloud computing node is added to the node resource pool until the network bandwidth of the node resource pool meets the network access speed of the target data source.
[0088] In an embodiment of the present application, the node resource pool may include at least one cloud server, and the cloud server may include at least one cloud computing node, and the cloud computing node may be used to run the operator in the operator tree generated in step 103. In an actual application scenario, the node resource pool has a limitation on network bandwidth. Therefore, in order to enable each operator to run normally in the cloud computing node, it is necessary to allocate the operator according to the network bandwidth of the node resource pool. In a specific embodiment, the network access speed of the target data source corresponding to the operator can be obtained, and then it is determined whether the network bandwidth of the node resource pool meets the network access speed. If it is determined that the network bandwidth does not meet the network access speed, a new cloud computing node is added to the node resource pool to increase the bandwidth of the node resource pool until the network bandwidth of the node resource pool meets the network access speed of the target data source.
[0089] In the embodiment of the present application, the network access speeds of the multiple target data sources can be detected respectively. Figure 6 This is an application scenario adopted by the embodiment of the present application, such as Figure 6 As shown, the Hangzhou data area of the first cloud platform can at least include Table Scan ECS Pool 1, Table Scan ECS Pool 2 and Table Scan ECS, and are respectively provided with NAT 1 gateway, NAT 2 gateway and Elastic Public IP 1 gateway. Both NAT gateway and Elastic Public IP gateway are gateway products of the first cloud platform with external network access capabilities. Of course, there are other products such as network dedicated lines, etc. Figure 6As shown, according to the detection results, the network speed of the first cloud platform Europe West 1 (FirstCloud eu-west-1) is 9MB / s, the network speed of the second cloud platform is 4MB / s, the network speed of the third cloud platform is 6MB / s, the network speed of the fourth cloud platform is 8MB / s, and the network speed of the private cloud is 0.8MB / s. According to the shared bandwidth of NAT 1 gateway, NAT 2 gateway, and elastic public IP 1 gateway (10MB, 20MB, and 1MB respectively), the relational scanning operator (such as Scan&Filter) corresponding to the target data source t4_a_r can be allocated to the cloud computing node in Table Scan ECS Pool 1, the relational scanning operator (such as Scan&Filter) corresponding to the target data source t5_second, t6_third, and t7_fourth can be allocated to the cloud computing node in Table Scan ECS Pool 2, and the relational scanning operator (such as Scan&Filter) corresponding to the target data source t8_onpremise can be allocated to the cloud computing node in the table scan cloud server. For Figure 4 The operators with the “FIRST” distance identifier in the table can be allocated to ordinary cloud computing nodes in the Hangzhou data area of the first cloud platform (i.e., cloud computing nodes that do not need to have external network access capabilities) because they are local calculations.
[0090] In an embodiment of the present application, after the operator is assigned to the cloud computing node, the cloud computing node can be used to execute the execution instruction in the order of the execution instruction. Of course, after the cloud computing node executes the execution instruction, a response query result can be generated and the query result can be returned. In an embodiment of the present application, after receiving the query result, the query result can be displayed.
[0091] In an embodiment of the present application, in the process of accessing the target data source through an external network gateway, certain network resources need to be consumed. The network resources and services of the cloud platform provider have high costs, so it is necessary to charge according to the amount of network resources consumed by the user. Based on this, in an embodiment of the present application, when it is determined that the location attribute information of the target data source is a long-distance identifier, the amount of network resources consumed by accessing the target data source is counted separately. For different gateways, the billing method of network resources is different. For example, the NAT gateway of the first cloud platform is charged according to the duration, and the elastic public IP is charged according to the amount of data transmitted. Of course, in other embodiments, it is also possible to charge according to both the duration and the amount of data transmitted, and this application does not limit this.
[0092] In an embodiment of the present application, after respectively obtaining the network resource amount consumed by the target data source, the network resource amount can be displayed. In an embodiment of the present application, the network resource amount can be displayed in the operator tree, and the network resource amount consumed by accessing the target data source can be displayed at the operator corresponding to the target data source. Figure 7 As shown, it can be Figure 4 The operator nodes in the operator tree shown show the amount of network resources consumed to access the target data source. According to statistics, the total number of bits consumed by all table scan operators in this query is 3435325689 bytes, of which NAT 1 consumes 8 seconds, NAT 2 consumes 6 seconds, and the elastic public IP consumes a total of 1935321684 bytes. It should be noted that the time overhead on NAT is the longest running time of the table scan operator on the NAT, rather than the cumulative time of all operators. Therefore, NAT 1 is 8 seconds, NAT 2 is 6 seconds, and the consumption of elastic public IP is related to the data transmission volume. The consumption of elastic public IP 1 is the total number of bits of the table scan operator under this resource, 935321684. Through this display method, users can understand the consumption of network resources more clearly, and then verify whether the relevant billing is correct.
[0093] Of course, in an embodiment of the present application, a resource consumption query request from a user terminal may also be received, and the resource consumption query request includes identification information of the data query. In one example, the identification information corresponding to this data query may include the string "q201902071530bjf88a42890039547", and the present application does not limit the format of the identification information. Based on this, after receiving a resource consumption query request containing the identification information, the amount of network resources consumed by the data query can be obtained according to the identification information, and the amount of network resources can be sent to the user terminal.
[0094] The data query method provided in the embodiments of the present application can realize cross-platform, cross-data region, and cross-cloud joint data query. In each embodiment of the present application, by using the location attribute information set in the metadata of the target data source, the data query module can determine which data sources are located across cloud platforms, across data regions, and across clouds, thereby setting the operators corresponding to these target data sources in computing nodes with external network access capabilities. Through the above-mentioned cross-cloud platform and cross-data region multi-data source joint query on the cloud, the needs of data joint query in most application scenarios in the cloud platform can be met.
[0095] like Figure 8As shown, the present application also provides a data query device, including a processor and a memory for storing processor executable instructions, and when the processor executes the instructions, it can achieve:
[0096] Obtain data query instructions for the target data source;
[0097] Parsing the data query instruction to obtain metadata corresponding to the target data source, wherein the metadata includes location attribute information of the target data source;
[0098] Generate an operator tree of the query plan, and mark the location attributes of operators in the operator tree according to the location attribute information of the target data source;
[0099] Based on the operator tree, the operators are allocated to cloud computing nodes in corresponding areas matching the location attributes.
[0100] Optionally, in one embodiment of the present application, the location attribute information includes a long-distance identifier, and the long-distance identifier is used to determine that the target data source is set in a cloud platform outside the local cloud platform, or, is set in a data area outside the local data area in the local cloud platform.
[0101] Optionally, in one embodiment of the present application, the metadata also includes at least one of the following information: data storage service type, data storage service address, database access account information, private cloud account information, and microservice account information.
[0102] Optionally, in one embodiment of the present application, the remote identification is configured as follows:
[0103] Determine whether the data storage service address includes cloud platform information and data area information;
[0104] If the data storage service address includes cloud platform information or data area information, the location attribute information of the target data source is set to a long-distance identifier.
[0105] Optionally, in one embodiment of the present application, when the processor implements the step of allocating the operator to the cloud computing node of the corresponding area matching the location attribute based on the operator tree, the step includes:
[0106] When it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, a node resource pool with external network access capability in the local data area is obtained, wherein the node resource pool includes a plurality of cloud computing nodes;
[0107] The operator is allocated to the cloud computing node in the node resource pool.
[0108] Optionally, in one embodiment of the present application, when the processor implements the step of allocating the operator to the cloud computing node in the node resource pool, the processor includes:
[0109] Obtaining the network access speed of the target data source corresponding to the operator;
[0110] Determine whether the network bandwidth of the node resource pool meets the network access speed;
[0111] If it is determined that the network bandwidth does not meet the network access speed, a new cloud computing node is added to the node resource pool until the network bandwidth of the node resource pool meets the network access speed of the target data source.
[0112] Optionally, in one embodiment of the present application, when the processor implements the step of allocating the operator to the cloud computing node of the corresponding area matching the location attribute based on the operator tree, the step includes:
[0113] In the case where it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, the target data sources located in the same data area of the same cloud platform are divided into the same group according to the metadata;
[0114] The operators corresponding to the target data sources in the same group are allocated to at least one cloud computing node in the same node resource pool with external network access capability.
[0115] Optionally, in one embodiment of the present application, the processor further implements:
[0116] The operator tree is executed using the cloud computing node.
[0117] Optionally, in one embodiment of the present application, the processor further implements:
[0118] When it is determined that the location attribute information of the target data source is a long-distance identifier, the amount of network resources consumed in accessing the target data source is counted.
[0119] Optionally, in one embodiment of the present application, the network resource amount includes access duration and / or transmission data amount.
[0120] Optionally, in one embodiment of the present application, the processor further implements:
[0121] The operator tree is displayed, and the amount of network resources consumed for accessing the target data source is displayed at the operator corresponding to the target data source.
[0122] Optionally, in one embodiment of the present application, the processor further implements:
[0123] receiving a resource consumption query request, wherein the resource consumption query request includes identification information of a data query;
[0124] Acquire the amount of network resources consumed by the data query according to the identification information;
[0125] The network resource consumption is sent.
[0126] On the other hand, the present application further provides a computer-readable storage medium having computer instructions stored thereon, which, when executed, implement the steps of the method described in any of the above embodiments.
[0127] The computer-readable storage medium may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical means. The computer-readable storage medium described in this embodiment may include: devices that store information using electrical energy, such as various memories, such as RAM, ROM, etc.; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and USB flash drives; devices that store information optically, such as CDs or DVDs. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, and so on.
[0128] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.
[0129] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.
[0130] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.
[0131] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.
[0132] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0133] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data query device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data query device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data query device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0135] These computer program instructions may also be loaded onto a computer or other programmable data query device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0136] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0137] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0138] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.
[0140] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.
[0141] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0142] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A data query method, characterized in that: The method comprises: Obtain data query instructions for the target data source; Parse the data query instruction to obtain metadata corresponding to the target data source, wherein the metadata includes location attribute information of the target data source, and the location attribute information includes a long-distance identifier, and the long-distance identifier is used to determine that the target data source is set in a cloud platform other than the local cloud platform, or is set in a data area other than the local data area in the local cloud platform; Generate an operator tree of the query plan, and mark the location attributes of operators in the operator tree according to the location attribute information of the target data source; When it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, the operator is allocated to a cloud computing node with external network access capability.
2. The data query method according to claim 1, characterized in that: The metadata also includes at least one of the following information: data storage service type, data storage service address, database access account information, private cloud account information, and microservice account information.
3. The data query method according to claim 2, characterized in that: The remote identification is configured as follows: Determine whether the data storage service address includes cloud platform information and data area information; If the data storage service address includes cloud platform information or data area information, the location attribute information of the target data source is set to a long-distance identifier.
4. The data query method according to claim 1, characterized in that: When it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, allocating the operator to a cloud computing node with external network access capability includes: When it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, a node resource pool with external network access capability in the local data area is obtained, wherein the node resource pool includes a plurality of cloud computing nodes; The operator is allocated to the cloud computing node in the node resource pool.
5. The data query method according to claim 4, characterized in that: The allocating the operator to the cloud computing node in the node resource pool includes: Obtaining the network access speed of the target data source corresponding to the operator; Determining whether the network bandwidth of the node resource pool meets the network access speed; If it is determined that the network bandwidth does not meet the network access speed, a new cloud computing node is added to the node resource pool until the network bandwidth of the node resource pool meets the network access speed of the target data source.
6. The data query method according to claim 1, characterized in that: When it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, allocating the operator to a cloud computing node with external network access capability includes: In the case where it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, the target data sources located in the same data area of the same cloud platform are divided into the same group according to the metadata; The operators corresponding to the target data sources in the same group are allocated to at least one cloud computing node in the same node resource pool with external network access capability.
7. The data query method according to claim 1, characterized in that: The method further comprises: The operator tree is executed using the cloud computing node.
8. The data query method according to claim 1, characterized in that: The method further comprises: When it is determined that the location attribute information of the target data source is a long-distance identifier, the amount of network resources consumed in accessing the target data source is counted.
9. The data query method according to claim 8, characterized in that: The network resource amount includes access duration and / or transmission data amount.
10. The data query method according to claim 8, characterized in that: The method further comprises: The operator tree is displayed, and the amount of network resources consumed for accessing the target data source is displayed at the operator corresponding to the target data source.
11. The data query method according to claim 7, characterized in that: The method further comprises: receiving a resource consumption query request, wherein the resource consumption query request includes identification information of a data query; Acquire the amount of network resources consumed by the data query according to the identification information; The network resource consumption is sent.
12. A data query device, characterized in that: The invention comprises a processor and a memory for storing processor-executable instructions, wherein when the processor executes the instructions, the following are achieved: Obtain data query instructions for the target data source; Parse the data query instruction to obtain metadata corresponding to the target data source, wherein the metadata includes location attribute information of the target data source, and the location attribute information includes a long-distance identifier, and the long-distance identifier is used to determine that the target data source is set in a cloud platform other than the local cloud platform, or is set in a data area other than the local data area in the local cloud platform; Generate an operator tree of the query plan, and mark the location attributes of operators in the operator tree according to the location attribute information of the target data source; When it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, the operator is allocated to a cloud computing node with external network access capability.
13. The data query device according to claim 12, characterized in that: The metadata also includes at least one of the following information: data storage service type, data storage service address, database access account information, private cloud account information, and microservice account information.
14. The data query device according to claim 13, characterized in that: The remote identification is configured as follows: Determine whether the data storage service address includes cloud platform information and data area information; If the data storage service address includes cloud platform information or data area information, the location attribute information of the target data source is set to a long-distance identifier.
15. The data query device according to claim 12, characterized in that: The processor, in the implementation step, when determining that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, allocates the operator to a cloud computing node with external network access capability, including: When it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, a node resource pool with external network access capability in the local data area is obtained, wherein the node resource pool includes a plurality of cloud computing nodes; The operator is allocated to the cloud computing node in the node resource pool.
16. The data query device according to claim 15, characterized in that: The processor, when implementing the step of allocating the operator to the cloud computing node in the node resource pool, includes: Obtaining the network access speed of the target data source corresponding to the operator; Determining whether the network bandwidth of the node resource pool meets the network access speed; If it is determined that the network bandwidth does not meet the network access speed, a new cloud computing node is added to the node resource pool until the network bandwidth of the node resource pool meets the network access speed of the target data source.
17. The data query device according to claim 12, characterized in that: The processor, in the implementation step, when determining that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, allocates the operator to a cloud computing node with external network access capability, including: In the case where it is determined that the location attribute information of the target data source corresponding to the operator is a long-distance identifier, the target data sources located in the same data area of the same cloud platform are divided into the same group according to the metadata; The operators corresponding to the target data sources in the same group are allocated to at least one cloud computing node in the same node resource pool with external network access capability.
18. The data query device according to claim 12, characterized in that: The processor also implements: The operator tree is executed using the cloud computing node.
19. The data query device according to claim 12, characterized in that: The processor also implements: When it is determined that the location attribute information of the target data source is a long-distance identifier, the amount of network resources consumed in accessing the target data source is counted.
20. The data query device according to claim 19, characterized in that: The network resource amount includes access duration and / or transmission data amount.
21. The data query device according to claim 19, characterized in that: The processor also implements: The operator tree is displayed, and the amount of network resources consumed for accessing the target data source is displayed at the operator corresponding to the target data source.
22. The data query device according to claim 18, characterized in that: The processor also implements: receiving a resource consumption query request, wherein the resource consumption query request includes identification information of a data query; Acquire the amount of network resources consumed by the data query according to the identification information; The network resource consumption is sent.
23. A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor, enables the processor to execute the data query method described in any one of claims 1-11.
