Data query method and apparatus, computer device, and storage medium

Abstract

The present disclosure provides a data query method and apparatus, a computer device, and a storage medium. Virtual data tables corresponding to a physical data table are generated by means of attribute information of the physical data table, then a plurality of target virtual data tables among the virtual data tables are associated to obtain a wide virtual data table, and data querying is performed by using the wide virtual data table. Data in the wide virtual data table refers to data in the virtual data tables, and the data in the virtual data tables refers to data in the physical data table, therefore, the virtual data tables and the wide virtual data table only store the logic relationship of the data rather than storing the actual data, and thus, no additional storage space is occupied. In addition, the wide virtual data table can concentrate various data in one data table. Moreover, generating corresponding virtual data tables for a physical data table and configuring a wide virtual data table to refer to the virtual data tables simplifies the reference relationship of the wide virtual data table and facilitates maintenance.

Description

Data query method, device, computer equipment and storage medium

[0001] This disclosure claims priority to a Chinese patent application filed with the Patent Office of China on July 14, 2023, with application number 2023108713241 and application name “A data query method, device, computer equipment and storage medium,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to a data query method, apparatus, computer equipment, and storage medium. Background Art

[0003] In databases, large amounts of data are stored in physical tables. These tables often come from different data sources, have widely varying data structures, and store a large number of different data types. This results in low efficiency when querying data directly from the physical tables. Simply consolidating these physical tables into a single wide physical table requires a significant amount of storage space. Furthermore, due to the large amount of data contained within a wide physical table, it is difficult to update the table model, add new fields, or set new table definitions.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure at least provide a data query method, apparatus, computer equipment, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a data query method, comprising:

[0007] Get attribute information of multiple physical data tables;

[0008] For any physical data table, based on the attribute information, a virtual data table corresponding to the physical data table is generated; the virtual data table has the same data structure as the physical data table, and each data in the virtual data table refers to the corresponding data in the physical data table;

[0009] Associating multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table; each data in the virtual data wide table references the corresponding data in the target virtual data table;

[0010] In response to a data query instruction for the virtual data wide table, target data corresponding to the data query instruction is determined based on the virtual data wide table.

[0011] In an optional implementation, associating multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table includes:

[0012] Selecting a reference data table from the target virtual data table;

[0013] Selecting a related data table from the remaining target virtual data tables;

[0014] Associating the reference data table with the associated data table to obtain a new reference data table;

[0015] The steps of selecting the associated data table and associating the reference data table with the associated data table are repeated until the number of the remaining target virtual data tables is zero.

[0016] In an optional implementation, associating the benchmark data table with the associated data table to obtain a new benchmark data table includes:

[0017] Selecting a target association method from a plurality of association methods based on the attribute information of the reference data table and the association data table;

[0018] The target association method is used to associate the reference data table with the associated data table to obtain a new reference data table.

[0019] In an optional implementation manner, the target virtual data table is filtered out from the virtual data table by the following steps:

[0020] Obtaining at least one preset query statement for a target query service;

[0021] Based on the preset query statement, a target virtual data table matching the preset query statement is determined.

[0022] In an optional implementation, in response to a data query instruction for the virtual data wide table, determining target data corresponding to the data query instruction based on the virtual data wide table includes:

[0023] In response to a data query instruction for the virtual data wide table, determining a data query path corresponding to the data query instruction based on a data reference relationship indicated in the virtual data wide table;

[0024] Searching for target query data matching any query node in the data query path from currently stored intermediate query data; the query node indicates data query logic for determining the intermediate query data;

[0025] When any of the target query data is found, starting from the query node corresponding to the target query data, a data query is performed based on the target query data and along the data query path to obtain the target data corresponding to the data query instruction.

[0026] In an optional implementation, after searching for target query data corresponding to any query node in the data query path, the method further includes:

[0027] In the case that any of the target query data is not found, starting from the initial query node of the data query path, data query is performed along the data query path to obtain the target data corresponding to the data query instruction.

[0028] In an optional implementation, the method further includes:

[0029] In the process of performing data query according to the data query path, for each query node, determining an importance evaluation value of the query node based on at least one of a predicted query frequency, a historical query frequency, and a computational complexity corresponding to the query node;

[0030] When the importance evaluation value is higher than or equal to a preset evaluation value, the intermediate query data corresponding to the query node is stored.

[0031] In a second aspect, an embodiment of the present disclosure further provides a data query device, comprising:

[0032] An acquisition module is used to obtain attribute information of multiple physical data tables;

[0033] a generating module configured to generate, for any physical data table, a virtual data table corresponding to the physical data table based on the attribute information; the virtual data table and the physical data table having the same data structure, and each data item in the virtual data table references the corresponding data item in the physical data table;

[0034] an associating module, configured to associate multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table; each data item in the virtual data wide table references the corresponding data in the target virtual data table;

[0035] The query module is configured to respond to a data query instruction for the virtual data wide table and determine target data corresponding to the data query instruction based on the virtual data wide table.

[0036] In an optional implementation manner, the generating module is specifically configured to:

[0037] Selecting a reference data table from the target virtual data table;

[0038] Selecting a related data table from the remaining target virtual data tables;

[0039] Associating the reference data table with the associated data table to obtain a new reference data table;

[0040] The steps of selecting the associated data table and associating the reference data table with the associated data table are repeated until the number of the remaining target virtual data tables is zero.

[0041] In an optional implementation, when the generating module associates the reference data table with the associated data table to obtain a new reference data table, it is configured to:

[0042] Selecting a target association method from a plurality of association methods based on the attribute information of the reference data table and the association data table;

[0043] The target association method is used to associate the reference data table with the associated data table to obtain a new reference data table.

[0044] In an optional implementation, the device further includes a determining module, configured to:

[0045] Obtaining at least one preset query statement for a target query service;

[0046] Based on the preset query statement, a target virtual data table matching the preset query statement is determined.

[0047] In an optional implementation, the query module is specifically configured to:

[0048] In response to a data query instruction for the virtual data wide table, determining a data query path corresponding to the data query instruction based on a data reference relationship indicated in the virtual data wide table;

[0049] Searching for target query data matching any query node in the data query path from currently stored intermediate query data; the query node indicates data query logic for determining the intermediate query data;

[0050] When any of the target query data is found, starting from the query node corresponding to the target query data, a data query is performed based on the target query data and along the data query path to obtain the target data corresponding to the data query instruction.

[0051] In an optional implementation, after searching for target query data corresponding to any query node in the data query path, the query module is further configured to:

[0052] In the case that any of the target query data is not found, starting from the initial query node of the data query path, data query is performed along the data query path to obtain the target data corresponding to the data query instruction.

[0053] In an optional implementation, the query module is further configured to:

[0054] In the process of performing data query according to the data query path, for each query node, determining an importance evaluation value of the query node based on at least one of a predicted query frequency, a historical query frequency, and a computational complexity corresponding to the query node;

[0055] When the importance evaluation value is higher than or equal to a preset evaluation value, the intermediate query data corresponding to the query node is stored.

[0056] In a third aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the machine-readable instructions perform the steps of the above-mentioned first aspect, or any possible implementation of the first aspect.

[0057] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, it executes the steps of the above-mentioned first aspect or any possible implementation of the first aspect.

[0058] For a description of the effects of the above-mentioned data query device, computer equipment, and computer-readable storage medium, please refer to the description of the above-mentioned data query method, which will not be repeated here.

[0059] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.

[0060] The data query method, apparatus, computer equipment and storage medium provided by the embodiments of the present disclosure generate a virtual data table corresponding to the physical data table through the attribute information of the physical data table, and then associate multiple target virtual data tables in the virtual data table to obtain a virtual data wide table, and use the virtual data wide table to perform data query. Since the data in the virtual data wide table references the virtual data table, and the data in the virtual data table references the physical data table, the virtual data wide table only stores the logical relationship of the data and does not store the actual data, so no additional storage space is required; the virtual data wide table can concentrate various data in one data table, and by generating a corresponding virtual data table for the physical data table and then having the virtual data wide table reference the virtual data table, the reference relationship of the virtual data wide table is simplified and easy to maintain.

[0061] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0063] FIG1 shows a flow chart of a data query method provided by some embodiments of the present disclosure;

[0064] FIG2 shows a schematic diagram of a data query path provided by some embodiments of the present disclosure;

[0065] FIG3 shows a schematic diagram of a data query system provided by some embodiments of the present disclosure;

[0066] FIG4 shows a schematic diagram of a data query device provided by some embodiments of the present disclosure;

[0067] FIG5 shows a schematic diagram of a computer device provided by some embodiments of the present disclosure. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0069] Research has found that databases typically store data in physical tables. When providing data query services, querying data directly from physical tables is inefficient because physical tables often come from different data sources, have significantly different data structures, and store a large number of different data types. Consolidating these physical tables into a single wide physical table requires a significant amount of storage space, and updating them is difficult due to the large amount of data contained within them.

[0070] Based on the above research, the present disclosure provides a data query method, which generates a virtual data table corresponding to the physical data table through the attribute information of the physical data table, and then associates multiple target virtual data tables in the virtual data table to obtain a virtual data wide table, and uses the virtual data wide table to perform data query. Since the data in the virtual data wide table references the virtual data table, and the data in the virtual data table references the physical data table, the virtual data wide table only stores the logical relationship of the data and does not store the actual data, so it does not need to occupy additional storage space; the virtual data wide table can concentrate various data in one data table, and by generating a corresponding virtual data table for the physical data table, and then the virtual data wide table references the virtual data table, the reference relationship of the virtual data wide table is simplified and easy to maintain.

[0071] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by this disclosure for the above problems below should be the contributions made by the inventors to this disclosure during the disclosure process.

[0072] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0073] To facilitate understanding of this embodiment, we first provide a detailed introduction to a data query method disclosed in this embodiment. The data query method provided in this embodiment is generally executed by a computer device with certain computing capabilities, such as a terminal device, a server, or other processing device. In some possible implementations, this data query method can be implemented by a processor invoking computer-readable instructions stored in a memory.

[0074] The data query method provided by the embodiment of the present disclosure is described below by taking the execution subject as a terminal device as an example.

[0075] 1 is a flowchart of a data query method provided by an embodiment of the present disclosure. The method includes steps S101 to S104, wherein:

[0076] S101: Acquire attribute information of multiple physical data tables.

[0077] In this step, the terminal device can obtain attribute information of multiple physical data tables in the database. The attribute information may include the storage location of the physical data table, such as the address information, port information, access protocol and other information of the physical data table; the attribute information may also include data structure information of the physical data table, such as header information, key value information, etc.

[0078] S102. For any physical data table, generate a virtual data table corresponding to the physical data table based on the attribute information; the virtual data table has the same data structure as the physical data table, and each data item in the virtual data table references the corresponding data in the physical data table.

[0079] In this step, a virtual data table corresponding to the physical data table may be generated based on the attribute information of the physical data table. The table structure schema of the virtual data table may match the schema of the physical data table.

[0080] Virtual tables share the same data structure as physical tables, including row, column, and key information. Each data item in a virtual table references the corresponding data item in the physical table. For example, the data in the second row and third column of a virtual table references the data in the second row and third column of a physical table. When data in the physical table is updated, the data in the virtual table is also updated because it references the physical table.

[0081] It's important to note that the key-value information and other data in the virtual data table are not the actual data in the physical data table. These key-value information references the data in the physical data table. In other words, the virtual data table records the correspondence (i.e., the reference relationship) between the key-value information in the virtual data table and the physical data table. The virtual data table does not maintain the specific key-value data.

[0082] The virtual data table may have the same index structure (such as a secondary index, a detailed index, a column index, a reverse index, etc.) as its corresponding physical data table.

[0083] S103 , associating multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table; each data item in the virtual data wide table references the corresponding data in the target virtual data table.

[0084] In this step, the virtual data tables may be screened to select target virtual data tables. When selecting the target virtual data tables, the screening may be performed according to the query requirements of the query business to be performed.

[0085] When filtering a target virtual data table, you can select an appropriate target filtering method from a variety of filtering methods. For example, when the user specifies a required virtual data table, the user-specified virtual data table can be directly used as the target virtual data table; when the user does not specify a virtual data table, you can filter according to the needs of the target business.

[0086] Exemplarily, at least one preset query statement of the target query service may be obtained first, and then based on the preset query statement, target virtual data matching the preset query statement may be determined.

[0087] For example, the target query service may include multiple preset query statements, and the preset query statements may indicate data to be queried. In this way, the virtual data table including the data to be queried may be used as the target virtual data table.

[0088] The above-mentioned preset query statements can be query statements of the relational database management system MySQL, that is, Structured Query Language (SQL), which can be used to access data and query, update and manage relational database systems. Query statements can be used to perform operations, conditional queries, and other processing on the data in the database.

[0089] Furthermore, virtual data tables can be screened using machine learning, deep learning and other models. Machine learning or deep learning models can use sample table pattern data provided by users, historical query statements and query information collected by the model for analysis and learning, thereby determining a target virtual data table that better meets user needs.

[0090] After obtaining the target virtual data table, each target virtual data table may be associated to generate a virtual data wide table corresponding to the target virtual data table.

[0091] It is worth noting that the above-mentioned target virtual data table can not only be obtained by filtering from the virtual data table, but can also be other virtual data tables specified by the user or business needs. For example, the target virtual data table may contain an intermediate virtual data table generated by the above-mentioned virtual data table, or an intermediate virtual data table generated by other means specified by the user.

[0092] The virtual data table corresponding to the above physical data table can be called the source data table, and other virtual data tables generated using the source data table can be called business data tables (such as the above intermediate virtual data table), wherein the above virtual data wide table can be one of the business data tables.

[0093] In a possible implementation, the source data table and / or the business data table may be used as the target virtual data table to generate the virtual data wide table. The generated virtual data wide table may also be used to generate other virtual data wide tables.

[0094] When associating the target virtual data table, you can first select a baseline data table from the target virtual data table, then select an associated data table from the remaining target virtual data tables, associate the baseline data table with the associated data table, and obtain a new baseline data table. After that, you can repeat the steps of selecting the associated data table and updating the baseline data table with the associated data table until the number of remaining target virtual data tables is zero, and use the final baseline data table as the above-mentioned virtual data wide table.

[0095] When associating a benchmark data table with an associated data table, a target association method can be selected from a variety of association methods based on the attribute information of the benchmark data table and the associated data table. Then, the benchmark data table and the associated data table are associated using the target association method to obtain a new benchmark data table.

[0096] For example, the association methods may include Join association, Union association, and Filter association.

[0097] Join associations: Use the primary and foreign keys of data tables to link data tables in different schemas. The resulting baseline data table can form a continuous data view. This is typically used to join fact tables and dimension tables, presenting the same schema as a complete, clearly structured physical wide table without generating physical intermediate data.

[0098] Union: By joining the same fields across different tables, you can combine partial data into a new set of data. This allows you to query data using different computation methods within the same schema (for example, combining streaming data with batch data, or combining batch data of different granularities) without changing the computational logic of the original table.

[0099] Filter association: A virtual data wide table created using filter association can solve the need for obtaining a convenient small table by conditional filtering on a large table. It presents the same schema as a small table with clear calculation logic and does not generate physical intermediate data.

[0100] After associating the target virtual data tables, a virtual data wide table is formed. The fields of the virtual data wide table are mapped from the target virtual data table. Logically, it is used for a clear schema and can be used by users as a normal data table query. At the same time, the virtual data wide table does not store any table data, only the table-related logic.

[0101] S104 : In response to a data query instruction for the virtual data wide table, determine target data corresponding to the data query instruction based on the virtual data wide table.

[0102] After obtaining the virtual data wide table, the user can initiate a data query instruction for the virtual data wide table to the terminal device at any time. The data query instruction can be a query statement for the virtual data wide table. The terminal device can parse and execute the data query instruction and feed back the queried target data to the user to realize the data query.

[0103] When querying target data, you can first determine the data query path corresponding to the data query instruction based on the data reference relationship indicated in the virtual data wide table, and then query the target query data that matches any query node in the query path from the currently stored intermediate query data; the query node only uses the data query logic used to determine the intermediate query data; when any of the target query data is found, you can start from the query node corresponding to the target query data, and based on the target query data, perform data query according to the data query path to obtain the target data corresponding to the data query instruction.

[0104] When a terminal device receives a data query instruction, due to the nested structure of virtual data tables, it can parse the data query instruction layer by layer until the data query instruction is parsed into a query statement based on the target virtual data table. Based on the obtained query statement, a multi-level data query path can be obtained. The data query path can include multiple query nodes, and the query node can indicate data query logic for determining intermediate query data. By executing the data query logic, the intermediate query data corresponding to the query node can be obtained.

[0105] The data query logic of the above query node usually requires the use of the intermediate query data corresponding to the previous query node in the query order. If the intermediate query data corresponding to the current query node can be directly obtained, there is no need to execute the query nodes before the current query node, which can effectively improve the query efficiency.

[0106] Referring to Figure 2, a schematic diagram of a data query path provided by an embodiment of the present disclosure is shown. The figure includes multiple data query instructions (query statements), namely Q1, Q2, Q3, Q4, and Q5. The data query path corresponding to each data query instruction is shown in the figure. The figure shows a virtual data wide table 21, multiple physical data tables 22, and query nodes 23. It can be seen that query nodes 231, query nodes 232, and query nodes 233 are simultaneously part of the data query paths of Q1 and Q2, that is, the common path of Q1 and Q2. Therefore, the intermediate query data corresponding to query nodes 231, query nodes 232, and query nodes 233 can be pre-calculated and stored, so that when Q1 and Q2 are received, the stored intermediate data can be directly used to achieve a fast response to the query statement.

[0107] Similarly, in Figure 2, query node 234, query node 235, and query node 236 are simultaneously in the data query paths of Q3 and Q4, and the intermediate query data of query node 234, query node 235, and query node 236 can be stored; query node 237, query node 238, and query node 239 are simultaneously in the data query paths of Q4 and Q5, and the intermediate query data of query node 237, query node 238, and query node 239 can also be stored.

[0108] For example, in the query path of Q4, query node 2310 needs to use the intermediate query data of query node 236, query node 239 and query node 2311 at the same time. Then, the intermediate query data of query node 235 can be used to determine the intermediate query data of query node 2311, and then the intermediate query data of query node 236, query node 239 and query node 2311 can be used to determine the intermediate query data of query node 2310, and then the target data corresponding to Q4 can be obtained, thereby greatly improving the data query efficiency.

[0109] As shown in FIG2 , during the data query process for Q1, the intermediate query data corresponding to query node 231 and query node 232 can be directly obtained, and starting from query node 231 and query node 232 , the data query is performed according to the data query path using the intermediate query data.

[0110] In the specific implementation process, the preset query statements of the target query business can be parsed in advance to determine the common query nodes (existing in the data query paths of multiple query statements) and the intermediate query data corresponding to the common query nodes can be pre-determined.

[0111] The pre-stored intermediate query data can be stored as a cache in the memory of the terminal device and set an expiration date, thereby reducing the probability of the queried data being inconsistent with the actual result due to changes in the physical data table.

[0112] Since the data of the virtual data wide table comes from multiple physical data wide tables, different types of computing engines can be used to calculate the split data query logic and assemble it according to the data query path. Users only need to query the virtual data wide table without being aware of the calculations of multiple computing engines.

[0113] When pre-storing the intermediate query data, in addition to filtering by query frequency (common query nodes), you can also filter based on the complexity of the data query logic corresponding to the query node. For example, if a data query logic contains relatively complex data query logic (such as sum, count, average, variance, etc.), you can pre-calculate it and store the resulting intermediate query data.

[0114] The stored intermediate query data can form a materialized view. Through the materialized view, the position of the corresponding query node in the data query path can be determined, thereby judging whether the intermediate query data is the data required by the data query instruction.

[0115] If the target query data cannot be found, you can directly start from the initial query node in the data query path, perform data query according to the data query path, and obtain the target data corresponding to the data query instruction.

[0116] During the data query process according to the data query path, some generated intermediate query data may also be stored. For example, for each query node, an importance evaluation value may be determined based on at least one of the query node's corresponding historical query frequency, computational complexity, or predicted query frequency. If the importance evaluation value is greater than or equal to a preset evaluation value, the intermediate query data corresponding to the query node is stored.

[0117] The above-mentioned intermediate query data may have some correlation relationships, and the intermediate query data may affect each other. For example, the intermediate query data corresponding to a query node may require the intermediate data of other query nodes to be determined. Therefore, when determining the importance evaluation value of the query node, the correlation relationship of the intermediate query data can also be introduced as an evaluation indicator to determine its importance evaluation value.

[0118] When screening query nodes, the query nodes can be evaluated individually based on their importance evaluation values ​​to screen out query nodes with importance evaluation values ​​higher than a preset evaluation value. Alternatively, a query node set can be screened out from each query node so that the sum of the importance evaluation values ​​corresponding to the query node set is the highest. This allows the optimal combination of intermediate query data to be selected while taking into account the relevance of the intermediate query data, thereby improving the utilization rate of the intermediate query data.

[0119] The above-mentioned historical query frequency can be the frequency of calculating the query node within a preset time period; the calculation complexity can be determined according to the operation type and the number of operations in the corresponding data query logic.

[0120] The aforementioned virtual data wide table can encapsulate the underlying multimodal physical data, freeing users from complex data modeling. Instead, users can perform data modeling through virtualization and build lightweight and flexible data models through virtual tables.

[0121] Because virtual data wide tables are logical structures based on query statements and relationship definitions (reference relationships within virtual data tables), they are used to organize and represent data relationships rather than actually storing the data itself. They can retrieve required data by executing query statements rather than maintaining actual data copies in the database. Furthermore, they can retrieve the latest data results in real time through query and association operations without having to maintain and synchronize large numbers of physical data copies. This saves storage resources and provides more flexible and dynamic data access. Query results from virtual data wide tables can be calculated and analyzed in real time as needed to meet specific business needs, without having to worry about the expansion of data assets and the complexity of storage management.

[0122] Virtual data wide tables can build more flexible and dynamic data models, embed data logical relationships and query logic into virtual data wide tables, and provide higher-level data abstraction and organization. This makes data processing and analysis easier, and allows for complex queries and analysis through hierarchical relationships and association operations.

[0123] During the nesting of virtual data wide tables, it is possible to clearly trace the upstream tables (physical data tables) from which the data in each virtual data table is derived, thus forming a data source and transmission path. Because the data lineage relationships of virtual data wide tables are clear, this information can be used to organize and merge asset calibers. Asset caliber refers to the definition and standardization of data assets (virtual data wide tables), including data naming conventions, structure definitions, data types, etc. By analyzing the data lineage relationships between virtual data wide tables, virtual data wide tables with similar data lineages and structures can be identified. These virtual data tables may correspond to similar business data assets.

[0124] When sorting and consolidating asset scopes, you can identify and merge virtual data wide tables with similar data lineage by comparing their similarity. A high similarity indicates that these virtual data wide tables share similar data sources and structures, potentially corresponding to the same or similar business entities or data views. By merging these similar virtual data wide tables, you can reduce duplication in definition and maintenance, improving data management efficiency and consistency.

[0125] Refer to Figure 3, which is a schematic diagram of a data query system provided by an embodiment of the present disclosure. In Figure 3, the data query system may include a logical layer, an acceleration layer, and a physical layer. A user may send a query statement for a virtual data wide table to a terminal device of the data query system. Each data column in the virtual data wide table comes from a physical data table corresponding to multiple virtual data tables. The data of each physical data table may come from different data platforms. The data query system may parse the query statement through the logical layer, search for appropriate intermediate query data in the acceleration layer, search for the required partial data from the physical layer, and return the data required by the user based on the intermediate query data and the data found in the physical layer.

[0126] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0127] Based on the same inventive concept, the embodiment of the present disclosure also provides a data query device corresponding to the data query method. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned data query method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0128] 4 is a schematic diagram of a data query device provided by an embodiment of the present disclosure, the device comprising:

[0129] An acquisition module 410 is used to acquire attribute information of multiple physical data tables;

[0130] A generating module 420 is configured to generate, for any physical data table, a virtual data table corresponding to the physical data table based on the attribute information; the virtual data table has the same data structure as the physical data table, and each data item in the virtual data table references the corresponding data item in the physical data table;

[0131] An association module 430 is configured to associate multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table; each data item in the virtual data wide table references the corresponding data in the target virtual data table;

[0132] The query module 440 is configured to respond to a data query instruction for the virtual data wide table and determine target data corresponding to the data query instruction based on the virtual data wide table.

[0133] In an optional implementation, the generating module 420 is specifically configured to:

[0134] Selecting a reference data table from the target virtual data table;

[0135] Selecting a related data table from the remaining target virtual data tables;

[0136] Associating the reference data table with the associated data table to obtain a new reference data table;

[0137] The steps of selecting the associated data table and associating the reference data table with the associated data table are repeated until the number of the remaining target virtual data tables is zero.

[0138] In an optional implementation, when the generating module 420 associates the reference data table with the associated data table to obtain a new reference data table, it is configured to:

[0139] Selecting a target association method from a plurality of association methods based on the attribute information of the reference data table and the association data table;

[0140] The target association method is used to associate the reference data table with the associated data table to obtain a new reference data table.

[0141] In an optional implementation, the device further includes a determining module, configured to:

[0142] Obtaining at least one preset query statement for a target query service;

[0143] Based on the preset query statement, a target virtual data table matching the preset query statement is determined.

[0144] In an optional implementation, the query module 440 is specifically configured to:

[0145] In response to a data query instruction for the virtual data wide table, determining a data query path corresponding to the data query instruction based on a data reference relationship indicated in the virtual data wide table;

[0146] Searching for target query data matching any query node in the data query path from currently stored intermediate query data; the query node indicates data query logic for determining the intermediate query data;

[0147] When any of the target query data is found, starting from the query node corresponding to the target query data, a data query is performed based on the target query data and along the data query path to obtain the target data corresponding to the data query instruction.

[0148] In an optional implementation, after searching for target query data corresponding to any query node in the data query path, the query module 440 is further configured to:

[0149] In the case that any of the target query data is not found, starting from the initial query node of the data query path, data query is performed along the data query path to obtain the target data corresponding to the data query instruction.

[0150] In an optional implementation, the query module 440 is further configured to:

[0151] In the process of performing data query according to the data query path, for each query node, determining an importance evaluation value of the query node based on a historical query frequency and / or a computational complexity corresponding to the query node;

[0152] When the importance evaluation value is higher than or equal to a preset evaluation value, the intermediate query data corresponding to the query node is stored.

[0153] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0154] The present disclosure also provides a computer device, as shown in FIG5 , which is a schematic diagram of the structure of the computer device provided by the present disclosure, including:

[0155] A processor 51 and a memory 52; the memory 52 stores machine-readable instructions executable by the processor 51, and the processor 51 is configured to execute the machine-readable instructions stored in the memory 52. ​​When the machine-readable instructions are executed by the processor 51, the processor 51 performs the following steps:

[0156] Get attribute information of multiple physical data tables;

[0157] For any physical data table, based on the attribute information, a virtual data table corresponding to the physical data table is generated; the virtual data table has the same data structure as the physical data table, and each data in the virtual data table refers to the corresponding data in the physical data table;

[0158] Associating multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table; each data in the virtual data wide table references the corresponding data in the target virtual data table;

[0159] In response to a data query instruction for the virtual data wide table, target data corresponding to the data query instruction is determined based on the virtual data wide table.

[0160] In an optional embodiment, the step executed by the processor 51 of associating multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table includes:

[0161] Selecting a reference data table from the target virtual data table;

[0162] Selecting a related data table from the remaining target virtual data tables;

[0163] Associating the reference data table with the associated data table to obtain a new reference data table;

[0164] The steps of selecting the associated data table and associating the reference data table with the associated data table are repeated until the number of the remaining target virtual data tables is zero.

[0165] In an optional embodiment, in the step executed by the processor 51, associating the reference data table with the associated data table to obtain a new reference data table includes:

[0166] Selecting a target association method from a plurality of association methods based on the attribute information of the reference data table and the association data table;

[0167] The target association method is used to associate the reference data table with the associated data table to obtain a new reference data table.

[0168] In an optional implementation manner, the steps executed by the processor 51 further include:

[0169] Obtaining at least one preset query statement for a target query service;

[0170] Based on the preset query statement, a target virtual data table matching the preset query statement is determined.

[0171] In an optional embodiment, the step executed by the processor 51, in response to the data query instruction for the virtual data wide table, determining the target data corresponding to the data query instruction based on the virtual data wide table, includes:

[0172] In response to a data query instruction for the virtual data wide table, determining a data query path corresponding to the data query instruction based on a data reference relationship indicated in the virtual data wide table;

[0173] Searching for target query data matching any query node in the data query path from currently stored intermediate query data; the query node indicates data query logic for determining the intermediate query data;

[0174] When any of the target query data is found, starting from the query node corresponding to the target query data, a data query is performed based on the target query data and along the data query path to obtain the target data corresponding to the data query instruction.

[0175] In an optional embodiment, the steps executed by the processor 51, after searching for target query data corresponding to any query node in the data query path, further include:

[0176] In the case that any of the target query data is not found, starting from the initial query node of the data query path, data query is performed along the data query path to obtain the target data corresponding to the data query instruction.

[0177] In an optional implementation manner, the steps executed by the processor 51 further include:

[0178] In the process of performing data query according to the data query path, for each query node, determining an importance evaluation value of the query node based on a historical query frequency and / or a computational complexity corresponding to the query node;

[0179] When the importance evaluation value is higher than or equal to a preset evaluation value, the intermediate query data corresponding to the query node is stored.

[0180] The above-mentioned memory 52 includes internal memory 521 and external memory 522; the memory 521 here is also called internal memory, which is used to temporarily store the calculation data in the processor 51 and the data exchanged with the external memory 522 such as the hard disk. The processor 51 exchanges data with the external memory 522 through the internal memory 521.

[0181] The specific execution process of the above instructions can refer to the steps of the data query method described in the embodiment of the present disclosure, which will not be repeated here.

[0182] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of the data query method described in the above method embodiment. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0183] The embodiments of the present disclosure also provide a computer program product, which carries program code. The instructions included in the program code can be used to execute the steps of the data query method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0184] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0188] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0189] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A data query method, characterized in that: include: Get the attribute information of multiple physical data tables; For any physical data table, based on the attribute information, generate a virtual data table corresponding to the physical data table; The virtual data table has the same data structure as the physical data table, and each item of data in the virtual data table refers to the corresponding data in the physical data table; Associating multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table; each data item in the virtual data wide table references the corresponding data in the target virtual data table; In response to a data query instruction for the virtual data wide table, target data corresponding to the data query instruction is determined based on the virtual data wide table.

2. The method according to claim 1, characterized in that The step of associating a plurality of target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table includes: Selecting a reference data table from the target virtual data table; Selecting a related data table from the remaining target virtual data tables; Associating the reference data table with the associated data table to obtain a new reference data table; The steps of selecting the associated data table and associating the reference data table with the associated data table are repeated until the number of the remaining target virtual data tables is zero.

3. The method according to claim 2, characterized in that The step of associating the reference data table with the associated data table to obtain a new reference data table includes: Based on the attribute information of the reference data table and the associated data table, selecting a target association method from a plurality of association methods; The reference data table is associated with the associated data table using the target association method to obtain a new reference data table.

4. The method according to claim 1, characterized in that: The target virtual data table is filtered out from the virtual data table by the following steps: Obtaining at least one preset query statement of a target query service; Based on the preset query statement, a target virtual data table matching the preset query statement is determined.

5. The method according to claim 1, characterized in that The step of responding to the data query instruction for the virtual data wide table and determining the target data corresponding to the data query instruction based on the virtual data wide table includes: In response to a data query instruction for the virtual data wide table, determining a data query path corresponding to the data query instruction based on a data reference relationship indicated in the virtual data wide table; Searching for target query data matching any query node in the data query path from the currently stored intermediate query data; the query node indicates data query logic for determining the intermediate query data; When any of the target query data is found, starting from the query node corresponding to the target query data, data query is performed based on the target query data and according to the data query path to obtain the target data corresponding to the data query instruction.

6. The method according to claim 5, characterized in that After searching for target query data corresponding to any query node in the data query path, the method further includes: In the case that any of the target query data is not found, starting from the initial query node of the data query path, data query is performed along the data query path to obtain the target data corresponding to the data query instruction.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: In the process of performing data query according to the data query path, for each query node, determining the importance evaluation value of the query node based on at least one of the predicted query frequency, the historical query frequency, and the calculation complexity corresponding to the query node; When the importance evaluation value is higher than or equal to a preset evaluation value, the intermediate query data corresponding to the query node is stored.

8. A computer device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory. When the machine-readable instructions are executed by the processor, the processor executes the following data query method: Get the attribute information of multiple physical data tables; For any physical data table, based on the attribute information, a virtual data table corresponding to the physical data table is generated; the virtual data table has the same data structure as the physical data table, and each data in the virtual data table refers to the corresponding data in the physical data table; Associating multiple target virtual data tables in the virtual data table to generate a virtual data wide table corresponding to the target virtual data table; each data item in the virtual data wide table references the corresponding data in the target virtual data table; In response to a data query instruction for the virtual data wide table, target data corresponding to the data query instruction is determined based on the virtual data wide table.

9. The computer device according to claim 8, characterized in that In the data query method executed by the processor, multiple target virtual data tables in the virtual data table are associated to generate a virtual data wide table corresponding to the target virtual data table, including: Selecting a reference data table from the target virtual data table; Selecting a related data table from the remaining target virtual data tables; Associating the reference data table with the associated data table to obtain a new reference data table; The steps of selecting the associated data table and associating the reference data table with the associated data table are repeated until the number of the remaining target virtual data tables is zero.

10. The computer device according to claim 9, characterized in that In the data query method executed by the processor, associating the reference data table with the associated data table to obtain a new reference data table includes: Based on the attribute information of the reference data table and the associated data table, selecting a target association method from a plurality of association methods; The reference data table is associated with the associated data table using the target association method to obtain a new reference data table.

11. The computer device according to claim 8, characterized in that In the data query method executed by the processor, a target virtual data table is screened out from the virtual data table in the following manner: Obtaining at least one preset query statement of a target query service; Based on the preset query statement, a target virtual data table matching the preset query statement is determined.

12. The computer device according to claim 8, characterized in that In the data query method executed by the processor, in response to a data query instruction for the virtual data wide table, determining target data corresponding to the data query instruction based on the virtual data wide table includes: In response to a data query instruction for the virtual data wide table, determining a data query path corresponding to the data query instruction based on a data reference relationship indicated in the virtual data wide table; Searching for target query data matching any query node in the data query path from the currently stored intermediate query data; the query node indicates data query logic for determining the intermediate query data; When any of the target query data is found, starting from the query node corresponding to the target query data, data query is performed based on the target query data and according to the data query path to obtain the target data corresponding to the data query instruction.

13. The computer device according to claim 12, characterized in that The data query method executed by the processor further includes, after searching for target query data corresponding to any query node in the data query path: In the case that any of the target query data is not found, starting from the initial query node of the data query path, data query is performed along the data query path to obtain the target data corresponding to the data query instruction.

14. The computer device according to claim 12 or 13, characterized in that: The data query method executed by the processor further includes: In the process of performing data query according to the data query path, for each query node, determining the importance evaluation value of the query node based on at least one of the predicted query frequency, the historical query frequency, and the calculation complexity corresponding to the query node; When the importance evaluation value is higher than or equal to a preset evaluation value, the intermediate query data corresponding to the query node is stored.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer device, the computer device executes the data query method according to any one of claims 1 to 7.

Images