Metadata generation method, data query method, system, device and storage medium

Abstract

One or more embodiments of this specification provide a metadata generation method, a data query method, a database system, an electronic device, and a computer-readable storage medium. The metadata generation method includes: obtaining several physical tables to be processed from at least one database; classifying the several physical tables according to the logical structure of each of the physical tables to obtain at least one category; wherein the logical structure is determined according to the data structure of the physical table and / or the logical data model to which the physical table belongs; different physical tables in any category have the same logical structure; metadata is created for each category; wherein the metadata of each category includes basic information and physical information, wherein the basic information is used to describe the logical structure of the category, and the physical information is used to describe the storage status of each physical table belonging to the category. Different physical tables with the same logical structure correspond to the same metadata.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a metadata generation method, a data query method, a database system, an electronic device, and a computer-readable storage medium. Background Art

[0002] Metadata, also known as intermediary data or relay data, is data about data. It primarily describes data properties and supports functions such as indicating storage location, historical data, resource search, and file logging. Its purpose is to identify resources, evaluate resources, track changes in resources during use, enable simple and efficient management of large amounts of networked data, and enable the effective discovery, retrieval, integrated organization, and management of used resources.

[0003] In databases used in related technologies, metadata is defined based on a single physical table, meaning each physical table corresponds to a piece of metadata. However, multiple physical tables stored in one or more databases may have the same source. Defining metadata based on a single physical table results in excessive fragmentation of metadata, which breaks the connection between data. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a metadata generation method, a data query method, a database system, an electronic device, and a computer-readable storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a metadata generation method is proposed, including:

[0007] Obtaining a plurality of physical tables to be processed from at least one database;

[0008] Classifying the plurality of physical tables according to the logical structure of each physical table to obtain at least one category; wherein the logical structure is determined according to the data structure of the physical table and / or the logical data model to which the physical table belongs; different physical tables in any category have the same logical structure;

[0009] Metadata is created for each category. The metadata of each category includes basic information and physical information. The basic information is used to describe the logical structure of the category, and the physical information is used to describe the storage conditions of each physical table belonging to the category.

[0010] Optionally, the plurality of physical tables include a first physical table physicalized from a logical data model and a second physical table that does not belong to any logical data model;

[0011] The classifying the plurality of physical tables according to the logical structure of each physical table to obtain at least one category includes:

[0012] Classify all first physical tables in the plurality of physical tables to obtain at least one category; different first physical tables in any category are obtained by physicalizing the same logical data model;

[0013] For a second physical table among the plurality of physical tables, it is detected whether a target category has a data structure identical to that of the second physical table; if so, the second physical table is added to the target category; if not, a new category is constructed based on the second physical table.

[0014] Optionally, for any category, if the category includes a first physical table physicalized from the same logical data model, the data structure of the category is determined according to the data structure of the first physical table physicalized from the same logical data model.

[0015] Optionally, if the data structures of different first physical tables obtained by physicalizing the same logical data model differ due to different deployment algorithms of different databases, after counting the number of first physical tables with the same data structure, the data structure of the category is determined based on the data structure of the first physical table with the highest number;

[0016] The physical information in the metadata includes difference information of the data structures of the different first physical tables.

[0017] Optionally, the same data structures of different physical tables in any category indicate that the field names, field types and / or field descriptions of the different physical tables are the same.

[0018] Optionally, the plurality of physical tables include a first physical table physicalized from a logical data model and a second physical table that does not belong to any logical data model;

[0019] All second physical tables belonging to the same category have the same field names, field types, and field descriptions;

[0020] If the two second physical tables come from different databases and the field types of the two second physical tables satisfy a preset field type mapping relationship between different databases, the field types of the two second physical tables are the same.

[0021] Optionally, the multiple physical tables are obtained from the same database; or,

[0022] The plurality of physical tables are obtained from at least two databases; wherein, if the at least two databases are deployed in different cloud environments; the metadata also includes environment information, and the environment information is used to describe the cloud environment in which each physical table in the category is located.

[0023] Optionally, also include:

[0024] In response to an operation of creating a new physical table in the database, searching, based on a logical structure of the new physical table, whether there is a category matching the new physical table;

[0025] If so, physical information indicating the storage status of the newly created physical table is added to the metadata of the matching category;

[0026] If not, a new category is constructed according to the newly created physical table, and metadata corresponding to the category is generated.

[0027] Optionally, the physical information includes at least one of the following: field information, statistical information, lineage relationship, and quality information of the physical table;

[0028] The statistical information includes at least one of the following: storage space of the physical table, access popularity, computing tasks generating the physical table, and resource consumption information; the statistical information is periodically updated according to a preset period.

[0029] According to a second aspect of one or more embodiments of this specification, a data query method is proposed, including:

[0030] In response to a data query request, determining a candidate physical table storing the data indicated by the data query request based on metadata; wherein the metadata is generated based on the metadata generation method according to any one of the first aspects;

[0031] A target physical table is selected from the candidate physical tables, and the data query request is sent to a target database engine corresponding to the target physical table, so that the target database engine queries the target physical table to obtain a data query result.

[0032] Optionally, the target physical table is selected from the candidate physical tables based on at least one of the following factors:

[0033] The real-time requirement of the data query request, the remaining bandwidth of the database node where each candidate physical table is located, and the query plan returned after the data query request is sent to the database engine corresponding to each candidate physical table; the query calculation indicates the query overhead of the data query request.

[0034] According to a third aspect of one or more embodiments of this specification, a database system is proposed, including a metadata generation device, a data query device, and at least one database; the plurality of physical tables are obtained from the database;

[0035] The metadata generating device is configured to execute the method described in any one of the first aspects;

[0036] The data query method is used to execute the method described in any one of the second aspects.

[0037] According to a fourth aspect of the embodiments of this specification, there is provided an electronic device, including:

[0038] processor;

[0039] a memory for storing processor-executable instructions;

[0040] Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0041] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.

[0042] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0043] In an embodiment of the present specification, several physical tables are classified based on the logical structure of the physical tables to obtain at least one category. Different physical tables in any category have the same logical structure, and metadata is generated for each category to achieve the same metadata corresponding to different physical tables with the same logical structure, thereby enabling more efficient and convenient management of data from the same source.

[0044] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of the physicalization of a logical data model provided by an exemplary embodiment.

[0046] Figure 2 and Figure 3 2 are different schematic diagrams of a database system provided by an exemplary embodiment.

[0047] Figure 4 This is a flowchart of a metadata generation method provided by an exemplary embodiment.

[0048] Figure 5 and Figure 6are different schematic diagrams of metadata provided by an exemplary embodiment.

[0049] Figure 7 It is a schematic diagram of the field type mapping relationship between different databases provided by an exemplary embodiment.

[0050] Figure 8 This is a flowchart of a data query method provided by an exemplary embodiment.

[0051] Figure 9 It is a structural diagram of a device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0053] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0054] Here is an explanation of the relevant terms:

[0055] Logical Data Model: The logical data model is an important fundamental model in database technology. It sits between physical data storage and user experience, representing the mapping between database information and the relational model. It supports user-defined, more general views and can be broken down into lower-level views to support mappings to specific databases, thereby extending application functionality. It allows application developers to more fully express their needs, enabling richer applications.

[0056] The most significant feature of a database's logical model is that it separates data from physical storage, providing a convenient way to implement complex relationships. The logical data model consists of three basic structures: entities, relationships, and attributes. An entity represents an independent physical object that can be created or modified. It can include one or more concepts or elements and can define relationships between entities. A relationship is a mapping relationship that defines the connection between different entities, allowing the database to connect and query data. Attributes are descriptions of entities, defining their characteristics and features, as well as the properties possessed by a specific entity.

[0057] In related database technologies, metadata is defined based on a single physical table, meaning each physical table corresponds to a piece of metadata. However, multiple physical tables stored in one or more databases may originate from the same source. Defining metadata based on a single physical table results in overly fragmented metadata, severing the connections between data. In other words, metadata makes it impossible to determine whether physical tables are actually related, making it impossible to uniformly analyze data distribution and storage based on metadata. Furthermore, when there are a large number of physical tables originating from the same source, defining metadata based on a single physical table can result in an excessive amount of metadata. For example, searching by table name may yield multiple pieces of metadata, making it difficult to locate the actual physical table.

[0058] For example, in a database (e.g., ADB, Analytic Database, or PostgreSQL), each partition is a physical table. Multiple partitions of the same table may be assigned different metadata, disrupting the data connection. Another example is that different temporary tables generated by periodically running the same algorithm may be assigned different metadata, also disrupting the data connection.

[0059] In another example, there is a need to store the same data in multiple databases in related technologies. For example, if there are online query requirements and offline analysis requirements for the same data, the data will usually be stored in the database that provides better query capabilities to meet the online query requirements; and the data will also be synchronized to the database that provides better analysis capabilities to meet the offline analysis requirements. For another example, the same logical data model may support multiple database storage and will be physicalized in different databases. Heterogeneous storage often exists. Please refer to Figure 1The same logical data model is physicalized in three databases: MaxCompute (a big data computing service), RDS for MySQL (a relational database management system), and PostgreSQL (an open source relational database management system). Metadata is defined for the physical table in each of these three databases, and these three sets of metadata are independent of each other. This breaks the connection between the data, making it impossible to determine from the metadata whether the physical tables are actually related, and making it impossible to uniformly analyze data distribution and storage based on the metadata.

[0060] In response to the problems in the related art, an embodiment of this specification provides a metadata generation method, which can obtain several physical tables to be processed from a database; then, according to the logical structure of each of the physical tables, the several physical tables are classified to obtain at least one category; wherein, the logical structure is determined according to the data structure of the physical table and / or the logical data model to which the physical table belongs; different physical tables in any category have the same logical structure; finally, metadata is created for each category; wherein, the metadata of each category includes basic information and physical information, wherein the basic information is used to describe the logical structure of the category, and the physical information is used to describe the storage status of each physical table belonging to the category. This embodiment implements different physical tables with the same logical structure corresponding to the same metadata, thereby enabling more efficient and convenient management of data from the same source.

[0061] In some embodiments, the metadata generation method provided in the embodiments of the present application can be performed by an electronic device, including but not limited to a server, a cloud server, a smart phone / mobile phone, a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer, or any other device with computing capabilities.

[0062] Exemplarily, the electronic device includes a processor and a memory, the memory stores executable instructions that can be run on the processor, and when the processor executes the executable instructions, the metadata generation method and data query method provided in the embodiments of this specification are implemented.

[0063] Exemplarily, the electronic device integrates a computer program product, and when the electronic device executes the computer program product, the metadata generation method provided in the embodiments of this specification is implemented.

[0064] In some embodiments, see Figure 2 and Figure 3 , Figure 2 and Figure 3A database system is shown, comprising a metadata generation device 10, a data query device 20, and at least one database 30. For example, the metadata generation device 10 and the data query device 20 can be integrated into the above-mentioned electronic device; it is understandable that the metadata generation device 10 and the data query device 20 can be installed in the same device or in different devices, and this embodiment does not impose any restrictions on this. For example, Figure 2 As shown, the electronic devices can be connected to the respective databases 30 for communication; or, as shown Figure 3 As shown, the electronic device can be connected to a database 30 with a central scheduling function, and the database 30 with a central scheduling function can be connected to other databases; this embodiment does not impose any restrictions on the communication connection method between devices, and specific settings can be made according to actual application scenarios.

[0065] See also Figure 2 and Figure 3 The metadata generation device 10 can obtain several physical tables to be processed from at least one database, classify the physical tables by executing the metadata generation method provided in the embodiments of this specification, and create metadata for each category. This allows different physical tables from the same source to correspond to the same metadata, thereby enabling more efficient and convenient management of data from the same source. The data query device 20 can perform data queries based on the metadata generated by the metadata generation device.

[0066] Next, the metadata generation method provided in the embodiment of this specification is exemplarily described: Figure 4 , Figure 4 A flowchart of a metadata generation method is shown. The method can be performed by a metadata generation device (hereinafter referred to as the device), and the method includes:

[0067] In S101 , several physical tables to be processed are obtained from at least one database.

[0068] In S102, the plurality of physical tables are classified according to the logical structure of each physical table to obtain at least one category; wherein the logical structure is determined according to the data structure of the physical table and / or the logical data model to which the physical table belongs; and different physical tables in any category have the same logical structure.

[0069] In S103 , metadata is created for each category. The metadata of each category includes basic information and physical information. The basic information is used to describe the logical structure of the category, and the physical information is used to describe the storage conditions of each physical table belonging to the category.

[0070] This embodiment implements classification based on the logical structure of the physical table and generates metadata for each category, so that different physical tables with the same logical structure correspond to the same metadata, thereby enabling more efficient and convenient management of data from the same source.

[0071] In some embodiments, based on actual scenario needs, the device can obtain multiple physical tables to be processed from the same database; it can also obtain multiple physical tables to be processed from at least two databases. The specific database can be specified by the user, for example, a selection page can be provided for the user to select the database to be processed from multiple databases displayed on the selection page. Exemplarily, the multiple physical tables can be all physical tables stored in the database, or they can be a portion of the physical tables stored in a database specified by the user. This embodiment does not impose any restrictions on this, and specific settings can be made based on the actual application scenario.

[0072] In some embodiments, see Figure 5 , Figure 5 The schematic diagram of metadata corresponding to each category is shown. The metadata of each category includes basic information and physical information. The basic information is used to describe the logical structure of the category. For example, see Figure 5 The basic information may include metadata unique identifier, metadata name, data structure information (such as field name, field type and field description in Table 5, etc.) and logical data model information (such as model name, model attribute information in Table 5, etc.).

[0073] The physical information is used to describe the storage conditions of each physical table belonging to this category. For example, see Figure 5 , the physical information includes at least one of the following: field information of the physical table ( Figure 5 Where n is an integer greater than 0), statistical information, lineage relationship and quality information of the physical table. Wherein, the statistical information includes but is not limited to at least one of the following: the storage space of the physical table, access popularity, computing tasks and resource consumption information of the output of the physical table; the statistical information can be periodically updated according to a preset period. The lineage relationship includes the upstream table of the physical table and the field relationship, etc.; the quality information is used to indicate the reliability of the data in the physical table. In addition, the physical information can also include the table name of the physical table, table description information, the database where the physical table is located (such as the unique identifier of the database), the person responsible for the table, and other information; but not limited to this. In this embodiment, different physical tables with the same logical structure correspond to the same metadata, and the statistical information of different physical tables with the same logical structure are aggregated together, so that the data value and data resource consumption of the data generated by the same source can be accurately evaluated.

[0074] In some embodiments, the device can obtain several physical tables from at least two different databases according to user needs. If at least two different databases are deployed in different cloud environments, the metadata of the same category can also include environment information, which is used to describe the cloud environment where each physical table in the category is located. For example, see Figure 6 , Figure 6 The metadata in also includes environment information, indicating the cloud environment in which different physical tables are located. This embodiment generates unified metadata across platforms and environments, thereby enabling efficient data governance.

[0075] Next, the classification process of the several physical tables is described:

[0076] In one possible implementation, assuming the logical structure is determined based on the data structure of the physical table, the device may classify the plurality of physical tables based on the data structure of each physical table to obtain at least one category, wherein different physical tables in any category have the same data structure. Exemplarily, the same data structure of different physical tables in any category indicates that different physical tables belonging to the same category have the same field names, field types, and / or field descriptions.

[0077] In another possible implementation, assuming that the logical structure is determined according to the logical data model to which the physical table belongs: the device can classify the multiple physical tables according to the logical data model to which each physical table belongs to obtain at least one category, and different physical tables in any category are physicalized by the same logical data model.

[0078] In another possible implementation, considering that some of the multiple physical tables may be physicalized from a logical data model, while others may not belong to any logical data model, this situation is described as follows: the logical structure is determined based on the data structure of the physical table and the logical data model to which the physical table belongs, and the multiple physical tables include a first physical table physicalized from a logical data model and a second physical table that does not belong to any logical data model. In the process of classifying the multiple physical tables, the device first classifies all first physical tables in the multiple physical tables to obtain at least one category, where different first physical tables in any category are physicalized from the same logical data model.

[0079] The metadata of any category includes basic information and physical information. The basic information includes the data structure information and logical data model information of the category (see Figure 5 and Figure 6For any category, if the category includes a first physical table physicalized from the same logical data model, the data structure of the category is determined according to the data structure of the first physical table physicalized from the same logical data model, that is, the data structure of the category is the data structure of the first physical table.

[0080] For example, considering that the data structures of different first physical tables obtained by physicalizing the same logical data model may differ due to different database deployment algorithms, when determining the data structure of a category, the device can count the number of first physical tables with the same data structure, and then determine the data structure of the category based on the data structure of the first physical table with the higher number. For example, assuming that Table 1, Table 2, Table 3, Table 4, and Table 5 are all physicalized from the same logical data model, Table 1, Table 2, and Table 3 all include Field A, Field B, and Field C; Table 4 and Table 5 include Field A and Field B, a comparison shows that the data structures of Table 1, Table 2, Table 3, Table 4, and Table 5 differ. The device can count the number of first physical tables with the same data structure. If there are three tables with data structures including Field A, Field B, and Field C, and two tables with data structures including Field A and Field B, the data structure of the category can be the data structure of the first physical table with the higher number, that is, {Field A, Field B, and Field C}.

[0081] Considering that the data structures of different first physical tables obtained by physicalizing the same logical data model may differ due to different database deployment algorithms, the data structure of a certain category determined in the above manner is not the same as all first physical tables under the category. Therefore, the physical information in the metadata of the category may also include the difference information of the data structures of the different first physical tables. For example, please refer to Figure 5 and Figure 6 The difference information can be reflected by the field information of the physical table recorded in the physical information, that is, the physical information corresponding to each physical table also records all the fields included in the physical table.

[0082] After classifying all first physical tables and determining the data structure of each category, the device next classifies the second physical tables among the several physical tables. For the second physical tables among the several physical tables, it is detected whether the data structure of the target category is the same as the data structure of the second physical table; if so, the second physical table is added to the target category; if not, a new category is constructed based on the second physical table.

[0083] Exemplarily, the field names, field types, and field descriptions of all second physical tables belonging to the same category are the same. The same field types here further take into account different representations in different databases. Therefore, if two second physical tables come from different databases, and the field types of the two second physical tables meet the preset field type mapping relationship between different databases, the device can also determine that the field types of the two second physical tables are the same, for example Figure 7 The field type mapping relationship between database A and database B is exemplified. For the same field name in the physical table, it is of String type in database A and of Varchar type in database B. Both are the same field type.

[0084] In some embodiments, after classifying several physical tables to determine at least one category, the device may create a table for each category. Figure 5 or Figure 6 The metadata shown may be stored in a database specified by the user, and this embodiment does not impose any limitation on this.

[0085] In some embodiments, after generating metadata corresponding to each category, if the user subsequently needs to create a new physical table in the database, the device also needs to determine whether the previously created metadata needs to be updated based on the actual situation. Exemplarily, in response to the operation of creating a new physical table in the database, the device searches for a category that matches the newly created physical table based on the logical structure of the newly created physical table; if so, physical information indicating the storage status of the newly created physical table is added to the metadata of the matching category; if not, a new category is constructed based on the newly created physical table, and metadata corresponding to the category is generated. This embodiment implements the same metadata corresponding to physical tables with the same logical structure, thereby enabling more efficient and convenient management of data from the same source.

[0086] In some embodiments, see Figure 8 , the embodiments of this specification also provide a data query method, which can be executed by a data query device, and the method includes:

[0087] In S201 , in response to a data query request, a candidate physical table storing the data indicated by the data query request is determined according to metadata; wherein the metadata is generated based on the above-mentioned metadata generation method.

[0088] In S202 , a target physical table is selected from the candidate physical tables, and the data query request is sent to a target database engine corresponding to the target physical table, so that the target database engine queries the target physical table to obtain a data query result.

[0089] In this embodiment, after receiving a data query request, relevant metadata can be determined based on the information carried in the data query request, and then data query can be performed based on the metadata. The metadata is generated based on the above-mentioned metadata generation method and can express physical tables with the same logical structure across platforms and environments; based on unified metadata, data queries can be completed very conveniently. The metadata records basic information and physical information of each physical table. Based on this information, the optimal storage can be automatically selected to obtain data.

[0090] In one possible implementation, after determining at least two candidate physical tables based on metadata, the data query device may select a target physical table from the candidate physical tables based on the real-time requirements of the data query request. For example, assume there are two candidate physical tables. Candidate physical table A is stored in a database that provides better query functionality to meet online query requirements; candidate physical table B is stored in a database that provides better analysis functionality to meet offline analysis requirements. Assuming that the data query request received by the data query device requires real-time query and result return, the data query device may send the data query request to the target database engine corresponding to candidate physical table A, so that the target database engine can query the target physical table to obtain the data query result and return it in a timely manner.

[0091] In another possible implementation, the data query device may select a target physical table from the candidate physical tables based on the remaining bandwidth of the database node where each candidate physical table is located, for example, the database node where the target physical table is located has the largest remaining bandwidth or is higher than a preset threshold.

[0092] In another possible implementation, the data query device may send the data query request to the database engine corresponding to each candidate physical table. The database engine corresponding to each candidate physical table may generate a query plan based on the data query request and return it to the data query device. The query calculation indicates the query overhead of the data query request. The data query device may then select a target physical table from the candidate physical tables based on the query plans corresponding to the candidate physical tables. For example, the query overhead indicated in the query plan of the target physical table is the smallest, or is less than a preset overhead.

[0093] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.

[0094] In some embodiments, see Figure 2 and Figure 3, an embodiment of this specification also provides a database system, including a metadata generation device, a data query device and at least one database; the plurality of physical tables are obtained from the database.

[0095] The metadata generation device is used to execute the metadata generation method described above; and the data query method is used to execute the data query method described above.

[0096] It is understandable that the embodiments of this specification do not impose any restrictions on the communication method between the metadata generation device, the data query device and the at least one database, and specific settings can be made according to actual application scenarios.

[0097] Exemplarily, the metadata generating device includes:

[0098] The physical table acquisition module is used to acquire several physical tables to be processed from at least one database.

[0099] A physical table classification module is used to classify the multiple physical tables according to the logical structure of each physical table to obtain at least one category; wherein the logical structure is determined according to the data structure of the physical table and / or the logical data model to which the physical table belongs; different physical tables in any category have the same logical structure.

[0100] The metadata generation module is used to create metadata for each category; wherein the metadata of each category includes basic information and physical information, the basic information is used to describe the logical structure of the category, and the physical information is used to describe the storage status of each physical table belonging to the category.

[0101] Exemplarily, the data query device includes:

[0102] The candidate physical table determination module is configured to determine, in response to a data query request, a candidate physical table storing the data indicated by the data query request based on metadata; wherein the metadata is generated based on the metadata generation method described above.

[0103] The data query module is used to select a target physical table from the candidate physical tables and send the data query request to a target database engine corresponding to the target physical table, so that the target database engine queries the target physical table to obtain a data query result.

[0104] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0105] Figure 9 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 9, at the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 902 reading the corresponding computer program from the non-volatile memory 910 into the memory 908 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logical devices or a combination of hardware and software, etc., that is, the execution subject of the following processing flow is not limited to each logical unit, but can also be hardware or logical devices. Exemplarily, the above-mentioned metadata generation device and the above-mentioned data query device can be applied to the following Figure 9 in the device shown.

[0106] In some embodiments, embodiments of this specification further provide an electronic device comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements any of the aforementioned methods by executing the executable instructions. In some embodiments, embodiments of this specification further provide a resource scheduling system comprising: a target resource node-side scheduler corresponding to any target resource node in a resource node cluster, configured to execute the steps of any of the aforementioned methods; and a central scheduler corresponding to the resource node cluster, configured to execute the steps of any of the aforementioned methods.

[0107] In some embodiments, the embodiments of this specification further provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0108] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0109] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0110] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0111] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0112] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0113] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0114] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0115] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0116] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0117] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A metadata generation method, comprising: Obtaining a plurality of physical tables to be processed from at least one database; Classifying the plurality of physical tables according to the logical structure of each physical table to obtain at least one category; wherein the logical structure is determined according to the data structure of the physical table and / or the logical data model to which the physical table belongs; different physical tables in any category have the same logical structure; Metadata is created for each category. The metadata of each category includes basic information and physical information. The basic information is used to describe the logical structure of the category, and the physical information is used to describe the storage conditions of each physical table belonging to the category.

2. The method according to claim 1, wherein the plurality of physical tables include a first physical table physicalized from a logical data model and a second physical table that does not belong to any logical data model; The classifying the plurality of physical tables according to the logical structure of each physical table to obtain at least one category includes: Classify all first physical tables in the plurality of physical tables to obtain at least one category; different first physical tables in any category are obtained by physicalizing the same logical data model; For a second physical table among the plurality of physical tables, it is detected whether a target category has a data structure identical to that of the second physical table; if so, the second physical table is added to the target category; if not, a new category is constructed based on the second physical table.

3. The method according to claim 2, wherein for any category, if the category includes a first physical table physicalized from the same logical data model, the data structure of the category is determined according to the data structure of the first physical table physicalized from the same logical data model.

4. According to the method of claim 3, if the data structures of different first physical tables obtained by physicalizing the same logical data model differ due to different deployment algorithms of different databases, after counting the number of first physical tables with the same data structure, the data structure of the category is determined based on the data structure of the first physical table with the highest number; in, The physical information in the metadata includes difference information of the data structures of the different first physical tables.

5. The method according to any one of claims 1 to 4, wherein the same data structures of different physical tables in any category indicate that field names, field types and / or field descriptions of the different physical tables are the same.

6. The method according to claim 5, wherein the plurality of physical tables include a first physical table physicalized from a logical data model and a second physical table that does not belong to any logical data model; All second physical tables belonging to the same category have the same field names, field types, and field descriptions; in, If the two second physical tables come from different databases, and the field types of the two second physical tables satisfy a preset field type mapping relationship between different databases, the field types of the two second physical tables are the same.

7. The method according to any one of claims 1 to 4, wherein the plurality of physical tables are obtained from the same database; or The physical tables are obtained from at least two databases; wherein, If the at least two databases are deployed in different cloud environments, the metadata further includes environment information, and the environment information is used to describe the cloud environment in which each physical table in the category is located.

8. The method according to claim 1, further comprising: In response to an operation of creating a new physical table in the database, searching, based on a logical structure of the new physical table, whether there is a category matching the new physical table; If so, physical information indicating the storage status of the newly created physical table is added to the metadata of the matching category; If not, a new category is constructed according to the newly created physical table, and metadata corresponding to the category is generated.

9. The method according to claim 1, wherein the physical information comprises at least one of the following: field information, statistical information, lineage relationship, and quality information of the physical table; in, The statistical information includes at least one of the following: storage space of the physical table, access popularity, computing tasks generating the physical table, and resource consumption information; the statistical information is periodically updated according to a preset period.

10. A data query method, comprising: In response to a data query request, determining a candidate physical table storing the data indicated by the data query request based on metadata; wherein the metadata is generated based on the metadata generation method according to any one of claims 1 to 9; A target physical table is selected from the candidate physical tables, and the data query request is sent to a target database engine corresponding to the target physical table, so that the target database engine queries the target physical table to obtain a data query result.

11. The method according to claim 10, wherein the target physical table is selected from the candidate physical tables based on at least one of the following factors: The real-time requirement of the data query request, the remaining bandwidth of the database node where each candidate physical table is located, and the query plan returned after the data query request is sent to the database engine corresponding to each candidate physical table; the query calculation indicates the query overhead of the data query request.

12. A database system comprising a metadata generation device, a data query device, and at least one database; the plurality of physical tables are obtained from the database; The metadata generating device is used to perform the method according to any one of claims 1 to 9; The data query method is used to execute the method according to claim 10 or 11.

13. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 11 by running the executable instructions.

14. A computer-readable storage medium having computer instructions stored thereon, which implement the steps of the method according to any one of claims 1 to 11 when executed by a processor.

Images