A data service method and related equipment

Abstract

The present specification provides a data service method and related devices, which are applied to a metadata service system, wherein the metadata service system stores metadata models and index information corresponding to the metadata models in a distributed manner. The data service method includes: obtaining a second metadata model corresponding to a first metadata model to be updated; updating the first metadata model stored in the metadata service system to the second metadata model based on the control of a distributed lock, and synchronously updating the first index information corresponding to the first metadata model stored in the metadata service system to the second index information.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of metadata technology, and more particularly, to a data service method and related devices. Background Art

[0002] Metadata is a type of data that describes data. In an enterprise, wherever there is data, there is corresponding metadata. Only with complete and accurate metadata can we better understand the data and make full use of the value of the data. Most Internet businesses and decisions often rely on the support of data services, and with the increasingly complex business development today, there are higher requirements for the organization and application of data on different platforms.

[0003] Therefore, how to provide users with efficient and convenient data services for massive data on different platforms is an urgent problem to be solved. Summary of the invention

[0004] In view of this, one or more embodiments of this specification provide a data service method and related devices.

[0005] To achieve the above-mentioned purpose, one or more embodiments of this specification provide a data service method, which is applied to a metadata service system, wherein the metadata service system stores a metadata model and index information corresponding to the metadata model in a distributed manner; the method includes:

[0006] Acquire a second metadata model corresponding to the first metadata model to be updated;

[0007] Based on the control of the distributed lock, the first metadata model stored in the metadata service system is updated to the second metadata model, and the first index information corresponding to the first metadata model stored in the metadata service system is synchronously updated to the second index information.

[0008] In yet another illustrated embodiment, the type of database used by the metadata service system when distributing and storing the metadata model and the index information corresponding to the metadata model is a Key-value database.

[0009] In another illustrated embodiment, the controlling based on the distributed lock, updating the first metadata model stored in the metadata service system to the second metadata model, and synchronously updating the first index information corresponding to the first metadata model stored in the metadata service system to the second index information, includes:

[0010] Based on the control of the distributed lock, the second metadata model is added to the metadata service system, the first metadata model is deleted, and the second metadata model is locked to prohibit matching index information for the second metadata model; and

[0011] The second index information corresponding to the second metadata model is added in the metadata service system, the first index information corresponding to the first metadata model is deleted, and the second metadata model is unlocked to allow matching of index information for the second metadata model.

[0012] In yet another illustrated embodiment, the invention further comprises:

[0013] Based on the multiple metadata models stored in the metadata service system, a topological map corresponding to the multiple metadata models is generated through topological configuration; wherein the topological map includes multiple nodes, each node corresponds to a metadata model, and the connections between the nodes correspond to the association relationship between the metadata models.

[0014] In yet another illustrated embodiment, the invention further comprises:

[0015] Acquire a target query statement, wherein the target query statement includes a query object and a query condition; wherein the query object includes a target metadata model stored in the metadata service system, and the query condition includes one or more index information stored in the metadata service system;

[0016] Based on the target query statement, generating a target query process for the target metadata model;

[0017] Based on the target query process, a query is performed in a plurality of metadata models stored in the metadata service system to obtain a query result; the query result includes the target metadata model.

[0018] In another illustrated embodiment, the target query process for generating the target metadata model based on the target query statement includes:

[0019] Based on the target query statement, determining one or more metadata models corresponding to the one or more index information;

[0020] Based on the one or more metadata models, a target query path for the target metadata model is determined in the topology map to generate the target query process; wherein,

[0021] The target query path includes a plurality of query nodes connected in sequence, the last query node among the plurality of query nodes is the target metadata model, and the plurality of query nodes also include at least part of the one or more metadata models.

[0022] In yet another illustrated embodiment, the plurality of query nodes further include metadata models associated with the target metadata model and / or the one or more metadata models in the topology map.

[0023] In yet another illustrated embodiment, determining a target query path for the target metadata model in the topology map based on the one or more metadata models includes:

[0024] Based on the one or more metadata models, determining an original query path for the target metadata model in the topology map;

[0025] Based on a preset path optimization rule, the original query path is optimized to obtain the target query path; wherein the target query path includes fewer query nodes than the original query path.

[0026] In yet another illustrated embodiment, the path optimization rule includes recursive optimization, and the recursive optimization includes one or more of an optimization based on association deduction and an optimization based on index discrimination.

[0027] In yet another illustrated embodiment, the invention further comprises:

[0028] The target query statement is parsed to determine whether the target query statement is legal; if the target query statement is legal, a target query process for the target metadata model is generated based on the target query statement.

[0029] In another illustrated embodiment, the target query statement is a query statement in the TQL language or a query statement in the SQL language.

[0030] Accordingly, the present specification also provides a data service device, which is applied to a metadata service system, wherein the metadata service system stores metadata models and index information corresponding to the metadata models in a distributed manner; the device comprises:

[0031] A model acquisition unit, configured to acquire a second metadata model corresponding to the first metadata model to be updated;

[0032] An updating unit is used to update the first metadata model stored in the metadata service system to the second metadata model based on the control of a distributed lock, and to synchronously update the first index information corresponding to the first metadata model stored in the metadata service system to the second index information.

[0033] In yet another illustrated embodiment, the type of database used by the metadata service system when distributing and storing the metadata model and the index information corresponding to the metadata model is a Key-value database.

[0034] In yet another illustrated embodiment, the updating unit is specifically configured to:

[0035] Based on the control of the distributed lock, the second metadata model is added to the metadata service system, the first metadata model is deleted, and the second metadata model is locked to prohibit matching index information for the second metadata model; and

[0036] The second index information corresponding to the second metadata model is added in the metadata service system, the first index information corresponding to the first metadata model is deleted, and the second metadata model is unlocked to allow matching of index information for the second metadata model.

[0037] In yet another illustrated embodiment, the apparatus further comprises:

[0038] A topology configuration unit is used to generate a topology map corresponding to the multiple metadata models through topology configuration based on the multiple metadata models stored in the metadata service system; wherein the topology map includes multiple nodes, each node corresponds to a metadata model, and the connections between the nodes correspond to the association relationship between the metadata models.

[0039] In yet another illustrated embodiment, the apparatus further comprises:

[0040] A query statement acquisition unit, used to acquire a target query statement, wherein the target query statement includes a query object and a query condition; wherein the query object includes a target metadata model stored in the metadata service system, and the query condition includes one or more index information stored in the metadata service system;

[0041] A query process generating unit, used for generating a target query process for the target metadata model based on the target query statement;

[0042] The query unit is used to query multiple metadata models stored in the metadata service system based on the target query process to obtain query results; the query results include the target metadata model.

[0043] In yet another illustrated embodiment, the query process generating unit is specifically configured to:

[0044] Based on the target query statement, determining one or more metadata models corresponding to the one or more index information;

[0045] Based on the one or more metadata models, a target query path for the target metadata model is determined in the topology map to generate the target query process; wherein,

[0046] The target query path includes a plurality of query nodes connected in sequence, the last query node among the plurality of query nodes is the target metadata model, and the plurality of query nodes also include at least part of the one or more metadata models.

[0047] In yet another illustrated embodiment, the plurality of query nodes further include metadata models associated with the target metadata model and / or the one or more metadata models in the topology map.

[0048] In yet another illustrated embodiment, the query process generating unit is specifically configured to:

[0049] Based on the one or more metadata models, determining an original query path for the target metadata model in the topology map;

[0050] Based on a preset path optimization rule, the original query path is optimized to obtain the target query path; wherein the target query path includes fewer query nodes than the original query path.

[0051] In yet another illustrated embodiment, the path optimization rule includes recursive optimization, and the recursive optimization includes one or more of an optimization based on association deduction and an optimization based on index discrimination.

[0052] In yet another illustrated embodiment, the apparatus further comprises:

[0053] The parsing unit is used to parse the target query statement to determine whether the target query statement is legal; if the target query statement is legal, a target query process for the target metadata model is generated based on the target query statement.

[0054] In another illustrated embodiment, the target query statement is a query statement in the TQL language or a query statement in the SQL language.

[0055] Accordingly, the present specification also provides a computer device, comprising: a memory and a processor; the memory stores a computer program that can be executed by the processor; when the processor runs the computer program, it executes the data service method described in the above embodiments.

[0056] Accordingly, the present specification also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the data service method described in the above embodiments is executed.

[0057] In summary, the present application not only stores the metadata model in a distributed manner, but also stores the index information corresponding to the metadata model in a distributed manner, thereby constructing a metadata service system that includes both the metadata model and the index information. In this way, the present application can not only ensure the storage of massive metadata models and their corresponding index information based on the extremely strong scalability of distributed storage, but also ensure that a strong indexing capability can still be achieved for massive metadata models to meet the query needs of users. Furthermore, the present application also controls the distributed lock to keep the update of the metadata model consistent with the update of its corresponding index information, thereby ensuring the accuracy of the index and the efficiency of data query.

[0058] In addition, this application also generates a topological map corresponding to multiple metadata models contained in the metadata service system through topological configuration. The topological map contains the association relationships between multiple metadata models, providing effective support for the subsequent implementation of more flexible, convenient and efficient data query services. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a schematic diagram of the architecture of a metadata service system provided by an exemplary embodiment;

[0060] Figure 2 is a schematic diagram of the architecture of another metadata service system provided by an exemplary embodiment;

[0061] Figure 3 is a flowchart of a data service method provided by an exemplary embodiment;

[0062] Figure 4 is a data update schematic diagram provided by an exemplary embodiment;

[0063] Figure 5 is a flowchart of a data query method provided by an exemplary embodiment;

[0064] Figure 6 is a structural schematic diagram of a data service device provided by an exemplary embodiment;

[0065] Figure 7It is a structural diagram of a computer device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0066] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0067] It should be noted that in other embodiments, the corresponding methods are not necessarily executed in the order shown and described in this specification.

[0068] In some other embodiments, the method may include more or fewer steps than those described in this specification. 5 In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments;

[0069] The multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0070] In addition, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all authorized by the user or

[0071] The collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0072] First, some of the terms in this specification are explained to facilitate understanding by technicians in this field.

[0073] (1) Metadata, which is defined as "data about data". In short, anything that can be used to describe a piece of data can be considered metadata. Enterprises need to manage metadata in a scientific and effective manner and provide metadata services to developers and end users to meet their business needs. For example, metadata can be the basic data of system operation from the perspective of technical risk5. The basic data can include, for example, underlying infrastructure, application layer data, and service layer data.

[0074] Etc. The so-called technical risks may include, for example, high availability (HA), which is one of the factors that usually need to be considered in the design of distributed system architecture. It generally refers to reducing the time that the system cannot provide services through design. Assuming that the system can always provide services, the system availability is 100%. If the system cannot provide services for 1 time unit for every 100 time units of operation, the system availability is 99%.

[0075] 0(2) Distributed storage refers to storing data in multiple independent devices. Distributed storage mostly uses multiple

[0076] The storage server shares the storage load and uses the location server to locate the storage information. This not only improves the reliability, availability and access efficiency of the system, but also makes it easy to expand and can achieve data storage of hundreds of millions of levels.

[0077] As mentioned above, with the development of increasingly complex Internet businesses, the organization and application of data on different platforms have become more and more complex.

[0078] However, the commonly used non-relational database storage, such as HBase and MangoDB, can flexibly change data attributes and can store large amounts of distributed data.

[0079] However, due to its distributed storage characteristics, its data indexing capabilities are limited and it cannot provide flexible and diverse data query services.

[0080] Therefore, in order to provide flexible, efficient and accurate metadata services, this specification provides a data service method that can be applied to the metadata service system provided in this specification.

[0081] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the architecture of a metadata service system provided by an exemplary embodiment. Figure 1 As shown, the metadata service system may include storage of metadata models and index storage corresponding to the metadata models. It should be understood that a metadata model may be an instance of metadata. Further, the metadata service system may store a large number of metadata models and index information corresponding to the large number of metadata models in a distributed storage manner.

[0082] Furthermore, the type of database used by the metadata service system when distributing the metadata model and the index information corresponding to the metadata model can be a key-value database. Optionally, the metadata service system can be composed of multiple servers, so that the metadata models of each platform and their corresponding index information are distributedly stored through multiple independent servers. In this way, based on the extremely strong expansion capability of distributed storage, the storage of massive metadata models and their corresponding index information can be effectively guaranteed. In addition, as mentioned above, the present application, on the basis of distributed storage of metadata models, also distributedly stores the index information corresponding to the metadata models, which is equivalent to building an index storage layer on top of the original metadata model storage layer, thereby ensuring that strong indexing capabilities can still be achieved for massive metadata models to meet user query needs.

[0083] Optionally, the storage of the index in the present application may be a memory-based database, which can effectively ensure the speed and accuracy of reading and writing index data due to the strong read and write performance of the memory.

[0084] like Figure 1 As shown, the metadata model storage of the metadata service system may include metadata model 1, metadata model 2, metadata model 3, and the like, and the index storage may include index information 1, index information 2, index information 3, and the like. Among them, metadata model 1, metadata model 2, and metadata model 3 may be data of different platforms and may be distributed and stored in different servers. Among them, index information 1 may be the index information corresponding to metadata model 1, index information 2 may be the index information corresponding to metadata model 2, and the like. Optionally, the index information corresponding to each metadata model may include multiple index conditions, and the metadata model may be queried through any one of the multiple index conditions. For example, index information 1 may include multiple index conditions such as age 25, unmarried, male, etc., index information 2 may include multiple index conditions such as age 27, married, male, etc., index information 3 may include multiple index conditions such as employee number X456, length of service 5 years, etc., and the examples are not further elaborated here.

[0085] For further information, see Figure 2 , Figure 2 FIG. 1 is a schematic diagram of another metadata service system provided by an exemplary embodiment. It should be noted that: Figure 2 The architecture shown is a software-level architecture, which is mainly used to represent the various configurations, features, and capabilities of the system. Figure 2 As shown, the metadata service system may include a configuration layer, a service layer, and a storage layer.

[0086] like Figure 2As shown, the configuration layer provides the ability to configure the metadata model and topology view. This application uses a configuration language to describe the metadata model and topology map, so that the metadata model and topology map can be flexibly and dynamically changed, and realizes the ability to quickly build and hot publish the metadata model, thereby ensuring the flexibility and efficiency of the overall metadata topology organization. Figure 1 The metadata service system shown, the so-called topology view configuration capability may include automatically inferring the association relationship between the multiple metadata models through topology configuration based on the multiple metadata models (e.g., metadata model 1, metadata model 2, metadata model 3, etc.) stored in the metadata service system, thereby generating a topology map corresponding to the multiple metadata models. The topology map may include multiple nodes, each node corresponds to a metadata model, and the lines between the nodes may correspond to the association relationship between the metadata models.

[0087] like Figure 2 As shown, the metadata model configuration may include multiple configuration items such as configuration item 1, configuration item 2, and configuration item 3. Exemplarily, the multiple configuration items may include model domain, field configuration, model function, field mapping, pipeline configuration, life cycle (Time To Live, TTL) configuration, etc., which are not specifically limited in this specification. Figure 2 As shown, the topology view configuration may include multiple configuration items such as configuration item 4, configuration item 5, and configuration item 6. Exemplarily, the multiple configuration items may specifically include a model list, association relationship, association weight, and view generation, etc., which is not specifically limited in this specification.

[0088] like Figure 2 As shown in the figure, the service layer mainly provides unified data services based on the underlying storage capacity and model configuration, such as changes to the metadata model, changes to index information, and consistency assurance of the changes of the two data. Optionally, the consistency of changes to the metadata model and its corresponding index information can be guaranteed through the control of distributed locks, thereby effectively avoiding the problem that the metadata model cannot correspond to the accurate index within a period of time (such as 20 minutes or 30 minutes, etc.), and the user cannot get accurate data when entering an index. This will not be elaborated here, and please refer to the subsequent Figure 3 The description of the corresponding embodiment. Figure 2 As shown, the service layer may include multiple service items such as service item 1, service item 2, service item 3, and service item 4. Exemplarily, the multiple service items may include data initialization, multi-channel incremental subscription, service routing, model change message, distributed lock management, index construction, topology service, sandbox environment, verification task, and change identification, etc. This specification does not specifically limit this.

[0089] like Figure 2As shown in Figure 1, the storage layer mainly includes metadata model storage and index storage. Figure 2 As shown, the metadata model storage may include multiple features such as feature 1, feature 2, and feature 3, which describe the underlying storage capabilities of the present application, or the logical characteristics of the storage. For example, the multiple features may include weak schema, version control, and dynamic columns, etc., which are not specifically limited in this specification. Figure 2 As shown, the index storage corresponding to the metadata model storage may also include multiple features such as feature 4, feature 5 and feature 6. Similarly, these multiple features describe the underlying storage capabilities of this application, or the logical characteristics of the storage. Exemplarily, these multiple features may include index configuration, index segmentation, concurrent updates, hotspot identification, performance monitoring and master-slave synchronization, etc. This specification does not specifically limit this. Figure 2 As shown, the data consistency between the metadata model storage and its corresponding index storage can be guaranteed through the control of distributed locks, that is, the consistency of changes to the metadata model and its index can be guaranteed.

[0090] Based on the above Figure 1 and Figure 2 The metadata service system described herein also provides a metadata service method. Figure 3 , Figure 3 is a flowchart of a data service method provided by an exemplary embodiment. The method can be applied to Figure 1 and Figure 2 The system architecture shown can be specifically applied to a computer device, which can be, for example, a laptop computer, a desktop computer, a server, a vehicle-mounted computer, etc., and this specification does not specifically limit this. Optionally, the computer device can be a device in the metadata service system, which can store part of the metadata model or index information, or the computer device can be a device independent of the metadata service system, and the computer device can establish a communication connection with the system via wired or wireless means. Figure 3 As shown, the method may specifically include the following steps S101 to S102.

[0091] Step S101: Acquire a second metadata model corresponding to the first metadata model to be updated.

[0092] Specifically, a second metadata model corresponding to the first metadata model to be updated is obtained, wherein the first metadata model is a metadata model originally stored in the metadata service system and currently needs to be updated, and the second metadata model is a metadata model after the first metadata model is updated.

[0093] Optionally, the computer device may directly obtain the second metadata model locally, or may obtain the second metadata model from an external device via a wireless method such as a network connection.

[0094] Step S102: based on the control of the distributed lock, the first metadata model stored in the metadata service system is updated to the second metadata model, and the first index information corresponding to the first metadata model stored in the metadata service system is synchronously updated to the second index information.

[0095] Specifically, based on the control of the distributed lock, the second metadata model is added to the metadata service system, the first metadata model is deleted, and the second metadata model is locked to prohibit matching index information for the second metadata model, so that the second metadata model cannot be indexed. Correspondingly, based on the control of the distributed lock, the second index information corresponding to the second metadata model is added to the metadata service system, the first index information corresponding to the first metadata model is deleted, and the second metadata model is unlocked to allow matching index information for the second metadata model, so that the second metadata model can be successfully indexed by the second index information.

[0096] In this way, before the index information corresponding to the updated metadata model is updated, the updated metadata model is always in a locked state. Only when the index information is updated, the updated metadata model can take effect normally. This achieves the synchronous construction and update of the metadata model and its index information, ensures the data consistency of the metadata model and its index information, guarantees the accuracy of subsequent data query services, and meets the business needs of users.

[0097] For example, see Figure 4 , Figure 4 FIG. 1 is a schematic diagram of data updating provided by an exemplary embodiment. Figure 4 As shown, the second metadata model is, for example, R (i.e., updated data (after-data)), and the first metadata model is, for example, R' (i.e., updated data (before-data)). Figure 4 As shown, when a data update instance Record-R enters the metadata service system, the metadata model R' originally stored in the metadata service system can be taken out first, and then the new metadata model R can be stored in the metadata service system.

[0098] like Figure 4As shown in the figure, for the original metadata model R', the content of its F0 field is X, and the content of its F1 field is M. When the metadata model R' is updated to the metadata model R, the content of its F0 field is updated to Y, and the content of its F1 field is updated to N. Among them, the field F0 and the field F1 can be used to express an attribute of the metadata model, such as its work number, time node, product category, etc. Figure 4 As shown, for the original metadata model R',

[0099] The index information may include F0=X, F1=M, etc., that is, the user may query the metadata model R' by inputting index conditions such as F0=X, F1=M, etc. Figure 4 As shown, for metadata model R, its index information may include F0=Y, F1=N, etc.

[0100] Index conditions, that is, the user can query the metadata model R by entering index conditions such as F0=Y, F1=N, etc.

[0101] like Figure 4 As shown, the present application locks the metadata model R after storing the new metadata model R in the metadata service system through the control of the distributed lock. Then, insert a new index into the index storage, for example, including inserting F0=Y

[0102] The index of metadata model R, and the index of metadata model R by F1=N, and so on, and the index of old 0 is deleted in the index storage, for example, including deleting the index of metadata model R' by F0=X, and the index of metadata model R' by F1=M,

[0103] Etc. Finally, the metadata model R is unlocked to achieve the consistency of the metadata model and its corresponding index information updates, thereby ensuring the accuracy of subsequent data queries.

[0104] like Figure 4 As shown in the figure, after completing the update of the metadata model and index information, the user enters the query condition F0=Y to perform the data

[0105] According to the query, the metadata model R can be queried. At this time, if the user enters the query condition F0=X, no 5 metadata model can be queried, which ensures the accuracy of data query.

[0106] Furthermore, based on the above metadata service system, in order to improve the user's data query service, this specification also provides a data query protocol. Figure 5 , Figure 5 FIG. 1 is a flow chart of a data query method provided by an exemplary embodiment. Figure 5As shown, the method includes the following steps S11 to S16.

[0107] 0 Step S11, obtaining a target query statement.

[0108] Specifically, the computer device obtains the target query statement, and the language used by the target query statement can be the topological query language (topo query language, TQL) of the present application, or the conventional structured query language (Structured Query Language, SQL), or the SQL dialect or any other possible language, etc., which is not specifically limited in this specification.

[0109] 5 Optionally, the target query statement may include a query object and a query condition, wherein the query object may include

[0110] The target metadata model stored in the metadata service system (for example, Figure 1 The metadata model 1 shown, or Figure 4 The query condition may include one or more index information stored in the metadata service system (e.g. Figure 1 The one or more index information

[0111] It may include the index information corresponding to the target metadata model (ie, the query object), or the one or more index information may also be completely the index information corresponding to other metadata models, which is not specifically limited in this specification.

[0112] like Figure 5 As shown, in the case of using TQL language, the target query statement can be "Query: A JOINDCondition: Ax=$0&&B.y=$1". In the target query statement, the query objects are A and D, and the query condition is Ax=$0&&B.y=$1, that is, the x field of A is $0, and the y field of B is $1.

[0113] like Figure 5 As shown, in the case of using SQL language, the target query statement can be "select from A, D where Ax = $0 and By = $1". In this target query statement, the query objects are also A and D, and the query conditions also include the two index information that the x field of A is $0 and the y field of B is $1.

[0114] It should be understood that TQL is the query language defined in this application, but considering that most users are more familiar with the traditional SQL language, in order to reduce the access cost, this application also transplants and adapts the SQL language, mainly by introducing Figure 5 The compatible tool calcite shown in the figure enables the target query statement in the SQL language to finally generate a query process that conforms to the logic of this application. In this way, it is ensured that users can choose any language to perform data query according to their actual needs, thereby improving the user experience.

[0115] Step S12, parsing and verifying the target query statement.

[0116] Specifically, the target query statement is parsed to verify whether the target query statement is legal. If the target query statement is legal, step S13 is executed. If the target query statement is illegal, the query can be stopped and the user is prompted.

[0117] Alternatively, if Figure 5 As shown, if the user uses the traditional SQL language, the target query statement can be parsed through Calcite and its legitimacy can be verified.

[0118] Optionally, by parsing the target query statement, it is also possible to check whether there is redundant and repeated content in the statement, thereby improving data query efficiency.

[0119] Step S13, generating an original query flow based on the target query statement.

[0120] Specifically, based on the target query statement, a corresponding original query process, or a query plan, is generated.

[0121] Optionally, based on the target query statement, one or more metadata models corresponding to one or more index information included in the target query statement are determined. Then, based on the one or more metadata models, an original query path for the target metadata model is determined in a pre-configured topology map to generate the original query process.

[0122] The original query path is equivalent to a part of the topological map, and the original query path may include multiple query nodes connected in sequence, each query node being a corresponding metadata model. Optionally, the last query node among the multiple query nodes may be the target metadata model, and the multiple query nodes may also include at least part of the one or more metadata models corresponding to the one or more index information. Optionally, the multiple query nodes also include metadata models associated with the target metadata model and / or the one or more metadata models in the topological map.

[0123] For example, take the target query statement "Query: A JOIN D Condition: Ax = $0 && B.y = $1" as an example. Figure 5 As shown, the query path may include metadata model A, metadata model B, metadata model C and metadata model D, wherein metadata model A is metadata model A that satisfies the condition Ax=$0, and metadata model B is metadata model B that satisfies the condition By=$1. Figure 5 As shown in , metadata model D is associated with metadata model B through metadata model C. Figure 5 As shown, under this query path, after querying metadata model A that satisfies the condition Ax=$0 and metadata model B that satisfies the condition By=$1, metadata model D can be finally queried through metadata model C associated with metadata model B.

[0124] Alternatively, if Figure 5 As shown, the present application has transplanted and adapted the SQL language, and through the compatible tool calcite, the target query statement under the SQL language can also generate the original query process.

[0125] Step S14, optimizing the original query process to obtain a target query process.

[0126] Specifically, based on the preset path optimization rule, the original query path in the original query process is optimized to obtain the target query path, so as to optimize the original query process to obtain the target query process. The target query path may include fewer query nodes than the original query path.

[0127] like Figure 5 As shown, the path optimization rules may include recursive optimization, and the recursive optimization may include one or more of optimization based on association derivation and optimization based on index discrimination, and may also include any other possible optimization methods, which are not specifically limited in this specification.

[0128] For example, if the query object only includes metadata model D, and the query conditions are still Ax=$0 and By=$1, then when metadata model A and metadata model B have a certain field in common (for example, An=Bm=$2), metadata model A can be optimized and the query starts directly from metadata model B, that is, the target query path obtained after optimization only includes the three query nodes of metadata model B, metadata model C and metadata model D. Furthermore, if metadata model B and metadata model C also have a certain field in common (for example, Bk=Ct=$3), metadata model B can be further optimized and the query starts directly from metadata model C, that is, the target query path obtained after optimization only includes the two query nodes of metadata model C and metadata model D, thereby further shortening the query path, simplifying the query process, shortening the query time, and greatly improving the data query efficiency.

[0129] Step S15, generating a corresponding physical query plan.

[0130] Specifically, a physical query plan corresponding to the target query process is generated.

[0131] like Figure 5 As shown in FIG. 1 , after obtaining metadata model A (for example, including A0, A1, A2) that satisfies the condition Ax=$0 and metadata model B (for example, including B0 and B1) that satisfies the condition By=$1, metadata model C associated with metadata model B that satisfies the condition By=$1 can be further limited. For example, Figure 5 As shown in the figure, the metadata models C associated with the metadata model B that satisfies the condition By=$1 all satisfy the condition Cz=$0. It should be understood that there may be multiple metadata models C, such as C1, C2, C3, C4, etc., but the metadata models C that satisfy the condition Cz=$0 may only include C1 and C2. In this way, the final physical query plan only needs to query the metadata models C that satisfy the condition Cz=$0, that is, Figure 5 As shown in “Query C with Condition Cz=$0”, further, it is sufficient to query the metadata model D associated with the metadata model C that satisfies the condition Cz=$0.

[0132] Step S16, execute the physical query plan and return the query result.

[0133] Specifically, the computer device executes the physical query plan, performs a query in the metadata service system, and returns a query result. The query result may include the target metadata model included in the target query statement. Optionally, the query result may also include one or more metadata models associated with the target metadata model.

[0134] For example, Figure 5 As shown, the computer device executes "index_key_c_z_{$0}" to query the metadata model C that meets the condition of Cz=$0 in the metadata model storage, and finally obtains metadata models C1 and C2. Then, based on the metadata models C1 and C2, the metadata models D0, D1, and D2 associated with the metadata models C1 and C2 are obtained. Figure 5 As shown, the query results finally returned may include metadata models A0, A1, A2, D0, D1 and D2, and the associations between them, for example, A0 is associated with D0 through B0 and C1, A1 is associated with D1 through B1 and C2, A1 is also associated with D2 through B1 and C2, and A2 is associated with D0 through B0 and C1. Exemplarily, the metadata models A0, A1, A2 may be buyer information, the metadata models D0, D1, D2 may be product information, the metadata models B0 and B1 may be merchant information, the metadata models C1 and C2 may be purchase time information, etc., which are not specifically limited in this specification.

[0135] As described above, the data query implemented by the present application is very flexible and efficient, and does not impose strict requirements on the query conditions. For example, the query conditions may not be the index information of the query object. In this case, the present application can also be based on the pre-configured topological view, through the association relationship between metadata models, to quickly and efficiently query the metadata model that the user wants to query, thereby meeting the user's actual business needs.

[0136] In summary, the present application not only stores the metadata model in a distributed manner, but also stores the index information corresponding to the metadata model in a distributed manner, thereby constructing a metadata service system that includes both the metadata model and the index information. In this way, the present application can not only ensure the storage of massive metadata models and their corresponding index information based on the extremely strong scalability of distributed storage, but also ensure that a strong indexing capability can still be achieved for massive metadata models to meet the query needs of users. Furthermore, the present application also controls the distributed lock to keep the update of the metadata model consistent with the update of its corresponding index information, thereby ensuring the accuracy of the index and the efficiency of data query.

[0137] In addition, this application also generates a topology map corresponding to multiple metadata models contained in the metadata service system through topology configuration. The topology map contains the association relationship between multiple metadata models, which provides effective support for the subsequent implementation of more flexible, convenient and efficient data query services. Thus, users are provided with flexible, efficient, accurate and low-cost metadata topology services.

[0138] Corresponding to the implementation of the above method flow, the embodiment of this specification also provides a data service device, which is applied to a metadata service system, wherein the metadata service system stores metadata models and index information corresponding to the metadata models in a distributed manner. Figure 6 , Figure 6 It is a structural diagram of a data service device provided by an exemplary embodiment.

[0139] like Figure 6 As shown, the device 300 includes:

[0140] The model acquisition unit 301 is used to acquire a second metadata model corresponding to the first metadata model to be updated;

[0141] The updating unit 302 is used to update the first metadata model stored in the metadata service system to the second metadata model based on the control of the distributed lock, and to synchronously update the first index information corresponding to the first metadata model stored in the metadata service system to the second index information.

[0142] In yet another illustrated embodiment, the type of database used by the metadata service system when distributing and storing the metadata model and the index information corresponding to the metadata model is a Key-value database.

[0143] In another illustrated embodiment, the updating unit 302 is specifically configured to:

[0144] Based on the control of the distributed lock, the second metadata model is added to the metadata service system, the first metadata model is deleted, and the second metadata model is locked to prohibit matching index information for the second metadata model; and

[0145] The second index information corresponding to the second metadata model is added in the metadata service system, the first index information corresponding to the first metadata model is deleted, and the second metadata model is unlocked to allow matching of index information for the second metadata model.

[0146] In yet another illustrated embodiment, the apparatus further comprises:

[0147] The topology configuration unit 303 is used to generate a topology map corresponding to the multiple metadata models through topology configuration based on the multiple metadata models stored in the metadata service system; wherein the topology map includes multiple nodes, each node corresponds to a metadata model, and the connections between the nodes correspond to the association relationship between the metadata models.

[0148] In yet another illustrated embodiment, the apparatus further comprises:

[0149] The query statement acquisition unit 304 is used to acquire a target query statement, wherein the target query statement includes a query object and a query condition; wherein the query object includes a target metadata model stored in the metadata service system, and the query condition includes one or more index information stored in the metadata service system;

[0150] A query process generating unit 306, configured to generate a target query process for the target metadata model based on the target query statement;

[0151] The query unit 307 is used to query multiple metadata models stored in the metadata service system based on the target query process to obtain a query result; the query result includes the target metadata model.

[0152] In another illustrated embodiment, the query process generating unit 306 is specifically configured to:

[0153] Based on the target query statement, determining one or more metadata models corresponding to the one or more index information;

[0154] Based on the one or more metadata models, a target query path for the target metadata model is determined in the topology map to generate the target query process; wherein,

[0155] The target query path includes a plurality of query nodes connected in sequence, the last query node among the plurality of query nodes is the target metadata model, and the plurality of query nodes also include at least part of the one or more metadata models.

[0156] In yet another illustrated embodiment, the plurality of query nodes further include metadata models associated with the target metadata model and / or the one or more metadata models in the topology map.

[0157] In another illustrated embodiment, the query process generating unit 306 is specifically configured to:

[0158] Based on the one or more metadata models, determining an original query path for the target metadata model in the topology map;

[0159] Based on a preset path optimization rule, the original query path is optimized to obtain the target query path; wherein the target query path includes fewer query nodes than the original query path.

[0160] In yet another illustrated embodiment, the path optimization rule includes recursive optimization, and the recursive optimization includes one or more of an optimization based on association deduction and an optimization based on index discrimination.

[0161] In yet another illustrated embodiment, the apparatus further comprises:

[0162] The parsing unit 305 is used to parse the target query statement to determine whether the target query statement is legal; if the target query statement is legal, a target query process for the target metadata model is generated based on the target query statement.

[0163] In another illustrated embodiment, the target query statement is a query statement in the TQL language or a query statement in the SQL language.

[0164] The implementation process of the functions and effects of each unit in the above device 300 is detailed in the above Figure 1-Figure 5 The description of the corresponding embodiment will not be repeated here. It should be understood that the above-mentioned device 300 can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the CPU (Central Process Unit) of the device where it is located reading the corresponding computer program instructions into the memory and running them. From the hardware level, in addition to the CPU and memory, the device where the above-mentioned device is located usually also includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards and cards for realizing network communication functions.

[0165] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the units or modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Those of ordinary skill in the art may understand and implement it without creative work.

[0166] The devices, units, and modules described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0167] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer device. Figure 7 , Figure 7 FIG. 1 is a schematic diagram of a computer device provided by an exemplary embodiment. Figure 7 As shown, the computer device 1000 includes a processor 1001 and a memory 1002, and may further include an input device 1004 (such as a keyboard, etc.) and an output device 1005 (such as a display, etc.). The processor 1001, the memory 1002, the input device 1004, and the output device 1005 may be connected via a bus or other means. Figure 7 As shown, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program that can be run by the processor 1001. The processor 1001 can be a general-purpose central processing unit, a microprocessor, or an integrated circuit for controlling the execution of the above method embodiment. When the processor 1001 runs the stored computer program, it can execute the various steps of the data service method in the embodiment of this specification, including: obtaining a second metadata model corresponding to the first metadata model to be updated; based on the control of a distributed lock, updating the first metadata model stored in the metadata service system to the second metadata model, and synchronously updating the first index information corresponding to the first metadata model stored in the metadata service system to the second index information, etc. For a detailed description of the various steps of the above-mentioned data service method, please refer to the previous content, which will not be repeated here.

[0168] Corresponding to the above method embodiment, the embodiment of this specification also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by the processor, it executes each step of the data service method in the embodiment of this specification. Figure 1-Figure 5 The description of the corresponding embodiments will not be repeated here.

[0169] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

[0170] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0171] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0172] Computer readable media include permanent and non-permanent, removable and non-removable media and can store information by any method or technology. The information can be computer readable instructions, data structures, program modules or other data.

[0173] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0174] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0175] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of complete hardware embodiments, complete software embodiments or embodiments combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Claims

1. A data service method, applied to a metadata service system, wherein the metadata service system stores a metadata model and index information corresponding to the metadata model in a distributed manner; The method comprises: Acquire a second metadata model corresponding to the first metadata model to be updated; Based on the control of the distributed lock, adding the second metadata model to the metadata service system, deleting the first metadata model, and locking the second metadata model; and In the metadata service system, second index information corresponding to the second metadata model is added, first index information corresponding to the first metadata model is deleted, and the second metadata model is unlocked.

2. According to the method described in claim 1, the type of database used by the metadata service system when distributing and storing the metadata model and the index information corresponding to the metadata model is a Key-value database.

3. The method according to claim 1, further comprising: Based on the multiple metadata models stored in the metadata service system, a topological map corresponding to the multiple metadata models is generated through topological configuration; wherein the topological map includes multiple nodes, each node corresponds to a metadata model, and the connections between the nodes correspond to the association relationship between the metadata models.

4. The method according to claim 3, further comprising: Acquire a target query statement, wherein the target query statement includes a query object and a query condition; wherein the query object includes a target metadata model stored in the metadata service system, and the query condition includes one or more index information stored in the metadata service system; Based on the target query statement, generating a target query process for the target metadata model; Based on the target query process, a query is performed in a plurality of metadata models stored in the metadata service system to obtain a query result; the query result includes the target metadata model.

5. The method according to claim 4, wherein generating a target query process for the target metadata model based on the target query statement comprises: Based on the target query statement, determining one or more metadata models corresponding to the one or more index information; Based on the one or more metadata models, a target query path for the target metadata model is determined in the topology map to generate the target query process; wherein, The target query path includes a plurality of query nodes connected in sequence, the last query node among the plurality of query nodes is the target metadata model, and the plurality of query nodes also include at least part of the one or more metadata models.

6. The method according to claim 5, wherein the plurality of query nodes further comprises metadata models associated with the target metadata model and / or the one or more metadata models in the topology map.

7. The method according to claim 5, wherein determining a target query path for the target metadata model in the topology map based on the one or more metadata models comprises: Based on the one or more metadata models, determining an original query path for the target metadata model in the topology map; Based on a preset path optimization rule, the original query path is optimized to obtain the target query path; wherein the target query path includes fewer query nodes than the original query path. 8 . The method according to claim 7 , wherein the path optimization rule comprises recursive optimization, and the recursive optimization comprises one or more of an optimization based on association deduction and an optimization based on index discrimination.

9. The method according to claim 4, further comprising: Parsing the target query statement to determine whether the target query statement is legal; If the target query statement is legal, a target query process for the target metadata model is generated based on the target query statement.

10. According to the method according to any one of claims 4 to 9, the target query statement is a query statement in the TQL language or a query statement in the SQL language.

11. A data service device, applied to a metadata service system, wherein the metadata service system stores a metadata model and index information corresponding to the metadata model in a distributed manner; The device comprises: A model acquisition unit, configured to acquire a second metadata model corresponding to the first metadata model to be updated; An updating unit is used to add the second metadata model to the metadata service system, delete the first metadata model, and lock the second metadata model based on the control of a distributed lock; and to add second index information corresponding to the second metadata model to the metadata service system, delete the first index information corresponding to the first metadata model, and unlock the second metadata model.

12. According to the device of claim 11, the type of database used by the metadata service system when distributing and storing the metadata model and the index information corresponding to the metadata model is a Key-value database.

13. The apparatus according to claim 11, further comprising: A topology configuration unit is used to generate a topology map corresponding to the multiple metadata models through topology configuration based on the multiple metadata models stored in the metadata service system; wherein the topology map includes multiple nodes, each node corresponds to a metadata model, and the connections between the nodes correspond to the association relationship between the metadata models.

14. The apparatus according to claim 13, further comprising: A query statement acquisition unit, used to acquire a target query statement, wherein the target query statement includes a query object and a query condition; wherein the query object includes a target metadata model stored in the metadata service system, and the query condition includes one or more index information stored in the metadata service system; A query process generating unit, used for generating a target query process for the target metadata model based on the target query statement; The query unit is used to query multiple metadata models stored in the metadata service system based on the target query process to obtain query results; the query results include the target metadata model.

15. The device according to claim 14, wherein the query process generating unit is specifically configured to: Based on the target query statement, determining one or more metadata models corresponding to the one or more index information; Based on the one or more metadata models, a target query path for the target metadata model is determined in the topology map to generate the target query process; wherein, The target query path includes a plurality of query nodes connected in sequence, the last query node among the plurality of query nodes is the target metadata model, and the plurality of query nodes also include at least part of the one or more metadata models.

16. The apparatus according to claim 15, wherein the plurality of query nodes further comprises metadata models associated with the target metadata model and / or the one or more metadata models in the topology map.

17. The device according to claim 15, wherein the query process generating unit is specifically configured to: Based on the one or more metadata models, determining an original query path for the target metadata model in the topology map; Based on the preset path optimization rules, the original query path is optimized to obtain the target query path; wherein, The target query path includes fewer query nodes than the original query path. 18 . The device according to claim 17 , wherein the path optimization rule comprises recursive optimization, and the recursive optimization comprises one or more of optimization based on association deduction and optimization based on index discrimination.

19. The apparatus according to claim 14, further comprising: A parsing unit, configured to parse the target query statement to determine whether the target query statement is legal; If the target query statement is legal, a target query process for the target metadata model is generated based on the target query statement.

20. The device according to any one of claims 14 to 19, wherein the target query statement is a query statement in TQL language or a query statement in SQL language.

21. A computer device comprising: Memory and processor; The memory stores a computer program executable by the processor; When the processor runs the computer program, the processor performs the method according to any one of claims 1 to 10.

22. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images