Method, apparatus, device and medium for storing and querying data

Abstract

Embodiments of the present disclosure relate to methods, apparatuses, devices and media for storing and querying data. A method for storing data includes, for data to be stored, generating a primary index for indexing the data. The method also includes writing the primary index and the data into a data row in a data table. The method further includes generating, based on a query rule of the data table and the data, a secondary index for indexing the primary index. In addition, the method also includes writing the secondary index and the primary index into an index row in the data table. Embodiments of the present disclosure can ensure that the performance of data writing and data querying does not degrade when entity data in a knowledge base is frequently updated.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computers, and more specifically, to methods, apparatuses, devices, and computer-readable storage media for storing and querying data. BACKGROUND

[0002] A knowledge graph, also known as a knowledge base, is used to describe various entities existing in the real world, the relationships between them, and their respective attributes. At present, knowledge graphs are widely used in search, artificial intelligence, deep learning, and other fields.

[0003] With the rapid growth of the scale of data in the knowledge graph and the application of data, entity data in the knowledge graph can face frequent updates. In addition, in various links of knowledge graph generation, incremental modification of entity data in the knowledge graph is often also needed. Therefore, it is desirable to provide a technology for storing and querying entity data in the knowledge graph, so as to ensure that the performance of data writing and data querying does not decrease in the above scenarios where the entity data is frequently updated. SUMMARY

[0004] According to example embodiments of the present disclosure, a scheme for storing and querying data is provided.

[0005] In a first aspect of the present disclosure, a method for storing data is provided. The method comprises: generating, for data to be stored, a primary index for indexing the data; writing the primary index and the data into a data row in a data table; generating, based on a query rule of the data table and the data, a secondary index for indexing the primary index; and writing the secondary index and the primary index into an index row in the data table.

[0006] In a second aspect of the present disclosure, a method for querying data is provided. The method comprises: in response to receiving a query request for data stored in a data table, determining, from one or more index rows of the data table, an index row matching the query request, the index row storing a primary index for indexing data and a secondary index for indexing the primary index; based on the primary index in the index row, determining, from one or more data rows of the data table, a data row associated with the index row, the data row storing the primary index and the data; determining whether the index row matches the data row; and in response to determining that the index row matches the data row, obtaining the data from the data row as a response to the query request.

[0007] In a third aspect of the present disclosure, an apparatus for storing data is provided. The apparatus includes: a first index generating module configured to generate, for data to be stored, a first index for indexing the data; a data row writing module configured to write the first index and the data into a data row in a data table; a second index generating module configured to generate, based on a query rule of the data table and the data, a second index for indexing the first index; and an index row writing module configured to write the second index and the first index into an index row in the data table.

[0008] In a fourth aspect of the present disclosure, an apparatus for querying data is provided. The apparatus includes: an index row determining module configured to determine, in response to receiving a query request for data stored in a data table, an index row matching the query request from one or more index rows of the data table, the index row storing a first index for indexing data and a second index for indexing the first index; a data row determining module configured to determine, based on the first index in the index row, a data row associated with the index row from one or more data rows of the data table, the data row storing the first index and the data; a matching module configured to determine whether the index row matches the data row; and a data obtaining module configured to obtain, in response to determining that the index row matches the data row, the data from the data row as a response to the query request.

[0009] In a fifth aspect of the present disclosure, an apparatus for storing data is provided. The apparatus includes: one or more processors; and a storage storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method according to the first aspect of the present disclosure.

[0010] In a sixth aspect of the present disclosure, an apparatus for querying data is provided. The apparatus includes: one or more processors; and a storage storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method according to the second aspect of the present disclosure.

[0011] In a seventh aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, which when executed by a processor, implements the method according to the first aspect of the present disclosure.

[0012] In an eighth aspect of the present disclosure, a computer-readable storage medium is provided, which stores a computer program, which when executed by a processor, implements the method according to the second aspect of the present disclosure.

[0013] It is to be understood that the description of the summary section is not intended to identify key or essential features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be apparent from review of the disclosure, which is as follows. BRIEF DESCRIPTION OF DRAWINGS

[0014] The above and other features, advantages and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0015] Figure 1 A block diagram of an example system according to embodiments of the present disclosure is shown;

[0016] Figure 2 A block diagram of an example server for storing and querying data according to embodiments of the present disclosure is shown;

[0017] Figure 3 A flow diagram of a method for storing data according to embodiments of the present disclosure is shown;

[0018] Figure 4A And 4B A schematic diagram of an example data table according to embodiments of the present disclosure is shown;

[0019] Figure 5 A flow diagram of a method for querying data according to embodiments of the present disclosure is shown;

[0020] Figure 6 A block diagram of an apparatus for storing data according to embodiments of the present disclosure is shown;

[0021] Figure 7 A block diagram of an apparatus for querying data according to embodiments of the present disclosure is shown; and

[0022] Figure 8 A block diagram of a computing device that can implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. While several embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in many forms and should not be construed as limited to the embodiments set forth herein, but rather should be read to include any embodiments falling within the scope of the present disclosure. It will be understood that the drawings and the embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure.

[0024] In the description of embodiments of the disclosure, the term "includes" and its synonyms shall be understood as open-ended including, i.e., "including but not limited to". The term "based on" shall be understood as "based at least in part on". The term "one embodiment" or "the embodiment" shall be understood as "at least one embodiment". The terms "first", "second", etc. can refer to different or same objects. Other explicit and implicit definitions can also be included below.

[0025] As described above, with the sharp increase in the scale of data in the knowledge graph and the application of data, the data in the knowledge graph can face frequent updates. In addition, in various links of knowledge graph generation, incremental modification of data in the knowledge graph is often also needed. These updates and modifications often rely on the query ability of the storage system to the data in the knowledge graph.

[0026] The data in the knowledge graph can usually be represented by triples in the subject-predicate-object (SPO) form. Taking the entity "green plant" as an example, the data (also referred to as "entity data") associated with it, for example, includes "the biological classification of green plant is monocotyledon class" (wherein "green plant" is the subject, "biological classification" is the predicate, and "monocotyledon class" is the object), "the subject of green plant is araceae" (wherein "green plant" is the subject, "biological classification" is the predicate, and "monocotyledon class" is the object), and the like.

[0027] Taking the SPO data "the biological classification of green plant is monocotyledon class" as an example, if the entity "green plant" and the entity "monocotyledon class" are regarded as two nodes, and the relationship "biological classification" is regarded as an edge connecting the two nodes, the edge can be referred to as an "out-edge" or a "positive edge" representing the node of the entity "green plant", and can be referred to as an "in-edge" or a "negative edge" representing the node of the entity "monocotyledon class". The entity "monocotyledon class" can be referred to as an "out-edge entity" or a "positive edge entity" of the entity "green plant". The entity "green plant" can be referred to as an "in-edge entity" or a "negative edge entity" of the entity "monocotyledon class". The relationship of the entity "green plant" with respect to the entity "monocotyledon class" is also referred to as an "out-edge relationship" or a "positive edge relationship", while the relationship of the entity "monocotyledon class" with respect to the entity "green plant" is also referred to as an "in-edge relationship" or a "negative edge relationship". In addition, for ease of illustration, the entity as the object in the SPO triple is referred to as an "object" or an "object entity" herein.

[0028] The entity data in the knowledge graph can be represented in various formats. For example, an example of entity data in a JASON format is shown as follows:

[0029]

[0030] In the above entity data in JASON format, the name of the subject is "green plant", and the corresponding entity identifier (id) is "01rzqh7j". The predicate is "biology class", and the value of the object is "monocotyledon class", and the corresponding entity identifier is "01702l79". Another predicate is "biology family", and the value of the object is "araceae", and the corresponding entity identifier is "01mmz9lr".

[0031] In the generation and use of the knowledge graph, it is expected to store the above entity data and perform various query operations on the stored entity data. For example, query all information related to green plant, that is, query all SPO data of a given subject (S). For another example, query the biology classification information of green plant, that is, query the SPO data related to the combination of the given subject and predicate (SP). For another example, query all entities (in-edge entities) with monocotyledon class as the object, that is, query the subject (S) associated with the given object (O). For another example, query all entities with monocotyledon class as the biology classification, that is, query the subject (S) related to the combination of the given object (O) and predicate (P), and so on.

[0032] Some conventional solutions use a database such as mongoDB to store and query entity data. The mongoDB-based storage system usually relies on establishing multiple indexes on a single data table, and queries through multiple indexes to realize the query for the out-edge entity. However, multiple indexes often seriously affect the write performance of the data table, and therefore cannot adapt to the application scenario where the entity data is frequently updated. Some other conventional solutions use a graph retrieval system such as GraphIndex to store and query entity data. Such solutions can implement graph query technology for a static entity data set. However, when any entity node is updated, the edges connected to the entity node also need to be updated, which brings a large update overhead, and therefore cannot adapt to the application scenario where the entity data is frequently updated.

[0033] According to embodiments of this disclosure, a scheme for storing and querying data is proposed. This scheme implements a low-cost and scalable storage and query technology for entity data in knowledge graphs, thereby addressing the aforementioned problems and / or other potential issues. The scheme stores entity data and a primary index for indexing the entity data in data rows of a data table, and builds a secondary index for indexing the primary index based on query rules for the data table. The secondary index and the primary index are stored in index rows of the same data table. When entity data is updated, a new secondary index is generated and written to a new index row of the data table along with the primary index. The removal of old index rows is delayed until before actual data querying. In this way, embodiments of this disclosure ensure that the performance of data writing and data querying does not degrade when entity data in the knowledge graph is frequently updated.

[0034] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0035] Figure 1 A block diagram of an example system 100 according to an embodiment of the present disclosure is shown. Figure 1 As shown, the example system 100 may generally include a client 120 and multiple servers 130-1, 130-2...130-N (collectively or individually referred to as "server 130", where N is a natural number). The multiple servers 130 are associated with multiple data tables 140-1, 140-2...140-N (collectively or individually referred to as "data table 140", where N is a natural number). It should be understood that in... Figure 1 The structure and function of system 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to environments with different structures and / or functions.

[0036] When storing entity data, user 110 can input the data to be stored (e.g., entity data in JASON format as described above) to client 120. Client 120 can forward the data to be stored to one or more servers 130, such as server 130-1. Server 130-1 can store the received entity data in data table 140-1.

[0037] Despite Figure 1The illustration shows each server 130 associated with a single data table 140. It should be understood that this is for illustrative purposes only and is not intended to limit the scope of this disclosure. In some embodiments, each server 130 may be associated with multiple data tables. In other embodiments, a single data table may be divided into multiple data shards, and each server 130 may be associated with only some of the multiple data shards. That is, a single data table may be stored across multiple servers 130.

[0038] When performing a query on entity data, user 110 can input a query request to client 120. Query requests may include, but are not limited to: querying entities with specific attribute values, querying entities with specific relationships to specific objects, querying outbound relationships between specific entities and specific objects, querying inbound relationships between specific entities and specific objects, querying data related to specific entities, querying entities with specific outbound relationships, querying entities with specific inbound relationships, and so on. Client 120 can forward the query request to multiple servers 130. Each server 130 can query entity data in its stored data table 140 to obtain query results. Each server 130 can return its query results to client 120. Client 120 can merge the query results from each server 130 and return the merged query result to user 110.

[0039] Figure 2 A block diagram of an example server 130 according to an embodiment of the present disclosure is shown. Figure 2 As shown, the example server 130 generally includes a data storage device 210, a data retrieval device 220, a caching device 230, and a storage device 240. It should be understood that... Figure 2 The structure and function of server 130 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to servers with different structures and / or functions.

[0040] like Figure 2 As shown, data storage device 210 can receive data to be stored from client 120 and store the data in data table 140 at server 130. Data table 140 can be stored in storage device 240. To improve the query performance of data table 140, at least a portion of the data in data table 140 can be cached in cache device 230. Further details will follow... Figure 3 The operation of data storage device 210 will be described in detail below.

[0041] like Figure 2As shown, the data query device 220 can receive a query request from the client 120, and perform a query operation on the data table 140 stored at the server 130 according to the query request. The operation of the data query device 220 will be described in detail below in connection with Figure 5

[0042] Figure 3 A flowchart of a method 300 for storing data according to an embodiment of the present disclosure is shown. For example, the method 300 can be performed by the data storage device 210 as shown. The method 300 will be described in detail below in connection with Figure 2 Figure 2 The scope of the present disclosure is not limited in this regard, however, as the method 300 can also include additional blocks not shown and / or can omit blocks shown.

[0043] At block 310, the data storage device 210 generates, for data to be stored, a primary index for indexing the data.

[0044] In some embodiments, the data to be stored can record information about an entity in a knowledge base. Hereinafter, data in JASON format as described above about the entity "Pothos" is taken as an example of the data to be stored. In some embodiments, for example, the data storage device 210 can extract the identifier (i.e., "01rzqh7j") of the entity "Pothos" from the data, and take the identifier as the primary index for indexing the data. It should be understood that this is only an example way of generating a primary index. In other embodiments, the data storage device 210 can also generate a primary index for indexing data based on the identifier and / or other information of an entity.

[0045] At block 320, the data storage device 210 writes the generated primary index and the data to be stored into a data row in the data table 140.

[0046] In some embodiments, the data table 140 is, for example, a data table (or a data slice in a HBase data table) in a distributed storage system HBase. HBase is a high-reliability, high-performance, column-oriented and scalable distributed storage system, which stores data in units of rows. In some embodiments, the rows in the data table 140 can be divided into data rows and index rows. "Data row" as described herein refers to a row storing target data, while "index row" refers to a row storing only index information. Figure 4A A schematic diagram of an example data table 140 according to an embodiment of the present disclosure is shown. As shown, the data table 140 includes a plurality of rows, each of which can be a data row or an index row. Figure 4A ​​As shown, the data table 140 (an HBase data table or a data shard of an HBase data table) can include an index region 440 and a data region 450. The data region 450 can include a plurality of data rows 451, 452, etc. The index region 440 can include a plurality of index rows 441, 442, etc.

[0047] In some embodiments, the data storage 210 can write the generated primary index and the data to be stored into a data row by taking the primary index as a row key of the data row and taking the data to be stored as a value of the data row. For example, in the data row 451 in Figure 4A In the data row 451 in Figure 4A In the data row 452 in

[0048] At block 330, the data storage 210 generates a secondary index for indexing the primary index based on the query rule of the data table 140 and the data to be stored.

[0049] In some embodiments, the data storage 210 can obtain the index configuration information of the data table 140. For example, the index configuration information of the data table 140 can be pre-stored in a relational database (e.g., MySQL) associated with the data table 140. In some embodiments, the index configuration information can indicate, for example, an identifier (e.g., a table name) of the associated data table 140, a query rule supported by the data table 140, a processing rule of a dirty index row (i.e., an index row that does not match the data row associated therewith) in the data table 140, etc. In some embodiments, the data storage 210 can generate the secondary index for indexing the primary index based on the query rule of the data table 140 indicated in the index configuration information and the data itself to be stored.

[0050] In some embodiments, the query rule of the data table 140 can indicate, for example, a query pattern supported by the data table 140. The target data stored in the data table 140 is entity data in the knowledge base. Take the entity data in JASON format about the entity “green plant” as an example described above, which describes one or more attribute values of the entity “green plant”, and / or the corresponding relationship between the entity “green plant” and one or more object entities (e.g., object entities “monocotyledon class”, “aroid family”). In some embodiments, the query pattern supported by the data table 140 can be one of the following, for example: querying entities with specific attribute values in the data table 140 (e.g., querying entities of type “plant”), querying entities with specific relationships with specific objects (e.g., querying entities whose “biological classification” is “monocotyledon class”), querying out-edge relationships between specific entities and specific objects (e.g., querying the out-edge relationship between “green plant” and “monocotyledon class”, i.e., querying the relationship of “green plant” with respect to “monocotyledon class”), querying in-edge relationships between specific entities and specific objects (e.g., the in-edge relationship between “green plant” and “monocotyledon class”, i.e., querying the relationship of “monocotyledon class” with respect to “green plant”), querying data related to specific entities (e.g., querying all SPO data of the entity “green plant”), querying entities with specific out-edge relationships (e.g., querying entities with out-edge relationship “biological classification”), querying entities with specific in-edge relationships (e.g., querying entities with in-edge relationship “biological classification”).

[0051] In some embodiments, the data storage device 210 can generate a secondary index for indexing the primary index by parsing the semantics of each SPO in the entity data based on the query rule of the data table 140. Additionally, in some embodiments, the data storage device 210 can generate the secondary index by taking the starting primary key of the data table 140 as the prefix of the secondary index.

[0052] For example, the secondary index can be divided into three segments, which are separated by the symbol "|". The first segment can be used to store the starting primary key of the data table 140 and the id of the object O, which are separated by the symbol "'". Assuming that the starting primary key of the data table 140 is "01akbg1g", the first segment is "01akbg1g'01702l79", for example. The second segment can be the expression of JasonPath related to the query rule. For this example, the second segment is "$.biologyClass.@id", for example. The third segment can be the id of the entity as the subject S, i.e., the row key of the entity data to be queried. For this example, the third segment is "01rzqh7j", for example. That is, for the query rule "query the entity whose biological classification is monocotyledon class", the generated secondary index is "01akbg1g'01702l79|$.biologyClass.@id|01rzqh7j". For another example, for the query rule "query the entity whose subject is Araceae", the generated secondary index is "01akbg1g'01mmz9lr|$.biologyFamily.@id|01rzqh7j".

[0053] At block 340, the data storage device 210 writes the secondary index and the primary index into the index row in the data table 140.

[0054] In some embodiments, the data storage device 210 can write the generated secondary index and the primary index into the index row by taking the secondary index as the row key of the index row and taking the primary index as the value of the index row. For example, for the index row 441 in the data table 140, the secondary index 410-1 is taken as the row key of the index row 441, and the primary index 420-1 as the row key of the data row 451 is taken as the value of the index row 441. For another example, for the index row 442 in the data table 140, the secondary index 410-2 is taken as the row key of the index row 442, and the primary index 420-2 as the row key of the data row 452 is taken as the value of the index row 442. Figure 4A Figure 4A In some embodiments, the data storage device 210 can write the generated secondary index and the primary index into the index row by taking the secondary index as the row key of the index row and taking the primary index as the value of the index row. For example, for the index row 441 in the data table 140, the secondary index 410-1 is taken as the row key of the index row 441, and the primary index 420-1 as the row key of the data row 451 is taken as the value of the index row 441. For another example, for the index row 442 in the data table 140, the secondary index 410-2 is taken as the row key of the index row 442, and the primary index 420-2 as the row key of the data row 452 is taken as the value of the index row 442.

[0055] Additionally or alternatively, in some embodiments, the data storage device 210 can perform the writing of the data row and the index row associated with each other as one batch transaction operation to ensure the atomicity of the writing of the data row and the index row and the consistency of the data row and the index row.

[0056] ​Additionally or alternatively, in some embodiments, when there are multiple data shards for the data table 140, the data storage 210 needs to ensure that the index row and the data row associated with each other are always stored in the same data shard, so that the index and the data are accessed at the same server when queried, thus avoiding the extra overhead caused by accessing the index and the data at different locations. To achieve this, for example, when generating the row key of the index row (i.e., the secondary index), the data storage 210 can prefix the starting primary key of the data shard where the data row is located to the row key of the index row. In this way, the data storage 210 can ensure that the index row and the data row associated with each other are always stored in the same data shard of the same data table. Furthermore, when the data shards of the data table 140 are split or multiple data shards are merged, the data storage 210 can split or merge the index row and the data row associated with each other synchronously, also ensuring that the index row and the data row associated with each other are always stored in the same data shard of the same data table.

[0057] In some embodiments, for example, when writing the data row and the index row to the HBase data table, the data storage 210 can make the row key of the index row always precede the row key of the data row in the dictionary order, so that the first half of the region of the same data table (or the same data shard of the data table) is the index region and the second half is the data region. Furthermore, the value of the index row and the value of the data row can be stored in different column families of the HBase data table respectively, thus achieving the physical isolation of the index row and the data row.

[0058] In this way, the entity data of the knowledge graph can be written into the data table 140 for subsequent querying.

[0059] In some embodiments, the data storage 210 can also receive a request to update the entity data from the client 120. For example, as shown in FIG. 4B, when the data storage 210 receives a request to update the original data 430-1 in the data row 451 to the data 430-3, the data storage 210 can write the data 430-3 into the data row 451. Based on the updated data 430-3 and the query rule of the data table 140, the data storage 210 can generate a new secondary index 420-3 for indexing the original primary index 420-1, and write the new secondary index 420-3 and the original primary index 420-1 into the new index row 443. As shown in FIG. 4B, the old index row 441 will not be removed from the data table 140 immediately. Figure 4B Figure 4A Figure 4B Figure 5 ​​​In further detail, the removal of old index row 441 will be delayed until before the actual query, thereby ensuring that the write performance of data table 140 will not degrade during frequent updates. Additionally or alternatively, in some embodiments, the update of data rows and the writing of new index rows can be performed as a batch transaction to ensure the consistency of updated data rows and new index rows.

[0060] In some embodiments, the data storage device 210 may store the data table 140 to, for example, Figure 2 The storage device 240 shown can be implemented, for example, using a first storage medium. To improve the query performance of the data table 140, the index area 440 in the data table 140 can be cached, for example, in a storage device such as... Figure 2 The first-level cache 231 is shown. The first-level cache 231 can be implemented, for example, using a second storage medium. Furthermore, the data area 450 in the data table 140 can be cached, for example, in a cache such as... Figure 2 The second-level cache 232 is shown. The second-level cache 232 can be implemented, for example, using a third storage medium.

[0061] In some embodiments, the access speed of the second storage medium used to implement the L1 cache 231 exceeds the access speed of the third storage medium used to implement the L2 cache 232, and the access speed of the third storage medium exceeds the access speed of the first storage medium used to implement the storage device 240. For example, an example of the first storage medium is a disk, an example of the second storage medium is a memory, and an example of the third storage medium is a solid-state drive. The examples listed above are merely examples of the first, second, and third storage media, and these examples are not intended to limit the scope of this disclosure. It should be understood that other storage media can also be used to implement, for example... Figure 2 The storage device 240 and cache device 230 are shown.

[0062] Figure 5 A flowchart of a method 500 for querying data according to an embodiment of the present disclosure is shown. For example, method 500 may be provided by, for example, Figure 2 The data query device 220 shown is used for execution. The following will be combined with... Figure 2 Method 500 will now be described in detail. It should be understood that method 500 may also include additional boxes not shown and / or the boxes shown may be omitted. The scope of this disclosure is not limited in this respect.

[0063] At box 510, in response to receiving a query request for data stored in data table 140, data query device 220 determines an index row from one or more index rows of data table 140 that matches the query request, the index row storing a primary index for indexing the data and a secondary index for indexing the primary index.

[0064] In some embodiments, the data query device 220 may determine at least a portion of the secondary index based at least on the keywords in the query request, and determine the index row that matches the determined portion of the secondary index from one or more index rows of the data table 140.

[0065] Taking the query request "Query anti-edge entity of monocotyledons" as an example, this query request is given, for instance, the id of object O in the SPO triple (i.e., 01702l79). The data query device 220 can determine the prefix of the secondary index "01akbg1g'01702l79" by concatenating the starting primary key (e.g., 01akbg1g) of the data table 140 (or a data partition of the data table) with the id of object O. The data query device 220 can then scan the index area 440 of the data table 140 based on the determined prefix of the secondary index to determine the index row that matches the determined prefix of the secondary index.

[0066] In some embodiments, at least a portion of the index area 440 of the data table 140 has been cached, for example, in a cache such as... Figure 2 The index row is stored in the first-level cache 231. In this case, the data query device 220 can first perform a scan of the index rows in the first-level cache 231. If an index row matching the prefix of the determined second-level index is already cached in the first-level cache 231, the data query device 220 can determine the index row in the first-level cache 231. Otherwise, the data query device 220 can perform a scan of the index rows in the storage device 240 to determine the index row from the storage device 240. In some embodiments, when the index row is determined from the storage device 240, the data query device 220 can backfill the index row into the first-level cache 231 to improve the efficiency of subsequent queries.

[0067] At box 520, the data query device 220 determines a data row associated with the index row from one or more data rows of the data table based on the first-level index in the index row, the data row storing the first-level index and data.

[0068] In some embodiments, the data query device 220 may scan the data area 450 of the data table 140 based on the first-level index in the determined index row to determine the data row that matches the first-level index.

[0069] In some embodiments, at least a portion of the data area 450 of the data table 140 has been cached, for example, in a cache such as... Figure 2The data is stored in the secondary cache 232. In this case, the data query device 220 can first perform a scan of the data rows in the secondary cache 232. If the data row that matches the determined primary index is already cached in the secondary cache 232, the data query device 220 can determine the data row in the secondary cache 232. Otherwise, the data query device 220 can perform a scan of the data rows in the storage device 240 to determine the data row that matches the determined primary index from the storage device 240. In some embodiments, when the data row is determined from the storage device 240, the data query device 220 can backfill the data row into the secondary cache 232 to improve the efficiency of subsequent queries.

[0070] At box 530, the data query device 220 determines whether the index row matches the data row.

[0071] In some embodiments, an index row may have a first tag indicating the time when the index row was written to the data table 140, and a data row may have a second tag indicating the time when the data row was last updated. In some embodiments, the data query device 220 may determine whether the first tag and the second tag match. When the first tag does not match the second tag, the data query device 220 may determine that the index row does not match the data row (e.g., in...). Figure 4B In this context, because the update time of data row 451 does not match the write time of index row 441, data row 451 and index row 441 are mismatched.

[0072] Alternatively or additionally, when the first tag matches the second tag, the data query device 220 may further perform a comparison of index information to determine whether the index row matches the data row. Figure 4B Taking data row 451 and index row 443 as an example, their timestamps match. In this case, data query device 220 can regenerate a set of secondary indexes based on data 430-3 in data row 451 and the query rules of data table 140. Data query device 220 can determine whether the secondary index in index row 443 is included in the regenerated set of secondary indexes. If the secondary index in index row 443 is included in the regenerated set of secondary indexes, then data query device 220 can determine that data row 451 and index row 443 match; otherwise, data query device 220 can determine that data row 451 and index row 443 do not match.

[0073] If an index row is determined to match a data row, then at box 540, the data query device 220 retrieves data from that data row as a response to the query request. For example, suppose the data query device 220 has determined that... Figure 4BIf both the data row 451 and the index row 443 match, the data query device 220 can obtain the data 430-3 from the data row 451 as a response to the query request.

[0074] If the index row does not match the data row, at block 550, the data query device 220 removes the index row from the data table 140. For example, assume that the data query device 220 has determined that the index row 441 does not match the data row 451 as shown in FIG. 4B. Figure 4B If both the data row 451 and the index row 441 match, the data query device 220 can obtain the data 430-3 from the data row 451 as a response to the query request.

[0075] As can be seen from the above description, the embodiments of the present disclosure implement a low-cost and scalable storage and query technique for entity data in a knowledge graph, thereby being able to solve the problems and / or other potential problems in the prior art. The embodiments of the present disclosure store entity data and a first index for indexing the entity data in a data row of a data table, and establish a second index for indexing the first index based on a query rule for the data table. The second index and the first index are stored in an index row in the same data table. When the entity data is updated, a new second index is generated and written into a new index row of the data table together with the first index. The removal of the old index row is delayed until actual data query is performed. In this way, the embodiments of the present disclosure can ensure that the performance of data writing and data query does not decrease when the entity data in the knowledge graph is frequently updated.

[0076] Figure 6 A schematic block diagram of a device 600 for storing data according to an embodiment of the present disclosure is shown. The device 600 can be used to implement the data storage device 210 as shown in FIG. 2A. Figure 2 As shown in FIG. 2A, the device 210 can include a first index generation module 210-1 configured to generate, for data to be stored, a first index for indexing the data; a data row writing module 210-2 configured to write the first index and the data into a data row in a data table; a second index generation module 210-3 configured to generate, based on a query rule of the data table and the data, a second index for indexing the first index; and an index row writing module 210-4 configured to write the second index and the first index into an index row in the data table. Figure 6 As shown in FIG. 2A, the device 210 can include a first index generation module 210-1 configured to generate, for data to be stored, a first index for indexing the data; a data row writing module 210-2 configured to write the first index and the data into a data row in a data table; a second index generation module 210-3 configured to generate, based on a query rule of the data table and the data, a second index for indexing the first index; and an index row writing module 210-4 configured to write the second index and the first index into an index row in the data table.

[0077] In some embodiments, the data record is related to information about an entity in a knowledge base, and the first index generation module 610 includes an identifier obtaining unit configured to obtain an identifier of the entity from the data, and a first index generation unit configured to generate the first index based on the identifier of the entity.

[0078] In some embodiments, the data row writing module comprises: a first data row writing unit configured to write the primary index as a key of the data row into the data row; and a second data row writing unit configured to write the data as a value of the data row into the data row.

[0079] In some embodiments, the secondary index generating module comprises: a configuration information obtaining unit configured to obtain index configuration information of the data table, the index configuration information indicating a query rule of the data table; and a first generating unit configured to generate the secondary index based on the index configuration information and the data.

[0080] In some embodiments, the data table is configured to store one or more attribute values of an entity in a knowledge base and / or a corresponding relationship between the entity and one or more objects, and the query rule for the data table comprises at least one of: querying an entity having a specific attribute value; querying an entity having a specific relationship with a specific object; querying a forward relationship of a specific entity with respect to a specific object; querying a reverse relationship of a specific object with respect to a specific entity; querying data related to a specific entity; querying an entity associated with a specific forward relationship; and querying an entity associated with a specific reverse relationship.

[0081] In some embodiments, the secondary index generating module comprises: a starting primary key determining unit configured to determine a starting primary key of the data table; and a second generating unit configured to generate the secondary index by taking the starting primary key as a prefix of the secondary index.

[0082] In some embodiments, the index row writing module comprises: a first index row writing unit configured to write the secondary index as a key of the index row into the index row; and a second index row writing unit configured to write the primary index as a value of the index row into the index row.

[0083] In some embodiments, the apparatus 600 further comprises: a data updating module configured to update the data stored in the data row in response to receiving a request to update the data; a new secondary index generating module configured to generate a new secondary index for indexing the primary index based on the updated data and the query rule; and a new index row writing module configured to write the new secondary index and the primary index into another index row in the data table.

[0084] In some embodiments, the apparatus 600 further comprises: a storage module configured to store the data table into a storage device implemented by a first storage medium; a first index row cache module configured to cache the index row into a primary cache implemented by a second storage medium; and a first data row cache module configured to cache the data row into a secondary cache implemented by a third storage medium.

[0085] In some embodiments, the access speed of the second storage medium exceeds the access speed of the third storage medium, and the access speed of the third storage medium exceeds the access speed of the first storage medium.

[0086] It should be understood that each module recited in the apparatus 600 corresponds to each step in the method 300 described with reference to Figure 3 and has the same effects as the corresponding operations and features in the method 300, and details are not described herein again.

[0087] Figure 7 A schematic block diagram of an apparatus 700 for querying data according to embodiments of the present disclosure is shown. The apparatus 700 can be used to implement the data querying apparatus 220 as shown in Figure 2 As shown in Figure 7 The apparatus 700 can include an index row determining module 710 configured to determine, in response to receiving a query request for data stored in a data table, an index row matching the query request from one or more index rows of the data table, the index row storing a primary index for indexing data and a secondary index for indexing the primary index; a data row determining module 720 configured to determine, based on the primary index in the index row, a data row associated with the index row from one or more data rows of the data table, the data row storing the primary index and data; a matching module 730 configured to determine whether the index row matches the data row; and a data obtaining module 740 configured to obtain, in response to determining that the index row matches the data row, the data from the data row as a response to the query request.

[0088] In some embodiments, the data table is used to store one or more attribute values of an entity in a knowledge base and / or a corresponding relationship between the entity and one or more objects, and the query request is used to query at least one of the following: an entity having a specific attribute value; an entity having a specific relationship with a specific object; an out-edge relationship between a specific entity and a specific object; an in-edge relationship between a specific entity and a specific object; data related to a specific entity; an entity having a specific out-edge relationship; and an entity having a specific in-edge relationship.

[0089] In some embodiments, the apparatus 700 further includes an index row removing module configured to remove, in response to determining that the index row does not match the data row, the index row from the data table.

[0090] In some embodiments, the index row determining module 710 includes a secondary index determining unit configured to determine at least a part of the secondary index based on a keyword in the query request, and an index row determining unit configured to determine, from the one or more index rows, an index row matching the at least a part of the secondary index.

[0091] In some embodiments, the matching module 730 includes a first marker obtaining unit configured to obtain a first marker of the index row, the first marker identifying a first time at which the index row was written to the data table, a second marker obtaining unit configured to obtain a second marker of the data row, the second marker identifying a second time at which the data row was updated, a marker matching unit configured to determine whether the first marker matches the second marker, and a first match determining unit configured to determine that the index row does not match the data row in response to determining that the first marker does not match the second marker.

[0092] In some embodiments, the matching module 730 further includes a secondary index generating unit configured to generate a set of secondary indexes based on data in the data row in response to determining that the first marker matches the second marker, a secondary index matching unit configured to determine whether a secondary index in the index row is included in the set of secondary indexes, a second match determining unit configured to determine that the index row matches the data row in response to determining that the secondary index in the index row is included in the set of secondary indexes, and a third match determining unit configured to determine that the index row does not match the data row in response to determining that the secondary index in the index row is missing from the set of secondary indexes.

[0093] In some embodiments, the data table is stored in a storage device implemented with a first storage medium, and at least a portion of the one or more index rows are cached in a primary cache implemented with a second storage medium. The index row determining module 710 includes a primary cache searching unit configured to determine whether an index row is cached in the primary cache, a first index row determining unit configured to determine the index row from the primary cache in response to determining that the index row is cached in the primary cache, and a second index row determining unit configured to determine the index row from the storage device in response to determining that the index row is missing from the primary cache.

[0094] In some embodiments, the apparatus 700 further includes a second index row caching module configured to cache an index row in the primary cache in response to the index row being determined from the storage device.

[0095] In some embodiments, at least a portion of the one or more data rows are cached in a secondary cache implemented with a third storage medium. The data row determining module 720 includes a secondary cache searching unit configured to determine whether a data row is cached in the secondary cache, a first data row determining unit configured to determine the data row from the secondary cache in response to determining that the data row is cached in the secondary cache, and a second data row determining unit configured to determine the data row from the storage device in response to determining that the data row is missing from the secondary cache.

[0096] In some embodiments, the apparatus 700 further includes a second data row caching module configured to cache the data row in a secondary cache in response to a data row being determined from the storage device.

[0097] In some embodiments, the access speed of the second storage medium exceeds the access speed of the third storage medium, and the access speed of the third storage medium exceeds the access speed of the first storage medium.

[0098] It should be understood that each module described in device 700 is related to a reference. Figure 5 Each step in the described method 500 corresponds to and has the same effect as the corresponding operation and feature in method 500; specific details will not be repeated here.

[0099] Furthermore, the modules and / or units included in apparatus 600 and / or 700 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in apparatus 600 and / or 700 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0100] Figure 6 and / or Figure 7 The modules and / or units shown can be implemented, in whole or in part, as hardware modules, software modules, firmware modules, or any combination thereof. In particular, in some embodiments, the processes, methods, or procedures described above can be implemented by hardware in a storage system, a host corresponding to the storage system, or other computing devices independent of the storage system.

[0101] Figure 8 A schematic block diagram of an example device 800 that can be used to implement embodiments of the present disclosure is shown. Device 800 can be used to implement, for example... Figure 1 Server 130 is shown. (As shown) Figure 8As shown, the device 800 includes a central processing unit (CPU) 801, which can perform various suitable actions and processes in accordance with computer program instructions stored in a read-only memory (ROM) 802 or computer program instructions loaded into a random access memory (RAM) 803 from a storage unit 808. Various programs and data required by the device 800 for operation are also stored in the RAM 803. The CPU 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0102] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, a magneto-optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0103] The processing unit 801 performs various methods and processes described above, such as the methods 300 and / or 500. For example, in some embodiments, the methods 300 and / or 500 can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the CPU 501, one or more steps of the methods 300 and / or 500 described above can be performed. Alternatively, in other embodiments, the CPU 801 can be configured to perform the methods 300 and / or 500 by any other suitable means, such as by means of firmware.

[0104] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0105] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, causes the machine to perform the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0106] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores program code for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include one or more lines of a system, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0107] Further, while operations are depicted in a particular order, this should not be understood as requiring such an order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, while several specific implementation details have been discussed, such details should not be construed as limiting the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in subcombination or as separate embodiments, apart from the other features described.

[0108] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for querying data, comprising: In response to receiving a query request for data stored in a data table, an index row matching the query request is determined from one or more index rows of the data table. The index row stores a primary index for indexing the data and a secondary index for indexing the primary index. The secondary index is generated based on the query rules of the data table and the data. The secondary index serves as the key of the index row, and the primary index serves as the value of the index row. The secondary index includes information associated with the data. Based on the first-level index in the index row, a data row associated with the index row is determined from one or more data rows of the data table, the data row storing the first-level index and the data; Determining whether the index row matches the data row includes: Obtain a first marker of the index row, the first marker identifying the first time the index row was written into the data table; Obtain a second marker for the data row, the second marker identifying a second time when the data row was updated; Determine whether the first tag matches the second tag; and In response to determining that the first tag does not match the second tag, it is determined that the index row does not match the data row; and In response to determining that the index row matches the data row, the data is retrieved from the data row as a response to the query request. The related data rows and index rows are written to the data table as a batch transaction, and The related index rows and data rows are stored in the same data shard.

2. The method according to claim 1, wherein the data table is used to store one or more attribute values ​​of entities in the knowledge base, and / or the corresponding relationships between the entities and one or more objects, and the query request is used to query at least one of the following: Entities with specific attribute values; An entity that has a specific relationship with a specific object; Outbound relationships between specific entities and specific objects; Inbound edges between a specific entity and a specific object; Data related to a specific entity; Entities with specific outgoing edge relationships; and Entities with specific incoming edge relationships.

3. The method according to claim 1, further comprising: In response to determining that the index row does not match the data row, the index row is removed from the data table.

4. The method of claim 1, wherein determining the index row from the one or more index rows comprises: At least a portion of the secondary index is determined based on the keywords in the query request. as well as Determine from the one or more index rows that match at least a portion of the secondary index.

5. The method according to claim 1, wherein determining whether the index row matches the data row further comprises: In response to determining that the first tag matches the second tag, a set of secondary indexes is generated based on the data in the data row; Determine whether the secondary index in the index row is included in the set of secondary indexes; In response to determining that the secondary index in the index row is included in the set of secondary indexes, the index row is determined to match the data row; and In response to determining that the secondary index in the index row is missing in the set of secondary indexes, it is determined that the index row does not match the data row.

6. The method of claim 1, wherein the data table is stored in a storage device implemented using a first storage medium, at least a portion of the one or more index rows are cached in a level-one cache implemented using a second storage medium, and determining the index row from the one or more index rows comprises: Determine whether the index row is cached in the first-level cache; In response to determining that the index row is cached in the first-level cache, the index row is determined from the first-level cache; as well as In response to determining that the index row is missing in the first-level cache, the index row is determined from the storage device.

7. The method according to claim 6, further comprising: In response to the index row being determined from the storage device, the index row is cached in the first-level cache.

8. The method of claim 6, wherein at least a portion of the one or more data rows is cached in a secondary cache implemented using a third storage medium, and determining the data row from the one or more data rows comprises: Determine whether the data row is cached in the second-level cache; In response to determining that the data row is cached in the second-level cache, the data row is determined from the second-level cache; as well as In response to determining that the data row is missing in the secondary cache, the data row is determined from the storage device.

9. The method according to claim 8, further comprising: In response to the data row being determined from the storage device, the data row is cached in the second-level cache.

10. The method of claim 8, wherein the access speed of the second storage medium exceeds the access speed of the third storage medium, and the access speed of the third storage medium exceeds the access speed of the first storage medium.

11. An apparatus for querying data, comprising: An index row determination module is configured to, in response to receiving a query request for data stored in a data table, determine an index row from one or more index rows of the data table that matches the query request. The index rows respectively store a primary index for indexing the data and a secondary index for indexing the primary index. The secondary index is generated based on the query rules of the data table and the data. The secondary index serves as the key of the index row, and the primary index serves as the value of the index row. The secondary index includes information associated with the data. The data row determination module is configured to determine, based on the first-level index in the index row, a data row associated with the index row from one or more data rows of the data table, wherein the data row stores the first-level index and the data; A matching module is configured to determine whether the index row matches the data row, wherein the matching module includes: The first tag acquisition unit is configured to acquire a first tag of the index row, the first tag identifying the first time the index row is written into the data table; The second tag acquisition unit is configured to acquire a second tag of the data row, the second tag identifying a second time when the data row was updated; A tag matching unit is configured to determine whether the first tag matches the second tag; and The first matching determination unit is configured to determine, in response to determining that the index row does not match the data row, the index row does not match the data row; and The data acquisition module is configured to, in response to determining that the index row matches the data row, acquire the data from the data row as a response to the query request. The related data rows and index rows are written to the data table as a batch transaction, and The related index rows and data rows are stored in the same data shard.

12. The apparatus of claim 11, wherein the data table is used to store one or more attribute values ​​of entities in a knowledge base, and / or corresponding relationships between the entities and one or more objects, and the query request is used to query at least one of the following: Entities with specific attribute values; An entity that has a specific relationship with a specific object; Outbound relationships between specific entities and specific objects; Inbound edges between a specific entity and a specific object; Data related to a specific entity; Entities with specific outgoing edge relationships; and Entities with specific incoming edge relationships.

13. The apparatus of claim 11, further comprising: An index row removal module is configured to remove the index row from the data table in response to determining that the index row does not match the data row.

14. The apparatus of claim 11, wherein the index row determination module comprises: The secondary index determination unit is configured to determine at least a portion of the secondary index based at least on the keywords in the query request; as well as The index row determination unit is configured to determine, from the one or more index rows, an index row that matches at least a portion of the secondary index.

15. The apparatus of claim 11, wherein the matching module further comprises: A secondary index generation unit is configured to generate a set of secondary indexes based on the data in the data row in response to determining that the first tag matches the second tag; A secondary index matching unit is configured to determine whether the secondary index in the index row is included in the set of secondary indexes; The second matching determination unit is configured to determine that the index row matches the data row in response to determining that the secondary index in the index row is included in the set of secondary indexes; as well as The third matching determination unit is configured to determine that the index row does not match the data row in response to determining that the secondary index in the index row is missing in the set of secondary indexes.

16. The apparatus of claim 11, wherein the data table is stored in a storage device implemented using a first storage medium, at least a portion of the one or more index rows are cached in a level-one cache implemented using a second storage medium, and the index row determination module comprises: A first-level cache search unit is configured to determine whether the index row is cached in the first-level cache; The first index row determining unit is configured to determine the index row from the first-level cache in response to determining that the index row is cached in the first-level cache; as well as The second index row determination unit is configured to determine the index row from the storage device in response to determining that the index row is missing in the first-level cache.

17. The apparatus of claim 16, further comprising: The second index row caching module is configured to cache the index row in the first-level cache in response to the index row being determined from the storage device.

18. The apparatus of claim 16, wherein at least a portion of the one or more data rows is cached in a secondary cache implemented using a third storage medium, and the data row determination module comprises: A second-level cache search unit is configured to determine whether the data row is cached in the second-level cache; The first data row determination unit is configured to determine the data row from the second-level cache in response to determining that the data row is cached in the second-level cache; as well as The second data row determination unit is configured to determine the data row from the storage device in response to determining that the data row is missing in the secondary cache.

19. The apparatus of claim 18, further comprising: The second data row caching module is configured to cache the data row in the second-level cache in response to the data row being determined from the storage device.

20. The apparatus of claim 18, wherein the access speed of the second storage medium exceeds the access speed of the third storage medium, and the access speed of the third storage medium exceeds the access speed of the first storage medium.

21. A device for querying data, comprising: One or more processors; as well as A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method according to any one of claims 1-10.

22. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method according to any one of claims 1-10.

Images