Distributed cache system and associated query and update method, device and storage medium

Abstract

The embodiment of the present application provides a distributed cache system, an associated query and update method, a device and a storage medium. In the embodiment of the present application, a distributed cache system is provided, which includes a read node and a write node. The memory cache of the read node stores the specified data in the data source, so that the associated query can be performed in the memory cache of the read node without querying the data source, which helps to improve the efficiency of the associated query; in addition, the write node can promptly notify the read node to update the specified data in the memory cache when the specified data in the data source changes, so as to ensure the accuracy and real-time performance of the associated query results based on the memory cache.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a distributed cache system and an associated query and update method, device and storage medium. Background Art

[0002] In today's information age, the application and research of information data has become a trend. Based on its advantages, databases are widely used in data storage, management, maintenance and query.

[0003] With the development of big data, database systems often need to provide different dimensions of data association queries for different users. However, the existing association query solution will trigger a table scan operation when the index is not hit, and the table scan performance is low, resulting in low efficiency of association query. In addition, the existing database system does not have the ability of distributed horizontal expansion, which limits the overall capacity of association query services. Summary of the invention

[0004] Multiple aspects of the present application provide a distributed cache system, an association query and update method, a device and a storage medium to improve the efficiency of association query and the overall capacity of association query services.

[0005] An embodiment of the present application provides a distributed cache system, including: a read node and a write node, wherein the memory cache of the read node stores specified data in a data source, and is used to provide external associated query operations based on the specified data in the memory cache, and to update the specified data in the memory cache based on a notification from the write node; the write node is used to notify the read node to update the specified data in its memory cache when the specified data in the data source changes, so that the read node can provide external associated query operations.

[0006] The embodiment of the present application also provides a data center network, comprising: a plurality of network devices, a database, and a distributed cache system; wherein the distributed cache system comprises a read node and a write node;

[0007] The database is used to store data related to the plurality of network devices;

[0008] The memory cache of the read node stores the specified data in the database, and is used to provide external associated query operations according to the specified data in the memory cache, and to update the specified data in the memory cache according to the notification of the write node;

[0009] The write node is used to notify the read node to update the specified data in its memory cache when the specified data in the database changes, so that the read node can provide external associated query operations.

[0010] An embodiment of the present application also provides an association query method, which is applicable to a reading node in a distributed cache system, and the method includes: receiving an association query request, the association query request including a filter condition and a query item; parsing a table item to be associated, a target table item, and an association path between the table item to be associated and the target table item from the filter condition; performing an association query in a local memory cache according to the association path between the table item to be associated and the target table item and the query item; wherein the local memory cache stores specified data in a data source, the specified data including the table item to be associated and the target table item.

[0011] An embodiment of the present application also provides a data source update method, which is applicable to a write node in a distributed cache system, comprising: receiving a data source update notification sent by an application system, wherein the data source update notification includes change information of specified data in the data source; sending the change information of the specified data to a read node in the distributed cache system, so that the read node can update the specified data in the local memory cache according to the change information.

[0012] An embodiment of the present application also provides an electronic device, which can be implemented as a read node in a distributed cache system, including: a memory, a processor and a communication component; the memory includes a memory cache and other storage space, the memory cache stores specified data in a data source, and the other storage space is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program for: receiving an association query request through the communication component, the association query request including a filter condition and a query item; parsing a to-be-associated table item, a target table item and an association path between the to-be-associated table item and the target table item from the filter condition; performing an association query in a local memory cache according to the association path between the to-be-associated table item and the target table item and the query item; wherein the local memory cache stores specified data in a data source, and the specified data includes the to-be-associated table item and the target table item.

[0013] An embodiment of the present application also provides an electronic device that can be implemented as a write node in a distributed cache system, including: a memory, a processor and a communication component; wherein the memory is used to store a computer program; the processor is coupled to the memory, and is used to execute the computer program for: receiving a data source update notification sent by an application system through the communication component, and the data source update notification includes change information of specified data in the data source; sending the change information of the specified data to a read node in the distributed cache system through the communication component, so that the read node can update the specified data in the local memory cache according to the change information.

[0014] The embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned association query method.

[0015] An embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned write node data update method.

[0016] In an embodiment of the present application, a distributed cache system is provided. The system includes a read node and a write node. The memory cache of the read node stores the specified data in the data source. In this way, an associated query can be performed in the memory cache of the read node without querying the data source, which helps to improve the efficiency of the associated query; in addition, when the specified data in the data source changes, the write node can promptly notify the read node to update the specified data in the memory cache, thereby ensuring the accuracy and real-time performance of the associated query results based on the memory cache. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0018] Figure 1a A schematic diagram of the structure of a distributed cache system provided in an embodiment of the present application;

[0019] Figure 1b A schematic diagram of the structure of a data center network provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of an association query method provided in an embodiment of the present application;

[0021] Figure 3 A flowchart of a data updating method provided in an embodiment of the present application;

[0022] Figure 4 A flowchart of another data updating method provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0026] In view of the technical problem that the existing associative query based on data source is inefficient, in some embodiments of the present application, a distributed cache system is provided. The system includes a read node and a write node, and the memory cache of the read node stores the specified data in the data source, so that the associative query can be performed in the memory cache of the read node without querying the data source, which helps to improve the efficiency of the associative query; in addition, when the specified data in the data source changes, the write node can promptly notify the read node to update the specified data in the memory cache, thereby ensuring the accuracy and real-time performance of the associative query results based on the memory cache.

[0027] Furthermore, in the distributed cache system of the embodiment of the present application, the read nodes can be horizontally expanded to improve the overall capacity of the associated query service, thereby solving the problem that the associated query based on the existing data source cannot be horizontally expanded.

[0028] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0029] Figure 1a A schematic diagram of the structure of a distributed cache system provided in an embodiment of the present application. Figure 1a As shown, the distributed cache system includes: a read node 11 and a write node 12. The read node 11 can also have the function of the write node 12 and be implemented as the write node 12; similarly, the write node 12 can also have the function of the read node 11 and be implemented as the read node 11.

[0030] In this embodiment, the number of read nodes 11 can be one or more. In each embodiment of the present application, multiple refers to 2 or more. The read node 11 can be a software module, application, service or a physical device that provides users with an association query function based on a data source. Multiple read nodes 11 can be deployed on different physical machines, or in different containers or virtual machines (VMs). Of course, these containers or virtual machines can be deployed on the same physical machine or on multiple different physical machines.

[0031] In this embodiment, the number of write nodes 12 may be one or more. The multiple write nodes 12 may be deployed on different physical machines or in different containers or virtual machines. These containers or virtual machines may be deployed on the same physical machine or on multiple different physical machines.

[0032] In this embodiment, each read node 11 has its own in-memory cache, and the in-memory cache of each read node 11 stores the specified data in the data source. Among them, the specified data in the data source stored in the in-memory cache of the read node 11 is a snapshot, mirror or copy of the specified data in the data source. Based on this, the read node 11 can provide external associated query operations based on the specified data in its in-memory cache, and the associated query operation mainly refers to the associated query operation that depends on the specified data. In an embodiment of the present application, the specified data in the data source can be part of the data in the data source, or it can be the full amount of data in the data source. Further, the specified data includes but is not limited to: metadata files, index files, and data files. Optionally, metadata information such as table item names, table item attributes, and attribute types are stored in the metadata file; specific instance data are stored in the data file, and these specific instance data can be encoded in bytes.

[0033] The data source may be any system that has data storage and management functions and supports associated queries, such as a database system, a data warehouse, a cloud storage system, etc.

[0034] In this embodiment, the memory cache of the reading node stores the specified data in the data source, and the associated query operation related to the specified data can be performed in the memory cache of the reading node without going to the data source, which helps to improve the efficiency of the associated query. In addition, in this embodiment, the memory cache of the reading node 11 is based on the memory implementation and supports the memory index structure, which can further improve the table lookup performance during the associated query, and further improve the efficiency of the associated query.

[0035] Furthermore, when the specified data in the data source changes, the write node 12 can notify the read node 11 to update the specified data stored in its local memory cache. Correspondingly, the read node 11 can also update the specified data in its memory cache according to the notification of the write node 12. Among them, when the specified data in the data source changes, the write node 12 notifies the read node 11 to update its memory cache, which can provide a basis for the read node 11 to provide external association query operations, which is conducive to ensuring the accuracy and real-time performance of the association query results based on the memory cache.

[0036] Furthermore, in the distributed cache system of this embodiment, the read nodes are independent of each other and all store designated data in the data source. The number of read nodes can be flexibly increased, and by increasing the number of read nodes, horizontal expansion of the system capacity can be achieved, thereby improving the overall capacity of the distributed cache system's associated query service and solving the problem that existing associated queries cannot be horizontally expanded.

[0037] In this embodiment, the distributed cache system can provide users with an associated query service, and the associated query service is specifically provided by the reading node 11. When a user needs to perform an associated query, an associated query request can be issued to the distributed cache system. For example, the distributed cache system can provide a query portal for users to issue an associated query request. Among them, the associated query request includes filtering conditions and query items. Among them, the filtering conditions include filtering conditions related to the query table items and filtering conditions related to the query data. Based on the filtering conditions related to the query table items, information such as the target table item of this associated query, the table item to be associated, and the association path between the table item to be associated and the target table item can be determined. The query item is used to describe the rule information for the subsequent assembly of the query results.

[0038] Furthermore, the query entry provided by the distributed cache system to the user may be a web page, an application page, or a command window, etc. After the associated query request reaches the distributed cache system, it will be assigned to any one or more read nodes 11 therein. The assigned read node 11 receives the associated query request and provides the user with an associated query service according to the associated query request. For any read node 11, the process of performing an associated query according to the associated query request is the same or similar. The following takes one of the read nodes 11 as an example to illustrate the process of performing an associated query on the read node.

[0039] When performing an association query, the read node 11 receives an association query request, which includes a filter condition and a query item. The read node 11 can parse the to-be-associated table item, the target table item, and the association path between the to-be-associated table item and the target table item from the filter condition. The to-be-associated table item refers to other table items other than the target table item that need to be associated in the association query, and the number can be 1 or more. In the embodiment of the present application, multiple refers to two or more. Further, since the specified data stored in the local memory cache of the read node 11 includes the to-be-associated table item and the target table item, the read node 11 can perform an association query in the local memory cache according to the association path between the to-be-associated table item and the target table item and the above-mentioned query item. In addition, the specified data stored in the memory cache of the read node 11 includes metadata files, and these metadata contain association relationships between different table items, which also provides a data basis for association queries based on memory caches.

[0040] In an optional embodiment, the user can input a corresponding query statement to the distributed cache system using a data source language supported by the distributed cache system, thereby initiating an associated query request to the distributed cache system. Optionally, the distributed cache system of this embodiment can support a cache query language (Linkcache query language, LQL), a structured query language (Structured Query Language, SQL) or a domain-specific language (Domain Specified Language, DSL), etc., but is not limited thereto. The query statement includes filter conditions and query items. The description language of the associated query request conforms to the grammatical format of the distributed cache system.

[0041] Correspondingly, the reading node 11 performs syntax analysis on the associated query request, and can parse out the filter conditions and query items of the associated query. Optionally, the syntax analyzer such as Antlr, Yacc, etc. can be used to perform syntax analysis on the associated query request, but it is not limited to this. Further, the reading node 11 parses the table item to be associated, the target table item, and the association path between the table item to be associated and the target table item from the filtering condition. Optionally, the reading node 11 parses the table item to be associated and the target table item from the filtering condition, and calculates the association path between the table item to be associated and the target table item according to the primary key and foreign key relationship between the table item to be associated and the target table item. Afterwards, the reading node 11 performs an associated query in the local memory cache according to the association path and the query item. Among them, the primary key is an identifier that can uniquely identify a record in a table item; the foreign key is one or more columns that establish or strengthen the link between two table items, and the one or more columns here are the primary key in one table item and the foreign key in another table item.

[0042] Among them, the association path between the to-be-associated table item and the target table item may be one or more. If there are multiple association paths between the to-be-associated table item and the target table item, the read node 11 can select the shortest association path from the multiple determined association paths as the target association path; and according to the target association path and the query item, perform an association query in the specified data of the data source stored in the memory cache to obtain the query result. Querying according to the shortest association path can further improve the query efficiency to a certain extent. It is worth noting that selecting the shortest association path is only one selection method and is not limited to this.

[0043] Furthermore, in order to enable the distributed cache system to more effectively utilize the memory cache of the read node 11, the memory cache of the read node 11 can be divided into an index area and a data area. The index area is used to store indexes in large table entries and small table entries; the data area is used to store data in large table entries. For example, when the read node 11 is implemented as a Java Virtual Machine (JVM), the index area can be implemented using the in-heap memory of the read node, and the data area can be implemented using the off-heap memory of the read node 11. The off-heap memory is the memory outside the heap of the JVM, which can more efficiently utilize the overall memory of the machine.

[0044] Optionally, small table items and large table items may be divided according to actual storage requirements, wherein a small table item refers to an item whose item capacity is less than a set capacity threshold, and a large table item refers to an item whose item capacity is greater than or equal to a set capacity threshold. The capacity threshold may be flexibly set according to actual storage requirements and is not limited here. For ease of description and distinction, in the embodiment of the present application, a small table item is defined as a first-category item, and a large table item is defined as a second-category item.

[0045] Based on the above, the reading node 11 can store the specified data in the data source in the memory cache in the following manner: store the index of the second type of table item in the specified data and the first type of table item in the index area, and store the data of the second type of table item in the data area. Among them, a table item includes a table item index and data, and the table item index includes: table item identifier, primary key ID, foreign key, etc., but is not limited to this.

[0046] Based on the above-mentioned index area and data area, when the reading node 11 performs an associated query in the local memory cache according to the associated path and query item between the table item to be associated and the target table item, it can query in the index area according to the associated path between the table item to be associated and the target table item, and obtain the list of primary key IDs in the target table item that meet the data screening condition. Among them, the data screening condition is a part of the filtering condition, which is used to describe the information of the data that needs to be screened in this associated query. Further, if the target table item is a first-class table item, the reading node 11 can directly obtain the data corresponding to each primary key ID in the primary key ID list from the index area; if the target table item is a second-class table item, the reading node 11 obtains the data corresponding to each primary key ID in the primary key ID list from the data area. Further, the reading node 11 assembles the data corresponding to each primary key ID in the obtained primary key ID list according to the query item in the associated query request to obtain the associated query result. Afterwards, the reading node 11 can output or return the associated query result to the user who initiated the associated query request.

[0047] It should be noted that in different application scenarios, users can set different filtering conditions and different query items according to their query needs. Optionally, the filtering conditions may include item identifiers of the items to be associated and the target items, primary key ID lists of the items to be associated and the target items, and foreign keys, but are not limited thereto.

[0048] In an optional embodiment, the read node 11 has a local persistent storage medium in addition to a memory cache. The local persistent storage medium can be a disk, a solid-state drive, a disk array, etc. The local persistent storage medium of the read node 11 stores a snapshot, a mirror image, or a copy of the specified data in the data source when the most recent build cycle arrives. For ease of description and distinction, the snapshot, mirror image, or copy of the specified data in the data source when the most recent build cycle arrives stored in the local persistent storage medium of the read node 11 is defined as a basic cache file. The purpose of storing the basic cache file is to enable the read node to build the specified data in the memory cache each time it starts. The basic cache file is a data foundation required for the read node to build the memory cache at startup. In addition, in this embodiment, the basic cache file can be periodically constructed according to the specified data in the data source according to the set build cycle.

[0049] Further, if Figure 1a As shown, the distributed cache system provided by the embodiment of the present application also includes an incremental version information library 13. The incremental version information library 13 is mainly used to store the change information of the specified data in the data source during the construction of two basic cache files. The changes here include the data added, updated, deleted, etc. in the specified data. The incremental version information library can be implemented using an existing relational database, a non-relational database, etc. The change information of the specified data is another data basis required for the read node to build a memory cache at startup. In this embodiment, the change information of the basic cache file and the specified data is stored separately, which can ensure that the read node 11 can efficiently update the memory cache, and the consistency of the memory cache of all read nodes can be maintained. In addition, the change information of the specified data is uniformly managed by persisting it to the incremental version information library, which can ensure the consistency of the specified data in the memory cache of the read node and the specified data in the data source when the service is restarted, which provides a basis for the accuracy of the associated query.

[0050] In this embodiment, the construction process of the basic cache file can be executed by any one or more write nodes 12. The write node 12 can periodically construct the basic cache file according to the specified data in the data source according to the construction cycle of the basic cache file, and send the basic cache file to the read node 11, so that the read node 11 can store the basic cache file in the local persistent storage medium. Correspondingly, the read node 11 stores the basic cache file sent by the write node 12 in the local persistent storage medium. Furthermore, when the write node 12 learns that the specified data in the data source has changed, for example, after receiving the data source update notification sent by the application system, it can also persist the change information of the data source to the incremental version information library 13.

[0051] Based on the above, the reading node 11 can load the change information of the basic cache file in the local persistent storage medium and the specified data in the incremental version information library each time it is started, and build the specified data in the local memory cache according to the loaded basic cache file and the change information of the specified data. Optionally, the change information of the loaded basic cache file and the specified data in the data source can be mapped to the memory cache, so that the data stored in the local memory cache of the reading node 11 can be consistent with the specified data in the data source, providing a data basis for performing associated queries in the local memory cache of the reading node 11.

[0052] In addition to providing an association query service based on the memory cache, the distributed cache system provided in the embodiment of the present application can also adaptively update the memory cache according to the update of the data source in order to ensure the accuracy of the association query results based on the memory cache.

[0053] In this embodiment, the application system can update the specified data in the data source. The application system here refers to a system that has the modification authority for the specified data of the data source. For example, when a new field is generated in the application system, the data of the new field can be written to the data source; for another example, for the same field, the application system may generate a new value; for another example, the application system may delete certain fields. After the application system updates the data in the data source, the application system can initiate a data source update notification to the write node 12, and the data source update notification includes the change information of the specified data in the data source. The change information is some information that can reflect the update status of the data source. Optionally, the change information of the specified data in the data source includes but is not limited to: the target data that has changed in the data source, the way the target data has changed (referred to as the change mode of the target data), the identifier of the table item where the target data is located, the primary key ID, etc. Among them, the target data can be: a row of data in the table item, some rows of data, or the data in the entire table item. The change mode of the target data, that is, the update mode of the data source, includes but is not limited to: the data row is replaced, the data row is deleted, a new data row is added to the table item, the entire table item is replaced, deleted or a new table item is added, etc., but is not limited to this.

[0054] The data source update notification sent by the application system will be distributed to any write node 12. For any write node 12, after receiving the data source update notification sent by the application system, it can notify the read node 11 to update the specified data in its memory cache according to the change information of the specified data contained in the data source update notification.

[0055] Optionally, the write node 12 can maintain an online read node list based on the initial read node list. Wherein, the initial read node list stores information such as the identification of each read node 11 stored in the distributed cache system, and the online read node list stores the identification information of the online read nodes. In an embodiment of the present application, the write node 12 can send a heartbeat message to the read node 11 according to a set cycle. If the read node 11 is online, it will receive the heartbeat message and return a heartbeat response message to the write node 12. The write node 12 receives the heartbeat response message, which read nodes 11 are online, and the identification information of these online read nodes is added or retained in the online read node list. If the read node fails to receive the heartbeat message sent by the write node 12 or fails during the heartbeat response message, the write node 12 cannot receive the heartbeat response message of these faulty read nodes, and the write node 12 can consider these read nodes to be temporarily offline according to a certain strategy, delete the identification information of these read nodes from the online read node list, and no longer send the change information of the specified data in the data source to these faulty read nodes in the future.

[0056] Furthermore, for a read node that is temporarily offline, the write node 12 will periodically detect whether it is back online through the above-mentioned heartbeat mechanism. If the read node is back online, the write node 12 will add its identification information to the online read node list; if a read node needs to be permanently offline, the information of the read node can be completely deleted from the initial read node list.

[0057] Based on the online read node list maintained by the write node 12, the write node 12 can determine the online read node according to the maintained online read node list; and send the change information of the specified data to the online read node.

[0058] Optionally, if there are multiple online read nodes 11 , the write node 12 may concurrently broadcast the change information of the specified data in the data source to the online read nodes 11 , thereby improving the transmission efficiency.

[0059] Optionally, the write node 12 may send a data source update notification to an online read node 11, and the read node 11 receives the data source update notification, parses the change information of the specified data in the data source, and updates the specified data in the data source stored in the memory cache according to the change information of the specified data in the data source. Alternatively, the write node 12 parses the change information of the specified data in the data source from the data source update notification, and then sends the change information of the specified data in the data source to the read node 11; the read node 11 receives the change information of the specified data in the data source, and updates the specified data in the data source stored in the memory cache according to the change information of the specified data in the data source.

[0060] Optionally, the change information of the specified data in the data source includes but is not limited to: the target data that has changed in the data source, the change method of the target data, the identifier of the table item where the target data is located, the primary key ID, etc., but is not limited to this. In this case, the process of the read node 11 updating the specified data of the data source stored in the memory cache according to the change information of the specified data in the data source includes: determining the target table item where the target data is located in the memory cache according to the identifier of the table item where the target data is located; and determining the target data row in the target table item according to the primary key ID of the target data; and then updating the target data row according to the change method of the target data.

[0061] Further, in the case where the memory cache of the read node 11 is divided into an index area and a data area, if the target data changes in the form of adding data, the read node 11 updates the target data row according to the change mode of the target data, including: if the table entry where the target data is located is a first-category table entry, it is only necessary to add a primary index with the primary key ID of the target data as the key and the target data as the value to the index area; if the table entry where the target data is located is a second-category table entry, it is necessary to add a primary index with the primary key ID of the target data as the key and the data area pointer corresponding to the target data as the value to the index area, and add the target data to the target data row in the target table entry in the data area. Further, if the change information of the specified data in the data source includes the attribute information of the target data, then regardless of whether the table entry where the target data is located is a first-category table entry or a second-category table entry, an attribute index with the attribute value of the target data as the key and the primary key ID of the target data as the value can be added to the index area.

[0062] If the target data is changed in such a way that the data is deleted, the read node 11 updates the target data row according to the target data change mode, including: if the table entry where the target data is located is a first-class table entry, the primary index in the index area with the primary key ID of the target data as the key and the target data as the value is deleted; if the table entry where the target data is located is a second-class table entry, the primary index in the index area with the primary key ID of the target data as the key and the data area pointer of the target data as the value is deleted, and the attribute index with the attribute value as the key and the primary key ID of the target data as the value of the attribute to be indexed is deleted, and the target data in the data memory area pointed to by the primary index pointer in the data area is deleted at the same time. If the index area also includes an attribute index, the attribute index with the attribute value of the target data as the key and the primary key ID of the target data as the value can be deleted from the index area regardless of whether the table entry where the target data is located is a first-class table entry or a second-class table entry.

[0063] If the target data is changed in such a way that the original data in the target data row is replaced, the read node 11 updates the target data row according to the target data change mode, including: if the table entry where the target data is located is a first-category table entry, the data content corresponding to the primary key ID of the target data in the index area is replaced with the target data; if the table entry where the target data is located is a second-category table entry, it is necessary to first find the data area pointer corresponding to the cached data item in the primary index through the primary key ID of the target data, and find the target data row in the data area according to the data area pointer, and then replace the data in the target data row with the target data. Further, if the index area also includes an attribute index, no matter whether the table entry where the target data is located is a first-category table entry or a second-category table entry, the attribute index in the index area with the attribute value of the target data as the key and the primary key ID of the target data as the value can be updated.

[0064] In another optional embodiment, in order to ensure that the read node 11 can generate the specified data of the data source based on the basic snapshot data and incremental version data of the data source when it is started next time, that is, when the read node 11 is restarted next time, the specified data of the data source in the memory cache is consistent with the specified data in the current data source, the write node 12 also persists the change information of the specified data in the data source as incremental version data to the centrally stored incremental version data source, that is, the write node 12 can persist the target data, the change method of the target data (addition, update, deletion), the target data version, the identifier of the table item where the target data is located, and the primary key ID to the incremental version data source.

[0065] Furthermore, in order to ensure that in the scenario where data is updated concurrently and out of order in a distributed environment, that is, in some application scenarios, the data source later update notification may arrive at the write node 12 earlier than the data source earlier update notification, in order to avoid the problem of cached data being overwritten and updated by mistake, a data version control mechanism is introduced in the embodiment of the present application. Each target data has a version field, which is used to write the version number of the target data. The version number corresponding to each data change is automatically incremented (which can be achieved through the data source). Accordingly, when the write node 12 persists the change information of the specified data in the data source as incremental version data to the incremental version data source cache update, the version number of the incremental data is also persisted to the incremental version data source.

[0066] Accordingly, when the read node 11 updates the memory cache according to the change information of the specified data in the data source, the version number of the target data can be compared with the version number of the original data in the target data row in the memory cache. If the version number of the target data is greater than the version number of the original data in the target data row, the target data row will be updated according to the change mode of the target data. Otherwise, the data source update notification is deemed to be expired and invalid.

[0067] The distributed cache system provided in the embodiment of the present application can be applied to various scenarios involving database association queries, for example, it can be applied to a data center. A data center may have millions of servers, and these servers have various operation and maintenance and configuration data, such as a series of related information about hardware and software assets (commonly referred to as configuration data) in a company or organization and the relationships between them. These data can be stored in a configuration management database (Configure Management Database, CMDB). Based on CMDB, a data center and its operation and maintenance system can query various information. For example, the operation and maintenance system can query server information according to applications and computer rooms, but the query process requires table items of dimensions such as associated applications, computer rooms, and servers, which involves associated queries. In order to improve the efficiency of associated queries, a distributed cache system is added in the data center of this embodiment.

[0068] Figure 1bThis is a schematic diagram of the structure of the data center network provided in the embodiment of the present application. Figure 1b As shown, the data center network includes: multiple network devices 10a, a database 10b and a distributed cache system 10c. The distributed cache system 10c includes a read node and a write node. The implementation form of the distributed cache system 10c can be found in the above embodiment. Figure 1a The relevant content will not be repeated here.

[0069] In this embodiment, the database 10b is used to store data related to multiple network devices 10a. These data include configuration data of these network devices 10a, the computer room they belong to, the applications installed on the network devices 10a, the device group information to which the network devices 10a belong, and metadata, but are not limited thereto. Due to the different services of the data center, the data stored in the database 10b is also different. Optionally, the database 10b can be a CMDB, which is used to store a series of information on hardware and software assets in a company or organization and the relationship between them.

[0070] In this embodiment, the number of read nodes and write nodes in the distributed cache system 10c can be one or more. The implementation form can be seen in the above Figure 1a The relevant contents of the read node 11 and the write node 12 are not repeated here.

[0071] In this embodiment, each reading node has its own memory cache, and the memory cache of each reading node stores the specified data in the database. The specified data in the database 10b stored in the memory cache of the reading node is a snapshot, mirror or copy of the specified data in the database 10b. The specified data in the database 10b can be part of the data in the database 10b, or it can be the full data in the database 10b. Further, the specified data includes but is not limited to: metadata files, index files, and data files. In this way, the reading node can provide external associated query operations based on the specified data in the memory cache, and the associated query operations mainly refer to associated query operations that depend on the specified data. For the description of metadata files and data files, please refer to the relevant content in the above embodiments, which will not be repeated here. For example, for the application scenario where the above database 10b is CMDB, the memory cache of the reading node in the distributed cache system 10c stores the specified data in the CMDB, which can be provided to the operation and maintenance system ( Figure 1b (not shown) provides associated query operations. For example, the operation and maintenance system can query server information according to applications and computer rooms. The query process can be performed on the reading node. The reading node can query the table items of the associated application, computer room, server and other dimensions based on the specified data in the CMDB stored in the memory cache, and finally obtain the information of the server that has a certain application installed and belongs to a certain computer room.

[0072] In this embodiment, the memory cache of the reading node stores the specified data in the database 10b, and the associated query operation related to the specified data can be performed in the memory cache of the reading node without going to the database 10b, which helps to improve the efficiency of the associated query. In addition, in this embodiment, the memory cache of the reading node is based on the memory implementation and supports the memory index structure, which can further improve the table lookup performance during the associated query, and further improve the efficiency of the associated query.

[0073] Furthermore, when the specified data in the database 10b changes, the write node can notify the read node to update the specified data stored in its local memory cache. Correspondingly, the read node can also update the specified data in its memory cache according to the notification of the write node. Among them, when the specified data in the database 10b changes, the write node notifies the read node to update its memory cache, which can provide a basis for the read node to provide external association query operations, which is conducive to ensuring the accuracy and real-time performance of the association query results based on the memory cache.

[0074] It should be noted that the specific implementation form of the distributed cache system, the implementation method of the read node providing external related queries, and the specific implementation method of the read node and the write node cooperating with each other to implement the read node cache update can be found in the above Figure 1a The relevant contents in will not be repeated here.

[0075] In addition to the above-mentioned distributed cache system and data center network embodiments, the embodiments of the present application also provide an association query and data update method based on a distributed cache system, which is exemplified below from the perspectives of the read node and the write node in the distributed cache system.

[0076] Figure 2 The present invention provides a flowchart of an association query method based on a distributed cache system. The method is applicable to a read node in a distributed cache system. Figure 2 As shown, the method includes:

[0077] 201. Receive an associated query request, where the associated query request includes a filtering condition and a query item.

[0078] 202. Parse the to-be-associated item, the target item, and the association path between the to-be-associated item and the target item from the filtering condition.

[0079] 203. Perform an association query in the local memory cache according to the association path and query item between the table item to be associated and the target table item; wherein the local memory cache stores designated data in the data source, and the designated data includes the table item to be associated and the target table item.

[0080] The distributed cache system includes at least one read node, and the execution subject of this embodiment can be any of the read nodes. The read node can be a functional module or program code that provides a data source query function for users, which can be deployed on a physical machine or in a container or virtual machine (VM) on a physical machine.

[0081] In this embodiment, each reading node has its own memory cache, and the memory cache of the reading node stores the specified data in the data source. Among them, the specified data in the data source stored in the local memory cache of the reading node is a backup of the specified data in the data source. Further, the specified data in the data source can be partial data in the data source, or it can be the full amount of data in the data source. Further, the specified data of the data source includes but is not limited to: metadata files, index files, data files, etc. Optionally, metadata information such as table item names, table item attributes, attribute types, etc. are stored in the metadata file; specific instance data are stored in the data file, and these specific instance data can be encoded in bytes.

[0082] Optionally, the memory cache of the read node is implemented based on memory, and can support a memory index structure, which can further improve the table lookup performance during associated queries, and further improve the efficiency of associated queries.

[0083] In this embodiment, the reading node receives an association query request, which includes the demand description information of the association query. Further, the reading node can determine the to-be-associated table items and the association path between the to-be-associated table items according to the demand description information of the association query. The to-be-associated table items refer to other table items other than the target table items that need to be associated in the association query, and the number can be 1 or more. Further, the reading node performs an association query in the local memory cache according to the association path between the to-be-associated table items and the target table items and the above-mentioned query items. In an embodiment of the present application, the specified data stored in the local memory cache of the reading node includes the to-be-associated table items and the target table items. Since the memory cache of the reading node stores the specified data of the data source, the specified data includes metadata files, and the association relationship between different table items is contained in these metadata. At the same time, when the data source is updated, the memory cache of each reading node will be updated synchronously in real time, which provides a data basis for the association query.

[0084] In this embodiment, the memory cache of the reading node stores the specified data in the data source. In this way, when performing an associated query with the specified data in the data source, it can be performed in the local memory cache of the reading node without querying the data source, thereby improving the efficiency of the associated query.

[0085] On the other hand, in an embodiment of the present application, the read nodes are independent of each other and all store designated data of the data source. The number of read nodes can be flexibly increased, and by increasing the number of read nodes, horizontal expansion of storage capacity can be achieved, thereby improving the overall query service capacity of the distributed cache system.

[0086] In an optional embodiment, when a user performs an associated query, an associated query request may be initiated to the distributed cache system. The user may input a corresponding query statement to the distributed cache system using a data source language supported by the distributed cache system, thereby initiating an associated query request to the distributed cache system. Optionally, the distributed cache system of this embodiment may support LQL, SQL, or DSL, etc., but is not limited thereto. The query statement includes filter conditions and query items for the associated query. The description language of the associated query request conforms to the grammatical format of the distributed cache system.

[0087] Based on the above-mentioned association query request, the reading node performs syntax analysis on the association query request, and can parse out the filter conditions and query items of the association query. Optionally, the association query request can be parsed using a syntax analyzer such as Antlr, Yacc, etc., but is not limited thereto. Accordingly, an optional implementation method of step 202 is: parse out the table item to be associated, the target table item, and the association path between the table item to be associated and the target table item from the filtering condition; and calculate the association path between the table item to be associated and the target table item according to the primary key and foreign key relationship between the table item to be associated and the target table item. Afterwards, the reading node 11 performs an association query in the local memory cache according to the association path and the query item. Among them, the primary key is an identifier that can uniquely identify a record in a table item; the foreign key is one or more columns that establish or strengthen the link between two table items, and the one or more columns here are the primary key in one table item and the foreign key in another table item.

[0088] Among them, the association path between the to-be-associated table item and the target table item may be one or more. If there are multiple association paths between the to-be-associated table item and the target table item, the reading node may select the shortest association path from the multiple determined association paths as the target association path; and perform an association query in the specified data of the data source stored in the memory cache according to the target association path and the query item to obtain the query result. Querying according to the shortest association path can further improve the query efficiency to a certain extent.

[0089] Furthermore, in order to enable the distributed cache system to more effectively utilize the memory of the physical machine where the read node is located, the memory cache of the read node can be divided into an index area and a data area. The index area is used to store indexes in large table entries and small table entries; the data area is used to store data in large table entries. For example, when the read node is implemented as a Java Virtual Machine (JVM), the index area can be implemented using the in-heap memory of the read node, and the data area can be implemented using the off-heap memory of the read node. The off-heap memory is the memory outside the JVM heap, which can more efficiently utilize the overall memory of the machine.

[0090] Optionally, small table items and large table items may be divided according to actual storage requirements, wherein a small table item refers to an item whose item capacity is less than a set capacity threshold, and a large table item refers to an item whose item capacity is greater than or equal to a set capacity threshold. The capacity threshold may be flexibly set according to actual storage requirements and is not limited here. For ease of description and distinction, in the embodiment of the present application, a small table item is defined as a first-category item, and a large table item is defined as a second-category item.

[0091] Accordingly, the reading node may store the specified data in the data source in the memory cache in the following manner: the index of the first type of table item in the second type of table item in the specified data is stored in the index area, and the data of the second type of table item is stored in the data area. Among them, a table item includes a table item index and data, and the table item index includes: table item identifier, primary key ID, foreign key, etc., but is not limited thereto.

[0092] Based on the above-mentioned index area and data area, an optional implementation method of step 203 is: query in the index area according to the association path between the table item to be associated and the target table item, and obtain the list of primary key IDs in the target table item that meet the data screening conditions. Further, if the target table item is a first-category table item, the read node obtains the data corresponding to each primary key ID in the primary key ID list that meets the data screening conditions from the index area; if the target table item is a second-category table item, the read node obtains the data corresponding to each primary key ID in the primary key ID list that meets the data screening conditions from the data area. Further, the read node assembles the data corresponding to each primary key ID in the primary key ID list according to the query items in the associated query request to obtain the associated query result. Afterwards, the read node can output the associated query result to return it to the user. Among them, the data screening condition is a part of the filtering condition, which is used to describe the information of the data that needs to be screened for this associated query.

[0093] It should be noted that in different application scenarios, users can set different filtering conditions and different query items according to their query needs. Optionally, the filtering conditions may include the item identifiers of the items to be associated and the target items, the primary key ID list of the items to be associated and the target items, and the foreign keys, but are not limited thereto.

[0094] In an optional embodiment, the read node also includes: a local persistent storage medium. The local persistent storage medium can be a disk, a solid-state drive, a disk array, etc. The local persistent storage medium of the read node stores a snapshot, mirror, or copy of the specified data in the data source when the most recent build cycle arrives. For ease of description and distinction, the snapshot, mirror, or copy of the specified data in the data source when the most recent build cycle arrives stored in the local persistent storage medium of the read node is defined as a basic cache file. The purpose of storing the basic cache file is to enable the read node to build the specified data in the memory cache each time it starts. The basic cache file is a data foundation required for the read node to build the memory cache at startup. In addition, in this embodiment, the basic cache file can be periodically constructed according to the specified data in the data source according to the set build cycle.

[0095] Furthermore, the distributed cache system provided in the embodiment of the present application also includes an incremental version information library. The incremental version information library is mainly used to store the change information of the specified data in the data source during the construction of two basic cache files. The changes here include the newly added, updated, and deleted data in the specified data. The incremental version information library can be implemented using an existing relational database, a non-relational database, etc. The change information of the specified data is another data basis required for the reader node to build a memory cache at startup. In this embodiment, the change information of the basic cache file and the specified data is stored separately, which can ensure that the reader node can efficiently update the memory cache and maintain the consistency of the memory cache of all reader nodes. In addition, the change information of the specified data is uniformly managed by persisting it to the incremental version information library, which can ensure the consistency of the specified data in the memory cache of the read node and the specified data in the data source when the service is restarted, which provides a basis for the accuracy of the associated query.

[0096] In this embodiment, the construction process of the basic cache file can be executed by any one or more write nodes. The write node can periodically construct the basic cache file according to the specified data in the data source according to the construction cycle of the basic cache file, and send the basic cache file to the read node so that the read node can store the basic cache file in the local persistent storage medium. Correspondingly, the read node stores the basic cache file sent by the write node in the local persistent storage medium. Furthermore, when the write node learns that the specified data in the data source has changed, for example, after receiving the data source update notification sent by the application system, it can also persist the change information of the data source to the incremental version information library.

[0097] Based on the above, each time the reader node is started, it can load the change information of the basic cache file in the local persistent storage medium and the specified data in the incremental version information library, and build the specified data in the local memory cache according to the loaded basic cache file and the change information of the specified data. Optionally, the change information of the loaded basic cache file and the specified data in the data source can be mapped to the memory cache, so that the data stored in the local memory cache of the reader node can be consistent with the specified data in the data source, providing a data basis for performing associated queries in the local memory cache of the reader node.

[0098] Furthermore, in order to keep the specified data in the data source stored in the memory cache of the read node consistent with the specified data in the data source, when the specified data in the data source is updated, the specified data stored in the memory cache of the read node needs to be updated in a timely manner. The following is an exemplary description of the updating process of the specified data in the data source stored in the memory cache of the read node.

[0099] Figure 3 The present invention provides a flow chart of an update method based on data source cache. The method is applicable to a read node in a distributed cache system. Figure 3 As shown, the method includes:

[0100] 301. Receive change information of specified data in a data source sent by a write node in a distributed cache system.

[0101] 302. Update the specified data in the local memory cache according to the change information of the specified data in the data source.

[0102] In the present embodiment, the read nodes all have their own memory cache, and the memory cache of the read nodes stores the specified data in the data source. In the present embodiment, after the write node sends the change information of the specified data in the data source to the read node, the read node receives the change information. Optionally, if the write node directly sends the change information of the specified data in the data source to the read node, the read node directly receives the change information of the specified data in the data source. If the write node sends a data source update notification to the read node, the read node receives the data source update notification and parses out the change information of the specified data in the data source; thereafter, the read node updates the specified data of the data source in the memory cache according to the change information of the specified data in the data source. Among them, the description of the specified data of the data source can be found in the relevant content in the above embodiment, which will not be repeated here.

[0103] In this embodiment, the reading node timely updates the specified data in the data source in the local memory cache according to the change information of the specified data in the data source. In this way, the specified data in the data source in the memory cache of the reading node can be kept consistent with the specified data in the data source, and then when performing an associated query, it can be performed in the memory cache of the reading node without querying the data source, which can improve the efficiency of the associated query.

[0104] Optionally, the change information of the specified data in the data source includes but is not limited to: the target data changed in the data source, the change mode of the target data, the identifier of the table item where the target data is located, the primary key ID, etc. Then an implementation of step 302 includes: determining the target table item where the target data is located in the memory cache according to the identifier of the table item where the target data is located; and determining the target data row in the target table item according to the primary key ID of the target data; and then, updating the target data row according to the change mode of the target data.

[0105] Furthermore, the memory cache of the read node may be divided into an index area and a data area. For the description of the index area and the data area, please refer to the relevant content of the above embodiment.

[0106] If the target data is changed by adding target data, the read node updates the target data row according to the target data change method, including: if the table item where the target data is located is a first-category table item, it is only necessary to add a primary index with the primary key ID of the target data as the key and the target data as the value to the index area; if the table item where the target data is located is a second-category table item, it is necessary to add a primary index with the primary key ID of the target data as the key and the data area pointer corresponding to the target data as the value to the index area, and add the target data to the target data row in the target table item in the data area. Furthermore, if the change information of the specified data in the data source includes the attribute information of the target data, then regardless of whether the table item where the target data is located is a first-category table item or a second-category table item, an attribute index with the attribute value of the target data as the key and the primary key ID of the target data as the value can be added to the index area.

[0107] If the target data is changed in such a way that the target data is deleted, the read node updates the target data row according to the target data change mode, including: if the table entry where the target data is located is a first-class table entry, the primary index in the index area with the primary key ID of the target data as the key and the target data as the value is deleted; if the table entry where the target data is located is a second-class table entry, the primary index in the index area with the primary key ID of the target data as the key and the data area pointer of the target data as the value is deleted, and the attribute index with the attribute value as the key and the primary key ID of the target data as the value of the attribute to be indexed is deleted, and the target data in the data memory area pointed to by the primary index pointer in the data area is deleted at the same time. If the index area also includes an attribute index, the attribute index with the attribute value of the target data as the key and the primary key ID of the target data as the value can be deleted from the index area regardless of whether the table entry where the target data is located is a first-class table entry or a second-class table entry.

[0108] If the target data is changed in such a way that the original data in the target data row is replaced, the read node updates the target data row according to the target data change method, including: if the table item where the target data is located is a first-category table item, the data content corresponding to the primary key ID of the target data in the index area is replaced with the target data; if the table item where the target data is located is a second-category table item, it is necessary to first find the data area pointer corresponding to the cached data item in the primary index through the primary key ID of the target data, and find the target data row in the data area according to the data area pointer, and then replace the data in the target data row with the target data. Furthermore, if the index area also includes an attribute index, whether the table item where the target data is located is a first-category table item or a second-category table item, the attribute index in the index area with the attribute value of the target data as the key and the primary key ID of the target data as the value can be updated.

[0109] Furthermore, in order to ensure that in the scenario where data is updated concurrently and out of order in a distributed environment, that is, in some application scenarios, the data source's later update notification may reach the write node earlier than the data source's earlier update notification, in order to avoid the problem of cached data being overwritten and updated by mistake, a data version control mechanism is introduced in the embodiment of the present application. Each target data has a version field, which is used to write the version number of the target data. The version number corresponding to each data change is automatically incremented (which can be achieved through the data source). Accordingly, when the write node persists the change information of the specified data in the data source as incremental version data to the incremental version data source cache update, the version number of the incremental data is also persisted to the incremental version data source.

[0110] Accordingly, when the reading node updates the memory cache according to the change information of the specified data in the data source, the version number of the target data can be compared with the version number of the original data in the target data row in the memory cache. If the version number of the target data is greater than the version number of the original data in the target data row, the target data row will be updated according to the change mode of the target data. Otherwise, this data source update notification is considered to be expired and invalid.

[0111] Accordingly, the embodiment of the present application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned association query method and / or the method for updating data by a reading node.

[0112] Figure 4 A flow chart of another data source update method provided in an embodiment of the present application. The method is applicable to a write node in a distributed cache system. Figure 4 As shown, the method includes:

[0113] 401. Receive a data source update notification sent by an application system, where the data source update notification includes change information of specified data in the data source.

[0114] 402. Send the change information of the specified data in the data source to the reading node in the distributed cache system, so that the reading node updates the specified data in the data source in the local memory cache according to the change information.

[0115] There may be one or more write nodes in the distributed cache system, and the execution subject of this embodiment may be any of the write nodes. The write node may be deployed on a physical machine, or in a container or a virtual machine.

[0116] In this embodiment, the application system can update the specified data in the data source. The application system here refers to a system that has modification authority for the specified data in the data source. After the application system updates the data in the data source, the application system can initiate a data source update notification to the write node, and the data source update notification includes the change information of the specified data in the data source. The change information is some information that can reflect the update status of the data source. Optionally, the change information of the specified data in the data source includes but is not limited to: the target data of the specified data in the data source, the change mode of the target data, the identifier of the table item where the target data is located, the primary key ID, etc. Among them, the target data can be: a row of data in the table item, some rows of data, or the data in the entire table item. The change mode of the target data, that is, the update mode of the specified data in the data source, includes but is not limited to: the data row is replaced, the data row is deleted, a new data row is added to the table item, the entire table item is replaced, deleted or a new table item is added, etc., but is not limited to this.

[0117] Accordingly, in step 401, the write node receives the data source update notification, and sends the change information of the specified data in the data source to the read node. Optionally, the write node may send the data source update notification to the read node; the read node receives the data source update notification, and parses the change information of the specified data in the data source; then, the read node updates the specified data in the data source in the memory cache according to the change information of the specified data in the data source. Alternatively, the write node parses the change information of the specified data in the data source from the data source update notification, and sends the change information of the specified data in the data source to the read node; the read node receives the change information of the specified data in the data source, and updates the specified data in the data source in the memory cache according to the change information of the specified data in the data source.

[0118] In this embodiment, the write node sends the change information of the specified data in the data source to the read node, so that the read node updates the specified data in the data source stored in the memory cache according to the change information. In this way, the specified data of the data source in the memory cache of the read node can be kept consistent with the specified data in the data source, and then when performing an associated query related to the specified data in the data source, it can be performed in the memory cache of the read node, which provides data support for cache-based associated queries and is conducive to ensuring the accuracy of the associated query results based on the memory cache.

[0119] In an optional embodiment, the write node may maintain an online read node list based on the initial read node list. The initial read node list stores information such as the identification of each read node in the distributed cache system, and the online read node list stores identification information of the online read nodes. For the description of the maintenance strategy of the write node for the online read node list, please refer to the relevant content in the above-mentioned system embodiment, which will not be repeated here. Based on the online read node list maintained by the above-mentioned write node, an optional implementation method of step 402 is: determine the read node in the online state according to the maintained online read node list; and send the change information of the specified data to the read node in the online state.

[0120] Optionally, if there are multiple online read nodes, the write node may concurrently broadcast the change information of the specified data in the data source to the online read nodes.

[0121] The write node has access rights to the incremental version information library of the distributed cache system. The write node is used to write data to the incremental version information library when the cache is built and the specified data in the data source is updated. Optionally, when the cache is built, the write node can periodically build a basic cache file according to the specified data in the data source according to the construction cycle of the basic cache file, and send the basic cache file to the read node so that the read node can store the basic cache file in a local persistent storage medium. Further, when the specified data in the data source is updated, the write node persists the change information of the specified data to the incremental version information library after receiving the data source update notification. Storing the basic data file and the incremental version data (change information of the specified data) in the data source separately can ensure that the read node can efficiently update the memory cache and maintain the complex distributed problem of file consistency in the memory cache of all read nodes. The incremental version data is uniformly managed by persisting it to the incremental version information library, which can ensure the consistency of the specified data in the memory cache of the read node and the specified data in the data source when the service is restarted, which provides the basis for the accuracy of the associated query.

[0122] Furthermore, in order to ensure that in the scenario where data is updated concurrently and out of order in a distributed environment, that is, in some application scenarios, the data source's later update notification may reach the write node earlier than the data source's earlier update notification, in order to avoid the problem of cached data being overwritten and updated by mistake, a data version control mechanism is introduced in the embodiment of the present application. Each target data has a version field, which is used to write the version number of the target data. The version number corresponding to each data change is automatically incremented (which can be achieved through the data source). Accordingly, when the write node persists the change information of the specified data in the data source as incremental version data to the incremental version data source cache update, the version number of the incremental data is also persisted to the incremental version data source.

[0123] Correspondingly, an embodiment of the present application also provides a computer-readable storage medium storing computer instructions, characterized in that when the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the above-mentioned method of writing nodes to update data.

[0124] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 201-203 can be device A; for another example, the execution subject of step 201 can be device A, and the execution subject of step 202 can be device B; and so on. In addition, in some processes described in the above embodiments and the accompanying drawings, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 401, 402, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0125] Figure 5 FIG. 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device can be used as a read node in the distributed cache system shown in FIG. Figure 5 As shown, the electronic device includes: a memory 50a, a processor 50b and a communication component 50c.

[0126] The memory 50a includes a memory cache and other storage spaces, wherein the memory cache stores the specified data in the data source, and the other storage spaces are used to store computer programs. The specified data in the data source may be part of the data in the data source, or may be the full amount of data in the data source.

[0127] The processor 50b is coupled to the memory 50a, and is used to execute a computer program for: receiving an association query request through the communication component 50c, the association query request including a filter condition and a query item; parsing a to-be-associated table item, a target table item, and an association path between the to-be-associated table item and the target table item from the filter condition; performing an association query in a local memory cache according to the association path and the query item between the to-be-associated table item and the target table item; wherein the specified data includes the to-be-associated table item and the target table item.

[0128] In an optional embodiment, when the processor 50b parses the to-be-associated table item, the target table item, and the association path between the to-be-associated table item and the target table item from the filtering conditions, it is specifically used to: parse the to-be-associated table item and the target table item from the filtering conditions; and calculate the association path between the to-be-associated table item and the target table item based on the primary key and foreign key relationship between the to-be-associated table item and the target table item.

[0129] In another optional embodiment, the designated data includes: first-category table items and second-category table items. The memory cache includes: an index area and a data area. The index area stores indexes in the second-category table items and first-category table items, and the data area stores data in the second-category table items; the first-category table items refer to table items whose table item capacity is less than a set threshold, and the second-category table items refer to table items whose table item capacity is greater than or equal to the set threshold.

[0130] Furthermore, when the processor 50b performs an associative query in the local memory cache, it is used to: perform a query in the index area according to the associative path to obtain a list of primary key IDs in the target table item that meet the data screening conditions; if the target table item is a first-category table item, obtain the data corresponding to each primary key ID in the primary key ID list from the index area; if the target table item is a second-category table item, obtain the data corresponding to each primary key ID in the primary key ID list from the data area; thereafter, according to the query item, assemble the data corresponding to each primary key ID in the primary key ID list to obtain the associative query result.

[0131] In another optional embodiment, other storage spaces in the memory 50a besides the memory cache also include: a persistent storage medium. The persistent storage medium stores a basic cache file. The basic cache file is a snapshot or backup of the specified data in the data source when the current cache is constructed. Accordingly, the processor 50b is also used to: load the basic cache file in the local persistent storage medium and the change information of the specified data in the incremental version information library in the distributed cache system when the read node is started; and construct the specified data in the local memory cache according to the basic cache file and the change information of the specified data. The change information of the specified data is the change information of the specified data in the data source persisted to the incremental version information library by the write node in the distributed cache system after the timestamp of the basic cache file.

[0132] Further, when the processor 50b constructs the specified data in the local memory cache based on the basic cache file and the change information of the specified data, it is specifically used to: construct the first category table items and the second category table items in the specified data based on the basic cache file and the change information of the specified data; store the indexes in the second category table items and the first category table items in the index area, and store the data in the second category table items in the data area.

[0133] In another optional embodiment, the processor 50b is also used to: receive change information of specified data in the data source sent by the write node in the distributed cache system through the communication component 50c; and update the specified data in the local memory cache according to the change information of the specified data in the data source.

[0134] Optionally, when the processor 50b receives change information of specified data in the data source sent by the write node in the distributed cache system through the communication component 50c, it is specifically used to: receive the data source update notification sent by the write node through the communication component 50c, where the data source update notification is sent by the application system after updating the specified data of the data source; and parse the change information of the specified data in the data source from the data source update notification.

[0135] Optionally, the change information of the specified data in the data source includes: the target data that has changed in the data source, the change mode of the target data, the identifier of the table item where the target data is located, and the primary key ID. Accordingly, when the processor 50b updates the specified data in the local memory cache, it is specifically used to: determine the target table item where the target data is located from the local memory cache according to the identifier of the table item where the target data is located; determine the target data row in the target table item according to the primary key ID of the target data; and update the target data row according to the change mode of the target data.

[0136] Furthermore, the memory cache includes: an index area and a data area. Accordingly, if the target data is changed in the form of newly added data, when the processor 50b updates the target data row, it includes: if the target table item is a first-type table item, adding a primary index with the primary key ID as the key and the target data as the value to the target table item in the index area; if the target table item is a second-type table item, adding a primary index with the primary key ID as the key and the data area pointer corresponding to the target data as the value to the index area, and adding the target data to the target data row in the target table item in the data area; the data area pointer points to the target data row.

[0137] Further, if the target data is changed by deleting data, then when the processor 50b updates the target data row, it is specifically used to: if the target table entry is a first-category table entry, delete the primary index with the primary key ID as the key and the target data as the value in the target table entry in the index area; if the target table entry is a second-category table entry, delete the primary index with the primary key ID as the key and the data area pointer corresponding to the target data as the value in the index area, and delete the target data row from the target table entry in the data area.

[0138] Further, optionally, the target data includes: a version number of the target data. When updating the target data row, the processor 50b is specifically configured to: compare the version number of the target data with the version number of the data of the target data row in the memory cache; if the version number of the target data is greater than the version number of the data of the target data row, then update the target data row according to the change mode of the target data.

[0139] In some optional embodiments, such as Figure 5 As shown, the electronic device may further include optional components such as a power supply component 50d. Figure 5The components are shown schematically only, and do not mean that the electronic device must include Figure 5 The components shown do not necessarily mean that the electronic device can only include Figure 5 Components shown.

[0140] The electronic device provided in this embodiment can be used as a reading node in a distributed cache system, and the memory cache of the reading node stores the specified data in the data source. In this way, when performing an associated query related to the specified data in the data source, it can be performed in the memory cache of the reading node, which can improve the table lookup performance and further improve the efficiency of the associated query.

[0141] Figure 6 FIG. 1 is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. The electronic device can be implemented as a write node in the distributed cache system shown in FIG. Figure 6 As shown, the electronic device includes: a memory 60a, a processor 60b and a communication component 60c.

[0142] The memory 60a is used to store computer programs.

[0143] The processor 60b is coupled to the memory 60a, and is used to execute a computer program for: receiving a data source update notification sent by the application system through the communication component 60c, the data source update notification including change information of specified data in the data source; and sending the change information of the specified data to a read node in the distributed cache system through the communication component 60c, so that the read node can update the specified data in the local memory cache according to the change information.

[0144] In an optional embodiment, when the processor 60b sends the change information of the specified data to the reading node in the distributed cache system through the communication component 60c, it is specifically used to: send the data source update notification to the reading node through the communication component 60c, so that the reading node can parse the change information of the specified data from the data source update notification.

[0145] In another optional embodiment, the processor 60b is further configured to persist the change information of the specified data in the data source into the incremental version information repository in the distributed cache system.

[0146] In another optional embodiment, before sending the change information of the specified data to the read node in the distributed cache system through the communication component 60c, the processor 60b is also used to: determine the read node in the online state based on the maintained online read node list; and send the change information of the specified data to the read node in the online state through the communication component 60c.

[0147] Furthermore, if there are multiple online reading nodes, when the processor 60b sends the change information of the specified data to the online reading nodes through the communication component 60c, it is specifically used to: concurrently broadcast the change information of the specified data to multiple online reading nodes through the communication component 60c.

[0148] In an optional embodiment, the processor 60b is also used to: periodically construct a basic cache file based on specified data in the data source according to the construction cycle of the basic cache file, and send the basic cache file to the reading node so that the reading node can store the basic cache file in a local persistent storage medium.

[0149] In some optional embodiments, such as Figure 6 As shown, the electronic device may further include optional components such as a power supply component 60d. Figure 6 The components are shown schematically only, and do not mean that the electronic device must include Figure 6 The components shown do not necessarily mean that the electronic device can only include Figure 6 Components shown.

[0150] The electronic device provided in this embodiment can be used as a write node in a distributed cache system. The electronic device sends the change information of the specified data in the data source to the read node, so that the read node updates the specified data in the memory cache according to the change information of the specified data in the data source. In this way, the specified data in the memory cache of the read node can be kept consistent with the specified data in the data source, and then when performing an associated query related to the specified data of the data source, it can be performed in the memory cache of the read node, which can improve the table lookup performance and thus improve the efficiency of the associated query.

[0151] In an embodiment of the present application, the memory may be configured to store various other data to support operations on the electronic device. Among them, the processor may execute a computer program stored in the memory to implement the corresponding control logic. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0152] In an embodiment of the present application, the communication component is configured to facilitate wired or wireless communication between the electronic device and other devices. The electronic device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component can also be implemented based on a near field communication (NFC) module, a radio frequency identification (RFID) technology, an infrared data association (IrDA) technology, an ultra-wideband (UWB) technology, a Bluetooth (BT) technology, and other technologies.

[0153] In an embodiment of the present application, a power supply assembly is configured to provide power to various components of an electronic device. The power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.

[0154] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0155] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0157] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0159] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0161] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0163] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A distributed cache system, characterized in that: include: Read nodes and write nodes; The memory cache of the read node stores the specified data in the data source, and is used to provide an associated query operation externally according to the specified data in the memory cache, and to update the specified data in the memory cache according to the notification of the write node; The write node is used to notify the read node to update the specified data in its memory cache when the specified data in the data source changes, so that the read node can provide external related query operations; The specified data includes the table items to be associated and the target table items. The table items to be associated refer to the table items other than the target table items that need to be associated in the associated query; the table items to be associated and the target table items respectively include table item indexes and data. The associated query operation is to perform an associated query on the table item indexes and data in the table items to be associated and the target table items, and assemble the queried data to obtain the associated query result.

2. The system according to claim 1, characterized in that The write node is specifically used for: Receiving a data source update notification sent by an application system, wherein the data source update notification includes change information of specified data in the data source; The change information of the specified data in the data source is sent to the reading node, so that the reading node can update the specified data in its memory cache.

3. The system according to claim 2, characterized in that Also includes: Incremental version repository; The incremental version information repository stores the change information of the specified data; the persistent storage medium of the reading node stores the basic cache file; The reading node is also used to: load the basic cache file in the local persistent storage medium and the change information of the specified data in the incremental version information library at startup, and construct the specified data in the memory cache according to the basic cache file and the change information of the specified data.

4. The system according to claim 3, characterized in that The write node is also used to: After receiving the data source update notification, persisting the change information of the specified data to the incremental version information library; as well as According to the construction cycle of the basic cache file, the basic cache file is periodically constructed according to the specified data in the data source, and the basic cache file is sent to the reading node so that the reading node stores the basic cache file in the local persistent storage medium.

5. The system according to claim 3, characterized in that The memory cache of the read node includes: an index area and a data area; The read node is further used to: store the indexes in the second category table items in the specified data and the first category table items in the index area, and store the data in the second category table items in the specified data in the data area; The first type of table entries refers to table entries whose table entry capacity is less than a set threshold, and the second type of table entries refers to table entries whose table entry capacity is greater than or equal to the set threshold.

6. The system according to any one of claims 1 to 5, characterized in that: The read node is specifically used for: receiving an associated query request, wherein the associated query request includes a filtering condition and a query item; Parsing the to-be-associated table item, the target table item, and the association path between the to-be-associated table item and the target table item from the filtering condition; as well as According to the association path between the to-be-associated table item and the target table item and the query item, an association query is performed in the local memory cache, and the specified data in the memory cache includes the to-be-associated table item and the target table item.

7. A correlation query method, applicable to a read node in a distributed cache system, characterized in that: The method comprises: receiving an associated query request, wherein the associated query request includes a filtering condition and a query item; Parsing the to-be-associated table item, the target table item, and the association path between the to-be-associated table item and the target table item from the filtering condition; According to the association path between the table item to be associated and the target table item and the query item, an association query is performed in the local memory cache; wherein the local memory cache stores specified data in the data source, the specified data includes the table item to be associated and the target table item, the table item to be associated refers to other table items other than the target table item that need to be associated in the association query; the table item to be associated and the target table item respectively include a table item index and data, the association query is to perform an association query on the table item index and data in the table item to be associated and the target table item, and assemble the queried data to obtain the association query result.

8. The method according to claim 7, characterized in that The specified data includes: first category table items and second category table items; The memory cache of the read node includes: an index area and a data area; the index area stores indexes in the second-category table items and the first-category table items, and the data area stores data in the second-category table items; The first type of table entries refers to table entries whose table entry capacity is less than a set threshold, and the second type of table entries refers to table entries whose table entry capacity is greater than or equal to the set threshold.

9. The method according to claim 8, characterized in that The performing an association query in the local memory cache according to the association path between the to-be-associated table item and the target table item and the query item includes: Searching the index area according to the association path to obtain a list of primary key IDs in the target table items that meet the data screening condition; If the target table entry is a first type table entry, obtaining data corresponding to each primary key ID in the primary key ID list from the index area; If the target table entry is a second type table entry, obtaining data corresponding to each primary key ID in the primary key ID list from the data area; According to the query item, the data corresponding to each primary key ID in the primary key ID list is assembled to obtain an associated query result.

10. The method according to claim 8, characterized in that Also includes: When the reading node is started, it loads the basic cache file in the local persistent storage medium and the change information of the specified data in the incremental version information library in the distributed cache system; wherein the change information of the specified data is the change information of the specified data after the timestamp of the basic cache file, which is persisted in the incremental version information library by the writing node in the distributed cache system; The designated data in the local memory cache is constructed according to the basic cache file and the change information of the designated data.

11. The method according to claim 10, characterized in that The step of constructing the designated data in the local memory cache according to the basic cache file and the change information of the designated data includes: Constructing first-category table entries and second-category table entries in the specified data according to the basic cache file and the change information of the specified data; The indexes in the second-category table entries and the first-category table entries are stored in the index area, and the data in the second-category table entries are stored in the data area.

12. The method according to any one of claims 8 to 11, characterized in that: Also includes: Receiving change information of specified data in the data source sent by a write node in the distributed cache system; According to the change information of the specified data in the data source, the specified data in the local memory cache is updated.

13. The method according to claim 12, characterized in that The change information of the specified data in the data source includes: the target data that has changed in the data source, the change mode of the target data, the identifier of the table item where the target data is located, and the primary key ID; The updating of the specified data in the local memory cache according to the change information of the specified data in the data source includes: Determine, from the local memory cache, the target table entry where the target data is located, according to the identifier of the table entry where the target data is located; Determine the target data row in the target table item according to the primary key ID of the target data; The target data row is updated according to the change mode of the target data.

14. The method according to claim 13, characterized in that If the target data is changed in a manner that new data is added, updating the target data row according to the target data change manner includes: If the target table entry is a first-category table entry, a primary index with the primary key ID as a key and the target data as a value is added to the target table entry in the index area; If the target table entry is a second-category table entry, a primary index with the primary key ID as the key and the data area pointer corresponding to the target data as the value is added in the index area, and the target data is added to the target data row in the target table entry in the data area; the data area pointer points to the target data row; If the target data is changed in a manner of deleting data, updating the target data row according to the target data change manner includes: If the target table entry is a first-category table entry, deleting a primary index with the primary key ID as a key and the target data as a value from the target table entry in the index area; If the target table entry is a second type of table entry, the primary index with the primary key ID as the key and the data area pointer corresponding to the target data as the value is deleted in the index area, and the target data row is deleted from the target table entry in the data area.

15. A data update method, applicable to a write node in a distributed cache system, characterized in that: The method comprises: Receiving a data source update notification sent by an application system, wherein the data source update notification includes change information of specified data in the data source; Sending the change information of the specified data to the reading node in the distributed cache system, so that the reading node can update the specified data in the local memory cache according to the change information and provide external related query operations based on the specified data; The specified data includes the table items to be associated and the target table items. The table items to be associated refer to the table items other than the target table items that need to be associated in the associated query; the table items to be associated and the target table items respectively include table item indexes and data. The associated query operation is to perform an associated query on the table item indexes and data in the table items to be associated and the target table items, and assemble the queried data to obtain the associated query result.

16. An electronic device, which can be implemented as a read node in a distributed cache system, characterized in that: include: memory, processor, and communications components; The memory includes a memory cache and other storage spaces, wherein the memory cache stores designated data in the data source, and the other storage spaces are used to store computer programs; The processor is coupled to the memory and is configured to execute the computer program for: Receiving, through the communication component, an associated query request, wherein the associated query request includes a filtering condition and a query item; Parsing the to-be-associated table item, the target table item, and the association path between the to-be-associated table item and the target table item from the filtering condition; According to the association path between the table item to be associated and the target table item and the query item, an association query is performed in the local memory cache; wherein the specified data includes the table item to be associated and the target table item, and the table item to be associated refers to other table items other than the target table item that need to be associated in the association query; the table item to be associated and the target table item respectively include a table item index and data, and the association query is to perform an association query on the table item index and data in the table item to be associated and the target table item, and assemble the queried data to obtain the association query result.

17. An electronic device, which can be implemented as a write node in a distributed cache system, characterized in that: include: Memory, processor and communication components; wherein the memory is used to store computer programs; The processor is coupled to the memory and is configured to execute the computer program for: Receiving, through the communication component, a data source update notification sent by the application system, wherein the data source update notification includes change information of specified data in the data source; Sending the change information of the specified data to the reading node in the distributed cache system through the communication component, so that the reading node can update the specified data in the local memory cache according to the change information and provide external related query operations based on the specified data; The specified data includes the table items to be associated and the target table items. The table items to be associated refer to the table items other than the target table items that need to be associated in the associated query; the table items to be associated and the target table items respectively include table item indexes and data. The associated query operation is to perform an associated query on the table item indexes and data in the table items to be associated and the target table items, and assemble the queried data to obtain the associated query result.

18. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the method according to any one of claims 7 to 14.

19. A computer-readable storage medium storing computer instructions, characterized in that: When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the steps in the method of claim 15 .

20. A data center network, characterized in that: include: A plurality of network devices, a database and a distributed cache system; wherein the distributed cache system includes a read node and a write node; The database is used to store data related to the plurality of network devices; The memory cache of the read node stores the specified data in the database, and is used to provide external associated query operations according to the specified data in the memory cache, and update the specified data in the memory cache according to the notification of the write node; The write node is used to notify the read node to update the specified data in its memory cache when the specified data in the database changes, so that the read node can provide external related query operations; The specified data includes the table items to be associated and the target table items. The table items to be associated refer to other table items other than the target table items that need to be associated in the associated query; the table items to be associated and the target table items respectively include table item indexes and data. The associated query operation is to perform an associated query on the table item data and data in the table items to be associated and the target table items, and assemble the queried data to obtain the associated query results.

Images