Data access method and system, and device and storage medium
By caching and verifying the data on the access side and combining RDMA technology to directly read the version number from the storage address, the storage node performance bottleneck and data consistency problems caused by data access hotspots are solved, and the data consistency and load are achieved balanced.
Patent Information
- Application Number
- PCT/CN2024/090133
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-22
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-28
AI Technical Summary
In a data access system, when the access popularity of a certain piece of data is high, the load of the storage node increases, resulting in a performance bottleneck. The access side may need to wait for a long time, and the prior art is difficult to ensure data when the asynchronous notification of failure cache. consistency.
By cached the target data and its version number on the access side and performing version number verification during reading, we ensure that the local cached data is consistent with the data of the storage node, and avoiding the access side's error reading of invalid data; at the same time, RDMA technology is used to Read the version number directly from the storage address without the participation of the storage node processor, reducing the load on the storage node.
It effectively reduces the load of storage nodes, ensures strong consistency and accuracy of data, avoids the problem of incorrect reading of failed data on the access side, and reduces the hotspot access load of storage nodes through load balancing and caching strategies.
Smart Images

Figure CN2024090133_28112024_PF_FP_ABST
Abstract
Description
Data access method, system, device and storage medium
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 22, 2023, with application number 202310581803.X and invention name “Data Access Method, System, Device and Storage Medium”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] One or more embodiments of this specification relate to the field of data access technology, and more particularly, to a data access method, system, device, and storage medium. Background Art
[0003] In a data access system of related art, data may be stored in one or more storage nodes, and different access terminals may access any storage node according to actual access needs and read required data from the storage node.
[0004] When a piece of data in a storage node is accessed with high popularity, the load on the storage node will increase, causing a performance bottleneck in the storage node. Some access terminals that access the data may need to wait for a long time.
[0005] Summary of the Invention
[0006] In view of this, one or more embodiments of this specification provide a data access method, system, device, and storage medium.
[0007] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:
[0008] According to a first aspect of one or more embodiments of the present specification, a data access method is proposed, which is applied to an access end, and the method includes: in the case of an access request, confirming whether the target data indicated by the access request is cached locally; if not, sending the access request to a storage node to receive the target data and its version number returned by the storage node, and caching them locally; wherein the version number of any data stored by the storage node is modified accordingly based on the modification operation on the data; if so, when it is determined that the version number of the target data cached locally is consistent with the version number of the target data stored by the storage node, reading the target data cached locally, otherwise, sending the access request to the storage node.
[0009] According to the second aspect of one or more embodiments of the present specification, a data access method is proposed, which is applied to a storage node, and the method includes: receiving an access request sent by any access terminal, and determining the target data indicated by the access request; returning the target data and its version number to the access terminal, so that the access terminal caches the target data and its version number locally; wherein the version number of any data stored in the storage node is modified accordingly based on the modification operation on the data; wherein the target data and its version number are used to enable the access terminal to read the locally cached target data when it receives an access request for the target data again, upon determining that the version number of the locally cached target data is consistent with the version number of the target data stored in the storage node.
[0010] According to a third aspect of an embodiment of the present disclosure, a data access method is provided, which is applied to a storage node, wherein any of the storage nodes is used to cache at least one piece of data and its version number of the storage node in other storage nodes and notify an access terminal; and / or, receive at least one piece of data and its version number sent by other storage nodes and store them locally; the version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the data; the method comprises: receiving an access request sent by any access terminal, determining whether the target data indicated by the access request is cached in the storage node by other storage nodes; if not, returning the target data to the access terminal; if so, upon determining that the version number of the target data stored in the storage node is consistent with the version number of the target data stored by the other storage nodes, returning the locally cached target data to the access terminal.
[0011] According to a fourth aspect of an embodiment of the present disclosure, there is provided a data access system, comprising at least one access terminal and at least one storage node; any one of the storage nodes is configured to receive an access request sent by any one of the access terminals, determine the target data indicated by the access request, and return the target data and its version number to the access terminal; wherein the version number of any one piece of data stored in the storage node is modified accordingly based on a modification operation on the piece of data; any one of the access terminals is configured to, in the event of an access request, confirm whether the target data indicated by the access request is cached locally; if not, send the access request to the storage node to receive the target data and its version number returned by the storage node, and cache them locally; if so, read the locally cached target data upon determining that the version number of the locally cached target data is consistent with the version number of the target data stored by the storage node, otherwise, send the access request to the storage node.
[0012] According to the fifth aspect of the embodiment of the present disclosure, there is provided a data access system, comprising at least one access terminal and at least two storage nodes; any one of the storage nodes is configured to cache at least one piece of data and its version number of the storage node in other storage nodes and notify the access terminal; and / or receive at least one piece of data and its version number sent by other storage nodes and store them locally; wherein the version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the piece of data; any one of the access terminals is configured to send an access request to the storage node; any one of the storage nodes is further configured to receive the access request sent by any one of the access terminals, determine whether the target data indicated by the access request is cached in the storage node by other storage nodes; if not, return the target data to the access terminal; if so, upon determining that the version number of the target data stored in the storage node is consistent with the version number of the target data stored by the other storage nodes, return the locally cached target data to the access terminal.
[0013] According to a sixth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.
[0014] According to a seventh aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above methods are implemented.
[0015] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG1 is a schematic structural diagram of a first data access system provided by an exemplary embodiment.
[0017] FIG2 is a schematic structural diagram of a second data access system provided by an exemplary embodiment.
[0018] FIG3 is a timing diagram of a data access process provided by an exemplary embodiment.
[0019] FIG4 is a schematic structural diagram of a third data access system provided by an exemplary embodiment.
[0020] FIG5 is a schematic structural diagram of a fourth data access system provided by an exemplary embodiment.
[0021] FIG6 is a flow chart of a data access method provided by an exemplary embodiment.
[0022] FIG7A is a flowchart of another data access method provided by an exemplary embodiment.
[0023] FIG7B is a flowchart of another data access method provided by an exemplary embodiment.
[0024] FIG8 is a schematic structural diagram of a device provided by an exemplary embodiment. DETAILED DESCRIPTION
[0025] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.
[0026] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.
[0027] In the related art, data can be stored in one or more storage nodes. Taking a data table in a database as an example, the data in the data table can be distributed on different storage nodes based on sharding rules. Different access terminals can access any storage node according to actual access needs and read the required data from the storage node. When the access popularity of a certain data in a storage node is high, the load of the storage node will increase, resulting in a performance bottleneck in the storage node. Some access terminals that access the data may need to wait for a long time. Among them, the storage node is a software product integrated in a physical device. A physical device can integrate one or more storage nodes. This embodiment does not impose any restrictions on this.
[0028] One solution to address increased storage node load in the related art is to cache hotspot data on the access client and then use asynchronous notifications to invalidate the cached hotspot data. For example, the access client can subscribe to data invalidation messages and obtain hotspot data information. The access client requests data A from the server. If A is hotspot data, the returned value of A is cached in local memory. When the access client receives a request to read data A again, it directly returns the cached result in memory, avoiding requests to the server. This reduces traffic for hotspot A. When data A is modified, the server asynchronously sends an invalidation message to the access client, deleting the local cache. Subsequent requests for data A will then request the latest data from the server again. In this solution, the invalidation message is sent asynchronously, resulting in inconsistent results between the time A is modified and the time the invalidation message is received. Furthermore, the server and access client are not on the same machine, and communication between the server and access client may experience disconnections, packet loss, and other anomalies. When data A is modified, the server cannot guarantee timely notification to all access clients, making data consistency impossible.
[0029] With respect to the problems in the related art, please refer to FIG1 . An embodiment of this specification provides a data access system, comprising at least one access terminal and at least one storage node. n in FIG1 is an integer greater than 0. Among them, any storage node is used to store at least one piece of data and the version number of each piece of data; the version number of any piece of data is modified accordingly based on the modification operation on the piece of data. For example, taking a data table in a database as an example, a piece of data here indicates a row of data in the data table. When one of the values in the row of data is modified, the version number of the row of data is modified accordingly. In other scenarios, such as in a function call scenario, a piece of data may refer to the values of all input parameters of a function. This embodiment does not impose any restrictions on the division of data, and specific settings can be made according to the actual application scenario.
[0030] Any of the storage nodes is further configured to receive an access request sent by any access terminal, determine the target data indicated by the access request, and return the target data and a version number of the target data to the access terminal.
[0031] Any of the access terminals is used to confirm whether the target data indicated by the access request is cached locally when there is an access request; if not, send the access request to the storage node to receive the target data and the version number of the target data returned by the storage node, and cache them locally; if so, read the locally cached target data when it is determined that the version number of the target data cached locally is consistent with the version number of the target data stored by the storage node; otherwise, send the access request to the storage node.
[0032] This embodiment enables the access end to cache target data, so that when the same demand is met later, the target data cached locally can be directly read without having to be obtained from the storage node, thereby reducing the load on the storage node; and considering that the target data stored in the storage node may be modified, resulting in the invalidation of the target data stored in the access end, in order to avoid the access end from mistakenly reading the invalid target data, a version number verification method is further provided. The storage node returns the version number of the target data when returning the target data. The access end can cache the target data and its version number. When it is determined that the version number of the target data cached locally is consistent with the version number of the target data stored in the storage node, it indicates that the target data cached by the access end is the latest and non-expired data. Then, the target data cached locally can be directly read, ensuring the accuracy and strong consistency of data reading. In some embodiments, referring to Figure 2, the access end includes a proxy node and a first client. The proxy node can be connected to the first client or the second client for communication. Exemplarily, the proxy node can receive an access request sent by the first client and forward it to the storage node; and after obtaining the target data indicated by the access request, it returns it to the first client. Exemplarily, the proxy node may receive an access request sent by the second client and forward it to the storage node; and after obtaining the target data indicated by the access request, return it to the second client.
[0033] In one possible scenario, the proxy node, the first client, and the storage node are maintained by the same manufacturer. Developers from that manufacturer can modify the code of the proxy node or the first client so that the modified proxy node and the modified first client have the aforementioned functionality. The second client is open source or maintained by another manufacturer, and its code cannot be modified. The second client can access the storage node through the modified proxy. Of course, the first client can also access the storage node through the modified proxy, and this embodiment does not impose any restrictions on this.
[0034] In another possible scenario, the proxy node, the first client, and the storage node may also be maintained by different manufacturers, and different manufacturers may make code modifications through negotiation to implement the functions provided in the embodiments of this specification. This embodiment does not impose any restrictions on this.
[0035] In some possible implementations, the version number verification process can be performed on the access end to reduce the load on the storage node caused by the verification operation. For example, the device hosting the access end is equipped with a first RDMA (Remote Direct Memory Access) network card, and the device hosting the storage node is equipped with a second RDMA network card. RDMA technology transfers data directly to a computer's storage area over a network, rapidly moving data from one system to a remote system's memory without impacting the operating system, thus minimizing the need for computer processing power.
[0036] Any of the storage nodes is further configured to receive an access request sent by any access terminal, determine the target data indicated by the access request, and return the target data, the version number of the target data, and the storage address of the version number to the access terminal.
[0037] Any of the access terminals is further used to, when there is an access request, confirm whether the target data indicated by the access request is cached locally; if not, send the access request to the storage node to receive the target data, the version number of the target data, and the storage address of the version number returned by the storage node, and cache them locally; if so, send a first verification request to the second RDMA network card through the first RDMA network card, the first verification request carrying the storage address of the version number of the target data, so that the second RDMA network card reads the version number of the target data stored in the storage node from the storage address and returns it; when it is determined that the version number of the locally cached target data is consistent with the version number returned by the second RDMA network card, read the locally cached target data; otherwise, send the access request to the storage node.
[0038] This embodiment can use RDMA technology to directly read the version number from the storage address and return it without the participation of the storage node's processor (such as CPU), which can greatly reduce the load of the storage node's processor.
[0039] In one example, please refer to the timing diagram of the data access system shown in FIG3 .
[0040] 11. At time T1, access terminal 1 has a data access request 1 for requesting data A, and the local storage space of access terminal 1 does not cache data A, so access terminal 1 sends access request 1 to storage node 1.
[0041] 12. In response to access request 1, storage node 1 returns data A (assuming A=1), version number ver of data A (assuming ver=5), and the storage address of the version number to access terminal 1.
[0042] 13. Access terminal 1 caches data A (A=1), version number ver (ver=5) of data A, and the storage address of the version number returned by storage node 1 in the local storage space.
[0043] 14. At time T2, access end 1 has a need to request data A again, that is, there is a data access request 2 for requesting data A. When it is confirmed that data A is in the local cache, a first verification request is sent to the second RDMA network card through the first RDMA network card. The first verification request carries the storage address of the version number of data A.
[0044] 15. The second RDMA network card directly reads the version number of the data A stored in the storage node from the storage address and returns it.
[0045] 16. Access terminal 1 verifies that the version number of the locally stored data A is consistent with the version number returned by the second RDMA network card, and directly reads the locally cached data A.
[0046] 17. At time T3, access terminal 2 sends a modification request for data A to storage node 1, requesting to modify A=1 to A=2.
[0047] 18. Storage node 1 modifies data A based on the modification request, and modifies the version number of data A (assuming that the modified ver=6).
[0048] 19. At time T4, access end 1 again requests data A, that is, there is a data access request 3 for requesting data A. When it is confirmed that data A is in the local cache, a first verification request is sent to the second RDMA network card through the first RDMA network card. The first verification request carries the storage address of the version number of data A.
[0049] 20. The second RDMA network card directly reads the version number of the data A stored in the storage node from the storage address and returns it.
[0050] 21. Access terminal 1 checks that the version number of locally stored data A (ver=5) is inconsistent with the version number (ver=6) returned by the second RDMA network card, and then sends access request 3 to storage node 1.
[0051] In another possible implementation, the version number verification process can also be performed in the storage node, and any of the storage nodes is further used to receive an access request sent by any access terminal, determine the target data indicated by the access request, and return the target data and its version number to the access terminal; wherein the version number of any data stored in the storage node is modified accordingly based on the modification operation on the data.
[0052] Any of the access terminals is used to confirm whether the target data indicated by the access request is cached locally when there is an access request; if not, send the access request to the storage node to receive the target data and its version number returned by the storage node, and cache them locally; if so, send a second verification request to the storage node, and the second verification request carries the version number of the target data cached locally.
[0053] Any of the storage nodes is also used to receive a second verification request sent by any access terminal, compare the version number of the target data carried in the second verification request with the version number of the target data stored in the storage node, and return verification consistency information to the access terminal if the two are consistent, so that the access terminal reads the target data cached locally at the access terminal; and return verification inconsistency information to the access terminal if the two are inconsistent, so that the access terminal sends an access request to this storage node.
[0054] In this embodiment, when the version numbers are verified to be consistent, the access end does not need to read data from the storage node, which is beneficial to reducing the load of the storage node while ensuring strong consistency of the data.
[0055] In some embodiments, considering that some data have a low access frequency, if these data are cached on the access end, the probability of subsequent access ends accessing them again is low, resulting in a problem of wasted storage resources on the access end. Therefore, any of the storage nodes is also used to determine whether the target data indicated by the access request is hot data after receiving an access request sent by any access end; hot data is data that is frequently accessed. For example, if the number of times the target data is accessed per unit time exceeds a preset number of times, the target data can be determined to be the target data; or if the number of times the target data is accessed per unit time is greater than the statistical value of the number of times other data is accessed per unit time, the target data can be determined to be the target data. The preset number of times can be specifically set according to the actual application scenario, and this embodiment does not impose any restrictions on this. The statistical value of the number of accesses includes but is not limited to the average, median, or maximum number, etc.
[0056] Any of the storage nodes is further configured to, if the target data is determined to be hot data, return the target data and its version number to the access terminal; if the target data is not hot data, directly return the target data to the access terminal. This embodiment stores the hot data of a storage node on the access terminal, which helps reduce the load of hot access on the storage node.
[0057] This specification provides a data access system, in which any storage node is used to store at least one piece of data and the version number of each piece of data; the version number of any piece of data is modified accordingly based on the modification operation on the piece of data, so that the modification of the data and the modification of the version number can be carried out synchronously and remain consistent. The access end can cache the target data and its version number, and will perform a version number check before reading the locally cached target data. The locally cached target data will only be read if the version number of the locally cached target data is the same as the version number of the target data stored by the storage node. This mechanism not only reduces the load on the storage node but also ensures that the target data read from the local storage node is the latest value, thus achieving strong data consistency.
[0058] In some embodiments, data can be cached not only on the access end but also on other storage nodes. Referring to FIG. 4 , an embodiment of this specification provides a data access system comprising at least one access end and at least two storage nodes. In FIG. 4 , n is an integer greater than 0. At least two storage nodes are communicatively connected to each other.
[0059] Any of the aforementioned storage nodes is used to cache at least one piece of data and its version number of the storage node in other storage nodes and notify the access end; and / or, receive at least one piece of data and its version number sent by other storage nodes and store them locally; wherein, the version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the data.
[0060] Exemplarily, after any of the storage nodes caches at least one piece of data and its version number in other storage nodes, it can notify some or all access terminals that have access requirements for the data, so that the access terminals that have access requirements for the data read the data from the other storage nodes, thereby reducing the load on the storage node.
[0061] Any access end is used to send access requests to storage nodes.
[0062] Any of the storage nodes is also used to receive an access request sent by any access terminal, and determine whether the target data indicated by the access request is cached in this storage node by other storage nodes; if not, return the target data to the access terminal; if so, when it is determined that the version number of the target data stored in this storage node is consistent with the version number of the target data stored by the other storage nodes, return the locally cached target data to the access terminal.
[0063] This embodiment enables the data of a certain storage node to be stored in other storage nodes, and the access terminal can read the target data from the other storage nodes, thereby helping to reduce the load on the certain storage node. In addition, considering that the target data stored in the storage node may be modified, causing the target data stored in other storage nodes to become invalid, in order to avoid the access terminal from mistakenly reading invalid target data from other storage nodes, a version number verification method is further provided. When the other storage nodes determine that the version number of the target data cached locally is consistent with the version number of the target data stored by the original storage node, indicating that the cached target data is the latest non-invalid data, the other storage nodes can return the locally cached target data to the access terminal, thereby ensuring the accuracy of data reading.
[0064] In some embodiments, referring to FIG5 , the access end includes a proxy node and a first client; the proxy node is capable of communicating with either the first client or the second client. Exemplarily, the proxy node may receive an access request from the first client and forward it to the storage node; and after obtaining the target data indicated by the access request, return it to the first client. Exemplarily, the proxy node may receive an access request from the second client and forward it to the storage node; and after obtaining the target data indicated by the access request, return it to the second client.
[0065] In some embodiments, after the data of a storage node is stored in other storage nodes, the version number verification process can be performed in either of the two storage nodes, and this embodiment does not impose any limitation on this.
[0066] In one possible implementation, the device where any of the storage nodes resides is equipped with an RDMA network card. Any of the storage nodes is further configured to cache at least one piece of data, the version number of the data, and the storage address of the version number on another storage node; and / or receive and locally store at least one piece of data, the version number of the data, and the storage address of the version number from another storage node.
[0067] Any access end is used to send access requests to storage nodes.
[0068] Any of the storage nodes is also used to receive an access request sent by any access terminal, determine whether the target data indicated by the access request is cached in this storage node by other storage nodes; if not, return the target data to the access terminal; if so, send a first verification request to the RDMA network card of the other storage node through the RDMA network card of this terminal, and the first verification request carries the storage address of the version number of the target data, so that the RDMA network card of the other storage node reads the version number of the target data from the storage address and returns it; when it is determined that the version number of the locally cached target data is consistent with the version number returned by the RDMA network card of the other storage node, return the locally cached target data to the access terminal.
[0069] This embodiment can make use of RDMA technology. Without the participation of processors (such as CPUs) of other storage nodes, the RDMA network cards of other storage nodes can directly read the version number from the storage address and return it, which can greatly reduce the load of the processors of other storage nodes (i.e., the original storage nodes storing the target data).
[0070] In another possible implementation, any of the storage nodes is used to cache at least one piece of data and its version number of the storage node in other storage nodes and notify the access end; and / or receive at least one piece of data and its version number sent by other storage nodes and store them locally; wherein the version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the data.
[0071] Any access end is used to send access requests to storage nodes.
[0072] Any of the above-mentioned storage nodes is also used to receive an access request sent by any access terminal, and determine whether the target data indicated by the access request is cached in the local storage node by other storage nodes; if not, return the target data to the access terminal; if so, send a second verification request to the other storage node, the second verification request carries the version number of the target data cached locally, so that the other storage node compares the version number of the target data carried by the second verification request with the version number of the target data stored by it, and returns verification consistency information when the two are consistent. Any of the above-mentioned storage nodes is also used to return the locally cached target data to the access terminal based on the verification consistency information returned by the other storage nodes. In addition, in the case of verification inconsistency information from the other storage nodes, the storage node can delete the locally cached target data and its version number, re-acquire the target data and its version number from the other storage nodes, store it locally, and return the target data to the access terminal.
[0073] In some embodiments, considering that some data are accessed less frequently, if these data are cached in other storage nodes, the probability of subsequent access terminals accessing them again is low, resulting in a waste of storage resources of other storage nodes; and other storage nodes also have load problems. Therefore, any of the storage nodes is also used to cache the hotspot data and the version number of the hotspot data in other storage nodes that meet the low load condition if it is determined that there is hotspot data in the data of this storage node, and notify the access terminal; and / or, when the load of this storage node meets the low load condition, receive the hotspot data and the version number of the hotspot data sent by other storage nodes and store them locally. This embodiment stores the hotspot data of a storage node in other storage nodes with low load, which is beneficial to reducing the load of hotspot access of the storage node and achieving load balancing.
[0074] The low-load condition can be specifically set based on the actual application scenario, and this embodiment does not impose any restrictions on this. For example, storage node 1 may first send a data storage request to storage node 2 and storage node 3; storage node 2 determines that its own load is high and returns a first response to storage node 1 stating that the data cannot be stored; storage node 3, when determining that its own load is low, returns a second response to storage node 1 stating that the data can be stored; storage node 1 can then store the hotspot data and its version number in storage node 3 based on the second response.
[0075] Hot data refers to frequently accessed data. For example, the number of times hot data is accessed per unit time exceeds a preset number; or the number of times hot data is accessed per unit time exceeds the statistical value of the number of times other data is accessed per unit time. The preset number of times can be set based on the actual application scenario and is not limited in this embodiment. The statistical value of the number of accesses includes, but is not limited to, the average, median, or maximum number.
[0076] This specification provides a data access system, in which any storage node is used to store at least one piece of data and the version number of each piece of data; the version number of any piece of data is modified accordingly based on the modification operation on the piece of data, so that the modification of the data and the modification of the version number can be carried out synchronously and remain consistent. When the storage node determines that the target data indicated by the access request is cached by other storage nodes in this storage node, it will perform a version number check before reading the locally cached target data. The locally cached target data will only be read if the version number of the locally cached target data is the same as the version number of the target data stored by other storage nodes. This mechanism ensures that the target data read is the latest value of other storage nodes, achieving strong data consistency.
[0077] The various technical features in the above embodiments can be combined arbitrarily as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of the various technical features in the above embodiments also falls within the scope of disclosure of this specification.
[0078] Accordingly, referring to FIG6 , an embodiment of this specification further provides a data access method, which is applied to the access terminal shown in FIG1 , and the method includes:
[0079] In S101 , when there is an access request, it is confirmed whether the target data indicated by the access request is cached locally.
[0080] In S102, if not, the access request is sent to the storage node to receive the target data and its version number returned by the storage node and cache them locally; wherein, the version number of any data stored in the storage node is modified accordingly based on the modification operation on the data.
[0081] In S103 , if yes, when it is determined that the version number of the target data in the local cache is consistent with the version number of the target data stored in the storage node, the target data in the local cache is read; otherwise, the access request is sent to the storage node.
[0082] In some embodiments, the device where the access end is located is installed with a first RDMA network card, and the device where the storage node is located is installed with a second RDMA network card.
[0083] The sending of the access request to the storage node to receive the target data and its version number returned by the storage node, and caching them locally, includes: sending the access request to the storage node to receive the target data, the version number of the target data and the storage address of the version number returned by the storage node, and caching them locally.
[0084] The reading of the locally cached target data when it is determined that the version number of the locally cached target data is consistent with the version number of the target data stored in the storage node includes: sending a first verification request to the second RDMA network card through the first RDMA network card, where the first verification request carries a storage address of the version number of the target data, so that the second RDMA network card reads the version number of the target data stored in the storage node from the storage address and returns it; and reading the locally cached target data when it is determined that the version number of the locally cached target data is consistent with the version number returned by the second RDMA network card.
[0085] In some embodiments, when it is determined that the version number of the target data cached locally is consistent with the version number of the target data stored by the storage node, reading the target data cached locally includes: sending a second verification request to the storage node, the second verification request carrying the version number of the target data cached locally, so that the storage node compares the version number of the target data carried by the second verification request with the version number of the target data stored by the storage node, and returning verification consistency information to the access end when the two are consistent; and reading the target data cached locally based on the returned verification consistency information.
[0086] In some embodiments, the access terminal includes a proxy node and a first client;
[0087] The proxy node can be connected to the first client or the second client for communication, receive an access request sent by the first client or the second client, and forward the access request to the storage node.
[0088] For details about the method, please refer to the description of the storage system above and will not be repeated here.
[0089] Correspondingly, referring to FIG. 7A , an embodiment of this specification further provides a data access method, which is applied to the storage node as shown in FIG. 1 , and the method includes:
[0090] In S201, an access request sent by any access terminal is received, and target data indicated by the access request is determined.
[0091] In S202, the target data and its version number are returned to the access end so that the access end caches the target data and its version number locally; wherein, the version number of any data stored in the storage node is modified accordingly based on the modification operation on the data; the target data and its version number are used to enable the access end to read the locally cached target data when it receives an access request for the target data again, and when it determines that the version number of the locally cached target data is consistent with the version number of the target data stored in the storage node.
[0092] In some embodiments, the device where the access end is located is installed with a first RDMA network card, and the device where the storage node is located is installed with a second RDMA network card; the access end is further used to send a first verification request to the second RDMA network card through the first RDMA network card, and the first verification request carries the storage address of the version number of the target data; the second RDMA network card is further used to read the version number of the target data stored in the storage node from the storage address carried in the received first verification request and return it.
[0093] In some embodiments, the method also includes: receiving a second verification request sent by any access end, comparing the version number of the target data carried by the second verification request with the version number of the target data stored by the storage node, and returning verification consistency information to the access end if the two are consistent, so that the access end reads the target data cached locally at the access end.
[0094] In some embodiments, returning the target data and its version number to the access end includes: if the target data is hotspot data, returning the target data and its version number to the access end; wherein, the number of times the hotspot data is accessed per unit time exceeds a preset number; or the number of times the hotspot data is accessed per unit time is greater than the statistical value of the number of times other data is accessed per unit time.
[0095] For details about the method, please refer to the description of the storage system above and will not be repeated here.
[0096] Accordingly, referring to FIG. 7B , an embodiment of this specification further provides a data access method, which is applied to the storage node shown in FIG. 4 , wherein any of the storage nodes is configured to cache at least one piece of data and its version number in another storage node and notify an access terminal; and / or receive at least one piece of data and its version number sent by another storage node and store them locally; the version number of any piece of data stored in the storage node is modified accordingly based on a modification operation on the piece of data; the method comprises:
[0097] In S301, an access request sent by any access terminal is received, and it is determined whether the target data indicated by the access request is cached in the current storage node by other storage nodes.
[0098] In S302, if not, the target data is returned to the access end.
[0099] In S303 , if yes, when it is determined that the version number of the target data stored in the current storage node is consistent with the version number of the target data stored in the other storage nodes, the locally cached target data is returned to the access end.
[0100] In some embodiments, the device where any of the storage nodes is located is installed with an RDMA network card; any of the storage nodes is also used to cache at least one piece of data of the storage node, the version number of the data, and the storage address of the version number in other storage nodes; and / or, receive at least one piece of data, the version number of the data, and the storage address of the version number sent by other storage nodes and store them locally.
[0101] The method of returning the locally cached target data to the access end when it is determined that the version number of the target data stored in the local storage node is consistent with the version number of the target data stored in the other storage nodes includes: sending a first verification request to the RDMA network card of the other storage node through the RDMA network card of the local end, where the first verification request carries the storage address of the version number of the target data, so that the RDMA network card of the other storage node reads the version number of the target data from the storage address and returns it; and returning the locally cached target data to the access end when it is determined that the version number of the locally cached target data is consistent with the version number returned by the RDMA network card of the other storage node.
[0102] In some embodiments, when it is determined that the version number of the target data stored in the current storage node is consistent with the version number of the target data stored in the other storage nodes, the locally cached target data is returned to the access end, including: sending a second verification request to the other storage node, the second verification request carrying the version number of the locally cached target data, so that the other storage node compares the version number of the target data carried in the second verification request with the version number of the target data stored by it, and returns verification consistency information when the two are consistent; based on the returned verification consistency information, the locally cached target data is returned to the access end.
[0103] In some embodiments, any of the storage nodes is configured to cache at least one piece of hot data and its version number on another storage node and notify an access terminal; and / or receive and locally store at least one piece of hot data and its version number sent by another storage node. The hot data may be accessed more than a preset number of times per unit time; or the hot data may be accessed more than a statistical value of the number of times other data is accessed per unit time.
[0104] For details about the method, please refer to the description of the storage system above and will not be repeated here.
[0105] This embodiment has the following beneficial effects:
[0106] A version number check is performed before reading the target data in the local cache. The target data in the local cache will be read only when the version number of the target data in the local cache is the same as the version number of the target data stored in the storage node. This mechanism not only reduces the load on the storage node but also ensures that the target data read from the local cache must be the latest value of the storage node, achieving strong data consistency and accuracy.
[0107] The version number of any piece of data is modified based on the modification operation on that piece of data. This ensures that data modifications and version number modifications are synchronized and consistent. The access end will perform a version number check before reading the locally cached target data. The locally cached target data will only be read if the version number of the locally cached target data matches the version number of the target data stored by the storage node. This mechanism not only reduces the load on the storage node but also ensures that the target data read from the local storage node is the latest value, achieving strong data consistency and accuracy.
[0108] Caching data to other storage nodes helps reduce the access load of this storage node, and the target data cached in other storage nodes needs to be version-checked before being returned to the access end. Only when it is determined that the version numbers of the target data stored in the two storage nodes are consistent will the target data be returned to the access end, achieving strong data consistency and accuracy.
[0109] FIG8 is a schematic structural diagram of a device provided by an exemplary embodiment. Referring to FIG8 , at the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 802 reading the corresponding computer program from the non-volatile memory 810 into the memory 808 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but may also be hardware or logic devices.
[0110] In some embodiments, an embodiment of this specification further provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor implements any of the above methods by running the executable instructions.
[0111] In some embodiments, the embodiments of this specification further provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.
[0112] Exemplarily, the electronic device integrates a computer program product, and when the electronic device executes the computer program product, the data access method provided in the embodiments of this specification is implemented.
[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0114] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.
[0115] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0116] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0117] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.
[0118] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0119] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0120] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0121] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."
[0122] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.
Claims
1. A data access method, applied to an access terminal, comprising: In case of an access request, confirming whether the target data indicated by the access request is cached locally; If not, send the access request to the storage node to receive the target data and its version number returned by the storage node, and cache them locally; wherein the version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the piece of data; If so, when it is determined that the version number of the target data in the local cache is consistent with the version number of the target data stored in the storage node, the target data in the local cache is read; otherwise, the access request is sent to the storage node.
2. According to the method of claim 1, the device where the access end is located is installed with a first RDMA network card, and the device where the storage node is located is installed with a second RDMA network card; In the case of sending the access request to the storage node, the storage node also returns the storage address of the version number; The step of reading the target data in the local cache when it is determined that the version number of the target data in the local cache is consistent with the version number of the target data stored in the storage node comprises: Sending a first verification request to the second RDMA network card through the first RDMA network card, where the first verification request carries a storage address of the version number of the target data, so that the second RDMA network card reads the version number of the target data stored in the storage node from the storage address and returns it; When it is determined that the version number of the target data cached locally is consistent with the version number returned by the second RDMA network card, the target data cached locally is read.
3. The method according to claim 1, wherein when it is determined that the version number of the target data cached locally is consistent with the version number of the target data stored in the storage node, reading the target data cached locally comprises: Sending a second verification request to the storage node, the second verification request carrying the version number of the target data cached locally; wherein the second verification request is used to enable the storage node to compare the version number of the target data carried by the second verification request with the version number of the target data stored by the storage node, and return verification consistency information to the access end when the two are consistent; Based on the returned verification consistency information, read the target data in the local cache.
4. The method according to any one of claims 1 to 3, wherein the access end comprises a proxy node and a first client; The proxy node can be connected to the first client or the second client for communication, receive an access request sent by the first client or the second client, and forward the access request to the storage node.
5. A data access method, applied to a storage node, comprising: Receiving an access request sent by any access terminal, and determining target data indicated by the access request; Returning the target data and its version number to the access end, so that the access end caches the target data and its version number locally; wherein the version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the piece of data; The target data and its version number are used to enable the access end to determine the version number of the target data cached locally and the target data stored in the storage node when receiving an access request for the target data again. If the version number is consistent, read the target data in the local cache.
6. The method according to claim 5, wherein the device where the access end is located is installed with a first RDMA network card, and the device where the storage node is located is installed with a second RDMA network card; The access end is further used to send a first verification request to the second RDMA network card through the first RDMA network card, where the first verification request carries a storage address of a version number of the target data; The second RDMA network card is further used to read the version number of the target data stored in the storage node from the storage address carried in the received first verification request and return it.
7. The method according to claim 5, further comprising: Receive a second verification request sent by any access terminal, compare the version number of the target data carried in the second verification request with the version number of the target data stored in the storage node, and return verification consistency information to the access terminal if the two are consistent, so that the access terminal can read the target data cached locally in the access terminal.
8. The method according to claim 5, wherein returning the target data and its version number to the access terminal comprises: If the target data is hot data, returning the target data and its version number to the access terminal; The number of times the hotspot data is accessed within a unit time exceeds a preset number; or the number of times the hotspot data is accessed within a unit time is greater than the statistical value of the number of times other data is accessed within a unit time.
9. A data access method, applied to a storage node, wherein any of the storage nodes is used to cache at least one piece of data and its version number of the storage node in other storage nodes, and notify the access end; and / or, receiving at least one piece of data and its version number sent by other storage nodes and storing them locally; The version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the piece of data; the method comprises: Receive an access request sent by any access terminal, and determine whether the target data indicated by the access request is cached in the current storage node by other storage nodes; If not, returning the target data to the access end; If so, when it is determined that the version number of the target data stored in the storage node is consistent with the version number of the target data stored in the other storage nodes, the locally cached target data is returned to the access end.
10. The method according to claim 9, wherein the device where any of the storage nodes is located is installed with an RDMA network card; Any of the storage nodes is further used to cache the storage address of the version number in other storage nodes; And / or, receiving the storage address of the version number sent by other storage nodes and storing it locally; The step of returning the locally cached target data to the access end when it is determined that the version number of the target data stored in the storage node is consistent with the version number of the target data stored in the other storage nodes includes: Sending a first verification request to the RDMA network card of the other storage node through the RDMA network card of the local end, wherein the first verification request carries the storage address of the version number of the target data, so that the RDMA network card of the other storage node reads the version number of the target data from the storage address and returns it; When it is determined that the version number of the target data cached locally is consistent with the version number returned by the RDMA network card of the other storage node, the target data cached locally is returned to the access end.
11. A data access system comprising at least one access terminal and at least one storage node; Any of the storage nodes is used to receive an access request sent by any access terminal, determine the target data indicated by the access request, and return the target data and its version number to the access terminal; wherein, The version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the piece of data; Any of the access terminals is used to confirm whether the target data indicated by the access request is cached locally when there is an access request; if not, send the access request to the storage node to receive the target data and its version number returned by the storage node and cache them locally; if so, read the locally cached target data when it is determined that the version number of the locally cached target data is consistent with the version number of the target data stored by the storage node; otherwise, send the access request to the storage node.
12. A data access system comprising at least one access terminal and at least two storage nodes; Any of the storage nodes is used to cache at least one piece of data and its version number in other storage nodes and notify the access end; and / or, receiving at least one piece of data and its version number sent by other storage nodes and storing them locally; wherein, The version number of any piece of data stored in the storage node is modified accordingly based on the modification operation on the piece of data; Any access end is used to send access requests to storage nodes; Any of the storage nodes is also used to receive an access request sent by any access terminal, and determine whether the target data indicated by the access request is cached in this storage node by other storage nodes; if not, return the target data to the access terminal; if so, when it is determined that the version number of the target data stored in this storage node is consistent with the version number of the target data stored in the other storage nodes, return the locally cached target data to the access terminal.
13. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 10 by running the executable instructions.
14. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.
Citation Information
Patent Citations
Web access method and device
CN107357876A
Data retrieval method and device, electronic equipment and readable storage medium
CN114461678A
Data processing method and device and nonvolatile storage medium
CN114567677A
Data access method, system and device and storage medium
CN116633616A
System and method of maintaining functional client side data cache coherence
US20030217081A1
Cited By
Query method and system of proxy storage engine under storage and calculation separation architecture
CN119441306A
