Data reading and writing method and system

Abstract

The invention provides a data reading and writing method and system, and relates to the technical field of data storage. According to the method, the reading strategy indication information is introduced into the data reading request, so that the client can flexibly specify to read the main storage node or the standby storage node, and the fine control of the reading strategy is realized. According to the design, the diversified requirements of different service scenes for data consistency and access delay are met, and the overall performance and the resource utilization rate of the system are optimized by reasonably utilizing storage node resources in a plurality of available areas.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and more specifically, to a data reading and writing method and system. Background Technology

[0002] In current distributed storage architectures, a master-slave replication architecture is typically used, where slave nodes synchronize data with the master node in real time to achieve redundant data storage. To distribute the load on the master node, a read-write separation strategy can be employed. Under this strategy, write operations are primarily handled by the master node, while read operations are distributed to the slave nodes. This strategy alleviates the read pressure on the master node to some extent and improves the overall throughput of the system.

[0003] However, due to data synchronization delays, the data on the slave node may not be completely consistent with the data on the master node. Especially in business scenarios with extremely high requirements for data consistency, this delay may lead to the reading of outdated data, affecting the accuracy of business decisions. Summary of the Invention

[0004] The purpose of this application is to provide a data reading and writing method and system to improve the problem that existing reading and writing strategies are inflexible and cannot be applied to more business scenarios.

[0005] In a first aspect, embodiments of this application provide a data read / write method, the method comprising: The client sends a data read request to the proxy component in the first availability zone. The data read request includes read policy indication information, which is used to indicate whether to read the primary storage node or the backup storage node. The multiple storage nodes include primary storage nodes and backup storage nodes, which are distributed in multiple availability zones. The multiple availability zones are used to indicate the data center corresponding to the client. The proxy component responds to the data read request and reads data according to the read strategy instruction information; The proxy component returns the read data to the client.

[0006] In the above implementation, by introducing read policy indication information into data read requests, clients can flexibly specify whether to read from the primary or backup storage node, thus achieving fine-grained control over the read policy. This design not only meets the diverse needs of different business scenarios for data consistency and access latency, but also optimizes the overall system performance and resource utilization by making reasonable use of storage node resources across multiple availability zones. For example, for scenarios with high data consistency requirements, the primary storage node can be read first; while for scenarios sensitive to access latency, the backup storage node within the local availability zone can be read first, thereby ensuring data availability while reducing latency caused by cross-availability zone access.

[0007] Optionally, before the client sends a data read request to the proxy component in the first availability zone, it further includes: The client generates read policy instruction information based on business requirements; The client generates a data read request based on the read policy instruction information.

[0008] In the above implementation process, the client can flexibly choose to read from the primary storage node or the backup storage node according to specific business needs, thereby achieving an optimal balance between data consistency and access latency. For example, in scenarios with high real-time requirements, the client can choose to prioritize reading from the backup storage node in the local availability zone to reduce latency; while in scenarios with extremely high data accuracy requirements, it can choose to read from the primary storage node to ensure data consistency. This flexible strategy customization not only improves the system's adaptability and performance but also enhances its scalability and reliability, better meeting complex and ever-changing business needs.

[0009] Optionally, the client sends a data read request to the proxy component in the first availability zone, including: The client sends a data read request to the proxy component in the first availability zone via the RESP protocol.

[0010] In the above implementation, the RESP protocol, as the standard communication protocol between Redis clients and servers, supports multiple data types, efficiently handles requests and responses, and ensures the accuracy and speed of data transmission. This design makes communication between the client and the proxy component more standardized and efficient, reduces development and maintenance costs, and also improves the overall performance and stability of the system.

[0011] Optionally, the client sends a data read request to the proxy component in the first availability zone via the RESP protocol, including: The client generates a data read request in RESP protocol format, wherein the read policy indication information in the data read request is represented by a custom field in the RESP protocol format.

[0012] In the above implementation, by introducing a custom field into the RESP protocol to represent read policy indication information, flexible extension within the standard protocol framework is achieved. This design not only retains the efficiency and compatibility of the RESP protocol, but also allows clients to dynamically specify read policies according to business needs without requiring large-scale modifications to the existing protocol.

[0013] Optionally, the first availability zone includes at least two proxy components, and the client sends a data read request to the proxy component in the first availability zone, including: The client sends a data read request to the nearest proxy component within the first availability zone.

[0014] In the above implementation process, by having the client send a data read request to the nearest proxy component in the first availability zone, network latency is significantly reduced and the response speed of data reading is improved.

[0015] Optionally, the storage node in the first availability zone is a backup storage node, and the proxy component responds to the data read request by reading data according to the read policy instruction information, including: If the read policy indication information in the data read request indicates reading the primary storage node, then the proxy component responds to the data read request and performs data reading across availability zones; If the read policy indication information in the data read request indicates reading from a backup storage node, then the proxy component responds to the data read request and reads data from a storage node in the first availability zone.

[0016] In the above implementation process, flexible read strategy instructions enable the proxy component to selectively read data based on the client's needs. When strong consistency data is required, the proxy component reads the primary storage node across availability zones to ensure data accuracy; while when data consistency requirements are lower but access latency is sensitive, the proxy component prioritizes reading data from the backup storage node in the local availability zone to reduce latency.

[0017] Optionally, the multiple storage nodes refer to nodes in the Redis system, or the multiple storage nodes refer to nodes in the KV system. This solution can be flexibly applied to various storage architectures, improving the system's scalability and compatibility. Whether it's a Redis-based in-memory storage system or other KV storage systems, this solution can achieve efficient data reading and load balancing through unified read policy indication information and intelligent routing by the proxy component.

[0018] Optionally, the method further includes: The client determines the second availability zone where the primary storage node is located; The client sends a data write request to the proxy component of the second availability zone; The proxy component of the second availability zone responds to the data write request and writes the data to the primary storage node.

[0019] In the above implementation process, the client actively determines the second availability zone where the primary storage node is located and sends the data write request to the proxy component of that availability zone, thus achieving precise routing of data write operations.

[0020] Optionally, the data read request may further include access policy indication information, which is used to indicate whether to access the local cache of the proxy component or to access the storage node of the availability zone.

[0021] In the above implementation, by introducing access policy indication information into data read requests, clients can flexibly choose to access the local cache of the proxy component or directly access the storage node of the availability zone. This design not only improves the system's flexibility and response speed but also optimizes resource utilization efficiency. For example, for frequently read data with low real-time requirements, accessing the local cache can significantly reduce latency and improve performance; while for scenarios requiring the latest data, directly accessing the storage node ensures data consistency and accuracy.

[0022] Secondly, embodiments of this application provide a data read / write system, the system comprising: a client, a proxy component, and a storage node; A client is used to send a data read request to a proxy component in the first availability zone. The data read request includes read policy indication information, which is used to indicate whether to read the primary storage node or the backup storage node. The multiple storage nodes include primary storage nodes and backup storage nodes, which are distributed in multiple availability zones. The multiple availability zones are used to indicate the data center corresponding to the client. The proxy component is used to read data in response to the data read request, according to the read strategy instruction information; The proxy component is also used to return the read data to the client.

[0023] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the structure of a data read / write system provided in an embodiment of this application; Figure 2 A flowchart illustrating a data read / write method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a data read / write RESP protocol format provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0027] It should be noted that the terms "system" and "network" in the embodiments of this invention can be used interchangeably. "Multiple" refers to two or more; therefore, in the embodiments of this invention, "multiple" can also be understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0028] It should also be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0029] This application provides a data read / write method that introduces read policy indication information into data read requests, enabling clients to flexibly specify whether to read from the primary or backup storage node, thereby achieving fine-grained control over the read policy. This design not only meets the diverse needs of different business scenarios for data consistency and access latency, but also optimizes the overall system performance and resource utilization by making reasonable use of storage node resources in multiple availability zones.

[0030] To facilitate understanding of the data read / write methods provided in this solution, the data read / write system involved in this solution will be introduced below, such as... Figure 1As shown, the data read / write system includes clients, agent components, and storage nodes, which can also be called replicas. Storage nodes can include primary storage nodes and backup storage nodes, with multiple storage nodes distributed across multiple availability zones. These availability zones characterize the data centers corresponding to the clients.

[0031] An availability zone can be understood as a data center that includes multiple storage nodes. Multiple availability zones can be distributed in different regions, such as the Beijing availability zone and the Hangzhou availability zone. Data within these availability zones can be synchronized.

[0032] The proxy component acts as an intermediary layer between the client and the storage node, responsible for receiving client requests and performing corresponding routing operations based on the instructions in the request.

[0033] In some implementations, at least one proxy component is deployed within an availability zone, and one or more proxy components may be deployed that can connect to various storage nodes within the availability zone.

[0034] In some implementations, at least one storage node is deployed within an availability zone. The primary storage node may be deployed in one availability zone (referred to as the primary availability zone). Multiple storage nodes may include one primary storage node and multiple backup storage nodes, and at least one backup storage node may be deployed in other availability zones. For example... Figure 1 As shown, primary storage nodes are deployed in availability zone 1, which can serve as the primary availability zone, while backup storage nodes are deployed in availability zone 2, which can serve as the backup availability zone.

[0035] The primary storage node can handle data write operations, and of course, it can also handle data read operations. The backup storage node can synchronize the data of the primary storage node in real time and is used to handle data read operations.

[0036] In this scenario, the proxy component within an availability zone with only backup storage nodes can establish a connection with the primary storage node, enabling communication between the proxy component and both primary and backup storage nodes. Conversely, if the availability zone containing the primary storage node does not have a backup storage node, the proxy component within that availability zone can establish a connection with backup storage nodes in any one or more other availability zones. If the availability zone has both a primary and a backup storage node, the proxy component can establish connections with both the primary and backup storage nodes within that availability zone, respectively.

[0037] Taking Availability Zone 1 and Availability Zone 2 as examples, if Proxy Component 1 and Primary Storage Node 1 are deployed in Availability Zone 1, and Proxy Component 2 and Backup Storage Node 2 are deployed in Availability Zone 2, then Proxy Component 1 can connect to Primary Storage Node 1 and Backup Storage Node 2, and Proxy Component 2 can connect to Primary Storage Node 1 and Backup Storage Node 2.

[0038] If proxy component 1, proxy component 2 and primary storage node 1 are deployed in availability zone 1, and proxy component 3, proxy component 4 and backup storage node 2 are deployed in availability zone 2, then proxy component 1 and proxy component 2 are both connected to primary storage node 1 and backup storage node 2, and proxy component 3 and proxy component 4 are both connected to primary storage node 1 and backup storage node 2.

[0039] If proxy component 1, proxy component 2, primary storage node 1, and backup storage node 2 are deployed in availability zone 1, and proxy component 3, proxy component 4, and backup storage node 3 and backup storage node 4 are deployed in availability zone 2, then proxy component 1 and proxy component 2 are both connected to primary storage node 1 and backup storage node 2. Of course, they can also be connected to backup storage node 3 and backup storage node 4. Proxy component 3 and proxy component 4 are both connected to primary storage node 1 and backup storage node 3 and backup storage node 4.

[0040] Understandably, if a primary storage node is deployed in an availability zone, since the primary storage node can handle both data read and write operations, the proxy component in that availability zone only needs to establish a connection with the primary storage node, and subsequent read and write operations can be performed in the primary storage node.

[0041] The aforementioned storage nodes can refer to servers used for storing data. The number of storage nodes deployed in each availability zone can be flexibly deployed according to actual needs, or the data volume of storage nodes in each availability zone can be planned based on the frequency of user requests in different regions. For example, more nodes can be deployed in availability zones with higher user request frequencies.

[0042] The functions of each module in the above data read / write system can be referred to the following method embodiment. In conjunction with the above description of the data read / write system, please refer to the following... Figure 2 , Figure 2 A flowchart of a data read / write method provided in this application embodiment, the method including the following steps: Step S110: The client sends a data read request to the proxy component in the first availability zone.

[0043] The data read request includes read policy indication information, which indicates whether to read from the primary storage node or the backup storage node. The multiple storage nodes include primary storage nodes and backup storage nodes, which are distributed in multiple availability zones. The multiple availability zones indicate the data center corresponding to the client.

[0044] A client can refer to an application, whose data can be stored across multiple storage nodes in multiple availability zones. Data is synchronized in real time between these storage nodes. Understandably, data is first written to the primary storage node, and backup storage nodes can synchronize data from the primary storage node in real time. A communication connection is established between each backup storage node and the primary storage node.

[0045] like Figure 1 As shown, taking Availability Zone 1 and Availability Zone 2 as examples, the primary storage node is deployed in Availability Zone 1, and the backup storage node is deployed in Availability Zone 2. When a client needs to read data, it can send a data read request to the proxy component in the first availability zone.

[0046] The first availability zone can refer to the availability zone where the client is located. For example, if the client's current location is Beijing, then the first availability zone would be the Beijing availability zone. In other words, the first availability zone can be understood as the availability zone closest to the client's current location (either physically or network-wise). Before sending a data read request, the client can first obtain information from various availability zones, then select the closest available availability zone as the first availability zone. The client can then click on the first availability zone on the relevant interface to trigger the corresponding data read request.

[0047] If the first availability zone contains a proxy component, the data read request is sent directly to that proxy component. If the first availability zone contains multiple proxy components, the client can choose any proxy component and send the data read request to it.

[0048] The client can generate read policy indication information based on its own data reading needs. The read policy indication information can be used to instruct whether to read from the primary storage node or the backup storage node.

[0049] Understandably, if the client's primary availability zone is the primary availability zone, the read policy indication information can be used to instruct the client to read the primary storage node. Of course, in some special scenarios, such as a primary storage node failure, it may also instruct the client to read the backup storage node. If the client's primary availability zone is the backup availability zone, the read policy indication information can be flexibly selected according to business needs. For example, for scenarios with high data consistency requirements, the primary storage node can be selected as the priority; while for scenarios sensitive to access latency, the backup storage node within the local availability zone can be selected as the priority, thereby ensuring data availability while reducing latency caused by cross-availability zone access.

[0050] Step S120: The proxy component responds to the data read request and reads the data according to the read policy instruction information.

[0051] After receiving a data read request, the proxy component parses the read policy indication information from it. If the read policy indication information indicates that the primary storage node is to be read, the proxy component reads the data from the primary storage node. If the read policy indication information indicates that the secondary storage node is to be read, the proxy component reads the data from the secondary storage node.

[0052] For example, the first availability zone refers to the above. Figure 1 In Availability Zone 1, where the primary storage node is located, if the read policy indication indicates reading from the primary storage node, the agent component can directly read data from the primary storage node in Availability Zone 1. If the read policy indication indicates reading from the backup storage node, and if a backup storage node also exists in the first Availability Zone, then the data is read from the backup storage node in the first Availability Zone; otherwise, it is read from the primary storage node in the first Availability Zone. If the first Availability Zone refers to... Figure 1 In Availability Zone 2, if the read policy indication information indicates reading from the primary storage node, the proxy component can read data from the primary storage node in Availability Zone 1 across zones. If the read policy indication information indicates reading from the backup storage node, the proxy component will directly read data from the backup storage node in Availability Zone 2.

[0053] Step S130: The proxy component returns the read data to the client.

[0054] The proxy component can send the data read from the corresponding storage node to the client, so that the client can obtain the data it needs.

[0055] In the above implementation process, by introducing read policy indication information into the data read request, the client can flexibly specify whether to read from the primary or backup storage node, thereby achieving fine-grained control over the read policy. This design not only meets the diverse needs of different business scenarios for data consistency and access latency, but also optimizes the overall system performance and resource utilization by making reasonable use of storage node resources in multiple availability zones.

[0056] Based on the above embodiments, before sending a data read request, the client can first generate read policy indication information according to business requirements, and then generate a data read request according to the read policy indication information.

[0057] The business requirements analysis module in the client can analyze and judge the current business operations in real time, thereby analyzing the current business requirements. For example, for the product details page display business of an e-commerce website, when a user is browsing products, the data consistency requirement is relatively low, but the access latency requirement is high. In this case, the business requirements analysis module can determine that this operation is more suitable to prioritize reading data from the backup storage node. This is because if there is a backup storage node in the availability zone, reading data within the local availability zone is more efficient. If the primary storage node is to be read, it may need to be read across zones, which may take longer. In this case, the business requirement is high access latency, and the generated read policy indication information can indicate that the backup storage node should be read.

[0058] For example, for operations involving transactions and payments that have high requirements for data consistency, the business requirement is high data consistency and low access latency. In this case, the generated read policy indication information can indicate reading the primary storage node, because the primary storage node is the first to write data, so the data in the primary storage node is more complete, while the data in other backup storage nodes may not be fully synchronized.

[0059] Therefore, corresponding read policy indication information can be generated for different business needs. Specifically, for business needs that require high data consistency but low access latency, the generated read policy indication information indicates reading the primary storage node. For business needs that require low data consistency but high access latency, the generated read policy indication information indicates reading the backup storage node.

[0060] In the above implementation process, the client can flexibly choose to read from the primary storage node or the backup storage node according to specific business needs, thereby achieving an optimal balance between data consistency and access latency. For example, in scenarios with high real-time requirements, the client can choose to prioritize reading from the backup storage node in the local availability zone to reduce latency; while in scenarios with extremely high data accuracy requirements, it can choose to read from the primary storage node to ensure data consistency. This flexible strategy customization not only improves the system's adaptability and performance but also enhances its scalability and reliability, better meeting complex and ever-changing business needs.

[0061] Based on the above embodiments, the communication protocol between the client and the proxy component can adopt the RESP (REdis Serialization Protocol) protocol, so the client can send a data read request to the proxy component in the first availability zone through the RESP protocol.

[0062] The client can first establish a stable network connection with the proxy component via the TCP / IP protocol, typically using the default Redis segment or other ports configured accordingly. Both the client and proxy components support the RESP protocol for data interaction. The RESP protocol defines the format of requests and responses, including simple strings, errors, integers, batch replies, and multiple batch replies, which can meet the transmission requirements of Redis commands.

[0063] When constructing a data read request using the RESP protocol, the "GET" command can be used to retrieve the value of a specified key, the "HGET" command can be used to retrieve the value of a specified field in a hash table, and so on. These commands can then be converted into a batch string format in the RESP protocol, starting with "$", followed by the string length and "\r\n", then the command string, and ending with "\r\n", such as "$3\r\nGET\r\n" which represents the GET command.

[0064] If a Redis command requires parameters, such as the key name to retrieve, the parameters are encoded according to the RESP protocol. The key is also encoded as a batch string and appended to the command, forming a string array for the entire command. Therefore, a complete `get key` command is encoded as "*2\r\n$3\r\nGET\r\n$3\r\nkey\r\n".

[0065] The constructed RESP request message can then be serialized according to the format specified by the protocol, converting commands and parameters into a byte stream for transmission over the network. During serialization, ensure that the format and special characters (such as \r\n) of each part are correct to guarantee that the proxy component can correctly parse the request message.

[0066] The client, through the established TCP connection with the proxy component, sends the serialized RESP request message (i.e., a data read request) as a data packet. During transmission, according to network communication rules, the data packet may need to be split and encapsulated to meet network transmission requirements. Simultaneously, various abnormal situations during network transmission, such as network interruptions and transmission timeouts, need to be handled to ensure that the request message is successfully delivered to the proxy component.

[0067] After sending the request message, the client waits for a response from the proxy component. The proxy component processes the received data read request, executes the corresponding Redis commands, and encapsulates the result (i.e., the read data) into a response message according to the RESP protocol format and returns it to the client.

[0068] After receiving the RESP protocol-formatted response from the proxy component, the client can parse the response byte by byte according to the RESP protocol specifications and extract the specific response content. The client can also associate the parsed response content with its business logic. For example, if the request is for user information data in a storage node, the client can update its local user information cache or directly display it to the user based on the returned data.

[0069] In the above implementation, the RESP protocol, as the standard communication protocol between Redis clients and servers, supports multiple data types, efficiently handles requests and responses, and ensures the accuracy and speed of data transmission. This design makes communication between the client and the proxy component more standardized and efficient, reduces development and maintenance costs, and also improves the overall performance and stability of the system.

[0070] Based on the above embodiments, in the method of communication between the client and the proxy component via the RESP protocol, the client first generates a data read request in the RESP protocol format, wherein the read policy indication information in the data read request is represented by a custom field in the RESP protocol format.

[0071] like Figure 3 As shown, this solution extends the original RESP protocol, generating an extended RESP protocol in the form of "@n\r\n$4\r\nnear\r\n...\r\n". The fields in the protocol are explained below:

[0072] Regarding the custom protocol header, it begins with a special character to indicate the start of the custom protocol. If this special character is parsed, it indicates that the following protocol content is a custom protocol. The special character indicating the custom protocol header can be any character except "+-:$*_,#!=(%~|>". Next, an integer n represents the number of custom protocol texts, and then "\r\n" indicates the end of the custom protocol header.

[0073] Regarding the nearest-neighbor read protocol, it begins with a single character "$" to indicate that the protocol content is a string. This is followed by an integer "n" representing the length of the protocol content string. Then, the "near" character can be used to indicate that the storage node in the availability zone is to be read first. Finally, it ends with "\r\n" to indicate a custom same-zone priority read protocol. Depending on the number of protocols defined in the custom protocol header, there can be n-1 other interaction protocols following it.

[0074] Understandably, the custom field "near" used in the above read policy indication information essentially indicates reading from the nearest available storage node. In some implementations, the client can know whether the storage node in its availability zone is the primary or secondary storage node. If it is a secondary storage node and the client needs to read from the secondary storage node, the read policy indication information uses the "near" field. If the client's availability zone is the primary storage node and the client's read request is also from the primary storage node, the read policy indication information also uses the "near" field. If the client needs to read data across zones—that is, if its availability zone is a secondary storage node but the client needs to read data from the primary storage node—the custom field can be empty. The proxy component will parse the custom field and find that it does not contain "near," thus knowing that it needs to read data from the primary storage node across zones.

[0075] It should be noted that if the availability zone where the client is located is the primary storage node, cross-zone reads are generally not performed. That is, in some implementations, even if the client's current business requirement is low data consistency but high access latency, data will still be read from the nearest primary storage node within the same availability zone, rather than cross-zone reads, because cross-zone reads have higher latency in this case. In other words, if the availability zone where the client is located is the primary storage node, regardless of its business requirement, the read policy indication information in its data read requests indicates that it is reading from the primary storage node, i.e., reading from the nearest node. The method of generating different read policy indication information for different business requirements in the above embodiments can be for the case where the storage node in the availability zone where the client is located is a backup storage node. Of course, if the primary storage node fails or other factors cause the primary storage node to be unreadable, the read policy indication information indicates that it is reading from the backup storage node.

[0076] In addition, the custom fields used in the read strategy indication information mentioned above can be flexibly set according to actual needs. Different custom fields can be used to represent reading the primary storage node or reading the backup storage node in order to distinguish them.

[0077] As an example, when generating a data read request according to the above format, the generated read strategy instruction information can be embedded into the data read request according to the format requirements of the private RESP protocol. Specifically, first add a custom protocol header, starting with a special character (such as "@"), followed by an integer n representing the length of the custom protocol text, and then add \r\n to indicate the end of the custom protocol header. Next, start with the character "$", followed by an integer representing the length of the protocol content string "near" (4), then the "near" string itself, and finally end with \r\n. The generated data read request then indicates that data should be read from the nearest available zone's storage node. If the storage node in the available zone where the client is located is a backup storage node, it indicates that data should be read from the backup storage node; if the storage node in the available zone where the client is located is a primary storage node, it indicates that data should be read from the primary storage node.

[0078] In the above implementation, by introducing a custom field into the RESP protocol to represent read policy indication information, flexible extension within the standard protocol framework is achieved. This design not only retains the efficiency and compatibility of the RESP protocol, but also allows clients to dynamically specify read policies according to business needs without requiring large-scale modifications to the existing protocol.

[0079] Based on the above embodiments, if an availability zone includes at least two proxy components, such as the first availability zone including at least two proxy components, then when the client sends a data read request, it can send the data read request to the proxy component that is closest to it in the first availability zone.

[0080] Before sending a data read request, the client can first read basic information such as the network location of all proxy components within its first availability zone from the pre-configured configuration file. Alternatively, the client can send requests to each proxy component to obtain its network location, and each proxy component will return its own network location. The client can obtain its own network location, which can refer to the actual physical address, or to specific network addresses such as data center, switch location, IP address, and subnet IP segment.

[0081] If it's an actual physical address, the nearest proxy component can be determined by calculating the physical distance. If it's a network address, the nearest proxy component can be evaluated based on information such as the shortest hop count of the physical network topology, network latency, and bandwidth.

[0082] If network latency is the primary criterion, the client can send probe requests to each proxy component to measure the round-trip time with each proxy component and select the proxy component with the shortest round-trip time as the closest proxy component.

[0083] Once the nearest proxy component is identified, a data read request can be sent to that proxy component.

[0084] In some implementations, after sending a data read request to the nearest proxy component, the client can set a reasonable timeout period to wait for the proxy component's response. If no response is received within the timeout period, the client can select another proxy component to make the request, or perform appropriate error handling according to specific business requirements.

[0085] In some implementations, the client needs to establish a connection with the proxy component before sending a data read request. To improve performance and reduce the overhead of frequent connection establishment and termination, the client can maintain a connection pool to store connections with various proxy components. For connections already established with the nearest proxy component, they are not immediately closed after the request is sent, but are returned to the connection pool for reuse in subsequent requests.

[0086] The client can periodically monitor the status of connections in the connection pool. For invalid connections (such as those caused by proxy component offline or network failure), the client can remove them from the connection pool in a timely manner and re-establish new connections, thereby ensuring that the client can continuously send data read requests to the proxy component in the availability zone.

[0087] In the above implementation process, by having the client send a data read request to the nearest proxy component in the first availability zone, network latency is significantly reduced and the response speed of data reading is improved.

[0088] Based on the above embodiments, if the storage node in the first availability zone is a backup storage node, when the proxy component reads data, if the read policy indication information in the data read request indicates reading the primary storage node, the proxy component responds to the data read request and reads data across the availability zone; if the read policy indication information in the data read request indicates reading the backup storage node, the proxy component responds to the data read request and reads data from the storage node in the first availability zone.

[0089] If the proxy component parses the read policy indication information in the data read request to indicate reading from the primary storage node, it can determine the availability zone where the primary storage node is located based on the pre-configured architecture information of the data read / write system. Then, it sends a data read request to the primary storage node within that availability zone to perform the data read. Upon receiving the request, the primary storage node executes the corresponding data read operation, retrieves the required data from its stored data, and returns the result to the proxy component.

[0090] If the proxy component parses the read policy indication information in the data read request to indicate reading from the backup storage node, the proxy component can select one of the backup storage nodes in the first availability zone to send a data acquisition request to acquire the data, based on the configuration information of the backup storage nodes. If there are multiple backup storage nodes, the proxy component can select one of the backup storage nodes to acquire the data.

[0091] In the above implementation process, flexible read strategy instructions enable the proxy component to selectively read data based on the client's needs. When strong consistency data is required, the proxy component reads the primary storage node across availability zones to ensure data accuracy; while when data consistency requirements are lower but access latency is sensitive, the proxy component prioritizes reading data from the backup storage node in the local availability zone to reduce latency.

[0092] Based on the above embodiments, multiple storage nodes can refer to nodes in the Redis system, or multiple storage nodes can refer to nodes in the KV system.

[0093] In a Redis system, the master storage node is the master Redis instance, and the slave storage node is the slave Redis instance. Data synchronization is achieved through the Redis master-slave replication mechanism. In a key-value (KV) system, the master storage node and the slave storage node are respectively master and slave KV storage nodes, using the corresponding master-slave replication mechanism to synchronize data.

[0094] In the system, each agent component can maintain a list of storage nodes, including information such as the network address, primary / standby role, and availability zone of each storage node. This list can be pre-configured through a configuration file or dynamically obtained and updated through the registry center.

[0095] The proxy component establishes pre-connections with each storage node to facilitate rapid request forwarding. For Redis systems, the proxy component uses a Redis client library to interact with the storage nodes; for key-value systems, it uses a corresponding client library or a custom protocol to communicate with the storage nodes.

[0096] Redis's master-slave replication mechanism automatically ensures data synchronization. The agent component can monitor the replication status; if a slave node's latency is too high, read requests to it can be temporarily reduced or an administrator can be alerted. Key-value (KV) systems also rely on their master-slave replication mechanism to synchronize data.

[0097] Data synchronization between the primary storage node in the primary availability zone and the backup storage nodes in other availability zones is achieved through Redis's master-slave replication mechanism. The primary storage node sends write operation records (such as writing new key-value pairs, updating data, etc.) to each backup storage node in the form of a command log. After receiving the command log, the backup storage nodes execute the same write operations locally, thus ensuring data consistency. However, it should be noted that there may be a certain delay in data synchronization, which may cause the data on the backup storage nodes to be slightly inconsistent with the data on the primary storage nodes at some point.

[0098] In the above implementation process, this solution can be flexibly applied to various storage architectures, improving the system's scalability and compatibility. Whether it's a Redis-based in-memory storage system or other key-value storage systems, this solution can achieve efficient data reading and load balancing through unified read policy instructions and intelligent routing by the proxy component.

[0099] Based on the above embodiments, when writing data, the client can first determine the second availability zone where the primary storage node is located, and the client sends a data write request to the proxy component of the second availability zone. The proxy component responds to the data write request and writes the data into the primary storage node.

[0100] The client can first obtain the second availability zone where the primary storage node is located from the architecture information of the data read and write system. This information can be stored in the client in advance through configuration files.

[0101] The client can locate the record corresponding to the primary storage node based on the storage node's configuration information. For example, the primary storage node can be determined based on the node's role field, and then the availability zone field value can be obtained to determine that the primary storage node is located in the second availability zone.

[0102] The client can obtain a list of proxy components within the second availability zone from a configuration file or the registry center. If there are multiple proxy components, the client can select the closest one. The selection method can refer to the relevant implementation methods in the above embodiments, which will not be repeated here. Then, the client can construct a data write request according to the RESP protocol format, similar to a data read request, and send the data write request to the proxy component.

[0103] The proxy component forwards data write requests to the primary storage node based on internal routing rules and storage node configuration. The primary storage node responds to the data write request and performs the data write. After completing the data write operation, the primary storage node can return the write result (such as a success or failure message) to the proxy component. Upon receiving this message, the proxy component parses the result information and returns it as a response to the client's write request.

[0104] After the primary storage node writes new data, it can synchronize the newly written data to other backup storage nodes according to the data synchronization mechanism described above.

[0105] In the above implementation process, the client actively determines the second availability zone where the primary storage node is located and sends the data write request to the proxy component of that availability zone, thus achieving precise routing of data write operations.

[0106] Based on the above embodiments, the proxy component can also cache some written data. When a client reads data, it can first read from the proxy component's cache; if the data is not found there, it will then read from the storage node. Therefore, the data read request can also include access policy indication information, which is used to indicate whether to access the proxy component's local cache or access the storage node of the availability zone.

[0107] The client can determine whether to access the local cache of the proxy component or the storage node of the availability zone for this data read operation based on the needs of the current business logic. For example, for frequently read and relatively stable data, accessing the local cache may be preferred to improve response speed, while for data that requires the latest data or critical business data, accessing the storage node of the availability zone may be chosen.

[0108] A data read request is constructed according to the RESP protocol, which also includes access policy indication information. Accessing local cache and accessing storage nodes in availability zones can be represented by different fields. The access policy indication information can be encapsulated in n-1 other protocol texts in the RESP protocol format.

[0109] Upon receiving a data read request, the proxy component can parse the access policy indication and read policy indication. If the access policy indication indicates access to the local cache, the proxy component searches for the corresponding data in the local cache. If the data is found, it is returned to the client. If not found, the data is read from the corresponding storage node according to the read policy indication. Alternatively, if the data is still not found, a miss message can be returned to the client. If the access policy indication indicates access to a storage node, the proxy component can read the data from the corresponding storage node according to the read policy indication.

[0110] Understandably, if data is read from the backup storage node first, since the data in the backup storage node may differ from the data in the primary storage node, if the proxy component fails to retrieve the data from the backup storage node, the proxy component can return a miss message to the client, or the proxy component can forward the request to the primary storage node to retrieve the data from the primary storage node.

[0111] In the above implementation, by introducing access policy indication information into data read requests, clients can flexibly choose to access the local cache of the proxy component or directly access the storage node of the availability zone. This design not only improves the system's flexibility and response speed but also optimizes resource utilization efficiency. For example, for frequently read data with low real-time requirements, accessing the local cache can significantly reduce latency and improve performance; while for scenarios requiring the latest data, directly accessing the storage node ensures data consistency and accuracy.

[0112] In summary, this application provides a data read / write method and system. This method introduces read policy indication information into data read requests, enabling clients to flexibly specify whether to read from the primary or backup storage node, thus achieving fine-grained control over the read policy. This design not only meets the diverse needs of different business scenarios for data consistency and access latency but also optimizes the overall system performance and resource utilization by rationally utilizing storage node resources across multiple availability zones. For example, for scenarios with high data consistency requirements, the primary storage node can be read first; while for scenarios sensitive to access latency, the backup storage node within the local availability zone can be read first, thereby ensuring data availability while reducing latency caused by cross-availability zone access.

[0113] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0114] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0115] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0116] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0117] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data read / write method, characterized in that, The method includes: The client sends a data read request to the proxy component in the first availability zone. The data read request includes read policy indication information, which is used to indicate whether to read the primary storage node or the backup storage node. The multiple storage nodes include primary storage nodes and backup storage nodes, which are distributed in multiple availability zones. The multiple availability zones are used to indicate the data center corresponding to the client. The proxy component responds to the data read request and reads data according to the read strategy instruction information; The proxy component returns the read data to the client.

2. The method according to claim 1, characterized in that, Before the client sends a data read request to the proxy component in the first availability zone, it also includes: The client generates read policy instruction information based on business requirements; The client generates a data read request based on the read policy instruction information.

3. The method according to claim 1, characterized in that, The client sends a data read request to the proxy component in the first availability zone, including: The client sends a data read request to the proxy component in the first availability zone via the RESP protocol.

4. The method according to claim 3, characterized in that, The client sends a data read request to the proxy component in the first availability zone via the RESP protocol, including: The client generates a data read request in RESP protocol format, wherein the read policy indication information in the data read request is represented by a custom field in the RESP protocol format.

5. The method according to claim 1, characterized in that, The first availability zone includes at least two proxy components. The client sends a data read request to the proxy component in the first availability zone, including: The client sends a data read request to the nearest proxy component within the first availability zone.

6. The method according to any one of claims 1-5, characterized in that, The storage nodes in the first availability zone are backup storage nodes. The proxy component responds to the data read request and reads data according to the read policy instruction information, including: If the read policy indication information in the data read request indicates reading the primary storage node, then the proxy component responds to the data read request and performs data reading across availability zones; If the read policy indication information in the data read request indicates reading from a backup storage node, then the proxy component responds to the data read request and reads data from a storage node in the first availability zone.

7. The method according to claim 1, characterized in that, The multiple storage nodes refer to nodes in the Redis system, or the multiple storage nodes refer to nodes in the KV system.

8. The method according to claim 1, characterized in that, The method further includes: The client determines the second availability zone where the primary storage node is located; The client sends a data write request to the proxy component of the second availability zone; The proxy component of the second availability zone responds to the data write request and writes the data to the primary storage node.

9. The method according to claim 1, characterized in that, The data read request also includes access policy indication information, which is used to indicate whether to access the local cache of the proxy component or to access the storage node of the availability zone.

10. A data read / write system, characterized in that, The system includes: a client, a proxy component, and a storage node; A client is used to send a data read request to a proxy component in the first availability zone. The data read request includes read policy indication information, which is used to indicate whether to read the primary storage node or the backup storage node. The multiple storage nodes include primary storage nodes and backup storage nodes, which are distributed in multiple availability zones. The multiple availability zones are used to indicate the data center corresponding to the client. The proxy component is used to read data in response to the data read request, according to the read strategy instruction information; The proxy component is also used to return the read data to the client.

Images