Method and apparatus for data storage and data retrieval

Abstract

The application discloses a data storage and data reading method and device, and relates to the technical field of computers. A specific implementation of the method comprises the following steps: saving a received order change message into a message queue of a plurality of clusters, and constructing order data by acquiring the order change message in the queue, and storing the order data in a storage medium of each cluster. The implementation realizes the reliability of data storage through the architecture of the plurality of clusters, so that subsequent data query can be realized through different clusters, and the disaster recovery of the system is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for data storage and data retrieval. Background Technology

[0002] In the field of internet technology, data storage primarily relies on various databases. In data query scenarios, when facing high-concurrency queries (High QPS) on the internet, relying on structured databases such as MySQL (Relational Database Management System) results in poor performance. Given the current scale of online retail traffic, relying on structured databases not only presents significant bottlenecks but also incurs high costs. Therefore, we will consider upgrading the structured database to a heterogeneous full-text search engine (Elasticsearch, ES) to address the aforementioned issues.

[0003] However, when using Elasticsearch (ES) to provide query services, if ES issues a warning and becomes unusable, the ES service will be shut down and downgraded to MySQL or other query services, providing basic query capabilities. Users will then be unable to use full-text search capabilities, resulting in a poor user experience. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for data storage and data retrieval, which can improve the reliability of data storage through a multi-cluster approach, and improve system disaster recovery through mutual backup relationships between clusters.

[0005] To achieve the above objectives, according to one aspect of the present invention, a data storage method is provided, comprising:

[0006] Receive order change messages and save them to message queues corresponding to multiple clusters, including multiple primary clusters and backup clusters deployed in different locations;

[0007] Retrieve order change messages from the message queue corresponding to each cluster, and construct order data based on the order change messages;

[0008] For each cluster, the storage medium corresponding to the cluster is determined based on the message topic of the cluster, and the order data is stored in the storage medium corresponding to the cluster.

[0009] Optionally, the order data includes the order number and the current order status; storing the order data in the corresponding storage medium of the cluster includes:

[0010] Lock the order based on the order number and check the latest status of the order;

[0011] In response to the latest status of the order being consistent with the current status of the order, the order data is stored in the corresponding storage medium of the cluster.

[0012] According to another aspect of the present invention, a method for reading data is provided, comprising:

[0013] Obtain the request forwarding strategy based on the order query request;

[0014] The target forwarding cluster is determined based on the request forwarding strategy. These multiple clusters are different service clusters that store the same order data through order change messages.

[0015] The order query request is forwarded to the target forwarding cluster, which then performs the order query based on the request to obtain the target order data.

[0016] Optionally, the request forwarding strategy includes a vertical invocation strategy, which determines the target forwarding cluster among multiple clusters, including:

[0017] In response to the request forwarding strategy being a vertical invocation strategy, the cluster corresponding to the source identifier of the order query request among multiple clusters is identified as the target forwarding cluster.

[0018] Optionally, the request forwarding strategy includes a percentage-based invocation strategy, which determines the target forwarding cluster among multiple clusters, including:

[0019] In response to the request forwarding strategy being a percentage-based call strategy, the traffic percentage of the cluster corresponding to the source identifier of the order query request is obtained;

[0020] The target forwarding cluster among multiple clusters is determined based on the traffic share.

[0021] Optionally, the request forwarding strategy includes a mutual backup call strategy, which determines the target forwarding cluster among multiple clusters based on the request forwarding strategy, including:

[0022] In response to the request forwarding strategy being a mutual backup call strategy, the cluster corresponding to the source identifier of the order query request among multiple clusters is determined as the target forwarding cluster;

[0023] The method also includes:

[0024] In response to the failure to forward the order query request to the target forwarding cluster, the target forwarding cluster is updated to a cluster with a mutual backup relationship with it.

[0025] Optionally, before determining the target forwarding cluster among multiple clusters based on the request forwarding strategy, the method further includes:

[0026] The failover strategy has been determined to be disabled.

[0027] The method also includes:

[0028] In response to the determination that the failover strategy has been initiated;

[0029] Order query requests are forwarded to backup clusters in multiple clusters, allowing the backup clusters to query orders based on the requests and obtain the target order data. The order data stored in the backup clusters is the same as that in the multiple clusters.

[0030] According to another aspect of the present invention, a data storage apparatus is provided, comprising:

[0031] The receiving module is used to receive order change messages and save the order change messages to message queues corresponding to multiple clusters, including multiple primary clusters and backup clusters deployed in different locations.

[0032] The acquisition module is used to retrieve order change messages from the message queue corresponding to each cluster and construct order data based on the order change messages;

[0033] The storage module is used to determine the corresponding storage medium for each cluster based on the message topic of the cluster, and to store the order data in the corresponding storage medium.

[0034] According to another aspect of the present invention, a data reading apparatus is provided, comprising:

[0035] The acquisition module is used to obtain the request forwarding strategy based on the order query request;

[0036] The determination module is used to determine the target forwarding cluster among multiple clusters based on the request forwarding strategy. These multiple clusters are different service clusters that store the same order data through order change messages.

[0037] The forwarding module is used to forward order query requests to the target forwarding cluster, so that the target forwarding cluster can perform order queries based on the order query requests and obtain the target order data.

[0038] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method or data retrieval method provided in the embodiments of the present invention.

[0039] According to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the data storage method or data reading method provided in the embodiments of the present invention.

[0040] According to another aspect of the present invention, a computer program product is provided, including a computer program that, when executed by a processor, implements the data storage method or data retrieval method provided in the embodiments of the present invention.

[0041] One embodiment of the above invention has the following advantages or beneficial effects: after receiving an order change message, it is stored in the message queues of multiple primary and backup clusters deployed in different locations. The order change message is further obtained through the message queues, the order data is constructed, and finally the order data is stored in the storage medium corresponding to each cluster, realizing multi-cluster storage of order data, improving the reliability of data storage, thereby ensuring the smooth progress of subsequent data reading processes and improving the system's disaster recovery capability.

[0042] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0043] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0044] Figure 1 This is a diagram illustrating the overall data processing architecture of the present invention.

[0045] Figure 2 This is a schematic diagram of the main steps of a data storage method according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the order data writing process;

[0047] Figure 4 This is a schematic diagram illustrating the main steps of a data reading method according to an embodiment of the present invention;

[0048] Figure 5 A flowchart illustrating the process of executing an order query request;

[0049] Figure 6 This is a schematic diagram of the main modules of a data storage device according to an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of the main modules of a data reading device according to an embodiment of the present invention;

[0051] Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0052] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0053] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0054] It should be noted that the technical solutions disclosed in this invention, regarding the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0055] It should be noted that the collection, use, storage, sharing and transfer of user personal information involved in the technical solution of the present invention all comply with the provisions of relevant laws and regulations, and require notification to users and obtaining their consent or authorization. When applicable, user personal information is subjected to de-identification and / or anonymization and / or encryption technical processing.

[0056] Figure 1 The diagram shows the overall data processing architecture of this invention. Order change messages from user terminals are forwarded to the order service cluster via a load balancer. The order service cluster obtains the message distribution strategy from the configuration center and distributes the message to the message middleware. The order service cluster stores the order data in a database to achieve persistent order data storage, and also stores the data in multiple Elasticsearch (ES) clusters. Subsequent order data queries are then performed based on the ES clusters.

[0057] Figure 2 This is a schematic diagram illustrating the main steps of a data storage method according to an embodiment of the present invention, as shown below. Figure 2 As shown, it includes steps S201 to S203.

[0058] Step S201: Receive order change messages and save the order change messages to message queues corresponding to multiple clusters, including multiple primary clusters and backup clusters deployed in different locations.

[0059] Upon receiving an order change message, the message is saved to message queues corresponding to multiple clusters, ensuring that the messages stored in each cluster's message queue are identical. These clusters include multiple primary clusters and backup clusters, with the primary clusters deployed in different locations. Specifically, one backup cluster can be selected.

[0060] Optionally, upon receiving an order change message, relevant business operations need to be performed first. After the operations are completed, it is determined whether to initiate heterogeneous Elasticsearch (ES) operations. If not, write message distribution will not be performed; if so, subsequent write message distribution operations will be performed. First, the heterogeneous ES operation information is forwarded to the internal message queue through the message distribution unit, enabling the system to perform cluster message writing operations based on the internal message queue.

[0061] The business operations include order placement and order dispatch notification to the system. Heterogeneous Elasticsearch (ES) processes can be determined by pre-configured parameters. For example, a switch parameter for heterogeneous ES can be set; a parameter of 1 starts the process, while a parameter of 0 does not.

[0062] Optionally, when saving order change messages, multiple message topics of the primary clusters are first obtained from the configuration center, and order change messages are sent to the message topics. After the message is successfully sent, the message topic of the backup cluster is obtained, and the order change message is sent to the message topic. After the message is sent, multiple message queues storing the same order change messages are obtained to ensure that multiple clusters can store the same data based on the order change messages.

[0063] Optionally, when sending order change messages to message topics in different master clusters, the messages can be sent sequentially to each message topic or sent to each message topic simultaneously.

[0064] Step S202: Obtain order change messages from the message queue corresponding to each cluster, and construct order data based on the order change messages.

[0065] For each cluster, when data needs to be written, the first step is to retrieve order change messages from the message queue and then construct order data based on these messages. Specifically, the corresponding change information is obtained based on the change operation in the order change message, and the order data is constructed by combining this information with the order number in the order change message.

[0066] Step S203: For each cluster, determine the storage medium corresponding to the cluster based on the message topic corresponding to the cluster, and store the order data in the storage medium corresponding to the cluster.

[0067] Within each cluster, the corresponding storage medium is determined based on the cluster's message topic. For example, message topic 'a' in cluster A corresponds to storage medium A. After determining the storage medium, the constructed order data is stored in the corresponding storage medium. Once data storage is complete, the order data stored in each cluster is identical.

[0068] Optionally, multiple master clusters can be mutually redundant. That is, when one master cluster fails and cannot handle order change messages, the order change messages can be forwarded to other master clusters. Specifically, the selection method of other master clusters can be set according to the actual situation, such as selecting based on the distance from the failed cluster, selecting a cluster with lower traffic based on the current traffic value of the cluster, etc., without any restrictions.

[0069] The data storage method provided by the embodiments of the present invention can, after receiving an order change message, store it in the message queues of multiple primary and backup clusters deployed in different locations, further obtain the order change message through the message queues, construct order data, and finally store the order data in the storage medium corresponding to each cluster, thereby realizing multi-cluster storage of order data, improving the reliability of data storage, ensuring the smooth progress of subsequent data reading processes, and improving the system's disaster recovery capability.

[0070] Optionally, the order data includes the order number and the current status of the order; storing the order data in the storage medium corresponding to the cluster includes: locking the order based on the order number and querying the latest status of the order; and storing the order data in the storage medium corresponding to the cluster in response to the latest status of the order being consistent with the current status of the order.

[0071] To ensure the accuracy of data writing, order status needs to be checked. Specifically, the order is locked based on the order number to prevent changes to the order data during the check process, and the latest order status is obtained. The latest status is then compared with the order status in the order data. If they match, the order data is stored in the corresponding storage medium; otherwise, no action is taken, and the old data is allowed to overwrite the new data, thus reverting the order status.

[0072] Optionally, before constructing the order data (storage entity), the latest status of the order is first determined based on the order number in the order change message, and the order data is constructed based on the latest status.

[0073] like Figure 3 The diagram illustrates the order data writing process. It receives and processes order change messages generated after order creation or modification. It determines whether heterogeneous Elasticsearch (ES) operations are needed; if not, the process ends; otherwise, it sends an asynchronous message to the message queue. Consuming asynchronous messages involves message consumption from multiple clusters. Order data is constructed based on messages from different clusters and written into the corresponding processes. The order data writing process ends after all clusters' order data has been written.

[0074] The data storage method provided by this embodiment of the invention can store received order change messages to message queues of multiple primary and backup clusters deployed in different locations. Furthermore, order data is constructed using these order change messages, and finally, the order data is stored in the storage medium corresponding to each cluster, improving data storage reliability and system disaster recovery capabilities. In addition, when constructing order data, the order status can be determined to ensure the order of message storage, further guaranteeing the accuracy of data storage.

[0075] Figure 4 This is a schematic diagram illustrating the main steps of a data reading method according to an embodiment of the present invention, as follows: Figure 4 As shown, it includes steps S401 to S403.

[0076] Step S401: Obtain the request forwarding strategy based on the order query request.

[0077] Upon receiving an order query request, the current request forwarding strategy is retrieved from the configuration center to determine the set of requests to forward. Specifically, the configuration center can determine different request forwarding strategies based on parameter values. For example, the parameter `switch_strategy` can be set, and different strategies correspond to different values; for instance, a value of 1 corresponds to strategy a, a value of 2 corresponds to strategy b, and so on.

[0078] The request forwarding strategies include vertical call strategy, percentage call strategy, and mutual backup call strategy.

[0079] Step S402: Determine the target forwarding cluster among multiple clusters based on the request forwarding strategy. The multiple clusters are different service clusters that store the same order data through order change messages.

[0080] Based on the established request forwarding strategy, it is determined that order query requests need to be forwarded to the target forwarding cluster among multiple clusters. These multiple clusters are different clusters storing the same order data, allowing the system to flexibly select different clusters for querying order data.

[0081] Optionally, the target forwarding cluster can be the cluster to which the order query request belongs. For example, an order query request generated in location A is forwarded to the cluster in location A.

[0082] Step S403: Forward the order query request to the target forwarding cluster, so that the target forwarding cluster can perform an order query based on the order query request and obtain the target order data.

[0083] The order query request is forwarded to the designated target forwarding cluster, which then performs a data query based on the order query request and generates the target order data based on the query results.

[0084] The data reading method according to embodiments of the present invention can determine the target forwarding cluster based on a multi-cluster architecture, combining order query requests and request forwarding strategies. This enables the cluster to determine and read target order data based on query requests, ensuring accurate data retrieval. Simultaneously, the multi-cluster architecture improves system reliability, disaster recovery capabilities, and stability. Furthermore, storing the same order data in multiple clusters enables data redundancy, reducing the risk of system failure.

[0085] Optionally, the request forwarding strategy includes a vertical invocation strategy, which determines the target forwarding cluster among multiple clusters based on the request forwarding strategy, including: in response to the request forwarding strategy being a vertical invocation strategy, determining the cluster among multiple clusters that corresponds to the source identifier of the order query request as the target forwarding cluster.

[0086] If the determined request forwarding strategy is a vertical call strategy, the target forwarding cluster is determined based on the source identifier of the order query. Specifically, if the source identifier of the order query is location A, then the cluster in location A is determined as the target forwarding cluster; if the source identifier is location B, then the target forwarding cluster is location B.

[0087] Optionally, the request forwarding strategy includes a percentage-based invocation strategy. Based on the request forwarding strategy, the target forwarding cluster among multiple clusters is determined, including: in response to the request forwarding strategy being a percentage-based invocation strategy, obtaining the traffic percentage of the cluster corresponding to the source identifier of the order query request; and determining the target forwarding cluster among multiple clusters based on the traffic percentage.

[0088] If the request forwarding strategy is percentage-based, the target forwarding cluster needs to be selected based on the traffic share among multiple clusters. Specifically, the traffic share of the cluster corresponding to the order query request source identifier can be obtained, and it can be determined whether it exceeds a preset threshold. If it exceeds the preset threshold, it indicates that the traffic of that cluster is high. In order to control the traffic value, the order query request needs to be forwarded to other clusters. Specifically, the cluster with the lowest traffic among other clusters can be selected, or a random cluster can be selected, etc.

[0089] Alternatively, the percentage-based call strategy can be as follows: after determining the cluster corresponding to the source identifier of the order query request, a random number in the range of 1-100 is randomly generated and compared with the current traffic percentage of that cluster. If the random number is less than the traffic percentage, then that cluster is determined as the target forwarding cluster; if it is not less than the traffic percentage, then any other cluster is determined as the target forwarding cluster.

[0090] Optionally, the request forwarding strategy includes a mutual backup call strategy. Based on the request forwarding strategy, a target forwarding cluster among multiple clusters is determined, including: in response to the request forwarding strategy being a mutual backup call strategy, determining the cluster among multiple clusters corresponding to the source identifier of the order query request as the target forwarding cluster; the method further includes: in response to the failure to forward the order query request to the target forwarding cluster, updating the target forwarding cluster to a cluster with a mutual backup relationship with it.

[0091] The mutual backup call strategy is an optimization strategy based on the vertical call strategy. That is, the cluster corresponding to the source identifier of the order query request is first determined as the target forwarding cluster. When the target forwarding cluster cannot perform the order query based on the request, that is, the order query operation fails, the order query request will be forwarded to the cluster with which it has a mutual backup relationship.

[0092] Optionally, before determining the target forwarding cluster among multiple clusters based on the request forwarding strategy, the method further includes: determining that the failover strategy is not enabled; the method further includes: in response to determining that the failover strategy is enabled, forwarding the order query request to a standby cluster of multiple clusters, so that the standby cluster performs an order query based on the order query request to obtain the target order data, and the order data stored in the standby cluster is the same as that in the multiple clusters.

[0093] Before determining the request forwarding strategy, it is necessary to check whether the system's failover strategy is enabled. If not, the request forwarding strategy process described above is executed. If enabled, the failover strategy is executed instead of the request forwarding strategy. Specifically, if not enabled, the order query request is forwarded to the backup cluster, allowing the backup cluster to query the order data based on the request. This backup cluster is a backup cluster for multiple clusters, storing order data from multiple clusters, and the order data storage process of the backup cluster is the same as that of the multiple clusters.

[0094] Specifically, when the Elasticsearch primary cluster fails, such as due to network outages in the dual-hot clusters, or due to large SQL queries from users causing memory and CPU usage to reach full capacity, the disaster recovery plan for the primary cluster becomes ineffective. Failover can only be achieved by switching to a standby cluster. Maintenance personnel can enable failover through configuration information. Once enabled, the system will obtain the standby cluster, i.e., the cluster Elasticsearch client, when obtaining storage clients (storage media). The requests will ultimately be processed by the standby cluster, achieving the goal of rapid fault recovery.

[0095] Optionally, such as Figure 5The diagram illustrates the execution flow of an order query request. Upon starting the order query and determining a non-failover strategy, the primary cluster is retrieved. The request forwarding strategy is then obtained to determine the target forwarding cluster. If the retrieved forwarding strategy is vertical call, the cluster corresponding to the request is retrieved; for example, if the corresponding cluster is A, the storage media of cluster A is retrieved and queried. If the retrieved forwarding strategy is percentage call, the traffic percentage of each cluster is obtained, and a random number between 1 and 100 is generated. First, it is determined whether the random number falls within the traffic percentage range of the cluster corresponding to the request. If it does, the storage media of that cluster is retrieved; otherwise, the storage media of other clusters are retrieved and queried. If the retrieved forwarding strategy is mutual backup call, a rule is added to the vertical call strategy: if the query fails, other clusters are invoked for the query.

[0096] Optionally, based on the mutual backup call strategy, a single-master circuit breaker automatic switching peer master cluster strategy can be further optimized. If a large number of order query requests cannot be processed in a single cluster within a short period of time, but requests continue to hit that cluster, switching to the backup cluster would increase the application cluster load and potentially cause a cascading failure. Therefore, this strategy was developed. The anomaly stratification corresponding to this strategy is a long-term failure of a single-hot ES cluster, such as continuous high CPU usage, continuous high memory usage, disk failure, machine failure, etc. When traffic hits a cluster and an anomaly occurs, the Hystrix component (circuit breaker and degradation component) records the anomaly event and the number of anomalies per unit time. If it exceeds a certain threshold, the Hystrix component directly performs a circuit breaker, switching the traffic to the peer hot cluster (master cluster). In subsequent requests, the Hystrix component will intermittently try the current cluster. If the availability reaches a certain threshold, the cluster call will be restarted, and the degradation will be removed. For example, if a current order query request originates from location A, it will first be distributed to the cluster in location A. If a large number of queries distributed to cluster A fail within a short period, traffic to cluster A will be circuit-broken. This means that order query requests originating from location A will also be forwarded to other clusters that have a mutual backup relationship with cluster A. This mutual backup relationship refers to each other acting as backup databases. Cluster A will only be reactivated after its availability is tested and reaches a preset threshold.

[0097] The data reading method according to embodiments of the present invention can realize order data query based on a multi-cluster architecture. Specifically, it determines the target forwarding cluster by combining the order query request and the request forwarding strategy, enabling it to query the target order data based on the query request, thus ensuring the reliability of data reading. Simultaneously, the multi-cluster architecture achieves mutual backup between clusters and free distribution of requests, improving the system's request processing capacity and disaster recovery capabilities, and reducing system risk. Furthermore, different request forwarding strategies can be applied to different scenarios, improving system flexibility and request processing efficiency.

[0098] Figure 6 This is a schematic diagram of a data storage device 600 provided in an embodiment of the present invention, as shown below. Figure 6 As shown, it includes a receiving module 601, an acquisition module 602, and a storage module 603.

[0099] The receiving module 601 is used to receive order change messages and save the order change messages to message queues corresponding to multiple clusters, including multiple primary clusters and backup clusters deployed in different locations.

[0100] The acquisition module 602 is used to retrieve order change messages from the message queue corresponding to each cluster and construct order data based on the order change messages;

[0101] Storage module 603 is used to determine the corresponding storage medium for each cluster based on the message topic of the cluster, and to store the order data in the corresponding storage medium.

[0102] The data storage apparatus provided in this embodiment of the invention can, after receiving an order change message, store it in the message queues of multiple primary and backup clusters deployed in different locations, further obtain the order change message through the message queues, construct order data, and finally store the order data in the storage medium corresponding to each cluster, thereby realizing multi-cluster storage of order data, improving the reliability of data storage, ensuring the smooth progress of subsequent data reading processes, and improving the system's disaster recovery capability.

[0103] Optionally, the order data includes the order number and the current status of the order; the storage module 603 is also used to lock the order based on the order number and query the latest status of the order;

[0104] In response to the latest status of the order being consistent with the current status of the order, the order data is stored in the corresponding storage medium of the cluster.

[0105] The data storage apparatus provided in this embodiment of the invention can store received order change messages to message queues of multiple primary and backup clusters deployed in different locations. Furthermore, it can construct order data using these order change messages and finally store the order data in the storage medium corresponding to each cluster, thereby improving data storage reliability and system disaster recovery capabilities. In addition, when constructing order data, the order status can be determined to ensure the order of message storage, further guaranteeing the accuracy of data storage.

[0106] Figure 7 This is a schematic diagram of a data reading device 700 provided in an embodiment of the present invention, as shown below. Figure 7 As shown, it includes an acquisition module 701, a determination module 702, and a forwarding module 703.

[0107] Module 701 is used to obtain the request forwarding strategy based on the order query request;

[0108] The determination module 702 is used to determine the target forwarding cluster among multiple clusters based on the request forwarding strategy. The multiple clusters are different service clusters that store the same order data through the order change message.

[0109] Forwarding module 703 is used to forward order query requests to the target forwarding cluster, so that the target forwarding cluster can perform order queries based on the order query requests and obtain the target order data.

[0110] The data reading apparatus according to embodiments of the present invention can determine the target forwarding cluster based on a multi-cluster architecture, combining order query requests and request forwarding strategies. This enables the cluster to determine and read target order data based on query requests, ensuring accurate data retrieval. Simultaneously, the multi-cluster architecture improves system reliability, disaster recovery capabilities, and stability. Furthermore, storing the same order data in multiple clusters enables data redundancy, reducing the risk of system failure.

[0111] Optionally, the request forwarding strategy includes a vertical invocation strategy. The determination module 702 is further configured to, in response to the request forwarding strategy being a vertical invocation strategy, determine the cluster among multiple clusters that corresponds to the source identifier of the order query request as the target forwarding cluster.

[0112] Optionally, the request forwarding strategy includes a percentage-based call strategy. The determination module 702 is further configured to, in response to the request forwarding strategy being a percentage-based call strategy, obtain the traffic percentage of the cluster corresponding to the source identifier of the order query request; and determine the target forwarding cluster among multiple clusters based on the traffic percentage.

[0113] Optionally, the request forwarding strategy includes a mutual backup call strategy. The determination module 702 is further configured to determine, in response to the request forwarding strategy being a mutual backup call strategy, the cluster corresponding to the source identifier of the order query request among multiple clusters as the target forwarding cluster. The device also includes an update module 704 (not shown in the figure), configured to update the target forwarding cluster to a cluster with a mutual backup relationship with it in response to the failure to forward the order query request to the target forwarding cluster.

[0114] Optionally, the determining module 702 is further configured to determine that the failover strategy is not enabled; the device further includes: the determining module 702 is further configured to respond to determining that the failover strategy is enabled; the forwarding module 703 is further configured to forward the order query request to a backup cluster of multiple clusters, so that the backup cluster performs an order query based on the order query request to obtain the target order data, and the order data stored in the backup cluster is the same as that in the multiple clusters.

[0115] The data reading apparatus according to embodiments of the present invention can realize order data query based on a multi-cluster architecture. Specifically, it determines the target forwarding cluster by combining the order query request and the request forwarding strategy, enabling it to query the target order data based on the query request, thus ensuring the reliability of data reading. Simultaneously, the multi-cluster architecture enables mutual backup between clusters and free distribution of requests, improving the system's request processing capacity and disaster recovery capabilities, and reducing system risk. Furthermore, different request forwarding strategies can be applied to different scenarios, improving system flexibility and request processing efficiency.

[0116] Figure 8 An exemplary system architecture 800 is shown, in which a data storage method, a data reading method, a data storage apparatus, or a data reading apparatus of embodiments of the present invention can be applied.

[0117] like Figure 8 As shown, system architecture 800 may include terminal devices 801, 802, and 803, a network 804, and a server 805. Network 804 serves as the medium for providing communication links between terminal devices 801, 802, and 803 and server 805. Network 804 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0118] Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 801, 802, and 803, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0119] Terminal devices 801, 802, and 803 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0120] Server 805 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 801, 802, and 803 (for example only). The backend management server can analyze and process received data such as data storage requests, and feed back the processing results (such as order data - for example only) to the terminal devices.

[0121] It should be noted that the data storage method and data reading method provided in the embodiments of the present invention are generally executed by the server 805, and correspondingly, the data storage device and the data reading device are generally located in the server 805.

[0122] It should be understood thatFigure 8 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0123] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing terminal devices or servers of the present invention. Figure 9 The terminal device or server shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0124] like Figure 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0125] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0126] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0127] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] The units or modules described in the embodiments of the present invention can be implemented in software or hardware. The described units or modules can also be housed in a processor; for example, a processor can be described as including a receiving module, an acquisition module, and a storage module. The names of these units or modules do not necessarily limit the specific unit or module itself; for example, a receiving module can also be described as "a module that receives order change messages and saves the order change messages to message queues corresponding to multiple clusters."

[0130] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:

[0131] Receive order change messages and save them to message queues corresponding to multiple clusters, including multiple primary clusters and backup clusters deployed in different locations;

[0132] Retrieve order change messages from the message queue corresponding to each cluster, and construct order data based on the order change messages;

[0133] For each cluster, the storage medium corresponding to the cluster is determined based on the message topic of the cluster, and the order data is stored in the storage medium corresponding to the cluster.

[0134] According to the technical solution of the present invention, after receiving an order change message, it is stored in the message queues of multiple primary and backup clusters deployed in different locations. The order change message is further obtained through the message queues, order data is constructed, and finally the order data is stored in the storage medium corresponding to each cluster, realizing multi-cluster storage of order data, improving the reliability of data storage, thereby ensuring the smooth progress of subsequent data reading processes and improving the system's disaster recovery capability.

[0135] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for data storage, characterized in that, include: Receive order change messages and save the order change messages to message queues corresponding to multiple clusters, including multiple primary clusters and backup clusters deployed in different locations; The order change message is retrieved from the message queue corresponding to each cluster, and order data is constructed based on the order change message; For each cluster, the storage medium corresponding to the cluster is determined according to the message topic corresponding to the cluster, and the order data is stored in the storage medium corresponding to the cluster.

2. The method according to claim 1, characterized in that, The order data includes the order number and the current order status; storing the order data in the storage medium corresponding to the cluster includes: Lock the order based on the order number and query the latest status of the order; In response to the latest status of the order being consistent with the current status of the order, the order data is stored in the storage medium corresponding to the cluster.

3. A method for reading data based on the data storage method as described in any one of claims 1 and 2, characterized in that, include: Obtain the request forwarding strategy based on the order query request; Based on the request forwarding strategy, a target forwarding cluster is determined among multiple clusters, which are different service clusters that store the same order data through order change messages. The order query request is forwarded to the target forwarding cluster, which then performs an order query based on the request to obtain the target order data.

4. The method according to claim 3, characterized in that, The request forwarding strategy includes a vertical invocation strategy, and determining the target forwarding cluster among multiple clusters based on the request forwarding strategy includes: In response to the request forwarding strategy being a vertical invocation strategy, the cluster corresponding to the source identifier of the order query request among the multiple clusters is determined as the target forwarding cluster.

5. The method according to claim 3, characterized in that, The request forwarding strategy includes a percentage-based invocation strategy, and determining the target forwarding cluster among multiple clusters based on the request forwarding strategy includes: In response to the request forwarding strategy being a percentage-based call strategy, the traffic percentage of the cluster corresponding to the source identifier of the order query request is obtained; The target forwarding cluster among the multiple clusters is determined based on the traffic share.

6. The method according to claim 3, characterized in that, The request forwarding strategy includes a mutual backup call strategy, and the step of determining the target forwarding cluster among multiple clusters based on the request forwarding strategy includes: In response to the request forwarding strategy being a mutual backup call strategy, the cluster corresponding to the source identifier of the order query request among the multiple clusters is determined as the target forwarding cluster; The method further includes: In response to the failure to forward the order query request to the target forwarding cluster, the target forwarding cluster is updated to a cluster with a mutual backup relationship with it.

7. The method according to any one of claims 3-6, characterized in that, Before determining the target forwarding cluster among multiple clusters based on the request forwarding strategy, the method further includes: The failover strategy has been determined to be disabled. The method further includes: In response to determining that the failover strategy is enabled; The order query request is forwarded to a backup cluster of the multiple clusters, so that the backup cluster performs an order query based on the order query request to obtain the target order data. The order data stored in the backup cluster is the same as that stored in the multiple clusters.

8. A data storage device, characterized in that, include: The receiving module is used to receive order change messages and save the order change messages to message queues corresponding to multiple clusters, including multiple primary clusters and backup clusters deployed in different locations. The acquisition module is used to retrieve the order change message from the message queue corresponding to each cluster, and construct order data based on the order change message; The storage module is used to determine the storage medium corresponding to each cluster based on the message topic corresponding to the cluster, and to store the order data in the storage medium corresponding to the cluster.

9. A data reading apparatus based on the data storage device as described in claim 8, characterized in that, include: The acquisition module is used to obtain the request forwarding strategy based on the order query request; The determination module is used to determine the target forwarding cluster among multiple clusters based on the request forwarding strategy. The multiple clusters are different service clusters that store the same order data through order change messages. The forwarding module is used to forward the order query request to the target forwarding cluster, so that the target forwarding cluster can perform an order query based on the order query request and obtain the target order data.

10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

Images