Data processing method and device based on active-active cluster architecture

Abstract

The invention relates to the field of computer networks, in particular to a data processing method and device based on an active-active cluster architecture. According to the method provided by the embodiment of the invention, by acquiring comprehensive state information, potential performance bottlenecks and fault points can be found in time, and optimization and processing are carried out in advance, so that the system performance is improved, and the problem that the system state is incomprehensively known in a traditional architecture is solved. And then the optimal writing target can be selected according to the actual demand and the real-time state, so that the efficiency and the performance of data storage and transmission are improved. And finally, writing the business data into the target object according to a writing strategy corresponding to the object association information, thereby ensuring high efficiency and reliability of the writing process, reducing performance reduction and data errors caused by improper strategies, and improving the overall system performance.

Description

Technical Field

[0001] The present invention relates to the field of computer networks, and in particular to a data processing method and device based on a dual-active cluster architecture. Background Art

[0002] Traditional network storage architecture relies on centralized storage servers to centrally manage data, which limits system performance and poses a risk of single point failure. It is difficult to meet the needs of modern data centers for high reliability, high availability, and efficient access. As a result, distributed network storage systems have emerged. They distribute multiple storage servers to bear the storage load, improve reliability, availability, and data access efficiency, and can be expanded on demand.

[0003] In active-active network planning, common strategies include reusing the service network and forwarding data through the service network to the replication node and then transmitting it through the replication network. The reusing service network solution is easy to implement active-active in the same cluster, but it occupies the service network bandwidth and affects the performance of non-active-active services; the separate replication network solution can ensure the independence of data replication, but it lengthens the data transmission path and may introduce additional network delays that affect performance. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a data processing method and device based on a dual-active cluster architecture to solve the deficiencies of the traditional network storage architecture in terms of performance, single point failure risk, and meeting data storage requirements.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method based on an active-active cluster architecture, the method being applied to a first cluster, where the first cluster is any cluster in the active-active cluster architecture, the method comprising:

[0006] Receiving a service request sent by a client, wherein the service request includes service data to be written and data writing requirements;

[0007] In response to the service request, a service status map of the active-active cluster architecture is obtained, wherein the service status map includes the cluster status of each cluster in the active-active cluster architecture, the network status between each cluster, and the operation status of the storage device in each cluster;

[0008] Determining a target object for writing the business data according to the data writing requirement, and acquiring object association information of the target object from the service status map;

[0009] The business data is written into the target object according to the writing strategy corresponding to the object association information and in accordance with the writing strategy.

[0010] Further, determining a target object for writing the business data according to the data writing requirement includes:

[0011] If the data writing requirement is local writing, the target storage device corresponding to the first cluster is used as the target object for writing the business data;

[0012] If the data writing requirement is remote writing, a second cluster specified by the data writing requirement is obtained from the data writing requirement, and the second cluster is used as a target object for writing the business data.

[0013] Further, if the target object is a target storage device of the first cluster, the object association information includes the operation status of the target storage device;

[0014] The step of writing the business data into the target object according to the writing strategy corresponding to the object association information includes:

[0015] Determine the current load of the target storage device according to the operation status of the target storage device;

[0016] If the current load is higher than a first threshold value and lower than a second threshold value, a waiting time period is obtained, a write operation is suspended during the waiting time period, and when the waiting time period expires, the service data is written into the target storage device;

[0017] Or, if the current load is higher than a second threshold, a target write timing is acquired, and when the target write timing is reached, the service data is written into the target storage device.

[0018] Further, if the target object is a second cluster, the object association information includes the target node status of each node in the second cluster, the target network status of the replication network between the first cluster and the second cluster;

[0019] The step of writing the business data into the target object according to the writing strategy corresponding to the object association information includes:

[0020] If the target object is the second cluster, obtaining an available node from the second cluster according to the target node state;

[0021] Obtaining node information corresponding to the available node, wherein the node information includes node load and node operation data;

[0022] Calculating the performance parameters of the available nodes using the node load and the node operation data;

[0023] Taking the available node with the highest performance parameter as the target available node;

[0024] Acquire a path performance indicator of each transmission path between the first cluster and the second cluster based on the target network condition;

[0025] Taking the transmission path with the highest path performance index as the target transmission path;

[0026] The service data is written into a target available node in the second cluster through the target transmission path.

[0027] Further, the obtaining of the service status map of the active-active cluster architecture includes:

[0028] Determine a pre-built active-active cluster architecture, wherein the active-active cluster architecture includes multiple clusters, multiple clusters are connected by a replication network, and each cluster includes a storage device;

[0029] Monitor the network status of the replication network between clusters;

[0030] Obtain the data replication and status of each cluster and the node status, and obtain the corresponding operation status of the storage devices in each cluster;

[0031] Building the cluster situation based on the data replication situation and node status of each cluster;

[0032] The service status map is constructed using the cluster status, the network status and the corresponding operation status of the storage device.

[0033] Furthermore, the method further comprises:

[0034] Monitoring the heartbeat signal of the replication network;

[0035] If the replication network is determined to be abnormal according to the heartbeat signal, a fault handling mechanism is triggered, and fault information of the replication network is obtained;

[0036] The fault information is updated to the service status map, and a fault notification is sent to the client corresponding to each cluster.

[0037] Furthermore, the method further comprises:

[0038] Acquire a path state parameter of a return path according to a preset period, wherein the return path is a network path through which service data is transmitted from the first cluster to other clusters in the active-active cluster architecture and then transmitted back to the first cluster;

[0039] Get pre-configured path health thresholds;

[0040] Comparing the path status parameter with the path health threshold to obtain the health status of the return path;

[0041] If the health status is unhealthy, the processing of the service request is suspended; or, if the health status is healthy, the processing of the service request continues.

[0042] In a second aspect, an embodiment of the present invention provides a data processing device based on a dual-active cluster architecture, the device comprising:

[0043] A receiving module, used to receive a service request sent by a client, wherein the service request includes the service data to be written and the data writing requirement;

[0044] A response module, used to respond to the service request and obtain a service status map of the active-active cluster architecture, wherein the service status map includes the cluster status of each cluster in the active-active cluster architecture, the network status between each cluster, and the operation status of the storage device in each cluster;

[0045] A processing module, used to determine a target object for writing the business data according to the data writing requirement, and obtain object association information of the target object from the service status map;

[0046] An execution module is used to write the business data into the target object according to the write strategy corresponding to the object association information and the write strategy.

[0047] In a third aspect, an embodiment of the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0048] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method of the first aspect or any corresponding embodiment thereof.

[0049] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the method for individually powering on and off a target component of a server according to the first aspect or any corresponding embodiment thereof.

[0050] The data processing method based on the active-active cluster architecture provided by the present invention has the following advantages:

[0051] The method provided in the embodiment of the present application can timely discover potential performance bottlenecks and fault points by obtaining comprehensive status information, and optimize and process them in advance, thereby improving system performance and overcoming the problem of incomplete understanding of system status in traditional architecture. Then, the optimal write target can be selected according to actual needs and real-time status to improve the efficiency and performance of data storage and transmission. Finally, according to the write strategy corresponding to the object association information, the business data is written to the target object in accordance with the write strategy to ensure the efficiency and reliability of the write process, reduce performance degradation and data errors caused by improper strategies, and improve the overall system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flow chart of a data processing method based on an active-active cluster architecture according to some embodiments of the present invention;

[0054] Figure 2 is a flow chart of another data processing method based on a dual-active cluster architecture according to some embodiments of the present invention;

[0055] Figure 3 is a structural block diagram of a data processing device based on a dual-active cluster architecture according to an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0058] According to an embodiment of the present invention, a data processing method and device based on a dual-active cluster architecture are provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0059] In this embodiment, a data processing method based on a dual-active cluster architecture is provided. Figure 1 is a flow chart of a data processing method based on a dual-active cluster architecture according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0060] Step S101: receiving a service request sent by a client, wherein the service request includes service data to be written and data writing requirements.

[0061] The method of the embodiment of the present application is applied to the first cluster, and the first cluster is any cluster located in the active-active cluster architecture. The active-active cluster architecture is a high-availability architecture mode, in which two or more clusters are active at the same time and can process business requests to achieve load balancing and disaster recovery of the business. The active-active cluster architecture can reasonably distribute business requests to different clusters according to the load conditions of each cluster, thereby improving the overall processing capacity and efficiency of the system. When a data center where a cluster is located encounters a disaster, hardware failure or other unforeseen problems, another cluster can seamlessly take over the business to ensure that the business is not affected. In addition, data can be synchronized in real time between clusters to ensure data consistency and integrity.

[0062] In an embodiment of the present application, a "business request" is an operation instruction initiated by a client to the first cluster in a dual-active cluster, which includes the business data to be written and the data writing requirements, wherein the "business data to be written" refers to the actual information that the client expects to store or process. For example, the content of a newly created document in an online document editing system, the transaction records of a user in an e-commerce platform, or the account operation data of a user in a financial system. The data writing requirement describes the client's specific requirements and expectations for data writing. The data writing requirement may include a specified data storage location (local or remote), data priority, writing timeliness requirements, data format requirements, etc.

[0063] Step S102, in response to the service request, obtain a service status map of the active-active cluster architecture, wherein the service status map includes the cluster status of each cluster in the active-active cluster architecture, the network status between each cluster, and the operation status of the storage device in each cluster.

[0064] In an embodiment of the present application, the service status map is a comprehensive and real-time status description of the entire active-active cluster architecture. Among them, the cluster status of each cluster covers the number of nodes in each cluster, the operating status of the nodes (whether it is running normally, whether it is busy or idle), the overall load level of the cluster (such as CPU, memory usage) and other information. The network conditions between each cluster include the stability of the network connection (whether there is packet loss or delay), the usage of the network bandwidth (whether it is close to saturation), the smoothness of the network path, etc. The operating conditions of the storage devices in each cluster include the size of the available space of the storage device, the read and write speed, whether there are faults or errors, etc.

[0065] Step S103: determining a target object for writing business data according to the data writing requirement, and acquiring object association information of the target object from the service status map.

[0066] In an embodiment of the present application, a target object for writing business data is determined based on a data writing requirement, including the following two situations: if the data writing requirement is for local writing, the target storage device corresponding to the first cluster is used as the target object for writing business data; or, if the data writing requirement is for remote writing, the second cluster specified by the data writing requirement is obtained from the data writing requirement, and the second cluster is used as the target object for writing business data.

[0067] It should be noted that in the active-active cluster architecture, when the first cluster receives a service request including a data writing requirement, it will perform analysis to determine a target object for writing the service data.

[0068] If the data writing requirement is clearly for local writing, this means that the client wants to store the business data within the first cluster itself. At this time, the first cluster will select its corresponding local target storage device as the target object for writing business data. For example, in an active-active cluster within an enterprise, an employee submits business data from a client in a local office and specifies it as local writing. The first cluster will identify its internal specific storage device (such as a local disk array) as the target and write the data directly into it. If the data writing requirement is for remote writing, this means that the client requires the business data to be stored in a cluster other than the first cluster. At this time, the first cluster will obtain the information of the specified second cluster from the data writing requirement. This specified second cluster is determined as the target object for writing business data.

[0069] In an embodiment of the present application, if the target object is a target storage device of the first cluster, the object-associated information includes the operating status of the target storage device; or, if the target object is a second cluster, the object-associated information includes the target node status of each node in the second cluster, and the target network status of the replication network between the first cluster and the second cluster.

[0070] It should be noted that when the target object is the target storage device of the first cluster, the object-related information includes the detailed operating status of the target storage device obtained from the service status map. The available space size of the storage device, the read and write speed, and whether there is a fault or error. When the target object is the second cluster, the object-related information includes: the target node status of each node in the second cluster obtained from the service status map. The target node status covers the operating status of the node (whether it is working normally, the load level), resource usage (such as CPU, memory usage), etc. It also includes: the target network status of the replication network between the first cluster and the second cluster obtained from the service status map, such as the network bandwidth, delay, packet loss rate, etc. This information is crucial to ensure that data can be efficiently and accurately transmitted from the first cluster to the second cluster.

[0071] Step S104: write the business data into the target object according to the write policy corresponding to the object association information and in accordance with the write policy.

[0072] In the embodiment of the present application, according to the writing strategy corresponding to the object association information, and in accordance with the writing strategy, writing the business data into the target object includes the following steps A1-A3:

[0073] Step A1, determining the current load of the target storage device according to the operation status of the target storage device.

[0074] Specifically, extract relevant data about the operation status of the target storage device from the service status map, including but not limited to the number of disk I / O operations, the amount of data read and written, the number of ongoing read and write tasks, etc. Obtain the CPU usage and memory usage of the target storage device. Count the number of disk read and write requests within a period of time (such as 1 minute). Calculate the average read and write speed of the disk. Based on the above-obtained data, comprehensively evaluate the current load of the target storage device.

[0075] For example: Number of disk I / O operations: 500 per minute. Amount of data read and written: 200MB per minute. Number of ongoing read and write tasks: 10. CPU utilization: 40%. Memory utilization: 35%. Number of disk read and write requests in 1 minute: 600. Average disk read and write speed: 80MB / s.

[0076] First, set weights for each metric. For example: Weight of disk I / O operations: 20%. Weight of data read and written: 20%. Weight of number of ongoing read and write tasks: 15%. Weight of CPU usage: 20%. Weight of memory usage: 15%. Weight of number of disk read and write requests: 5%. Weight of average disk read and write speed: 5%.

[0077] Then, calculate the score of each indicator within its range. Take the set load level as an example:

[0078] Disk I / O operation times: 0-300 times per minute means low load, 0 points, 301-600 times per minute means medium load, 50 points, and more than 600 times per minute means high load, 100 points. Currently, 500 times per minute means 50 points.

[0079] Data volume read and written: 0-100MB per minute is low load and gets 0 points, 101-200MB is medium load and gets 50 points, and more than 200MB is high load and gets 100 points. Currently, 200MB per minute means 50 points.

[0080] The number of ongoing read and write tasks: 0-5 is low load and gets 0 points, 6-15 is medium load and gets 50 points, and more than 15 is high load and gets 100 points. Currently there are 10, so the score is 50 points.

[0081] CPU usage: 0-30% is low load and gets 0 points, 31%-70% is medium load and gets 50 points, and more than 70% is high load and gets 100 points. If the current load is 40%, the score is 50 points. Memory usage: Similarly, if the current load is 35%, the score is 50 points.

[0082] The number of disk read and write requests: 0-400 times is low load and gets 0 points, 401-700 times is medium load and gets 50 points, and more than 700 times is high load and gets 100 points. The current number is 600 times, so the score is 50 points.

[0083] Average read and write speed of the disk: 0-50MB / s means low load, 0 points; 51-100MB / s means medium load, 50 points; and over 100MB / s means high load, 100 points. The current speed is 80MB / s, so the score is 50 points.

[0084] Finally, multiply the score of each indicator by its corresponding weight, and then add them together to get the comprehensive score: Comprehensive score = (number of disk I / O operations score × 20%) + (amount of data read and written score × 20%) + (number of ongoing read and write tasks score × 15%) + (CPU utilization score × 20%) + (memory utilization score × 15%) + (number of disk read and write requests score × 5%) + (average disk read and write speed score × 5%). That is, comprehensive score = (50×20%) + (50×20%) + (50×15%) + (50×20%) + (50×15%) + (50×5%) + (50×5%) = 50 points.

[0085] Since the comprehensive score is 50 points, which is between 30% and 70%, the current load is determined to be a medium load.

[0086] Step A2: if the current load is higher than the first threshold and lower than the second threshold, a waiting time period is obtained, the write operation is suspended during the waiting time period, and the business data is written to the target storage device when the waiting time period expires.

[0087] Specifically, the first threshold and the second threshold are pre-set standard values ​​for measuring the degree of load. When it is determined that the current load is higher than the first threshold, it means that the target storage device is already in a relatively high load state, but has not reached an extremely serious level (i.e., lower than the second threshold). In this case, a pre-set waiting time period will be obtained. The length of this waiting time period is determined in advance based on factors such as performance, business requirements, and experience. During the waiting time period, the write operation of business data will be suspended. This is to avoid continuing to write data when the target storage device is highly loaded, which may result in slower write speeds, data loss, or other issues that may affect stability and data integrity. When the waiting time period expires, the load of the target storage device is re-evaluated. If the load has been reduced to an appropriate level at this time, the business data that has not been written before will be written to the target storage device.

[0088] For example, in an active-active cluster of an online video platform, the first threshold is set to 70%, the second threshold is set to 90%, and the waiting time period is set to 5 minutes. When it is detected that the load of the target storage device reaches 80%, the writing of new video data will be suspended and a 5-minute wait will begin. After 5 minutes, if the load drops below 70%, the video data that has not been written before will be written.

[0089] Or, in step A3, if the current load is higher than the second threshold, a target write timing is obtained, and when the target write timing is reached, the business data is written into the target storage device.

[0090] Specifically, when the current load is higher than the second threshold, it indicates that the target storage device is in a very high load state. In this case, the data write operation will not be performed immediately, but the target write timing will be obtained. The target write timing is a suitable time point determined according to a series of conditions and policies for writing business data to the target storage device. The following strategies can be used to determine the target write timing, including:

[0091] Strategy ①: Determine the write timing based on the load change trend of the target storage device. If the load continues to rise, you may need to wait for the load to decrease. If the load has stabilized at a high level, you may need to determine it based on the availability of other related devices or resources. Strategy ②: Determine the write timing based on the overall business priority. Certain urgent or critical business data may be given a higher write priority. Even if the target storage device is heavily loaded, it will be written as soon as possible at the appropriate time. ③ Determine the write timing based on the status of other related resources, such as the idleness of network bandwidth, the availability of backup storage devices, etc. When the determined target write timing is reached, it means that the conditions at this time (such as the load is reduced to an acceptable range, the relevant resources are ready, etc.) are suitable for writing business data to the target storage device, thereby performing the write operation.

[0092] The method provided in the embodiment of the present application can flexibly arrange write operations according to the load conditions of the target storage device, avoid performance degradation caused by excessive writing under high load, and realize the reasonable allocation and optimal utilization of storage resources. It helps to maintain the load balance of the entire storage system and improve the overall performance and stability. When the load is between the first threshold and the second threshold, by waiting for the appropriate time period before writing, conflicts with other ongoing operations can be avoided, thereby improving the efficiency of actual writing. When the load is higher than the second threshold, the target write timing is accurately obtained to ensure that the data is written at a time when the system is relatively idle or more suitable for writing, further improving the write efficiency. Avoiding writing data when the storage device is overloaded reduces the risk of data errors, losses or inconsistencies that may be caused by a busy system, and ensures the integrity and consistency of the data.

[0093] In the embodiment of the present application, according to the writing strategy corresponding to the object association information, and in accordance with the writing strategy, writing the business data into the target object includes the following steps B1-B7:

[0094] Step B1: if the target object is the second cluster, obtain available nodes from the second cluster according to the target node status.

[0095] Specifically, the target node status includes detailed operation and availability information about each node in the second cluster, such as the load level of the node, whether it is running normally, whether it is idle or busy, etc. First, the target node status of each node in the second cluster will be fully scanned. Then, according to the preset rules and conditions, it is determined which nodes are available. These rules can be based on the load of the node (selecting nodes with lower load), the health status of the node (excluding nodes with faults or abnormalities), the resource usage of the node (such as remaining storage space, available memory, etc.). For example, if the preset rule is to select nodes with a load of less than 50% and a remaining storage space of more than 100GB as available nodes, then the nodes that meet these conditions will be screened out from all the nodes in the second cluster.

[0096] Assume that there are nodes A, B, and C in the second cluster, and the load of node A is 30%, and the remaining storage space is 150GB; the load of node B is 70%, and the remaining storage space is 80GB; the load of node C is 40%, and the remaining storage space is 120GB. According to the above rules, nodes A and C will be determined as available nodes.

[0097] Step B2, obtaining node information corresponding to the available nodes, wherein the node information includes node load and node operation data.

[0098] Specifically, node load refers to the degree of work pressure that the node is currently under. It includes indicators such as CPU usage (for example, the percentage of CPU occupied), memory usage (the proportion of used memory to total memory), and disk I / O busyness (the number of reads and writes per second or the amount of data read and written). By understanding the node load, you can determine whether the node has enough resources to handle new write tasks, and estimate the impact that write operations may have on its performance. Node operation data covers a wider range of information related to the node operation status. Including but not limited to the following: network bandwidth usage, process operation status, error and warning information, recent performance data, etc.

[0099] Step B3, calculating the performance parameters of the available nodes using the node load and node operation data.

[0100] Specifically, in this step, the performance parameters of each available node are calculated using the various indicators in the node load and node operation data. First, weights need to be set for each indicator to reflect its importance in evaluating node performance. For example, the weight of CPU usage may be 30%, the weight of memory usage may be 25%, the weight of network bandwidth usage may be 20%, the weight of process operation may be 15%, the weight of error and warning information may be 5%, and the weight of recent performance data may be 5%. Then, calculations are performed based on the actual values ​​of each indicator and the set weights. For example, if the CPU usage of a node is 40% (the full score is 100%, the lower the value, the better), the score calculated according to the weight of 30% is 0.3×(100-40)=18 points. Similar calculations are performed for each indicator in the node load and node operation data, and all scores are added together to obtain the total performance parameter score of the available node.

[0101] Step B4: Use the available node with the highest performance parameter as the target available node.

[0102] Specifically, after calculating the performance parameters of all available nodes, the scores are compared.

[0103] The available node with the highest score is identified as the node with the best performance, and is selected as the target available node. For example, there are three available nodes A, B, and C, and their calculated performance parameters are 75 points, 80 points, and 60 points, respectively. Then node B, which has the highest performance parameter score of 80 points, will be determined as the target available node for subsequent data write operations.

[0104] Step B5: obtaining path performance indicators of each transmission path between the first cluster and the second cluster based on the target network situation.

[0105] Specifically, when it is determined that business data is to be written from the first cluster to the second cluster, it is first necessary to understand the target network conditions of the replication network between the two. This includes information such as bandwidth, delay, packet loss rate, jitter, etc. of each transmission path. Based on these network condition data, the corresponding path performance indicators are calculated for each transmission path. For example, a higher weight can be given to bandwidth because higher bandwidth generally means faster data transmission speed. Factors such as delay, packet loss rate, and jitter are also given certain weights according to their impact on the quality of data transmission. By comprehensively considering these factors and weights, the performance indicator score of each transmission path is calculated.

[0106] Step B6: taking the transmission path with the highest path performance index as the target transmission path.

[0107] Specifically, after calculating the performance indicators of each transmission path, these scores are compared. The transmission path with the highest score is selected as the target transmission path. This means that the path can provide the fastest, most stable and most reliable data transmission service under the current network conditions. For example, suppose there are three transmission paths A, B, and C, and their path performance indicator scores are 85 points, 90 points, and 70 points. Then transmission path B is selected as the target transmission path because it has the highest score of 90 points.

[0108] Step B7: writing the service data to the target available node in the second cluster through the target transmission path.

[0109] Specifically, once the target transmission path is determined, the service data is sent from the first cluster to the "target available node" in the second cluster using this path. During the data transmission process, the network resources and performance advantages provided by the target transmission path are utilized to ensure that the data can be efficiently, accurately and completely transmitted to the designated node.

[0110] The method provided in the embodiment of the present application can select the available nodes and transmission paths with the best performance, maximize the speed and efficiency of data writing, and reduce the time cost of data transmission. Selecting reliable nodes and paths reduces the risk of errors, loss or inconsistency in data transmission and writing. Avoid writing data to nodes with excessive load or potential problems, and enhance the stability of the entire active-active cluster architecture.

[0111] The method provided in the embodiment of the present application can timely discover potential performance bottlenecks and fault points by obtaining comprehensive status information, and optimize and process them in advance, thereby improving system performance and overcoming the problem of incomplete understanding of system status in traditional architecture. Then, the optimal write target can be selected according to actual needs and real-time status to improve the efficiency and performance of data storage and transmission. Finally, according to the write strategy corresponding to the object association information, the business data is written to the target object in accordance with the write strategy to ensure the efficiency and reliability of the write process, reduce performance degradation and data errors caused by improper strategies, and improve the overall system performance.

[0112] In the embodiment of the present application, a service status map of the active-active cluster architecture is obtained, such as Figure 2 As shown, including:

[0113] Step S201 , determining a pre-built active-active cluster architecture, wherein the active-active cluster architecture includes a plurality of clusters, the plurality of clusters are connected by a replication network, and each cluster includes a storage device.

[0114] In the embodiment of the present application, the active-active cluster architecture is a highly available, high-performance architecture design designed to ensure business continuity and data security. In this architecture, multiple clusters are included. Each cluster is a relatively independent but mutually cooperative system unit. These clusters are interconnected through a replication network. The replication network is responsible for transmitting data and synchronizing status information between clusters to ensure data consistency and business collaboration of each cluster. Each cluster is equipped with storage devices. These storage devices are used to store important data such as business data and system configuration information.

[0115] Step S202: monitor the network status of the replication network between each cluster.

[0116] In the embodiment of the present application, monitoring the network status of the replication network between each cluster means continuously and actively monitoring and collecting various performance and status information related to the replication network, including but not limited to the following aspects: network bandwidth, network delay, packet loss rate, traffic distribution, etc.

[0117] Step S203, obtaining data replication and status and node status of each cluster, and obtaining the corresponding operation status of the storage devices in each cluster.

[0118] In the embodiments of the present application, the required information is collected through various monitoring means and interfaces. For data replication, detailed information such as the progress of replication, the amount of data replicated, the frequency of replication, whether there are replication errors or interruptions, etc. will be obtained. This helps to understand the synchronization status of data between various clusters. The node status includes whether the node is running normally, the load level of the node (such as CPU utilization, memory utilization, etc.), whether the node is in failure or maintenance mode, etc. The corresponding operation status of the storage device covers information such as the available space of the storage device, the read and write speed, and whether there are disk errors.

[0119] Step S204: constructing cluster status based on the data replication status and node status of each cluster.

[0120] In the embodiment of the present application, after obtaining the above information, the data replication situation and node status are comprehensively analyzed and sorted. For example, if the load of multiple nodes is very high, or data replication is frequently interrupted or erroneous, it can be judged that the cluster may be in a relatively tense or unstable state. By integrating and evaluating this information, a description of the overall status of each cluster is formed, including characteristics of the cluster's health, performance level, stability, etc.

[0121] Step S205, constructing a service status map using the cluster status, network status and the corresponding operation status of the storage device.

[0122] In the embodiment of the present application, the cluster status is integrated with the acquired network status (such as bandwidth, latency, packet loss rate, etc.) and the operation status of the storage device (available space, read and write speed, etc.). This information is organized in an intuitive, comprehensive, easy-to-understand and query way to form a service status map. This map is a visual interface, or a structured data set, which can clearly show the real-time status and relationship of each part in the entire active-active cluster architecture.

[0123] In an embodiment of the present application, the method also includes: monitoring the heartbeat signal of the replication network; if the replication network is determined to be abnormal based on the heartbeat signal, triggering a fault handling mechanism and obtaining fault information of the replication network; updating the fault information to the service status map, and sending a fault notification to the client corresponding to each cluster.

[0124] Specifically, the heartbeat signal is a signal that is sent periodically to indicate that the network or component is operating normally. By continuously monitoring this signal, the activity and normal status of the replication network can be monitored in real time. If the received heartbeat signal does not conform to the expected pattern, such as signal interruption, long delay, signal content error, etc., it can be determined that an abnormality has occurred in the replication network.

[0125] Once a network anomaly is identified, a pre-set fault handling process will be automatically initiated. This may include a series of diagnostic tests on the network to obtain more detailed fault information, such as the specific fault location (a certain network node, link, or device), fault type (hardware failure, software error, or network congestion), and impact range. The obtained detailed information about the replication network failure is added to the service status map, so that the service status map can reflect the latest fault situation in real time, so that relevant personnel can clearly understand the severity and specific situation of the problem.

[0126] In addition, in order to let users or administrators of each cluster know the failure of the replication network in time, notifications will be sent to the clients associated with each cluster. The notifications can be in the form of pop-up prompts, emails, text messages, etc. The notification content usually includes a basic description of the failure, the expected scope of impact, and the possible resolution time, so that users can prepare countermeasures or arrangements in advance.

[0127] In an embodiment of the present application, the method also includes: obtaining path status parameters of the return path according to a preset period, wherein the return path is the network path through which the business data is transmitted from the first cluster to other clusters in the active-active cluster architecture and then returned to the first cluster; obtaining a pre-configured path health threshold; comparing the path status parameters with the path health threshold to obtain the health status of the return path; if the health status is unhealthy, suspending processing of business requests; or, if the health status is healthy, continuing to process business requests.

[0128] Specifically, the return path is the network path that the business data passes through after being transmitted from the first cluster to other clusters in the active-active cluster architecture and then back to the first cluster, that is, the network channel used when the first cluster sends business data to other clusters and the other clusters need to return the results or related information to the first cluster after processing the data.

[0129] At pre-set time intervals (e.g., every 5 minutes, every 10 minutes, etc.), some key performance and status data about the return path are collected. These parameters may include network delay, packet loss rate, bandwidth utilization, error rate, etc., which can reflect the operating status of the return path.

[0130] Then obtain a series of standard values ​​pre-defined by the user to indicate that the return path is in a healthy state. These standard values ​​are the path health thresholds. For example, the network delay threshold is set to 50 milliseconds, the packet loss rate threshold is set to 2%, and the bandwidth utilization threshold is set to 80%. Compare the collected path status parameters with the pre-set health thresholds. If all parameters are within the threshold range, the return path is judged to be healthy; if any one or more parameters exceed the threshold range, it is judged to be unhealthy.

[0131] When the return path is judged to be unhealthy, in order to avoid possible data errors or incompleteness, the system will temporarily stop processing new business requests to prevent the problem from further deteriorating. Alternatively, if the return path is in good health, the system will continue to process business requests according to the normal process to ensure the smooth progress of the business.

[0132] In this embodiment, a data processing device based on a dual-active cluster architecture is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0133] This embodiment provides a data processing device based on a dual-active cluster architecture. Figure 3 As shown, including:

[0134] The receiving module 301 is used to receive a service request sent by a client, wherein the service request includes service data to be written and data writing requirements;

[0135] A response module 302 is used to respond to the service request and obtain a service status map of the active-active cluster architecture, wherein the service status map includes the cluster status of each cluster in the active-active cluster architecture, the network status between each cluster, and the operation status of the storage device in each cluster;

[0136] Processing module 303, used to determine the target object for writing business data according to the data writing requirement, and obtain object association information of the target object from the service status map;

[0137] The execution module 304 is used to write the business data into the target object according to the write policy corresponding to the object association information and the write policy.

[0138] In an embodiment of the present application, the processing module 303 is used to, if the data writing requirement is local writing, use the target storage device corresponding to the first cluster as the target object for writing business data; if the data writing requirement is remote writing, obtain the second cluster specified by the data writing requirement from the data writing requirement, and use the second cluster as the target object for writing business data.

[0139] In the embodiment of the present application, if the target object is a target storage device of the first cluster, the object association information includes the operation status of the target storage device;

[0140] The processing module 303 is used to determine the current load of the target storage device according to the operation status of the target storage device; if the current load is higher than the first threshold and lower than the second threshold, obtain the waiting time period, suspend the write operation during the waiting time period, and write the business data to the target storage device when the waiting time period expires; or, if the current load is higher than the second threshold, obtain the target write timing, and write the business data to the target storage device when the target write timing is reached.

[0141] In the embodiment of the present application, if the target object is the second cluster, the object association information includes the target node status of each node in the second cluster, the target network status of the replication network between the first cluster and the second cluster;

[0142] Processing module 303 is used to obtain available nodes from the second cluster according to the target node status if the target object is the second cluster; obtain node information corresponding to the available nodes, wherein the node information includes node load and node operation data; calculate the performance parameters of the available nodes using the node load and the node operation data; use the available node with the highest performance parameter as the target available node; obtain the path performance index of each transmission path between the first cluster and the second cluster based on the target network situation; use the transmission path with the highest path performance index as the target transmission path; and write the business data to the target available node in the second cluster through the target transmission path.

[0143] In an embodiment of the present application, the response module 302 is used to determine a pre-built active-active cluster architecture, wherein the active-active cluster architecture includes multiple clusters, multiple clusters are connected by a replication network, and each cluster includes a storage device; monitors the network status of the replication network between each cluster; obtains the data replication and status of each cluster and the node status, and obtains the corresponding operating status of the storage device in each cluster; constructs the cluster status based on the data replication status and node status of each cluster; and constructs a service status map using the cluster status, network status and the corresponding operating status of the storage device.

[0144] In an embodiment of the present application, the device also includes: an anomaly detection module, which is used to monitor the heartbeat signal of the replication network; if the replication network is determined to be abnormal based on the heartbeat signal, a fault handling mechanism is triggered, and fault information of the replication network is obtained; the fault information is updated to the service status map, and a fault notification is sent to the client corresponding to each cluster.

[0145] In an embodiment of the present application, the device also includes: a monitoring module, which is used to obtain path status parameters of the return path according to a preset period, wherein the return path is the network path through which the business data is transmitted from the first cluster to other clusters in the active-active cluster architecture and then returned to the first cluster; obtain a pre-configured path health threshold; compare the path status parameters with the path health threshold to obtain the health status of the return path; if the health status is unhealthy, suspend processing of business requests; or, if the health status is healthy, continue processing business requests.

[0146] See also Figure 4 , Figure 4 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 4 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).

[0147] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0148] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0149] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0151] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0152] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0153] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data processing method based on active-active cluster architecture, characterized in that: The method is applied to a first cluster, where the first cluster is any cluster in an active-active cluster architecture, and the method includes: Receiving a service request sent by a client, wherein the service request includes service data to be written and data writing requirements; In response to the service request, a service status map of the active-active cluster architecture is obtained, wherein the service status map includes the cluster status of each cluster in the active-active cluster architecture, the network status between each cluster, and the operation status of the storage device in each cluster; Determining a target object for writing the business data according to the data writing requirement, and acquiring object association information of the target object from the service status map; The business data is written into the target object according to the write strategy corresponding to the object association information and in accordance with the write strategy.

2. The method according to claim 1, characterized in that The determining, according to the data writing requirement, a target object for writing the business data comprises: If the data writing requirement is local writing, the target storage device corresponding to the first cluster is used as the target object for writing the business data; If the data writing requirement is remote writing, a second cluster specified by the data writing requirement is obtained from the data writing requirement, and the second cluster is used as a target object for writing the business data.

3. The method according to claim 2, characterized in that If the target object is a target storage device of the first cluster, the object association information includes the operation status of the target storage device; The step of writing the business data into the target object according to the writing strategy corresponding to the object association information includes: Determine the current load of the target storage device according to the operation status of the target storage device; If the current load is higher than a first threshold value and lower than a second threshold value, a waiting time period is obtained, a write operation is suspended during the waiting time period, and when the waiting time period expires, the service data is written into the target storage device; Or, if the current load is higher than a second threshold, a target write timing is acquired, and when the target write timing is reached, the service data is written into the target storage device.

4. The method according to claim 2, characterized in that: If the target object is the second cluster, the object association information includes the target node status of each node in the second cluster, the target network status of the replication network between the first cluster and the second cluster; The step of writing the business data into the target object according to the writing strategy corresponding to the object association information includes: If the target object is the second cluster, obtaining an available node from the second cluster according to the target node state; Obtaining node information corresponding to the available node, wherein the node information includes node load and node operation data; Calculating the performance parameters of the available nodes using the node load and the node operation data; Taking the available node with the highest performance parameter as the target available node; Acquire a path performance indicator of each transmission path between the first cluster and the second cluster based on the target network condition; Taking the transmission path with the highest path performance index as the target transmission path; The service data is written into a target available node in the second cluster through the target transmission path.

5. The method according to claim 1, characterized in that: The obtaining of the service status map of the active-active cluster architecture includes: Determine a pre-built active-active cluster architecture, wherein the active-active cluster architecture includes multiple clusters, multiple clusters are connected by a replication network, and each cluster includes a storage device; Monitor the network status of the replication network between clusters; Obtain the data replication and status of each cluster and the node status, and obtain the corresponding operation status of the storage devices in each cluster; Building the cluster situation based on the data replication situation and node status of each cluster; The service status map is constructed using the cluster status, the network status and the corresponding operation status of the storage device.

6. The method according to claim 5, characterized in that The method further comprises: Monitoring the heartbeat signal of the replication network; If the replication network is determined to be abnormal according to the heartbeat signal, a fault handling mechanism is triggered, and fault information of the replication network is obtained; The fault information is updated to the service status map, and a fault notification is sent to the client corresponding to each cluster.

7. The method according to claim 1, characterized in that The method further comprises: Acquire a path state parameter of a return path according to a preset period, wherein the return path is a network path through which service data is transmitted from the first cluster to other clusters in the active-active cluster architecture and then transmitted back to the first cluster; Get pre-configured path health thresholds; Comparing the path status parameter with the path health threshold to obtain the health status of the return path; If the health status is unhealthy, the processing of the service request is suspended; or, if the health status is healthy, the processing of the service request continues.

8. A data processing device based on a dual-active cluster architecture, characterized in that: The device comprises: A receiving module, used to receive a service request sent by a client, wherein the service request includes the service data to be written and the data writing requirement; A response module, used to respond to the service request and obtain a service status map of the active-active cluster architecture, wherein the service status map includes the cluster status of each cluster in the active-active cluster architecture, the network status between each cluster, and the operation status of the storage device in each cluster; A processing module, used to determine a target object for writing the business data according to the data writing requirement, and obtain object association information of the target object from the service status map; An execution module is used to write the business data into the target object according to the write strategy corresponding to the object association information and in accordance with the write strategy.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images