A distributed service management method and system

Abstract

The application discloses a kind of distributed service management methods, method is applied to the distributed service management system of multiple clusters, any cluster includes one master node and multiple read-only nodes, and the synchronization of service data is realized by creating link between the read-only node of different cluster and target end;Method includes: obtaining target user information, in the service involved in the target user of management system query;Based on the service obtained by query, the data involved in each service to the target user is combed out to form a synchronization link template, the synchronization link template includes the cluster where the target synchronization data is located, the form involved in the service and the number of links to be created, based on the synchronization link template, the link between the target end and at least one read-only node of the corresponding cluster is created, to extract the corresponding business data from different clusters and synchronize to target end.The application can solve the problem of DTS link creation cumbersome and inefficient, greatly improve the efficiency and quality of link creation.

Description

Technical Field

[0001] This application relates to the field of distributed data technology, and in particular to a distributed business management method and system. Background Technology

[0002] The existing technology for extracting and synchronizing data to the target in real time is mainly DTS (Data Transmission Service). DTS is a real-time data streaming service provided by Alibaba Cloud. Its underlying infrastructure adopts Alibaba's Double 11 multi-site active-active architecture, which can synchronize ongoing data changes between two data sources.

[0003] In the current system architecture, multiple business systems share multiple clusters, serving hundreds of companies in the gas and water industry. These companies use different combinations of business systems according to their own needs, generating a large amount of data. Using DTS technology to extract data from these multiple PolarDB MySQL clusters for numerous customers presents several technical and business challenges. For example, creating DTS links is cumbersome and inefficient: in the current system, the process of creating DTS links is tedious. Users need to select and configure the synchronization conditions for each table individually on a web interface. This is extremely inefficient and error-prone for customers dealing with a large number of tables and complex data models. Especially in large database clusters, where the number of tables often reaches thousands or even tens of thousands, the time cost of link creation is extremely high, and the risk of configuration errors increases. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application adopts the following technical solution:

[0005] In a first aspect, this application provides a distributed business management method, which is applied to a multi-cluster distributed business management system. Each cluster includes a master node and multiple read-only nodes. Business data synchronization is achieved by creating links between the read-only nodes in different clusters and the target end.

[0006] The method includes:

[0007] Obtain target user information, and based on the target user information, query the business related to the target user in the distributed business management system;

[0008] Based on the obtained business, data related to the target user of each business is sorted out to form a synchronization link template, the synchronization link template including a cluster where target synchronization data is located, a form related to the business and a required number of links to be created, and based on the synchronization link template, links between the target end and at least one read-only node of the corresponding cluster are automatically created to extract corresponding business data from different clusters and synchronize the business data to the target end.

[0009] In summary, the present application provides a distributed business management method, which can automatically sort out business data requested by a target end to form a synchronization link template, automatically create links between a read-only node of a corresponding cluster and the target end based on the synchronization link template, and realize synchronization of business data. Through the method provided by the present application, users do not need to select and configure the synchronization conditions of each table one by one on a Web interface, and through the automatic link creation process, the tedious manual configuration process is reduced. Especially for large-scale database clusters, a large amount of time and effort can be saved, and configuration errors caused by manual operation are avoided, ensuring that each synchronization link is correctly created according to the business model, thereby improving the accuracy and stability of the system. The method solves the problems of tediousness and inefficiency of DTS link creation, and greatly improves the efficiency and quality of link creation.

[0010] Further, when the target synchronization data of the target end involves data of at least two different businesses in the same cluster, the data of the at least two different businesses in the same cluster are combined, and link creation is performed based on the combined business data, including: combining dictionary tables in the data of the at least two different businesses into one link, and combining business tables in the data of the at least two different businesses and dividing the links according to the number of forms of the business tables in a load-balanced manner.

[0011] Further, the method further comprises:

[0012] The method further comprises:

[0013] Further, when the proportion of the used links exceeds 90% and the proportion of the created links exceeds 95%, a new read-only node is automatically created in the cluster.

[0014] Further, the method further comprises:

[0015] The number of created links of different read-only nodes in the cluster is obtained, and when a link needs to be created for a read-only node in the cluster, a read-only node with the least number of created links is selected to create the link.

[0016] In a second aspect, the application further provides a distributed business management system, which applies the distributed business management method. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A step flowchart of the distributed business management method provided by an embodiment of the application is shown in the following.

[0018] Figure 2 A step flowchart of automatic expansion of the resource warehouse in the distributed business management method provided by an embodiment of the application is shown in the following.

[0019] Figure 3 A whole flowchart of the actual application of the distributed business management method provided by an embodiment of the application is shown in the following. DETAILED DESCRIPTION

[0020] The application will be described in detail below with reference to the specific embodiments shown in the drawings, but these embodiments do not limit the application, and the changes made by those skilled in the art to the structure, method, or function of these embodiments are included in the protection scope of the application.

[0021] In order to solve the problems in the prior art, in a first aspect, the application provides a distributed business management method, which is applied to a distributed business management system of multiple clusters, any cluster including one master node and multiple read-only nodes, and the business data synchronization is realized by creating links between the read-only nodes of different clusters.

[0022] As an optional implementation manner, the management method provided by the application realizes the automation of business chain creation, reduces the cumbersome process of manual configuration, and improves the efficiency and quality of link creation. As shown in the following, Figure 1 The method includes the following steps:

[0023] In step S11, target user information is obtained, and based on the target user information, the business involved by the target user is queried in the distributed business management system.

[0024] In step S12, based on the queried business, the data involved by each business to the target user is sorted out to form a synchronization link template, the synchronization link template including the cluster where the target synchronization data is located, the form involved by the business, and the number of links to be created, and based on the synchronization link template, the link between the target end and at least one read-only node of the corresponding cluster is automatically created to extract the corresponding business data from different clusters and synchronize to the target end.

[0025] Specifically, in the case of a customer with synchronization requirements, target user information is obtained, and the business scenarios involved by the target user are queried in the distributed business management system according to the target user information. For example, the target user is involved in a1 business of system A cluster, c1 business and c2 business of C cluster, a total of three businesses.

[0026] Based on the business information obtained by the query, each business is analyzed, and according to the data model characteristics of each business, the data of the target user involved in each business is sorted out to form a related synchronization link template. The synchronization link template includes the cluster where the target synchronization data is located, the form involved by the business, and the number of links to be created. After the target customer confirms the synchronization link template, the data synchronization links related to various businesses are automatically created, and the full and incremental data synchronization is started, reducing manual intervention, avoiding errors that may occur during manual configuration, and ensuring efficient start and execution of data synchronization tasks.

[0027] For example, two business links are automatically created for the a1 business of the A cluster. One link synchronizes multiple forms such as dictionary tables and non-customer data that do not require customer-level filtering, and the second link synchronizes multiple business forms that require different filtering conditions. This way, the tedious work of manually entering conditions for each synchronization link is eliminated, the tedious process of manual configuration is reduced, configuration errors caused by manual operation are avoided, and each synchronization link is correctly created according to the business model, thereby improving the accuracy and stability of the system.

[0028] According to the above description, the present application provides a distributed business management method, which can automatically sort out the business data requested by the target end to form a synchronization link template, and automatically create links between the read-only nodes of the corresponding cluster and the target end based on the synchronization link template, thereby realizing synchronization of business data. Through the method provided by the present application, users do not need to select and configure the synchronization conditions of each table one by one on the Web interface. Through the automatic link creation process, the tedious process of manual configuration is reduced, especially for large-scale database clusters, a lot of time and effort can be saved, and configuration errors caused by manual operation are avoided, ensuring that each synchronization link is correctly created according to the business model, thereby improving the accuracy and stability of the system. The problem of tedious and inefficient DTS link creation is solved, and the efficiency and quality of link creation are greatly improved.

[0029] As an optional implementation, the management method provided in the application further combines control pressure through a business link, optimizes resource allocation, and realizes load balancing of each link. The method comprises: when target synchronization data of a target end involves data of at least two different businesses in a same cluster, combining the data of the at least two different businesses in the same cluster, and creating a link based on the combined data of the businesses, comprising: combining a dictionary table in the data of the at least two different businesses into one link, and dividing and creating the link according to the number of forms of the business table in a load-balanced manner after combining the data of the at least two different businesses.

[0030] Specifically, before creating a synchronization link, the business range involved by a customer is sorted out, the number of links and the amount of data of all businesses of the target customer are obtained through association of the synchronization link template in the above description, it is judged which links belong to low-load links according to the number of table data in the link, and the data links of multiple businesses in a same cluster are combined to form a single link, thereby reducing the number of links, reducing redundant links, improving the utilization efficiency of resources, avoiding the synchronization pressure of each link being similar, ensuring that the load of each link is within a reasonable range, avoiding link overload, and improving the stability of the system.

[0031] For example, when a target user uses c1 business and c2 business in a C cluster in the system, there may be relatively less data amount and forms of the c1 business than the c2 business. If only the c1 business is synchronized, only 2 links may be needed, and if only the c2 business is synchronized, only 3 links may be needed. In this case, if the links are directly created according to the simple 2+3 principle, the load pressure of the 2 links of the c1 business will be small, and the load pressure of the 3 links of the c2 business will be large. At this time, the dictionary forms of the c1 business and the c2 business are combined into 1 link, and the business forms are combined and divided into multiple links according to a certain number of forms, so that the number of required links can be reduced from 5 to 3-4, realizing the reduction of the number of links, and through the combination and division of the business forms into multiple links, the load balancing of each link is realized.

[0032] According to the above description, the method provided in the application optimizes the synchronization link template, reduces redundant links, avoids waste of excess resources, improves the utilization efficiency of resources, and ensures that the load of each link is within a reasonable range after optimization of the link configuration, thereby avoiding link overload, improving the stability and performance of the system, solving the problem of uneven link pressure caused by business differentiation of customers, realizing load balancing, and maximizing resource utilization.

[0033] As an optional implementation, the management method provided in the application further balances synchronization pressure through automatic expansion of a resource warehouse, and avoids resource overload of a single read-only node. As shown in Figure 2 The method further comprises the following steps:

[0034] Step S31, monitor the number of used links and the number of created links of the cluster, and compare the number of used links and the number of created links with the total link capacity of the cluster.

[0035] Step S32, based on the proportion of the number of used links and the proportion of the number of created links, determine whether to create a new read-only node in the cluster, and after creating the new read-only node, recalculate the total link capacity and the remaining available link capacity of the cluster.

[0036] Specifically, all read-only nodes of all clusters are centralized into a resource warehouse, and the available capacity in the resource warehouse decreases with the increase of customers. The number of used links and the number of created links of the cluster are monitored, and the number of used links and the number of created links are compared with the total link capacity of the cluster.

[0037] As an optional implementation manner, when the number of used links of the cluster exceeds 80% of the total link capacity, the system prompts that the resource is tight. If the number of created links is still less than 95% of the total link capacity of the cluster, a new link can be created at this time. When the number of used links of the cluster exceeds 90% of the total link capacity and the number of created links exceeds 95% of the total link capacity of the cluster, the system prompts that a new link cannot be created, and the resource needs to be increased. The new read-only node is automatically created in the cluster.

[0038] The information of the newly added read-only node is input and submitted to the resource warehouse for expansion. The resource warehouse reanalyzes and calculates the latest total link capacity and the remaining available link capacity of the cluster according to the information of the newly added node and the original business resource, to provide accurate basis for subsequent business allocation and resource management. At the same time, the information of the newly added node is used as a selectable data source node when creating a synchronous link. When a customer creates a synchronous link, the number of synchronous links of each node is calculated based on all available nodes of the cluster. The subsequent synchronous link selects a node with low load from multiple available nodes of the same business for data synchronization, so as to balance the system synchronization pressure, improve the reliability and stability of the system, and reduce the need for manual intervention, so that the resource management is more flexible, and the system's response ability in resource shortage is enhanced.

[0039] For example, A cluster includes 3 read-only nodes and 2 businesses, and the total capacity is expected to be 30 business links. When a total of 25 link synchronization data are created by multiple customers, the number of links that can be accommodated by A cluster is used to exceed 80%, and there are 5 available resources left. At this time, a 4-core 8G read-only node with lower configuration than the read-write node is added, and the total capacity of the available links of A cluster is calculated to be 40, and there are 17 available resources left. According to the method provided in the application, the system can automatically expand when the link pressure is too large, avoiding the cumbersome process of manual intervention, and improving the flexibility and response speed of the system.

[0040] As an optional implementation, the management method further comprises: obtaining the number of created links of different read-only nodes in the cluster, and when a link needs to be created for a read-only node in the cluster, selecting a read-only node with the least number of created links to create the link. According to the method provided in the application, the system automatically selects a synchronization link from a node with low load, balances the synchronization pressure, and avoids overloading a single node resource.

[0041] Specifically, there are multiple read-only nodes in the cluster for data synchronization, and different nodes use different resources. Before creating a synchronization link, a data source node is selected, the current link usage of each available data node is calculated, and the read-only node with the least number of created links is selected as the data source node to create the link according to the usage ratio of the total link of the node, so as to reduce the load pressure.

[0042] For example, the C cluster has 3 read-only nodes, each of which has the same configuration, and each node can create a maximum of 15 links. The C cluster has a total capacity of 45 links. Currently, C1 node has created 12 links, C2 node has created 10 links, and C3 node has created 2 links. The new customer needs to use the c1, c2, c3 and c4 services of the C cluster, and the system needs to create 6 links after balancing the data model combination. At this time, among the three read-only nodes of the C cluster, C3 node has the least number of existing links and load. Therefore, after selecting the three read-only nodes for judgment during link creation, C3 node is selected as the main source of data synchronization, and 6 links are created for synchronization on C3 node, thereby realizing route balancing of the data source node, reasonably allocating link load, avoiding excessive concentration of resources on a certain node, and improving the reliability and stability of the system.

[0043] As shown in Figure 3 The system accesses new resources (such as database nodes, etc.), which are warehoused together with the existing database resources to realize centralized integration and management of resources, laying a foundation for subsequent operations. The customer-related information is input into the system to trigger identification of the business involved by the customer and to clarify the business scope of the customer in the system. In combination with the customer information and the identified business, multi-business link merging and balancing are performed to balance the link load among different businesses and optimize resource allocation. At the same time, low-load resource nodes are selected according to resource usage to avoid node overload and ensure balanced system performance. Through the above steps, a link creation scheme is determined and data synchronization is performed. After synchronization is completed, the result is fed back to the resource node warehousing to update the resource state (such as resource load, available link number, etc.), so that the system can dynamically adapt to business changes, form a closed-loop management, and ensure efficient and stable operation of the business management system.

[0044] According to the above description, the distributed business management method provided by the application has the following advantages. First, the resource node warehouse is used to calculate the number of creatable links of the read-only node based on the influence characteristics of multiple factors, so that the resource is quantified and controlled, the confusion of resource allocation is avoided, and the management efficiency is improved. By centrally managing all resource nodes, the resource status of each business can be more intuitively mastered, the system can intelligently allocate the synchronization link according to the current resource load, the performance problem caused by insufficient resources of some links is avoided, and the synchronization process is ensured to be smooth and efficient. Second, the resource warehouse is automatically expanded and the business link is automatically created, the current resource details are automatically calculated and refreshed, the control of the resource boundary is realized, the cumbersome process of manual intervention is avoided, the flexibility and response speed of the system are improved, and the correct creation of each synchronization link according to the business model is ensured, so that the accuracy and stability of the system are improved. Third, the pressure of the business link combination control and the data node routing pressure equalization are reduced, the redundant links of the system are reduced, the waste of excess resources is avoided, the node with less pressure is automatically selected when creating a link to balance the load pressure of the source node, the link overload is avoided, the stability and performance of the system are improved, and the load balancing and resource utilization maximization are realized.

[0045] In the second aspect, the application further provides a distributed business management system, which applies the distributed business management method described above. The system can efficiently process a large number of synchronization links, simplify resource scheduling and management, and improve the efficiency and quality of link creation.

[0046] It should be understood that the word "exemplary" used herein means "serving as an example, instance, or illustration". Any embodiment described as "exemplary" is not necessarily preferred or advantageous over other embodiments and / or does not exclude features combining other embodiments. It should be understood that certain features of the application described in the context of separate embodiments can also be provided in combination. Conversely, various features of the application described in the context of a single embodiment can also be provided separately or in any appropriate combination or as part of any other described embodiment of the application.

[0047] In the description of the application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" herein is only a description of the association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, "at least one" means one or more, and "multiple" means two or more. "First", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different.

[0048] The above disclosed is only the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Those skilled in the art can understand that changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and scope of the present application and the appended claims should be equivalent replacements and still fall within the scope of the present application.

Claims

1. A method of distributed service management, characterized by, The method is applied to a multi-cluster distributed business management system, any cluster comprising one master node and multiple read-only nodes, and the business data synchronization is achieved by creating a link between the read-only nodes of different clusters and a target end; The method comprises: obtaining target user information, and querying a business involved by a target user in the distributed business management system based on the target user information; based on the queried business, combing data related to the target user of each business to form a synchronization link template, the synchronization link template comprising a cluster where target synchronization data is located, a form related by the business, and a required number of links, and based on the synchronization link template, automatically creating a link between the target end and at least one read-only node of a corresponding cluster to extract corresponding business data from different clusters and synchronize the business data to the target end.

2. The distributed business management method of claim 1, wherein when the target synchronization data of the target end involves data of at least two different businesses in the same cluster, the data of the at least two different businesses in the same cluster are combined, and the link is created based on the combined data of the business, comprising: combining dictionary tables in the data of the at least two different businesses into one link, and combining business tables in the data of the at least two different businesses and dividing the link according to the number of forms of the business tables in a load balancing manner. The method further comprises:

3. The distributed traffic management method of claim 1, wherein, monitoring the number of used links and the number of created links of a cluster, comparing the number of used links and the number of created links with the total number of links that can be accommodated by the cluster, and based on the proportion of the number of used links and the proportion of the number of created links, determining whether to create a new read-only node in the cluster, and when a new read-only node is created, recalculating the total number of links that can be accommodated by the cluster and the number of remaining available links.

4. The distributed business management method of claim 3, wherein when the proportion of the number of used links exceeds 90% and the proportion of the number of created links exceeds 95%, a new read-only node is automatically created in the cluster. The method further comprises: obtaining the number of created links of different read-only nodes in a cluster, and when a link needs to be created for a read-only node in the cluster, selecting a read-only node with the least number of created links for link creation.

5. The distributed traffic management method of claim 1, wherein, The system applies the distributed business management method of any one of claims 1 to 5. ​ 6. A distributed business management system characterized by, ​

Images