Automatic and efficient grouping method and system

Abstract

One technical scheme of the invention discloses an automatic and efficient grouping method. Another technical scheme of the invention provides an automatic and efficient grouping system which is used for realizing the automatic and efficient grouping method. Through a grouping method based on demands, table data, table data weights and server states, scenes with high data increment and high demand iteration in financial scenes are supported, the defects caused by common grouping schemes are reduced to the maximum extent, and the grouping algorithm is more suitable for actual application scenes from demands, so that the method is suitable for large-scale popularization and application. On the basis of reducing the manual maintenance cost, automatic reasonable allocation and efficient utilization of resources can be realized, resource waste is reduced, and the performance and reliability of the system are improved.

Description

Technical Field

[0001] The invention relates to a distributed grouping method based on request and server configuration, belonging to the technical field of distributed systems. Background Art

[0002] Nowadays, in financial scenarios, software services have to face the situation of high data growth and high iteration of customer needs. To deal with such situations, how to comprehensively cover the data while ensuring rapid iteration and rapid expansion is very difficult for a software service.

[0003] Currently, the common solution is to implement distributed software services. The general distributed system expansion solutions include the following two:

[0004] First) Vertical Scaling: This refers to increasing the system's processing power by increasing the resources of a single node (such as CPU, memory, disk space, etc.).

[0005] The disadvantages of the first solution are: high hardware cost, physical baseline, and hardware compatibility issues after upgrade.

[0006] Second) Horizontal expansion: It is to share the load by adding more nodes and improve the overall processing capacity of the system.

[0007] The disadvantages of the second solution are: increased management and maintenance costs, network latency, and bandwidth issues;

[0008] Based on the premise that all the above solutions have their own shortcomings, we found that in such financial scenarios, not all needs require all data tables, and different needs require access to different data tables. Summary of the Invention

[0009] The purpose of the present invention is to propose a technical solution for automatically and evenly distributing requirements and table data as a combination according to a set of strategies.

[0010] In order to achieve the above object, the technical solution of the present invention is to disclose an automated and efficient grouping method, which is characterized by comprising the following steps:

[0011] Step 1: Data preprocessing, including:

[0012] Step 101: Obtain dependency data between requests and data tables, and then for any request, obtain all data tables used by the request, and for any data table, obtain all requests that depend on the data table;

[0013] Step 102: Calculate the weight of each data table. The larger the data table file, the greater its weight.

[0014] Step 103, transform all requests and data tables into vectors;

[0015] Step 104, remove requests that can be contained by other requests, wherein for any two requests, when all data tables that request one depends on are contained by all data tables that request two depends on, then request one is contained by request two, remove request one;

[0016] Step 2, grouping, including:

[0017] Step 201, create n initial groups TableGroup according to the number of servers or server groups n, n≥1;

[0018] Step 202, optimize grouping:

[0019] Take out the m data tables farthest from the center point in each group TableGroup, and re-group, wherein the value of m is an empirical parameter;

[0020] Step 203, request grouping:

[0021] Traverse all requests and match requests to appropriate groups TableGroup, wherein if the current group TableGroup has all the data tables required by the current request, it is considered that the current group TableGroup matches the current request, otherwise, it is considered that the current group TableGroup does not match the current request;

[0022] Step 3, grouping implementation, including:

[0023] Group according to the grouping strategy of step 2 to generate n groups TableGroup, each group TableGroup contains its own request set and data table set, combine the obtained grouping result, configure the routing to ensure that each request is requested to the corresponding server or server cluster, and configure the data table for the n servers or server clusters, so that the n servers or server clusters load the required data tables according to the grouping result.

[0024] Preferably, in step 101, Intermediate Language analysis is used to obtain request and data table dependency from source code.

[0025] Preferably, in step 102, the weight corresponding to each data table is the quantitative value of the file size of the current data table.

[0026] Preferably, in step 103, when transforming requests into vectors, the number of data tables that the current request depends on is used as the dimension of the component, and the weight corresponding to each data table is used as the component value under the corresponding component dimension.

[0027] Preferably, in step 103, when converting the data table into a vector, the number of associated requests is used as a component dimension, and the component value under each component dimension is (100 / the number of associated requests).

[0028] Preferably, the step 201 further includes the following steps:

[0029] Traverse all data tables and select n data tables with the longest vector distance. Create n initial groupings, TableGroup, using these n data tables as the initial core.

[0030] Based on the principle of shortest distance between vectors, the remaining data tables are assigned to n initial groupings, and the weights corresponding to all data tables in each group are ensured to be as evenly distributed as possible. The maximum and minimum weight thresholds are set to control the weight difference between any two groups of TableGroups.

[0031] Preferably, in step 203, if all the groups TableGroup cannot match the current request, matching is performed according to the minimum cost principle, first completing the data tables in the current group TableGroup, and then allocating the request that has a dependency relationship with the data tables to the current group TableGroup.

[0032] Another technical solution of the present invention is to provide an automated and efficient grouping system for implementing the above-mentioned automated and efficient grouping method, which is characterized by comprising:

[0033] Request analysis module: used to analyze the requirements of each request for the data table and obtain the dependency relationship between the request and the data table;

[0034] Data cache module: used to store requests, data tables, data table weights and dependencies after request analysis;

[0035] Grouping strategy execution module: used to execute the grouping strategy described in step 2 of the above-mentioned automated and efficient grouping method;

[0036] Grouping implementation module: used to configure the results to the routing module and each server or server cluster based on the grouping results. The servers will perform subsequent actions including data loading based on the grouping results.

[0037] Preferably, a routing system module is exposed to the outside to transfer different requests to different service groups according to the grouping situation.

[0038] The application supports the scene of high increment of data and high iteration of demand in the financial scene by a grouping method based on demand, table data, table data weight and server state, minimizes the shortcomings brought by common grouping schemes, and the grouping algorithm is more suitable for the actual application scene from the demand, can realize the automatic and reasonable allocation and efficient use of resources on the basis of reducing the artificial maintenance cost, reduces the resource waste, and improves the performance and reliability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The request, table grouping logic is illustrated;

[0040] Figure 2 The request and table dependency relationship is illustrated;

[0041] Figure 3 The request flow is illustrated. DETAILED DESCRIPTION

[0042] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not used to limit the scope of the application. In addition, it should be understood that after reading the content taught by the application, those skilled in the art can make various modifications or modifications to the application, and these equivalent forms also fall within the scope defined by the appended claims of the application.

[0043] In the embodiments of the application, the "request" is referred to as "Req", the "data table" is referred to as "Table", the "weight" is referred to as "Weight", and the "single server or server cluster (used as load balancing)" is referred to as "group". The automatic and efficient grouping method disclosed in the embodiments of the application aims to: let the total Weight of each group be balanced, let the sum of the Table Weight of all groups be relatively small, and let the redundancy of Table between different groups be low.

[0044] As shown in Figure 1 The automatic and efficient grouping method disclosed in the embodiments of the application includes the following steps:

[0045] Step 1, preprocessing before grouping, further including:

[0046] Step 101, preparing the dependency relationship data of Req and Table: using IL (Intermediate Language) analysis to obtain the Req and Table dependency relationship from the source code (as shown in Figure 2The specific method is to use Reflection to obtain all types, methods of all types, and dependencies of all methods in the assembly. Based on the dependencies of the obtained Req and Table: for any Req, all n Tables used by the Req can be known; for any Table, the number of Reqs that depend on the Table can be known, providing basic data for subsequent grouping.

[0047] Step 102, Table Weight data preparation: quantifying the size of Table according to the size of Table data file, that is, obtaining the Weight corresponding to each Table data file. The larger the Table data file, the larger the Weight, that is, the larger the magnitude. When converting Req to vector later, it is used as the component value.

[0048] Step 103, vectorization:

[0049] Convert Req to vector, where the number of dependent Tables is the component dimension, and the Weight corresponding to each Table data file is the component value in the corresponding component dimension.

[0050] Convert Table to vector, where the number of associated Reqs is the component dimension, and the component value in each component dimension is 100 / Req number.

[0051] The distance between vectors is the cosine distance, that is, calculated using 1-(inner product between vectors) / (modulus product).

[0052] Step 104, dimension reduction operation: ignore those Reqs that can be contained by other Reqs, that is, if Req A depends on Tables A and B, and Req B depends on Table A, then Req B is considered to be contained by Req A and can be marked as ignored.

[0053] Step 2, start grouping, further including:

[0054] Step 201, initially group Tables, assuming there are n servers or server groups, then create n initial group Tables TableGroup. The TableGroup itself is a vector, and its components are the de-duplicated summary of all Table components inside it. The specific steps are as follows:

[0055] Select the n Tables that are as far apart as possible from all Tables by traversing, and use the n Tables as the initial core of the initial n TableGroups.

[0056] Based on the principle of shortest distance between table vectors, the remaining tables are assigned to each TableGroup, while ensuring that the sum of the table weights of each TableGroup is as even as possible. In this embodiment of the present invention, the average weight = the total weight of all tables / n, and maximum and minimum weight thresholds are set to control the differences between groups to be small.

[0057] Step 202: Table grouping optimization:

[0058] After the initial grouping, since the center of a TableGroup changes dynamically as more tables are added, there is room for optimization for the table farthest from the center in each TableGroup. Therefore, the m tables farthest from the center in each TableGroup are removed and the grouping logic from the previous step is repeated. The value of m is an empirical parameter. A preferred implementation is m = n / 10, where n is the number of TableGroups.

[0059] Step 203: Req grouping:

[0060] Based on the preliminary grouping, all Reqs are traversed and matched to appropriate TableGroups. That is, a TableGroup that has all the tables required by the Req is considered suitable. However, if no TableGroups are matched, the minimum cost principle (the TableGroup with the smallest sum of the weights of the tables to be filled) is used for matching. First, the tables in the current TableGroup are filled, and then the Req is assigned to the current TableGroup.

[0061] Step 3, grouping implementation, further includes:

[0062] According to the grouping strategy, after grouping, n groups have been generated (each group contains its own Req set and Table set). Based on this grouping result, the routing is configured (to ensure that each Req request is sent to the corresponding server or server cluster), and the Table is configured for the n servers or server clusters (according to the grouping results, each server loads the required Table data according to the grouping result).

[0063] To implement the above grouping strategy, the corresponding system should include the following modules:

[0064] 1. Request analysis module: used to analyze the requirements of each Req on the Table, that is, the dependency relationship between the Req and the Table.

[0065] 2. Data cache module: used for storing data such as Req, Table, Table Weight and dependency relationship after request analysis.

[0066] 3. Grouping policy execution module: grouping according to preliminary Table, grouping optimization, total flow of Req grouping, and execution of grouping policy based on all data.

[0067] 4. Grouping implementation module: according to grouping results, the results are configured to the routing module and each server or server cluster, and each server performs subsequent data loading and other actions according to the results.

[0068] 5. A routing system module is exposed externally to transfer different request requests to different service groups according to the grouping situation.

[0069] After the service request and service data are distributed and combined in the form of groups according to the above grouping algorithm, the optimization of the distribution of required data is achieved from the actual demand angle of the service request.

[0070] A specific implementation case is:

[0071] Suppose there is a distributed system:

[0072] including Req A, B, C and Table1, 2, 3, 4, wherein Table1 weight is 3, Table2 and Table3 weight is 1, and Table4 weight is 4:

[0073] The requirements are as follows:

[0074] Req A needs to access Table1 and Table2;

[0075] Req B needs to access Table2 and Table3;

[0076] Req C needs to access Table3 and Table4.

[0077] The system currently contains two servers S1 and S2:

[0078] According to the grouping policy described above:

[0079] Req A, Req B are allocated to S1;

[0080] Req C is allocated to S2;

[0081] Table1, Table2, Table3 are allocated to S1;

[0082] Table3, Table4 are allocated to S2.

[0083] The process of receiving access is as follows Figure 3 shown.

Claims

1. An automated and efficient grouping method, characterized in that: The following steps are involved: Step 1: Data preprocessing, including: Step 101: Obtain dependency data between requests and data tables, and then for any request, obtain all data tables used by the request, and for any data table, obtain all requests that depend on the data table; Step 102: Calculate the weight of each data table. The larger the data table file, the greater its weight. Step 103: Convert all requests and data tables into vectors. Step 104: Discard requests that can be included in other requests. For any two requests, if all data tables that request one depends on are included in all data tables that request two depends on, then request one is included in request two, and request one is discarded. Step 2: Grouping, including: Step 201: Create n initial groups TableGroup according to the number n of servers or server groups, where n≥1. Step 202: Optimize grouping: Take out the m data tables farthest from the center point in each group TableGroup and regroup them. The value of m is a logical empirical parameter. Step 203: Request grouping: Traverse all requests and match them to the appropriate TableGroup. If the current TableGroup contains all the data tables required by the current request, the current TableGroup is considered to match the current request. Otherwise, the current TableGroup is considered to not match the current request. Step 3: Group implementation, including: According to the grouping strategy of step 2, grouping is performed to generate n groups TableGroup. Each group TableGroup contains its own request set and data table set. Combined with the obtained grouping results, the routing is configured to ensure that each request is sent to the corresponding server or server cluster, and the data tables are configured for the n servers or server clusters so that the data tables that need to be loaded are loaded on the n servers or server clusters according to the grouping results.

2. An automated and efficient grouping method according to claim 1, characterized in that: In step 101, Intermediate Language analysis is used to obtain the request and data table dependencies from the source code.

3. An automated and efficient grouping method according to claim 1, characterized in that: In step 102 , the weight corresponding to each data table is the quantitative value of the file size of the current data table.

4. An automated and efficient grouping method according to claim 1, characterized in that: In step 103 , when converting the request into a vector, the number of data tables that the current request depends on is used as a component dimension, and the weight corresponding to each data table is used as a component value under the corresponding component dimension.

5. An automated and efficient grouping method according to claim 1, characterized in that: In step 103 , when the data table is converted into a vector, the number of associated requests is used as a component dimension, and the component value under each component dimension is (100 / the number of associated requests).

6. An automated and efficient grouping method according to claim 1, characterized in that: The step 201 further includes the following steps: Traverse all data tables and select n data tables with the longest vector distance. Create n initial groupings, TableGroup, using these n data tables as the initial core. Based on the principle of shortest distance between vectors, the remaining data tables are assigned to n initial groupings, and the weights corresponding to all data tables in each group are ensured to be as evenly distributed as possible. The maximum and minimum weight thresholds are set to control the weight difference between any two groups of TableGroups.

7. An automated and efficient grouping method according to claim 1, characterized in that: In step 203, if all the TableGroups cannot match the current request, matching is performed according to the minimum cost principle, first completing the data tables in the current TableGroup, and then allocating the request that has a dependency relationship with the data tables to the current TableGroup.

8. An automated and efficient grouping system for implementing the automated and efficient grouping method according to claim 1, characterized in that: include: Request analysis module: used to analyze the requirements of each request for the data table and obtain the dependency relationship between the request and the data table; Data cache module: used to store requests, data tables, data table weights and dependencies after request analysis; A grouping strategy execution module: configured to execute the grouping strategy described in step 2 of the automated and efficient grouping method according to claim 1; Grouping implementation module: used to configure the results to the routing module and each server or server cluster based on the grouping results. The servers will perform subsequent actions including data loading based on the grouping results.

9. An automated and efficient grouping system as claimed in claim 8, characterized in that: A routing system module is exposed to the outside world to transfer different requests to different service groups according to the grouping situation.

Images