A data distribution method, apparatus, electronic device, chip, and medium

By monitoring server traffic data and using clustering algorithms to optimize data distribution, this technology solves the problem of relying on human experience in existing technologies, achieving efficient and low-cost data distribution, and is applicable to fields such as data analysis, cloud computing, and distributed databases.

CN117170964BActive Publication Date: 2026-11-17CHINA MOBILE GRP HENAN CO LTD +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310997672.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-08
Publication Date
2026-11-17
Estimated Expiration
2043-08-08

AI Technical Summary

Technical Problem

Existing data distribution methods rely on human experience and cannot accurately determine the distribution key, resulting in low data processing efficiency and high cost, especially when the data volume is large, requiring higher-configuration equipment.

Method used

By monitoring traffic data between servers, clustering algorithms are used to optimize data distribution, reduce data broadcasting or redistribution, and a self-developed three-tier architecture is adopted to automatically collect network card traffic data and distribute it optimally according to the data bitmap, thus eliminating reliance on human experience.

Benefits of technology

It improves the real-time performance and efficiency of data processing, reduces data broadcasting or redistribution, lowers equipment costs, and is suitable for scenarios such as data analysis, cloud computing, big data, and distributed databases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117170964B_ABST
    Figure CN117170964B_ABST
Patent Text Reader

Abstract

This disclosure provides a data distribution method, apparatus, electronic device, chip, and medium, relating to the field of database technology. The method includes: monitoring data flow between multiple servers and obtaining monitoring results, the monitoring results including the flow path of the data in the multiple servers, the data flow including multiple mobile network traffic data; determining the distribution degree of the data flow in the multiple servers based on the monitoring results and a clustering algorithm; determining a first data distribution bitmap based on the distribution degree, and distributing the data corresponding to the data set in the database to the multiple servers according to the first data distribution bitmap, the first data distribution bitmap including the data distribution of the data set in the database, reducing data broadcasting or redistribution, not relying on related documents and manual operations, eliminating the dependence on selecting distribution keys based on human experience, and improving the real-time performance and efficiency of data processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of database technology, and in particular to a data distribution method, apparatus, electronic device, chip, and medium. Background Technology

[0002] Data distribution refers to storing or distributing data across different locations or nodes for computation, processing, or storage. Data distribution is a crucial technology for data processing and storage, requiring design and implementation tailored to specific needs and scenarios. It also necessitates consideration of data integrity and consistency to ensure data security and reliability. Greenplum is a relational MPP (Massively Parallel Processing) database cluster, a logical database composed of several independent database services. Greenplum distributes massive amounts of data across multiple nodes according to a distribution strategy based on the table's distribution key.

[0003] However, data distribution methods in related technologies rely on the user's understanding and technical skill in determining the accuracy of the distribution key. Furthermore, when dealing with large volumes of data, more sophisticated equipment is required to improve network transmission efficiency. Summary of the Invention

[0004] This disclosure provides a data distribution method, apparatus, electronic device, chip, and medium to address the problem of excessive reliance on human experience in related technologies. By monitoring traffic data between various servers and optimizing data distribution based on aggregation algorithms, it reduces data broadcasting or redistribution, eliminates reliance on related documents and manual operations, and removes the dependence on selecting distribution keys based on human experience, thereby improving the real-time performance and efficiency of data processing.

[0005] A first aspect of this disclosure proposes a data distribution method, which includes: monitoring circulating data among multiple servers, obtaining monitoring results, the monitoring results including the circulation path of the circulating data among the multiple servers, the circulating data including multiple mobile network traffic data; determining the distribution degree of the circulating data among the multiple servers based on the monitoring results and a clustering algorithm; determining a first data distribution bitmap based on the distribution degree, and distributing the data corresponding to the data set in the database to the multiple servers according to the first data distribution bitmap, the first data distribution bitmap including the data distribution situation corresponding to the data set in the database.

[0006] In some embodiments of this disclosure, the method further includes: receiving data distribution information sent by a database administrator; determining whether the data distribution bitmap information is stored in a database and obtaining a data distribution bitmap task; and based on the data distribution bitmap task, connecting to the control node of the database and obtaining a second data distribution bitmap.

[0007] In some embodiments of this disclosure, determining whether the data distribution bitmap information is stored in the database and obtaining the data distribution bitmap task includes: if the data distribution information is stored in the database, retrieving the data distribution task from the database based on the data distribution information and sending a notification message, the notification message being used to prompt the database administrator that the data distribution bitmap task is stored in the database; if the data distribution bitmap information is not stored in the database, creating the data distribution bitmap task based on the data distribution information.

[0008] In some embodiments of this disclosure, monitoring the flow of data between multiple servers and obtaining monitoring results includes: obtaining the identifiers corresponding to the flow of data between multiple servers and storing the identifiers in a database.

[0009] In some embodiments of this disclosure, determining the distribution degree of circulating data across multiple servers based on monitoring results and clustering algorithms includes: performing central clustering on the circulating data based on a preset number of categories and a clustering algorithm to determine the distribution degree of circulating data across multiple servers, wherein the preset number of categories is the same as the number of multiple servers.

[0010] In some embodiments of this disclosure, determining a first data distribution bitmap based on the distribution degree, and distributing the data corresponding to the data set in the database to multiple servers according to the first data distribution bitmap includes: distributing circulating data to multiple servers corresponding to the maximum distribution degree of the circulating data according to the distribution degree, to obtain a third data distribution bitmap; distributing static data in the second data distribution bitmap to the third data distribution bitmap according to the second data distribution bitmap, to obtain a fourth data distribution bitmap; determining the computational cost of multiple nodes in the multiple servers, wherein the computational cost data includes storage cost and processing cost; and determining the first data distribution bitmap based on the computational cost, the second data distribution bitmap, and the fourth data distribution bitmap.

[0011] In some embodiments of this disclosure, determining the first data distribution bitmap based on computational cost, the second data distribution bitmap, and the fourth data distribution bitmap includes: if the computational cost of some or all nodes is less than a preset threshold, using the data distribution positions of some or all nodes in the fourth data distribution bitmap as the data distribution positions of some or all nodes in the first data distribution bitmap; if the computational cost of some or all nodes is greater than the preset threshold, using the data distribution positions of some or all nodes in the second data distribution bitmap as the data distribution positions of some or all nodes in the first data distribution bitmap.

[0012] A second aspect of this disclosure provides a data distribution apparatus, comprising: an acquisition unit for monitoring data flow between multiple servers and acquiring monitoring results, the monitoring results including the flow path of the data flow among the multiple servers, the data flow including multiple mobile network traffic data; a determination unit for determining the distribution degree of the data flow among the multiple servers based on the monitoring results and a clustering algorithm; and a distribution unit for adjusting a second data distribution bitmap to a first data distribution bitmap based on the distribution degree, and distributing data corresponding to a data set in a database to the multiple servers according to the first data distribution bitmap, the second data distribution bitmap including the data distribution information corresponding to the data set stored in the database.

[0013] A third aspect of this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0014] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0015] A fifth aspect of this disclosure provides a chip including one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processors, cause the electronic device to perform the methods described in the first aspect of this disclosure.

[0016] In summary, the data distribution method proposed in this disclosure monitors the data flow between multiple servers and obtains monitoring results. These results include the flow path of the data across the multiple servers, and the data includes multiple mobile network traffic data. Based on the monitoring results and a clustering algorithm, the distribution degree of the data across the multiple servers is determined. Based on the distribution degree, a first data distribution bitmap is determined, and the data corresponding to the data set in the database is distributed to the multiple servers according to the first data distribution bitmap. The first data distribution bitmap includes the data distribution corresponding to the data set in the database. By monitoring the traffic data between the servers and optimizing the data distribution using an aggregation algorithm, the broadcasting or redistribution of data is reduced. This method does not rely on related documents or manual operations, eliminates the dependence on selecting distribution keys based on human experience, and improves the real-time performance and efficiency of data processing.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 A flowchart of a data distribution method provided in an embodiment of this disclosure; Figure 2 A flowchart of a data distribution method provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram illustrating the monitoring of data flow between three servers, as provided in an embodiment of this disclosure. Figure 4 This is a schematic diagram of a second data distribution bitmap provided in an embodiment of the present disclosure, using three servers Node1, Node2, and Node3 as an example; Figure 5 This is a schematic diagram of a third data distribution bitmap provided in an embodiment of the present disclosure, using three servers Node1, Node2, and Node3 as an example; Figure 6 This is a schematic diagram of a fourth data distribution bitmap provided in an embodiment of the present disclosure, using three servers Node1, Node2, and Node3 as an example; Figure 7 An architecture diagram of a specific data distribution method provided in this embodiment of the disclosure; Figure 8 A flowchart illustrating a specific data distribution method provided in this embodiment of the disclosure; Figure 9 This is a schematic diagram of the structure of a data distribution device provided in an embodiment of the present disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0020] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments are described below with reference to the accompanying drawings.

[0021] Data distribution refers to storing or distributing data across different locations or nodes for computation, processing, or storage. Data distribution is a crucial technology for data processing and storage, requiring design and implementation tailored to specific needs and scenarios. It also necessitates consideration of data integrity and consistency to ensure data security and reliability. Greenplum is a relational MPP (Massively Parallel Processing) database cluster, essentially a logical database composed of several independent database services. Unlike Oracle RAC's share-disking architecture, Greenplum employs a Shared-Nothing architecture. The entire cluster consists of numerous data nodes (Segment Hosts) and control nodes (MasterHosts), with each data node capable of running multiple databases. In a typical Shared-Nothing architecture, all resources (CPU, memory, disk) on each node are independent; each node possesses only a portion of the total data and can only utilize its own resources.

[0022] Greenplum database distributes massive amounts of data across multiple nodes according to a distribution strategy based on the table's distribution key. For example, in a cluster of one hundred nodes, each node stores only one percent of the total data, and all one hundred nodes process data in parallel, resulting in performance tens of times faster than a single, well-configured node. However, evenly distributing data does not necessarily lead to a complete performance improvement. During SQL (Structured Query Language) queries, when performing a join query between two tables, if the join field contains the distribution key, the data can be processed within their respective nodes. If the join field does not contain the distribution key, data redistribution is required, moving the data to a single node for the join to obtain the necessary data. Since remote network data transmission is much more expensive than local retrieval, this process incurs significant network overhead, leading to a decrease in query performance. Moreover, as the data volume increases, the amount of data that needs to be transmitted will also increase.

[0023] There are two types of SQL joins in distributed databases: Single-database association: The association key is the same as the distribution key, and only a single database needs to be associated to obtain the result.

[0024] Cross-database joins: When the join key and distribution key are inconsistent, the data needs to be redistributed. This can be converted into a single-database join to achieve the table relationship.

[0025] Taking the relationship between tables A and B in Table 1 below and the inner join as an example, if the join key of one table is not the distribution key, then the table will be broadcast or redistributed, moving the data to a node for association, thereby obtaining the data.

[0026]

[0027] Table 2 select from A,B where A.id=B.id; The distribution key is the same as the association key, which is a single-database association and will not cause broadcasting or redistribution.

[0028] select from A,B where A.id=B.id2; In this case, the join key of table A is a distribution key, while the join key of table B is not a distribution key. Therefore, the join can be achieved in two ways.

[0029] Redistribute the data from table B to a single node based on the id2 field, and then join it with table A. The amount of data redistributed is N. Broadcast table A, placing a full copy of the data on each node, and then join it with table B to obtain the result. The amount of data broadcast is M. Number of nodes. Therefore, when N > M When determining the number of nodes, choose table A for broadcasting; otherwise, choose table B for redistribution.

[0030] select from A,B where A.id2=B.id2; The two tables have different join keys and distribution keys.

[0031] Redistributing data from tables A and B across each node based on the id2 field incurs a cost of M+N. Alternatively, one table can be broadcast and then joined; broadcasting the smaller table reduces the cost. The cost of broadcasting is min(M, N). The number of nodes. Therefore, when M+N>min(M,N) When determining the number of nodes, choose to broadcast from the smaller table; otherwise, choose to redistribute to both tables.

[0032] Other related situations are similar, and as the amount of data in the table and the number of related conditions increase, the cost of transmitting data over the network will become increasingly higher.

[0033] Therefore, to address the above issues, the main data distribution methods for related technologies are: 1) If resources are sufficient, redefine the new distribution key of the table and redistribute the data to minimize data redistribution.

[0034] 2) Choose better network switches and network interface cards (NICs) to improve network transmission efficiency.

[0035] However, these technologies rely on personnel experience and the distribution key. Determining the accuracy of the distribution key depends entirely on the personnel's understanding and technical skill regarding the database table data and structure. Furthermore, the diversity of related fields and the limitations of hash distribution rules mean that optimal data distribution cannot be achieved solely through the distribution key. Additionally, these technologies are costly; as the data volume increases, higher-configuration equipment is required, leading to a linear increase in cost.

[0036] To address the problems existing in related technologies, this disclosure proposes a data distribution method. It employs a self-developed three-layer architecture task distribution scheme, a set of standard mechanisms including task scheduling, optimal data distribution analysis and confirmation, and optimal distribution implementation based on data bitmaps. Furthermore, by automatically collecting network interface card traffic data from each segment (data node), and based on aggregation algorithms and data location maps, it optimizes to obtain an accurate and optimal data distribution under the condition of data hash distribution. This reduces data broadcasting or redistribution, does not rely on related documents and manual operations, eliminates the dependence on experience in selecting distribution keys, and improves the real-time performance and efficiency of processing.

[0037] The method proposed in this disclosure can achieve efficient and reliable data distribution and can be applied to various scenarios, including data analysis, cloud computing, big data, distributed databases, parallel computing, etc., which are not limited in the embodiments of this disclosure.

[0038] The data distribution method provided in this application will be described in detail below with reference to the accompanying drawings.

[0039] Figure 1 A flowchart illustrating a data distribution method provided in an embodiment of this disclosure. Figure 1 As shown, the data distribution method includes steps 101-103.

[0040] Step 101: Monitor the data flow between multiple servers and obtain the monitoring results. The monitoring results include the data flow path between multiple servers and the data flow includes multiple mobile network traffic data.

[0041] In the embodiments of this disclosure, "multiple servers" refers to servers corresponding to multiple network devices, where the network device can be a network interface card (NIC). "Traffic data" refers to network traffic data moving between multiple servers, and this traffic data can be stored in a database. The flow path of traffic data across multiple servers refers to whether the traffic data appears on each server.

[0042] In this disclosure, network interface card (NIC) traffic monitoring is enabled. For data initiated by SQL, an SQL parser can be used to parse the SQL statement, obtain network traffic data between servers, and save the obtained network traffic data to the database.

[0043] SQL (Structured Query Language) is a programming language specifically designed for managing relational databases. It is a standard language used to manipulate and query databases, including inserting, deleting, updating, and querying data.

[0044] Step 102: Based on the monitoring results and clustering algorithm, determine the distribution of circulating data across multiple servers.

[0045] In the embodiments of this disclosure, the clustering algorithm can be the k-means clustering algorithm. The k-means clustering algorithm is a common unsupervised learning algorithm used to divide data points into k different clusters, i.e., k preset categories. By randomly selecting k initial centroids, each data point is then assigned to the cluster containing its nearest centroid. Based on the cluster assigned to each data point, the centroid of each cluster is recalculated. This process is repeated until the cluster centroids no longer change or a predetermined number of iterations is reached. This allows the circulating data to be centrally clustered onto different servers, thus obtaining the distribution degree of the circulating data.

[0046] The degree of distribution of circulating data across multiple servers can refer to the distribution of circulating data among different servers, as well as the quantity and density of circulating data on each server.

[0047] Step 103: Determine the first data distribution bitmap based on the distribution degree, and distribute the data corresponding to the data set in the database to multiple servers according to the first data distribution bitmap. The first data distribution bitmap includes the data distribution of the data set in the database.

[0048] In the embodiments of this disclosure, a third data distribution bitmap corresponding to the flowing data can be obtained based on the degree of distribution of the circulating data across different servers. This third data distribution bitmap represents the optimal location distribution map of the flowing data. Simultaneously, combined with a second data distribution bitmap obtained through the control node of the database, network traffic data that has not moved is distributed according to the second data distribution bitmap to form an optimal location distribution map of the network traffic data, i.e., a fourth data distribution bitmap. Based on the fourth and second data distribution bitmaps, a first data distribution bitmap is obtained, and the movement of network traffic data between server nodes is controlled using this first data distribution bitmap. The data distribution bitmap represents the distribution of network traffic data across the servers.

[0049] It is important to note that in the process of obtaining the first data distribution bitmap, the computational cost of each node involved in adjusting the data distribution also needs to be considered. Computational cost includes storage cost and processing cost, namely the capacity cost and CPU cost of moving the data distribution.

[0050] If the computation cost of a node is greater than or equal to a preset threshold, the node will still be distributed according to the position in the second data distribution bitmap; if the computation cost of a node is less than the preset threshold, the node can be distributed according to the position in the fourth data distribution bitmap, so as to control the cost and avoid high costs while satisfying the optimal data distribution.

[0051] In summary, the data distribution method proposed in this disclosure monitors the data flow between multiple servers and obtains monitoring results. These results include the flow path of the data across the multiple servers, and the data includes multiple mobile network traffic data. Based on the monitoring results and a clustering algorithm, the distribution degree of the data across the multiple servers is determined. Based on the distribution degree, a first data distribution bitmap is determined, and the data corresponding to the data set in the database is distributed to the multiple servers according to the first data distribution bitmap. The first data distribution bitmap includes the data distribution corresponding to the data set in the database. By monitoring the traffic data between the servers and optimizing the data distribution using an aggregation algorithm, the broadcasting or redistribution of data is reduced. This method does not rely on related documents or manual operations, eliminates the dependence on selecting distribution keys based on human experience, and improves the real-time performance and efficiency of data processing.

[0052] based on Figure 1 The embodiment shown, Figure 2 A flowchart of a data distribution method proposed in this disclosure is further shown. Figure 2 based on Figure 1 The illustrated embodiment further defines steps 101 and 102. Figure 2 In the illustrated embodiment, steps 201, 202, and 203 precede step 101, and step 102 includes step 205. For example... Figure 2 As shown, the method includes the following steps: Step 201: Receive data distribution information sent by the database administrator.

[0053] In the embodiments of this disclosure, the Database Administrator (DBA) is a general term for staff who manage and maintain a Database Management System (DBMS), and is mainly responsible for the full lifecycle management of the business database from design and testing to deployment and delivery.

[0054] In this disclosure, data distribution information is received from a database administrator (DBA) defining a data distribution bitmap task. The data distribution bitmap task can be stopped or stopped based on the current system conditions. The data distribution information may include the operation flow, parameter settings, and execution method of the data distribution bitmap task.

[0055] Step 202: Determine whether the data distribution bitmap information is stored in the database and obtain the data distribution bitmap task.

[0056] In the embodiments of this disclosure, it is determined whether the current database stores data distribution bitmap information sent by the database administrator. If the data distribution information is stored in the database, the data distribution task in the database is retrieved based on the data distribution information, and a notification message is sent. The notification message is used to prompt the database administrator that the data distribution bitmap task is stored in the database, that is, to notify the database administrator that there is currently a data distribution bitmap task. If the data distribution bitmap information is not stored in the database, a data distribution bitmap task is created based on the data distribution information.

[0057] Step 203: Based on the data distribution bitmap task, connect to the control node of the database and obtain the second data distribution bitmap.

[0058] In embodiments of this disclosure, the database may be a Greenplum database, and the control node may be the master node of the Greenplum database.

[0059] In this disclosure, the task scheduling layer can connect to the Master node of the target Greenplum database based on the data distribution bitmap task to obtain the data distribution bitmap of the existing tables in the Greenplum database, i.e., the hash distribution map of each table in the multi-partition database, and perform periodic updates, i.e., updates according to a certain preset period, which is not limited in this disclosure. Here, a table in the database refers to a collection of data stored in the database, one of the basic objects of the database, composed of rows and columns, each row representing a record, and each column representing an attribute or field. The task scheduling layer is mainly responsible for the management of the data distribution bitmap task, including operations such as creating, canceling, updating, deleting, querying, and background scheduling of the data distribution bitmap task. A monitoring task is defined, and a reasonable data distribution bitmap is generated through network interface card data parsing, and the data is redistributed according to the bitmap information.

[0060] Step 204: Monitor the data flow between multiple servers and obtain the monitoring results.

[0061] In embodiments of this disclosure, a data bit layer can be created in the data distribution architecture to collect network traffic data moving between multiple servers, as well as the movement path of the moving network traffic data among the multiple servers. Simultaneously, the identifiers corresponding to the circulating data between the multiple servers are obtained and stored in a database.

[0062] like Figure 3As shown, network interface card (NIC) data collection, taking three servers as an example, involves monitoring the network interface card (NIC) traffic flow. This involves collecting the traffic volume between the NIC servers during SQL query execution to determine if traffic flow occurs on each NIC and then storing this data in the database. The primary information collected in this traffic flow data is the rowid, which is the unique identifier for each record in the database. This rowid can be used to find the table name, the corresponding field location, and the records that appear through the database system metadata of the control node (Master node).

[0063] Step 205: Based on the preset number of categories and clustering algorithm, perform central clustering on the circulation data to determine the distribution degree of the circulation data in multiple servers. The preset number of categories is the same as the number of multiple servers.

[0064] In the embodiments of this disclosure, the collected circulation data is analyzed using a clustering algorithm to determine the distribution of the circulation data across multiple servers.

[0065] In one optional embodiment of this disclosure, the collected circulation data is classified and estimated using the k-means clustering algorithm, which is an unsupervised learning algorithm. Unlike regression, Naive Bayes, and SVM, which all have class labels y (providing the classification of examples), k-means clustering does not provide y in the samples; it only uses features x. The purpose of clustering is to find the potential class y for each sample x and group samples x of the same class y together.

[0066] In clustering algorithms, training samples are { ,… }, each There is no category y. The K-means algorithm clusters samples into k clusters, where k is the preset number of categories. The specific algorithm description is as follows: Randomly select k cluster centroids as , ,…, .

[0067] Repeat the following process until convergence. { For each sample i, calculate the class it should belong to. arg

[0068] For each class j, recalculate the centroid of that class.

[0069] } Where K is the pre-given number of clusters, This represents the class that is closest to sample i among the k classes. The value is one from 1 to k. Centroid This represents a guess about the centroids of samples belonging to the same class. All data is clustered across k servers. First, k centroids (or k servers) are randomly selected as the k centroids. Then, for each data point, the distance to each of the k centroids is calculated, and the server with the closest distance is selected as the centroid. In this way, after the first step, each piece of data has an assigned server; the second step is to recalculate the centroid of each server. (Average the coordinates of all the data within). Repeat steps one and two until the centroid remains constant or changes very little. This clusters all circulating data onto different servers, forming a data distribution map.

[0070] Step 206: Determine the first data distribution bitmap based on the distribution degree, and distribute the data corresponding to the data set in the database to multiple servers according to the first data distribution bitmap.

[0071] In the embodiments of this disclosure, based on the distribution degree, circulating data is distributed to multiple servers corresponding to the maximum distribution degree of the circulating data, resulting in a third data distribution bitmap; based on the second data distribution bitmap, static data in the second data distribution bitmap is distributed to the third data distribution bitmap, resulting in a fourth data distribution bitmap; the computational cost of multiple nodes in the multiple servers is determined, and the computational cost data includes storage cost and processing cost; based on the computational cost, the second data distribution bitmap, and the fourth data distribution bitmap, a first data distribution bitmap is determined. If the computational cost of some or all nodes is less than a preset threshold, the data distribution positions of some or all nodes in the fourth data distribution bitmap are used as the data distribution positions of some or all nodes in the first data distribution bitmap; if the computational cost of some or all nodes is greater than the preset threshold, the data distribution positions of some or all nodes in the second data distribution bitmap are used as the data distribution positions of some or all nodes in the first data distribution bitmap.

[0072] In one optional embodiment of this disclosure, clustering calculations can determine the distribution degree of circulating data across different servers. This allows us to determine which server has the highest hit rate for a given piece of mobile network traffic data and distribute it there. By distributing the mobile network traffic data to servers based on the maximum distribution degree, a distribution pattern corresponding to each piece of mobile network traffic data can be formed. If the distribution degree is the same, the distribution will be consistent regardless of the server. This method forms an optimal distribution map of the circulating data, i.e., the third data distribution bitmap. These circulating network traffic data also correspond to different tables. The hash distribution map corresponding to the original tables, i.e., the second data distribution bitmap, shows that some hash-based data has not moved. By combining the second and third data distribution bitmaps, an optimal data distribution map, i.e., the fourth data distribution bitmap, is formed.

[0073] Taking three servers, Node1, Node2, and Node3, as an example, to obtain... Figure 4 The second data distribution bitmap is shown in Table 2. In Table 2, Tab 1.1 represents the first record of Tab 1, Tab 1.2 represents the second record of Tab 1, and so on. Tab 1.3, Tab 1.4, Tab 1.5, and Tab 1.6 are all records in Tab 1, i.e., the circulating data in Tab 1. Based on the distribution degree of each circulating data in Table 3, the following can be determined: Figure 5 The third data distribution bitmap shown. Combined with the second data distribution bitmap (the original Hash distribution bitmap), it forms the following: Figure 6 The fourth data distribution bitmap shown is the new data distribution bitmap. Among them, Figure 4 Tab1.2, Tab1.3, and Tab1.4 are all static data, so they do not need to be moved.

[0074]

[0075] Table 2 After forming the new data bitmap, i.e., obtaining the fourth data distribution bitmap, an uneven distribution of network traffic data may occur. Although the fourth data distribution bitmap reduces a significant amount of network traffic overhead at the network layer, it is necessary to determine the computational cost of each server, i.e., the computational capacity cost and CPU cost. This is done by judging the relationship between the computational cost and preset thresholds to avoid excessive costs. The preset thresholds can include a first preset threshold and a second preset threshold. The first preset threshold is used to determine storage costs, and the second preset threshold is used to determine processing costs.

[0076] When calculating capacity cost, first obtain the total capacity Mcap of each node in the server using the capacity viewing command (df). Then, obtain the capacity Ncap used by each node using the fourth data bitmap and the size of the network traffic data. The capacity cost, denoted as Ccap, can be determined using Formula 1.

[0077] Formula 1 An alarm is triggered when the cost of capacity is greater than or equal to 50%, and the optimal bitmap optimization is terminated when it is greater than or equal to 80%. Here, 50% and 80% are both first preset thresholds, which are capacity utilization thresholds. The first preset thresholds can be adjusted by parameters, but are not limited in this embodiment.

[0078] When calculating CPU cost, the current system load is first required. The higher the load, the greater the cost. The load is estimated using Formula 2: Formula 2 Formula 2 normalizes the current CPU load, where Vcpu represents the current system load, obtained using the load lookup command (uptimie command), φmin represents the minimum system load, and Kmax represents the maximum system load. After obtaining the CPU load, Formula 3 is needed to estimate the cost of CPU processing records. Formula 3 Formula 3 is normalized, where α is the number of records in the current node. βmin represents the minimum number of records in the node, and βmax represents the maximum number of records in the node. Since βmin can be 0, the right half of the equation can be derived.

[0079] After obtaining the load and latency values, the CPU cost can be calculated using Formula 4: Ccpu = 100+ Formula 4 Since the load has a significant impact on CPU cost, the load is multiplied by 100. As can be seen from the formula above, the higher the CPU load, the higher the CPU cost; similarly, the more records to be processed, the higher the CPU cost.

[0080] To improve query efficiency by reducing data broadcasting or inter-data transfer during SQL queries, it's inevitable that some nodes will experience excessively high utilization. Figure 6As shown, Node2 stores more network traffic data than Node3. During automatic optimization, there's a possibility of a single node's capacity becoming excessive. To ensure normal system operation, the process will terminate when the cost exceeds 50%, or alternatively, an alarm can be triggered at 50% and the system can be stopped at 80%, using dynamic parameter adjustments for control. Here, 50% and 80% are both second preset thresholds. It's evident that the first preset threshold can be the same as the second preset threshold, meaning control can be achieved through a single preset threshold.

[0081] Understandably, by determining the computational cost of each node in each server under the fourth data distribution bitmap, nodes with computational costs less than a preset threshold are distributed according to the network traffic data in the fourth data distribution bitmap. That is, the locations of nodes with computational costs less than the preset threshold in the fourth data distribution bitmap are used as the locations of those nodes in the first data distribution bitmap. If the computational cost is greater than or equal to the preset threshold, the current node still distributes network traffic data according to the first data bitmap; that is, nodes with computational costs greater than or equal to the preset threshold are used as the locations of those nodes in the first data distribution bitmap. This is how the optimal first data distribution bitmap is obtained.

[0082] Furthermore, after evaluating system resources through calculations, it is necessary to determine whether to implement the proposed solution. This is achieved through the solution implementation layer in the data distribution architecture. Once the data bitmap is confirmed, the solution implementation layer is responsible for exporting the data that needs to be shifted, moving the table data according to the first data distribution bitmap, persisting it to disk, executing the data movement, and recording the implementation log. This solution implementation requires user confirmation that the data distribution solution is correct before it can be executed, and the implementation log will be recorded.

[0083] In summary, the method provided in this disclosure, by scanning database and network traffic data and analyzing sample data, yields a better data distribution while meeting computational cost requirements, without relying on the experience of maintenance personnel. This improves query efficiency and addresses the issue of decreased query efficiency caused by the increasing scale of distributed databases. Furthermore, a scheduled task can be configured to scan newly added tables in the database. When data distribution optimization is needed, the task can be invoked without requiring a full database scan, thus minimizing time constraints and meeting time requirements.

[0084] based on Figure 2 , Figure 3 The illustrated embodiments, such as Figure 7 , Figure 8 As shown, this disclosure provides an architecture diagram of a specific data distribution method, as well as a flowchart of a specific data distribution method.

[0085] In this embodiment, the database DBA defines the information for creating a data distribution bitmap task. The task can be stopped or closed based on the current system status. If the task already exists, a prompt is displayed indicating that a data distribution task already exists, and the task is directly invoked; otherwise, a data distribution task is created. The system connects to the Master node of the target Greenplum database to obtain the data distribution bitmap of the existing tables in the database, i.e., the hash distribution map of each table in the multi-partition database, and updates it periodically. After collection, the program starts network interface card (NIC) traffic collection, uniformly collecting the data flow between partitions initiated by SQL, and saving it to the database. The collected NIC flow data is parsed to determine the original tables that need data distribution optimization. Based on the newly formed data bitmap, the capacity and CPU consumption are estimated; if it exceeds the system's preset threshold, the process is terminated. With the tables whose data bitmaps have been determined and system resources are sufficient, the bitmap information is distributed to each segment node (data node) through the Master node (control node), the data of each node is moved to the new data node, persisted to disk, and the original table data is cleaned up. The task is registered and feedback is sent to the task allocation database.

[0086] In summary, the method proposed in this disclosure utilizes a self-developed three-layer architecture device, a standard mechanism for task scheduling, optimal data distribution analysis and confirmation, and optimal distribution implementation based on data bitmaps. Furthermore, by automatically collecting network interface card traffic data from each segment, and optimizing the data distribution based on aggregation algorithms and data location maps, it reduces data broadcasting or redistribution under the condition of data hash distribution. It does not rely on relevant documents or manual operations, and eliminates the dependence on experience in selecting distribution keys, thus greatly improving the real-time performance and efficiency of processing.

[0087] Figure 9 This is a schematic diagram of the structure of a data distribution device 900 provided in an embodiment of this disclosure. Figure 9 As shown, the data distribution device includes: The acquisition unit 910 is used to monitor the data flow between multiple servers and acquire monitoring results. The monitoring results include the flow path of the data flow between multiple servers, and the data flow includes multiple mobile network traffic data. Unit 920 is used to determine the distribution of circulating data across multiple servers based on monitoring results and clustering algorithms. Distribution unit 930 is used to adjust the second data distribution bitmap to the first data distribution bitmap according to the distribution degree, and distribute the data corresponding to the data set in the database to multiple servers according to the first data distribution bitmap. The second data distribution bitmap includes the data distribution of the data set stored in the database.

[0088] In some embodiments, the apparatus 900 further includes: a receiving unit, configured to receive data distribution information sent by a database administrator; determine whether the data distribution bitmap information is stored in a database, obtain a data distribution bitmap task; and, based on the data distribution bitmap task, connect to the control node of the database to obtain a second data distribution bitmap.

[0089] In some embodiments, the receiving unit is configured to: if the data distribution information is stored in the database, retrieve the data distribution task in the database based on the data distribution information and send a notification message, the notification message being used to prompt the database administrator that the data distribution bitmap task is stored in the database; if the data distribution bitmap information is not stored in the database, create a data distribution bitmap task based on the data distribution information.

[0090] In some embodiments, the acquisition unit 910 is used to: acquire the identifier corresponding to the data flowing between multiple servers, and store the identifier in a database.

[0091] In some embodiments, the determining unit 920 is configured to: perform central clustering on the circulating data based on a preset number of categories and a clustering algorithm, and determine the distribution degree of the circulating data in multiple servers, wherein the preset number of categories is the same as the number of multiple servers.

[0092] In some embodiments, the distribution unit 930 is configured to: distribute circulating data to multiple servers corresponding to the maximum distribution degree of the circulating data according to the distribution degree, to obtain a third data distribution bitmap; distribute static data in the second data distribution bitmap to the third data distribution bitmap according to the second data distribution bitmap, to obtain a fourth data distribution bitmap; determine the computing cost of multiple nodes in the multiple servers respectively, the computing cost data including storage cost and processing cost; and determine a first data distribution bitmap based on the computing cost, the second data distribution bitmap, and the fourth data distribution bitmap.

[0093] In some embodiments, the distribution unit 930 is configured to: if the computational cost of some or all nodes is less than a preset threshold, use the data distribution positions of some or all nodes in the fourth data distribution bitmap as the data distribution positions of some or all nodes in the first data distribution bitmap; if the computational cost of some or all nodes is greater than the preset threshold, use the data distribution positions of some or all nodes in the second data distribution bitmap as the data distribution positions of some or all nodes in the first data distribution bitmap.

[0094] In summary, by using a data distribution device to monitor the data flow between multiple servers and obtain monitoring results, including the flow path of the data across the servers and the data itself including multiple mobile network traffic data, the distribution degree of the data across the servers is determined based on the monitoring results and clustering algorithms. Based on the distribution degree, a first data distribution bitmap is determined, and the data corresponding to the data sets in the database is distributed to the multiple servers according to the first data distribution bitmap. The first data distribution bitmap includes the data distribution of the data sets in the database. By monitoring the traffic data between the servers and optimizing the data distribution using aggregation algorithms, the broadcasting or redistribution of data is reduced. This approach does not rely on related documents or manual operations, eliminates the dependence on selecting distribution keys based on human experience, and improves the real-time performance and efficiency of data processing.

[0095] Corresponding to the methods provided in the above embodiments, this disclosure also provides a data distribution device. Since the device provided in this disclosure corresponds to the methods provided in the above embodiments, the implementation of the methods is also applicable to the device provided in this embodiment, and will not be described in detail in this embodiment.

[0096] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the electronic device may include a hardware structure and software modules, and may implement the above functions in the form of a hardware structure, software modules, or a hardware structure plus software modules. One of the above functions may be executed in the form of a hardware structure, software modules, or a hardware structure plus software modules.

[0097] Figure 10 This is a block diagram illustrating an electronic device 1000 for implementing the above-described data distribution method according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, computer, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0098] Reference Figure 10 The electronic device 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power supply component 1006, a multimedia component 1008, an audio component 1010, an input / output (I / O) interface 1012, a sensor component 1014, and a communication component 1016.

[0099] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0100] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of such data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0101] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0102] Multimedia component 1008 includes a screen that provides an output interface between electronic device 1000 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When electronic device 1000 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0103] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0104] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0105] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 may detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or a component of electronic device 1000, the presence or absence of user contact with electronic device 1000, the orientation or acceleration / deceleration of electronic device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0106] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0107] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0108] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0109] Embodiments of this disclosure also propose a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the data distribution method described in the above embodiments of this disclosure.

[0110] Embodiments of this disclosure also provide a computer program product, including a computer program that is executed by a processor using the data distribution method described in the above embodiments of this disclosure.

[0111] Embodiments of this disclosure also propose a chip including one or more interface circuits and one or more processors; the interface circuits are used to receive signals from the memory of an electronic device and send signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processor, cause the electronic device to perform the data distribution method described in the above embodiments of this disclosure.

[0112] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0113] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0114] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0115] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0116] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0117] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0118] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0119] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A data distribution method, characterized in that, The method includes: Receive data distribution information sent by the database administrator; Determine whether the data distribution information is stored in the database, and obtain the data distribution bitmap task; Based on the data distribution bitmap task, connect to the control node of the database to obtain a second data distribution bitmap, the second data distribution bitmap including the data distribution corresponding to the data set stored in the database; Monitor the data flow between multiple servers and obtain monitoring results. The monitoring results include the flow path of the data flow among the multiple servers, and the data flow includes multiple mobile network traffic data. Based on the monitoring results and clustering algorithm, the distribution of the circulating data among the multiple servers is determined; Based on the distribution degree, the circulating data is distributed to the multiple servers corresponding to the maximum distribution degree of the circulating data to obtain a third data distribution bitmap; based on the second data distribution bitmap, the static data in the second data distribution bitmap is distributed to the third data distribution bitmap to obtain a fourth data distribution bitmap; the computing cost of multiple nodes in the multiple servers is determined respectively, the computing cost including storage cost and processing cost; based on the computing cost, the second data distribution bitmap, and the fourth data distribution bitmap, a first data distribution bitmap is determined, and the data corresponding to the data set in the database is distributed to the multiple servers according to the first data distribution bitmap, the first data distribution bitmap including the data distribution of the data set in the database.

2. The method according to claim 1, characterized in that, The task of determining whether the data distribution information is stored in the database and obtaining the data distribution bitmap includes: If the data distribution information is stored in a database, the data distribution task in the database is retrieved based on the data distribution information, and a notification message is sent. The notification message is used to remind the database administrator that the data distribution bitmap task is stored in the database. If the data distribution information is not stored in the database, the data distribution bitmap task is created based on the data distribution information.

3. The method according to claim 1, characterized in that, The monitoring of data flow between multiple servers and the acquisition of monitoring results include: Obtain the identifier corresponding to the circulating data between the multiple servers, and store the identifier in the database.

4. The method according to claim 1, characterized in that, Determining the distribution of the circulating data across the multiple servers based on the monitoring results and clustering algorithm includes: Based on the preset number of categories and the clustering algorithm, the circulation data is centrally clustered to determine the distribution degree of the circulation data among the multiple servers. The preset number of categories is the same as the number of multiple servers.

5. The method according to claim 1, characterized in that, The process of determining the first data distribution bitmap based on the computational cost, the second data distribution bitmap, and the fourth data distribution bitmap includes: If the computational cost of some or all nodes is less than a preset threshold, the data distribution position of some or all nodes in the fourth data distribution bitmap is taken as the data distribution position of some or all nodes in the first data distribution bitmap. If the computational cost of some or all of the nodes exceeds a preset threshold, the data distribution position of some or all of the nodes in the second data distribution bitmap is taken as the data distribution position of some or all of the nodes in the first data distribution bitmap.

6. A data distribution device, characterized in that, The device includes: A receiving unit is configured to receive data distribution information sent by a database administrator; determine whether the data distribution information is stored in a database; obtain a data distribution bitmap task; and, based on the data distribution bitmap task, connect to the control node of the database to obtain a second data distribution bitmap, wherein the second data distribution bitmap includes the data distribution corresponding to the data set stored in the database. An acquisition unit is used to monitor the flow of data between multiple servers and acquire monitoring results. The monitoring results include the flow path of the flow of data among the multiple servers, and the flow of data includes multiple mobile network traffic data. The determining unit is used to determine the distribution degree of the circulating data among the multiple servers based on the monitoring results and clustering algorithms. A distribution unit is configured to distribute the circulating data to the plurality of servers corresponding to the maximum distribution degree of the circulating data according to the distribution degree, thereby obtaining a third data distribution bitmap; distribute the static data in the second data distribution bitmap to the third data distribution bitmap according to the second data distribution bitmap, thereby obtaining a fourth data distribution bitmap; determine the computing cost of the plurality of nodes in the plurality of servers respectively, wherein the computing cost includes storage cost and processing cost; determine a first data distribution bitmap based on the computing cost, the second data distribution bitmap, and the fourth data distribution bitmap, and distribute the data corresponding to the data set in the database to the plurality of servers according to the first data distribution bitmap, wherein the first data distribution bitmap includes the data distribution of the data set in the database.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

9. A chip, characterized in that, The device includes one or more interface circuits and one or more processors; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, the signals including computer instructions stored in the memory, which, when executed by the processors, cause the electronic device to perform the method of any one of claims 1-5.

Citation Information

Patent Citations

  • Data placement control for distributed computing environment

    CN107710193A

  • Data redistribution method and apparatus, and database cluster

    CN108319623A