Data distribution method and apparatus, electronic device, and readable storage medium

Abstract

The application discloses a data distribution method and device, electronic equipment and readable storage medium, and belongs to the technical field of information. The method comprises the following steps: clustering data to obtain n data shards and corresponding n center points; obtaining the distance between each center point and each center point in k center points to obtain k*k distances, wherein the k center points comprise: a center point corresponding to an unassigned data shard, a center point corresponding to a server satisfying a first condition, and the first condition comprises that the data shard is assigned, and the number of assigned data shards is less than a first threshold; determining a first center point and a second center point corresponding to the maximum distance; and assigning a first data shard and a second data shard to the same server, wherein the first data shard is a data shard corresponding to the first center point, and the second data shard is a data shard corresponding to the second center point.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and specifically relates to a data allocation method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the development of information technology, the amount of data has exploded. Distributed data retrieval systems distribute and store data on multiple nodes, and retrieve data from multiple nodes during retrieval.

[0003] Existing distributed data retrieval systems randomly distribute and store data across multiple nodes, resulting in low retrieval efficiency because similar data needs to be searched across all nodes during data retrieval. Summary of the Invention

[0004] The purpose of this application is to provide a data allocation method, apparatus, electronic device, and readable storage medium that can solve the problem of low data retrieval efficiency caused by randomly allocating and storing data on multiple nodes.

[0005] In a first aspect, embodiments of this application provide a data allocation method, the method comprising:

[0006] The data is clustered to obtain n data segments and n center points corresponding to the n data segments, where n is an integer greater than 1;

[0007] Obtain the distances between each of the k center points and each of the other center points, resulting in k×k distances. The k center points include: center points corresponding to unallocated data shards among the n data shards, and center points corresponding to servers among the m servers that satisfy a first condition. The first condition includes that data shards are allocated, and the number of allocated data shards is less than a first threshold; m is an integer greater than 1 and less than n; k is an integer greater than 1.

[0008] Determine the first and second center points corresponding to the maximum distance among the k×k distances;

[0009] The first data shard and the second data shard are allocated to the same server among the m servers, wherein the first data shard is the data shard corresponding to the first center point, and the second data shard is the data shard corresponding to the second center point.

[0010] Secondly, embodiments of this application provide a data allocation apparatus, the apparatus comprising:

[0011] The clustering module is used to cluster the data to obtain n data fragments and n centroids corresponding to the n data fragments, where n is an integer greater than 1;

[0012] The first acquisition module is used to acquire the distances between each of the k center points and each other, resulting in k×k distances. The k center points include: center points corresponding to unallocated data shards among the n data shards, and center points corresponding to servers among the m servers that satisfy a first condition. The first condition includes that data shards are allocated, and the number of allocated data shards is less than a first threshold; m is an integer greater than 1 and less than n; k is an integer greater than 1.

[0013] The first determining module is used to determine the first center point and the second center point corresponding to the maximum distance among the k×k distances;

[0014] The first allocation module is used to allocate the first data shard and the second data shard to the same server among the m servers, wherein the first data shard is the data shard corresponding to the first center point, and the second data shard is the data shard corresponding to the second center point.

[0015] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0016] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0018] In a sixth aspect, embodiments of this application provide a program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0019] In this embodiment, the data can first be clustered to group similar data into the same data shard, resulting in n data shards. Then, the distances between each of the k centroids and each of the other centroids can be obtained, resulting in k×k distances. The k centroids include: the centroid corresponding to a currently unassigned data shard, and the centroid corresponding to a server among the m servers that has been assigned data shards, where the number of assigned data shards is less than a first threshold. The first centroid and the second centroid corresponding to the largest distance among the k×k distances are determined. The data shard corresponding to the first centroid and the centroid corresponding to the second centroid are then assigned to the same server among the m servers. This ensures that data shards with larger distances (i.e., not very close) are assigned to the same server, and vice versa, data shards with smaller distances (i.e., very close) are assigned to different servers. In this way, on the one hand, data retrieval traffic can be evenly distributed across different servers; on the other hand, since each data shard corresponds to a central point, when retrieving data from the first data, the retrieval range (server and data shards within the server) can be quickly located by calculating the distance between the first data and the central point corresponding to each data shard, thereby improving data retrieval efficiency. Attached Figure Description

[0020] Figure 1 This is one of the flowcharts of the data allocation method provided in the embodiments of this application;

[0021] Figure 2 This is a schematic diagram of data fragmentation provided in an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of data search provided in an embodiment of this application;

[0023] Figure 4 This is the second flowchart of the data allocation method provided in the embodiments of this application;

[0024] Figure 5 This is a flowchart of the data search provided in the embodiments of this application;

[0025] Figure 6 This is a structural diagram of the data distribution device provided in the embodiments of this application;

[0026] Figure 7 This is one of the structural diagrams of the electronic device provided in the embodiments of this application;

[0027] Figure 8 This is the second structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0029] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0030] The data allocation method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0031] Figure 1 This is one of the flowcharts of the data allocation method provided in the embodiments of this application. The data allocation method may include:

[0032] Step 101: Cluster the data to obtain n data segments and n center points corresponding to the n data segments.

[0033] In this embodiment, data can be represented by vectors, meaning that a data point can be represented as a vector. Specifically, the data can be text, images, videos, relationship networks, or chemical structures, etc.

[0034] In step 101, the data can be clustered to group similar data into the same data shard, resulting in n data shards. Here, n is an integer greater than 1, and the specific value of n can be preset according to actual needs; this embodiment does not limit this. The clustering method can be K-Means clustering, but is not limited to it.

[0035] In practice, if it is determined that the data needs to be clustered into n classes, n-1 distance thresholds can be set to obtain n distance intervals. The distance between each data point and all other data points is obtained, and data points within the same distance interval are clustered into one class, resulting in a data partition (also called a cluster).

[0036] It should be noted that, in the embodiments of this application, the distance between A and B represents the relative distance between A and B. The larger the distance between A and B, the farther apart A and B are, and the smaller the similarity between A and B; the smaller the distance between A and B, the closer A and B are, and the greater the similarity between A and B.

[0037] The distance between A and B can be determined by calculating the Euclidean distance, cosine distance (COS distance), or inner product distance between A and B. It's important to note that the Euclidean distance between A and B is positively correlated with the actual distance between them; that is, the larger the Euclidean distance, the greater the distance between A and B, and vice versa. Conversely, the cosine distance between A and B is negatively correlated with the actual distance between them; that is, the larger the cosine distance, the smaller the distance between A and B, and vice versa.

[0038] In this embodiment, each data shard corresponds to a center point (which may be called a data shard center point or cluster center point). The center point is a vector, and its dimensions are the same as the dimensions of the vectors in the data shard. The values ​​of each dimension of the center point corresponding to the data shard can be the average of the corresponding dimensions of the vectors in the data shard.

[0039] Step 101 yields n data partitions [C1, C2, C3, ..., C n ], and n data shards corresponding to n center points [c1,c2,c3,…,c n For easier understanding, please refer to [link / reference]. Figure 2 .exist Figure 2 In this context, numbers represent data fragment identifiers (also known as cluster numbers), and small circles represent vectors.

[0040] After obtaining n data shards, at least one allocation operation can be performed to distribute the n data shards to m servers, where n and m are both integers greater than 1, and n is greater than m.

[0041] Understandably, when n is less than or equal to m, each of the n data shards can be allocated to different servers, meaning each data shard occupies one server.

[0042] When n is greater than m, the total number of data shards is greater than the total number of servers, and each server needs to store at least one data shard. To ensure that the number of data shards stored on each server is balanced and that data retrieval traffic is evenly distributed across each server, in this embodiment of the application, the allocation operation may include steps 102 to 104.

[0043] Step 102: Obtain the distance between each of the k center points and each of the other center points to obtain k×k distances. The k center points include: the center points corresponding to the unassigned data shards in the n data shards, and the center points corresponding to the servers that meet the first condition among the m servers.

[0044] The first condition may include: data shards have been allocated, and the number of allocated data shards is less than a first threshold. k is an integer greater than 1.

[0045] In this embodiment of the application, as described above, each data shard corresponds to a central point. Additionally, each server that has been allocated data shards also corresponds to a central point (which can be referred to as a hybrid central point).

[0046] The hybrid centroid corresponding to a server is the center point of all cluster centroids corresponding to all data shards stored on the server. For example, if server 1 stores 10 data shards, then the hybrid centroid of server 1 is the center point of the 10 cluster centroids corresponding to the 10 data shards.

[0047] The mixed center point can be obtained by weighted averaging of the center points. For example, the mixed center point cc of center points a and b can be calculated by formula (1):

[0048]

[0049] Where d represents the dimension of the center point.

[0050] Additionally, to prevent the allocated data shards from exceeding the server's memory capacity, an upper limit, or first threshold, can be set for the server. Once the number of data shards allocated to a server reaches the first threshold, further allocation of data shards to that server can be stopped.

[0051] The first threshold is the maximum number of data shards that the server is allowed to allocate, and can be set according to actual needs. This application embodiment does not limit this. In an optional implementation, in order to ensure that the data shards stored on m servers are balanced, the first threshold can be determined based on n and m. Specifically, if n is divisible by m, the first threshold can be n / m; if n is not divisible by m, the first threshold can be floor(n / m)+1, where floor represents the floor function of n / m.

[0052] Therefore, the allocation object of the allocation operation can include at least one of the following:

[0053] Unallocated data fragments;

[0054] The server stores data shards that meet the first condition, wherein the first condition includes that data shards are allocated and the number of allocated data shards is less than a first threshold.

[0055] For unallocated data shards, the allocation operation can be used to assign them to servers. For data shards stored on servers, the allocation operation can be used to transfer them to another server.

[0056] Considering that the data allocation method aims to assign data fragments that are far apart (i.e., not close) to the same server, and data fragments that are close together (i.e., not close) to different servers, the data retrieval traffic can be evenly distributed across different servers. Therefore, in each allocation operation, the distance between the unallocated data fragment and the center point corresponding to the server that meets the first condition can be obtained, and the first center point and the second center point corresponding to the maximum distance can be determined. The data fragments corresponding to the first center point and the second center point can then be allocated to the same server among the m servers.

[0057] Understandably, the unallocated data shards, the servers satisfying the first condition, and / or the data shards stored on servers satisfying the first condition may change for different allocation operations. Therefore, the k center points in step a may be different for different allocation operations. For example, in the first allocation operation, all m servers are empty and have no data shards allocated. In this allocation operation, the k center points are the n cluster center points corresponding to the n data shards. In the second allocation operation, one of the m servers is allocated 2 data shards. Assuming the first threshold is greater than 2, in this allocation operation, the k center points include the n-2 cluster center points corresponding to the remaining unallocated n-2 data shards, and the mixed center point corresponding to the server storing 2 data shards.

[0058] Step 103: Determine the first center point and the second center point corresponding to the maximum distance among the k×k distances.

[0059] In step 103, the first center point may be a cluster center point or a mixed center point; the second center point may be a cluster center point or a mixed center point. The first center point and the second center point may be of the same type (such as a cluster center point or a mixed center point) or different types.

[0060] Specifically, it can include the following possibilities:

[0061] Case 1: Both the first and second center points are cluster center points;

[0062] Case 2: One of the first and second center points is a cluster center point, and the other is a mixed center point;

[0063] Case 3: Both the first and second center points are mixed center points.

[0064] Step 104: Assign the first data shard and the second data shard to the same server among the m servers, wherein the first data shard is the data shard corresponding to the first center point, and the second data shard is the data shard corresponding to the second center point.

[0065] In step 104, the number of data shards corresponding to a cluster center point is 1. For a mixed center point, the corresponding data shard is all the data shards stored on its corresponding server, and the number of its corresponding data shards is greater than or equal to 1.

[0066] The data allocation method in this embodiment first clusters the data, placing similar data into the same data shard to obtain n data shards. Then, it obtains the distances between each of the k center points and each of the other k center points, resulting in k×k distances. The k center points include: the center point corresponding to a currently unassigned data shard, and the center point corresponding to a server among m servers that has been assigned data shards, where the number of assigned data shards is less than a first threshold. It then determines the first center point and the second center point corresponding to the largest distance among the k×k distances. Finally, it assigns the data shard corresponding to the first center point and the center point corresponding to the second center point to the same server among the m servers. This ensures that data shards with larger distances (i.e., not very close) are assigned to the same server, and vice versa, data shards with smaller distances (i.e., very close) are assigned to different servers. In this way, on the one hand, data retrieval traffic can be evenly distributed across different servers; on the other hand, since each data shard corresponds to a central point, when retrieving data from the first data, the retrieval range (server and data shards within the server) can be quickly located by calculating the distance between the first data and the central point corresponding to each data shard, thereby improving data retrieval efficiency.

[0067] As can be seen from the foregoing, there are multiple possible forms of the first and second center points. The specific implementation of step c differs for different forms of the first and second center points, as explained below.

[0068] Regarding the aforementioned situation 1

[0069] Both the first and second center points are cluster center points. It can be determined whether any of the m servers lacks a data shard, and based on this determination, the specific implementation of step c is determined.

[0070] In some embodiments, the first center point and the second center point are both center points corresponding to data shards; among the m servers, there are servers that have not been allocated data shards;

[0071] The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes:

[0072] The first data shard and the second data shard are allocated to the first server;

[0073] The first server is any one of the m servers that has not been allocated a data shard.

[0074] In this embodiment, since there are still servers among the m servers that have not been allocated data shards, the first data shard and the second data shard can be allocated to any of these servers to make the data shards allocated to each server balanced.

[0075] Of course, in some implementations, the m servers can be sorted, and then the first server that does not store any data shards after sorting can be determined as the first server.

[0076] In some embodiments, the first center point and the second center point are both center points corresponding to data shards; each of the m servers is allocated a data shard;

[0077] The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes:

[0078] Calculate the distance between the target center point and the corresponding center point of each server to obtain m distances, where the target center point is the center point of the first center point and the second center point; the second condition is: data shards are allocated, and the number of stored data shards plus 2 is less than or equal to the first threshold;

[0079] The first data shard and the second data shard are allocated to the second server;

[0080] The second server is the server corresponding to the largest distance among the m distances.

[0081] In this embodiment, since there are no servers among the m servers that have not been allocated data shards, a new hybrid center point, i.e., the target center point, can be determined based on the first center point and the second center point.

[0082] From the m servers, select the servers that can still store at least 2 more data shards, i.e., the servers that meet the second condition.

[0083] Next, the distances between the target center point and the center points of each server that satisfy the second condition are calculated to obtain m distances. The first data shard and the second data shard are then assigned to the server with the largest distance among the m distances.

[0084] In this way, data fragments that are far apart (i.e., not close) can be assigned to the same server, while data fragments that are close together can be assigned to different servers, thus balancing data retrieval traffic across different servers.

[0085] Regarding the aforementioned situation 2

[0086] In some embodiments, the first center point is the center point corresponding to the data shard, and the second center point is the center point corresponding to the server;

[0087] The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes:

[0088] The first data shard is allocated to the third server;

[0089] The third server is the server corresponding to the second central point.

[0090] In this embodiment, the example is given where the first center point is a cluster center point and the second center point is a hybrid center point. However, it is understood that when the first center point is a hybrid center point and the second center point is a cluster center point, the second data shard can be directly assigned to the server corresponding to the first center point, so that the first data shard and the second data shard are allocated to the same server among the m servers.

[0091] In cases where one of the first and second central points is a cluster-type central point and the other is a hybrid central point, the data shards corresponding to the cluster-type central point can be directly allocated to the servers corresponding to the hybrid central point. This simplifies the allocation of data shards.

[0092] Regarding the aforementioned situation 3

[0093] In some embodiments, the first center point is the center point corresponding to the fourth server, and the second center point is the center point corresponding to the fifth server;

[0094] The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes:

[0095] If the first number is less than the first threshold, the first data fragment is allocated to the fifth server, or the second data fragment is allocated to the fourth server;

[0096] Wherein, the first quantity is the sum of the number of data fragments included in the first data fragment and the second data fragment.

[0097] In scenario 3, both the first and second central points are hybrid central points. The first and second data slices can be transferred from two servers to the same server. However, considering that the server has a data slice storage limit, before determining the allocation strategy for the first and second data slices, it can be determined whether the sum of the number of data slices included in the first and second data slices is less than the first threshold.

[0098] In this case, the first and second data slices can be transferred from the two servers to either of the two servers.

[0099] In some embodiments, the method further includes:

[0100] If the first quantity is greater than or equal to the first threshold, determine the third center point and the fourth center point corresponding to the second largest distance among the k×k distances;

[0101] The third data shard and the fourth data shard are allocated to the same server among the m servers, wherein the third data shard is the data shard corresponding to the third center point, and the fourth data shard is the data shard corresponding to the fourth center point.

[0102] If not, the allocation of the first and second data shards can be abandoned, and the third and fourth center points corresponding to the second largest distance in step a can be determined. The same allocation method as for the first and second data shards can be used to allocate the third and fourth data shards, and so on, until an allocation operation is successfully completed.

[0103] This ensures that the data shards stored on the server do not exceed the upper limit, and that data shards that are far apart (i.e., not close to each other) are assigned to the same server, while data shards that are close together are assigned to different servers, so that data retrieval traffic can be evenly distributed across different servers.

[0104] In this application embodiment, to simplify the allocation operation, in some embodiments, before obtaining the distance between each of the k center points and each of the other center points to obtain k×k distances, the method further includes:

[0105] Get the distance table and statistics array;

[0106] The statistical array includes m values ​​that correspond one-to-one with the m servers, where the value corresponding to a server represents the number of data shards allocated to that server.

[0107] The distance table includes row and column labels corresponding to q center points, and q rows and q columns of values; the q center points include: the n data shards, and the center points corresponding to the servers among the m servers that are allocated data shards; the value in the i-th row and j-th column represents the distance between the center point corresponding to the i-th row label and the center point corresponding to the j-th row label; q is an integer greater than n, and i and j are both positive integers less than or equal to q;

[0108] In the distance table, the values ​​corresponding to the fifth center point are all set to invalid values; the fifth center point includes: the center point corresponding to the data shards that have been allocated among the n data shards, and the center point corresponding to the server whose value in the statistical array is greater than or equal to the first threshold;

[0109] The step of obtaining the distance between each of the k center points and each of the other center points, resulting in k×k distances, includes:

[0110] Query the valid values ​​in the distance table to obtain k×k distances.

[0111] In this embodiment, a distance table and a statistical array can be used to perform the allocation operation. Specifically, the distance table can be updated using the statistical array; and k×k distances can be retrieved from the distance table. This simplifies the allocation operation.

[0112] The following explains the distance table and statistical array.

[0113] Neither the distance table nor the statistics array is fixed. The size of the distance table (the length and width of the table) will continuously grow as the assignment operation is executed, and the values ​​in the statistics array will also continuously change, but the length of the statistics array will not change.

[0114] The length of the statistics array is the number of servers, and the i-th value in the statistics array represents the number of data shards allocated to the i-th server. Initially, the array is [0,0,0,…,0]. The maximum value in the array is the first threshold.

[0115] In one optional implementation, the statistical data may specifically include m tags corresponding one-to-one with m servers, and m values ​​corresponding one-to-one with m servers. The tags corresponding to the servers can be understood as server identifiers, and the values ​​corresponding to the servers represent the number of data shards stored on the servers.

[0116] In the distance table, the row and column labels are the same. The label corresponding to the cluster centroid can be understood as the identifier of the data shard, and the label corresponding to the mixed centroid can be understood as the identifier of the server corresponding to the mixed centroid. In addition, in the distance table, each row label and column label corresponds to a value, which represents the distance between the centroid corresponding to the row label and the column label. Specifically, it can be a Euclidean distance value or a cosine distance value.

[0117] The Euclidean distance between center point a and center point b can be calculated using formula (2):

[0118]

[0119] The cosine distance between center point a and center point b can be calculated using formula (3):

[0120]

[0121] In the initial state, the distance table includes n row labels and n column labels corresponding to n cluster centroids, as well as n×n values, representing the distance between each cluster centroid and the other cluster centroids.

[0122] As the allocation operation is executed, data shards are distributed to the server, resulting in new centroids, or hybrid centroids. New row and column labels, along with their corresponding values, are added to the distance table.

[0123] Additionally, to ensure accurate determination of the maximum distance between data shards requiring allocation from the distance table, the values ​​corresponding to data shards that do not require allocation need to be set to invalid values. Specifically, data shards that do not require allocation can include: allocated data shards among the n data shards, and data shards stored on servers whose number of data shards stored on the m servers reaches a first threshold.

[0124] The data sharding operation of this embodiment will be described below.

[0125] 1. During the first allocation operation, find the two points farthest from the table and use c. i and c j This indicates that the two clusters represented by these two points are assigned to the server with the fewest shards. Initially, all servers have no shards, so the clusters are directly assigned to the first server. The statistics array then becomes [2,0,0,0,…,0]. i and c j Combined into a new mixing center point cc1, which is c i and c j The midpoint.

[0126] 2. Calculate the distances of the new mixing center point cc1 to all other points, and update the distance table by adding cc1. Since c... i and c j It has already been assigned, c i and c j The corresponding values ​​are set to invalid in the distance table, that is, the i-th row, j-th row, i-th column, and j-th column are all set to invalid, and these invalid values ​​are not considered in subsequent calculations.

[0127] 3. Check the servers assigned to the new hybrid center point cc1, and check if the value at the corresponding position in the array has reached the limit. If it has, then the row and column of cc1 in the distance table need to be set to invalid, because it will definitely not be merged with other points onto a server in the future.

[0128] 4. Continue allocating the remaining clusters. Find the two points with the greatest distance among the values ​​in the distance table that have not been invalidated. At this point, we need to discuss five cases:

[0129] 1) If both points are cluster center points (as opposed to mixed center points), and there are currently empty servers without allocated shards, then proceed as in step 1: assign both points to an empty server, and then update the relevant distance table and statistics array.

[0130] 2) If both points are cluster center points, but data has already been allocated to all servers, and we need to assign these two clusters to the same server, we first find a list of servers that can accommodate two or more clusters. This is calculated by adding two to the values ​​in the count array, and finding those values ​​that are still less than or equal to the first threshold. If there is only one such server, we directly assign the data to that server. If there are multiple such servers, we still need to find the server with the furthest distance. These two cluster center points form a mixed center point, and we calculate the distance between this mixed center point and the server's mixed center point.

[0131] 3) If one is a cluster centroid and the other is a hybrid centroid, then directly assign the cluster centroid to the server corresponding to the hybrid centroid. At the same time, update the statistics array, calculate the latest hybrid centroid for that server, update the distance table, and invalidate the relevant positions in the distance table (the cluster centroid and hybrid centroid before merging). If the server load reaches its limit after merging, then the new hybrid centroid should also be invalidated in the distance table.

[0132] 4) If both are mixed centers, and the sum of the number of cluster centers corresponding to the two mixed centers exceeds the limit of a single server, then find a point with a distance second only to the distance between the two mixed centers for allocation, and return to the beginning of step 4.

[0133] 5) If both are mixed centroids, and the sum of the number of cluster centers corresponding to the two mixed centroids does not exceed the limit of a single server, then merge one of the mixed centroids into the other. For example, if server S1 is allocated 2 cluster centers and server S2 is allocated 4 cluster centers, merge the data from S2 into S1, then modify the corresponding positions in the statistics array, making S1 6 and S2 0. Simultaneously, modify the corresponding positions in the distance table, invalidating the original two mixed centroids, and generating a new mixed centroid.

[0134] In this way, after the data is distributed, the number of shards on each server is relatively even, while ensuring that the distance between shards on the same server is as far as possible.

[0135] For ease of understanding, the following example is provided:

[0136] Assume there are 1000 cluster centers and 10 servers.

[0137] Table 1: Statistical array of the initial state

[0138] No. 1 No. 2 No. 3 No. 4 No. 5 No. 6 No. 7 No. 8 No. 9 No. 10 none none none none none none none none none none

[0139] Table 2: Distance table for initial state

[0140]

[0141]

[0142] The distance table is a symmetric matrix, where larger values ​​represent greater distances (using Euclidean distance as an example). Clusters with large distances need to be assigned to the same server.

[0143] It is obvious that, as shown in the matrix above, c2 and c 1000 The furthest value is 584.3. These two clusters are combined (the vector data of the corresponding two clusters are placed on the same server). After combination, a new point cc1 is obtained, which can be calculated using formula (1). This point is c2 and c... 1000 The midpoint in space.

[0144] After the calculations were completed, we obtained the updated statistical array (see Table 3) and the new distance table (see Table 4).

[0145] Table 3: Updated statistical array

[0146] No. 1 No. 2 No. 3 No. 4 No. 5 No. 6 No. 7 No. 8 No. 9 No. 10 <![CDATA[c2,c 1000 ]]> none none none none none none none none none

[0147] In Table 3, clusters (c2, c 1000 It was assigned to server number 1.

[0148] Table 4: Updated Distance Table

[0149]

[0150] In Table 4, c2,c 1000 The value has already been assigned, and the corresponding value has been set to an invalid value. For ease of demonstration, c2 and c are directly used as the references. 1000 Cross out all data in the row and column where it is located.

[0151] At the same time, cc1 should also be added to the distance table, because cc1 represents c2 and c. 1000 Furthermore, server 1 has currently only been allocated c2 and c2. 1000 There are two shards. In this example, one server needs to allocate 100 shards, so cc1, as a new mixing center point, needs to participate in the subsequent calculation. After adding cc1 and calculating the distance between cc1 and other points, it is filled into the distance table.

[0152] Continuing to search for the two furthest points, as shown in Table 4, c4 and cc1 are 581.5 apart, making them the furthest two points. These two clusters are combined (the vector data of the corresponding two clusters are placed on the same server). The combined result is a new point cc2, which is a combination of cc1 and c4, i.e., c2, c4, and c... 1000 The midpoint of these three points in space. The value of this point can be calculated using formula (1).

[0153] At this point, we obtain new statistical arrays (see Table 5) and distance tables (see Table 6).

[0154] Table 5: Updated statistical array

[0155]

[0156] Table 6: Updated Distance Table

[0157]

[0158] Following the same logic as before, the clusters to be combined need to be invalidated; in this example, they are simply crossed out. At this point, c1 and c3 are the furthest apart. These two clusters are combined (the vector data of the corresponding two clusters are placed on the same server). The combined clusters result in a new point cc3, which is the midpoint between c1 and c3 in space. This yields a new statistical array (see Table 7) and a distance table (see Table 8).

[0159] Table 7: Updated statistical array

[0160] No. 1 No. 2 No. 3 No. 4 No. 5 No. 6 No. 7 No. 8 No. 9 No. 10 <![CDATA[c2,c4,c 1000 ]]> <![CDATA[c1,c3]]> none none none none none none none none

[0161] Table 8: Updated Distance Table

[0162] <![CDATA[c5]]> <![CDATA[c6]]> <![CDATA[c7]]> <![CDATA[c8]]> … <![CDATA[cc3]]> <![CDATA[c5]]> 0 15.5 12.8 11.8 … 25.1 <![CDATA[c6]]> 15.5 0 2.5 11.2 … 77.8 <![CDATA[c7]]> 12.8 2.5 0 7.6 … 2.3 <![CDATA[c8]]> 11.8 11.2 7.6 0 … 8.5 … … … … … … … <![CDATA[cc3]]> 25.1 77.8 2.3 8.5 … 0

[0163] Continue merging and allocating according to the previous logic until all data shards have been allocated.

[0164] In some embodiments, after performing at least one allocation operation to distribute the n data shards to m servers, the method further includes:

[0165] Establish a routing table, which includes: the data fragment identifier and server address corresponding to each of the n data fragments;

[0166] Receive a search request for the first data;

[0167] In response to the search request, the search results corresponding to the first data are output according to the routing table.

[0168] In this embodiment, after allocating n data shards to m servers, a routing table can be established to facilitate quick location of the data search range. Specifically, the routing table can include at least the data shard identifier and server address corresponding to each of the n data shards, where the server address corresponding to the data shard is the address of the server storing the data shard. Furthermore, the routing table can also include the center point corresponding to each data shard; however, in some embodiments, the center point corresponding to each data shard can be stored independently outside the routing table.

[0169] In response to a search request for the first data, the system searches the server for f data items similar to the first data, where f is a positive integer. The specific value of f can be preset according to actual needs, and this embodiment does not limit this.

[0170] In this embodiment of the application, the server to be searched and the data shards in the server can be quickly located through the routing table, thereby improving the efficiency of data search.

[0171] In some embodiments, responding to the search request and outputting the search results corresponding to the first data according to the routing table includes:

[0172] Calculate the distances between the first data and the n center points corresponding to the n data segments respectively to obtain n distances;

[0173] Using the center points corresponding to the h largest distances among the n distances, the data fragment identifier and server address corresponding to each of the h distances are obtained from the routing table, where h is a positive integer;

[0174] Based on the data fragment identifier and server address corresponding to each of the h distances, at least one target server and the target data fragment in each target server are determined.

[0175] Search for the f data pieces that are closest to the first data in the target data shard in each target server, where f is a positive integer;

[0176] An aggregation operation is performed on the f data points obtained from each of the target servers to obtain f second data points;

[0177] Output the search results corresponding to the first data, wherein the search results include the f second data.

[0178] In this embodiment, to search for similar data to the first data, the distances between the first data and the n center points corresponding to the n data segments can be calculated to obtain n distances. Then, the largest h distances are selected from the n distances, where h is a positive integer. The specific value of h can be preset according to actual needs, and this embodiment does not limit it.

[0179] After determining the maximum h distances, we can identify the h center points corresponding to those h distances. Then, by looking up the routing table, we can obtain the data fragment identifiers and server addresses corresponding to each of the h center points, i.e., the data fragment identifiers and server addresses corresponding to the first data segment. The server addresses corresponding to different center points among the h center points may be the same or different.

[0180] Then, based on the data shard identifier corresponding to the first data and the server address, f data items that are similar to the first data can be searched in the corresponding data shard of the corresponding server. In specific implementation, f data items that are similar to the first data can be searched in each corresponding data shard.

[0181] Then, an aggregation operation can be performed on all the data that are close to the first data to obtain f second data. In specific implementation, the f data with the smallest distance from the first data can be selected from all the data that are close to the first data and determined as the f second data, and then the f second data are output.

[0182] For ease of understanding, combined with Figure 3 The example is explained below:

[0183] exist Figure 3 In the system, the cluster center vector group includes n cluster center points corresponding to n clusters; the routing table includes the cluster number and node (i.e., server) address corresponding to n clusters.

[0184] Upon receiving a search request for the first data, n cluster center points can be obtained from the cluster center vector group. The distance between the first data and the n cluster center points can be calculated by the routing calculation component to obtain the maximum h distances.

[0185] Next, the routing calculation component can obtain h node addresses and cluster numbers corresponding to distances from the routing table. Based on the node addresses and cluster numbers obtained from the routing, the routing calculation component routes the request to the relevant retrieval node and specific vector group, and searches for f data items that are close to the first data item in the corresponding node and corresponding cluster. The search results are input to the aggregation component, which aggregates them to obtain f second data items and outputs the search results.

[0186] As can be seen, the data allocation method of this application embodiment can quickly locate the range of search data during data retrieval without having to search on each server and each data shard, thereby improving data search efficiency.

[0187] It should be noted that the various optional embodiments of the data allocation method described in this application can be implemented in combination or independently without conflict with each other. The specific implementation can be determined according to the actual situation, and this application does not limit this.

[0188] To facilitate understanding of the data allocation method provided in the above embodiments, the following describes the data allocation method using a specific scenario embodiment.

[0189] like Figure 4 As shown, data allocation may include the following steps:

[0190] Step 401: Cluster the original vectors.

[0191] Clustering is performed on the data in the vector library, grouping similar vectors into the same partition. This application requires that the vector distance formula used during the search must be consistent with the formula used during clustering.

[0192] Step 402: Allocate data according to cluster type.

[0193] The clustered vectors are then sharded. The sharding scheme directly impacts the performance of the distributed system and the load on individual machines. Neighboring clusters are highly likely to be searched in the same search, so it's necessary to break up neighboring clusters. Different data shards are distributed to different servers, and a single server can load multiple shards simultaneously. Ideally, each shard should occupy its own server, but this would waste significant server resources. Since the number of data shards is often large, exceeding the total number of servers, a single server needs to load multiple shards. Therefore, a more efficient scheme is needed to distribute data shards across different servers, ensuring that request traffic is evenly distributed across different servers while also maintaining relatively even server memory usage.

[0194] This step distributes the data evenly across each search server.

[0195] Step 403: Create a routing table.

[0196] Since step 402 has already completed the allocation of each shard, this step needs to import the data into the corresponding server according to this allocation rule, and record the shard number (i.e., cluster number), cluster center vector, and the server address where the shard is located, storing this information in the database. This step facilitates accurate routing to the corresponding server for vector retrieval during searches.

[0197] As can be seen, this application embodiment uses a clustering algorithm to cluster vectors based on their spatial distribution characteristics, and then fragments the vector data according to the cluster number. The fragmented vectors can be distributed across different servers to achieve distributed storage and retrieval.

[0198] like Figure 5 As shown, data searching may include the following steps:

[0199] Step 501: Calculate the neighbor class of the request vector.

[0200] The purpose of this step is to locate the approximate range of the target vector. The vector to be searched is first compared with the distances of all cluster center vectors to obtain one or more nearest cluster centers for further, more precise searching. In practical systems, to achieve greater accuracy, more than one nearest cluster is often selected for further precise searching. At the same time, for better performance, the number of nearest clusters selected should not be too large. Different systems have different requirements for performance and accuracy, and the number of cluster centers selected will be dynamically adjusted. In this invention, h represents the number of nearest cluster centers selected.

[0201] Step 502: Obtain the corresponding server list.

[0202] For the selected h nearest neighbor clusters, obtain the server address and cluster number corresponding to these clusters from the routing table.

[0203] Step 503: Search for sharded data.

[0204] After obtaining the server address, search for the f nearest neighbors to the target vector in the corresponding cluster of the server.

[0205] Step 504, aggregation.

[0206] The search results returned by multiple servers in the previous step are aggregated to obtain the final search results.

[0207] In this embodiment, when performing vector retrieval, the distance between the request vector and the cluster center is first calculated. After obtaining the nearest cluster number, the corresponding server is then searched. Compared to existing distributed vector retrieval, this embodiment can accurately locate the specific vector data server for retrieval, without needing to search all vector data servers. Through this embodiment, request traffic is routed to different servers, significantly improving system performance.

[0208] The data allocation method provided in this application can be executed by a data allocation device. This application uses an example of a data allocation device executing the data allocation method to illustrate the data allocation device provided in this application.

[0209] like Figure 6 As shown, the data distribution device 600 may include:

[0210] Clustering module 601 is used to cluster the data to obtain n data fragments and n center points corresponding to the n data fragments, where n is an integer greater than 1;

[0211] The first acquisition module 602 is used to acquire the distances between each of the k center points and each other, obtaining k×k distances. The k center points include: center points corresponding to unallocated data shards among the n data shards, and center points corresponding to servers among the m servers that satisfy a first condition. The first condition includes that data shards are allocated, and the number of allocated data shards is less than a first threshold; m is an integer greater than 1, m is less than n; and k is an integer greater than 1.

[0212] The first determining module 603 is used to determine the first center point and the second center point corresponding to the maximum distance among the k×k distances;

[0213] The first allocation module 604 is used to allocate the first data fragment and the second data fragment to the same server among the m servers.

[0214] In some embodiments, the first center point and the second center point are both center points corresponding to data shards; among the m servers, there are servers that have not been allocated data shards;

[0215] The first allocation module is specifically used for:

[0216] The first data shard and the second data shard are allocated to the first server;

[0217] The first server is any one of the m servers that has not been allocated a data shard.

[0218] In some embodiments, the first center point and the second center point are both center points corresponding to data shards; each of the m servers is allocated a data shard;

[0219] The first allocation module is specifically used for:

[0220] Calculate the distances between the target center point and the center points of each server that meet the second condition, resulting in m distances. The target center point is the center point of the first center point and the second center point. The second condition is: data shards are allocated, and the number of stored data shards plus 2 is less than or equal to the first threshold.

[0221] The second server is the server corresponding to the largest distance among the m distances.

[0222] In some embodiments, the first center point is the center point corresponding to the cluster class, and the second center point is the center point corresponding to the server;

[0223] The first allocation module is specifically used for:

[0224] The first data shard is allocated to the third server;

[0225] The third server is the server corresponding to the second central point.

[0226] In some embodiments, the first center point is the center point corresponding to the fourth server, and the second center point is the center point corresponding to the fifth server;

[0227] The first allocation module is specifically used for:

[0228] If the first number is less than the first threshold, the first data fragment is allocated to the fifth server, or the second data fragment is allocated to the fourth server;

[0229] Wherein, the first quantity is the sum of the number of data fragments included in the first data fragment and the second data fragment.

[0230] In some embodiments, the apparatus further includes:

[0231] The second determining module is used to determine the third center point and the fourth center point corresponding to the second largest distance among the k×k distances when the first quantity is greater than or equal to the first threshold.

[0232] The second allocation module is used to allocate the third data shard and the fourth data shard to the same server among the m servers, wherein the third data shard is the data shard corresponding to the third center point, and the fourth data shard is the data shard corresponding to the fourth center point.

[0233] In some embodiments, the apparatus further includes:

[0234] The second acquisition module is used to acquire the distance table and the statistics array;

[0235] The statistical array includes m values ​​that correspond one-to-one with the m servers, where the value corresponding to a server represents the number of data shards allocated to that server.

[0236] The distance table includes row and column labels corresponding to q center points, and q rows and q columns of values; the q center points include: the n data shards, and the center points corresponding to the servers among the m servers that are allocated data shards; the value in the i-th row and j-th column represents the distance between the center point corresponding to the i-th row label and the center point corresponding to the j-th row label; q is an integer greater than n, and i and j are both positive integers less than or equal to q;

[0237] In the distance table, the values ​​corresponding to the fifth center point are all set to invalid values; the fifth center point includes: the center point corresponding to the data shards that have been allocated among the n data shards, and the center point corresponding to the server whose value in the statistical array is greater than or equal to the first threshold;

[0238] The first allocation module is specifically used for:

[0239] Query the valid values ​​in the distance table to obtain k×k distances.

[0240] In some embodiments, the apparatus further includes:

[0241] A module is established to create a routing table, which includes: the data fragment identifier and server address corresponding to each of the n data fragments;

[0242] The receiving module is used to receive search requests for the first data;

[0243] The module is used to calculate the distance between the first data and the n center points corresponding to the n data slices, respectively, to obtain n distances;

[0244] The routing module is used to obtain the data fragment identifier and server address corresponding to each of the h distances from the routing table by using the center points corresponding to the h largest distances among the n distances, where h is a positive integer;

[0245] The third determining module is used to determine at least one target server and the target data fragment in each of the h distances based on the data fragment identifier and server address corresponding to each distance.

[0246] The search module is used to search for f data that are closest to the first data in the target data shard in each target server, where f is a positive integer;

[0247] An aggregation module is used to perform an aggregation operation on the f data obtained from each of the target servers to obtain f second data.

[0248] An output module is used to output the search results corresponding to the first data, wherein the search results include the f second data.

[0249] The data allocation device provided in this application embodiment can implement the various processes implemented in the foregoing method embodiments, and will not be described again here to avoid repetition.

[0250] In the embodiments of this application Data distribution device It can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or any other device besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application does not specifically limit the scope of the embodiments.

[0251] The data distribution device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0252] Optionally, such as Figure 7 As shown, this application embodiment also provides an electronic device 700, including a processor 701 and a memory 702. The memory 702 stores a program or instructions that can run on the processor 701. When the program or instructions are executed by the processor 701, they implement the various steps of the above-described data allocation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0253] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0254] Figure 8 A hardware structure diagram of an electronic device to implement an embodiment of this application.

[0255] The electronic device 800 includes, but is not limited to, components such as: radio frequency unit 801, network module 802, audio output unit 803, input unit 804, sensor 805, display unit 806, user input unit 807, interface unit 808, memory 809, and processor 810.

[0256] Those skilled in the art will understand that the electronic device 800 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 810 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0257] The processor 810 is used for:

[0258] The data is clustered to obtain n data segments and n center points corresponding to the n data segments, where n is an integer greater than 1;

[0259] Obtain the distances between each of the k center points and each of the other center points, resulting in k×k distances. The k center points include: center points corresponding to unallocated data shards among the n data shards, and center points corresponding to servers among the m servers that satisfy a first condition. The first condition includes that data shards are allocated, and the number of allocated data shards is less than a first threshold; m is an integer greater than 1 and less than n; k is an integer greater than 1.

[0260] Determine the first and second center points corresponding to the maximum distance among the k×k distances;

[0261] The first data shard and the second data shard are allocated to the same server among the m servers, wherein the first data shard is the data shard corresponding to the first center point, and the second data shard is the data shard corresponding to the second center point.

[0262] The electronic device provided in this application embodiment can implement the various processes implemented in the foregoing method embodiments, and will not be described again here to avoid repetition.

[0263] It should be understood that, in this embodiment, the input unit 804 may include a graphics processing unit (GPU) 8041 and a microphone 8042. The GPU 8041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 806 may include a display panel 8061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 807 includes at least one of a touch panel 8071 and other input devices 8072. The touch panel 8071 is also called a touch screen. The touch panel 8071 may include a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0264] The memory 809 can be used to store software programs and various data. The memory 809 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 809 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 809 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0265] Processor 810 may include one or more processing units; optionally, processor 810 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 810.

[0266] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data allocation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0267] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0268] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above data allocation method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0269] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0270] This application provides a computer program product that is stored in a storage medium and executed by at least one processor to implement the various processes of the data allocation method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0271] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0272] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0273] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data allocation method, characterized in that, include: The data is clustered to obtain n data segments and n center points corresponding to the n data segments, where n is an integer greater than 1; Obtain the distances between each of the k center points and each of the other center points, resulting in k×k distances. The k center points include: center points corresponding to unallocated data shards among the n data shards, and center points corresponding to servers among the m servers that satisfy a first condition. The first condition includes that data shards are allocated, and the number of allocated data shards is less than a first threshold; m is an integer greater than 1 and less than n; k is an integer greater than 1. Determine the first and second center points corresponding to the maximum distance among the k×k distances; The first data shard and the second data shard are allocated to the same server among the m servers, wherein the first data shard is the data shard corresponding to the first center point, and the second data shard is the data shard corresponding to the second center point.

2. The method according to claim 1, characterized in that, Both the first center point and the second center point are center points corresponding to data shards; among the m servers, there are servers that have not been allocated data shards; The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes: The first data shard and the second data shard are allocated to the first server; The first server is any one of the m servers that has not been allocated a data shard.

3. The method according to claim 1, characterized in that, Both the first center point and the second center point are center points corresponding to data shards; each of the m servers is allocated a data shard; The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes: Calculate the distances between the target center point and the center points of each server that meet the second condition, resulting in m distances. The target center point is the center point of the first center point and the second center point. The second condition is: data shards are allocated, and the number of stored data shards plus 2 is less than or equal to the first threshold. The first data shard and the second data shard are allocated to the second server; The second server is the server corresponding to the largest distance among the m distances.

4. The method according to claim 1, characterized in that, The first center point is the center point corresponding to the data shard, and the second center point is the center point corresponding to the server; The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes: The first data shard is allocated to the third server; The third server is the server corresponding to the second central point.

5. The method according to claim 1, characterized in that, The first center point is the center point corresponding to the fourth server, and the second center point is the center point corresponding to the fifth server; The step of allocating the first data fragment and the second data fragment to the same server among the m servers includes: If the first number is less than the first threshold, the first data fragment is allocated to the fifth server, or the second data fragment is allocated to the fourth server; Wherein, the first quantity is the sum of the number of data fragments included in the first data fragment and the second data fragment.

6. The method according to claim 5, characterized in that, The method further includes: If the first quantity is greater than or equal to the first threshold, determine the third center point and the fourth center point corresponding to the second largest distance among the k×k distances; The third data shard and the fourth data shard are allocated to the same server among the m servers, wherein the third data shard is the data shard corresponding to the third center point, and the fourth data shard is the data shard corresponding to the fourth center point.

7. The method according to claim 1, characterized in that, Before obtaining the distances between each of the k center points and all other center points to obtain k×k distances, the method further includes: Get the distance table and statistics array; The statistical array includes m values ​​that correspond one-to-one with the m servers, where the value corresponding to a server represents the number of data shards allocated to that server. The distance table includes row and column labels corresponding to q center points, and q row and q column values; the q center points include: the n data shards, and the center points corresponding to the servers among the m servers that are allocated data shards; the value in the i-th row and j-th column represents the distance between the center point corresponding to the i-th row label and the center point corresponding to the j-th row label; q is an integer greater than or equal to n, and i and j are both positive integers less than or equal to q; In the distance table, the values ​​corresponding to the fifth center point are all set to invalid values; the fifth center point includes: the center point corresponding to the data shards that have been allocated among the n data shards, and the center point corresponding to the server whose value in the statistical array is greater than or equal to the first threshold; The step of obtaining the distance between each of the k center points and each of the other center points, resulting in k×k distances, includes: Query the valid values ​​in the distance table to obtain k×k distances.

8. The method according to any one of claims 1 to 7, characterized in that, After allocating the first data shard and the second data shard to the same server among the m servers, the method further includes: Establish a routing table, which includes: the data fragment identifier and server address corresponding to each of the n data fragments; Receive a search request for the first data; Calculate the distances between the first data and the n center points corresponding to the n data segments respectively to obtain n distances; Using the center points corresponding to the h largest distances among the n distances, the data fragment identifier and server address corresponding to each of the h distances are obtained from the routing table, where h is a positive integer; Based on the data fragment identifier and server address corresponding to each of the h distances, at least one target server and the target data fragment in each target server are determined. Search for the f data pieces that are closest to the first data in the target data shard in each target server, where f is a positive integer; An aggregation operation is performed on the f data points obtained from each of the target servers to obtain f second data points; Output the search results corresponding to the first data, wherein the search results include the f second data.

9. A data distribution device, characterized in that, include: The clustering module is used to cluster the data to obtain n data fragments and n centroids corresponding to the n data fragments, where n is an integer greater than 1; The first acquisition module is used to acquire the distances between each of the k center points and each other, resulting in k×k distances. The k center points include: center points corresponding to unallocated data shards among the n data shards, and center points corresponding to servers among the m servers that satisfy a first condition. The first condition includes that data shards are allocated, and the number of allocated data shards is less than a first threshold; m is an integer greater than 1 and less than n; k is an integer greater than 1. The first determining module is used to determine the first center point and the second center point corresponding to the maximum distance among the k×k distances; The first allocation module is used to allocate the first data shard and the second data shard to the same server among the m servers, wherein the first data shard is the data shard corresponding to the first center point, and the second data shard is the data shard corresponding to the second center point.

10. The apparatus according to claim 9, characterized in that, Both the first center point and the second center point are center points corresponding to data shards; among the m servers, there are servers that have not been allocated data shards; The first allocation module is specifically used for: The first data shard and the second data shard are allocated to the first server; The first server is any one of the m servers that has not been allocated a data shard.

11. The apparatus according to claim 9, characterized in that, Both the first center point and the second center point are center points corresponding to data shards; each of the m servers is allocated a data shard; The first allocation module is specifically used for: Calculate the distances between the target center point and the center points of each server that meet the second condition, obtaining m distances, where the target center point is the center point of the first center point and the second center point; the second condition is: data shards are allocated, and the number of stored data shards plus 2 is less than or equal to the first threshold; allocate the first data shard and the second data shard to the second server; The second server is the server corresponding to the largest distance among the m distances.

12. The apparatus according to claim 9, characterized in that, The first center point is the center point corresponding to the data shard, and the second center point is the center point corresponding to the server; The first allocation module is specifically used for: The first data shard is allocated to the third server; The third server is the server corresponding to the second central point.

13. The apparatus according to claim 9, characterized in that, The first center point is the center point corresponding to the fourth server, and the second center point is the center point corresponding to the fifth server; The first allocation module is specifically used for: If the first number is less than the first threshold, the first data fragment is allocated to the fifth server, or the second data fragment is allocated to the fourth server; Wherein, the first quantity is the sum of the number of data fragments included in the first data fragment and the second data fragment.

14. The apparatus according to claim 13, characterized in that, The device further includes: The second determining module is used to determine the third center point and the fourth center point corresponding to the second largest distance among the k×k distances when the first quantity is greater than or equal to the first threshold. The second allocation module is used to allocate the third data shard and the fourth data shard to the same server among the m servers, wherein the third data shard is the data shard corresponding to the third center point, and the fourth data shard is the data shard corresponding to the fourth center point.

15. The apparatus according to claim 9, characterized in that, The device further includes: The second acquisition module is used to acquire the distance table and the statistics array; The statistical array includes m values ​​that correspond one-to-one with the m servers, where the value corresponding to a server represents the number of data shards allocated to that server. The distance table includes row and column labels corresponding to q center points, and q row and q column values; the q center points include: the n data shards, and the center points corresponding to the servers among the m servers that are allocated data shards; the value in the i-th row and j-th column represents the distance between the center point corresponding to the i-th row label and the center point corresponding to the j-th row label; q is an integer greater than or equal to n, and i and j are both positive integers less than or equal to q; In the distance table, the values ​​corresponding to the fifth center point are all set to invalid values; the fifth center point includes: the center point corresponding to the data shards that have been allocated among the n data shards, and the center point corresponding to the server whose value in the statistical array is greater than or equal to the first threshold; The first allocation module is specifically used for: Query the valid values ​​in the distance table to obtain k×k distances.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The device further includes: A module is established to create a routing table, which includes: the data fragment identifier and server address corresponding to each of the n data fragments; The receiving module is used to receive search requests for the first data; The calculation module is used to calculate the distance between the first data and the n center points corresponding to the n data segments respectively, so as to obtain n distances; The routing module is used to obtain the data fragment identifier and server address corresponding to each of the h distances from the routing table by using the center points corresponding to the h largest distances among the n distances, where h is a positive integer; The third determining module is used to determine at least one target server and the target data fragment in each of the h distances based on the data fragment identifier and server address corresponding to each distance. The search module is used to search for f data that are closest to the first data in the target data shard in each target server, where f is a positive integer; An aggregation module is used to perform an aggregation operation on the f data obtained from each of the target servers to obtain f second data. An output module is used to output the search results corresponding to the first data, wherein the search results include the f second data.

17. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the data allocation method as described in any one of claims 1 to 8.

18. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the data allocation method as described in any one of claims 1 to 8.

Images