Distributed vector processing method, apparatus, system, and medium

Abstract

A distributed vector processing method, device, system and medium are disclosed. The system includes at least two clusters, each cluster includes a plurality of nodes, and the number of nodes in each cluster of the at least two clusters is the same. In each cluster, the plurality of nodes include two types of nodes, aggregation nodes and computing nodes. The aggregation nodes in each cluster are used to manage the data in the cluster, and the computing nodes are used to store and calculate the data of at least one partition. Each computing node in each cluster in the scheme provided by the application can perform parallel storage and calculation on data, can quickly transplant local algorithms without a distributed foundation, can realize distributed calculation, and can accelerate the development efficiency of federated algorithms.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a distributed vector processing method, apparatus, system, and medium. Background Technology

[0002] With the continuous development of computer technology, federated learning (also known as federated machine learning) has been widely used in various technical fields. Federated learning is a machine learning framework that can effectively help multiple organizations to use data and perform machine learning modeling while meeting the requirements of user privacy protection, data security and government regulations.

[0003] In the current field of federated learning, there are differences between the single-machine development model and the multi-machine distributed development model when developing a federated learning algorithm. The single-machine version is simple and intuitive to program, allowing participants to better organize and process complete data from all parties. It also allows for referencing existing algorithm implementations in unencrypted scenarios to complete the federated algorithm development. However, its drawback is that it can only process small amounts of data. Furthermore, the single-machine version currently developed by researchers cannot be directly migrated to the distributed version.

[0004] In summary, there is currently no simple and convenient distributed development architecture or solution for handling large amounts of data in federated learning. Summary of the Invention

[0005] The main objective of this invention is to provide a distributed vector processing method, apparatus, system, and medium, and to provide a simple and convenient distributed development system. Through the development and application of this system, not only can the problem of processing large amounts of data in federated learning scenarios be solved, but it can also be directly migrated to a distributed version for application.

[0006] To achieve the above objectives, the present invention provides a distributed vector processing system, comprising:

[0007] At least two clusters, each cluster comprising multiple nodes, wherein the number of nodes in each of the at least two clusters is the same;

[0008] In each cluster, the multiple nodes include two types of nodes: aggregation nodes and compute nodes;

[0009] In each cluster, the aggregation node is used to manage the data in its cluster, and the compute node is used to store and compute data in at least one partition.

[0010] In one specific implementation, the aggregation node in each cluster is also used to execute and / or distribute computation commands.

[0011] In one specific implementation, the aggregation node in each cluster is also used to communicate with other clusters.

[0012] In one specific implementation, the aggregation node in each cluster is specifically used for:

[0013] Obtain a computation instruction, the computation instruction being used to instruct computation on a data vector in the cluster, the computation instruction including an operation type and a data identifier;

[0014] Based on the operation type and the data identifier, an operation instruction is determined, wherein the operation instruction includes an operator and a vector key value;

[0015] The computation instructions are sent to each computing node in the cluster respectively;

[0016] Accordingly, the compute nodes in each cluster are specifically used for:

[0017] Receive computation instructions sent by the aggregation node of the cluster, wherein the computation instructions include operators and vector key values;

[0018] According to the operator, the data vector corresponding to the vector key value in at least one local partition is operated on to obtain the operated data vector.

[0019] In one specific implementation, the aggregation node in each cluster is also used for:

[0020] Based on the number of data partitions in the cluster, the received data to be stored is split to obtain multiple data vectors;

[0021] The vector key value corresponding to each data vector is determined according to preset rules;

[0022] Based on the number of data partitions on each computing node, the multiple data vectors and vector key values ​​are sent to each computing node in the cluster respectively;

[0023] Correspondingly, the compute nodes in each cluster are also used for:

[0024] Receive at least one data vector and a vector key value for each data vector sent by the aggregation node;

[0025] The at least one data vector is stored in at least one data partition based on a vector key, wherein each data partition stores one data vector.

[0026] In one specific implementation, the aggregation node in each cluster is also used for:

[0027] Read the data vectors of each data partition in each computing node of the cluster to obtain multiple data vectors;

[0028] The multiple data vectors are merged based on vector keys to obtain a local vector.

[0029] In one specific implementation, the aggregation node in each cluster is also used for:

[0030] Send the data vector to the aggregation node of other clusters;

[0031] or,

[0032] Receive data vectors sent by aggregation nodes in other clusters.

[0033] In one specific implementation, the compute nodes in each cluster are also used for:

[0034] The data vector is encrypted and decrypted using the stored key to obtain the processed data vector.

[0035] In one specific implementation, the distributed vector processing system is used to process data vectors in a federated learning scenario.

[0036] This invention also provides a distributed vector processing method, applied to the aggregation node of any cluster in the aforementioned distributed vector processing system, the method comprising:

[0037] Obtain a computation instruction, the computation instruction being used to instruct computation on a data vector in the cluster, the computation instruction including an operation type and a data identifier;

[0038] Based on the operation type and the data identifier, an operation instruction is determined, wherein the operation instruction includes an operator and a vector key value;

[0039] The computation instructions are sent to each computing node in the cluster.

[0040] In one specific implementation, if the operation type is a binary operation, the operation instruction further includes an lvalue variable and an rvalue variable.

[0041] In one specific embodiment, the method further includes:

[0042] Based on the number of data partitions in the cluster, the received data to be stored is split to obtain multiple data vectors;

[0043] The vector key value corresponding to each data vector is determined according to preset rules;

[0044] Based on the number of data partitions on each computing node, the multiple data vectors and vector key values ​​are sent to each computing node in the cluster.

[0045] In one specific embodiment, the method further includes:

[0046] Read the data vectors of each data partition in each computing node of the cluster to obtain multiple data vectors;

[0047] The multiple data vectors are merged based on vector keys to obtain a local vector.

[0048] In one specific embodiment, the method further includes:

[0049] Send the data vector to the aggregation node of other clusters;

[0050] or,

[0051] Receive data vectors sent by aggregation nodes in other clusters.

[0052] In one specific implementation, the data vector is a data vector in a federated learning scenario.

[0053] This invention also provides a distributed vector processing method, applied to the computing nodes of any cluster in the aforementioned distributed vector processing system, the method comprising:

[0054] Receive computation instructions sent by the aggregation node of the cluster, wherein the computation instructions include operators and vector key values;

[0055] According to the operator, the data vector corresponding to the vector key value in at least one local partition is operated on to obtain the operated data vector.

[0056] In one specific implementation, if the operation type is a binary operation, the operation instruction further includes an lvalue variable and an rvalue variable;

[0057] Accordingly, the step of performing operations on the data vector corresponding to the vector key value in at least one local partition according to the operator to obtain the processed data vector includes:

[0058] Based on the operator, the lvalue variable, and the rvalue variable, the data vector corresponding to the vector key value in at least one local partition is operated on to obtain the operated data vector.

[0059] In one specific embodiment, the method further includes:

[0060] Receive at least one data vector and a vector key value for each data vector sent by the aggregation node;

[0061] The at least one data vector is stored in at least one data partition based on a vector key, wherein each data partition stores one data vector.

[0062] In one specific implementation, before performing the operation on the data vector corresponding to the vector key value in at least one local partition according to the operator to obtain the operated data vector, the method further includes:

[0063] If the data vector corresponding to the vector key value in at least one local partition is an encrypted vector, then the data vector corresponding to the vector key value in at least one local partition is decrypted using a pre-stored key to obtain the decrypted data vector.

[0064] In one specific embodiment, the method further includes:

[0065] The processed data vector is encrypted using a pre-stored key.

[0066] In one specific implementation, the data vector is a data vector in a federated learning scenario.

[0067] The present invention also provides a distributed vector processing apparatus, comprising:

[0068] A receiving module is used to acquire calculation instructions, which are used to instruct the calculation of data vectors in the cluster. The calculation instructions include operation type and data identifier.

[0069] The processing module is used to determine the operation instruction based on the operation type and the data identifier, wherein the operation instruction includes an operator and a vector key value;

[0070] The sending module is used to send the operation instructions to each computing node in the cluster.

[0071] The present invention also provides a distributed vector processing apparatus, comprising:

[0072] The receiving module is used to receive the operation instructions sent by the aggregation node of the cluster, wherein the operation instructions include operators and vector key values;

[0073] The processing module is used to perform operations on the data vector corresponding to the vector key value in at least one local partition according to the operator, so as to obtain the processed data vector.

[0074] The present invention also provides an electronic device, comprising: a memory, a processor, and an interactive interface, wherein the memory stores a computer program executable on the processor, and the computer program, when executed by the processor, implements the steps of the distributed vector processing method as described in any of the preceding claims.

[0075] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the distributed vector processing method described in any of the preceding claims.

[0076] The present invention also provides a computer program product, including a computer program, which is executed by a processor to implement the steps of the distributed vector processing method described in any of the preceding claims.

[0077] In this invention, the distributed vector processing system includes at least two clusters, each containing multiple nodes, with the number of nodes in each cluster being the same. Within each cluster, the multiple nodes include both aggregation nodes and compute nodes. The aggregation nodes in each cluster manage the data within their respective cluster, while the compute nodes store and compute data from at least one partition. The compute nodes in each cluster of the solution provided by this invention can perform parallel storage and computation of data, enabling rapid porting of local algorithms without the need for a distributed infrastructure, thus achieving distributed computing and accelerating the development efficiency of federated algorithms. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the architecture of the distributed vector processing system provided by the present invention;

[0079] Figure 2 A flowchart of an embodiment of the distributed vector processing method provided by the present invention;

[0080] Figure 3 A flowchart of Embodiment 2 of the distributed vector processing method provided by the present invention;

[0081] Figure 4 A flowchart of Embodiment 3 of the distributed vector processing method provided by the present invention;

[0082] Figure 5 This is a schematic diagram of the computing interface in the distributed vector processing system provided by the present invention;

[0083] Figure 6 This is a schematic diagram of the structure of a first embodiment of the distributed vector processing device provided by the present invention;

[0084] Figure 7 This is a schematic diagram of the structure of a second embodiment of the distributed vector processing device provided by the present invention;

[0085] Figure 8 This is a schematic diagram of the structure of an electronic device according to a first embodiment of the present invention.

[0086] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0087] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0088] Existing single-machine versions of technologies cannot handle large datasets. Developing distributed versions requires experience in distributed development, resulting in a high barrier to entry and significant difficulty. Furthermore, current federated learning distributed frameworks only provide interfaces such as map, reduce, and aggregate, which are too low-level for algorithm developers, further increasing the difficulty. In conclusion, there is currently no suitable distributed vector solution or implementation for federated learning scenarios.

[0089] To address the aforementioned issues and facilitate the development of federated learning algorithms, this invention presents a distributed vector architecture and design method suitable for federated learning scenarios. Using this invention, developers can program using distributed vectors in a manner similar to using single-machine vectors, without being limited by distributed programming fundamentals. The operation is simple and convenient, and it can be directly migrated to a distributed version.

[0090] The following detailed description of the distributed vector processing scheme for federated learning scenarios provided by the present invention will be based on specific embodiments.

[0091] Figure 1 This is a schematic diagram of the architecture of the distributed vector processing system provided by the present invention, as shown below. Figure 1 As shown, the distributed vector processing system includes:

[0092] At least two clusters, each containing multiple nodes, wherein the number of nodes in each of the at least two clusters is the same.

[0093] In each cluster, the plurality of nodes include two types of nodes: aggregation nodes and compute nodes. The aggregation nodes in each cluster are used to manage the data in their respective clusters, while the compute nodes are used to store and compute data in at least one partition.

[0094] like Figure 1 The cluster on the left includes one aggregation node and two compute nodes, while the cluster on the right also includes one aggregation node and two compute nodes. The number of compute nodes in a cluster can be three or more; this solution does not impose any restrictions on this.

[0095] Optionally, the aggregation nodes in each cluster are also used to communicate with other clusters. Figure 1 The two clusters interact through the aggregation nodes in their respective clusters. In practical applications, clusters can also communicate through compute nodes, and this solution does not restrict this.

[0096] In this system, the aggregation node in each cluster is responsible for managing the work of the entire cluster. It can partition received data and distribute it to various compute nodes for storage. It can also integrate or merge data vectors on compute nodes before sending them to other clusters or other entities. Optionally, during vector operations, the aggregation node in each cluster is also used to execute and / or distribute computation commands. For example, it can obtain the corresponding operators and / or related parameters based on the computation command and then instruct them on each compute node.

[0097] In a specific application, taking a federated learning scenario as an example, a federated learning scenario may include a guest party and a host party. In this scheme, Figure 1 Any one of the clusters can act as the guest, and the other can act as the host; this scheme does not impose any restrictions on this.

[0098] For each cluster, both compute nodes and aggregation nodes can be implemented as electronic devices with data processing and communication functions, such as computers, computer equipment, and servers. This solution does not restrict the specific type of equipment.

[0099] Figure 2 This is a flowchart of an embodiment of the distributed vector processing method provided by the present invention. Figure 1 Based on the system architecture shown, the distributed vector processing method provided by this invention mainly involves two execution entities in the cluster: the aggregation node and the computing node, specifically including the following steps:

[0100] S101: Obtain calculation instructions.

[0101] In this step, the calculation instruction is used to instruct the calculation of data vectors in the cluster, and the calculation instruction includes the operation type and data identifier.

[0102] In the application of this distributed vector processing scheme, when developers need to perform calculations on the data vectors stored in the cluster, they can send calculation instructions to the aggregation node through external devices or operating devices. Generally, the calculation instructions should at least clearly indicate the data to be calculated and what operation to perform, that is, the aforementioned data identifier and operation type.

[0103] S102: Determine the operation instruction based on the operation type and data identifier. The operation instruction includes the operator and the vector key value.

[0104] In this step, after obtaining the operation type and data identifier, the aggregation node determines all data vectors corresponding to that data identifier, as well as the key values ​​of those data vectors. Since the data vectors are stored in different partitions and managed by the aggregation node, the vector key values ​​of the data vectors to be operated on can be determined within the aggregation node to instruct the computing nodes.

[0105] Aggregation nodes can also determine the operator based on the operation type, and indicate it along with the compute node.

[0106] Optionally, in one specific implementation, if the operation type is a binary operation, the operation instruction also includes an lvalue variable and an rvalue variable. That is, the aggregation node needs to determine the relevant parameters associated with the binary operation. For example, if it is a binary operation between two data vectors, then the vector key values ​​of the two vectors need to be explicitly indicated; if it is a binary operation between a certain data and other constants, then the vector key value and the constant need to be indicated.

[0107] S103: Send computation instructions to each compute node in the cluster.

[0108] In this step, the aggregation node sends the obtained computational quality to each compute node in the cluster that includes the data vector corresponding to the data identifier.

[0109] Each compute node receives computation instructions sent by the aggregation node in its cluster. These instructions include operators and vector key values.

[0110] S104: Based on the operator, perform operations on the data vectors corresponding to the vector key values ​​in at least one local partition to obtain the processed data vector.

[0111] In this step, there may be multiple partitions in the same computing node, each of which stores a data vector, or there may be only one partition on a computing node, which stores a data vector corresponding to the data identifier.

[0112] When a compute node has only one partition, after receiving a computation instruction, it retrieves the corresponding data vector based on the vector key, performs the computation on the data vector according to the operator specified, and obtains the processed data vector. At this point, it should be understood that the compute node contains two data vectors: one before the computation and one after. The processed data vector can be identified by a new vector key, and this new identifier can be returned to the aggregation node for storage.

[0113] When a compute node contains multiple partitions, after receiving an operation instruction, the compute node retrieves the corresponding data vectors based on the multiple vector keys. Then, according to the specific operation indicated by the operator, it performs the operation on each of these data vectors, resulting in multiple processed data vectors. These processed data vectors are still stored in the partition where they were originally stored. At this point, it should be understood that the compute node contains two sets of data vectors: one before the operation and one after. Each processed data vector can be identified by a new vector key, and this new identifier can be returned to the aggregation node for storage.

[0114] Optionally, the operation types generally include unary operations, binary operations, and aggregation algorithms. In specific implementations, if the operation type is a binary operation, the operation instruction also includes lvalue variables and rvalue variables.

[0115] Accordingly, this step can be specifically implemented as follows: based on the operator, the lvalue variable, and the rvalue variable, perform operations on the data vector corresponding to the vector key value in at least one local partition to obtain the processed data vector.

[0116] The meaning of this scheme is that the computation instructions obtained by the computing node also include other relevant parameters required for binary operations. Similarly, operations can be performed on the relevant data vectors in each partition based on operators, lvalue variables, and rvalue variables to obtain the processed data vectors.

[0117] The distributed vector processing method in this embodiment is based on Figure 1 The system architecture shown allows each computing node in each cluster to perform parallel computations on the data. It enables the rapid porting of local algorithms without the need for a distributed infrastructure, thus achieving distributed computing. This can accelerate the development efficiency of some complex algorithms, especially in the development of federated learning algorithms, where it can effectively improve development efficiency.

[0118] Figure 3 The flowchart of Embodiment 2 of the distributed vector processing method provided by the present invention is as follows: Figure 3 As shown, based on the above embodiments, the aggregation node can also be used to partition the received data and then store it separately on each compute node in the cluster, specifically including the following steps:

[0119] S201: Based on the number of data partitions in the cluster, the received data to be stored is split into multiple data vectors.

[0120] In this step, when there is data that needs to be stored in the cluster using distributed vectors, after the aggregation node obtains the data to be stored, it can split the data to be stored according to the number of data partitions in the current cluster, either in order or according to other rules, to obtain multiple data vectors that are the same as the number of partitions.

[0121] S202: Determine the vector key value corresponding to each data vector according to the preset rules.

[0122] In this step, in order to manage each data vector, the aggregation node needs to identify each data vector. Specifically, the vector key value of each data vector can be set according to preset rules. This vector key value is mainly used to query and determine the data vector during the operation or read and write process.

[0123] In specific implementation, the preset rule can be numbered in a certain order, or the vector key value can be set according to the row number or other parameters used in the splitting rule. This solution does not restrict this.

[0124] S203: Based on the number of data partitions on each compute node, send multiple data vectors and vector key values ​​to each compute node in the cluster.

[0125] In this step, for the aggregation node, after splitting into multiple data vectors and a vector key for each vector, the multiple data vectors and vector keys can be sent to various partitions for storage. For each compute node, it receives at least one data vector and a vector key for each data vector sent by the aggregation node.

[0126] S204: Store at least one data vector to at least one data partition based on a vector key, wherein each data partition stores one data vector.

[0127] In this step, after receiving the data vector and the vector key, the compute node stores the data vector in the data partition.

[0128] It should be understood that in this scheme, since a computing node may contain one or more partitions, a computing node can receive one data vector and vector key, or more data vectors and vector key. For data corresponding to the same data identifier, each partition stores only one data vector of that data. This scheme does not limit the number of data partitions in a computing node.

[0129] Figure 4The flowchart below shows a third embodiment of the distributed vector processing method provided by the present invention. Based on the above embodiments, the aggregation node can also be used to merge or aggregate vectors from various computing nodes in the cluster, specifically including the following steps:

[0130] S301: Reads the data vectors of each data partition in each compute node of the cluster to obtain multiple data vectors.

[0131] S302: Merge multiple data vectors based on vector key values ​​to obtain a local vector.

[0132] As can be seen from the foregoing embodiments, in the above steps, the multiple clusters in the distributed vector processing system provided by the present invention can not only perform data distribution, storage and computation within the cluster, but the aggregation nodes within the cluster can also acquire and aggregate data from various computing nodes in order to transmit it to other devices or other clusters.

[0133] Specifically, when it is necessary to obtain the data corresponding to a certain data identifier in a cluster, the aggregation node can read the data vectors in each data partition from each computing node based on the vector key value corresponding to the data identifier managed locally, and then obtain multiple data vectors. These data vectors generally belong to the same overall data.

[0134] For aggregation nodes, there are two methods for transmitting data to other devices or clusters. In one implementation, the aggregation node can merge multiple data vectors based on vector keys to obtain a local vector. In other words, it merges the data vectors into a single, unified data set, i.e., a local vector. The merging pattern corresponds to that used when splitting the data into multiple vectors.

[0135] Alternatively, in another implementation, if other devices or clusters also ultimately need to use distributed vectors, then the aggregation node does not need to merge the multiple data vectors. It only needs to package the multiple vectors for transmission, that is, transmit them as a whole, and its data still includes multiple distributed data vectors.

[0136] In practical applications, whether the aggregation node needs to merge multiple data vectors can be configured according to the actual situation, and this solution does not impose any restrictions on this.

[0137] Optionally, the system typically contains at least two clusters, which can interact with each other using data or vectors. Data interaction between clusters can occur through compute nodes and / or aggregation nodes. This embodiment uses aggregation nodes for data interaction as an example. Figure 5 For the aggregation node in cluster 1, it can also be used to achieve the following steps:

[0138] S401: Send the data vector to the aggregation node of other clusters.

[0139] Alternatively, S501: Receives data vectors sent by aggregation nodes in other clusters.

[0140] In other words, the aggregator node can send the data vector it obtains to aggregator nodes in other clusters, and it can also receive data vectors sent by aggregator nodes in other clusters. The system architecture provided in this solution offers interaction interfaces between different clusters.

[0141] Based on any of the foregoing embodiments, before the computing node performs operations on the data vector according to the operator to obtain the processed data vector, if the data vector corresponding to the vector key value in at least one local partition is an encrypted vector, then it is necessary to decrypt the data vector corresponding to the vector key value in at least one local partition using a pre-stored key before performing the calculation to obtain the decrypted data vector.

[0142] Correspondingly, after the computing node performs operations on the data vector according to the operator to obtain the processed data vector, it can also encrypt the processed data vector using a pre-stored key to obtain the final processed data vector.

[0143] Based on the above embodiments, the following uses a federated learning scenario as an example to provide a detailed description of the distributed vector processing scheme provided by the present invention through specific examples.

[0144] With the development of internet and big data technologies, federated learning is widely used for data analysis and computation in fields such as joint risk control, auto insurance pricing, e-commerce data processing, user data analysis, smart medical diagnosis, and smart cities. However, due to the massive amounts of data required in these scenarios, the current single-machine development approach cannot be directly ported and is inefficient. This solution, however, provides a distributed vector processing method that can handle the large-scale data vector operations in federated learning and can be directly migrated to a distributed version for application. Its development and processing efficiency are superior to existing technologies.

[0145] Specifically, such as Figure 1 In the architecture shown, the aggregation node is primarily used to manage the data in the current cluster and execute computation commands. It can also send and receive data from another party. The compute nodes are used to compute data for each partition. In addition, the compute nodes in each cluster are also used to encrypt and decrypt data vectors according to stored keys to obtain processed data vectors.

[0146] Since data in general application scenarios can be converted into a two-dimensional vector representation, the distributed vector here specifically refers to a two-dimensional vector. The data distribution rules for distributed vectors in a cluster are as follows: First, both clusters must agree on the same number of partitions, for example, n. Specifically, when the total size of the stored data is a rows and b columns, the aggregation node can divide the complete data into n equal parts.

[0147] Then, the aggregation node determines the vector key value for each piece of data according to preset rules. For example, the start and end row numbers of the original dataset are used to form a tuple as the key (an identifier, i.e., the key value in the aforementioned scheme). The specific split dataset can then be treated as a vector called a data vector (or simply a value). For example, a dataset with 10,000 rows and 20 columns, split into 4 partitions, would have the key (0, 2500) for the first partition, (2500, 5000) for the second partition, and so on. Alternatively, the split datasets (i.e., data vectors) can be numbered according to the order of the split. For example, a dataset with 10,000 rows and 20 columns, split into 4 partitions, could have the key of the first partition set to 1, the key of the second partition set to 2, the key of the third partition set to 3, and so on, to obtain the vector key value for each partition.

[0148] After obtaining the key-value pair of the data vector corresponding to each partition, the hash value of each key is calculated and then modulo the number of partitions to obtain the final partition identifier (id), i.e., partition id = hash(key) % n. This partition id and the corresponding key (or key-value pair) of the data vector are mainly for the convenience of the aggregation node in managing the data in the cluster.

[0149] In the solution provided by this invention, distributed vectors are capable of concurrently executing multiple local data vectors, with each partition's execution equivalent to a single local data vector. The data vector design differs for federated learning scenarios, supporting two types of data vectors: unencrypted vectors and encrypted vectors. Besides the basic vector's storage data and dimension member variables, both types of vectors also have a password management variable storing the same public or private key information, used for encrypting and decrypting the two types of local data vectors. In specific implementations, for encrypted vectors that do not require decryption, only one public key needs to be stored. The public and private keys differ depending on the specific encryption algorithm used. Taking the homomorphic encryption algorithm Paillier's fast key generation method as an example, the public key contains two elements (n, g), where n = p * q, g = n + 1, and p and q are two large prime numbers with a length of at least 512 bits. The private key contains two elements (λ, μ), where λ = (p - 1) * (q - 1), and μ = ((p - 1) * (q - 1) - 1) % n.

[0150] The encryption algorithm used in federated learning is homomorphic encryption, but it's not limited to this type. Encryption and decryption are performed on each value of the data vector, without changing the dimension of the data vector. Local vectors support common vector operations, such as quadratic operations, slicing, and matrix multiplication. Furthermore, homomorphically encrypted data vectors also support operations, and the calculation process for an encrypted vector and a non-encrypted vector involves converting the non-encrypted vector into an encrypted vector before performing the operation; the final result is an encrypted vector.

[0151] The programming interface for distributed vectors should be as consistent as possible with that for local vectors. This solution also incorporates some extensions based on the characteristics of federated learning:

[0152] 1. Multi-party data transmission interface push / pull commands (data transmission between clusters).

[0153] based on Figure 4 In the illustrated embodiment, the push / pull command is primarily used for data transfer between clusters.

[0154] Specifically, "push" refers to sending data from one party to another, while "pull" refers to retrieving data sent by another party. Figure 1 Taking the cluster on the left as the guest and the cluster on the right as the host as an example, when the guest executes a push command, data is sent to the host's aggregation node through the guest's aggregation node. The host's aggregation node then splits and distributes the data to the disks of each compute node in the host cluster. When the host executes a pull command, each compute node accesses its current disk. If data is found, the transfer is successful. If no data is found, the host enters a blocked state, waiting for the guest to send data.

[0155] II. Data transmission interface within the cluster: put / get commands (data transmission within the cluster)

[0156] based on Figure 2 or Figure 3 In the illustrated embodiment, the put / get command is primarily used for data transfer between clusters.

[0157] Specifically, `put` and `get` are cluster-level commands. The `put` command represents the data from the aggregation node as a local vector, splits it into multiple data nodes, and distributes them to the compute nodes, following the same distribution rules as mentioned above. The `get` command sends the data vectors from each partition to the aggregation node, merging them into local vectors. These two interfaces are often used to assist in algorithm development. When developing algorithms using distributed vectors, experiments are often conducted with a small amount of data; the `put` interface can quickly prepare distributed data. To verify the correctness of the distributed vector calculation results, the `get` interface can be used to pull the distributed data vectors to the aggregation node, representing them as local vectors, allowing you to check the correctness of the results.

[0158] III. Calculation Interface

[0159] Figure 5 This is a schematic diagram of the computing interface in the distributed vector processing system provided by the present invention. Figure 5 As shown, distributed vector computation refers to the computation of data vectors within a cluster. Distributed vector computation follows the underlying map-reduce approach, and is generally applicable... Figure 5 Vector operations can be broadly categorized into several types, including unary operations, binary operations, and aggregation algorithms. Common unary operations include calculating the absolute value (abs), extracting the square root (sqrt), calculating the step function (sign), calculating the logarithm (log), negation, and encryption / decryption. In a unary operation, the input parameters are only the current lvalue variable, which is the key of the data vector (i.e., the name of the data vector), and the specified operator. The vector key and operator are sent to each computation node. Upon receiving the operator, each computation node loads the data vector from its disk, performs the corresponding operation, generates a new local data vector, and stores it on disk, thus completing the unary operation.

[0160] Binary operations include addition, subtraction, multiplication, division, modulo, and idempotency. Compared to unary operations, binary operations require an additional input parameter: an rvalue variable. This rvalue variable can be a single value, a vector, or a distributed vector. If the rvalue variable is a single value, such as rtw1 = rwt + 1, where variables starting with rtw represent a distributed data vector, then the lvalue variable, operator, and rvalue variable are directly sent to the compute node. The compute node loads the data vector from disk, performs operations with the rvalue variable, and generates a new data vector. If the rvalue is a local vector, such as rtw1 = rtw + loc, where variables starting with loc represent another local data vector, then this data vector is serialized into binary before the lvalue variable (i.e., the key of the data vector on the compute node), operator, and the serialized rvalue are sent to the compute node. Upon receiving this, the compute node deserializes the rvalue into a data vector, performs operations with the lvalue and rvalue, and generates a new data vector. If the rvalue is also a distributed vector, such as rtw2 = rtw + rtw1, then the key (name of one data vector) and operator of the lvalue variable, and the key (name of another data vector) of the rvalue variable, are sent to the computing nodes. Each computing node, upon receiving these, loads the lvalue and rvalue data vectors from disk based on the distributed vector keys, performs the corresponding operations, and generates a new data vector. Finally, the resulting data vector is stored on disk, thus completing the distributed binary operation.

[0161] Aggregation operations include summation (sum) and mean (mean). Summing by row can be categorized as a unary operation, but it's rarely used in practice; summing by column is more common. The input parameter is the lvalue sum operator. Upon receiving this, the compute nodes load the data vector from disk, sum or mean by column, resulting in a new data vector with only one row. Each compute node serializes this data vector before sending it back to the aggregate node. The aggregate node, after receiving all the data vectors, sums or averages them all, and the resulting local vector is the final result.

[0162] In summary, this invention designs a distributed vector architecture and processing scheme suitable for federated learning scenarios. It retains the computation interface of local vectors but performs parallel data storage and computation to achieve distributed computing. This significantly reduces the development difficulty for developers and solves the problem of processing large amounts of data in federated learning scenarios. Versions developed using this architecture can be directly migrated to distributed versions for application.

[0163] Figure 6 This is a schematic diagram of the structure of a first embodiment of the distributed vector processing device provided by the present invention, as shown below. Figure 6 As shown, the distributed vector processing device 10 includes:

[0164] Receiving module 11 is used to acquire calculation instructions, which are used to instruct calculations on data vectors in the cluster. The calculation instructions include operation type and data identifier.

[0165] Processing module 12 is used to determine an operation instruction based on the operation type and the data identifier, wherein the operation instruction includes an operator and a vector key value;

[0166] The sending module 13 is used to send the operation instructions to each computing node in the cluster.

[0167] The distributed vector processing device 10 provided in this embodiment is used to execute the technical solution of the aggregation node in each cluster in the aforementioned system and method embodiments. Its implementation principle and technical effect are similar. By managing each computing node in the cluster, data communication and control of each computing node to perform parallel storage and computation of data are realized. Local algorithms can be quickly ported without a distributed foundation to realize distributed computing, which can accelerate the development efficiency of federated algorithms.

[0168] Based on the above embodiments, if the operation type is a binary operation, the operation instruction further includes an lvalue variable and an rvalue variable.

[0169] Optionally, the processing module 12 is further configured to:

[0170] Based on the number of data partitions in the cluster, the received data to be stored is split to obtain multiple data vectors;

[0171] The vector key value corresponding to each data vector is determined according to preset rules;

[0172] The sending module 13 is further configured to send the plurality of data vectors and vector key values ​​to each computing node in the cluster, respectively, according to the number of data partitions on each computing node.

[0173] Optionally, the processing module 12 is further configured to:

[0174] Read the data vectors of each data partition in each computing node of the cluster to obtain multiple data vectors;

[0175] The multiple data vectors are merged based on vector keys to obtain a local vector.

[0176] Optional,

[0177] The sending module is also used to send data vectors to aggregation nodes of other clusters;

[0178] or,

[0179] The receiving module is also used to receive data vectors sent by aggregation nodes of other clusters.

[0180] Optionally, the data vector is a data vector in a federated learning scenario.

[0181] The distributed vector processing device 10 provided in this embodiment is used to execute the technical solution of the aggregation node in each cluster in the aforementioned system and method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0182] Figure 7 This is a schematic diagram of the structure of a second embodiment of the distributed vector processing device provided by the present invention, as shown below. Figure 7 As shown, the distributed vector processing device 20 includes:

[0183] The receiving module 21 is used to receive the operation instructions sent by the aggregation node of the cluster, wherein the operation instructions include operators and vector key values;

[0184] The processing module 22 is used to perform operations on the data vector corresponding to the vector key value in at least one local partition according to the operator, so as to obtain the processed data vector.

[0185] The distributed vector processing device 20 provided in this embodiment is used to execute the technical solution of the computing nodes in each cluster in the aforementioned system and method embodiments. Its implementation principle and technical effect are similar. Each computing node in each cluster can perform parallel storage and computation of data. It can quickly port local algorithms without a distributed foundation to achieve distributed computing and can accelerate the development efficiency of federated algorithms.

[0186] Based on the above embodiments, in the specific implementation of the distributed vector processing device 20, if the operation type is a binary operation, the operation instruction further includes an lvalue variable and an rvalue variable;

[0187] Accordingly, the processing module 22 is specifically used for:

[0188] Based on the operator, the lvalue variable, and the rvalue variable, the data vector corresponding to the vector key value in at least one local partition is operated on to obtain the operated data vector.

[0189] Optionally, the receiving module 21 is further configured to receive at least one data vector and a vector key value of each data vector sent by the aggregation node;

[0190] The processing module 22 is further configured to store the at least one data vector into at least one data partition based on a vector key, wherein each data partition stores one data vector.

[0191] Optionally, the processing module 22 is further configured to: if the data vector corresponding to the vector key value in at least one local partition is an encrypted vector, then use a pre-stored key to decrypt the data vector corresponding to the vector key value in at least one local partition to obtain a decrypted data vector.

[0192] Optionally, the processing module is further configured to:

[0193] The processed data vector is encrypted using a pre-stored key.

[0194] Optionally, the data vector is a data vector in a federated learning scenario.

[0195] The distributed vector processing device 20 provided in this embodiment is used to execute the technical solutions of computing nodes in each cluster in the aforementioned system and method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0196] In this invention, it should be understood that each computing node or aggregation node in the cluster can be implemented by a computer device, such as a computing server, or different functions can be implemented on different devices. These devices, as a whole, implement the functions of one or more nodes and can be collectively referred to as electronic devices. This solution does not limit this.

[0197] Figure 8 This is a schematic diagram of the structure of an electronic device according to a first embodiment of the present invention, as shown below. Figure 8 As shown, the electronic device 100 includes:

[0198] The system includes a memory 111, a processor 112, and an interaction interface 113. The memory 111 stores a computer program that can run on the processor 112. When the computer program is executed by the processor 112, it implements the method steps of the aggregation node or the computing node in any of the foregoing method embodiments.

[0199] Optionally, the various devices of the electronic device 100 can be connected to each other via bus 114.

[0200] The memory 111 can be a separate storage unit or a storage unit integrated into the processor 112. The number of processors 112 can be one or more.

[0201] In the above implementation of the electronic device 100, the memory 111 and the processor 112 are electrically connected directly or indirectly to realize data transmission or interaction. That is, the memory 111 and the processor 112 can be connected through an interface or integrated together. For example, these components can be electrically connected to each other through one or more communication buses or signal lines, such as through bus connections. The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The memory is used to store programs, and the processor executes the program after receiving execution instructions. Furthermore, the software programs and modules within the aforementioned memory may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.) and may communicate with various hardware or software components to provide an operating environment for other software components.

[0202] Processor 112 can be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processor, etc., which can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0203] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the distributed vector processing method provided in any of the foregoing method embodiments.

[0204] The present invention also provides a computer program product, including a computer program, which is executed by a processor to implement the steps of the distributed vector processing method provided in any of the foregoing method embodiments.

[0205] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0206] The sequence numbers of the above embodiments of the present invention are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause an electronic device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0207] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A distributed vector processing system, characterized in that, include: At least two clusters, each cluster comprising multiple nodes, wherein the number of nodes in each of the at least two clusters is the same; In each cluster, the multiple nodes include two types of nodes: aggregation nodes and compute nodes; In each cluster, the aggregation node is used to manage the data in its cluster, and the compute node is used to store and compute data in at least one partition. Specifically, the aggregation nodes in each cluster are used for: Obtain a computation instruction, the computation instruction being used to instruct computation on a data vector in the cluster, the computation instruction including an operation type and a data identifier; Based on the operation type and the data identifier, an operation instruction is determined, wherein the operation instruction includes an operator and a vector key value; The computation instructions are sent to each computing node in the cluster respectively; Accordingly, the compute nodes in each cluster are specifically used for: Receive computation instructions sent by the aggregation node of the cluster, wherein the computation instructions include operators and vector key values; According to the operator, the data vector corresponding to the vector key value in at least one local partition is operated on to obtain the operated data vector; The aggregation nodes in each cluster are also used for: Based on the number of data partitions in the cluster, the received data to be stored is split to obtain multiple data vectors; The vector key value corresponding to each data vector is determined according to preset rules; Based on the number of data partitions on each computing node, the multiple data vectors and vector key values ​​are sent to each computing node in the cluster respectively; Correspondingly, the compute nodes in each cluster are also used for: Receive at least one data vector and a vector key value for each data vector sent by the aggregation node; The at least one data vector is stored in at least one data partition based on a vector key, wherein each data partition stores one data vector.

2. The system according to claim 1, characterized in that, The aggregation node in each cluster is also used for: Read the data vectors of each data partition in each computing node of the cluster to obtain multiple data vectors; The multiple data vectors are merged based on vector keys to obtain a local vector.

3. The system according to claim 1, characterized in that, The aggregation node in each cluster is also used for: Send the data vector to the aggregation node of other clusters; or, Receive data vectors sent by aggregation nodes in other clusters.

4. The system according to claim 1, characterized in that, The compute nodes in each cluster are also used for: The data vector is encrypted and decrypted using the stored key to obtain the processed data vector.

5. The system according to any one of claims 1 to 4, characterized in that, The distributed vector processing system is used to process data vectors in federated learning scenarios.

6. A distributed vector processing method, characterized in that, The method, applied to an aggregation node of any cluster in the distributed vector processing system according to any one of claims 1 to 5, comprises: Obtain a computation instruction, the computation instruction being used to instruct computation on a data vector in the cluster, the computation instruction including an operation type and a data identifier; Based on the operation type and the data identifier, an operation instruction is determined, wherein the operation instruction includes an operator and a vector key value; The computation instructions are sent to each computing node in the cluster.

7. The method according to claim 6, characterized in that, If the operation type is a binary operation, the operation instruction also includes an lvalue variable and an rvalue variable.

8. The method according to claim 6 or 7, characterized in that, The method further includes: Based on the number of data partitions in the cluster, the received data to be stored is split to obtain multiple data vectors; The vector key value corresponding to each data vector is determined according to preset rules; Based on the number of data partitions on each computing node, the multiple data vectors and vector key values ​​are sent to each computing node in the cluster.

9. The method according to claim 8, characterized in that, The method further includes: Read the data vectors of each data partition in each computing node of the cluster to obtain multiple data vectors; The multiple data vectors are merged based on vector keys to obtain a local vector.

10. The method according to claim 6 or 7, characterized in that, The method further includes: Send the data vector to the aggregation node of other clusters; or, Receive data vectors sent by aggregation nodes in other clusters.

11. The method according to claim 6 or 7, characterized in that, The data vector refers to the data vector in the federated learning scenario.

12. A distributed vector processing method, characterized in that, The method, applied to any computing node of any cluster in the distributed vector processing system according to any one of claims 1 to 5, comprises: Receive computation instructions sent by the aggregation node of the cluster, wherein the computation instructions include operators and vector key values; According to the operator, the data vector corresponding to the vector key value in at least one local partition is operated on to obtain the operated data vector.

13. The method according to claim 12, characterized in that, If the operation type is a binary operation, the operation instruction also includes an lvalue variable and an rvalue variable; Accordingly, the step of performing operations on the data vector corresponding to the vector key value in at least one local partition according to the operator to obtain the processed data vector includes: Based on the operator, the lvalue variable, and the rvalue variable, the data vector corresponding to the vector key value in at least one local partition is operated on to obtain the operated data vector.

14. The method according to claim 12 or 13, characterized in that, The method further includes: Receive at least one data vector and a vector key value for each data vector sent by the aggregation node; The at least one data vector is stored in at least one data partition based on a vector key, wherein each data partition stores one data vector.

15. The method according to claim 12 or 13, characterized in that, Before performing the operation on the data vector corresponding to the vector key value in at least one local partition according to the operator to obtain the operated data vector, the method further includes: If the data vector corresponding to the vector key value in at least one local partition is an encrypted vector, then the data vector corresponding to the vector key value in at least one local partition is decrypted using a pre-stored key to obtain the decrypted data vector.

16. The method according to claim 15, characterized in that, The method further includes: The processed data vector is encrypted using a pre-stored key.

17. The method according to claim 12 or 13, characterized in that, The data vector refers to the data vector in the federated learning scenario.

18. A distributed vector processing device, characterized in that, An aggregation node applied to any cluster in the distributed vector processing system according to any one of claims 1 to 5, comprising: A receiving module is used to acquire calculation instructions, which are used to instruct the calculation of data vectors in the cluster. The calculation instructions include operation type and data identifier. The processing module is used to determine the operation instruction based on the operation type and the data identifier, wherein the operation instruction includes an operator and a vector key value; The sending module is used to send the operation instructions to each computing node in the cluster.

19. A distributed vector processing device, characterized in that, A computing node applied to any cluster in the distributed vector processing system according to any one of claims 1 to 5, comprising: The receiving module is used to receive computation instructions sent by the aggregation nodes of the cluster, wherein the computation instructions include operators and vector key values; The processing module is used to perform operations on the data vector corresponding to the vector key value in at least one local partition according to the operator, so as to obtain the processed data vector.

20. An electronic device, characterized in that, include: The system includes a memory, a processor, and an interface, wherein the memory stores a computer program that can run on the processor, and the computer program, when executed by the processor, implements the steps of the distributed vector processing method as described in any one of claims 6 to 17.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the distributed vector processing method as described in any one of claims 6 to 17.

22. A computer program product, characterized in that, It includes a computer program, which is executed by a processor to implement the steps of the distributed vector processing method according to any one of claims 6 to 17.

Images