Data distribution and calculation method and system fused with block chain

The integration of blockchain with GPU servers for data distribution and calculation addresses the challenge of processing large data volumes by ensuring efficient data integrity and correctness across a cluster, reducing server crashes.

CN119105862AActive Publication Date: 2024-12-10RUNJIAN COMM +3
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202411044336.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-12-10
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

Existing server systems fail to efficiently process large amounts of data without crashing, necessitating high performance requirements or risking server failure.

Method used

A data distribution and calculation method and system integrating blockchain, utilizing a GPU server as a root server, dividing data into blocks, performing reduction operations, and verifying integrity using an improved Merkle tree to distribute and verify data across multiple GPU servers.

Benefits of technology

Enables efficient data processing by sharing server computing power, ensuring data integrity and correctness across a cluster, and reducing the risk of server crashes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119105862A_ABST
    Figure CN119105862A_ABST
Patent Text Reader

Abstract

The invention provides a block chain fused data distribution and calculation method and system. The method comprises the following steps: selecting a GPU server from cluster servers as a root server; dividing the to-be-processed data into n data blocks on the root server; according to the performance of each GPU server, the corresponding data block is subjected to first protocol operation and then distributed to the corresponding GPU server, a verification block chain is formed through packaging of an improved Merkel tree rule, and the verification block chain is used for verifying the integrity and correctness of the data block received by each GPU server; and each GPU server calculates the data blocks passing the verification, performs second protocol operation on the calculation result, and returns the calculation result to the root server. According to the method, large-scale data are stored on the root server, and when data processing is carried out, the data stored on the root server are sliced and then distributed to all the servers, and server computing power resource sharing calling cluster type computing is carried out.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data distribution and processing, and more specifically, to a data distribution and calculation method and system integrating blockchain. Background Art

[0002] With the rapid development of data, the amount of data that needs to be processed is getting larger and larger. If a server is used to process large amounts of data, the performance requirements of the server are very high, or the server is likely to crash directly. How to solve this problem is worth exploring. Summary of the invention

[0003] In view of the technical problems existing in the prior art, the present invention provides a data distribution and calculation method and system integrating blockchain.

[0004] According to a first aspect of the present invention, a data distribution and calculation method integrating blockchain is provided, comprising:

[0005] Select a GPU server as the root server from a cluster server consisting of n GPU servers, where n is a positive integer;

[0006] Divide the data to be processed into n data blocks corresponding to the n GPU servers on the root server;

[0007] Performing a first reduction operation on the corresponding data block according to the performance of each GPU server, distributing the data block after the first reduction operation to the corresponding GPU server, and packaging it to form a verification blockchain through an improved Merkle tree rule, wherein the verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server;

[0008] Each GPU server calculates the data blocks that have passed the verification, performs a second reduction operation on the calculation results, and returns them to the root server.

[0009] According to a second aspect of the present invention, a data distribution and computing system integrating blockchain is provided, comprising:

[0010] A determination module, used to determine a GPU server as a root server from a cluster server consisting of n GPU servers, where n is a positive integer;

[0011] A partitioning module, used for partitioning the data to be processed into n data blocks corresponding to the n GPU servers on the root server;

[0012] A distribution module is used to perform a first reduction operation on the corresponding data block according to the performance of each GPU server, distribute the data block after the first reduction operation to the corresponding GPU server, and package it into a verification blockchain through an improved Merkle tree rule, wherein the verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server;

[0013] The processing module is used for each GPU server to calculate the data blocks that have passed the verification, and return the calculation results to the root server after performing a second reduction operation.

[0014] The present invention provides a data distribution and calculation method and system that integrates blockchain. Large-scale data is stored on a root server. When processing data, the data stored on the root server is sliced ​​and distributed to each server to share server computing power resources and call cluster training. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flowchart of a data allocation and calculation method for integrating blockchain provided by the present invention;

[0016] Figure 2 A schematic diagram of the interaction of multiple GPU servers;

[0017] Figure 3 Schematic diagram for verifying blockchain;

[0018] Figure 4 A schematic diagram of the overall process of the data allocation and calculation method of the integrated blockchain provided by the present invention;

[0019] Figure 5 A structural diagram of a data distribution and computing system integrating blockchain provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0021] Figure 1 A flow chart of a data allocation and calculation method for integrating blockchain provided by the present invention, such as Figure 1 As shown, the method includes:

[0022] Step 1: Select a GPU server as a root server from a cluster server consisting of n GPU servers, where n is a positive integer.

[0023] It is understandable that all servers are numbered consecutively starting from the root server to form a numbered server sequence. For a server cluster G with n GPU servers, {GPU1, GPU2, ..., GPU n}, n = 1, 2, 3, ..., in order to achieve the rapid concentration and aggregation effect of the root server, the root server is selected according to the middle position of the number of server clusters G, that is, GPU j is the root server, where n is an odd number, n is an even number, and the rest are sub-servers. The data interaction diagram between the root server and each sub-server is as follows: Figure 2 As shown, the root server sends data to each sub-server, and each sub-server calculates the data and returns the calculation results to the root server for aggregation.

[0024] It should be noted that, for a large-scale server cluster that is organized across regions and media, such as a server cluster with thousands of servers distributed in multiple regions, with both virtual servers in the cloud and physical media servers, the present invention links the cross-regional physical media servers and virtual servers together in a consecutive numbering manner to form a larger-scale server cluster, forming a unified server cluster with cross-regional servers indexed by numbers. m}, m = 1, 2, 3, ..., is a physical media server cluster distributed in region m, {GPU m+1 ,…,GPU n}, m,n=1,2,3,…,n≥m+1 are cloud storage server clusters distributed in n regions, then the two server clusters are numbered consecutively, {GPU1,…,GPU m ,GPU m+1 ,…,GPU n}, m, n = 1, 2, 3, ..., n ≥ m + 1. According to the method of the present invention, the root server is a GPU j ,in, n is an odd number, n is an even number, and the rest are sub-servers.

[0025] Step 2: divide the data to be processed into n data blocks corresponding to the n GPU servers on the root server.

[0026] It is understandable that on the root server where the large model is deployed, in order to improve the computing training efficiency and realize computing power sharing, the data to be processed on the root server is divided into n data blocks. Among them, the data to be processed can be divided into data blocks of equal size by an average method. For the data to be processed D on the root server, it is divided into D according to the number of GPU servers n in the server cluster G. i =D / n,i=1,2,…,n.

[0027] The data to be processed may also be divided according to the comprehensive performance of each GPU. The division of the data to be processed according to the comprehensive performance of each GPU server includes: calculating the comprehensive capacity of each GPU server according to the response time, transmission rate, computing power and required cost of each GPU server:

[0028] P j =1 / n×t j ×v j ×p j ×e j ;

[0029] Among them, P j represents the comprehensive capability of the jth GPU server, t j represents the response time of the jth GPU server, v j represents the transmission rate of the jth GPU server, p j represents the computing power of the jth GPU server, e j represents the cost required for the j-th GPU server;

[0030] According to the comprehensive capabilities of each GPU server, the data to be processed is divided into n data blocks in proportion.

[0031] Take a specific application scenario as an example. In banking business processing, it is usually necessary to allocate multiple data blocks to GPU servers with different processing capabilities for calculation according to the different business volumes of customers. For a data block set {D1, D2, ..., D n}, define the Comprehensive Validating Algorithm of Server (CVAS), P = 1 / n*t*v*p*e, n is the number of servers in the server cluster, t is the response time of the GPU server, v is the transmission rate of the server, p is the computing power of the server, and e is the cost of using the server.

[0032] The comprehensive capability of each server is calculated according to the comprehensive capability evaluation algorithm of the GPU server, and the servers are arranged in descending order, {GPU1, GPU2, ..., GPU n}. When allocating data blocks, D1 is allocated to GPU1, D2 is allocated to GPU2, and so on. n Assign to GPU n .

[0033] Step 3: Perform the first reduction operation on the corresponding data block according to the performance of each GPU server, distribute the data block after the first reduction operation to the corresponding GPU server, and package it into a verification blockchain through the improved Merkle tree rule. The verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server.

[0034] It is understandable that after the data to be processed is divided and the GPU servers are allocated in step S2, the corresponding data blocks are subjected to the first reduction operation according to the performance of each GPU server to reduce the impact of invalid and erroneous data, reduce the data storage volume, and improve the computing time.

[0035] For server GPUs in a server cluster i , i = 1, 2, ..., n, the sum reduction operator can represent the weight balance of sum reduction operation on the data of any server in the server cluster, then the first reduction operator can be defined as t j represents the response time of the jth GPU server, v j represents the transmission rate of the jth GPU server, a j It represents the first reduction operator of the jth GPU server. The amount of data in the data block after the first reduction operation is: D i is the amount of data in the ith data block after segmentation, D i ' is the data size of the i-th data block after the first reduction operation.

[0036] Distribute the data blocks after the first reduction operation to the corresponding GPU servers.

[0037] In the process of the root server distributing the divided data blocks to the corresponding GPU servers, the data blocks are encrypted by an asymmetric encryption method before they are distributed. The common RSA asymmetric encryption method is used, in which the public key is distributed to the entire network, and the private key is kept by each GPU server. After each GPU server receives the data block, it is decrypted with its own private key. After the data block is encrypted, the encrypted data blocks that have been equally divided after the sum operation are distributed to all servers in the server cluster in the order of server numbers to share server computing power. According to the distribution results of the distributed servers, together with the server number and distribution time, they are packaged according to the rules of the Advanced Merkle Tree to form a verification blockchain.

[0038] Among them, W i Indicates the data after division D i Distribute to the corresponding server GPU j Post-GPU j Whether to respond to the result of successful distribution. The value is 0 or 1. If it is 0, it means that the corresponding data block is distributed successfully and the corresponding GPU server can start computing. If it is 1, it means that the corresponding data block distribution fails and the corresponding GPU server cannot start computing.

[0039] Among them, W i ={0, 1|(D i →GPU j )},i=1,2,…,n.

[0040] W i The value of together with the GPU server number j and the response time T i Packed to form verification block C i = {W i ,T i ,j},i=1,2,…,n.

[0041] According to the Advanced MerkleTrees rules, create a verification blockchain, see Figure 3 , each verification block node is the sum of the hash of itself and the previous child verification block node, that is:

[0042] C i ={h(C i-1 )+h(C i )}, i=1,2,…,n, h represents hash calculation.

[0043] All validation blocks are linked to form a validation blockchain.

[0044] C={C i},i=1,2,…,n.

[0045] When the GPU server receives the data block, it verifies the integrity and correctness of the data in the received data block according to the verification block node. If the data block verification passes, the corresponding GPU server will be turned on for calculation; if the data block verification fails, the corresponding GPU server will be turned off and will not participate in the data calculation.

[0046] Step 4: Each GPU server processes the verified data blocks, performs a second reduction operation on the calculation results, and returns them to the root server.

[0047] It is understandable that the divided data blocks are assigned to the corresponding GPU servers, and after data verification, the GPU servers calculate the received data blocks to obtain calculation results. Each GPU server performs a second reduction operation on the calculation results and returns the calculation results after the reduction operation to the root server.

[0048] Assume that the calculation results of each GPU server on the data block are represented as R1, R2, ...R j ;

[0049] According to the response time and transmission rate of each GPU server, the second protocol operation operator of each GPU server is calculated. Then the calculation result after the second reduction operation is

[0050] It should be noted that in step S2, the multiple data blocks {D1, D2, ..., D n}Distributed to the server cluster {GPU1, GPU2, ..., GPU n}, D1 is assigned to GPU1, D2 is assigned to GPU2, and so on. n Assign to GPU n It is necessary to verify whether the allocation plan is acceptable.

[0051] Specifically, in banking business processing, in order to reduce the calculation results, they must be verified to ensure the correctness of the results. In practice, when allocating the above, the "Verification Method of Strong and Weak Relation" (VMSWR) is adopted, that is, the data block with the smallest data volume D n Assign to the GPU with the weakest overall capability n , and also the data block D with the smallest amount of data nAllocate the largest data block D1 to GPU1 with the strongest comprehensive capability, and also allocate D1 to GPU with the weakest comprehensive capability. n , the calculated result R n1 and R 1n As the verification result.

[0052] As mentioned above, the allocation scheme for allocating multiple data blocks to multiple GPU servers is {D1, D n ->GPU1,D2->GPU2,…,D n 、D1->GPU n}, where D1, D n ->GPU1 refers to the combination of D1 and D n Assign to GPU1, D2->GPU2 means assign D2 to GPU2, ..., D n 、D1->GPU n D n and D1 is assigned to the GPU n .

[0053] The corresponding calculation result is {R 11 , R 22 , ..., R nn , R n1 , R 1n}, R 11 represents the calculation result after the second reduction operation of assigning D1 to GPU1, R 22 represents the calculation result after the second reduction operation of assigning D2 to GPU2, ..., R nn Indicates that D n Assign to GPU n The calculation result after the second reduction operation, R n1 Indicates that D n The calculation result after the second reduction operation assigned to GPU1.

[0054] Based on the data, compared with R 11 Is it related to R 1n Consistent, and R n1 Is it related to R nn Consistent, define and establish the Verification Model of Strong and Weak Relation of Servers Distributing Data (VMSWRSDD), Po=(|R 11 -R 1n |≤b)&(|R n1 -R nn|≤b), where b is the deviation threshold, b can be any number. If |R 11 -R 1n |≤b and (|R n1 -R nn |≤b), that is, when Po=1, it means that the business processing calculation is correct and the result can be output, then the corresponding data block allocation plan is acceptable. If one of the two is inconsistent, there is an error, that is, Po≠1, then the result cannot be output and an error is returned. Then the data block allocation plan is also unacceptable and the data block needs to be reallocated.

[0055] See also Figure 4 , which is a schematic diagram of the overall process of the data allocation and calculation method of the integrated blockchain provided by the present invention, mainly comprising the following steps:

[0056] The GPU servers in the server cluster are numbered and certain strategies are adopted to divide the data to be processed. The data to be processed can be divided equally or according to the performance of different GPU servers and the data volume in different proportions. After the data to be processed is divided, the divided data blocks are reduced and distributed to the corresponding GPU servers. The GPU server calculates the received data blocks, reduces the calculation results again, and returns them to the root server.

[0057] See also Figure 5 , which is a structural diagram of a data distribution and computing system integrating blockchain provided by an embodiment of the present invention, the system includes a determination module 501, a division module 502, a distribution module 503 and a computing module 504, wherein:

[0058] A determination module 501 is used to determine a GPU server as a root server from a cluster server consisting of n GPU servers, where n is a positive integer;

[0059] A division module 502 is used to divide the data to be processed into n data blocks corresponding to the n GPU servers on the root server;

[0060] The distribution module 503 is used to perform a first reduction operation on the corresponding data block according to the performance of each GPU server, distribute the data block after the first reduction operation to the corresponding GPU server, and package it into a verification blockchain through an improved Merkle tree rule, wherein the verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server;

[0061] The calculation module 504 is used for each GPU server to calculate the data block that has passed the verification, and return the calculation result to the root server after performing a second reduction operation.

[0062] It can be understood that the data distribution and calculation system of the fused blockchain provided by the present invention corresponds to the data distribution and calculation method of the fused blockchain provided by the aforementioned embodiments. The relevant technical features of the data distribution and calculation system of the fused blockchain can refer to the relevant technical features of the data distribution and calculation method of the fused blockchain, which will not be repeated here.

[0063] The data distribution and calculation method and system of the integrated blockchain provided by the embodiment of the present invention have the following beneficial effects:

[0064] (1) Large-scale data is stored on the root server. When processing data, the data stored on the root server is sliced ​​and distributed to each server to share server computing resources and call cluster computing;

[0065] (2) The data can be divided equally or unequally according to the comprehensive performance of different GPU servers, making full use of the computing performance of different GPU servers. When allocating data, the allocation plan can be verified to ensure the correctness of the allocation plan.

[0066] (3) When distributing data, the blockchain is verified to verify whether the GPU server has received complete and correct data, ensuring the success of the final data calculation.

[0067] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0068] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0072] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0073] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data distribution and calculation method integrating blockchain, characterized in that: include: Select a GPU server as the root server from a cluster server consisting of n GPU servers, where n is a positive integer; Divide the data to be processed into n data blocks corresponding to the n GPU servers on the root server; Performing a first reduction operation on the corresponding data block according to the performance of each GPU server, distributing the data block after the first reduction operation to the corresponding GPU server, and packaging it to form a verification blockchain through an improved Merkle tree rule, wherein the verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server; Each GPU server calculates the data blocks that have passed the verification, performs a second reduction operation on the calculation results, and returns them to the root server.

2. The data distribution and calculation method of the fusion blockchain according to claim 1 is characterized in that: The step of selecting a GPU server as a root server from a cluster server consisting of n GPU servers includes: Number the n GPU servers consecutively, and take n is an odd number, If n is an even number, then GPU j is the root server, j is the number of the root server, and the remaining GPU servers are sub-servers.

3. The data distribution and calculation method of the fusion blockchain according to claim 1 is characterized in that: The step of dividing the data to be processed into n data blocks corresponding to the n GPU servers on the root server includes: The data to be processed is divided into n data blocks, and the data volume of each data block is D i =D / n,i=1,2,...,n; or, Calculate the comprehensive capacity of each GPU server and divide the data to be processed according to the comprehensive capacity of n GPU servers.

4. The data distribution and calculation method of the fusion blockchain according to claim 3 is characterized in that: The calculating of the comprehensive capability of each GPU server and the dividing of the data to be processed according to the comprehensive capabilities of the n GPU servers include: Based on the response time, transmission rate, computing power and required cost of each GPU server, calculate the comprehensive capacity of each GPU server: P j =1 / n×t j ×v j ×p j ×e j ; Among them, P j represents the comprehensive capability of the jth GPU server, t j represents the response time of the jth GPU server, v j represents the transmission rate of the jth GPU server, p j represents the computing power of the jth GPU server, e j represents the cost required for the j-th GPU server; According to the comprehensive capabilities of each GPU server, the data to be processed is divided into n data blocks with different data sizes.

5. The data distribution and calculation method of the fusion blockchain according to claim 4 is characterized in that: If the data to be processed is n data blocks to be allocated, the n data blocks to be allocated are arranged in descending order according to the data size of each data block to form a set of data blocks to be allocated {D1, D2, ..., D n }; Arrange n GPU servers in descending order according to the comprehensive capabilities of each GPU server to form a server cluster {GPU1, GPU2, ..., GPU n }; Assign D1 to GPU1, D2 to GPU2, and so on. n Assign to GPU n .

6. The data distribution and calculation method of the fusion blockchain according to claim 2 is characterized in that: The method of performing a first reduction operation on the corresponding data block according to the performance of each GPU server and distributing the data block after the first reduction operation to the corresponding GPU server includes: According to the response time and transmission rate of each GPU server, the first reduction operation operator is calculated Then the amount of data in the data block after the first reduction operation is: Among them, t j represents the response time of the jth GPU server, v j represents the transmission rate of the jth GPU server, a j represents the first reduction operator of the jth GPU server, D i is the amount of data in the ith data block after segmentation, D i ' is the data volume of the i-th data block after the first reduction operation; Distribute the data blocks after the first reduction operation to the corresponding GPU server; The improved Merkle tree rule is used to pack and form a verification blockchain, including: According to the response distribution result of the GPU server after distributing the data block, the number of the GPU server and the data block distribution time, the verification blockchain is packaged according to the improved Merkle book rule, where: W i Indicates data fast D i Distribute to the corresponding GPU server GPU j Post-GPU j Whether to respond to the result of successful distribution. The value is 0 or 1. If it is 0, it means that the corresponding data is distributed successfully and the corresponding GPU server can start computing. If it is 1, it means that the corresponding data is distributed unsuccessfully and the corresponding GPU server cannot start computing. W i The value of together with the GPU server number j and the response time T i Packed to form verification block C i = {W i ,j,T i },i=1,2,...,n,j=1,2,...,n; According to the improved Merkle tree rule, each verification block node is the sum of its own hash and the hash of the previous verification block node, that is: C i ={h(C i-1 )+h(C i )}, h represents hash calculation; All verification block nodes are linked to form a verification blockchain: C={C i },i=1,2,...,n。 7. The data distribution and calculation method of the fusion blockchain according to claim 5 is characterized in that: Each GPU server calculates the data block that has passed the verification, performs a second reduction operation on the calculation result, and returns it to the root server, including: The calculation result of each GPU server on the data block is represented as R1, R2, ...R j ; According to the response time and transmission rate of each GPU server, the second protocol operation operator of each GPU server is calculated. Then the calculation result after the second reduction operation is 8. The data distribution and calculation method of the fusion blockchain according to claim 7 is characterized in that: D1 is assigned to GPU1, D2 is assigned to GPU2, and so on. n While assigning it to GPUn, D1 is also assigned to GPU n , and D n Assigned to GPU1, that is, the data block allocation scheme is {D1, D n ->GPU1,D2->GPU2,…,D n 、D1->GPU n }, where D1, D n ->GPU1 refers to the combination of D1 and D n Assign to GPU1, D2->GPU2 means assign D2 to GPU2, ..., D n 、D1->GPU n D n and D1 is assigned to the GPU n ; Calculate the data block allocation scheme corresponding to {R 11 , R 22 , ..., R nn , R n1 , R 1n }, where R 11 represents the calculation result after the second reduction operation of assigning D1 to GPU1, R 22 represents the calculation result after the second reduction operation of assigning D2 to GPU2, ..., R nn Indicates that D n Assign to GPU n The calculation result after the second reduction operation, R n1 Indicates that D n The calculation result after the second reduction operation assigned to GPU1; Let Po = (|R 11 -R 1n |≤b)&(|R n1 -R nn |≤b), where b is the deviation threshold. If Po=1, the data block allocation scheme is acceptable. If Po=0, the data block allocation scheme is unacceptable.

9. A data distribution and computing system integrating blockchain, characterized in that: include: A determination module, used to determine a GPU server as a root server from a cluster server consisting of n GPU servers, where n is a positive integer; A partitioning module, used for partitioning the data to be processed into n data blocks corresponding to the n GPU servers on the root server; A distribution module is used to perform a first reduction operation on the corresponding data block according to the performance of each GPU server, distribute the data block after the first reduction operation to the corresponding GPU server, and package it into a verification blockchain through an improved Merkle tree rule, wherein the verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server; The processing module is used for each GPU server to calculate the data blocks that have passed the verification, and return the calculation results to the root server after performing a second reduction operation.

Citation Information

Patent Citations

  • Method and system for data communication between cluster system devices

    CN105183692A

  • Big data processing method and big data processing device

    CN105488134A

  • Data processing method and apparatus

    CN107040582A

  • Metadata verification method, system server and computer readable storage medium

    CN109657499A

  • Model training method, server and computer readable storage medium

    CN110134636A