A blockchain-fused data distribution and computing method and system

By dividing data into blocks within a GPU server cluster and using improved Merkle tree rules to verify data integrity, the performance bottleneck of a single server processing large-scale data is resolved, achieving efficient data computation and resource sharing.

CN119105862BActive Publication Date: 2025-11-25RUNJIAN COMM +3
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411044336.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2025-11-25
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

In existing technologies, single servers face high performance requirements or are prone to crashing when processing large-scale data, making it difficult to effectively handle large-scale data.

Method used

A cluster consisting of n GPU servers is used. A root server is selected, the data is divided into n data blocks, and a verification blockchain is formed using an improved Merkle tree rule to verify the integrity and correctness of the data blocks. Each GPU server performs calculations and returns the results to the root server.

Benefits of technology

It enables efficient processing of large-scale data, utilizes shared server computing resources, and ensures the success and accuracy of data computation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119105862B_ABST
    Figure CN119105862B_ABST
Patent Text Reader

Abstract

The application provides a kind of fusion blockchain's data distribution and computing method and system, method includes: selecting a GPU server as root server from cluster server;In root server, the data to be processed is divided into n data blocks;According to the performance of each GPU server, the corresponding data block is subjected to first reduction operation and distributed to the corresponding GPU server, and is packaged to form a check block chain by improved merkle tree rule, check block chain is used to check the integrity and correctness of the data block received by each GPU server;Each GPU server calculates the data block that passes the check, and returns the calculation result to the root server after the second reduction operation.The large-scale data is stored in the root server in the application, and when data processing is performed, the data stored in the root server is sliced and distributed to each server for server computing resource sharing and cluster computing.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data distribution processing, and more particularly, to a data distribution and calculation method and system fusing a blockchain. BACKGROUND

[0002] With the rapid development of data, the amount of data that needs to be calculated and processed is increasing. If a server is used to process large-scale data, the performance requirement of the server is very high, or in other words, the server may directly crash. How to solve this problem is worth exploring. SUMMARY

[0003] The present application provides a data distribution and calculation method and system fusing a blockchain to solve the technical problems in the prior art.

[0004] According to a first aspect of the present application, a data distribution and calculation method fusing a blockchain is provided, comprising:

[0005] selecting one GPU server from a cluster server composed of n GPU servers as a root server, n being a positive integer;

[0006] dividing the to-be-processed data into n data blocks corresponding to the n GPU servers on the root server;

[0007] performing a first reduction operation on the corresponding data blocks according to the performance of each GPU server, distributing the data blocks after the first reduction operation to the corresponding GPU servers, and packing to form a check blockchain through an improved Merkle tree rule, the check blockchain being used to check the integrity and correctness of the data blocks received by each GPU server;

[0008] each GPU server performs calculation on the data blocks that pass the check, and returns the calculation result after a second reduction operation to the root server.

[0009] According to a second aspect of the present application, a data distribution and calculation system fusing a blockchain is provided, comprising:

[0010] a determination module for determining one GPU server from a cluster server composed of n GPU servers as a root server, n being a positive integer;

[0011] a division module for dividing the to-be-processed data into n data blocks corresponding to the n GPU servers on the root server;

[0012] The distribution module is used for performing a first reduction operation on the corresponding data block according to the performance of each GPU server, distributing the data block after the first reduction operation to the corresponding GPU server, and packing a check block chain through an improved Merkle tree rule, and the check block chain is used for checking the integrity and correctness of the data block received by each GPU server.

[0013] The processing module is used for each GPU server to perform calculation on the data block that passes the check, and returns the calculation result after a second reduction operation to the root server.

[0014] The application provides a data distribution and calculation method and system combined with a block chain, which stores large-scale data on a root server, slices the data stored on the root server when performing data processing, and distributes the data to each server for server computing resource sharing and cluster training. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 The application provides a data distribution and calculation method combined with a block chain.

[0016] Figure 2 The application provides an interaction diagram of multiple GPU servers.

[0017] Figure 3 The application provides a check block chain diagram.

[0018] Figure 4 The application provides a whole process diagram of the data distribution and calculation method combined with a block chain.

[0019] Figure 5 The application provides a data distribution and calculation system structure diagram combined with a block chain. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application. In addition, the technical features in each embodiment or single embodiment provided by the application can be combined with each other to form a feasible technical scheme, and this combination is not restricted by the order of steps and / or structure mode, but should be based on the realization by those skilled in the art, and when the combination of technical schemes appears contradictory or unfeasible, it should be considered that the combination of technical schemes does not exist and is not within the protection scope of the application.

[0021] Figure 1 A fusion block chain data allocation and calculation method flow chart is provided for the present application, as shown in Figure 1 The method comprises the following steps:

[0022] Step 1: Select a GPU server as a root server from a cluster server composed of n GPU servers, n is a positive integer.

[0023] It can be understood that all servers are numbered one by one starting from the root server to form a numbered server sequence. For a server cluster G with n GPU servers, {GPU1, GPU2,..., GPU n}, n = 1, 2, 3,..., in order to realize the rapid central aggregation effect of the root server, the root server is selected according to the middle bit of the number of the server cluster G, that is, GPU j is the root server, wherein, n is an odd number, n is an even number, and the rest are sub-servers. The data interaction diagram between the root server and each sub-server is shown in Figure 2 The root server sends data to each sub-server, and each sub-server returns the calculation result to the root server for summarization after calculating the data.

[0024] It should be noted that for a large server cluster organized across regions and media, such as a server cluster with thousands of servers distributed in multiple regions, there are both virtual servers in the cloud and physical media servers. The present application links the cross-regional physical media servers and virtual servers together in a consecutive numbering manner to form a larger server cluster, forming a unified server cluster indexed by numbering across regions. Wherein {GPU1,..., GPU m}, m = 1, 2, 3,..., is a physical media server cluster in m region, {GPU m+1 ,..., GPU n}, m, n = 1, 2, 3,..., n ≥ m + 1 is a cloud storage server cluster distributed in n regions, then the two server clusters are numbered consecutively in a consecutive numbering manner, {GPU1,..., GPU m , GPU m+1 ,..., GPU n}, m, n = 1, 2, 3,..., n ≥ m + 1. According to the method of the present application, the root server is GPU j , wherein, n is an odd number, n is an even number, and the rest are sub-servers.

[0025] Step 2, dividing the to-be-processed data on the root server into n data blocks corresponding to n GPU server.

[0026] It can be understood that, on the root server deployed with a large model, in order to improve the calculation training efficiency and realize the sharing of computing power, the to-be-processed data on the root server is divided into n data blocks. Among them, the to-be-processed data can be divided into data blocks of equal size by the average method. For the to-be-processed data D on the root server, it is divided into D i =D / n, i=1, 2, …, n according to the number n of GPU servers in the server cluster.

[0027] The to-be-processed data can also be divided according to the comprehensive performance of each GPU, wherein dividing the to-be-processed data according to the comprehensive performance of each GPU server includes: calculating the comprehensive performance of each GPU server according to the response time, transmission rate, computing power and required cost of each GPU server:

[0028] P j =D / n, i=1, 2, …, n according to the number n of GPU servers in the server cluster. j ×v j ×p j ×e j ;

[0029] Wherein, P j represents the comprehensive performance of the jth GPU server, t j represents the response time of the jth GPU server, v j represents the transmission rate of the jth GPU server, p j represents the computing power of the jth GPU server, and e j represents the cost required by the jth GPU server.

[0030] According to the comprehensive performance of each GPU server, the to-be-processed data is divided into n data blocks in proportion.

[0031] A specific application scenario is described. In bank business processing, multiple data blocks are usually allocated to GPU servers with different processing capabilities for calculation according to the different sizes of customer business. For the data block set {D1, D2, …, D n} arranged in descending order according to the data size, the comprehensive validating algorithm of server (CVAS) is defined as P=1 / n*t*v*p*e, n is the number of servers in the server cluster, t represents the response time of the GPU server, v represents the transmission rate of the server, p represents the computing power of the server, and e represents the cost required by the server.

[0032] The comprehensive ability of each server is calculated by the comprehensive ability evaluation algorithm of the GPU server, and the servers are ranked in descending order, {GPU1, GPU2, …, GPU n When the data blocks are allocated, D1 is allocated to GPU1, D2 is allocated to GPU2, and so on, and D n is allocated to GPU n .

[0033] Step 3: According to the performance of each GPU server, the corresponding data block is subjected to a first reduction operation, the data block after the first reduction operation is distributed to the corresponding GPU server, and the improved Merkel tree rule is used to package to form a check block chain, the check block chain is used to check the integrity and correctness of the data block received by each GPU server.

[0034] It can be understood that after step S2 divides and allocates the GPU servers to the data to be processed, the corresponding data block is subjected to a first reduction operation according to the performance of each GPU server, so as to reduce the influence of invalid and error data, reduce the data storage amount, and improve the calculation time.

[0035] For the servers GPU i i=1, 2, …, n in the server cluster, the summation reduction operation operator can represent the weight balance of the summation reduction operation on the data of any server in the server cluster, and the first reduction operation operator can be defined as t j represents the response time of the jth GPU server, v j represents the transmission rate of the jth GPU server, a j represents the first reduction operation operator of the jth GPU server, and the data amount of the data block after the first reduction operation is: D i is the data amount of the ith data block after the first reduction operation. i

[0036] The data block after the first reduction operation is distributed to the corresponding GPU server.

[0037] ​In the process of distributing the divided data blocks to the corresponding GPU servers, the data blocks are encrypted by an asymmetric encryption method before being distributed. The common RSA asymmetric encryption method is adopted, in which the public key is distributed to the entire network, and the private key is kept by each GPU server. After each GPU server receives the data block, it is decrypted with the private key kept by itself. After the data block is encrypted, the evenly divided encrypted data block after the summation operation is distributed to all servers in the server cluster in the order of server number for server computing power sharing. According to the distribution result of the distributed server, the server number and the distribution time are packaged according to the improved Merkle tree (Advanced Merkle Trees) rule to form a check block chain.

[0038] wherein W i represents the divided data block D i distributed to the corresponding server GPU j GPU j whether the distribution is successful, which takes the value of 0 or 1. If it is 0, it means that the corresponding data block distribution is successful, and the corresponding GPU server can start computing. If it is 1, it means that the corresponding data block distribution fails, and the corresponding GPU server cannot start computing.

[0039] wherein W i = {0, 1 | (D i → GPU j ), i = 1, 2, …, n.

[0040] The value of W i is packaged together with the GPU server number j and the response time T i to form a check block C i = {W i , T i , j}, i = 1, 2, …, n.

[0041] According to the improved Merkle tree (Advanced Merkle Trees) rule, a check block chain is created, which can be referred to in Figure 3 Each check block node is the sum of its own and the hash of the previous child check block node, that is:

[0042] C i = {h(C i-1 ) + h(C i )}, i = 1, 2, …, n, h represents hash calculation.

[0043] All check block links form a check block chain.

[0044] C = {C i}, i = 1, 2, …, n.

[0045] When the GPU server receives the data block, the integrity and correctness of the data in the received data block are checked according to the check block node. If the data block passes the check, the corresponding GPU server is started to perform the calculation work; if the data block fails the check, the corresponding GPU server is closed and does not participate in the calculation work of the data.

[0046] Step 4, each GPU server processes the data block that passes the check, and returns the calculation result to the root server after the second reduction operation.

[0047] It can be understood that after the divided data block is allocated to the corresponding GPU server and passes the data check, the GPU server performs calculation on the received data block to obtain the calculation result. Each GPU server performs a second reduction operation on the calculation result and returns the calculation result after the reduction operation to the root server.

[0048] Suppose the calculation result of each GPU server on the data block is represented as R1, R2,... R j ;

[0049] According to the response time and transmission rate of each GPU server, the second reduction operation operator of each GPU server is calculated Then the calculation result after the second reduction operation is

[0050] It should be noted that when the multiple data blocks {D1, D2,..., D n} in step S2 are distributed to the server cluster {GPU1, GPU2,..., GPU n}, for the allocation scheme of assigning D1 to GPU1, D2 to GPU2,..., and D n to GPU n , it is necessary to verify whether the allocation scheme can be accepted.

[0051] Specifically, in the bank business processing, in order to reduce the calculation result, the result must be verified to ensure the correctness of the result. The practice is that when the above allocation is performed, the "Verification Method of Strong and Weak Relation" (VMSWR) is adopted, that is, the data block D n with the smallest data volume is allocated to the GPU n with the weakest comprehensive ability, and the data block D nD1 is allocated to the GPU 1 with the strongest comprehensive capability, both the largest data block D1 is allocated to the GPU 1 with the strongest comprehensive capability, and D1 is also allocated to the GPU with the weakest comprehensive capability n , the calculated result R n1 and R 1n as the verification result.

[0052] As described above, the allocation scheme of the multiple data blocks to the multiple GPU servers is {D1, D n ->GPU1, D2->GPU2, …, D n , D1->GPU n}, wherein D1, D n ->GPU1 means that D1 and D n are allocated to the GPU 1, D2->GPU2 means that D2 is allocated to the GPU 2, …, D n , D1->GPU n means that D n and D1 are allocated to the GPU n .

[0053] The corresponding calculation results are {R 11 , R 22 , …, R nn , R n1 , R 1n}, R 11 represents the calculation result after the second reduction operation of D1 allocated to the GPU 1, R 22 represents the calculation result after the second reduction operation of D2 allocated to the GPU 2, …, R nn represents the calculation result after the second reduction operation of D n allocated to the GPU n , and R n1 represents the calculation result after the second reduction operation of D n allocated to the GPU 1.

[0054] Based on the data, whether R 11 is consistent with R 1n , and whether R n1 is consistent with R nn are compared, and a verification model of strong and weak relation of servers distributing data (VMSWRSDD) is defined, Po= (|R 11 -R 1n |≤b) & (|R n1 -R nn|≤b), where b is a bias threshold, b can be any number, if |R 11 -R 1n |≤b and (|R n1 -R nn |≤b), that is, when Po=1, it indicates that the current business processing calculation is correct, and the result can be output, then the corresponding data block allocation scheme is acceptable, if one of them is inconsistent, there is an error, that is, Po≠1, then the result cannot be output, and an error is returned, then the data block allocation scheme is also unacceptable, and the data block needs to be allocated again.

[0055] Referring to Figure 4 The overall flowchart of the fusion blockchain-based data allocation and calculation method provided by the application mainly includes the following steps:

[0056] The GPU servers in the server cluster are numbered, and a certain strategy is used to divide the to-be-processed data, wherein the to-be-processed data can be divided in an equal division manner, or the to-be-processed data can be divided according to the performance of different GPU servers in different proportions of data size. After the to-be-processed data is divided, the divided data blocks are reduced and distributed to the corresponding GPU servers. The GPU servers calculate the received data blocks, and return the calculation results to the root server after the calculation results are reduced again.

[0057] Referring to Figure 5 The system structure diagram of the fusion blockchain-based data allocation and calculation system provided by the embodiment of the application, the system includes a determination module 501, a division module 502, a distribution module 503 and a calculation module 504, wherein:

[0058] The determination module 501 is used for determining a GPU server as a root server from a cluster server composed of n GPU servers, and n is a positive integer;

[0059] The division module 502 is used for dividing the to-be-processed data into n data blocks corresponding to the n GPU servers on the root server;

[0060] The distribution module 503 is used for performing a first reduction operation on the corresponding data blocks according to the performance of each GPU server, distributing the data blocks after the first reduction operation to the corresponding GPU servers, and packing a check blockchain through an improved Merkle tree rule, wherein the check blockchain is used for checking the integrity and correctness of the data blocks received by each GPU server;

[0061] The calculation module 504 is used for each GPU server to calculate the data blocks that pass the check, and returns the calculation results to the root server after a second reduction operation.

[0062] It can be understood that the fusion blockchain data distribution and computing system provided by the present application corresponds to the fusion blockchain data distribution and computing method provided by the foregoing embodiments, and the technical features of the fusion blockchain data distribution and computing system can refer to the technical features of the fusion blockchain data distribution and computing method, which will not be repeated here.

[0063] The fusion blockchain data distribution and computing method and system provided by the embodiments of the present application have the following beneficial effects:

[0064] (1) Large-scale data is stored on a root server, and when data processing is performed, the data stored on the root server is sliced and distributed to each server for server computing resource sharing and cluster computing.

[0065] (2) The data can be divided in an equal division manner, or can be divided in an unequal division manner according to the comprehensive performance of different GPU servers, so as to fully utilize the computing performance of different GPU servers, and the distribution scheme is verified to ensure the correctness of the distribution scheme.

[0066] (3) When the data is distributed, the blockchain is used to verify whether the GPU server receives complete and correct data, so as to ensure the success of the final data calculation.

[0067] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in one embodiment can be referred to the relevant description of other embodiments.

[0068] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0069] The present application is described in reference to the flowchart illustrations and / or block diagrams according to the methods, apparatus (systems) and computer program products of embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart Figure 1 one or more of the flowcharts and / or blocks Figure 1 one or more of the flowcharts and / or blocks

[0070] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart Figure 1 one or more of the flowcharts and / or blocks Figure 1 one or more of the flowcharts and / or blocks

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more of the flowcharts and / or blocks Figure 1 one or more of the flowcharts and / or blocks

[0072] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those of skill in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, the attached claims are intended to cover all such additional variations and modifications as falling within the scope of the application.

[0073] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A data allocation and computation method integrating blockchain, characterized in that, include: Select one GPU server from a cluster of n GPU servers as the root server, where n is a positive integer; On the root server, the data to be processed is divided into n data blocks corresponding to n GPU servers; Based on the performance of each GPU server, the corresponding data block is subjected to the first reduction operation. The data block after the first reduction operation is distributed to the corresponding GPU server and packaged into a verification blockchain using improved Merkle tree rules. The verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server. Each GPU server performs calculations on the verified data blocks, performs a second reduction operation on the calculation results, and returns them to the root server. The process of packaging and forming a verification blockchain using improved Merkle tree rules includes: Based on the GPU server's response after distributing data blocks, the GPU server's number, and the data block distribution time, and following the improved Merkel rules, a verification blockchain is formed. Each verification block node is the sum of its own hash and the hash of the previous verification block node. All verification block nodes are linked together to form the verification blockchain. The method further includes: when allocating data blocks, allocating the data block with the smallest data volume to the GPU server with the weakest overall capability, and also allocating the data block with the smallest data volume to the GPU server with the strongest overall capability, and allocating the data block with the largest data volume to the GPU server with the strongest overall capability, and also allocating the data block with the largest data volume to the GPU server with the weakest overall capability, and using the calculated result as the verification result. The step of dividing the data to be processed into n data blocks corresponding to n GPU servers on the root server includes: Calculate the overall capability of each GPU server, and based on the overall capability of the n GPU servers, divide the data to be processed, specifically including: Calculate the overall capability of each GPU server based on its response time, transfer rate, computing power, and required cost: P j =1 / n × t j × v j × p j × e j ; in, P j This represents the overall capability of the j-th GPU server. t j This represents the response time of the j-th GPU server. v j This represents the transmission rate of the j-th GPU server. p j This represents the computing power of the j-th GPU server. e j This represents the cost required for the j-th GPU server; Based on the overall capabilities of each GPU server, the data to be processed is divided into n data blocks of varying sizes.

2. The data allocation and calculation method integrating blockchain according to claim 1, characterized in that, The step of selecting a GPU server as the root server from a cluster of n GPU servers includes: Number the n GPU servers consecutively and take... n is an odd number. If n is even, then the GPU j 'j' is the root server, and 'j' is the root server number. The other GPU servers are sub-servers.

3. The data allocation and calculation method integrating blockchain according to claim 1, characterized in that, If the data to be processed consists of n data blocks to be allocated, then the n data blocks to be allocated are sorted in descending order according to the size of each data block, forming a set of data blocks to be allocated {D1, D2, ..., D...}. n }; The n GPU servers are arranged in descending order according to their overall capabilities, forming a server cluster {GPU1, GPU2, ..., GPU...}. n }; Assign D1 to GPU1, assign D2 to GPU2, ..., and so on. n Allocate to GPU n .

4. The data allocation and calculation method integrating blockchain according to claim 2, characterized in that, The step of performing a first reduction operation on the corresponding data block based on the performance of each GPU server, and distributing the data block after the first reduction operation to the corresponding GPU server includes: Calculate the first reduction operator based on the response time and transmission rate of each GPU server. The data size of the data block after the first reduction operation is: ; in, t j This represents the response time of the j-th GPU server. v j This represents the transmission rate of the j-th GPU server. This represents the first reduction operator of the j-th GPU server. Let i be the data size of the i-th data block after splitting. The data size of the i-th data block after the first reduction operation; Distribute the data blocks after the first reduction operation to the corresponding GPU servers; The step of packaging data blocks into a verification blockchain according to the GPU server's response after the data blocks are distributed, the GPU server's ID, and the data block distribution time, and following the improved Merkel rules, includes: Indicates data speed Distributed to the corresponding GPU server back Whether to respond to the result of successful distribution, its value is 0 or 1. If it is 0, it means that the corresponding data block was successfully distributed and the corresponding GPU server can start computing. If it is 1, it means that the corresponding data block was not distributed and the corresponding GPU server cannot start computing. The value of , along with the GPU server ID j and response time Pack into a verification block ; Each check block node is the sum of its own hash and the hash of the previous check block node, that is: , h Indicates hash calculation; All verification block nodes are linked together to form a verification blockchain: 。 5. The data allocation and calculation method integrating blockchain according to claim 3, characterized in that, Each GPU server performs calculations on the verified data blocks, performs a second reduction operation on the calculation results, and returns them to the root server, including: The computation result of each GPU server for the data block is represented as , ... ; Calculate the second reduction operator for each GPU server based on its response time and transmission rate. The result of the calculation after the second reduction operation is: , .

6. The data allocation and calculation method integrating blockchain according to claim 5, characterized in that, Assigning D1 to GPU1, D2 to GPU2, ..., and so on, assigning D... n While allocating to GPUn, D1 is also allocated to GPU. n , and D n The allocation scheme for data blocks to GPU1 is {D1, D...} n ->GPU1,D2->GPU2,…,D n D1->GPU n }, where D1 and D n ->GPU1 refers to the combination of D1 and D n Assigning to GPU1, D2->GPU2 means assigning D2 to GPU2, ..., D n D1->GPU n This refers to D n and D1 allocated to the GPU n ; The computational data block allocation scheme corresponds to {R} 11 R 22 , ..., R nn R n1 R 1n }, where R 11 R represents the computation result after the second reduction operation that assigns D1 to GPU1. 22 This represents the computation result after the second reduction operation that assigns D2 to GPU2, ..., R nn Indicates that D n Allocate to GPU n The calculation result after the second reduction operation, R n1 Indicates that D n The computation result after the second reduction operation assigned to GPU1; Let Po = (|R) 11 -R 1n |≤b)&(|R n1 -R nn |≤b), where b is the deviation threshold. If Po=1, the data block allocation scheme is acceptable; if Po=0, the data block allocation scheme is unacceptable.

7. A data allocation and computing system integrating blockchain, characterized in that, include: The determination module is used to determine one GPU server as the root server from a cluster of n GPU servers, where n is a positive integer. The partitioning module is used to divide the data to be processed into n data blocks corresponding to n GPU servers on the root server; The distribution module is used to perform a first reduction operation on the corresponding data block according to the performance of each GPU server, distribute the data block after the first reduction operation to the corresponding GPU server, and package it into a verification blockchain using an improved Merkle tree rule. The verification blockchain is used to verify the integrity and correctness of the data block received by each GPU server. The processing module is used by each GPU server to perform calculations on the verified data blocks, and then return the calculation results to the root server after performing a second reduction operation. The distribution module is specifically used for: Based on the GPU server's response after distributing data blocks, the GPU server's number, and the data block distribution time, and following the improved Merkel rules, a verification blockchain is formed. Each verification block node is the sum of its own hash and the hash of the previous verification block node. All verification block nodes are linked together to form the verification blockchain. The system also includes a verification module for: When allocating data blocks, the smallest data block is allocated to the GPU server with the weakest overall capability, and the largest data block is allocated to the GPU server with the strongest overall capability. The calculated result is used as the verification result. The partitioning module is specifically used for: Calculate the overall capability of each GPU server, and based on the overall capability of the n GPU servers, divide the data to be processed, specifically including: Calculate the overall capability of each GPU server based on its response time, transfer rate, computing power, and required cost: P j =1 / n × t j × v j × p j × e j ; in, P j This represents the overall capability of the j-th GPU server. t j This represents the response time of the j-th GPU server. v j This represents the transmission rate of the j-th GPU server. p j This represents the computing power of the j-th GPU server. e j This represents the cost required for the j-th GPU server; Based on the overall capabilities of each GPU server, the data to be processed is divided into n data blocks of varying sizes.

Citation Information

Patent Citations

  • Big data processing method and big data processing device

    CN105488134A

  • Model training method, server and computer readable storage medium

    CN110134636A