Storage method and system for large-scale risk data

Abstract

The invention discloses a large-scale risk data-oriented evidence storage method and system, and belongs to the technical field of block chain security. The method comprises the following steps: dividing multi-modal risk data into a data block set, and adding each data block into a corresponding queue; performing loop expansion processing on a loop body in a compression function of the BLAKE3 algorithm to generate a BLAKE3 optimization algorithm; after calculating a hash value of the data block in each queue based on a BLAKE3 optimization algorithm, generating a hash value of the multi-modal risk data; storing the multi-modal risk data in a local database, and obtaining risk data identification features; and storing the hash value of the multi-modal risk data and the risk data identification feature on a block chain. The method has the advantages of good block chain evidence storage efficiency, low block chain storage pressure and high data security.

Description

Technical Field

[0001] The present invention relates to a method and system for storing evidence of large-scale risk data, and belongs to the field of blockchain security technology. Background Art

[0002] Blockchain technology is widely used in various real-world evidence storage scenarios, such as finance and law, due to its key features such as decentralization, immutability and transparency. Each piece of evidence will be recorded in a block, which is linked to form a chain through an encryption algorithm. Each block contains the hash value of the previous block to ensure the integrity and traceability of the data. Smart contracts can automatically execute the evidence storage process, ensuring that information is automatically recorded and verified when conditions are met, thereby providing an efficient trust mechanism.

[0003] Although blockchain evidence storage can ensure data integrity, security, and traceability, it has the following shortcomings when facing large-scale multimodal risk data evidence storage: (1) Storage limitations: Each node in the blockchain needs to store a complete copy of the data. As the amount of data increases, the storage pressure increases. (2) Performance bottlenecks: As the amount of data increases, transaction processing time and block generation time may be extended. In addition, verifying each transaction requires time and computing resources, which affects the overall evidence storage efficiency.

[0004] In order to speed up the storage efficiency of large-scale multimodal risk data and reduce the storage pressure of blockchain, an efficient storage technology must be designed.

[0005] At present, the methods for storing large-scale multimodal risk data include the following:

[0006] Solution 1: Risk data evidence based on centralized storage and backup. Store risk data on an encrypted server and back it up regularly to prevent data loss.

[0007] However, this solution requires a trusted third party to manage the server, which poses a risk of data tampering.

[0008] Solution 2: Blockchain-based risk data storage. Use blockchain technology to record, store and verify risk data. This process ensures the data is tamper-proof, transparent and traceable, thereby providing data authenticity and legal validity.

[0009] However, this solution has the problems of high storage pressure and low evidence storage efficiency. Summary of the invention

[0010] The purpose of the present invention is to overcome the technical defects of existing large-scale multimodal risk data storage methods, such as high storage pressure and low storage efficiency. In order to solve the technical problems of storing large-scale multimodal risk data, a storage method and system for large-scale risk data are proposed.

[0011] In order to achieve the above object, the present invention adopts the following technical scheme.

[0012] A method for storing large-scale risk data, the method comprising:

[0013] Divide the multimodal risk data into a set of data blocks, and add each data block to a corresponding queue;

[0014] Generate a BLAKE3 optimization algorithm by performing loop expansion processing on the loop body in the compression function of the BLAKE3 algorithm;

[0015] After calculating the hash value of the data block in each queue based on the BLAKE3 optimization algorithm, the hash value of the multimodal risk data is generated;

[0016] Store multimodal risk data in a local database and obtain risk data identification features;

[0017] The hash value of the multimodal risk data and the risk data identification feature are stored on a blockchain.

[0018] Furthermore, the queue itself has a mutex lock.

[0019] Furthermore, the BLAKE3 optimization algorithm is generated by performing loop expansion processing on the loop body in the compression function of the BLAKE3 algorithm, including:

[0020] Add hash calculation functions g(v,0,4,8,12,m[(i+8)%16],m[(i+9)%16]), hash calculation function g(v,1,5,9,13,m[(i+10)%16],m[(i+11)%16]), hash calculation function g(v,2,6,10,14,m[(i+12)%16],m[(i+13)%16]) and hash calculation function g(v,3,7,11,15,m[(i+14)%16],m[(i+15)%16]) to the loop body of the BLAKE3 algorithm;

[0021] and,

[0022] Modify the loop judgment condition; wherein, the loop judgment condition before the modification is to execute i=i+1 when i<12, and the loop judgment condition after the modification is to execute i=i+2 when i<12;

[0023] and,

[0024] Dynamically allocate buffers for BLAKE3 optimization algorithm.

[0025] Furthermore, the calculation of the hash value of the data block in each queue based on the BLAKE3 optimization algorithm includes:

[0026] Obtaining a data block size threshold, and dynamically allocating the number of threads based on the data block size threshold;

[0027] The thread reads a block of data from the queue;

[0028] The hash value of the data block is calculated based on the BLAKE3 optimization algorithm, and the two-dimensional array Local_Hash[num1][num2] of the uint32_t type is used to store the hash value of the data block in real time; wherein num1+1 represents the total number of local hash values, which are the hash values ​​of the data block, and num2+1 represents the number of 32-bit hash blocks.

[0029] Furthermore, the dynamically allocating the number of threads according to the data block size threshold includes:

[0030] Get the set minimum number of threads, maximum number of threads, and data block size threshold;

[0031] Obtaining the size of multimodal risk data, and generating an optimal number of threads by combining the minimum number of threads and the data block size threshold;

[0032] When the optimal number of threads is not greater than the maximum number of threads, the number of threads allocated to the queue is the optimal number of threads;

[0033] When the optimal number of threads is greater than the maximum number of threads, the number of threads allocated to the queue is the maximum number of threads.

[0034] Further, generating a hash value of the multimodal risk data includes:

[0035] Initialize a hash array h[num2] of uint32_t type;

[0036] Traverse the two-dimensional array Local_Hash[num1][num2], and perform bitwise XOR calculation on the 32-bit hash value in the hash array h[num2] and the 32-bit hash value corresponding to the two-dimensional array Local_Hash[p][q], and assign the result to the hash array h[num2]; wherein p∈[0,num1), q∈[0,num2);

[0037] After the traversal is completed, the hash value of the multimodal risk data is obtained.

[0038] Furthermore, the storage identification features include: storage address, risk data description and timestamp.

[0039] Furthermore, the hash value of the multimodal risk data and the risk data identification feature are stored on a blockchain, including:

[0040] Generate a smart contract, which uses mapping to store and query the risk data identification features and the hash value of the multimodal risk data, uses the event keyword to declare a risk data storage event, uses the emit keyword to declare that the event defined by event has been triggered and executed, and records the event in the blockchain log;

[0041] Based on the smart contract, the hash value of the multimodal risk data and the risk data identification feature are stored on the blockchain.

[0042] A notarization system for large-scale risk data, the system comprising:

[0043] A data partitioning module, used to partition the multimodal risk data into a set of data blocks and add each data block to a corresponding queue;

[0044] An algorithm optimization module, used for generating a BLAKE3 optimization algorithm by performing loop expansion processing on a loop body in a compression function of the BLAKE3 algorithm;

[0045] A hash calculation module, used to generate a hash value of the multimodal risk data after calculating the hash value of the data block in each queue based on the BLAKE3 optimization algorithm;

[0046] A first storage module is used to store the multimodal risk data in a local database and obtain risk data identification features;

[0047] The second storage module is used to store the hash value of the multimodal risk data and the risk data identification feature on the blockchain.

[0048] An electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements any of the above-mentioned methods for storing large-scale risk data.

[0049] Compared with the prior art, the present invention has at least the following beneficial effects.

[0050] 1. This method has good blockchain evidence storage efficiency. Specifically: BLAKE3 optimization algorithm and multi-threaded parallel computing accelerate data hash calculation; multimodal risk data identification features and global data hash are stored on the blockchain. The smaller the data volume, the faster the transaction speed and the higher the evidence storage efficiency.

[0051] 2. This method has lower blockchain storage pressure. Specifically, the real multimodal risk data is stored off-chain, and the multimodal risk data identifier and hash value are stored on-chain. The joint storage on and off-chain reduces the pressure of storing large-scale data on the blockchain.

[0052] 3. This method has high data security. Specifically, it uses smart contracts to automatically store multimodal risk data on the blockchain, avoiding the risk of third-party tampering with data and improving data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of the evidence storage method for large-scale risk data. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, the notarization method for large-scale risk data of the present invention includes the following steps.

[0056] Step 1: Perform partitioning on multimodal risk data.

[0057] Specifically, step 1 includes the following steps:

[0058] Step 1.1: Divide the risk data into a set of data blocks according to a fixed size and add the data blocks to the queue. The queue is a shared resource and a mutex is assigned to it. When a thread reads a data block in the queue, the mutex is locked to ensure that the current data block is only accessed by one thread. When the thread finishes reading the data blocks in the queue, the mutex is unlocked. When the thread completes the calculation of the current data block, it continues to read the remaining data blocks in the queue for execution, which can make full use of the computing resources of idle threads.

[0059] Step 2: Perform optimization processing on the BLAKE3 algorithm.

[0060] Specifically, step 2 includes the following steps:

[0061] Step 2.1: Unroll the loop body in the core "compression function" of the BLAKE3 algorithm to improve the code execution efficiency. The loop body contains four g functions, namely g(v,0,4,8,12,m[i%16],m[(i+1)%16]), g(v,1,5,9,13,m[(i+2)%16],m[(i+3)%16]), g(v,2,6,10,14,m[(i+4)%16],m[(i+5)%16]), g(v,3,7,11,15,m[(i+6)%16],m[(i+7)%16]). Among them, v is an array of length 16, which is used to maintain the current state of hash calculation. m is an array of length 16, which is used to store the information of the input data block. g is a hash calculation function. i represents the current number of loops. The loop body is executed 12 times in total, that is, each time the loop body is executed, (i+1) and (i<12) are calculated once. In order to reduce the number of loop judgments, add g(v,0,4,8,12,m[(i+8)%16],m[(i+9)%16]), g(v,1,5,9,13,m[(i+10)%16],m[(i+11)%16]), g(v,2,6,10,14,m[(i+12)%16],m[(i+13)%16]), g(v,3,7,11,15,m[(i+14)%16],m[(i+15)%16] to the loop body. Each time the loop body is executed, (i+2) and (i<12) are calculated once. At this time, the loop body only needs to be executed 6 times, reducing 6 loop judgments. Moreover, this method makes full use of the information of data blocks.

[0062] Step 2.2: Dynamically allocate buffers to avoid memory waste caused by fixed-size buffers and improve program execution efficiency. When a thread needs to use a buffer, allocate the buffer in real time. When the buffer is no longer in use, release the buffer in real time.

[0063] Step 3: Assign multiple threads to parallelly calculate the local data hash.

[0064] Specifically, step 3 includes the following steps:

[0065] Step 3.1: Dynamically allocate the number of threads based on the data size. The specific steps include: 1) Set the minimum number of threads to MIN_THREADS; 2) Set the maximum number of threads to MAX_THREADS; 3) Calculate the optimal number of threads num_threads for the current data based on formula (1). Among them, file_size represents the size of the file. CHUNK_SIZE represents the custom data block size threshold. The idea of ​​formula (1) is that when the file is large (greater than the data block threshold), more threads are needed to improve computing efficiency. 4) Use the maximum number of threads to constrain num_threads. When the number of threads is too large, it will cause context switching overhead, resource competition overhead and other problems. Therefore, when the calculated num_threads is greater than MAX_THREADS, assign MAX_THREADS to num_threads.

[0066] num_threads =MIN_THREADS + (file_size / CHUNK_SIZE) (1)

[0067] Step 3.2: The thread reads the data block from the queue;

[0068] Step 3.3: Calculate the hash value of the data block using the BLAKE3 optimization algorithm in step 2;

[0069] Step 3.4: Use the uint32_t type two-dimensional array Local_Hash[num1][num2] to store the thread-local hash value in real time. Since the BLAKE3 algorithm calculates the hash value of the data as 256 bits, and one block of Local_Hash stores a 32-bit value. Therefore, num2=8 blocks are required to store the hash value of the data, that is, the two-dimensional array used in this embodiment is Local_Hash[num1][8]. By passing the address of the Local_Hash array and the index value of the data block in the queue to the thread, the thread-local hash value can be stored in Local_Hash in real time.

[0070] Step 4: Calculate the global data hash through the local data hash.

[0071] Specifically, step 4 includes the following steps:

[0072] Step 4.1: Initialize the hash array h[8] of type uint32_t to store the global data hash. Traverse the array Local_Hash[num1][8] and calculate the global data hash based on the bitwise XOR operation expressed in formula (2). Among them, the value of p is [0, num1), indicating the pth local hash value. The value of q is [0, 8), indicating the qth 32-bit hash block.

[0073] h[q] ^=local_hash[p][q] (2)

[0074] The traditional method is to use global hash to merge local hash in real time. However, as a shared resource, global hash requires a mutex to ensure that it is accessed by only one thread. When there are too many threads, the time for the thread to wait for the mutex to be unlocked will be longer, thereby reducing computing efficiency.

[0075] Step 5: Multimodal risk data is stored off-chain and identification features are recorded.

[0076] Specifically, step 5 includes the following steps:

[0077] Step 5.1: Store the multimodal risk data in the local database and obtain identification features such as storage address, risk data description, and timestamp.

[0078] Step 6: Store the identification characteristics and global data hash on the blockchain via a smart contract.

[0079] Specifically, step 6 includes the following steps:

[0080] Step 6.1: Write a smart contract, whose functions include storing multimodal risk data identification features and hash values, querying multimodal risk data related information, etc. In the smart contract, use mapping to store and query risk data identification features and hashes. In the smart contract, use the event keyword to declare a risk data storage event. Events are a mechanism for recording information in smart contracts, allowing contracts to publish logs to the outside world (such as front-end applications) during execution. Use the emit keyword to declare that the event defined by event has been triggered and recorded in the blockchain log. The advantage of using events is that it can provide transparency and queryability of on-chain data without consuming too much Gas.

[0081] Step 6.2: After receiving the multimodal risk data identification features and hash values, the smart contract publishes them as a transaction to the blockchain. After verification by the consensus protocol, the evidence is automatically stored on the blockchain.

[0082] Below, a specific experiment is used to illustrate the evidence storage method for large-scale risk data provided by the present invention.

[0083] Hardware configuration: 12 CPU cores, 16 logical processors, 1.70GHz frequency, 16GB memory.

[0084] Software configuration: #include<stdint.h> 、#include<string.h> 、#include<stdio.h> 、#include<stdlib.h> 、#include<time.h> 、#include<pthread.h> .

[0085] Experimental data: MP4 file of 1.01GB in size.

[0086] Experimental results:

[0087]

[0088]

[0089] The above is one of the embodiments of the present invention, and the present invention should not be limited to the contents disclosed in the embodiment and the drawings. Any equivalent or modification completed without departing from the spirit disclosed in the present invention shall fall within the scope of protection of the present invention.

Claims

1. A method for storing large-scale risk data, characterized in that: The method comprises: Divide the multimodal risk data into a set of data blocks, and add each data block to a corresponding queue; Generate a BLAKE3 optimization algorithm by performing loop expansion processing on the loop body in the compression function of the BLAKE3 algorithm; After calculating the hash value of the data block in each queue based on the BLAKE3 optimization algorithm, the hash value of the multimodal risk data is generated; Store multimodal risk data in a local database and obtain risk data identification features; The hash value of the multimodal risk data and the risk data identification feature are stored on a blockchain.

2. The method according to claim 1, characterized in that The queue itself has a mutex lock.

3. The method according to claim 1, characterized in that The method generates a BLAKE3 optimization algorithm by performing loop expansion processing on the loop body in the compression function of the BLAKE3 algorithm, including: Add hash calculation functions g(v,0,4,8,12,m[(i+8)%16],m[(i+9)%16]), hash calculation function g(v,1,5,9,13,m[(i+10)%16],m[(i+11)%16]), hash calculation function g(v,2,6,10,14,m[(i+12)%16],m[(i+13)%16]) and hash calculation function g(v,3,7,11,15,m[(i+14)%16],m[(i+15)%16]) to the loop body of the BLAKE3 algorithm; and, Modify the loop judgment condition; wherein, the loop judgment condition before the modification is to execute i=i+1 when i<12, and the loop judgment condition after the modification is to execute i=i+2 when i<12; and, Dynamically allocate buffers for BLAKE3 optimization algorithm.

4. The method according to claim 1, characterized in that: The calculation of the hash value of the data block in each queue based on the BLAKE3 optimization algorithm includes: Obtaining a data block size threshold, and dynamically allocating the number of threads based on the data block size threshold; The thread reads a block of data from the queue; The hash value of the data block is calculated based on the BLAKE3 optimization algorithm, and the two-dimensional array Local_Hash[num1][num2] of the uint32_t type is used to store the hash value of the data block in real time; wherein num1+1 represents the total number of local hash values, which are the hash values ​​of the data block, and num2+1 represents the number of 32-bit hash blocks.

5. The method according to claim 4, characterized in that The dynamically allocating the number of threads according to the data block size threshold comprises: Get the set minimum number of threads, maximum number of threads, and data block size threshold; Obtaining the size of multimodal risk data, and generating an optimal number of threads by combining the minimum number of threads and the data block size threshold; When the optimal number of threads is not greater than the maximum number of threads, the number of threads allocated to the queue is the optimal number of threads; When the optimal number of threads is greater than the maximum number of threads, the number of threads allocated to the queue is the maximum number of threads.

6. The method according to claim 4, characterized in that Generating a hash value of the multimodal risk data includes: Initialize a hash array h[num2] of uint32_t type; Traverse the two-dimensional array Local_Hash[num1][num2], and perform bitwise XOR calculation on the 32-bit hash value in the hash array h[num2] and the 32-bit hash value corresponding to the two-dimensional array Local_Hash[p][q], and assign the result to the hash array h[num2]; wherein p∈[0,num1), q∈[0,num2); After the traversal is completed, the hash value of the multimodal risk data is obtained.

7. The method according to claim 1, characterized in that The storage identification features include: storage address, risk data description and timestamp.

8. The method according to claim 1, characterized in that Storing the hash value of the multimodal risk data and the risk data identification feature on a blockchain, comprising: Generate a smart contract, which uses mapping to store and query the risk data identification features and the hash value of the multimodal risk data, uses the event keyword to declare a risk data storage event, uses the emit keyword to declare that the event defined by event has been triggered and executed, and records the event in the blockchain log; Based on the smart contract, the hash value of the multimodal risk data and the risk data identification feature are stored on the blockchain.

9. A proof system for large-scale risk data, characterized in that: The system comprises: A data partitioning module, used to partition the multimodal risk data into a set of data blocks and add each data block to a corresponding queue; An algorithm optimization module, used for generating a BLAKE3 optimization algorithm by performing loop expansion processing on a loop body in a compression function of the BLAKE3 algorithm; A hash calculation module, used to generate a hash value of the multimodal risk data after calculating the hash value of the data block in each queue based on the BLAKE3 optimization algorithm; A first storage module is used to store the multimodal risk data in a local database and obtain risk data identification features; The second storage module is used to store the hash value of the multimodal risk data and the risk data identification feature on the blockchain.

10. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for storing large-scale risk data as described in any one of claims 1 to 8 is implemented.

Images