Evidence storage methods and systems for large-scale risk data

Abstract

This invention discloses a method and system for storing evidence of large-scale risk data, belonging to the field of blockchain security technology. The method includes: dividing multimodal risk data into a set of data blocks and adding each data block to a corresponding queue; generating an optimized BLAKE3 algorithm by loop unrolling the loop body within the compression function of the BLAKE3 algorithm; generating the hash value of the multimodal risk data by calculating the hash value of each data block in the queue based on the optimized BLAKE3 algorithm; storing the multimodal risk data in a local database and obtaining risk data identification features; and storing the hash value and the risk data identification features of the multimodal risk data on the blockchain. This invention offers good blockchain evidence storage efficiency, low blockchain storage pressure, and high data security.

Description

Technical Field

[0001] This invention relates to a method and system for storing evidence of large-scale risky data, belonging to the field of blockchain security technology. Background Technology

[0002] Blockchain technology, with its key characteristics of decentralization, immutability, and transparency, is widely used in various real-world evidence preservation scenarios, such as finance and law. Each piece of evidence is recorded in a block, and these blocks are linked together using cryptographic algorithms to form a chain. Each block contains the hash value of the previous block, ensuring data integrity and traceability. Smart contracts can automatically execute the evidence preservation process, ensuring that information is automatically recorded and verified when conditions are met, thus providing an efficient trust mechanism.

[0003] Although blockchain-based evidence storage can ensure data integrity, security, and traceability, it has the following shortcomings when dealing with large-scale, multimodal risk data: (1) Storage limitations: Each node in the blockchain needs to store a complete copy of the data, and the storage pressure increases with the amount of data. (2) Performance bottlenecks: With the increase in the amount of data, transaction processing time and block generation time may be prolonged. In addition, verifying each transaction requires time and computing resources, thus affecting the overall evidence storage efficiency.

[0004] To accelerate the efficiency of storing large-scale multimodal risk data and reduce the storage pressure on blockchain, an efficient storage technology must be designed.

[0005] Currently, the methods for storing large-scale multimodal risk data include the following:

[0006] Option 1: Risk data notarization based on centralized storage and backup. Storing risk data on an encrypted server and backing it up regularly to prevent data loss.

[0007] However, this solution requires a trusted third party to manage the server, which poses a risk of data tampering.

[0008] Option 2: Blockchain-based risk data storage. This involves using blockchain technology to record, store, and verify risk data. This process ensures the immutability, transparency, and traceability of the data, thereby providing authenticity and legal validity.

[0009] However, this solution suffers from high storage pressure and low evidence preservation efficiency. Summary of the Invention

[0010] The purpose of this invention is to overcome the technical defects of existing methods for storing large-scale multimodal risk data, such as high storage pressure and low storage efficiency. In order to solve the technical problems of storing large-scale multimodal risk data, a method and system for storing large-scale risk data is proposed.

[0011] To achieve the above objectives, the present invention adopts the following technical solution.

[0012] A method for storing evidence of large-scale risk data, the method comprising:

[0013] Multimodal risk data is divided into a set of data blocks, and each data block is added to a corresponding queue;

[0014] The BLAKE3 optimized algorithm is generated by unrolling the loop body within the compression function of the BLAKE3 algorithm.

[0015] After calculating the hash value of the data block in each queue based on the BLAKE3 optimization algorithm, the hash value of the multimodal risk data is generated.

[0016] Multimodal risk data is stored in a local database, and risk data identification features are obtained;

[0017] The hash value of the multimodal risk data and the risk data identification feature are stored on the blockchain.

[0018] Furthermore, the queue itself is equipped with a mutex lock.

[0019] Furthermore, the step of generating the BLAKE3 optimized algorithm by loop unrolling the loop body within the compression function of the BLAKE3 algorithm includes:

[0020] Add the hash calculation functions g(v,0,4,8,12,m[(i+8)%16],m[(i+9)%16]), g(v,1,5,9,13,m[(i+10)%16],m[(i+11)%16]), g(v,2,6,10,14,m[(i+12)%16],m[(i+13)%16]), and g(v,3,7,11,15,m[(i+14)%16],m[(i+15)%16]) to the loop body of the BLAKE3 algorithm;

[0021] and,

[0022] Modify the loop condition; previously, the loop condition was to execute i = i + 1 if i < 12, and now it is to execute i = i + 2 if i < 12.

[0023] and,

[0024] Dynamically allocate buffers for the BLAKE3 optimization algorithm.

[0025] Furthermore, the calculation of the hash value of the data block in each queue based on the BLAKE3 optimization algorithm includes:

[0026] Obtain the data block size threshold and dynamically allocate the number of threads based on the data block size threshold;

[0027] The thread reads data blocks from the queue;

[0028] The hash value of the data block is calculated based on the BLAKE3 optimization algorithm, and the hash value of the data block is stored in real time using a two-dimensional array of type uint32_t, Local_Hash[num1][num2]. Here, num1+1 represents the total number of local hash values, which are the hash values ​​of the data block, and num2+1 represents the number of 32-bit hash blocks.

[0029] Furthermore, the dynamic allocation of the number of threads based on the data block size threshold includes:

[0030] Get the set minimum number of threads, maximum number of threads, and data block size threshold;

[0031] Obtain the size of the multimodal risk data, and combine it with the minimum number of threads and the data block size threshold to generate the optimal number of threads;

[0032] If the optimal number of threads is not greater than the maximum number of threads, the number of threads allocated to the queue shall be the optimal number of threads.

[0033] If the optimal number of threads is greater than the maximum number of threads, the number of threads allocated to the queue shall be the maximum number of threads.

[0034] Furthermore, the hash value used to generate the multimodal risk data includes:

[0035] Initialize a hash array h[num2] of type uint32_t;

[0036] Iterate through the two-dimensional array Local_Hash[num1][num2], and perform a bitwise XOR operation between the 32-bit hash value in the hash array h[num2] and the 32-bit hash value corresponding to the two-dimensional array Local_Hash[p][q], and assign the result to the hash array h[num2]; where p∈[0,num1), q∈[0,num2);

[0037] After the traversal is complete, the hash value of the multimodal risk data is obtained.

[0038] Furthermore, the storage identification features include: storage address, risk data description, and timestamp.

[0039] Furthermore, storing the hash value of the multimodal risk data and the risk data identification feature on the blockchain includes:

[0040] A smart contract is generated, which uses mapping to store and query the risk data identification features and the hash value of the multimodal risk data, uses the event keyword to declare a risk data storage event, uses the emit keyword to declare that the event defined by the event has been triggered and executed, and records the event to the blockchain log;

[0041] The hash value of the multimodal risk data and the risk data identification feature are stored on the blockchain based on the smart contract.

[0042] A system for storing evidence of large-scale risk data, the system comprising:

[0043] The data partitioning module is used to divide multimodal risk data into a set of data blocks and add each data block to the corresponding queue;

[0044] The algorithm optimization module is used to generate an optimized BLAKE3 algorithm by performing loop unrolling on the loop body within the compression function of the BLAKE3 algorithm.

[0045] The hash calculation module is used to calculate the hash value of the data block in each queue based on the BLAKE3 optimization algorithm, and then generate the hash value of the multimodal risk data.

[0046] The first storage module is used to store multimodal risk data in a local database and obtain risk data identification features;

[0047] The second storage module is used to store the hash value of the multimodal risk data and the risk data identification feature on the blockchain.

[0048] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the evidence storage method for large-scale risk data as described above.

[0049] Compared with the prior art, the present invention has at least the following beneficial effects.

[0050] 1. This method has good blockchain evidence storage efficiency. Specifically: the BLAKE3 optimization algorithm and multi-threaded parallel computing accelerate data hash calculation; multimodal risk data identification features and global data hash are stored on the blockchain. The smaller the data volume, the faster the transaction speed and the higher the evidence storage efficiency.

[0051] 2. This method has lower blockchain storage pressure. Specifically, it stores real multimodal risk data off-chain, while storing multimodal risk data identifiers and hash values ​​on-chain. This combined on-chain and off-chain storage reduces the pressure of storing large-scale data on the blockchain.

[0052] 3. This method offers high data security. Specifically, it uses smart contracts to automatically store multimodal risk data on the blockchain, avoiding the risk of third-party data tampering and improving data security. Attached Figure Description

[0053] Figure 1 A flowchart for a method of evidence storage for large-scale risk data. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, the evidence storage method for large-scale risk data of the present invention includes the following steps.

[0056] Step 1: Perform segmentation processing on the multimodal risk data.

[0057] Specifically, step 1 includes the following steps:

[0058] Step 1.1: Divide the risk data into a set of data blocks of fixed size and add the data blocks to a queue. The queue is a shared resource, and a mutex lock is assigned to it. When a thread reads a data block from the queue, the mutex locks, ensuring that only one thread accesses the current data block. When a thread finishes reading the data block from the queue, the mutex locks unlocks. When a thread finishes calculating the current data block, it continues to read the remaining data blocks from the queue for execution, which makes full use of the computing resources of idle threads.

[0059] Step 2: Perform optimization processing on the BLAKE3 algorithm.

[0060] Specifically, step 2 includes the following steps:

[0061] Step 2.1: Unroll the loop body within the core "compression function" of the BLAKE3 algorithm to improve code execution efficiency. The loop body contains four g functions: g(v,0,4,8,12,m[i%16],m[(i+1)%16]), g(v,1,5,9,13,m[(i+2)%16],m[(i+3)%16]), g(v,2,6,10,14,m[(i+4)%16],m[(i+5)%16]), and g(v,3,7,11,15,m[(i+6)%16],m[(i+7)%16]). Here, v is an array of length 16 used to maintain the current hash calculation state. m is an array of length 16 used to store information about the input data block. g is a hash calculation function. i represents the current loop count. The loop body executes 12 times in total, meaning that (i+1) and (i<12) are calculated once for each execution of the loop body. To reduce the number of loop checks, the following are added to the loop body: g(v,0,4,8,12,m[(i+8)%16],m[(i+9)%16]), g(v,1,5,9,13,m[(i+10)%16],m[(i+11)%16]), g(v,2,6,10,14,m[(i+12)%16],m[(i+13)%16]), g(v,3,7,11,15,m[(i+14)%16],m[(i+15)%16]). Each time the loop body is executed, (i+2) and (i<12) are calculated once. Therefore, the loop body only executes 6 times, reducing the number of loop checks by 6. Furthermore, this method makes full use of the information in the data blocks.

[0062] Step 2.2: Dynamically allocate buffers to avoid memory waste caused by fixed-size buffers and improve program execution efficiency. Buffers are allocated in real-time when a thread needs to use them. Buffers are released in real-time when they are no longer needed.

[0063] Step 3: Allocate multiple threads to compute local data hashes in parallel.

[0064] Specifically, step 3 includes the following steps:

[0065] Step 3.1: Dynamically allocate the number of threads based on the data size. Specific steps include: 1) Setting the minimum number of threads to MIN_THREADS; 2) Setting the maximum number of threads to MAX_THREADS; 3) Calculating the optimal number of threads num_threads for the current data based on formula (1). Here, file_size represents the file size. CHUNK_SIZE represents a custom data block size threshold. The idea behind formula (1) is that when the file is very large (greater than the data block threshold), more threads are needed to improve computational efficiency. 4) Constraining num_threads using the maximum number of threads. Too many threads can cause context switching overhead and resource contention overhead. Therefore, when the calculated num_threads is greater than MAX_THREADS, MAX_THREADS is assigned to num_threads.

[0066] num_threads =MIN_THREADS + (file_size / CHUNK_SIZE) (1)

[0067] Step 3.2: The thread reads data blocks from the queue;

[0068] Step 3.3: Calculate the hash value of the data block using the BLAKE3 optimization algorithm from Step 2;

[0069] Step 3.4: Use a two-dimensional array of type uint32_t, Local_Hash[num1][num2], to store the thread-local hash value in real time. Since the BLAKE3 algorithm calculates a 256-bit hash value, and one block of Local_Hash stores a 32-bit value, num2 = 8 blocks are needed to store the hash value of the data. Therefore, in this embodiment, the two-dimensional array used is Local_Hash[num1][8]. By passing the address of the Local_Hash array and the index of the data block in the queue to the thread, the thread-local hash value can be stored in Local_Hash in real time.

[0070] Step 4: Calculate the global data hash using the local data hash.

[0071] Specifically, step 4 includes the following steps:

[0072] Step 4.1: Initialize a hash array h[8] of type uint32_t to store the global data hash. Traverse the array Local_Hash[num1][8] and calculate the global data hash based on the bitwise XOR operation represented in formula (2). Where p takes the value [0, num1), representing the p-th local hash value. q takes the value [0, 8), representing the q-th 32-bit hash block.

[0073] h[q] ^=local_hash[p][q] (2)

[0074] The traditional approach is to use a global hash to merge local hashes in real time. However, as a shared resource, the global hash requires a mutex lock to ensure that it is accessed by only one thread. When there are too many threads, the time it takes for a thread to wait for the mutex lock to unlock increases, thus reducing computational efficiency.

[0075] Step 5: Store and record the identification features of the multimodal risk data chain.

[0076] Specifically, step 5 includes the following steps:

[0077] Step 5.1: Store the multimodal risk data in a local database and obtain identification features such as storage address, risk data description, and timestamp.

[0078] Step 6: Store the identifier features and global data hash on the blockchain via a smart contract.

[0079] Specifically, step 6 includes the following steps:

[0080] Step 6.1: Write a smart contract whose functions include storing multimodal risk data identifiers and hash values, and querying multimodal risk data related information. In the smart contract, a mapping is used to store and query risk data identifiers and hashes. The `event` keyword is used in the smart contract to declare a risk data storage event. Events are a mechanism in smart contracts for recording information, allowing the contract to publish logs to external sources (such as front-end applications) during execution. The `emit` keyword is used to declare that the event defined by `event` has been triggered and to record the event in the blockchain log. The advantage of using events is that they provide transparency and queryability of on-chain data without consuming excessive Gas.

[0081] Step 6.2: After receiving the multimodal risk data identifier features and hash value, the smart contract publishes it as a transaction to the blockchain. After verification by the consensus protocol, it is automatically stored on the blockchain.

[0082] The following is a specific experiment illustrating the evidence storage method for large-scale risk data provided by this invention.

[0083] Hardware configuration: 12 CPU cores, 16 logical processing units, 1.70GHz frequency, 16GB memory.

[0084] Software configuration: #include<stdint.h> #include<string.h> #include<stdio.h> #include<stdlib.h> #include<time.h> #include<pthread.h> .

[0085] Experimental data: MP4 file of 1.01GB size.

[0086] Experimental results:

[0087]

[0088]

[0089] The above description is one embodiment of the present invention, and the present invention should not be limited to the content disclosed in this embodiment and the accompanying drawings. Any equivalent or modified versions made without departing from the spirit of the present invention fall within the scope of protection of the present invention.

Claims

1. A method for storing evidence of large-scale risk data, characterized in that, The method includes: Multimodal risk data is divided into a set of data blocks, and each data block is added to a corresponding queue; The BLAKE3 optimized algorithm is generated by unrolling the loop body within the compression function of the BLAKE3 algorithm. After calculating the hash value of the data block in each queue based on the BLAKE3 optimization algorithm, the hash value of the multimodal risk data is generated; wherein, the calculation of the hash value of the data block in each queue based on the BLAKE3 optimization algorithm includes: Obtain the data block size threshold and dynamically allocate the number of threads based on the data block size threshold; The thread reads data blocks from the queue; The hash value of a data block is calculated based on the BLAKE3 optimization algorithm, and the hash value of the data block is stored in real time using a two-dimensional array of type uint32_t, Local_Hash[num1][num2]; where num1 represents the total number of local hash values, the local hash value is the hash value of the data block, and num2 represents the number of 32-bit hash blocks; Multimodal risk data is stored in a local database, and risk data identification features are obtained; The hash value of the multimodal risk data and the risk data identification feature are stored on the blockchain.

2. The method according to claim 1, characterized in that, The queue itself is protected by a mutex lock.

3. The method according to claim 1, characterized in that, The process of generating an optimized BLAKE3 algorithm by loop unrolling the loop body within the compression function of the BLAKE3 algorithm includes: In the loop body of the BLAKE3 algorithm, add hash calculation functions g(v,0,4,8,12,m[(i+8)%16],m[(i+9)%16]), g(v,1,5,9,13,m[(i+10)%16],m[(i+11)%16]), g(v,2,6,10,14,m[(i+12)%16],m[(i+13)%16]), and g(v,3,7,11,15,m[(i+14)%16],m[(i+15)%16]). Here, v is an array of length 16 used to maintain the current hash calculation state; m is an array of length 16 used to store information about the input data blocks; g is a hash calculation function; and i represents the current loop count. and, Modify the loop condition; previously, the loop condition was to execute i = i + 1 if i < 12, and now it is to execute i = i + 2 if i < 12. and, Dynamically allocate buffers for the BLAKE3 optimization algorithm.

4. The method according to claim 1, characterized in that, The dynamic allocation of the number of threads based on the data block size threshold includes: Get the set minimum number of threads, maximum number of threads, and data block size threshold; Obtain the size of the multimodal risk data, and combine it with the minimum number of threads and the data block size threshold to generate the optimal number of threads; If the optimal number of threads is not greater than the maximum number of threads, the number of threads allocated to the queue shall be the optimal number of threads. If the optimal number of threads is greater than the maximum number of threads, the number of threads allocated to the queue shall be the maximum number of threads.

5. The method according to claim 1, characterized in that, The hash value used to generate the multimodal risk data includes: Initialize a hash array h[num2] of type uint32_t; Iterate through the two-dimensional array Local_Hash[num1][num2], and perform a bitwise XOR operation between the 32-bit hash value in the hash array h[num2] and the 32-bit hash value corresponding to the two-dimensional array Local_Hash[p][q], and assign the result to the hash array h[num2]; where p∈[0,num1), q∈[0,num2); After the traversal is complete, the hash value of the multimodal risk data is obtained.

6. The method according to claim 1, characterized in that, The storage identification features include: storage address, risk data description, and timestamp.

7. The method according to claim 1, characterized in that, Storing the hash value of the multimodal risk data and the risk data identification feature on the blockchain includes: A smart contract is generated, which uses mapping to store and query the risk data identification features and the hash value of the multimodal risk data, uses the event keyword to declare a risk data storage event, uses the emit keyword to declare that the event defined by the event has been triggered and executed, and records the event to the blockchain log; The hash value of the multimodal risk data and the risk data identification feature are stored on the blockchain based on the smart contract.

8. A data storage system for large-scale risk data, characterized in that, The system includes: The data partitioning module is used to divide multimodal risk data into a set of data blocks and add each data block to the corresponding queue; The algorithm optimization module is used to generate an optimized BLAKE3 algorithm by performing loop unrolling on the loop body within the compression function of the BLAKE3 algorithm. The hash calculation module is used to generate the hash value of the multimodal risk data after calculating the hash value of the data block in each queue based on the BLAKE3 optimization algorithm; wherein, the calculation of the hash value of the data block in each queue based on the BLAKE3 optimization algorithm includes: Obtain the data block size threshold and dynamically allocate the number of threads based on the data block size threshold; The thread reads data blocks from the queue; The hash value of a data block is calculated based on the BLAKE3 optimization algorithm, and the hash value of the data block is stored in real time using a two-dimensional array of type uint32_t, Local_Hash[num1][num2]; where num1 represents the total number of local hash values, the local hash value is the hash value of the data block, and num2 represents the number of 32-bit hash blocks; The first storage module is used to store multimodal risk data in a local database and obtain risk data identification features; the second storage module is used to store the hash value of the multimodal risk data and the risk data identification features on the blockchain.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the evidence storage method for large-scale risk data as described in any one of claims 1-7.

Images