Blockchain and multi-data tag based data collection method and system

Abstract

This invention discloses a data acquisition and storage method and system based on blockchain and multiple data tags. The method includes: a client determining the source data file to be uploaded, a core metadata tag set, and an extended metadata tag set; deduplicating and merging the two tag sets to obtain a tag set; calculating the hash value of the tag set as its identifier; dividing the source data file into N file fragments, calculating their hash values, and constructing a Merkle tree; using the root hash value of the Merkle tree and the tag set identifier to construct another Merkle tree and obtaining the root hash value as a global integrity identifier; and uploading the source data file, its tag set, and the global integrity identifier to an application server, which then submits the data to the blockchain network as a notarization transaction to achieve tamper-proof notarization. This invention aims to ensure the security of source data files and tag sets during the acquisition and storage process, as well as the traceability of the acquisition and storage process, in distributed, geographically dispersed data acquisition and storage scenarios.

Description

Technical Field

[0001] This invention belongs to the field of data acquisition and storage technology, specifically relating to a data acquisition and storage method and system based on blockchain and multiple data tags. Background Technology

[0002] In the construction of data centers, most projects require the collection and storage of both structured and unstructured data to establish corresponding databases and data file systems, while also meeting the need to improve data query and retrieval efficiency. Chinese patent application CN118227846A discloses a method and system for distributing substation monitoring data based on data tags. This solution adds multiple tags based on metadata while uploading source data files, which can effectively address needs such as fuzzy data retrieval. However, it has significant shortcomings in ensuring data security. For the collection and storage of distributed, geographically dispersed data, strategies are needed to ensure the security of the data collection and transmission process, prevent data tampering, and ensure traceability of the collection and storage process. Clearly, this solution fails to meet these requirements. Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a data acquisition and storage method and system based on blockchain and multiple data tags, addressing the aforementioned problems in the prior art. This invention aims to ensure the security of source data files and tag set data during the acquisition and storage process, as well as the traceability of the acquisition and storage process, for distributed and geographically dispersed data acquisition and storage scenarios.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A data acquisition and storage method based on blockchain and multi-data tags includes the following steps: S101, the client determines the source data file to be uploaded, as well as the core metadata tag set and extended metadata tag set of the source data file. The core metadata tag set and extended metadata tag set are deduplicated and merged to obtain a tag set, and the hash value of the tag set is calculated as the tag set identifier TagHash. S102, the client divides the source data file into N file fragments, constructs a Merkle tree based on the hash values ​​of the N file fragments, and uses the root hash value of the Merkle tree as the file integrity identifier FileRoot of the source data file; the client then constructs a Merkle tree again based on the integrity identifier FileRoot and the tag set identifier TagHash, and uses the root hash value of the newly constructed Merkle tree as the global integrity identifier FileUID of the source data file; S103, the client uploads the source data file, its tag set, and the global integrity identifier FileUID to the remote application server, so that the application server can store the source data file and its tag set and submit the global integrity identifier FileUID as a notarization transaction to the blockchain network to achieve tamper-proof notarization.

[0005] Optionally, the tag set obtained by deduplicating and merging the core metadata tag set and the extended metadata tag set in step S101 includes: S201, each metadata tag in the core metadata tag set and the extended metadata tag set is converted into a corresponding tag vector using word vector tools; S202, For each metadata tag in the core metadata tag set, calculate the cosine similarity between it and the tag vector of each metadata tag in the extended metadata tag set, and delete the metadata tags in the extended metadata tag set whose cosine similarity exceeds a preset threshold, and finally obtain the deduplicated extended metadata tag set. S203, combine the core metadata tag set with the deduplicated extended metadata tag set to obtain the tag set.

[0006] Optionally, in step S102, the client splits the source data file into N file fragments, including: S301, Calculate the base number of shards N_base: N_base = ceil(S / S_base); Where ceil is the floor function, S is the file size of the source data file, and S_base is the base fragment size; S302, calculate the adjusted number of shards N_adjusted based on the base number of shards N_base: N_adjusted=round(N_base×α×β); α=max(0.5,min(2.0,α_bandwidth×α_latency×α_stability)); β=max(0.6,min(1.3,(β_cpu×β_memory×β_network) 1 / 3 )); Where `round` is the rounding function, `α` is the network quality factor, `β` is the system load factor, `α_bandwidth`, `α_latency`, and `α_stability` are the bandwidth factor, latency factor, and stability factor, respectively. The bandwidth factor is obtained by mapping the available bandwidth of the network, the latency factor is obtained by mapping the network latency, and the stability factor is obtained by mapping the network packet loss rate. `β_cpu`, `β_memory`, and `β_network` are the CPU load factor, memory load factor, and network load factor, respectively. The CPU load factor is obtained by mapping the CPU utilization rate, the memory load factor is obtained by mapping the memory utilization rate, and the network load factor is obtained by mapping the network load. S303, the initial number of fragments N0 is obtained by applying the adjusted fragment number N_adjusted constraint: N0=max(1,min(N_max,N_adjusted)); Where max and min are the maximum and minimum value operation functions, respectively, and N_max is the preset maximum number of partitions; S304, calculate the actual fragment size S / N0 based on the initial fragment size N0. If the actual fragment size S / N0 is greater than the preset maximum fragment size S_max, then determine the final fragment size N according to the following formula: N = max(1, ceil(S / S_max)); If the actual fragment size S / N0 is less than the preset minimum fragment size S_min, then the final number of fragments N is determined according to the following formula: N=min(N_max,floor(S / S_min)); Where floor is the floor function; if the actual fragment size S / N0 is between the minimum fragment size S_min and the maximum fragment size S_max, then the actual fragment size S / N0 is taken as the final number of fragments N; S305, divide the source data file into N file fragments according to the final number of fragments N.

[0007] Optionally, the calculation function expressions for the bandwidth factor, delay factor, and stability factor are as follows: α_bandwidth=0.6+0.9×(1-e -B / 15 ); α_latency = 1.2 × e -L / 120 ; α_stability=1.2×e -P / 4 ; Where B is the available bandwidth, e is the natural constant, L represents the network latency, and P represents the network packet loss rate.

[0008] Optionally, the calculation function expressions for the CPU load factor, memory load factor, and network load factor are as follows: β_cpu = 1.3 - 0.7 × (1 - e -3×C ); β_memory = 1.3 - 0.7 × (1 - e -3×M ); β_network = 1.3 - 0.7 × (1 - e -3×K ); Where C represents CPU utilization, M represents memory utilization, and K represents network load.

[0009] Optionally, the Merkle tree construction based on the hash values ​​of the N file fragments in step S102 includes: calculating the hash value H_i of each of the N file fragments, and merging the hash value H_i of each file fragment and its fragment index Index_i to generate an enhanced hash value STHash. STHash=SHA3-256(Index_i:H_i); SHA3-256 is a 256-bit hash function; the enhanced hash values ​​STHash of the N file fragments are used as leaf nodes of the Merkle tree, and the Merkle tree is constructed in index order.

[0010] Optionally, in step S102, reconstructing the Merkle tree based on the integrity identifier FileRoot and the tag set identifier TagHash includes: using the integrity identifier FileRoot and the tag set identifier TagHash as two nodes of the Merkle tree respectively, and constructing a Merkle tree composed of the two nodes.

[0011] Optionally, step S103, which involves storing the source data file and its tag set via the application server and submitting the global integrity identifier FileUID as a notarization transaction to the blockchain network to achieve tamper-proof notarization, includes: S401, The application server verifies the N file fragments using the global integrity identifier FileUID. If the verification passes, it proceeds to step S402; otherwise, it sends a retransmission request to the client, ends the process, and exits. S402, the application server reassembles N file fragments into a complete data file; S403, the application server uploads the complete data file to the distributed file system and generates a file content identifier (FileCID). It stores the FileCID and the global integrity identifier (FileUID) in a relational database to establish a mapping relationship between the FileCID and the FileUID. The application server also stores the tag set of the source data file and the FileUID in the search engine database to establish a mapping relationship between the tag set and the FileUID. S404, the application server submits the global integrity identifier FileUID, the client's address and timestamp as evidence storage transactions to the blockchain network, realizing the immutable evidence storage of the source data file and its tag set.

[0012] Optionally, in step S401, the application server verifies the N file fragments using the global integrity identifier FileUID, including: S501, the application server calculates the hash value of each of the N received file fragments, and calculates the hash value of the received tag set to obtain the server tag set identifier TagHash_server; S502, the application server constructs a Merkle tree based on the hash values ​​of the received N file fragments, and uses the root hash value of the Merkle tree as the server-side file integrity identifier FileRoot_server of the source data file; it then constructs the Merkle tree again based on the server-side integrity identifier FileRoot_server and the server-side tag set identifier TagHash_server, and uses the root hash value of the newly constructed Merkle tree as the server-side global integrity identifier FileUID_server of the source data file. S503: Determine whether the server-side global integrity identifier FileUID_server and the received global integrity identifier FileUID are consistent. If they are consistent, the verification is deemed to have passed; otherwise, the verification is deemed to have failed.

[0013] The present invention also provides a data acquisition and storage system based on blockchain and multiple data tags, including interconnected client and application server, both of which include a microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the data acquisition and storage method based on blockchain and multiple data tags.

[0014] Compared with existing technologies, the present invention mainly achieves the following beneficial effects: The core of the data collection and storage method based on blockchain and multi-data tags in the present invention lies in the seamless integration of file content tagging, fragmented uploading, distributed storage, and blockchain evidence storage through a multi-layered, collaborative technical architecture. It generates a unique identifier by calculating the hash value of the source data file fragments. This identifier, combined with the file tag hash value and the collection process information, constructs a blockchain, which can solve the shortcomings of existing remote distributed data collection and storage processes that cannot ensure the security of source data file data and tag set data. It effectively avoids illegal operations such as data tampering and deletion during data collection, transmission, and storage, thereby ensuring data security and traceability of the collection and storage process. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the Merkle tree constructed in an embodiment of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions of the present invention, the technical solutions of the present invention will be further described in detail below with reference to the accompanying drawings in the embodiments of the present invention.

[0018] like Figure 1 As shown, the data acquisition and storage method based on blockchain and multi-data tags in this embodiment includes the following steps: S101, the client determines the source data file to be uploaded, as well as the core metadata tag set and extended metadata tag set of the source data file. The core metadata tag set and extended metadata tag set are deduplicated and merged to obtain a tag set, and the hash value of the tag set is calculated as the tag set identifier TagHash. S102, the client divides the source data file into N file fragments, constructs a Merkle tree based on the hash values ​​of the N file fragments, and uses the root hash value of the Merkle tree as the file integrity identifier FileRoot of the source data file; the client then constructs a Merkle tree again based on the integrity identifier FileRoot and the tag set identifier TagHash, and uses the root hash value of the newly constructed Merkle tree as the global integrity identifier FileUID of the source data file; S103, the client uploads the source data file, its tag set, and the global integrity identifier FileUID to the remote application server, so that the application server can store the source data file and its tag set and submit the global integrity identifier FileUID as a notarization transaction to the blockchain network to achieve tamper-proof notarization.

[0019] In this embodiment, the source data file is a PDF file with a size of 500MB.

[0020] After the user selects a file, the core metadata tag set C is determined as follows: C = {2026 / 1 / 6, Data Acquisition Device A, Device B, Importance Level C, Text, Item D, Level E, Unit F}; The newly added extended metadata tag set D is: D = {General Test Report Template, for Data Mining, Organization F}.

[0021] In step S101 of this embodiment, the tag set obtained by deduplicating and merging the core metadata tag set and the extended metadata tag set includes: S201, each metadata tag in the core metadata tag set and the extended metadata tag set is converted into a corresponding tag vector using word vector tools (such as word2vec); S202, For each metadata tag in the core metadata tag set, calculate the cosine similarity between it and the tag vector of each metadata tag in the extended metadata tag set, and delete the metadata tags in the extended metadata tag set whose cosine similarity exceeds a preset threshold, and finally obtain the deduplicated extended metadata tag set. S203, combine the core metadata tag set with the deduplicated extended metadata tag set to obtain the tag set.

[0022] In step S201 of this embodiment, the vector is trained based on the word2vec model. The word vectors of multiple words in a single tag are arithmetically averaged dimension by dimension to obtain the vector representation of the tag. The vector dimension is 50-dimensional to ensure computational efficiency and semantic accuracy. In the following embodiment, the first 5 dimensions of the 50-dimensional vector are used for verification during calculation. In actual use, the full vector is calculated. After calculation, the tag vector set of the core metadata tag set C={2026 / 1 / 6, Collection Device A, Device B, Importance Level C, Text, Item D, Level E, Unit F} is: Cvec={ [0.080, 0.150, -0.030, 0.220, -0.050], [0.240, 0.233, -0.127, 0.280, 0.117], [0.200, 0.215, -0.100, 0.240, 0.120], [0.260, 0.225, 0.080, 0.190, 0.200], [-0.450, 0.220, 0.580, -0.230, 0.520], [0.220, 0.180, -0.100, 0.215, 0.120], [0.255, 0.195, 0.075, 0.175, 0.200], [0.195, 0.335, -0.135, 0.450, -0.100]}.

[0023] After calculation, the tag vector set of the extended metadata tag set D = {General Test Report Template, Data Mining Use, Organization F} is: Dvec={ [0.260, 0.198, -0.108, 0.258, 0.155], [-0.450, 0.173, 0.547, -0.257, 0.507], [0.205, 0.325, -0.125, 0.440, -0.095]}.

[0024] In step S202 of this embodiment, when calculating the cosine similarity between each metadata tag in the core metadata tag set and the tag vector of each metadata tag in the extended metadata tag set, the cosine similarity between the tag vectors in the tag vector sets Cvec and Dvec is calculated iteratively: re_ij=cos(Cvec_i,Dvec_j)=(Cvec_i·Dvec_j) / (‖Cvec_i‖×‖Dvec_j‖) Where re_ij and cos(Cvec_i, Dvec_j) are the cosine similarity between label vectors Cvec_i and Dvec_j, where Cvec_i is the i-th element in the label vector set Cvec, Dvec_j is the j-th element in the label vector set Dvec, Cvec_i·Dvec_j is the dot product between label vectors Cvec_i and Dvec_j, and ||Cvec_i|| and ||Dvec_j|| are the Euclidean norms (L2 norms) of label vectors Cvec_i and Dvec_j, respectively. For example, the cosine similarity re_11 between the first label "2026 / 1 / 6" in the core metadata label set C and the first label "General Test Report Template" in the extended metadata label set D is approximately 0.791. The cosine similarity re_83 between the 8th tag "Unit F" in the core metadata tag set C and the 3rd tag "Organization F" in the extended metadata tag set D is approximately 0.373225 / 0.3733≈0.9998≈1.

[0025] Then, the cosine similarity re_ij in the extended metadata tag set is compared with a preset threshold rc, where the preset threshold rc is an empirical value, which can be taken as rc=0.95. If the cosine similarity re_ij in the extended metadata tag set exceeds the preset threshold rc, it is determined that the tag vector Dvec_j and the tag vector Cvec_i are semantically duplicated, and the metadata tags in the extended metadata tag set whose cosine similarity exceeds the preset threshold are deleted, finally obtaining the deduplicated extended metadata tag set. For example, for the cosine similarity re_11, since 0.791 < 0.95, it is determined that C1 and D1 are different tags. Also, since 0.791 is close to 0.95, it can be determined that the two tags have a strong semantic relationship, because in real-world scenarios, test reports usually need to be associated with the test date. For the cosine similarity re_83, since 1 > 0.95, C8 and D3 are considered semantically redundant labels. "Unit" and "organization" are synonyms, and when combined with the common word "F," their core meaning is "organizations / units identified by F," indicating a high degree of semantic consistency. Therefore, the third label in D needs to be deleted. The deduplicated extended metadata label set DN = {General Test Report Template, for Data Mining}. At this point, merging the two label sets yields the final label set: SLCFinal={C+DN}={2026 / 1 / 6, Data Acquisition Device A, Data Acquisition Object B, Importance Level C, Text, Item D, Hierarchy E, Unit F, General Test Report Template, For Data Mining} Then calculate the hash value of the tag set, TagHash.

[0026] After the user selects the source data file, determines the corresponding tag set, and clicks upload, the optimal number of file fragments can be calculated using a dynamic model that comprehensively considers file size, real-time network quality (including bandwidth, latency, and stability), and system load (including CPU, memory, and network). This process divides the source data file into N fragments. Specifically, in step S102 of this embodiment, the client dividing the source data file into N file fragments includes: S301, Calculate the base number of shards N_base: N_base = ceil(S / S_base); Where ceil is the rounding function, S is the file size of the source data file, and S_base is the base fragment size. In this embodiment, the file size of the source data file is 500M, and the base fragment size S_base is 32MB. Therefore, N_base = ceil(500 / 32) = 16. S302, calculate the adjusted number of shards N_adjusted based on the base number of shards N_base: N_adjusted=round(N_base×α×β); α=max(0.5,min(2.0,α_bandwidth×α_latency×α_stability)); β=max(0.6,min(1.3,(β_cpu×β_memory×β_network) 1 / 3 )); Where `round` is the rounding function, `α` is the network quality factor, `β` is the system load factor, `α_bandwidth`, `α_latency`, and `α_stability` are the bandwidth factor, latency factor, and stability factor, respectively. The bandwidth factor is obtained by mapping the available bandwidth of the network, the latency factor is obtained by mapping the network latency, and the stability factor is obtained by mapping the network packet loss rate. `β_cpu`, `β_memory`, and `β_network` are the CPU load factor, memory load factor, and network load factor, respectively. The CPU load factor is obtained by mapping the CPU utilization rate, the memory load factor is obtained by mapping the memory utilization rate, and the network load factor is obtained by mapping the network load. S303, the initial number of fragments N0 is obtained by applying the adjusted fragment number N_adjusted constraint: N0=max(1,min(N_max,N_adjusted)); Where max and min are the maximum and minimum value operation functions, respectively, and N_max is the preset maximum number of shards; in this embodiment, the preset maximum number of shards N_max is set to 512 to avoid excessive shards causing management complexity; S304, calculate the actual fragment size S / N0 based on the initial fragment size N0. If the actual fragment size S / N0 is greater than the preset maximum fragment size S_max, then determine the final fragment size N according to the following formula: N = max(1, ceil(S / S_max)); If the actual fragment size S / N0 is less than the preset minimum fragment size S_min, then the final number of fragments N is determined according to the following formula: N=min(N_max,floor(S / S_min)); Where floor is the floor function; if the actual fragment size S / N0 is between the minimum fragment size S_min and the maximum fragment size S_max, then the actual fragment size S / N0 is taken as the final number of fragments N; in this embodiment, the minimum fragment size S_min is set to 2MB to avoid management overhead caused by fragments that are too small; the maximum fragment size S_max is set to 128MB to avoid fragments that are too large affecting parallel transmission; S305, divide the source data file into N file fragments according to the final number of fragments N.

[0027] The bandwidth factor is obtained by mapping the available bandwidth of the network, the latency factor by mapping the network latency, and the stability factor by mapping the network packet loss rate. These network quality parameters, such as available bandwidth, network latency, and network packet loss rate, are obtained in real time through the browser's Network Information API. The bandwidth factor, latency factor, and stability factor can be mapped using the desired method, such as lookup tables, functions, or machine learning models. As an optional implementation, the calculation function expressions for the bandwidth factor, latency factor, and stability factor in this embodiment are as follows: α_bandwidth=0.6+0.9×(1-e -B / 15 ); α_latency = 1.2 × e -L / 120 ; α_stability=1.2×e -P / 4 ; Where B is the available bandwidth, e is the natural constant, L represents the network latency, and P represents the network packet loss rate. Assuming good network quality (available bandwidth B = 100 Mbps, network latency L = 20 ms, network packet loss rate P = 0.1%), then substituting the values ​​of each factor, the network quality factor α = max(0.5, min(2.0, α_bandwidth × α_latency × α_stability)) = max(0.5, min(2.0, 1.782)) = 1.782.

[0028] The CPU load factor is obtained by mapping CPU utilization, the memory load factor by mapping memory utilization, and the network load factor by mapping network load. These system load parameters, such as CPU utilization, memory utilization, and network load, are obtained through a RESTful API provided by the application server. The CPU load factor, memory load factor, and network load factor can be mapped using the desired method, such as table lookup, function, or machine learning model. As an optional implementation, the calculation function expressions for the CPU load factor, memory load factor, and network load factor in this embodiment are as follows: β_cpu = 1.3 - 0.7 × (1 - e -3×C ); β_memory = 1.3 - 0.7 × (1 - e -3×M ); β_network = 1.3 - 0.7 × (1 - e -3×K ); Where C represents CPU utilization, M represents memory utilization, and K represents network load. Assuming a relatively light system load (CPU utilization C = 15% = 0.15, memory utilization M = 20% = 0.20, network load K = 20% = 0.20), then the system load factor β can be calculated by substituting the values ​​of each factor: β = max(0.6, min(1.3, (β_cpu × β_memory × β_network)). 1 / 3 ))=max(0.6,min(1.3,1.0045))=1.0045≈1.

[0029] Based on this, we have: N_adjusted=round(N_base×α×β)=round(16×1.782×1.0)=29.

[0030] Using quantity constraints, the initial number of fragments is N0 = max(1, min(512, 29)) = 29. Verifying that the fragment size 500 / 29 ≈ 17.2 MB, which is between S_min = 2 MB and S_max = 128 MB, we arrive at the final number of fragments N = N0 = 29.

[0031] In this embodiment, step S102, constructing a Merkle tree based on the hash values ​​of N file fragments, includes: calculating the hash value H_i of each of the N file fragments, and merging the hash value H_i of each file fragment with its fragment index Index_i to generate an enhanced hash value STHash. STHash=SHA3-256(Index_i:H_i); SHA3-256 is a 256-bit hash function. The enhanced hash values ​​(STHash) of the N file fragments are used as leaf nodes of the Merkle tree, and the Merkle tree is constructed in index order. When constructing the Merkle tree, if the number of leaf nodes is insufficient, the last leaf node is copied multiple times, for example... Figure 2 Taking a source data file containing 5 file fragments as an example, Leaf_1 to Leaf_5 represent the enhanced hash value STHash calculated based on the hash values ​​of the 5 file fragments. For example, the calculation method for Leaf_1 is as follows: Leaf_1=SHA3-256(Index_1:H_1); Here, H_1 is the hash value of the first file fragment, Index_1 is the fragment index of the first file fragment, and so on. After constructing the required leaf nodes, intermediate nodes can be constructed according to the index order, for example... Figure 2 The 8 leaf nodes can be used to construct 4 intermediate nodes M1 to M4, where the hash value of intermediate node M1 is calculated as follows: M1 = SHA3-256(Leaf_1:Leaf_2); Similarly, four intermediate nodes M1 to M4 can be used to construct two intermediate nodes M5 and M6. The hash value of intermediate node M5 is calculated as follows: M5 = SHA3-256(M1:M2); Ultimately, the integrity identifier FileRoot can be constructed, which can be represented as: FileRoot=SHA3-256(M5:M6); In this embodiment, step S102, reconstructing the Merkle tree based on the integrity identifier FileRoot and the tag set identifier TagHash, includes: using the integrity identifier FileRoot and the tag set identifier TagHash as two nodes of the Merkle tree, respectively, and constructing a Merkle tree composed of the two nodes. Figure 2 As shown, the root hash value of the reconstructed Merkle tree, used as the global integrity identifier FileUID of the source data file, can be represented as: FileUID=SHA3-256(FileRoot: TagHash).

[0032] In step S102, the client can temporarily store the source data file, its tag set, and the global integrity identifier FileUID in the browser's IndexedDB. In step S103, the client can quickly upload the source data file, its tag set, and the global integrity identifier FileUID stored in the browser's IndexedDB to the remote application server. The browser sequentially uploads the 29 fragments and their associated data to the application server via multiple HTTP POST requests.

[0033] In this embodiment, step S103 involves storing the source data file and its tag set via the application server and submitting the global integrity identifier FileUID as a notarization transaction to the blockchain network to achieve tamper-proof notarization. S401, The application server verifies the N file fragments using the global integrity identifier FileUID. If the verification passes, it proceeds to step S402; otherwise, it sends a retransmission request to the client, ends the process, and exits. S402, the application server reassembles N file fragments into a complete data file; S403, the application server uploads the complete data file to the distributed file system and generates a file content identifier (FileCID). It stores the FileCID and the global integrity identifier (FileUID) in a relational database to establish a mapping relationship between the FileCID and the FileUID. The application server also stores the tag set of the source data file and the FileUID in the search engine database to establish a mapping relationship between the tag set and the FileUID. S404, the application server submits the global integrity identifier FileUID, the client's address and timestamp as evidence storage transactions to the blockchain network, realizing the immutable evidence storage of the source data file and its tag set.

[0034] In step S401 of this embodiment, the application server verifies the N file fragments using the global integrity identifier FileUID, including: S501, the application server calculates the hash value of each of the N received file fragments, and calculates the hash value of the received tag set to obtain the server tag set identifier TagHash_server; S502, the application server constructs a Merkle tree based on the hash values ​​of the received N file fragments, and uses the root hash value of the Merkle tree as the server-side file integrity identifier FileRoot_server of the source data file; it then constructs the Merkle tree again based on the server-side integrity identifier FileRoot_server and the server-side tag set identifier TagHash_server, and uses the root hash value of the newly constructed Merkle tree as the server-side global integrity identifier FileUID_server of the source data file. S503: Determine whether the server-side global integrity identifier FileUID_server and the received global integrity identifier FileUID are consistent. If they are consistent, the verification is deemed to have passed; otherwise, the verification is deemed to have failed.

[0035] In this embodiment, the application server immediately performs SHA3-256 hash verification on the tag set and each arriving fragment to ensure data integrity during transmission. After all fragments are uploaded and verified, a Merkle tree is constructed on the server side using the received Index_i, H_i, and tag set, yielding the MerkleRoot_Server. This MerkleRoot_Server is then compared with the FileUID sent by the client. If they match, the final integrity verification is passed. The server reassembles the 29 fragments into the original 500MB file in index order; otherwise, a retransmission request is made. The reassembled complete file is then uploaded to the IPFS cluster. IPFS uses the Rabin fingerprint algorithm to divide the file into content blocks and constructs an IPLD directed acyclic graph based on the data blocks. Finally, it calculates and returns the content identifier (FileCID) of the root node. The application server writes the key mapping relationship (FileUID, FileCID) into a PostgreSQL database. The file's FileUID and the plaintext tag "2026 / 1 / 6, Collection Device A, Collection Object B, Importance Level C, Text, Item D, Level E, Unit F, General Test Report Template, For Data Mining" are indexed into Elasticsearch. After confirming the file has been successfully stored in IPFS (obtaining the FileCID) and the mapping relationship is determined, the application server immediately performs blockchain notarization. The application server submits the `FileUID` (unique file identifier), `TagHash` (file tag set hash value), `uploader` (user's address), and `timestamp` (timestamp) to the blockchain network NET1, creating a record with the FileUID as the key in the on-chain storage mapping. Simultaneously, a `FileRecordStored` event is triggered, indicating successful notarization. The front-end application listens for the `FileRecordStored` event and, upon receiving it, displays "Upload and notarization successful" to the user. Through the above implementation methods, this embodiment realizes the entire process of file generation, fragmentation, uploading, integrity verification, distributed storage and final blockchain trusted evidence storage, taking into account efficiency, security, cost and user experience.

[0036] This embodiment also provides a data acquisition and storage system based on blockchain and multiple data tags, including interconnected clients and application servers. Both the client and application server include a microprocessor and a memory. The microprocessor is programmed or configured to execute the data acquisition and storage method based on blockchain and multiple data tags. As an optional implementation, the application server can be an application server cluster, deploying at least two high-performance servers and using frameworks such as Spring Boot to deploy the front-end and back-end applications. A PostgreSQL database is deployed to reliably store core structured metadata such as the mapping relationship between FileUID and FileCID. An Elasticsearch cluster is deployed to store basic file information and a complete list of plaintext tags, supporting efficient full-text search and multi-tag combined queries with its powerful inverted index capabilities. A unified API interface is provided externally through a load balancer such as Nginx or F5. This cluster is the business hub of the system, responsible for executing all the core business logic described below. As an optional implementation, the blockchain can be deployed as an independent blockchain node server, for example, configuring at least four servers to build a private chain using a PoA (Proof-of-Authority) consensus mechanism. These four nodes jointly maintain the full data of the blockchain, ensuring low-latency confirmation and high reliability of notarized transactions. Smart contracts are written in Solidity and deployed on the chain. Furthermore, the blockchain can also be integrated and deployed on application servers or application server clusters. Since application servers need to upload complete data files to the distributed file system, as an optional implementation, the distributed file system can be implemented using IPFS cluster nodes. At least three servers can be configured to form a private IPFS cluster. This cluster is responsible for persistently storing the complete file data and ensures high availability and reliability through built-in data redundancy and replication mechanisms. The distributed file system can also be integrated and deployed on application servers or application server clusters. In this configuration, clients, application servers, the blockchain, and the distributed file system can communicate securely via a dedicated network, using encryption protocols such as TLS / SSL to ensure the security of data transmission.

[0037] Those skilled in the art will understand that the technical solutions provided by this invention may take the form of a method, system, or computer program product. Therefore, this invention may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this invention may take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce an implementation of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0038] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A data acquisition and storage method based on blockchain and multi-data tags, characterized in that, Includes the following steps: S101, the client determines the source data file to be uploaded, as well as the core metadata tag set and extended metadata tag set of the source data file. The core metadata tag set and extended metadata tag set are deduplicated and merged to obtain a tag set, and the hash value of the tag set is calculated as the tag set identifier TagHash. S102, the client divides the source data file into N file fragments, constructs a Merkle tree based on the hash values ​​of the N file fragments, and uses the root hash value of the Merkle tree as the file integrity identifier FileRoot of the source data file; the client then constructs a Merkle tree again based on the integrity identifier FileRoot and the tag set identifier TagHash, and uses the root hash value of the newly constructed Merkle tree as the global integrity identifier FileUID of the source data file; S103, the client uploads the source data file, its tag set, and the global integrity identifier FileUID to the remote application server, so that the application server can store the source data file and its tag set and submit the global integrity identifier FileUID as a notarization transaction to the blockchain network to achieve tamper-proof notarization.

2. The data acquisition and storage method based on blockchain and multiple data tags according to claim 1, characterized in that, In step S101, the tag set obtained by deduplicating and merging the core metadata tag set and the extended metadata tag set includes: S201, each metadata tag in the core metadata tag set and the extended metadata tag set is converted into a corresponding tag vector using word vector tools; S202, For each metadata tag in the core metadata tag set, calculate the cosine similarity between it and the tag vector of each metadata tag in the extended metadata tag set, and delete the metadata tags in the extended metadata tag set whose cosine similarity exceeds a preset threshold, and finally obtain the deduplicated extended metadata tag set. S203, combine the core metadata tag set with the deduplicated extended metadata tag set to obtain the tag set.

3. The data acquisition and storage method based on blockchain and multiple data tags according to claim 1, characterized in that, In step S102, the client splits the source data file into N file fragments, including: S301, Calculate the base number of shards N_base: N_base = ceil(S / S_base); Where ceil is the floor function, S is the file size of the source data file, and S_base is the base fragment size; S302, calculate the adjusted number of shards N_adjusted based on the base number of shards N_base: N_adjusted=round(N_base×α×β); α=max(0.5,min(2.0,α_bandwidth×α_latency×α_stability)); β=max(0.6,min(1.3,(β_cpu×β_memory×β_network) 1 / 3 )); Where `round` is the rounding function, `α` is the network quality factor, `β` is the system load factor, `α_bandwidth`, `α_latency`, and `α_stability` are the bandwidth factor, latency factor, and stability factor, respectively. The bandwidth factor is obtained by mapping the available bandwidth of the network, the latency factor is obtained by mapping the network latency, and the stability factor is obtained by mapping the network packet loss rate. `β_cpu`, `β_memory`, and `β_network` are the CPU load factor, memory load factor, and network load factor, respectively. The CPU load factor is obtained by mapping the CPU utilization rate, the memory load factor is obtained by mapping the memory utilization rate, and the network load factor is obtained by mapping the network load. S303, the initial number of fragments N0 is obtained by applying the adjusted fragment number N_adjusted constraint: N0=max(1,min(N_max,N_adjusted)); Where max and min are the maximum and minimum value operation functions, respectively, and N_max is the preset maximum number of partitions; S304, calculate the actual fragment size S / N0 based on the initial fragment size N0. If the actual fragment size S / N0 is greater than the preset maximum fragment size S_max, then determine the final fragment size N according to the following formula: N = max(1, ceil(S / S_max)); If the actual fragment size S / N0 is less than the preset minimum fragment size S_min, then the final number of fragments N is determined according to the following formula: N=min(N_max,floor(S / S_min)); Where floor is the floor function; if the actual fragment size S / N0 is between the minimum fragment size S_min and the maximum fragment size S_max, then the actual fragment size S / N0 is taken as the final number of fragments N; S305, divide the source data file into N file fragments according to the final number of fragments N.

4. The data acquisition and storage method based on blockchain and multiple data tags according to claim 3, characterized in that, The calculation function expressions for the bandwidth factor, delay factor, and stability factor are as follows: α_bandwidth=0.6+0.9×(1-e -B / 15 ); α_latency=1.2×e -L / 120 ; α_stability=1.2×e -P / 4 ; Where B is the available bandwidth, e is the natural constant, L represents the network latency, and P represents the network packet loss rate.

5. The data acquisition and storage method based on blockchain and multiple data tags according to claim 3, characterized in that, The calculation function expressions for the CPU load factor, memory load factor, and network load factor are as follows: β_cpu=1.3-0.7×(1-e -3×C ); β_memory=1.3-0.7×(1-e -3×M ); β_network=1.3-0.7×(1-e -3×K ); Where C represents CPU utilization, M represents memory utilization, and K represents network load.

6. The data acquisition and storage method based on blockchain and multiple data tags according to claim 1, characterized in that, Step S102, which involves constructing a Merkle tree based on the hash values ​​of the N file fragments, includes: calculating the hash value H_i of each of the N file fragments, and merging the hash value H_i of each file fragment with its fragment index Index_i to generate an enhanced hash value STHash. STHash=SHA3-256(Index_i:H_i); SHA3-256 is a 256-bit hash function; the enhanced hash values ​​STHash of the N file fragments are used as leaf nodes of the Merkle tree, and the Merkle tree is constructed in index order.

7. The data acquisition and storage method based on blockchain and multiple data tags according to claim 1, characterized in that, Step S102, which involves reconstructing the Merkle tree based on the integrity identifier FileRoot and the tag set identifier TagHash, includes: using the integrity identifier FileRoot and the tag set identifier TagHash as two nodes of the Merkle tree, and constructing a Merkle tree composed of the two nodes.

8. The data acquisition and storage method based on blockchain and multiple data tags according to claim 1, characterized in that, Step S103, which involves storing the source data file and its tag set via the application server and submitting the global integrity identifier FileUID as a notarization transaction to the blockchain network to achieve tamper-proof notarization, includes: S401, The application server verifies the N file fragments using the global integrity identifier FileUID. If the verification passes, it proceeds to step S402; otherwise, it sends a retransmission request to the client, ends the process, and exits. S402, the application server reassembles N file fragments into a complete data file; S403, the application server uploads the complete data file to the distributed file system and generates a file content identifier (FileCID). It stores the FileCID and the global integrity identifier (FileUID) in a relational database to establish a mapping relationship between the FileCID and the FileUID. The application server also stores the tag set of the source data file and the FileUID in the search engine database to establish a mapping relationship between the tag set and the FileUID. S404, the application server submits the global integrity identifier FileUID, the client's address and timestamp as evidence storage transactions to the blockchain network, realizing the immutable evidence storage of the source data file and its tag set.

9. The data acquisition and storage method based on blockchain and multiple data tags according to claim 8, characterized in that, In step S401, the application server verifies the N file fragments using the global integrity identifier FileUID, including: S501, the application server calculates the hash value of each of the N received file fragments, and calculates the hash value of the received tag set to obtain the server tag set identifier TagHash_server; S502, the application server constructs a Merkle tree based on the hash values ​​of the received N file fragments, and uses the root hash value of the Merkle tree as the server-side file integrity identifier FileRoot_server of the source data file; it then constructs the Merkle tree again based on the server-side integrity identifier FileRoot_server and the server-side tag set identifier TagHash_server, and uses the root hash value of the newly constructed Merkle tree as the server-side global integrity identifier FileUID_server of the source data file. S503: Determine whether the server-side global integrity identifier FileUID_server and the received global integrity identifier FileUID are consistent. If they are consistent, the verification is deemed to have passed; otherwise, the verification is deemed to have failed.

10. A data acquisition and storage system based on blockchain and multi-data tags, comprising interconnected client and application server, wherein both the client and application server include a microprocessor and a memory, characterized in that, The microprocessor is programmed or configured to execute the data acquisition and storage method based on blockchain and multiple data tags as described in any one of claims 1 to 9.

Images