Heterogeneous data source right confirmation method based on block chain

Abstract

The invention belongs to the technical field of data management, and particularly relates to a block chain-based heterogeneous data source right confirmation method, which comprises the steps of multi-source heterogeneous data acquisition, digital fingerprint generation, block chain hash anchoring, data storage and circulation and ownership tracing and verification. According to the invention, original data is collected from various terminals and system platforms by adopting a multi-source heterogeneous data acquisition technology so as to ensure the diversity and universality of data sources, and the collected original data is processed and unique digital fingerprints are generated. The method comprises the following steps: processing a digital fingerprint through a hash algorithm to generate a corresponding block chain hash value, anchoring the block chain hash value on a block chain, ensuring authority and permanency of a data ownership record, storing original data in a safe data warehouse, recording the hash value and ownership information of the data on the block chain at the same time, and storing the data in the block chain. Through combination of multi-source heterogeneous data acquisition, digital fingerprint generation and block chain hash anchoring technologies, automatic registration and efficient management of data ownership are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and specifically to a method for authenticating the rights of heterogeneous data sources based on blockchain. Background Art

[0002] In today's digital age, the wave of information technology is developing rapidly at an unprecedented speed. Emerging technologies such as the Internet, Internet of Things, cloud computing, big data, and artificial intelligence are intertwined and deeply integrated, driving profound changes in various fields of society. In this process, data, as the core product of information technology, has become increasingly prominent and has become a key production factor driving economic development, social progress, and technological innovation;

[0003] With the wide application of data in various fields, the integration and circulation of multi-source heterogeneous data have become an inevitable trend. Multi-source data comes from different data sources, such as different business systems within an enterprise, external partners, social media platforms, etc. There are significant differences in aspects such as data format, data structure, and data quality. Heterogeneous data covers various types such as text, images, audio, and video, increasing the complexity of data processing; in the process of integrating and circulating multi-source heterogeneous data, the confirmation and management of data ownership have become a key problem that needs to be solved urgently. The traditional centralized data management method has the defects that data is easily tampered with and the ownership is difficult to trace, and it cannot meet the complex needs of the modern data ecosystem;

[0004] In view of the above technical defects, a solution is proposed. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for authenticating the rights of heterogeneous data sources based on blockchain, which solves the problems in the prior art that it is difficult to effectively confirm and manage data ownership in the process of integrating and circulating multi-source heterogeneous data, there are defects that data is easily tampered with and the ownership is difficult to trace, and it cannot meet the complex needs of the modern data ecosystem.

[0006] To achieve the above purpose, the present invention provides the following technical solutions:

[0007] A method for authenticating the rights of heterogeneous data sources based on blockchain, comprising the following steps:

[0008] Step 1: Multi-source heterogeneous data collection: clarify the goals, scope, and specific business scenario requirements of data collection, classify data sources into structured data, semi-structured data, and unstructured data, evaluate the reliability, update frequency, and compliance of data sources, and screen data sources that meet the collection criteria;

[0009] Select the corresponding acquisition tool according to the data source type, design a distributed acquisition node to support high-concurrency data acquisition, select real-time streaming acquisition or batch timed acquisition based on the characteristics of the data source, remove duplicate, incorrect or invalid data, fill in missing values, unify the data format, and convert data in different formats into a unified data model;

[0010] Step 2: Digital fingerprint generation: Extract key features from the preprocessed data, select the corresponding hash algorithm according to the data type, input the feature data into the hash algorithm to generate a unique digital fingerprint, and store the digital fingerprint in a local database or a distributed file system;

[0011] Step 3: Blockchain hash anchoring: Select a public chain, consortium chain or private chain according to business requirements, build a blockchain network, deploy multiple consensus nodes, define a smart contract for data ownership registration, deploy the smart contract to the blockchain network, and initialize test data; Submit the digital fingerprint to the blockchain through the smart contract to generate an on-chain transaction record, and use the immutable timestamp of the blockchain to record the generation time of data ownership;

[0012] Step 4: Data storage and circulation;

[0013] Step 5: Ownership traceability and verification.

[0014] Furthermore, in the step of selecting the corresponding acquisition tool according to the data source type in Step 1, for structured data, use an ETL tool or a database connector, for semi-structured data, adopt API interface calls, and for unstructured data, utilize the file transfer protocol or object storage service.

[0015] Furthermore, in the step of selecting the corresponding hash algorithm according to the data type in Step 2, for text data, adopt the SHA-256 or MD5 algorithm to generate a hash value of a fixed length, and for image or video data, use the perceptual hash algorithm or deep learning feature extraction.

[0016] Furthermore, in the step of defining the smart contract for data ownership registration in Step 3, it includes data registration, ownership change and query interface functions. Among them, data registration is used to receive digital fingerprints and metadata, ownership change is used to support the transfer and update of data ownership, and the query interface is used to provide a hash value query interface and return ownership information.

[0017] Furthermore, the specific process of Step 4 is as follows:

[0018] Data warehouse design: Adopt a hot and cold data separation strategy, store frequently accessed data in high-speed storage, and store low-frequency data in object storage; Generate metadata for each data file, including file path, size and format, and associate it with the blockchain hash value;

[0019] Data circulation and sharing: Based on the blockchain ownership information, implement fine-grained data access control, formulate data exchange standards, record the usage logs of data during the data flow process, including access time and user ID, and associate them with the blockchain records.

[0020] Further, the specific process of step five is as follows:

[0021] Ownership query and traceability: Provide RESTful API or SDK to support users to query data ownership information through blockchain hash values, and generate a complete traceability path of data according to the blockchain records;

[0022] Ownership verification and anti-tampering: Use the public key of the data owner to verify the digital signature. Before using the data, recalculate the data fingerprint and compare it with the hash value stored in the blockchain to verify the data integrity.

[0023] Further, in step five, when there is an ownership dispute, the arbitration logic is automatically executed through the blockchain records and smart contracts, and all data ownership changes and access operations are recorded to generate an immutable audit log.

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] In the present invention, through five steps of multi-source heterogeneous data collection, digital fingerprint generation, blockchain hash anchoring, data storage and circulation, and ownership traceability and verification, the full life cycle management of data ownership can be carried out. By combining multi-source heterogeneous data collection, digital fingerprint generation, and blockchain hash anchoring technologies, the automatic registration and efficient management of data ownership are realized, ensuring the safe circulation and traceability of data in a complex ecosystem, providing a reliable guarantee for the efficient circulation of data elements, and having broad application prospects. Brief Description of the Drawings

[0026] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with the drawings;

[0027] Figure 1 is the overall method flow chart of the present invention;

[0028] Figure 2 is the flow block diagram of step one in the present invention;

[0029] Figure 3 is the flow block diagram of step two in the present invention;

[0030] Figure 4 is the flow block diagram of step three in the present invention;

[0031] Figure 5 is the flow block diagram of step four in the present invention;

[0032] Figure 6 This is the flowchart of step five in the present invention. Detailed implementation manners

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] Embodiment 1: As Figures 1-6 shown, the method for authenticating heterogeneous data sources based on blockchain proposed by the present invention includes the following steps:

[0035] Step 1. Multi-source heterogeneous data collection:

[0036] Requirement analysis and data source identification: Business requirement research: clarify the goals, scope, and specific business scenario requirements of data collection; data source classification: classify data sources into structured data (such as databases, ERP systems), semi-structured data (such as XML, JSON files), and unstructured data (such as documents, images, videos); data source evaluation: evaluate the reliability, update frequency, and compliance of data sources, and screen data sources that meet the collection standards;

[0037] Technical selection and tool deployment: Collection tool selection: Select corresponding collection tools according to the data source type. For structured data, use ETL tools (such as Apache NiFi, Informatica) or database connectors. For semi-structured data, use API interface calls, and for unstructured data, use file transfer protocols (such as FTP, SFTP) or object storage services (such as AWS S3); Distributed collection architecture: Design distributed collection nodes to support high-concurrency data collection and ensure data real-time performance;

[0038] Data collection and preprocessing: Select real-time streaming collection or batch timed collection based on the characteristics of the data source, remove duplicate, incorrect, or invalid data, fill in missing values, unify the data format (such as timestamp standardization), and convert data in different formats into a unified data model (such as JSON Schema) for subsequent processing;

[0039] Step 2. Digital fingerprint generation:

[0040] Feature Extraction and Hash Algorithm Selection: Feature Extraction: Extract key features from the preprocessed data (such as document content summaries, image pixel matrices, video frame sequences); Hash Algorithm Selection: Select the corresponding hash algorithm according to the data type. For text data, use the SHA-256 or MD5 algorithm to generate a fixed-length hash value. For image or video data, use perceptual hash algorithms (such as aHash, dHash) or deep learning feature extraction (such as ResNet feature vectors);

[0041] Digital Fingerprint Generation and Storage: Fingerprint Generation: Input the feature data into the hash algorithm to generate a unique digital fingerprint (such as a 128-bit binary string); Fingerprint Storage: Store the digital fingerprint in a local database or a distributed file system (such as HDFS) for subsequent queries;

[0042] Step Three: Blockchain Hash Anchoring:

[0043] Blockchain Platform Selection and Deployment: Platform Selection: Select a public blockchain (such as Ethereum), a consortium blockchain (such as Hyperledger Fabric), or a private blockchain according to business requirements, and build a blockchain network; Node Deployment: Deploy multiple consensus nodes (such as PoW, PBFT) to ensure network decentralization and high availability;

[0044] Smart Contract Development: Contract Design: Define a smart contract for data ownership registration, including data registration, ownership change, and query interface functions. Among them, data registration is used to receive digital fingerprints and metadata (such as data source, generation time), ownership change is used to support the transfer and update of data ownership, and the query interface is used to provide a hash value query interface and return ownership information; Contract Deployment: Deploy the smart contract to the blockchain network and initialize test data;

[0045] Hash Value Anchoring and On-chain Storage: Hash Value Submission: Submit the digital fingerprint to the blockchain through the smart contract to generate an on-chain transaction record; Timestamp Service: Use the immutable timestamp of the blockchain to record the generation time of data ownership;

[0046] Step Four: Data Storage and Circulation:

[0047] Data Warehouse Design: Hierarchical Storage: Adopt a hot and cold data separation strategy, store frequently accessed data in high-speed storage (such as Redis), and store infrequently accessed data in object storage (such as MinIO); Metadata Management: Generate metadata (such as file path, size, format) for each data file and associate it with the blockchain hash value;

[0048] Data Circulation and Sharing: Permission Control: Implement fine-grained data access control based on blockchain ownership information (such as the RBAC model); Data Exchange Protocol: Develop data exchange standards (such as OAuth2.0, OpenID Connect); Data Traceability: During the data flow process, record the usage logs of data (such as access time, user ID) and associate them with blockchain records;

[0049] Step Five: Ownership Traceability and Verification:

[0050] Ownership Query and Traceability: Query Interface: Provide RESTful API or SDK to support users in querying data ownership information through blockchain hash values; Traceability Path Generation: Generate the complete traceability path of data based on blockchain records (such as data source → processing node → end user);

[0051] Ownership Verification and Anti-Tampering: Digital Signature Verification: Use the public key of the data owner to verify the digital signature to ensure that the data has not been tampered with; Hash Value Comparison: Before using the data, recalculate the data fingerprint and compare it with the hash value stored on the blockchain to verify the data integrity;

[0052] Dispute Resolution and Audit: Dispute Handling: When ownership disputes occur, automatically execute arbitration logic through blockchain records and smart contracts; Audit Log: Record all data ownership changes and access operations to generate an immutable audit log.

[0053] The working principle of the present invention: When in use, collect raw data from various terminals and system platforms by adopting multi-source heterogeneous data collection technology to ensure the diversity and extensiveness of data sources, providing a basis for subsequent data processing and analysis. And process the collected raw data to generate unique digital fingerprints to distinguish different data contents to ensure the uniqueness and immutability of the data. Process the digital fingerprints through a hash algorithm to generate corresponding blockchain hash values and anchor them on the blockchain. The immutable feature of the blockchain ensures the authority and permanence of the data ownership records, and store the raw data in a secure data warehouse. At the same time, record the hash values and ownership information of the data on the blockchain. During the data flow process, the ownership and circulation path of the data can be quickly confirmed by querying the records on the blockchain. By combining multi-source heterogeneous data collection, digital fingerprint generation, and blockchain hash anchoring technology, automatic registration and efficient management of data ownership are achieved.

[0054] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only the specific embodiments. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A method for confirming the ownership of heterogeneous data sources based on blockchain, characterized in that: The following steps are involved: Step 1: Multi-source heterogeneous data collection: clarify the goals, scope and specific business scenario requirements of data collection, classify data sources into structured data, semi-structured data and unstructured data, evaluate the reliability, update frequency and compliance of data sources, and select data sources that meet the collection standards; Select the corresponding collection tool according to the data source type, design distributed collection nodes, support high-concurrency data collection, select real-time streaming collection or batch timing collection based on the characteristics of the data source, remove duplicate, erroneous or invalid data, fill in missing values, unify data formats, and convert data in different formats into a unified data model; Step 2: Digital fingerprint generation: extract key features from the preprocessed data, select the corresponding hash algorithm according to the data type, input the feature data into the hash algorithm, generate a unique digital fingerprint, and store the digital fingerprint in a local database or distributed file system; Step 3: Blockchain hash anchoring: Select a public chain, consortium chain or private chain according to business needs, build a blockchain network, deploy multiple consensus nodes, define smart contracts for data ownership registration, deploy smart contracts to the blockchain network, and initialize test data; Submit the digital fingerprint to the blockchain through a smart contract to generate a transaction record on the chain, and use the blockchain's tamper-proof timestamp to record the generation time of the data ownership; Step 4: Data storage and circulation; Step 5: Ownership tracing and verification.

2. The method for confirming the ownership of heterogeneous data sources based on blockchain according to claim 1 is characterized in that: In step one, for structured data, use ETL tools or database connectors, for semi-structured data, use API interface calls, and for unstructured data, use file transfer protocols or object storage services.

3. The method for confirming the ownership of heterogeneous data sources based on blockchain according to claim 1 is characterized in that: In step 2, for text data, the SHA-256 or MD5 algorithm is used to generate a fixed-length hash value, and for image or video data, the perceptual hash algorithm or deep learning feature extraction is used.

4. The method for confirming the ownership of heterogeneous data sources based on blockchain according to claim 1 is characterized in that: In the smart contract for defining data ownership registration in step three, data registration, ownership change and query interface functions are included.

5. The method for confirming the ownership of heterogeneous data sources based on blockchain according to claim 1 is characterized in that: The specific process of step 4 is as follows: Data warehouse design: adopt a cold and hot data separation strategy to store high-frequency access data in high-speed storage and low-frequency data in object storage; generate metadata for each data file and associate it with the blockchain hash value; Data circulation and sharing: Implement fine-grained data access control based on blockchain ownership information, formulate data exchange standards, record data usage logs during data circulation, and associate them with blockchain records.

6. The method for confirming the ownership of heterogeneous data sources based on blockchain according to claim 1 is characterized in that: The specific process of step five is as follows: Ownership query and traceability: provide RESTful API or SDK to support users to query data ownership information through blockchain hash value, and generate a complete traceability path for data based on blockchain records; Ownership verification and tamper-proofing: use the public key of the data owner to verify the digital signature, recalculate the data fingerprint before using the data and compare it with the hash value stored in the blockchain to verify the data integrity.

7. The method for confirming the ownership of heterogeneous data sources based on blockchain according to claim 6 is characterized in that: In step five, when an ownership dispute occurs, the arbitration logic is automatically executed through blockchain records and smart contracts, and all data ownership changes and access operations are recorded to generate an unalterable audit log.

Images