Content addressable hybrid file storage method and device and storage medium

Abstract

The invention discloses a content addressable hybrid file storage method and device and a storage medium. According to the invention, through combination of specific hardware design and software architecture, efficient, safe and reliable distributed storage and sharing of data in scenes such as families, Internet of Things, enterprises and the like are realized. According to the method, efficient positioning and deduplication are achieved through content addressable, the P2P network is used for optimizing data direct transmission efficiency and reducing center dependence, data reliability is guaranteed through local and network redundancy, and data safety is guaranteed through refined authority control.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a content-addressable hybrid file storage method, device, and storage medium. Background Art

[0002] Traditional centralized cloud storage technologies (such as Amazon S3 and Google Drive): These technologies rely on centralized server clusters in large data centers to store and manage user data. Data transmission, access, and sharing are all handled through central servers. Their key features are centralized management and ease of deploying universal services. However, inherent drawbacks include the risk of single points of failure, data privacy and security being relatively dependent on service provider policies, and a strong reliance on network bandwidth, which can result in high operating costs. Furthermore, centralized transmission can lead to high access latency, especially in edge computing scenarios, resulting in delayed responses.

[0003] Distributed storage technology based on IPFS (InterPlanetary File System): IPFS uses content addressability (locating files using content hashes or CIDs) and a peer-to-peer (P2P) network for distributed file storage and retrieval. Its core features are decentralization, content addressability, and data distribution. In theory, this can improve storage efficiency and censorship resistance. However, in practice, especially in resource-constrained edge environments like homes and the Internet of Things, it faces limitations such as insufficient data persistence (depending on node availability), a lack of mature, fine-grained permission management mechanisms, low cold data access efficiency, and insufficient response latency for real-time applications (such as smart home control).

[0004] General-purpose P2P (peer-to-peer) network file sharing technologies (such as BitTorrent): These technologies directly leverage the resources (storage and bandwidth) of user nodes within the network for data transmission and sharing. They are characterized by decentralized resource sharing and high bandwidth utilization (especially when distributing popular resources). However, their primary design goal is file distribution, not persistent, reliable storage. Significant challenges exist in data consistency, reliability, version management, fast retrieval (especially for less popular content), and cross-device state synchronization, making them difficult to directly meet the complex data storage and management needs of homes, IoT, or enterprises.

[0005] Traditional distributed file systems (e.g., HDFS): These systems are designed to store very large datasets on a cluster of multiple servers, achieving high fault tolerance and high throughput through data partitioning and redundancy. They are characterized by high scalability and fault tolerance, making them suitable for big data analytics. However, their architecture is complex, typically designed for data centers, and unsuitable for resource-constrained home edge devices or IoT terminals. Furthermore, their design focuses on internal cluster management, with weak support for content addressability, cross-domain P2P sharing, and real-time permission control, making it difficult to meet the flexibility and real-time requirements of the scenarios targeted by this invention.

[0006] In summary, existing technologies are either overly centralized, posing performance, cost, and privacy risks. While adopting a distributed approach (e.g., IPFS and P2P), they lack data persistence, permission management, usability, real-time performance, and adaptability to specific scenarios (home, IoT, and enterprise). Or, like traditional distributed file systems, they are unsuitable for resource-constrained edge environments. Existing technologies fail to provide a comprehensive solution that effectively integrates content addressability, P2P transmission, data redundancy, and secure permission control, and can efficiently adapt to diverse edge or distributed application scenarios, such as home, IoT, and enterprise.

[0007] When existing technologies are applied to distributed scenarios such as home edge computing, the Internet of Things, and enterprise collaboration, they have the following technical problems that have not been effectively solved: Inherent risks and bottlenecks of centralized architecture: The centralized nature of traditional cloud storage leads to inherent bottlenecks in reliability (single point failure), security (risk of privacy leakage), cost (long-term storage and bandwidth fees), and access performance (high latency, dependence on backbone networks), making it difficult to meet the edge scenarios' requirements for high reliability, low latency, and high privacy protection.

[0008] Limitations of existing distributed technologies such as IPFS in specific scenarios: Insufficient data persistence and availability: IPFS relies on nodes to contribute storage. Nodes going offline or exiting the network may cause data to become inaccessible. It lacks a strong data persistence guarantee mechanism and is not suitable for storing critical home data or corporate files.

[0009] Imperfect permission management and secure sharing mechanisms: The lack of a native, flexible, and easy-to-use fine-grained permission control mechanism makes it difficult to meet the needs of granular management of data access rights within family members or corporate teams. Directly sharing hashes can lead to uncontrollable data exposure.

[0010] Real-time and performance bottlenecks: For scenarios such as IoT monitoring and smart home control that require low-latency responses, the discovery and transmission mechanisms of existing IPFS / P2P networks may not guarantee sufficient real-time performance.

[0011] Deficiencies of P2P technology in reliable storage and management: General P2P technology focuses on temporary sharing rather than long-term reliable storage. It lacks effective data redundancy, version control, consistency maintenance, and efficient retrieval mechanisms, making it difficult to serve as a reliable data storage infrastructure.

[0012] Traditional distributed file systems have poor adaptability to edge environments: Systems such as HDFS have complex architectures and high resource overhead, making them unsuitable for resource-constrained homes or IoT devices. At the same time, their design concepts are at odds with the ease of use and flexibility (such as content addressing and P2P sharing) required in home and enterprise scenarios.

[0013] In summary, existing technical solutions have obvious shortcomings in integrating the reliability of distributed storage, the efficiency of content addressability, the low latency of P2P transmission, the security of refined permission control, and the adaptability to resource-constrained edge environments. They are unable to provide a unified and efficient data storage and sharing solution that fully meets the needs of scenarios such as home, Internet of Things, and enterprise collaboration.

[0014] Therefore, the existing technology has defects and needs to be improved. Summary of the Invention

[0015] The technical problem to be solved by the present invention is to provide a content-addressable hybrid file storage method, device and storage medium to achieve efficient, secure and reliable distributed storage and sharing of data in scenarios such as home, Internet of Things, and enterprise.

[0016] The technical solution of the present invention is as follows: providing a content-addressable hybrid file storage method based on the application interface layer, data processing and routing layer, storage and node management layer, security and encryption layer: including the following processes: data writing (storage) process, data reading (retrieval) process, data sharing and permission control process.

[0017] The data writing (storage) process includes the following steps: S1: The user uploads a file through the application interface layer; S2: The data processing and routing layer receives the file and splits it into data blocks; S3: The data processing and routing layer calculates a hash value for each data block and generates a unique data block CID. The data processing and routing layer calculates a hash value for the entire file and generates a top-level CID. S4: Security and encryption layer encrypts the data blocks; S5: The storage and node management layer stores the encrypted data blocks in the local storage unit; S6: Based on the preset redundancy strategy, the storage and node management layer selects other nodes in the network and distributes copies of the data block through the enhanced distributed transmission protocol stack; S7: The storage and node management layer stores and manages metadata.

[0018] The data reading (retrieval) process includes the following steps: A1: The user requests to access the file through the application interface layer; A2: The application interface layer searches for the corresponding top-level CID. A3: The data processing and routing layer parses the top-level CID to obtain the required data block CID list; A4: The data processing and routing layer queries the node network (including local nodes) for each required data block CID through the content routing mechanism to locate the node holding the data block; A5: The storage and node management layer obtains the encrypted data blocks from the nearest or fastest node; A6: The security and encryption layer decrypts and verifies the obtained encrypted data blocks; A7: The data processing and routing layer reorganizes the data blocks into original files in sequence and returns them to the application interface layer.

[0019] The data sharing and permission control process includes the following steps: B1: The user selects the file to be shared; B2: The data processing and routing layer generates a secure sharing credential associated with the top-level CID of the file; B3: The user shares this secure sharing credential with the authorized recipient; B4: The recipient initiates an access request using the secure shared credentials; B5: The data processing and routing layer verifies the validity of the secure sharing credentials and the requester's authority; B6: After verification, data access is provided according to the data reading process (usually through direct P2P transmission).

[0020] Furthermore, in step S3, the data block CID and the top-level CID adopt a Merkle DAG structure.

[0021] Furthermore, in step S7, the metadata includes: source data of the file, mapping relationship between the file and the top-level CID, mapping relationship between the data block and the data block CID, mapping relationship between the top-level CID and the data block CID, and permission information.

[0022] Furthermore, in step A1, the user requests to access the file through the application interface layer using the file name or the top-level CID of the file.

[0023] Furthermore, in step A5, the nearest or fastest node refers to: first selecting a local node, and then selecting other nodes in the P2P network.

[0024] Furthermore, in step B1, it also includes: selecting a target recipient to be shared, and the target recipient is limited by limiting conditions, and the limiting conditions are at least one of the unique identifier of the device, the IP address, and the recipient ID.

[0025] Furthermore, the secure sharing credential is an encrypted link or token, which contains access permission information; the permission information includes: at least one of validity period, number of visits, read, write, and download permissions.

[0026] Furthermore, the present invention also provides a content-addressable hybrid file storage device, comprising: Core processing unit: used to perform data sharding, hash calculation, data encryption / decryption, network protocol processing, and device management tasks; Storage unit: used to store file data blocks, metadata, cached data, and some redundant data from other nodes in the network; Network interface unit: used for the device to access the local network and exchange data with other network nodes; Power management unit: used to provide stable power to the entire device; Expansion interface unit: used to connect external storage devices to expand capacity or import and export data.

[0027] The storage unit stores a computer program, and the processor is used to read the computer program and execute the aforementioned content-addressable hybrid file storage method.

[0028] Furthermore, the present invention also provides a computer-readable storage medium, characterized in that the computer program is stored in the computer-readable storage medium, and the computer program implements the aforementioned content-addressable hybrid file storage method when read and executed by one or more processors.

[0029] By adopting the above solution, the present invention provides a content-addressable hybrid file storage method, device, and storage medium, which have the following technical effects: High reliability: Through data sharding and multi-node redundant storage: local RAID (disk array) optional + network distributed redundancy, the risk of single point failure is significantly reduced and data persistence is improved.

[0030] Enhanced security and privacy protection: End-to-end encryption, content-based addressing (not easily scanned directly), and refined permission control are combined to effectively protect user data privacy and transmission security.

[0031] Low-latency access and efficient transmission: Prioritize local access, combined with an optimized P2P direct transmission protocol, to reduce dependence on central servers and lower access latency, especially suitable for LAN or edge environments.

[0032] High scalability and flexibility: Nodes can be added on demand, and storage capacity is easily expanded. The hybrid architecture can flexibly adapt to different network environments and application requirements (home, IoT, enterprise, etc.).

[0033] Storage space optimization: The content-addressing feature naturally supports data deduplication. The same content block only needs to be stored once, saving storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Flowchart of the data writing process of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] This invention provides a hybrid distributed storage solution that combines content addressability (similar to IPFS), peer-to-peer (P2P) transmission, local redundant storage, and optional cloud backup / coordination mechanisms. It also provides specialized hardware devices and supporting methods for implementing this solution. This solution aims to leverage content addressability for efficient location and deduplication, utilize P2P networks to optimize direct data transmission efficiency and reduce central reliance, ensure data reliability through local and network redundancy, and safeguard data security through refined permission control.

[0037] The present invention provides a content-addressable hybrid file storage device, which is intended to serve as a node in a distributed storage network. Its typical structure includes: Core processing unit: used to perform data sharding, hash calculation, data encryption / decryption, network protocol processing, and device management tasks; specifically: configured with a processor (for example, an embedded multi-core processor) that can efficiently perform tasks such as data sharding, hash calculation (generating content identifier CID), data encryption / decryption, network protocol processing (such as P2P protocol stack, content routing protocol), and device management.

[0038] Storage unit: Used to store file data blocks, metadata, cached data, and partially redundant data from other nodes in the network. Specifically, it integrates non-volatile storage media with a certain capacity (such as SSD, HDD, or a combination thereof, with configurable capacity ranging from 2TB to 12TB or higher) to store user data blocks, metadata, cached data, and partially redundant data from other nodes in the network. Local redundancy strategies such as RAID can be configured to enhance single-point data reliability.

[0039] Network Interface Unit: This unit allows the device to access the local network and exchange data with other network nodes. Specifically, it includes at least one high-speed wired network interface (e.g., Gigabit Ethernet) for accessing the local network and exchanging data with other network nodes. Wireless interfaces (e.g., Wi-Fi, Bluetooth, etc.) can be configured as needed.

[0040] Power management unit: Used to provide stable power to the entire device; and may include a power adapter interface (such as DC input) and an optional backup power mechanism (such as a built-in small-capacity battery or supercapacitor) to cope with brief power outages and protect ongoing operations and data integrity.

[0041] Expansion interface unit: may include standard interfaces such as USB for connecting external storage devices to expand capacity or import and export data.

[0042] The storage unit stores a computer program, and the processor is used to read the computer program and execute the aforementioned content-addressable hybrid file storage method.

[0043] Brief description of the hardware working mechanism: The core processing unit runs the software logic defined in this invention, handles data reception, sharding, hashing, encryption, and storage decisions (local storage, initiating P2P distribution), communicates with other nodes through the network interface unit, responds to data retrieval requests (based on CID), performs data transmission (prioritizing P2P), and manages local storage space and device status.

[0044] The data management method of the present invention for implementing the above-mentioned hardware device functions has a core computer program (software) architecture that can be described in layers and logically divided into the following: Application interface layer: Provides user interaction interfaces (such as Web UI, mobile app API, and desktop client interface) and APIs for third-party application integration, processing user requests for file upload, download, browsing, search, sharing, and permission settings.

[0045] Data Processing and Routing Layer: This core layer is responsible for: receiving application-layer data and performing data sharding (slicing files into fixed- or variable-sized data blocks); hashing each data block to generate a unique content identifier (CID); managing the mapping relationships (metadata) between files and CIDs, and between data blocks and CIDs; implementing CID-based content routing to locate network nodes holding specific data blocks; and performing data block encryption. This layer can be based on or compatible with the IPFS protocol and can integrate optimized distributed transport protocols (for example, enabling faster node discovery and better transmission path selection, referred to here as an enhanced distributed transport protocol stack) to optimize performance.

[0046] Storage and node management layer: Manages space allocation, data block reading and writing, and garbage collection of local storage units; maintains the status of devices in the distributed network (online, offline, resource status); implements data redundancy strategies (for example, copying data blocks to N other nodes in the network); manages connections and communications with other nodes; implements permission control logic and verifies the legitimacy of access requests.

[0047] Security and encryption layer: Provides underlying support for data encryption (such as using the AES-256 algorithm) to ensure the confidentiality of data during local storage and network transmission; manages encryption keys; and implements security authentication mechanisms.

[0048] The present invention provides a content-addressable hybrid file storage method based on an application interface layer, a data processing and routing layer, a storage and node management layer, and a security and encryption layer. The method includes the following processes: a data writing (storage) process, a data reading (retrieval) process, and a data sharing and permission control process.

[0049] See also Figure 1 , the data writing (storage) process includes the following steps: S1: The user uploads a file through the application interface layer; S2: The data processing and routing layer receives the file and splits it into data blocks; S3: The data processing and routing layer calculates a hash value for each data block and generates a unique data block CID. The data processing and routing layer calculates a hash value for the entire file and generates a top-level CID. In this step, the data block CID and top-level CID use the Merkle DAG structure. S4: Security and encryption layer encrypts the data blocks; S5: The storage and node management layer stores the encrypted data blocks in the local storage unit; S6: Based on the preset redundancy strategy, the storage and node management layer selects other nodes in the network and distributes copies of the data block through the enhanced distributed transmission protocol stack; S7: The storage and node management layer stores and manages metadata. In this step, the metadata includes: the source data of the file, the mapping relationship between the file and the top-level CID, the mapping relationship between the data block and the data block CID, the mapping relationship between the top-level CID and the data block CID, and permission information.

[0050] The data reading (retrieval) process includes the following steps: A1: The user requests to access the file through the application interface layer. In this step, the user uses the file name or the top-level CID of the file to request to access the file through the application interface layer.

[0051] A2: The application interface layer searches for the corresponding top-level CID. A3: The data processing and routing layer parses the top-level CID to obtain the required data block CID list; A4: The data processing and routing layer queries the node network (including local nodes) for each required data block CID through the content routing mechanism to locate the node holding the data block; A5: The storage and node management layer obtains the encrypted data block from the nearest or fastest node; in this step, the nearest or fastest node refers to: first selecting the local node, and then selecting other nodes in the P2P network.

[0052] A6: The security and encryption layer decrypts and verifies the obtained encrypted data blocks; A7: The data processing and routing layer reorganizes the data blocks into original files in sequence and returns them to the application interface layer.

[0053] The data sharing and permission control process includes the following steps: B1: The user selects the file to be shared and the target recipient to be shared. The target recipient is limited by a limiting condition, which is at least one of the unique identifier of the device, the IP address, and the recipient ID. B2: The data processing and routing layer generates a secure sharing credential associated with the top-level CID of the file; B3: The user shares this secure sharing credential with the authorized recipient; B4: The recipient initiates an access request using the secure shared credentials; B5: The data processing and routing layer verifies the validity of the secure sharing credentials and the requester's authority; B6: After verification, data access is provided according to the data reading process (usually through direct P2P transmission).

[0054] The secure sharing credential is an encrypted link or token that contains access permission information; the permission information includes: validity period, number of visits, read and write, and download permissions.

[0055] In summary, the present invention provides a content-addressable hybrid file storage method, device, and storage medium, which have the following technical effects: High reliability: Through data sharding and multi-node redundant storage: local RAID (disk array) optional + network distributed redundancy, the risk of single point failure is significantly reduced and data persistence is improved.

[0056] Enhanced security and privacy protection: End-to-end encryption, content-based addressing (not easily scanned directly), and refined permission control are combined to effectively protect user data privacy and transmission security.

[0057] Low-latency access and efficient transmission: Prioritize local access, combined with an optimized P2P direct transmission protocol, to reduce dependence on central servers and lower access latency, especially suitable for LAN or edge environments.

[0058] High scalability and flexibility: Nodes can be added on demand, and storage capacity is easily expanded. The hybrid architecture can flexibly adapt to different network environments and application requirements (home, IoT, enterprise, etc.).

[0059] Storage space optimization: The content-addressing feature naturally supports data deduplication. The same content block only needs to be stored once, saving storage space.

[0060] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hybrid file storage method based on content addressability, characterized in that: Based on application interface layer, data processing and routing layer, storage and node management layer, security and encryption layer: Includes the following processes: data writing process, data reading process, data sharing and permission control process; The data writing process includes the following steps: S1: The user uploads a file through the application interface layer; S2: The data processing and routing layer receives the file and splits it into data blocks; S3: The data processing and routing layer calculates a hash value for each data block and generates a unique data block CID. The data processing and routing layer calculates a hash value for the entire file and generates a top-level CID. S4: Security and encryption layer encrypts the data blocks; S5: The storage and node management layer stores the encrypted data blocks in the local storage unit; S6: Based on the preset redundancy strategy, the storage and node management layer selects other nodes in the network and distributes copies of the data block through the enhanced distributed transmission protocol stack; S7: The storage and node management layer stores and manages metadata. The data reading process includes the following steps: A1: The user requests to access the file through the application interface layer; A2: The application interface layer searches for the corresponding top-level CID. A3: The data processing and routing layer parses the top-level CID to obtain the required data block CID list; A4: The data processing and routing layer queries the node network for each required data block CID through the content routing mechanism to locate the node holding the data block; A5: The storage and node management layer obtains the encrypted data blocks from the nearest or fastest node; A6: The security and encryption layer decrypts and verifies the obtained encrypted data blocks; A7: The data processing and routing layer reorganizes the data blocks into original files in sequence and returns them to the application interface layer. The data sharing and permission control process includes the following steps: B1: The user selects the file to be shared; B2: The data processing and routing layer generates a secure sharing credential associated with the top-level CID of the file; B3: The user shares this secure sharing credential with the authorized recipient; B4: The recipient initiates an access request using the secure shared credentials; B5: The data processing and routing layer verifies the validity of the secure sharing credentials and the requester's authority; B6: After verification, data access is provided according to the data reading process.

2. A content-addressable hybrid file storage method according to claim 1, characterized in that: In step S3, the data block CID and the top-level CID adopt the Merkle DAG structure.

3. The hybrid file storage method based on content addressability according to claim 1, characterized in that: In step S7, the metadata includes: source data of the file, mapping relationship between the file and the top-level CID, mapping relationship between data blocks and data block CIDs, mapping relationship between the top-level CID and data block CIDs, and permission information.

4. The hybrid file storage method based on content addressability according to claim 1, characterized in that: In step A1, the user requests access to a file through the application interface layer using the file name or the top-level CID of the file.

5. The hybrid file storage method based on content addressability according to claim 1, characterized in that: In step A5, the nearest or fastest node refers to: first selecting a local node, and then selecting other nodes in the P2P network.

6. A content-addressable hybrid file storage method according to claim 1, characterized in that: In step B1, it also includes: selecting a target recipient to be shared, and the target recipient is limited by a limiting condition, and the limiting condition is at least one of the unique identifier of the device, the IP address, and the recipient ID.

7. The hybrid file storage method based on content addressability according to claim 1, characterized in that: The secure sharing credential is an encrypted link or token that contains access permission information; the permission information includes: validity period, number of visits, read and write, and download permissions.

8. A hybrid file storage device based on content addressability, characterized in that: include: Core processing unit: used to perform data sharding, hash calculation, data encryption / decryption, network protocol processing, and device management tasks; Storage unit: used to store file data blocks, metadata, cached data, and some redundant data from other nodes in the network; Network interface unit: used for the device to access the local network and exchange data with other network nodes; Power management unit: used to provide stable power to the entire device; The storage unit stores a computer program, and the processor is used to read the computer program and execute the content-addressable hybrid file storage method according to any one of claims 1 to 7.

9. The hybrid file storage device based on content addressability according to claim 8, characterized in that: Also includes: Expansion interface unit: used to connect external storage devices to expand capacity or import and export data.

10. A computer-readable storage medium, characterized in that The readable storage medium stores a computer program, which, when read and executed by one or more processors, implements the content-addressable hybrid file storage method according to any one of claims 1 to 7.

Images