Distributed data storage method, system and terminal based on open-source gap system

Abstract

The invention discloses a distributed data storage method, a distributed data storage system and a terminal based on an open-source Gawan-Mongolia-System, and the method comprises the steps: obtaining a target demand of distributed data storage, and designing a distributed data storage architecture based on the Gawan-Mongolia-system; a distributed storage module is developed based on the Gawan Mongolia system, data fragmentation, redundancy storage and node management are carried out, and a distributed consistency protocol is integrated; obtaining target data needing to be stored, encrypting the target data, and storing the encrypted target data by using a distributed data storage system; and when receiving that the user accesses the encrypted target data, carrying out identity verification and authorization check on the user, and if the user passes the verification and obtains authorization, decrypting the encrypted target data by using a corresponding secret key, and allowing the user to access. According to the distributed data storage technology based on the open-source Gaan Mongolia system, efficient, reliable and safe distributed data storage service can be provided, and the increasing data storage and processing requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of distributed data storage technology, and in particular to a distributed data storage method, system, terminal and computer-readable storage medium based on the open source Hongmeng system. Background Art

[0002] Distributed data storage is a data management solution that stores data in multiple physically independent storage nodes. These nodes are interconnected through a network to form a logical storage system. The characteristics of distributed data storage are as follows:

[0003] Storage nodes: A distributed data storage system consists of multiple storage nodes, each of which can be a physical server, a virtual machine, or an edge computing device. Each node is responsible for storing a portion of the data and can independently process read and write requests.

[0004] Data Sharding: Data sharding is to divide data into small pieces (called data slices or shards) and distribute these shards to different storage nodes. This sharding can be based on the hash value, range or other rules of the data.

[0005] Redundant storage: In order to improve the reliability and fault tolerance of the system, distributed data storage usually adopts redundant storage mechanisms, such as replication or erasure coding. The replication mechanism saves multiple copies of data on different storage nodes, while erasure coding allows the original data to be restored when part of the data is lost through encoding technology.

[0006] Consistency model: In a distributed system, the consistency model defines the degree to which data consistency is guaranteed between multiple replicas. Common consistency models include strong consistency and eventual consistency.

[0007] Distributed consistency protocol: In order to maintain data consistency in a distributed environment, distributed consistency protocols are usually adopted, such as Paxos, Raft, etc. These protocols ensure that the system can reach a consistent state even in the event of node failure or network partition.

[0008] Scalability: Distributed data storage systems can scale horizontally by adding more storage nodes to support growing data volumes and concurrent access requirements.

[0009] Fault recovery: The distributed data storage system has fault recovery capabilities. When a storage node fails, the system can automatically switch to other healthy nodes to ensure service continuity and data integrity.

[0010] Data security: Distributed data storage systems need to implement a variety of security measures, including data encryption, access control, identity authentication, etc., to protect data from unauthorized access or leakage.

[0011] Through these concepts, distributed data storage systems can provide high-availability, high-reliability, and high-scalability data storage solutions suitable for large-scale data centers and cloud computing environments.

[0012] Distributed data storage systems lack scalability. With the rapid growth of data volume, traditional distributed storage systems are often difficult to expand smoothly, resulting in performance bottlenecks and rising costs; data consistency is difficult to ensure: In a distributed environment, maintaining data consistency is a complex and critical issue, and existing solutions may perform poorly in high-concurrency scenarios; the system is highly complex, and the deployment, configuration, and management of distributed storage systems are usually complex, increasing operation and maintenance costs; security and reliability challenges, data in distributed storage faces high security threats and failure risks, requiring strong security mechanisms and fault tolerance capabilities.

[0013] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0014] The main purpose of the present invention is to provide a distributed data storage method, system, terminal and computer-readable storage medium based on the open source Hongmeng system, aiming to solve the problems in the prior art of insufficient scalability of distributed data storage, difficulty in ensuring data consistency, high system complexity, and high security threats and failure risks.

[0015] To achieve the above object, the present invention provides a distributed data storage method based on an open source Hongmeng system, and the distributed data storage method based on an open source Hongmeng system comprises the following steps:

[0016] Obtain target requirements for distributed data storage, and design a distributed data storage architecture based on the Hongmeng system according to the target requirements;

[0017] Develop a distributed storage module based on the distributed data storage architecture based on the Hongmeng system, perform data sharding, redundant storage and node management through the distributed storage module, and integrate a distributed consistency protocol into the distributed data storage architecture;

[0018] Acquire target data to be stored, determine an encryption algorithm and a key management scheme, encrypt the target data to obtain encrypted target data, and use a distributed data storage system to store the encrypted target data;

[0019] When receiving a request from a user to access the encrypted target data, the user is authenticated and authorized. If the user passes the authentication and is authorized, the encrypted target data is decrypted using the corresponding key, and the user is allowed to access.

[0020] Optionally, in the distributed data storage method based on the open source Hongmeng system, the target requirements include: performance requirements, scalability requirements, consistency requirements, availability requirements, reliability requirements, security requirements, and maintenance and operation requirements;

[0021] The performance requirements include read and write speed, response time, and throughput;

[0022] The scalability requirements include horizontal expansion and dynamic adjustment;

[0023] The consistency requirements include data consistency and consistency model;

[0024] The availability requirements include fault tolerance and disaster recovery;

[0025] The reliability requirements include data reliability;

[0026] The security requirements include data security and network security;

[0027] The maintenance and operations requirements include manageability, monitoring tools, and logging.

[0028] Optionally, in the distributed data storage method based on the open source Hongmeng system, the distributed data storage architecture includes:

[0029] Storage nodes are used to store data, perform data read and write operations, and maintain data copies;

[0030] The data sharding manager is responsible for the data sharding strategy and distributes the data to different storage nodes;

[0031] The consistency protocol layer is used to ensure the consistency of data between multiple storage nodes;

[0032] Metadata management component, used to store and manage information about data location, data sharding, and node status;

[0033] Load balancer, used to distribute client requests to different storage nodes;

[0034] Fault detection and recovery components, which monitor node health, detect faults, and trigger recovery processes;

[0035] Security modules, which implement security policies, including data encryption, access control, and authentication;

[0036] The network layer is used to provide the communication basis between nodes and support data transmission between storage nodes;

[0037] Monitoring and logging components, which are used to collect system performance indicators and log information for monitoring, troubleshooting, and performance analysis;

[0038] Configuration management components are used to manage and distribute system configurations, supporting automated deployment and configuration updates.

[0039] Optionally, in the distributed data storage method based on the open source Hongmeng system, the data sharding includes sharding strategy, sharding algorithm and sharding transparency;

[0040] The redundant storage includes replica management, replica synchronization and fault recovery;

[0041] The node management includes node discovery, load balancing and dynamic adjustment.

[0042] Optionally, the distributed data storage method based on the open source Hongmeng system, wherein the data sharding, redundant storage and node management are performed by the distributed storage module, specifically includes:

[0043] Use distributed database middleware to build sharding and redundant storage mechanisms;

[0044] Develop a custom storage engine and implement sharding logic and redundancy mechanism on Hongmeng system through the storage engine;

[0045] Use the service framework provided by Hongmeng system to communicate and synchronize data between nodes;

[0046] Deploy and manage storage nodes using containers and orchestration tools to automate node management and failure recovery.

[0047] Optionally, the distributed data storage method based on the open source Hongmeng system, wherein the distributed consistency protocol includes the Raft protocol, the Paxos protocol, and the Zab protocol;

[0048] A distributed consistency protocol is integrated into the distributed data storage architecture to ensure data consistency, handle node failures, and support system expansion.

[0049] Optionally, the distributed data storage method based on the open source Hongmeng system, wherein the distributed data storage method based on the open source Hongmeng system further includes:

[0050] Regularly monitor the operating status of the distributed data storage system, detect and handle potential problems and failures;

[0051] The potential problems and failures include network problems, data consistency problems, performance problems, security problems, hardware failures, and software failures;

[0052] The processing of potential problems and failures specifically includes:

[0053] Using monitoring tools to continuously monitor various indicators of the distributed data storage system;

[0054] Set up an alarm system to immediately notify the operation and maintenance team if an anomaly is detected;

[0055] Diagnose alerts and reported problems to determine the root cause of the problem;

[0056] Take appropriate response measures based on the nature of the problem;

[0057] Record the handling process of all problems and faults;

[0058] Develop preventive measures based on the frequency and impact of the problem.

[0059] In addition, to achieve the above-mentioned purpose, the present invention further provides a distributed data storage system based on the open source Hongmeng system, wherein the distributed data storage system based on the open source Hongmeng system includes:

[0060] A storage architecture design module is used to obtain the target requirements for distributed data storage and design a distributed data storage architecture based on the Hongmeng system according to the target requirements;

[0061] A distributed storage development module, which is used to develop a distributed storage module on the distributed data storage architecture based on the Hongmeng system, perform data sharding, redundant storage and node management through the distributed storage module, and integrate a distributed consistency protocol into the distributed data storage architecture;

[0062] A data encryption storage module is used to obtain target data to be stored, determine an encryption algorithm and a key management scheme, encrypt the target data to obtain encrypted target data, and use a distributed data storage system to store the encrypted target data;

[0063] The data access control module is used to perform identity authentication and authorization checks on the user when receiving a request to access the encrypted target data. If the user passes the authentication and obtains authorization, the encrypted target data is decrypted using the corresponding key to allow the user to access.

[0064] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a distributed data storage program based on the open source Hongmeng system stored in the memory and run on the processor, and when the distributed data storage program based on the open source Hongmeng system is executed by the processor, the steps of the distributed data storage method based on the open source Hongmeng system as described above are implemented.

[0065] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a distributed data storage program based on the open source Hongmeng system, and when the distributed data storage program based on the open source Hongmeng system is executed by the processor, the steps of the distributed data storage method based on the open source Hongmeng system as described above are implemented.

[0066] In the present invention, the target demand for distributed data storage is obtained, and a distributed data storage architecture based on the Hongmeng system is designed according to the target demand; a distributed storage module is developed on the distributed data storage architecture based on the Hongmeng system, and data sharding, redundant storage and node management are performed through the distributed storage module, and a distributed consistency protocol is integrated in the distributed data storage architecture; the target data to be stored is obtained, the encryption algorithm and key management scheme are determined, the target data is encrypted to obtain encrypted target data, and the encrypted target data is stored using a distributed data storage system; when a user accesses the encrypted target data, the user is authenticated and authorized. If the user passes the verification and obtains authorization, the encrypted target data is decrypted using the corresponding key, and the user is allowed to access. The present invention is based on the distributed data storage technology of the open source Hongmeng system, and can provide efficient, reliable and secure distributed data storage services to meet the growing needs of data storage and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flowchart of a preferred embodiment of the distributed data storage method based on the open source Hongmeng system of the present invention;

[0068] Figure 2 It is a structural diagram of a preferred embodiment of the distributed data storage system based on the open source Hongmeng system of the present invention;

[0069] Figure 3 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0071] In the existing technology, distributed data storage systems face insufficient scalability. With the rapid growth of data volume, traditional distributed storage systems are often difficult to expand smoothly, resulting in performance bottlenecks and rising costs. Smooth Scalability refers to the ability to increase or decrease system resources without interruption while maintaining system performance and availability to adapt to changing workloads and demands. In distributed data storage technology, smooth expansion usually refers to the following capabilities: Non-interruption service: During the expansion process, the system can continue to operate normally without service interruption or performance degradation due to the addition or removal of resources; Dynamic resource management: The system can dynamically add or remove storage nodes according to the current workload without shutting down or restarting the entire system; Performance consistency: After the expansion operation, the system can maintain consistent read and write performance to ensure that users do not feel performance fluctuations; Transparency: For users and applications, the expansion operation is transparent, that is, they do not need to know the changes in the underlying resources to continue to access data and enjoy services; The key features of smooth expansion include: Horizontal expansion (Scaling Out): Increase the storage capacity and processing power of the system by adding more storage nodes, rather than by improving the performance of a single node (vertical expansion); Automation: The expansion process is automated as much as possible to reduce manual intervention, reduce operational complexity and the probability of error; Load balancing: After expansion, the system can automatically redistribute data and workload to ensure that all nodes bear tasks evenly; Data migration: When adding new nodes, the system can efficiently migrate data from old nodes to new nodes while maintaining data integrity and consistency; Smooth expansion is crucial for distributed data storage systems because it allows the system to flexibly adjust resources as the amount of data grows and user demand increases, ensuring the long-term availability and economy of the system.

[0072] Distributed data storage systems face difficulties in ensuring data consistency: In a distributed environment, maintaining data consistency is a complex and critical issue, and existing solutions may not perform well in high-concurrency scenarios; Data consistency refers to the synchronization and accurate state of data on multiple replicas (or nodes) in a distributed system. In a distributed data storage system, data consistency is a key factor in ensuring system reliability. The following are several core points of data consistency: Consistency model: Distributed systems usually implement different consistency models to balance performance and consistency requirements. Common consistency models include: Strong consistency: Any read and write operations are immediately reflected on all replicas of the system. Eventual consistency: The system guarantees that all replicas will eventually reach a consistent state after a period of time without subsequent updates. Weak consistency: The system does not guarantee that the data of all replicas is up to date and may return outdated data. Replication and data synchronization: In a distributed system, data is usually replicated to multiple nodes to improve availability and reliability. Data consistency involves the data synchronization mechanism between these replicas. Concurrency control: When multiple clients read and write data at the same time, the system needs to take measures to ensure data consistency, such as using locks, transactions, timestamps, and other mechanisms. Fault recovery: In the event of node failure or network partition, the system needs to be able to recover to a consistent state, which may involve complex fault recovery and replica repair processes. The importance of data consistency is reflected in the following aspects: Reliability: Users expect to get correct and up-to-date information whenever they read data. Correctness: Applications rely on consistent data to make correct business decisions. User experience: Inconsistent data may lead to a poor user experience. For example, in an e-commerce system, users may see outdated inventory information. When achieving data consistency, distributed data storage systems need to overcome the following challenges: Network latency and partitioning: Network problems may cause communication delays or complete interruption between nodes. Concurrent operations: Multiple clients operating on the same data at the same time may cause data inconsistency. System load: In high-concurrency scenarios, maintaining data consistency may increase the burden on the system and affect performance. Therefore, when designing a distributed data storage system, it is necessary to consider how to achieve and maintain data consistency while also considering the system's performance, scalability, and fault tolerance.

[0073] Distributed data storage systems face high system complexity. The deployment, configuration and management of distributed storage systems are usually complex, which increases the operation and maintenance costs. The deployment, configuration and management of distributed storage systems are usually complex. The main reasons include the following: Complexity of system architecture: Distributed storage systems are composed of multiple nodes, which may be distributed in different physical locations and need to communicate through the network. The system needs to design a complex network topology, such as clusters and interconnections between data centers, to ensure high availability and fault tolerance of data. Data sharding and redundancy: Data needs to be sharded and redundantly stored between multiple nodes, which involves complex sharding strategies and redundant mechanism design. Ensuring data consistency and fault recovery requires the implementation of complex algorithms and protocols, such as consistent hashing, Paxos, Raft, etc. Node management and coordination: The system needs to manage the addition, removal and fault handling of nodes, which usually requires automated node management tools and coordination mechanisms. Coordination between nodes involves state synchronization, task allocation, load balancing, etc., which all require careful design and maintenance. Network communication issues: Distributed systems rely on stable network communication, but network delays, packet loss, partitions and other issues will increase the difficulty of deployment and management. A network failure detection and recovery mechanism needs to be implemented to ensure the stable operation of the system. Security considerations: Distributed storage systems need to consider data security and privacy protection, which involves complex encryption, authentication, and access control mechanisms. Security configuration and management require expertise to ensure that data is not accessed or leaked without authorization. Monitoring and maintenance: The system requires continuous monitoring and maintenance to detect performance bottlenecks, resource utilization, potential security threats, etc. Regular system upgrades and backups are also a complex task that requires ensuring data integrity and service availability during operations. Compatibility and scalability: The system needs to be designed to be compatible with different hardware and software environments while supporting future expansion. As business needs grow, the system may need to be constantly adjusted and optimized to adapt to new challenges. Due to the above reasons, the deployment, configuration, and management of distributed storage systems are much more complex than traditional monolithic storage systems, requiring expertise and experience to ensure system stability and performance.

[0074] Distributed data storage systems face security and reliability challenges. Data in distributed storage faces high security threats and failure risks, requiring strong security mechanisms and fault tolerance. In distributed data storage systems, security threats and failure risks mainly include the following aspects: Data leakage and unauthorized access: Network attacks: Distributed systems may be subject to network attacks, such as DDoS attacks, man-in-the-middle attacks, etc. These attacks may lead to data leakage or service interruption. Unauthorized access: If there is no effective access control mechanism, attackers may illegally access and steal stored data. Data tampering and integrity damage: Data tampering: During data transmission or storage, if there are no appropriate protection measures, data may be tampered with. Integrity damage: System failure or malicious attack may cause data integrity to be damaged. Denial of Service (DoS) attack: Denial of Service: Attackers may launch a denial of service attack, making the system unable to respond normally to requests from legitimate users. Node failure and data loss: Hardware failure: Hardware failure of storage nodes may lead to data loss or service interruption. Software error: Errors in system software may cause node failure or data corruption. Network partition and communication failure: Network partition: Network partition may cause some nodes in the distributed system to be unable to communicate with other nodes, affecting data consistency and system availability. Communication failure: Network communication problems may cause data synchronization failure or problems in coordination between nodes. Distributed Denial of Service (DDoS) Attack: Distributed Denial of Service: Attackers use multiple damaged nodes to launch attacks, paralyzing the entire system. Malware and Viruses: Malware: Systems may be infected with malware that can destroy data or steal sensitive information. Viruses: Viruses may spread in distributed systems and affect multiple nodes. Configuration Errors and Operational Errors: Configuration Errors: Improper system configuration may lead to security vulnerabilities or service interruptions. Operational Errors: Incorrect operations by operation and maintenance personnel may cause system instability or data loss. In order to cope with these security threats and failure risks, distributed data storage systems need to implement a series of security measures, including but not limited to data encryption, access control, identity authentication, audit logs, fault detection and recovery mechanisms, redundant storage, regular backups, and network security protection.

[0075] The present invention relates to the field of distributed data storage technology, and in particular to a distributed data storage solution combined with the open source Hongmeng system. Future application directions include but are not limited to cloud computing, big data processing, the Internet of Things (IoT), and edge computing, and aim to provide efficient, reliable, and scalable distributed data storage services.

[0076] The distributed data storage method based on the open source Hongmeng system described in the preferred embodiment of the present invention is as follows Figure 1 As shown, the distributed data storage method based on the open source Hongmeng system includes the following steps:

[0077] Step S10: Obtain the target requirements for distributed data storage, and design a distributed data storage architecture based on the Hongmeng system according to the target requirements.

[0078] Specifically, the target requirements include: performance requirements, scalability requirements, consistency requirements, availability requirements, reliability requirements, security requirements, and maintenance and operation requirements; the performance requirements include read and write speeds, response time, and throughput; the scalability requirements include horizontal expansion and dynamic adjustment; the consistency requirements include data consistency and consistency model; the availability requirements include fault tolerance and disaster recovery; the reliability requirements include data reliability; the security requirements include data security and network security; the maintenance and operation requirements include manageability, monitoring tools, and logging.

[0079] (1) Performance requirements:

[0080] Read and write speed: The system needs to achieve a certain data read and write speed to meet the demand for fast data access.

[0081] Response time: The system should have a low latency response time to ensure user experience.

[0082] Throughput: The system should be able to handle a large number of concurrent read and write requests to ensure high throughput.

[0083] (2) Scalability requirements:

[0084] Horizontal expansion: The system should support increasing storage capacity and processing power by adding more nodes without interrupting service.

[0085] Dynamic adjustment: The system should be able to dynamically adjust resources based on workload to optimize performance and cost.

[0086] (3) Consistency requirements:

[0087] Data consistency: The system needs to ensure that data on all replicas remains consistent, especially in the case of concurrent access.

[0088] Consistency model: Select an appropriate consistency model (such as strong consistency, eventual consistency, etc.) based on application requirements.

[0089] (4) Availability requirements:

[0090] Fault tolerance: The system should be able to continue to provide services even if some nodes fail.

[0091] Disaster recovery: The system should be able to recover from catastrophic failures.

[0092] (5) Reliability requirements:

[0093] Data reliability: The system needs to ensure data persistence and reliability to prevent data loss.

[0094] (6) Security requirements:

[0095] Data security: The system should have security measures such as data encryption and access control.

[0096] Network security: The system should be able to resist network attacks and protect data from illegal access.

[0097] (7) Maintenance and operation requirements:

[0098] Manageability: The system should be easy to manage and maintain to reduce operation and maintenance costs.

[0099] Monitoring and logging: The system should provide effective monitoring tools and logging to facilitate problem diagnosis and performance analysis.

[0100] These indicators come from the following sources:

[0101] Business requirements: The requirements of specific application scenarios for data storage systems, such as e-commerce platforms requiring fast data reading and writing capabilities, and financial systems requiring strong data consistency and security.

[0102] Technical standards: technical standards and best practices within the industry, such as the CAP theorem (consistency, availability, partition tolerance) for distributed systems.

[0103] User expectations: User expectations of system performance, stability, and security.

[0104] Laws and regulations: Data protection regulations and requirements for data processing and storage.

[0105] During the system requirements analysis and design phase, these requirements will be listed and analyzed in detail to ensure that the designed system architecture can meet the predetermined performance, scalability and consistency goals.

[0106] A distributed data storage architecture based on the Hongmeng system is designed to clarify the functions and interaction methods of each component. The distributed data storage architecture includes: storage nodes, which are used to store data, perform data read and write operations, and maintain data copies; a data sharding manager, which is responsible for the data sharding strategy and distributes data to different storage nodes; a consistency protocol layer, which is used to ensure the consistency of data between multiple storage nodes; a metadata management component, which is used to store and manage information about data location, data sharding, and node status; a load balancer, which is used to distribute client requests to different storage nodes; a fault detection and recovery component, which is used to monitor the health status of nodes, detect faults, and trigger the recovery process; a security module, which is used to implement security policies, including data encryption, access control, and authentication; a network layer, which is used to provide a communication basis between nodes and support data transmission between storage nodes; a monitoring and logging component, which is used to collect system performance indicators and log information for monitoring, troubleshooting, and performance analysis; a configuration management component, which is used to manage and distribute system configurations, and supports automated deployment and configuration updates.

[0107] Specifically, the components are described as follows:

[0108] (1) Storage Nodes:

[0109] Function: Responsible for storing data, performing data read and write operations, maintaining data copies, and ensuring data persistence.

[0110] Description: Storage nodes can be physical servers or virtual machines, which are connected through a network to form a storage cluster.

[0111] (2) Sharding Manager:

[0112] Function: Responsible for the data sharding strategy, distributing data to different storage nodes to achieve load balancing and horizontal expansion.

[0113] Description: Manages how data is divided into multiple fragments and how they are mapped to specific storage nodes.

[0114] (3) Consistency Protocol Layer:

[0115] Function: Implement distributed consistency protocols, such as Raft or Paxos, to ensure data consistency among multiple storage nodes.

[0116] Description: Handles communication between nodes, coordinates update operations, and ensures the eventual consistency of data on different nodes.

[0117] (4) Metadata Management:

[0118] Function: Store and manage metadata about data location, data sharding information, node status, etc.

[0119] Description: The metadata management component is the "brain" of the system; it knows where the data is stored and coordinates various operations.

[0120] (5) Load Balancer:

[0121] Function: Distribute client requests to different storage nodes to avoid overloading a single node.

[0122] Description: Intelligently routes requests to the most appropriate storage node based on node load and performance indicators.

[0123] (6) Fault Detection and Recovery:

[0124] Function: Monitor node health, detect failures, and trigger recovery processes.

[0125] Description: This component is responsible for the fault tolerance of the system, ensuring that the system can automatically recover when a node fails.

[0126] (7) Security Module:

[0127] Function: Implement security policies, including data encryption, access control, identity authentication, etc.

[0128] Description: Protect data from unauthorized access and cyber attacks, ensuring data security and privacy.

[0129] (8) Network Layer:

[0130] Function: Provide communication basis between nodes to ensure the reliability and efficiency of data transmission.

[0131] Description: The network layer can be a physical network or a virtual network, which supports data transmission between storage nodes.

[0132] (9) Monitoring and Logging:

[0133] Function: Collect system performance indicators and log information for monitoring, troubleshooting and performance analysis.

[0134] Description: This component helps operation and maintenance personnel understand the operating status of the system and discover and solve problems in a timely manner.

[0135] (9) Configuration Management:

[0136] Function: Manage and distribute system configurations, support automated deployment and configuration updates.

[0137] Description: Simplify the system configuration process and ensure the consistency and correctness of the configuration.

[0138] Together, these components form a distributed data storage architecture, with each component playing a specific role to ensure the performance, scalability, consistency, and reliability of the entire system.

[0139] Step S20: Develop a distributed storage module on the distributed data storage architecture based on the Hongmeng system, perform data sharding, redundant storage and node management through the distributed storage module, and integrate a distributed consistency protocol into the distributed data storage architecture.

[0140] Specifically, the data sharding includes sharding strategy, sharding algorithm and sharding transparency; the redundant storage includes replica management, replica synchronization and fault recovery; the node management includes node discovery, load balancing and dynamic adjustment. The distributed consistency protocol includes Raft protocol, Paxos protocol and Zab protocol; the distributed consistency protocol is integrated in the distributed data storage architecture to ensure data consistency, handle node failures and support system expansion.

[0141] The distributed storage module is a key part of the distributed data storage architecture. This module is usually responsible for implementing the core functions of distributed data storage, including the following aspects:

[0142] Data Sharding: Divide large amounts of data into smaller pieces to facilitate management and storage while improving access efficiency.

[0143] Redundant Storage: Improve data reliability and system fault tolerance by storing copies of data on multiple storage nodes.

[0144] Node Management: Manage the addition, removal, and status monitoring of storage nodes to ensure the stable operation of the entire storage cluster.

[0145] As part of the architecture, the distributed storage module works together with other components to form a complete distributed data storage system. Its main purpose is to achieve distributed storage and processing of data to ensure high performance, high availability and high reliability of the system. Developing a distributed storage module is a key step in building the entire distributed data storage system, which directly affects the performance and functionality of the system.

[0146] Implementing data sharding, redundant storage, and node management usually involves the following technologies and methods:

[0147] (1) Data Sharding:

[0148] Sharding strategy: defines the method of data sharding, such as hash-based sharding, range sharding, or consistent hash sharding. These strategies determine how data is distributed to different storage nodes.

[0149] Sharding algorithm: Develop or integrate an algorithm to determine the shard to which data belongs when it is written or read, and route it to the corresponding storage node.

[0150] Sharding transparency: ensures that upper-layer applications do not need to worry about the details of data sharding when writing or reading data, which is automatically handled by the storage system.

[0151] (2) Redundant Storage:

[0152] Replica management: Create multiple copies of each data shard and store these copies on different storage nodes to achieve redundancy.

[0153] Replica synchronization: Implement mechanisms to ensure that all replicas are consistent at all times, usually through a distributed consensus protocol such as Raft or Paxos.

[0154] Fault recovery: When a storage node fails, the system can automatically switch to other copies to ensure service continuity and data integrity.

[0155] (3) Node Management:

[0156] Node discovery: Implement an automatic node discovery mechanism so that newly added storage nodes can be identified by the system and added to the cluster.

[0157] Load balancing: Read and write requests are distributed through a load balancing algorithm to ensure that all storage nodes are loaded evenly and avoid overloading a single node.

[0158] Node monitoring: Real-time monitoring of storage node status, including CPU, memory, storage space, and network status, to facilitate timely discovery and resolution of problems.

[0159] Dynamic adjustment: Dynamically add or remove storage nodes based on system load and performance requirements to achieve horizontal expansion.

[0160] The following are the specific technical implementation methods:

[0161] Use distributed database middleware: You can build sharding and redundant storage mechanisms based on existing distributed database middleware (such as Apache Cassandra, MongoDB Sharding, etc.).

[0162] Custom storage engine: Develop a custom storage engine to implement sharding logic and redundancy mechanism directly on the Hongmeng system.

[0163] Utilize the service framework of the Hongmeng system: Utilize the service framework provided by the Hongmeng system, such as the Distributed Service Framework (DSF), to achieve communication and data synchronization between nodes.

[0164] Containerization and orchestration technology: Use containers (such as Docker) and orchestration tools (such as Kubernetes) to deploy and manage storage nodes to achieve automated node management and fault recovery.

[0165] Through the above method, the distributed storage module can achieve efficient data sharding, redundant storage and node management on the open source Hongmeng system, providing stable and reliable storage services for upper-level applications.

[0166] Integrate distributed consistency protocols to ensure data consistency across multiple nodes. Distributed consistency protocols are a key concept in distributed systems. They define a series of rules and algorithms to ensure that data on multiple nodes in a distributed environment remain consistent. In distributed data storage systems, since data may be distributed in different physical locations, these protocols become particularly important to ensure that all nodes have a consistent view of the data and maintain data consistency even in the face of network partitions, node failures, or other system anomalies.

[0167] Here are some common distributed consensus protocols:

[0168] (1) Raft Protocol: Raft is a consensus algorithm for managing replicated logs. It simplifies the replication of state machines to make the algorithm easier to understand. It ensures consistency through leader election, log replication, and security, and can maintain system availability in the face of network partitions and node failures.

[0169] (2) Paxos protocol: Paxos is an algorithm for solving the consistency problem in distributed systems. It can ensure consistency in the case of network delays and message loss. It reaches consensus through proposal numbers and majority voting to ensure that all nodes eventually reach a consensus on a certain value.

[0170] (3) Zab Protocol: Zab is the abbreviation of ZooKeeper Atomic Broadcast Protocol, which provides a simple broadcast algorithm for achieving consistency in distributed systems. It ensures transactional nature in distributed systems and maintains consistency even in the event of network partitions.

[0171] The purpose of integrating a distributed consensus protocol is to:

[0172] (1) Ensure data consistency: When multiple clients read and write data concurrently, the protocol ensures that all nodes can eventually see the same view of the data.

[0173] (2) Handling node failures: When a node fails for various reasons, the protocol can coordinate the remaining nodes to maintain the overall consistency of the system.

[0174] (3) Support system expansion: The protocol allows the system to add or remove nodes without interrupting service, thereby supporting horizontal expansion of the system.

[0175] Distributed consistency protocols are the key to achieving reliability and availability of distributed data storage systems. They use a series of mechanisms to ensure that even when some nodes fail or there are network communication problems, the entire system can still correctly process read and write requests and maintain data consistency.

[0176] Step S30: Acquire target data to be stored, determine an encryption algorithm and a key management scheme, encrypt the target data to obtain encrypted target data, and use a distributed data storage system to store the encrypted target data.

[0177] Step S40: When receiving a request from a user to access the encrypted target data, the user is authenticated and authorized. If the user passes the authentication and obtains authorization, the encrypted target data is decrypted using the corresponding key, and the user is allowed to access.

[0178] Specifically, implement security measures such as data encryption and access control to protect the security of stored data. Design and implement data redundancy and fault recovery mechanisms to improve system reliability; implementing data encryption and access control is an important measure to ensure the security of distributed data storage systems. The following are specific implementation methods of these measures:

[0179] (1) Data encryption:

[0180] Encryption policy definition: Determine the encryption algorithm (such as AES, RSA, etc.) and key management scheme. Define when and where encryption is applied, such as before data is written to the storage node or when data is transmitted between nodes. Encryption implementation: Encryption when data is written: Encrypt the data using an encryption algorithm before it is written to the storage system. This ensures that the data stored on the node always exists in encrypted form. Key management: Securely generate, store, distribute, and rotate encryption keys. Hardware security modules (HSM) or key management services can be used to enhance the security of keys. Decryption process: When a legitimate user or application needs to read the data, the system will use the corresponding key to decrypt the encrypted data.

[0181] (2) Access control:

[0182] Authentication: Ensure that only authenticated users and applications can access the system. Implement user authentication mechanisms such as username / password, digital certificates, tokens, or biometrics. Authorization policies: Define the access rights of users and applications, determine what data they can access and what operations they can perform. Implement policies such as role-based access control (RBAC) or attribute-based access control (ABAC). Storage access control: Implement access control at the data storage layer to ensure that only users with appropriate permissions can read or write specific data. Transmission access control: Implement access control during data transmission, such as through secure channels such as VPN, SSL / TLS. Combining data encryption and access control: Combination of encryption and access control: Based on data encryption, access control ensures that only authorized users can access even if the data is decrypted. Access to stored data: For stored encrypted data, when a user requests access, the system first performs authentication and authorization checks. If the user is authenticated and authorized, the system will decrypt the data using the appropriate key and then allow the user access.

[0183] In this way, data encryption and access control work together to provide strong security protection for distributed data storage systems, ensuring the confidentiality, integrity and availability of data.

[0184] In a distributed data storage system, data redundancy and fault recovery mechanisms are key components to ensure system reliability and data persistence. The following are specific explanations and implementation methods of these concepts:

[0185] Data redundancy: Data redundancy refers to storing multiple copies of data on different storage nodes, so that even if a node fails, the data can still be recovered from other copies. The following are some common ways to implement data redundancy:

[0186] Replica: Copies of data are stored on multiple storage nodes, and a certain strategy (such as consistent hashing) is usually used to determine where to place the data copies.

[0187] Sharding: Split the data into multiple fragments (shards), each shard has multiple copies, distributed and stored on different nodes in the cluster.

[0188] Erasure coding: Using erasure coding technology, such as Reed-Solomon coding, data is divided into fragments and additional verification fragments are generated. Even if some data fragments are lost, the original data can be restored through the verification fragments.

[0189] Fault recovery mechanism: Fault recovery mechanism refers to the ability of the system to automatically take measures to restore service when a node or component in the system fails. The following are several implementation methods of fault recovery mechanism:

[0190] Fault detection: Real-time monitoring of node status, including heartbeat detection, system resource monitoring, etc., to quickly detect node failures.

[0191] Failover: When a node failure is detected, the system automatically transfers the tasks and load of the failed node to other healthy nodes.

[0192] Data recovery: Use redundant data copies or erasure coding technology to recover data on failed nodes.

[0193] Automatic rebalancing: After a node is restored, the system automatically rebalances data distribution to ensure that all nodes are load balanced.

[0194] Use data redundancy and failure recovery mechanisms:

[0195] Implement redundancy strategy: During the system design phase, determine the data redundancy strategy and implement corresponding technologies, such as setting the number of data copies and configuring erasure code parameters.

[0196] Automated failure handling: Develop or integrate automated scripts and tools to automate failover and data recovery processes when a failure is detected.

[0197] Monitoring and alerting: Implement a monitoring system to issue alerts in time when failures occur and trigger failure recovery processes.

[0198] Regular testing: Perform fault simulation tests regularly to ensure that the fault recovery mechanism works effectively when a fault actually occurs.

[0199] Through these mechanisms, distributed data storage systems can quickly recover and continue to provide stable and reliable services when encountering node failures or other system anomalies, thereby ensuring data persistence and system availability.

[0200] Furthermore, the distributed data storage system is subjected to comprehensive functional testing, performance testing, and stress testing to ensure the stability and performance of the system. The distributed data storage system is actually gradually built and improved throughout the implementation process. The following is the formation process of the distributed data storage system:

[0201] Step 1: System requirements analysis and design:

[0202] In this step, we do not directly build a distributed data storage system, but conduct preliminary demand analysis and system design. This includes determining the key indicators that the system needs to meet, such as performance, scalability, consistency, and designing the system architecture and component interaction methods. This stage is a theoretical planning and design that provides guidance for subsequent actual development work.

[0203] Step 2: Develop distributed storage module:

[0204] In this step, actual code development and module construction begin. The development of distributed storage modules is the core part of building a distributed data storage system. Key technologies such as data sharding, redundant storage, node management, and distributed consistency protocols will be implemented here, and distributed consistency protocols will be integrated. After completing this step, the basic distributed data storage system has been formed, but it may not be perfect and robust enough.

[0205] Step 3: Enhance security and reliability:

[0206] After having a basic distributed data storage system, the next step is to enhance the security and reliability of the system. In this step, by implementing measures such as data encryption, access control, data redundancy and fault recovery mechanisms, the distributed data storage system is made more secure and reliable. These security and reliability enhancement measures are an indispensable part of the distributed data storage system.

[0207] Step 4: System testing and optimization:

[0208] At this step, the distributed data storage system has been basically built, but it still needs to be fully tested to verify its performance and stability. Some problems may be found during the test, and the system needs to be optimized based on the test results. This step is a key link to ensure the quality of the system. Through testing and optimization, the distributed data storage system is further improved.

[0209] Therefore, it can be said that the distributed data storage system is built in step 2 and is further improved and optimized in steps 3 and 4. Each step is part of building a distributed data storage system until an efficient, reliable and secure distributed data storage service is finally formed. In step 1, although there is design and planning, there is no actual physical system; only in step 2 and subsequent steps, with the deepening of development and the increase of functions, the distributed data storage system gradually takes shape.

[0210] According to the test results, the system is optimized, the configuration parameters and algorithms are adjusted, and the overall system performance is improved. Functional testing, performance testing, and stress testing are key links to ensure that the distributed data storage system meets the design requirements and performance indicators. The following are the specific implementation methods of these tests:

[0211] (1) Functional test:

[0212] Test plan development: Determine the test objectives and test scope. Design test cases to cover all expected functions, including data reading and writing, sharding, redundancy, fault recovery, etc. Test execution: Perform tests according to the test cases, simulating normal operation processes and abnormal situations. Verify whether the system works as expected, including user interface, API, internal logic, etc. Result verification: Check whether the test results are consistent with the expected results. Record and analyze any functional anomalies or failures.

[0213] (2) Performance test:

[0214] Benchmark testing: measure basic performance indicators, such as read / write speed, response time, etc., without system load. Load testing: gradually increase system load to simulate multiple users accessing and manipulating data at the same time. Measure the performance of the system under different loads. Performance indicator collection: collect performance data, including throughput, response time, number of concurrent users, etc. Analyze system resource usage, such as CPU, memory, network I / O, etc.

[0215] (3) Stress test:

[0216] Extreme load testing: Apply a load beyond the normal operating range to the system until the system reaches the point of collapse. Observe the system's behavior and recovery capabilities under extreme conditions.

[0217] (4) Stability test:

[0218] Run the system for a long time under high load to check whether the system runs stably and has no problems such as memory leaks.

[0219] (5) Fault recovery test:

[0220] Intentionally introduce faults during stress testing, such as node failure, network delay, etc. Verify whether the system's fault recovery mechanism is effective, including automatic switching, data recovery, etc.

[0221] Test methods and tools:

[0222] Automated testing tools: Use automated testing tools to perform repetitive testing tasks, such as JMeter, Gatling, etc. Monitoring tools: Use monitoring tools to collect system performance data, such as Prometheus, Grafana, etc. Log analysis: Analyze system logs to identify potential performance bottlenecks or errors.

[0223] Through these tests, the functions, performance and stability of the distributed data storage system can be fully evaluated to ensure that it can meet the predetermined business requirements and technical indicators after actual deployment. The test results will provide a basis for the optimization and improvement of the system.

[0224] Optimizing the system based on test results is an iterative process that involves analyzing test data, identifying performance bottlenecks or functional defects, and taking appropriate measures to improve the system. The following are the specific optimization steps:

[0225] (1) Analyze the test results:

[0226] Collect test data: Collect detailed data of functional testing, performance testing and stress testing, including response time, throughput, error rate, resource utilization (CPU, memory, disk I / O, network I / O), etc.

[0227] Identify problems: Analyze test data to identify system performance bottlenecks, functional defects, or security risks. Determine which problems are caused by improper configuration, inefficient code, unreasonable resource allocation, or inadequate system architecture design.

[0228] (2) System optimization measures:

[0229] Configuration optimization: Adjust system configuration parameters, such as cache size, thread pool size, load balancing strategy, etc., to optimize resource utilization and system performance. Optimize network configuration to reduce latency and packet loss. Code optimization: Refactor inefficient code segments found in performance testing to improve algorithm efficiency. Optimize data structures and database queries to reduce unnecessary calculations and I / O operations. Resource allocation: Adjust resource allocation based on test results, such as increasing or decreasing node resources to balance the load. Use more efficient data storage formats or compression algorithms to reduce storage and transmission overhead. Architecture adjustment: If the test shows that the system architecture has design flaws, consider redesigning some components or introducing new architecture patterns. For example, introduce a more efficient data sharding strategy or improve the implementation of distributed consistency protocols. Security reinforcement: According to the results of security tests, strengthen the system's security protection measures, such as enhancing encryption algorithms, improving access control logic, etc. Fault recovery mechanism: According to the results of stress testing and fault recovery testing, optimize the fault detection and recovery process to ensure the high availability of the system.

[0230] (3) Iterative testing and optimization:

[0231] Retest: After implementing optimization measures, re-perform functional testing, performance testing, and stress testing to verify the optimization effect. Ensure that the optimization measures have not introduced new problems. Continuous monitoring: After the system is deployed, continuously monitor the performance and stability of the system to identify and solve problems in a timely manner. Documentation: Record all optimization processes and results, including changes made, reasons, and effects, for future review and continuous improvement.

[0232] Through these steps, we can ensure that the distributed data storage system can better meet the design goals and business needs after optimization, and provide efficient, reliable and secure storage services. Deploy the distributed data storage system in the actual environment and provide necessary operation and maintenance support and training. Regularly monitor the operating status of the distributed data storage system, handle potential problems and failures in a timely manner, and ensure the continuous and stable operation of the system.

[0233] The following are potential problems and types of failures you may encounter, and how to deal with them:

[0234] Potential problems and failure types:

[0235] Hardware failure: storage node hard disk damage. Network equipment failure, such as switch or router problems. Software failure: system software crashes or becomes unresponsive. Application errors, resulting in data corruption or service interruption. Network problems: network delays or packet loss. Inter-node communication interruption. Data consistency issues: inconsistencies between data copies. Update operations are not correctly synchronized to all nodes. Performance issues: system response time is too long. Throughput is lower than expected. Security issues: unauthorized access attempts. Data leakage or encryption measures are cracked.

[0236] Treatment method:

[0237] Hardware fault handling: Use redundant hardware and failover mechanisms to automatically switch to backup devices when hardware failures are detected. Perform hardware maintenance and replace old equipment regularly. Software fault handling: Implement monitoring of applications and system software, and immediately restart services or repair them once an abnormality is found. Regularly update and patch to fix known software defects. Network problem handling: Monitor network performance and use network diagnostic tools to locate problems. Configure network redundant paths to deal with single point failures. Data consistency problem handling: Use distributed consistency protocols and algorithms to ensure data consistency, such as Paxos or Raft. Regularly perform data verification and repair inconsistent data copies. Performance problem handling: Analyze performance bottlenecks and optimize resource allocation and load balancing strategies. Adjust system configuration based on performance test results, such as increasing node resources or optimizing caching strategies. Security problem handling: Implement strict access control and authentication mechanisms. Perform security audits and vulnerability scans regularly and repair security vulnerabilities in a timely manner.

[0238] Specific steps:

[0239] (1) Monitoring: Use monitoring tools to continuously monitor various system indicators, including hardware status, software logs, network traffic, data consistency, etc.

[0240] (2) Alarm: Set up an alarm system to immediately notify the operation and maintenance team once an abnormality is detected.

[0241] (3) Diagnosis: Diagnose alarms and reported problems to determine the root cause of the problem.

[0242] (4) Response: Take appropriate response measures based on the nature of the problem, such as restarting services, replacing hardware, applying patches, etc.

[0243] (5) Records: Record the handling process of all problems and faults, including causes, handling methods and results, to facilitate future analysis and improvement.

[0244] (6) Prevention: Develop preventive measures based on the frequency and impact of the problem to reduce the probability of similar problems occurring in the future.

[0245] Through these methods, it can be ensured that the distributed data storage system can respond and recover quickly when encountering potential problems and failures, ensuring the continuous and stable operation of the system.

[0246] Through the above steps, the distributed data storage technology based on the open source Hongmeng system will be able to provide efficient, reliable and secure distributed data storage services to meet the growing data storage and processing needs.

[0247] Beneficial effects of the present invention:

[0248] (1) High scalability: Through the distributed architecture of the Hongmeng system, storage nodes can be dynamically added and removed, supporting horizontal expansion to cope with the growing demand for data storage. The dynamic expansion capability enables the system to flexibly respond to storage needs of different scales.

[0249] (2) Strong consistency guarantee: A distributed consistency protocol is introduced to ensure high consistency of data among multiple storage nodes. This ensures data accuracy even in high-concurrency scenarios. The strong consistency protocol ensures accurate synchronization of data in a distributed environment.

[0250] (3) Simplify system deployment and management: Utilize the automated configuration and management functions of the Hongmeng system to simplify the deployment and operation and maintenance process of the distributed storage system, reduce management complexity, and simplify the deployment and management process to reduce manpower and resource investment.

[0251] (4) Enhanced security and reliability: The security mechanisms of the Hongmeng system, such as data encryption and access control, are used to improve the security of data storage. At the same time, data redundancy and fault recovery strategies are adopted to enhance the reliability of the system. Multi-level security protection and fault-tolerant mechanisms protect the integrity and availability of data.

[0252] Furthermore, if Figure 2 As shown, based on the above-mentioned distributed data storage method based on the open source Hongmeng system, the present invention also provides a distributed data storage system based on the open source Hongmeng system, wherein the distributed data storage system based on the open source Hongmeng system includes:

[0253] A storage architecture design module 51 is used to obtain target requirements for distributed data storage and design a distributed data storage architecture based on the Hongmeng system according to the target requirements;

[0254] A distributed storage development module 52 is used to develop a distributed storage module on the distributed data storage architecture based on the Hongmeng system, perform data sharding, redundant storage and node management through the distributed storage module, and integrate a distributed consistency protocol into the distributed data storage architecture;

[0255] The data encryption storage module 53 is used to obtain target data to be stored, determine an encryption algorithm and a key management scheme, encrypt the target data to obtain encrypted target data, and use a distributed data storage system to store the encrypted target data;

[0256] The data access control module 54 is used to perform identity authentication and authorization check on the user when receiving a request to access the encrypted target data. If the user passes the authentication and obtains authorization, the encrypted target data is decrypted using the corresponding key to allow the user to access.

[0257] Furthermore, if Figure 3 As shown, based on the above-mentioned distributed data storage method and system based on the open source Hongmeng system, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 3 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0258] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a distributed data storage program 40 based on an open source Hongmeng system is stored on the memory 20, and the distributed data storage program 40 based on the open source Hongmeng system can be executed by the processor 10, thereby realizing the distributed data storage method based on the open source Hongmeng system in this application.

[0259] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code stored in the memory 20 or process data, such as executing the distributed data storage method based on the open source Hongmeng system.

[0260] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0261] In one embodiment, when the processor 10 executes the distributed data storage program 40 based on the open source Hongmeng system in the memory 20, the steps of the distributed data storage method based on the open source Hongmeng system as described above are implemented.

[0262] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a distributed data storage program based on the open source Hongmeng system, and when the distributed data storage program based on the open source Hongmeng system is executed by a processor, the steps of the distributed data storage method based on the open source Hongmeng system as described above are implemented.

[0263] In summary, the present invention provides a distributed data storage method, system, terminal and computer-readable storage medium based on the open source Hongmeng system, the method comprising: obtaining the target requirements of distributed data storage, designing a distributed data storage architecture based on the Hongmeng system according to the target requirements; developing a distributed storage module based on the Hongmeng system on the distributed data storage architecture, performing data sharding, redundant storage and node management through the distributed storage module, and integrating a distributed consistency protocol in the distributed data storage architecture; obtaining the target data to be stored, determining the encryption algorithm and key management scheme, encrypting the target data to obtain encrypted target data, and using a distributed data storage system to store the encrypted target data; when receiving a user access to the encrypted target data, performing identity authentication and authorization checks on the user, if the user passes the verification and obtains authorization, using the corresponding key to decrypt the encrypted target data, allowing the user to access. The present invention is based on the distributed data storage technology of the open source Hongmeng system, and can provide efficient, reliable and secure distributed data storage services to meet the growing needs of data storage and processing.

[0264] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0265] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0266] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A distributed data storage method based on the open source Hongmeng system, characterized in that: The distributed data storage method based on the open source Hongmeng system includes: Obtain target requirements for distributed data storage, and design a distributed data storage architecture based on the Hongmeng system according to the target requirements; Develop a distributed storage module based on the distributed data storage architecture based on the Hongmeng system, perform data sharding, redundant storage and node management through the distributed storage module, and integrate a distributed consistency protocol into the distributed data storage architecture; Acquire target data to be stored, determine an encryption algorithm and a key management scheme, encrypt the target data to obtain encrypted target data, and use a distributed data storage system to store the encrypted target data; When receiving a request from a user to access the encrypted target data, the user is authenticated and authorized. If the user passes the authentication and is authorized, the encrypted target data is decrypted using the corresponding key, and the user is allowed to access.

2. According to the distributed data storage method based on the open source Hongmeng system according to claim 1, it is characterized in that: The target requirements include: performance requirements, scalability requirements, consistency requirements, availability requirements, reliability requirements, security requirements, and maintenance and operation requirements; The performance requirements include read and write speed, response time, and throughput; The scalability requirements include horizontal expansion and dynamic adjustment; The consistency requirements include data consistency and consistency model; The availability requirements include fault tolerance and disaster recovery; The reliability requirements include data reliability; The security requirements include data security and network security; The maintenance and operations requirements include manageability, monitoring tools, and logging.

3. According to the distributed data storage method based on the open source Hongmeng system according to claim 1, it is characterized in that: The distributed data storage architecture includes: Storage nodes are used to store data, perform data read and write operations, and maintain data copies; The data sharding manager is responsible for the data sharding strategy and distributes the data to different storage nodes; The consistency protocol layer is used to ensure the consistency of data between multiple storage nodes; Metadata management component, used to store and manage information about data location, data sharding, and node status; Load balancer, used to distribute client requests to different storage nodes; Fault detection and recovery components, which monitor node health, detect faults, and trigger recovery processes; Security modules, which implement security policies, including data encryption, access control, and authentication; The network layer is used to provide the communication basis between nodes and support data transmission between storage nodes; Monitoring and logging components, which are used to collect system performance indicators and log information for monitoring, troubleshooting, and performance analysis; Configuration management components are used to manage and distribute system configurations, supporting automated deployment and configuration updates.

4. The distributed data storage method based on the open source Hongmeng system according to claim 1 is characterized in that: The data sharding includes sharding strategy, sharding algorithm and sharding transparency; The redundant storage includes replica management, replica synchronization and fault recovery; The node management includes node discovery, load balancing and dynamic adjustment.

5. The distributed data storage method based on the open source Hongmeng system according to claim 4 is characterized in that: The data sharding, redundant storage and node management by the distributed storage module specifically include: Use distributed database middleware to build sharding and redundant storage mechanisms; Develop a custom storage engine and implement sharding logic and redundancy mechanism on Hongmeng system through the storage engine; Use the service framework provided by Hongmeng system to communicate and synchronize data between nodes; Deploy and manage storage nodes using containers and orchestration tools to automate node management and failure recovery.

6. The distributed data storage method based on the open source Hongmeng system according to claim 1 is characterized in that: The distributed consistency protocols include Raft protocol, Paxos protocol and Zab protocol; A distributed consistency protocol is integrated into the distributed data storage architecture to ensure data consistency, handle node failures, and support system expansion.

7. The distributed data storage method based on the open source Hongmeng system according to claim 1 is characterized in that: The distributed data storage method based on the open source Hongmeng system also includes: Regularly monitor the operating status of the distributed data storage system, detect and handle potential problems and failures; The potential problems and failures include network problems, data consistency problems, performance problems, security problems, hardware failures, and software failures; The processing of potential problems and failures specifically includes: Using monitoring tools to continuously monitor various indicators of the distributed data storage system; Set up an alarm system to immediately notify the operation and maintenance team if an anomaly is detected; Diagnose alerts and reported problems to determine the root cause of the problem; Take appropriate response measures based on the nature of the problem; Record the handling process of all problems and faults; Develop preventive measures based on the frequency and impact of the problem.

8. A distributed data storage system based on the open source Hongmeng system, characterized in that: The distributed data storage system based on the open source Hongmeng system includes: A storage architecture design module is used to obtain the target requirements for distributed data storage and design a distributed data storage architecture based on the Hongmeng system according to the target requirements; A distributed storage development module, which is used to develop a distributed storage module on the distributed data storage architecture based on the Hongmeng system, perform data sharding, redundant storage and node management through the distributed storage module, and integrate a distributed consistency protocol into the distributed data storage architecture; A data encryption storage module is used to obtain target data to be stored, determine an encryption algorithm and a key management scheme, encrypt the target data to obtain encrypted target data, and use a distributed data storage system to store the encrypted target data; The data access control module is used to perform identity authentication and authorization checks on the user when receiving a request to access the encrypted target data. If the user passes the authentication and obtains authorization, the encrypted target data is decrypted using the corresponding key to allow the user to access.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a distributed data storage program based on the open source Hongmeng system stored in the memory and executable on the processor. When the distributed data storage program based on the open source Hongmeng system is executed by the processor, the steps of the distributed data storage method based on the open source Hongmeng system as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a distributed data storage program based on the open source Hongmeng system. When the distributed data storage program based on the open source Hongmeng system is executed by the processor, the steps of the distributed data storage method based on the open source Hongmeng system as described in any one of claims 1 to 7 are implemented.

Images