A distributed data storage and access processing method

Abstract

The present application relates to the technical field of data storage, and particularly relates to a distributed data storage and access processing method. The method comprises the following steps: presetting a guide configuration file in a distributed storage cluster to perform identity authentication and communication link establishment on each storage node and a cluster management center, thereby generating a secure data interaction channel; a data sending end device sends a data storage request to the cluster management center through the secure data interaction channel and distributes data shards to appropriate storage nodes; a data receiving end device sends a data access request to the cluster management center through the secure data interaction channel and generates an access path and feeds back the access path to the data receiving end device; by monitoring the corresponding resource usage in each storage node in real time, when a preset threshold is reached, based on the result of insufficient resources, according to a preset data migration strategy and a load balancing algorithm, the data is redistributed to other storage nodes. The present application can realize efficient storage and rapid access of data in a distributed environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and in particular to a distributed data storage and access processing method. BACKGROUND

[0002] With the rapid development of information technology, the rapid growth of data volume and the continuous improvement of computing demand, the traditional centralized data storage and processing mode faces many challenges. Distributed data storage system can effectively solve the single point failure and load balancing problem, and improve the reliability and scalability of the system by dispersing the data storage in multiple nodes. At present, many large Internet companies and cloud computing platforms have adopted distributed storage architecture, such as Google's Bigtable, Amazon's Dynamo and Hadoop's HDFS, which have achieved remarkable results in handling large-scale data storage and high concurrency access.

[0003] In addition, the same patent CN113542439A discloses a distributed data storage and access method and device, which randomly generates an election committee including M nodes from N nodes, elects a unique block node from the M nodes according to the election committee, elects Y trusted nodes from the N nodes according to the election committee, and forms a distributed storage blockchain system with the unique block node. Based on the distributed storage blockchain system, the meta information of the entity promotion material is received; the unique block node links and stores the meta information of the entity promotion material; the query statement of the access end is received and parsed based on the distributed storage blockchain system; according to the contact request of the access end, the contact method of the entity data providing end is provided to the access end, so that the access end can obtain the private key of the entity data providing end. This can effectively balance the relationship between data security, data value independent control and the processing efficiency of the blockchain system, but the description of its security is relatively simple, mainly focusing on protecting data through blockchain storage and query. Although the blockchain has inherent security, it does not describe the details of node identity authentication and secure data interaction in depth, which may face some potential attack risks, especially in the security of storage node identity and communication link establishment, so as to effectively organize and index data, resulting in reduced data retrieval access efficiency. SUMMARY

[0004] In order to solve the above technical problems existing in the prior art distributed data access process, the present application provides a distributed data storage and access processing method through the management and scheduling process of distributed storage cluster, which can effectively improve the storage efficiency and data access speed of the system, including the following steps:

[0005] In the system initialization stage, a boot configuration file is preset in each storage node of the distributed storage cluster, and each storage node and the cluster management center are authenticated and a communication link is established based on the node identity key and the cluster topology information stored in the boot configuration file, thereby generating a secure data interaction channel between the storage node and the cluster management center.

[0006] In the data storage stage, the data sending end device sends a data storage request to the cluster management center through the secure data interaction channel, the cluster management center performs sharding processing on the data based on the data storage request according to a preset data sharding storage strategy, and distributes the data shards to appropriate storage nodes according to the load, storage capacity and data access frequency of each storage node.

[0007] In the data access stage, the data receiving end device sends a data access request to the cluster management center through the secure data interaction channel, the cluster management center generates an optimal data access path based on the data access request and the real-time state of each storage node according to the index information corresponding to the data, and feeds back the data access path to the data receiving end device, and the data receiving end device acquires the data from the corresponding storage node according to the data access path.

[0008] In the resource dynamic adjustment stage, the data storage capacity and the resource usage corresponding to the read / write performance in each storage node are monitored in real time, and when the resource usage of a certain storage node reaches a preset threshold, the cluster management center reassigns the data shards stored in the node to other storage nodes according to a preset data migration strategy and a load balancing algorithm based on the result of insufficient resource usage.

[0009] Through the management and scheduling mechanism of the above distributed storage cluster in the application, efficient resource utilization and data access performance can be realized while ensuring data security. In the system initialization stage, through the boot configuration file and identity authentication, the secure communication between each storage node and the cluster management center is ensured, and potential security risks are avoided. In the data storage stage, the cluster management center intelligently allocates data shards according to the load condition, storage capacity and data access heat, avoids the overload of a single storage node, improves the storage efficiency, and ensures the reliable storage of data. In the data access stage, the cluster management center generates the optimal access path according to the real-time index information and the state of the storage node, so that the data can be quickly and accurately transmitted to the data receiving end device, optimizing the data access speed and improving the user experience. Especially in the resource dynamic adjustment stage, the cluster management center can timely discover resource bottlenecks or overload problems by real-time monitoring of the resource usage of each node, and automatically trigger data migration and load balancing strategies to ensure that the system can still operate stably under high load conditions, avoiding the risk of performance degradation or system crash. This optimization scheme of the distributed storage cluster not only improves the processing capacity and response speed of the system, but also effectively improves the utilization rate of resources, reduces the maintenance cost, and provides higher flexibility and sustainability for future expansion.

[0010] Preferably, the boot configuration file specifically contains the communication key between the storage node and the cluster management center, the node unique identifier and the cluster network topology information, which are used to activate the secure data interaction communication between the storage node and the cluster management center.

[0011] The boot configuration file plays a crucial role in the distributed storage cluster in the application, specifically containing the communication key between the storage node and the cluster management center, the node unique identifier and the cluster network topology information. Through this configuration file, the identity of each storage node can be accurately identified, and the secure communication with the cluster management center is ensured. This mechanism enables each storage node to obtain the necessary identity information and network topology structure during initialization, ensuring the interoperability and security between nodes in the cluster. At the same time, it ensures the scalability of the system, and quickly completes the security configuration and communication link establishment when adding new nodes, so that the entire system can still operate efficiently and stably when facing a large number of storage nodes. Through effective boot configuration, not only the security of data transmission is enhanced, but also the complexity of deployment is reduced, and the maintenance efficiency of the system is improved.

[0012] Preferably, the identity authentication and communication link establishment based on the boot configuration file include the following steps:

[0013] Read the communication key and unique identifier of the node between the storage node and the cluster management center embedded in the boot configuration file;

[0014] Based on the communication key between the storage node and the cluster management center, asymmetric encryption authentication is performed on the identity information between the storage node and the cluster management center to generate a temporary authentication token between the storage node and the cluster management center.

[0015] Based on the temporary authentication token, the session keys of the storage node and the cluster management center are synchronized according to the two-way handshake verification mechanism, and a TLS encryption protocol is established based on the session keys of both parties.

[0016] A secure data transmission link is generated based on the TLS encryption protocol, and the authentication and communication link between the storage node and the cluster management center are completed based on the secure data transmission link to create a secure data interaction channel between the storage node and the cluster management center.

[0017] This invention enables highly secure data exchange between storage nodes and the cluster management center through the steps of identity authentication and communication link establishment via a bootstrap configuration file. During this stage, the identity information of the storage node and the cluster management center is authenticated using an asymmetric encryption mechanism, generating a temporary authentication token to ensure the authenticity and security of both parties' identities. Simultaneously, the session key synchronously generated based on a two-way handshake verification mechanism further enhances security, and the TLS encryption protocol ensures the confidentiality and integrity of data during transmission. This identity authentication and communication link establishment not only prevents data theft and tampering but also ensures the reliability and efficiency of data exchange, providing a secure and stable communication foundation for the system. This process is crucial for improving the security of the cluster system, especially when data interaction occurs across multiple nodes, effectively avoiding potential security threats.

[0018] Preferably, the data sending device sends a data storage request to the cluster management center through a secure data interaction channel, including the following steps:

[0019] The data sending device generates a raw data storage request message containing data element information and storage requirements;

[0020] The original data storage request message is transmitted with AES-256 secure encryption using a secure data exchange channel to obtain an encrypted data storage request data packet;

[0021] The encrypted data storage request data packet is sent to the cluster management center through a secure data interaction channel, and the data storage request data packet is decrypted and restored to generate a data storage request.

[0022] In this invention, during the transmission of data storage requests, the original request is encrypted using AES-256 through a secure data interaction channel before transmission. This effectively ensures the security of the request message during transmission. The AES-256 encryption algorithm has strong encryption capabilities, resisting various attack methods and ensuring the confidentiality and integrity of data storage requests. Simultaneously, the cluster management center can restore the original request information through decryption, ensuring that the received request content has not been tampered with or damaged. This step avoids external malicious interference or data leakage, enhancing the system's security and reliability, and ensuring the efficient transmission and accurate execution of data storage requests. With increasingly severe network attacks, this encryption mechanism provides stronger protection for distributed storage systems, enabling the system to maintain efficient and secure operation even in complex network environments.

[0023] Preferably, the original data storage request message includes the device identifier corresponding to the data sending device, the data type, the data size, and the data storage priority requirement information.

[0024] In this invention, the original data storage request message contains key information such as device identifier, data type, data size, and priority requirements. This provides the cluster management center with a detailed understanding of data storage needs. Through this information, the cluster management center can perform precise resource scheduling and priority processing based on the requirements of the data sending device. For example, data priority requirements help the system rationally allocate storage resources when faced with a large number of storage requests, prioritizing high-priority data requests. This mechanism ensures that the system can dynamically adjust resource allocation according to actual conditions, enabling high-priority requests to receive timely responses, thereby improving the efficiency and response speed of the entire system. Furthermore, information on data type and size helps optimize data storage strategies, providing necessary support for subsequent sharding and data access.

[0025] Preferably, the step of processing data into data shards based on data storage requests and according to a preset data sharding storage strategy, and then distributing the data shards to appropriate storage nodes, includes the following steps:

[0026] The cluster management center is used to perform data characteristic analysis on the data type and data size in the data storage request in order to determine the data shard size.

[0027] Based on the device identifier corresponding to the data sending device and the priority requirement information for data storage, the priority of data storage requests is classified.

[0028] Based on the load status, storage capacity, and data access frequency reported in real time by each storage node, combined with the data shard size and request priority, the target storage node for each data shard is calculated through a load balancing algorithm.

[0029] The data is sharded according to the determined data shard size, and the data shards are sent to the corresponding target storage nodes through a secure data exchange channel. At the same time, data index information is established in the cluster management center.

[0030] In the data storage request processing process, the cluster management center analyzes the data type, size, and priority requirements of the sending device to rationally determine the data sharding strategy. Based on the shard size, the cluster management center can efficiently allocate data to appropriate storage nodes, thereby avoiding resource overload on individual nodes. Simultaneously, the load balancing algorithm dynamically adjusts the data storage location based on the load, storage capacity, and data access frequency of each storage node, ensuring maximum resource utilization for each node. In this way, the system not only improves storage resource utilization but also avoids performance bottlenecks caused by excessive load on certain nodes, enhancing the scalability and stability of the entire cluster. Furthermore, the cluster management center establishes data index information, laying the foundation for fast data access and improving data access efficiency.

[0031] Preferably, the data receiving device sends a data access request to the cluster management center through a secure data interaction channel, including the following steps:

[0032] The data receiving device generates a raw data access request message containing the data identifier and access requirements;

[0033] The original data access request message is encrypted using the national standard SM4 encryption method through a secure data exchange channel to obtain an encrypted data access request data packet.

[0034] The encrypted data access request data packet is sent to the cluster management center through a secure data interaction channel, and the data packet is decrypted and restored to generate a data access request.

[0035] This invention ensures secure and reliable data transmission by using a secure data interaction channel to send data access requests. First, the receiving device generates an original data access request message containing data identifiers and access requirements, guaranteeing the accuracy and integrity of the request content. Next, the original data request is encrypted using the SM4 encryption algorithm, effectively preventing data theft or tampering during transmission and enhancing system security. Finally, upon receiving the encrypted data access request, the cluster management center decrypts and restores it, ensuring the accurate transmission of the original data access request content. The SM4 encryption technology used in this process complies with domestic information security policies, providing legal and technical protection for the transmission of sensitive data and avoiding the risk of information leakage. Furthermore, the secure encryption transmission mechanism throughout the entire process protects the data privacy of users and enterprises, reduces the possibility of external attacks or data leakage, and improves the system's credibility and stability.

[0036] Preferably, the original data access request message includes the device identifier corresponding to the data receiving device, the data identifier to be accessed, and the urgency information of the data access.

[0037] In this invention, the original data access request message includes a device identifier, a data identifier, and urgency information. This helps the cluster management center accurately identify the source and nature of the access request. The device identifier ensures that the data request comes from a legitimate device, preventing access requests from unauthorized devices from entering the system. The data identifier allows the cluster management center to clearly know the location of the data requested by the user, reducing performance loss caused by data access errors. The urgency information provides the cluster management center with a basis for evaluating request priorities, rationally arranging the order of data access, and ensuring that urgent data access requests are processed first. This process helps improve system response speed and service efficiency, especially in high-concurrency environments, effectively reducing latency and ensuring timely response and processing of user data requests.

[0038] Preferably, the step of generating the optimal data access path based on the data access request, the corresponding index information of the data, and the real-time status of each storage node includes the following steps:

[0039] The cluster management center uses the data identifier required in the data access request to obtain the storage location information of the data shards from the data index information;

[0040] The priority of data access requests is evaluated based on the device identifier corresponding to the data receiving device and the urgency information of data access.

[0041] Based on the network bandwidth, read / write performance, and load reported in real time by each storage node, combined with the storage location information and request priority of the data shards, the optimal data access path from the data receiving device to each data shard storage node is calculated using the improved Dijkstra algorithm.

[0042] This invention utilizes a cluster management center to generate an optimal data access path by combining index information, device identifiers, and urgency levels from data access requests. This significantly improves data access efficiency and system resource utilization. First, the cluster management center retrieves storage location information from the data index based on the data access request identifier, ensuring accurate request location and avoiding unnecessary addressing processes, thus reducing latency. Second, prioritization is achieved by combining device identifiers and urgency levels, ensuring high-priority requests receive responses in the shortest possible time, avoiding access delays caused by network congestion or busy storage nodes. The optimal path is calculated using an improved Dijkstra algorithm, which flexibly adjusts data transmission routes based on real-time network bandwidth, read / write performance, and load conditions, ensuring data requests are completed on the optimal path. This reduces bottlenecks in transmission, further optimizes the entire data access process, ensures efficient system operation and stability, and enhances user experience.

[0043] Preferably, the step of redistributing the data shards stored on the node to other storage nodes based on insufficient resource utilization, according to a preset data migration strategy and load balancing algorithm, includes the following steps:

[0044] The cluster management center is used to respond to insufficient resource utilization by data migration, thereby generating data migration instructions.

[0045] Based on data migration instructions, data heat analysis and dependency analysis are performed on data shards stored on nodes with insufficient resource utilization to obtain the access frequency, correlation degree of data shards and resource utilization of other storage nodes;

[0046] Based on the access frequency and correlation of data shards, as well as the resource usage of other storage nodes, the data shards are redistributed to other storage nodes according to the preset data migration strategy and load balancing algorithm, and the data index information in the cluster management center is updated synchronously.

[0047] This invention utilizes data migration based on insufficient resource utilization to effectively optimize storage resource allocation within a cluster. By leveraging data migration commands generated by the cluster management center, the system can automatically respond to nodes, reducing manual intervention and improving overall cluster performance and efficiency. During data migration, combining data heat analysis and dependency analysis, data can be intelligently redistributed to other storage nodes based on the access frequency and correlation of data shards, avoiding resource idleness or over-congestion. The application of load balancing algorithms ensures more even data distribution, thereby improving cluster response speed and storage efficiency. Synchronously updating the data index information in the cluster management center ensures information consistency between storage nodes, preventing access errors or performance degradation caused by data migration. This method significantly improves cluster resource utilization, reduces resource waste and system bottleneck risks, and enhances system resilience and scalability.

[0048] The present invention has the following specific beneficial effects:

[0049] (1) During the system initialization phase, by pre-setting boot configuration files for each storage node in the distributed storage cluster, secure communication and authentication between the cluster management center and each storage node are ensured. The identity key and cluster topology information of each storage node are stored in the boot configuration file. In this way, the cluster management center can perform effective authentication when a node connects. This process ensures the legitimacy of each node in the cluster, avoids unauthorized node access, and ensures data security. In addition, through the topology information in the boot configuration file, the cluster management center can obtain the location and status of each node in a timely manner, thereby establishing a stable and secure communication link. This authentication and communication link establishment is the foundation for the reliable operation of the entire distributed storage system, effectively avoiding unnecessary conflicts and communication delays between nodes, and providing reliable guarantees for subsequent data storage, access, and dynamic resource adjustment. As a result, the nodes in the cluster can achieve efficient collaboration, reduce the management difficulty of the system, and ensure the efficiency and security of data exchange.

[0050] (2) In the data storage stage, the cluster management center performs data sharding based on the data storage request and selects storage nodes reasonably according to the preset data sharding storage strategy and load balancing algorithm. The key to this stage is to ensure that the data can be stored efficiently and evenly on different storage nodes through the intelligent scheduling capability of the cluster management center. First, the data sending device initiates a data storage request to the cluster management center through a secure data interaction channel. The cluster management center then determines which storage nodes are most suitable for storing these data shards based on the content of the data request, combined with the load of the storage nodes, storage capacity, and data access frequency. The reasonable application of load balancing and data sharding strategies can avoid overloading of some nodes or waste of storage resources, ensuring full utilization of cluster resources. At the same time, this strategy also effectively improves the reliability of data storage because different nodes store different shards of data. If a node fails, other nodes can still provide data backup, thereby reducing the overall failure risk of the system. This stage lays a solid foundation for efficient and reliable data storage, effectively organizes and indexes data, and improves the scalability and fault tolerance of the storage process.

[0051] (3) During the data access phase, a secure data interaction channel ensures the security of data transmission between the data receiving device and the cluster management center. Based on the received data access request, the cluster management center combines data index information and the real-time status of storage nodes to generate the optimal data access path. In this way, the cluster management center can intelligently select the optimal data access path based on real-time data such as the load, storage status, and network latency of each node, thereby improving the speed and efficiency of data access. After receiving the access path, the data receiving device retrieves the required data from the designated storage node according to the path indication. This process not only accelerates the data transmission process and reduces unnecessary waiting time, but also optimizes the stability of data access through reasonable path planning, avoiding access bottlenecks caused by network congestion or storage node overload. Furthermore, this intelligent scheduling mechanism enhances the user experience of the system, thereby improving the efficiency of data retrieval and access. In summary, the optimization of the data access phase ensures that the distributed storage system can maintain high performance and stability even in a high-concurrency environment.

[0052] (4) Dynamic Resource Adjustment Phase: By monitoring the real-time resource usage of each storage node, the system can flexibly adjust resources according to load changes. When the resource usage of a storage node reaches a preset threshold, the cluster management center will respond promptly and initiate a data migration mechanism to migrate some data fragments from that node to other storage nodes. The key to dynamic resource adjustment is to ensure that the resource allocation of each node in the cluster is as balanced as possible through load balancing algorithms and data migration strategies, avoiding overload of a single node or waste of storage resources. This mechanism improves the flexibility and scalability of the storage system, and can dynamically adjust resource allocation according to the actual situation of node storage capacity, read and write performance, etc., thereby ensuring efficient data access and stable system operation. Through this dynamic adjustment mechanism, the system can automatically respond to load fluctuations in different time periods, avoiding the need for human intervention, improving the automation level of resource management, and reducing maintenance costs. At the same time, this mechanism also improves the fault tolerance of the system. When a storage node cannot continue to provide services due to insufficient resources or failure, the data can be quickly migrated to other nodes, thereby avoiding service interruption and improving the reliability and accuracy of data storage access. Attached Figure Description

[0053] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0054] Fig. 1 This is a flowchart illustrating the steps of the distributed data storage and access processing method of the present invention;

[0055] Fig. 2 This is a schematic diagram illustrating the steps of the present invention for identity authentication and communication link establishment based on a boot configuration file;

[0056] Fig. 3 This is a flowchart illustrating the steps of the data sending device of the present invention sending a data storage request to the cluster management center through a secure data interaction channel. Detailed Implementation

[0057] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0058] To achieve the above objectives, please refer to Figs. 1-3 This invention provides a distributed data storage and access processing method, such as... Fig. 1 As shown, it includes the following steps:

[0059] S1: During the system initialization phase, a boot configuration file is pre-installed in each storage node of the distributed storage cluster. Based on the node identity key and cluster topology information stored in the boot configuration file, the system performs identity authentication and establishes a communication link between each storage node and the cluster management center, thereby generating a secure data interaction channel between the storage node and the cluster management center.

[0060] In this embodiment of the invention, a boot configuration file containing node identity keys and cluster topology information is pre-programmed into the storage medium of each storage node in the distributed storage cluster. The node identity key is a 2048-bit RSA key pair generated using an asymmetric encryption algorithm. The private key is stored in the Trusted Execution Environment (TEE) of the storage node, while the public key is uploaded to the cluster management center. The cluster topology information includes the IP address, port number, hardware configuration, and other information of the storage nodes, and is stored in the boot configuration file in JSON format. After each storage node starts up, it establishes a connection with the cluster management center via the Secure Shell Protocol (SSH). The storage node uses its private key to sign a message containing its own node information. The cluster management center uses the corresponding public key to verify the signature and complete the identity authentication. After successful authentication, a communication link is established based on the Transport Layer Security Protocol (TLS 1.3). By exchanging certificates and negotiating encryption algorithms and keys, a secure data interaction channel is generated between the storage node and the cluster management center. This channel uses the AES-256-GCM symmetric encryption algorithm to encrypt the transmitted data.

[0061] S2: Data storage stage. The data sending device sends a data storage request to the cluster management center through a secure data interaction channel. Based on the data storage request, the cluster management center performs data sharding processing according to the preset data sharding storage strategy, and distributes the data shards to the appropriate storage nodes according to the load, storage capacity and data access frequency of each storage node.

[0062] In this embodiment of the invention, assuming that the data sending device is an enterprise server that needs to store 10GB of business data, the server sends a data storage request containing data element information (such as data type and creation time) to the cluster management center through an established secure data interaction channel using the secure version of Hypertext Transfer Protocol (HTTPS). After receiving the request, the cluster management center divides the 10GB of data into 1000 10MB data fragments according to a preset fixed-size fragmentation strategy. Subsequently, the cluster management center obtains real-time information on the load of each storage node (by monitoring CPU utilization, memory utilization, and disk I / O rate), storage capacity (remaining available space), and data access frequency (the number of times the node's data has been accessed in the past week). For example, storage node A currently has a CPU utilization of 80%, a remaining storage capacity of 100GB, and an access frequency of 100 times per week; storage node B currently has a CPU utilization of 30%, a remaining storage capacity of 500GB, and an access frequency of 20 times per week. Finally, the data is sharded and distributed to storage node B, which has a lower load and sufficient storage capacity. The distribution process uses a reliable file transfer protocol (such as FTP over TLS) to ensure complete data transmission.

[0063] S3: During the data access phase, the data receiving device sends a data access request to the cluster management center through a secure data interaction channel. Based on the data access request, the cluster management center generates the optimal data access path according to the index information corresponding to the data and the real-time status of each storage node, and feeds back the data access path to the data receiving device. The data receiving device then retrieves the data from the corresponding storage node according to the data access path.

[0064] In this embodiment of the invention, the data receiving device is a workstation. When it needs to access the aforementioned stored business data, the workstation sends a data access request containing data identifiers (such as filename and creation time) to the cluster management center through a secure data interaction channel. Upon receiving the request, the cluster management center queries the data index table to obtain the corresponding index information. This index information records the storage location of the data fragments on each storage node. Simultaneously, the status of each storage node is monitored in real time, including network latency (obtained via ping command round-trip time) and disk response time (obtained via I / O performance monitoring tools). Based on this information, the optimal data access path is calculated using Dijkstra's algorithm. Assuming there are storage nodes C, D, and E, and the data fragments are stored on nodes C and E respectively, the path with the minimum sum of network latency and disk response time from the workstation to node C and then to node E is the optimal path. The cluster management center feeds this path back to the workstation, which then retrieves the data fragments from storage nodes C and E via the NFSv4 protocol according to the path and reassembles the data locally.

[0065] S4: In the dynamic resource adjustment phase, the resource usage corresponding to the data storage capacity and read / write performance of each storage node is monitored in real time. When the resource usage of a certain storage node reaches a preset threshold, the cluster management center, based on the result of insufficient resource usage, will redistribute the data shards stored on that node to other storage nodes according to the preset data migration strategy and load balancing algorithm.

[0066] In this embodiment of the invention, a performance monitoring agent is deployed on each storage node to collect data storage capacity and read / write performance data in real time. For example, the remaining space size, disk read / write operations per second, and average read / write latency of the storage nodes are collected every 5 minutes. When the remaining storage capacity of storage node F drops to 10GB (the preset threshold is 20GB), the cluster management center initiates a data migration process. Based on a preset data migration strategy based on data access popularity, the data shards with the lowest access popularity are migrated first. At the same time, a consistent hash load balancing algorithm is used to calculate the target storage node. Assuming that storage node G currently has low load and sufficient storage capacity, a network file replication tool (such as Rsync) is used to copy the data shards that meet the migration strategy from storage node F to storage node G, and the storage location information of the data shards in the data index table is updated, completing the redistribution of data among the storage nodes.

[0067] Furthermore, the boot configuration file specifically includes the communication key between the storage node and the cluster management center, the node's unique identifier, and the cluster network topology information, which are used to activate secure data interaction communication between the storage node and the cluster management center.

[0068] In this embodiment of the invention, during the system initialization phase, the burning and activation process of the boot configuration file involves multiple precise operations. The boot configuration file is stored in a read-only memory (ROM) chip, which contains a communication key pair generated by an elliptic curve cryptography algorithm (ECC, P-256 curve). The private key is 256 bits long, the public key is encoded in X.509 format, and the node unique identifier is generated using the UUIDv4 standard with a length of 128 bits to ensure global uniqueness. The cluster network topology information is stored in Protocol Buffers format, including Layer 3 BGP routing information, AS number allocation, and multi-path equal cost routing (ECMP) configuration. After the storage node is powered on, the Trusted Platform Module (TPM 2.0) first verifies the digital signature of the boot configuration file using the Ed25519 signature algorithm. After successful verification, the configuration file is loaded into the SecureEnclave.

[0069] Furthermore, the authentication and communication link establishment based on the boot configuration file includes the following steps:

[0070] S101: Read the communication key and unique identifier of the node between the storage node and the cluster management center embedded in the boot configuration file;

[0071] In this embodiment of the invention, when reading the boot configuration file, the secure boot loader (UEFI secure boot mode) of the storage node first verifies the integrity hash value of the file. This hash value is calculated using the SHA-384 algorithm. The private key in the communication key pair is initialized by a hardware random number generator (HRNG) and stored in a dedicated key slot of the TPM. Access permissions are strictly limited to be called only by secure communication services. The node's unique identifier is loaded into a protected area of ​​memory (isolated by memory protection key MPK technology) and an integrity monitoring flag is attached. Any unauthorized modification will trigger a system reset. The entire reading process protects data transmission through memory encryption technology (AES-256-XTS), and the bus adopts a trusted path mechanism to prevent man-in-the-middle attacks.

[0072] S102: Based on the communication key between the storage node and the cluster management center, perform asymmetric encryption authentication on the identity information between the storage node and the cluster management center to generate a temporary authentication token between the storage node and the cluster management center.

[0073] In this embodiment of the invention, the authentication process employs an elliptic curve-based Diffie-Hellman key exchange (ECDHE) algorithm, specifically using the Curve 25519 curve. The storage node first generates a temporary elliptic curve key pair, encapsulating the public key and the node's unique identifier in a TLS handshake message. It then uses its stored private key to perform an ECDSA signature (P-256 curve). Upon receiving the message, the cluster management center verifies the signature using its pre-stored public key and generates its own temporary key pair. Both parties negotiate a 32-byte master secret using the ECDHE algorithm. This master secret is further derived into session keys (including an encryption key, an integrity verification key, and an initialization vector). The temporary authentication token uses the JSON Web Token (JWT) format, containing the node identifier, timestamp, and permission declaration. It is signed using the HMAC-SHA3-256 algorithm, with a validity period of 5 minutes.

[0074] S103: Based on the temporary authentication token, synchronize the corresponding session keys of the storage node and the cluster management center according to the two-way handshake verification mechanism, and establish a TLS encryption protocol based on the corresponding session keys of both parties;

[0075] In this embodiment of the invention, a two-way message exchange mode of the TLS 1.3 standard is adopted through a two-way handshake verification mechanism. The storage node first sends a ClientHello message, which includes a list of supported cipher suites (ChaCha20-Poly1305 is preferred) and extended fields (such as Certificate Status Request and Application Layer Protocol Negotiation ALPN). The cluster management center responds with a ServerHello message, selects a specific cipher suite, and sends a certificate chain (using OCSP stapling technology to improve verification efficiency). The storage node verifies the validity of the certificate (including certificate chain verification, CRL check, and time validity check) and generates a Finished message (using HMAC to verify the integrity of the handshake process). The cluster management center also verifies the client's Finished message. After both parties synchronize the session key, the handshake is completed. The entire process uses Perfect ForwardSecrecy technology to ensure that historical communication content is untraceable.

[0076] S104: Generate a secure data transmission link according to the TLS encryption protocol, and complete the identity authentication and communication link establishment between the storage node and the cluster management center based on the secure data transmission link, so as to generate a secure data interaction channel between the storage node and the cluster management center.

[0077] In this embodiment of the invention, a secure data transmission link is established based on the TLS 1.3 encryption protocol, using a 128-bit AES-GCM symmetric encryption algorithm to protect data transmission. The data frame format follows the DTLS 1.3 standard (suitable for connectionless protocols), including a record layer header (containing version, length, and content type), an encrypted data payload, and an AEAD authentication tag. Each data packet contains a 32-bit sequence number and a 64-bit timestamp to prevent replay attacks. After the link is established, the storage node sends an authentication request containing a hardware fingerprint (generated via the TPM PCR register) and runtime measurements. The cluster management center verifies the consistency of these values ​​with the pre-registration information. If they match, a long-term session credential (using SAML 2.0 format) is issued, and the communication channel is marked as a trusted channel. The log recording of the entire process uses blockchain technology to ensure immutability. Each block contains 2048 transactions linked using the SHA-256 hash algorithm.

[0078] Furthermore, the data sending device sending a data storage request to the cluster management center through a secure data interaction channel includes the following steps:

[0079] S201: The data sending device generates a raw data storage request message containing data element information and storage requirements;

[0080] In this embodiment of the invention, during the data storage request generation stage, the data sending device generates an original data storage request message based on a pre-configured message format specification. This message adopts the Protocol Buffers serialization format and contains a strictly defined data structure: the device identifier adopts a 128-bit UUIDv4 format, which is generated by the Hardware Security Module (HSM) and bound to the device's TPM (Trusted Platform Module); the data type field follows the ISO / IEC 11179 metadata registration standard and uses predefined classification codes (e.g., 001 represents structured data, 002 represents semi-structured data); the data size field is accurate to the byte and is represented by a 64-bit unsigned integer; the storage priority requirement information uses 3-bit binary encoding (001-high priority, 010-medium priority, 100-low priority), and all fields adopt the TLV (tag-length-value) encoding method to ensure the self-descriptiveness and scalability of the message structure.

[0081] S202: Use the secure data exchange channel to transmit the original data storage request message with AES-256 secure encryption to obtain the encrypted data storage request data packet;

[0082] In this embodiment of the invention, after the original data storage request message is generated, it is transmitted in encrypted form through a secure data interaction channel. The encryption is performed using the AES-256-GCM algorithm, and the key derivation follows the HKDF (HMAC-based Key Derivation Function) standard. The encryption key is derived from the master secret generated during the TLS handshake process. The initialization vector (IV) uses a 256-bit random number generated by a hardware random number generator (HRNG). The encryption process strictly follows the NIST SP 800-38D standard. The additional authentication data (AAD) includes a message timestamp (Unix time accurate to milliseconds), a message sequence number (32-bit incrementing counter), and a message type identifier (fixed at 0x0002). The encrypted data storage request data packet contains a 12-byte GCM authentication tag for verifying data integrity and authenticity.

[0083] S203: Send the encrypted data storage request data packet to the cluster management center through the secure data interaction channel, and decrypt and restore the data storage request data packet to generate a data storage request.

[0084] In this embodiment of the invention, when the encrypted data packet is sent through a secure data interaction channel, the application data transmission mechanism of the TLS 1.3 protocol is adopted. The data packet is encapsulated in the TLS record layer, with each record having a maximum length of 16384 bytes and containing a 24-byte record header (version number, content type, and length). TCP Fast Open technology is enabled during transmission to reduce connection establishment latency. After receiving the data packet, the cluster management center first verifies the validity of the TLS session (through a session ticket mechanism), and then decrypts the data packet using a pre-shared session key. The decryption process verifies the GCM authentication tag. If the verification fails, the data packet is discarded and a security audit is triggered. After successful decryption, the values ​​of each field are extracted from the Protocol Buffers message, and finally, a data storage request in the internal format is generated.

[0085] Furthermore, the original data storage request message includes the device identifier corresponding to the data sending device, the data type, the data size, and the data storage priority requirement information.

[0086] In this embodiment of the invention, each component of the original data storage request message has explicit technical constraints. The device identifier is verified by signature verification using an HSM to ensure its authenticity; the data type field is validated by comparison with a predefined classification dictionary; and the data size field uses range validation (minimum value 1 byte, maximum value 2 bytes). 64 -1 byte); storage priority encoding is validated through bitmask operations, the timestamp field is used to detect message replay attacks, a 5-minute acceptance window is set, messages outside the window will be rejected, all field validation rules are implemented through a hardware-accelerated regular expression engine to ensure the efficiency and determinism of the validation process, the entire message processing flow is controlled by a state machine, each processing stage is set with explicit timeout parameters (e.g., 100 milliseconds for decryption timeout, 50 milliseconds for field validation timeout), and a retry mechanism or error handling process is automatically triggered in case of timeout.

[0087] Furthermore, the step of processing data into data shards based on data storage requests and according to a preset data sharding storage strategy, and then distributing the data shards to appropriate storage nodes, includes the following steps:

[0088] The cluster management center is used to perform data characteristic analysis on the data type and data size in the data storage request in order to determine the data shard size.

[0089] In this embodiment of the invention, the cluster management center processes data based on a predefined data type and sharding strategy mapping table. For structured data (data type code 001), a fixed-size sharding strategy is adopted, with a shard size of 64MB. For semi-structured data (data type code 002), a block-based adaptive sharding strategy is adopted, with a base shard size of 32MB, which can be dynamically adjusted according to the internal structure of the data. For unstructured data (data type code 003), a content-aware sharding strategy is adopted, with a shard size of 128MB. Data size verification uses an interval judgment mechanism. If the data is less than 1GB, a small data optimization strategy is activated (shard size is halved). If the data exceeds 100GB, a big data processing flow is initiated (multi-threaded sharding is enabled). All judgment rules are executed through a hardware-accelerated decision tree engine to ensure microsecond-level response time.

[0090] Furthermore, based on the device identifier corresponding to the data sending device and the priority requirement information for data storage, the priority of data storage requests is classified.

[0091] In this embodiment of the invention, the priority classification process strictly follows a three-level classification system. The device identifier is verified by comparing it with the list of privileged devices. Requests initiated by privileged devices (such as core business servers) are directly assigned high priority (code 001). For ordinary devices, the priority is mapped according to the storage priority requirement information: the high priority (001) in the original request is mapped to the system high priority, the medium priority (010) is mapped to the system medium priority, and the low priority (100) is mapped to the system low priority. Each priority is associated with a specific resource quota: high priority requests are allocated 50% of the available bandwidth, medium priority 30%, and low priority 20%. The priority tag uses metadata injection technology to add a 4-byte priority tag to the request message header to ensure that it is visible throughout the entire processing flow.

[0092] Furthermore, based on the real-time load, storage capacity, and data access frequency reported by each storage node, combined with the data shard size and request priority, the target storage node for each data shard is calculated using a load balancing algorithm.

[0093] In this embodiment of the invention, the target storage node calculation adopts an improved consistent hashing algorithm. The load of each storage node is comprehensively evaluated by real-time collection of CPU utilization (accurate to 0.1%), memory utilization (accurate to MB), and network bandwidth utilization (accurate to kbps). Storage capacity monitoring is accurate to the byte level and is obtained through file system quota management technology. Data access popularity is statistically analyzed by the number of accesses within a sliding time window (past 72 hours). An exponential decay algorithm is used to reduce the weight of historical data. The algorithm maps each storage node to 256 virtual nodes to form a virtual node ring. Data sharding calculates hash values ​​using the MurmurHash3 algorithm. The nearest virtual node is found clockwise as the target storage node. High-priority requests are given an additional 50% weight during calculation to ensure that they are preferentially allocated to the node with the best performance.

[0094] Furthermore, the data is sharded according to the determined data shard size, and the data shards are distributed to the corresponding target storage nodes through a secure data exchange channel. At the same time, data index information is established in the cluster management center.

[0095] In this embodiment of the invention, the data sharding and distribution process adopts zero-copy technology. The original data is transmitted from the sending device to the memory buffer of the cluster management center via DMA (Direct Memory Access). The buffer adopts a circular queue structure with a size of 4GB. The sharding process is implemented by a hardware block engine, which cuts the data according to the determined shard size. Each shard generates a 64-bit CRC checksum. The sharded data is directly transmitted to the target storage node via RDMA (Remote Direct Memory Access) technology, skipping kernel space copying. The transmission process adopts the QUIC protocol, which provides multiplexing and fast packet loss recovery capabilities. The data index information is established in the cluster management center using a B+ tree structure, which includes information such as shard hash value, storage node location, and timestamp. The index data is stored in a RAID10 array to ensure high availability and read / write performance. The transmission status of each shard is recorded through a distributed transaction log, and the Paxos algorithm is used to ensure log consistency.

[0096] Furthermore, the data receiving device sending a data access request to the cluster management center through a secure data interaction channel includes the following steps:

[0097] The data receiving device generates a raw data access request message containing the data identifier and access requirements;

[0098] In this embodiment of the invention, the data receiving device generates an original data access request message according to a predetermined message format specification. The device identifier uses a globally unique 128-bit UUID encoding, which is generated by the device's built-in Trusted Platform Module (TPM) in combination with hardware fingerprints (such as network card MAC address and CPU serial number) to ensure that it cannot be forged. The identifier of the data to be accessed uses a SHA-256 hash value, which is generated by the cluster management center and recorded in the index information during the data storage stage. The data receiving device obtains this identifier through the business system. The urgency information of the data access is represented by a 3-bit binary code: 001 represents urgent (such as real-time transaction data query), 010 represents normal (such as historical data statistics), and 100 represents low priority (such as offline data mining). The message is structured and encoded using the ASN.1 standard to form binary data with clearly defined fields.

[0099] Furthermore, the original data access request message is encrypted using the national standard SM4 encryption method through a secure data exchange channel to obtain an encrypted data access request data packet.

[0100] In this embodiment of the invention, the encryption of the original data access request message is implemented using the national cryptographic SM4 algorithm. The key management system allocates a pair of SM4 keys to each data receiving device. The key length is 128 bits and is stored in the device's security chip. It is generated by a hardware random number generator (HRNG). The encryption process strictly follows the GM / T 0002-2012 standard. The message is encrypted in blocks using the electronic codebook (ECB) mode. Each block is 128 bits long. Before encryption, a 16-byte initialization vector (IV) is added to the message header. This vector is generated by the device in real time and transmitted through a secure channel. The encrypted message is appended with an 8-byte MAC (Message Authentication Code), which is calculated using the HMAC-SM3 algorithm to verify the integrity and authenticity of the message. Finally, the encrypted data access request data packet is formed.

[0101] Furthermore, the encrypted data access request data packet is sent to the cluster management center through a secure data interaction channel, and the data packet is decrypted and restored to generate a data access request.

[0102] In this embodiment of the invention, the encrypted data packets are transmitted through an established secure data interaction channel. A General Cryptographic Communication Protocol (GMTLS) based on TLS 1.3 is used for transport layer protection. Before transmission, the data packets are encapsulated into GMTLS records. Each record contains a 2-byte version number, a 1-byte content type, a 2-byte length field, and an encrypted message body. During transmission, the data receiving device establishes a connection using the TCP protocol and enables fast retransmission and fast recovery mechanisms to ensure transmission stability. After receiving the data packets, the cluster management center first verifies the validity of the TLS session, then retrieves the corresponding SM4 key from the security chip to decrypt the data packets. The decrypted data is restored to the original message format using an ASN.1 decoder, extracting the device identifier, data identifier, and urgency information, ultimately generating a data access request that can be processed.

[0103] Furthermore, the original data access request message includes the device identifier corresponding to the data receiving device, the data identifier to be accessed, and the urgency information of the data access.

[0104] In this embodiment of the invention, after the device identifier is generated at the receiving end, it needs to be verified by the device registration system to ensure that the device has been registered in the cluster management system and has data access permissions. During transmission and processing, the uniqueness and accuracy of the data identifier are verified by hash value comparison to prevent the data identifier from being tampered with or transmitted incorrectly. The urgency information of data access directly affects the priority scheduling of subsequent processing flows. For example, urgent requests will skip the queue and directly enter the data access path calculation stage, while regular and low-priority requests will be processed sequentially according to the system load, following the first-in-first-out principle or a weighted scheduling strategy.

[0105] Furthermore, the step of generating the optimal data access path based on the data access request, the corresponding index information of the data, and the real-time status of each storage node includes the following steps:

[0106] The cluster management center uses the data identifier required in the data access request to obtain the storage location information of the data shards from the data index information;

[0107] In this embodiment of the invention, after receiving a data access request through the cluster management center, the data shards are located through the data index information system. The data index adopts a distributed hash table (DHT) structure, which maps data identifiers to corresponding physical storage locations according to a consistent hashing algorithm. When a data access request is received, the required data identifier is first hashed using SHA-256, and then the physical node storing the data shard is quickly located through the DHT routing table. The storage location information of each data shard includes metadata such as node IP address, port number, storage path, and replica location. This information is stored in the index system in Protocol Buffers format, and microsecond-level query response is achieved through memory-mapped file technology.

[0108] Furthermore, based on the device identifier corresponding to the data receiving device and the urgency information of data access, the priority of data access requests is evaluated;

[0109] In this embodiment of the invention, the cluster management center performs priority evaluation based on device identifiers and urgency information. The device identifier is verified by comparing it with a pre-registered list of trusted devices to confirm the device's access permissions. The urgency information is directly mapped to the system's three-level scheduling priority: 001 urgency level corresponds to the real-time scheduling queue, 010 normal level corresponds to the round-robin scheduling queue, and 100 low priority corresponds to the background scheduling queue. Each queue is associated with a specific resource quota. The urgency level queue is allocated 50% of the system bandwidth resources, the normal level 30%, and the low priority 20%. The evaluation result is achieved by adding a 4-byte priority tag to the request message header. This tag is used throughout the entire processing flow to ensure the consistency of the priority policy.

[0110] Furthermore, based on the network bandwidth, read / write performance, and load reported in real time by each storage node, combined with the storage location information and request priority of the data shards, the optimal data access path from the data receiving device to each data shard storage node is calculated using the improved Dijkstra algorithm.

[0111] In this embodiment of the invention, path calculation employs an improved Dijkstra algorithm. Each storage node reports network bandwidth (accurate to kbps), read / write performance (IOPS and throughput), and load status (CPU utilization and memory utilization) to the cluster management center every 500 milliseconds via a heartbeat mechanism. This real-time data is integrated into the global topology graph. The edge weight of each node is calculated by combining network latency, bandwidth utilization, and node load. Network latency is measured by round-trip time (RTT) using ICMP echo requests (ping), with a probe packet sent every 500 milliseconds for 10 consecutive data collections. A Kalman filter is used to filter the raw data, eliminating burst noise interference, resulting in a smoothed network latency value T (in milliseconds). Bandwidth utilization is calculated using NetFlow traffic monitoring technology, collecting inbound / outbound traffic of the link every 2 seconds, and calculating the average bandwidth utilization U (percentage) within a 5-minute sliding window. The formula is: U = (Actual traffic / Total link bandwidth). ×100%; Node load is calculated using three metrics: CPU utilization (C), memory utilization (M), and disk I / O queue length (Q). CPU and memory utilization are obtained in real-time through the / proc / stat and / proc / meminfo interfaces of the node's operating system, while I / O queue length is collected using the iostat tool. After normalizing the three metrics to the [0,1] interval, the node load L is calculated using a weighted summation formula: L = 0.5C + 0.3M + 0.2Q. Network latency is quantized by mapping the filtered latency value T to the [0,1] interval using a piecewise linear function:

[0112] plaintext

[0113] If T≤10ms, S_T=0

[0114] if 10ms <T≤100ms, S_T=(T-10) / 90

[0115] If T>100ms, S_T=1

[0116] Bandwidth utilization quantization maps the utilization rate U to the [0, 1] interval using a non-linear function:

[0117] plaintext

[0118] S_U=U 2 / 10000 (when U≤70%)

[0119] S_U = 1 - (100 - U) 2 / 900 (when U>70%)

[0120] Node load quantization directly uses the normalized L value as the quantization result S_L. The base weight is calculated using a weighted summation formula: W_base = 0.4S_T + 0.3S_U + 0.3S_L. QoS factors are used to adjust the weights based on the priority of data access requests. For urgent requests, an acceleration factor α = 0.7 is introduced; for regular requests, α = 1.0; and for low-priority requests, α = 1.3. The adjusted weight is: W_adjusted = W_base × α. Additionally, to avoid path congestion, a penalty factor is added to heavily used paths. The current traffic percentage P of each path is recorded. When P > 30%, a penalty term is introduced: penalty = (P - 0.3). 2 ×5, W_final = W_adjusted + penalty, edge weights are updated every 1 second to ensure rapid response to network status changes. When a major network topology change occurs (such as link interruption or node failure), a full weight recalculation process is triggered. A weight smoothing factor β = 0.8 can be introduced, and the weights are updated using an exponential smoothing formula: W(t) = β × W(t-1) + (1-β) × W_calculated. This allows for priority selection of paths with fewer hops for urgent requests and for low-priority requests, a bandwidth cost factor is added to prioritize paths with lower bandwidth but lower cost. The calculation results generate a list of multiple alternative paths, including the primary path and at least two backup paths. Each path is labeled with estimated latency, available bandwidth, and reliability metrics.

[0121] Furthermore, based on the result of insufficient resource utilization, the data shards stored on this node are redistributed to other storage nodes according to a preset data migration strategy and load balancing algorithm, which includes the following steps:

[0122] The cluster management center is used to respond to insufficient resource utilization by data migration, thereby generating data migration instructions.

[0123] In this embodiment of the invention, the cluster management center monitors the resource usage of each node in real time through the resource monitoring system. When it is detected that the CPU usage of a certain node is below 20% for 10 consecutive minutes, the memory usage is below 30%, and the IOPS is below 15% of the design peak, the node is determined to be a node with insufficient resource usage. At this time, the system automatically generates a data migration instruction. The instruction includes information such as the source node identifier, resource usage status report, and migration trigger threshold. The instruction is encoded in JSON format and sent to the migration execution module through a secure message queue (such as RabbitMQ).

[0124] Furthermore, based on data migration instructions, data heat analysis and dependency analysis are performed on the data shards stored on nodes with insufficient resource utilization to obtain the access frequency, correlation degree, and resource utilization of other storage nodes of the data shards.

[0125] In this embodiment of the invention, data heat analysis is performed using access records within a sliding time window (the past 72 hours). The access frequency of each data shard is counted as the number of accesses per hour, and the heat value is calculated using the Exponentially Weighted Moving Average (EWMA) algorithm. Dependency analysis is achieved through association rule mining technology, analyzing the co-occurrence frequency between data shards, setting a minimum support of 10% and a minimum confidence of 80%, and identifying strongly correlated data shard groups. Simultaneously, the system collects real-time resource usage data from other storage nodes, including available storage capacity, CPU idle rate, network bandwidth, and other metrics, which are updated every 2 minutes.

[0126] Furthermore, based on the access frequency and correlation of data shards and the resource usage of other storage nodes, the data shards are redistributed to other storage nodes according to the preset data migration strategy and load balancing algorithm, and the data index information of the cluster management center is updated synchronously.

[0127] In this embodiment of the invention, the data migration strategy adopts a two-layer scheduling algorithm based on popularity and load. For cold data with an access frequency below a threshold (e.g., less than 10 times per week), it is preferentially migrated to low-cost storage nodes. For data shard groups with strong correlations, it is ensured that they are migrated to nodes in the same physical rack or adjacent network areas. The load balancing algorithm uses an improved minimum connection number algorithm to dynamically calculate the load pressure of the target node, including factors such as the current number of connections, CPU load, and disk IOPS. The migration process adopts incremental synchronization technology, first copying the snapshot of the data shard, and then transmitting the changed part by comparing the hash value. The update of data index information is guaranteed to be consistent through a two-phase commit protocol, ensuring that data access requests can be correctly routed to the new storage location during the migration process.

[0128] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalents of the application are intended to be included within the invention.

[0129] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features of the invention herein.

Claims

1. A method for distributed data storage and access processing, characterized by: The method comprises the following steps: The system initialization stage, the boot configuration file is preset in each storage node of the distributed storage cluster, and the node identity key and the cluster topology information stored in the boot configuration file are used for identity authentication and communication link establishment between each storage node and the cluster management center, so that a secure data interaction channel between the storage node and the cluster management center is generated; The data storage stage, the data sending end device sends a data storage request to the cluster management center through the secure data interaction channel, the cluster management center processes the data according to a preset data sharding storage strategy based on the data storage request, and distributes the data shards to appropriate storage nodes according to the load, storage capacity and data access frequency of each storage node; The data access stage, the data receiving end device sends a data access request to the cluster management center through the secure data interaction channel, the cluster management center generates an optimal data access path according to the index information of the data and the real-time state of each storage node based on the data access request, and feeds back the data access path to the data receiving end device, and the data receiving end device obtains data from the corresponding storage node according to the data access path; wherein the generation of the optimal data access path comprises the following steps: The cluster management center obtains the storage location information of the data shard from the data index information according to the data identifier required to be accessed in the data access request; The priority of the data access request is evaluated based on the device identifier corresponding to the data receiving end device and the urgency information of the data access; The optimal data access path from the data receiving end device to each data shard storage node is calculated by combining the storage location information of the data shard and the request priority based on the network bandwidth, read-write performance and load of each storage node reported in real time, and through the Dijkstra improved algorithm; The resource dynamic adjustment stage, by monitoring the resource usage of the data storage capacity and read-write performance in each storage node in real time, when the resource usage of a certain storage node reaches a preset threshold, the cluster management center reassigns the data shards stored in the node to other storage nodes according to the preset data migration strategy and load balancing algorithm based on the resource usage result.

2. The method of claim 1, wherein: The boot configuration file specifically contains the communication key between the storage node and the cluster management center, the node unique identifier and the cluster network topology structure information, which is used to activate the secure data interaction communication between the storage node and the cluster management center.

3. The method of claim 2, wherein: The identity authentication and communication link establishment based on the boot configuration file comprises the following steps: The communication key and the node unique identifier between the storage node and the cluster management center in the boot configuration file are read; The identity information between the storage node and the cluster management center is asymmetrically encrypted and authenticated based on the communication key between the storage node and the cluster management center, so as to generate a temporary authentication token between the storage node and the cluster management center; According to the two-way handshake verification mechanism, the corresponding session keys of the storage node and the cluster management center are synchronized based on the temporary authentication token, and the TLS encryption protocol is established according to the corresponding session keys of both sides; According to the TLS encryption protocol, a secure data transmission link is generated, and identity authentication and communication link establishment of the storage node and the cluster management center are completed based on the secure data transmission link, to generate a secure data interaction channel between the storage node and the cluster management center.

4. The method of claim 1, wherein: The data sending end device sends a data storage request to the cluster management center through the secure data interaction channel, including the following steps: The data sending end device generates an original data storage request message containing data element information and storage requirements; The original data storage request message is transmitted by AES-256 secure encryption through the secure data interaction channel to obtain an encrypted data storage request data packet; The encrypted data storage request data packet is sent to the cluster management center through the secure data interaction channel, and the data storage request data packet is decrypted and restored to generate a data storage request.

5. The method of claim 4, wherein: The original data storage request message includes the device identifier corresponding to the data sending end device, the data type, the data size, and the priority requirement information of data storage.

6. The method of claim 5, wherein: Based on the data storage request, the data is processed according to a preset data sharding storage strategy, and the data shards are distributed to appropriate storage nodes, including the following steps: The cluster management center analyzes the data characteristics of the corresponding data type and data size in the data storage request to determine the data shard size; Based on the device identifier corresponding to the data sending end device and the priority requirement information of data storage, the priority of the data storage request is classified; According to the real-time reporting of the load situation, storage capacity and data access heat of each storage node, combined with the data shard size and request priority, the target storage node of each data shard is calculated by a load balancing algorithm; The data is sharded according to the determined data shard size, and the data shards are distributed to the corresponding target storage nodes through the secure data interaction channel, and data index information is established in the cluster management center.

7. The method of claim 1, wherein: The data receiving end device sends a data access request to the cluster management center through the secure data interaction channel, including the following steps: The data receiving end device generates an original data access request message containing data identifier and access requirements; The original data access request message is transmitted by national secret SM4 secure encryption through the secure data interaction channel to obtain an encrypted data access request data packet; The encrypted data access request data packet is sent to the cluster management center through the secure data interaction channel, and the data packet is decrypted and restored to generate a data access request.

8. The method of claim 7, wherein: The original data access request message includes the device identifier corresponding to the data receiving end device, the data identifier required to be accessed, and the urgency information of data access.

9. The distributed data storage and access processing method of claim 1, wherein: Based on the resource usage insufficient result, the data shards stored in the node are redistributed to other storage nodes according to the preset data migration strategy and load balancing algorithm, including the following steps: The cluster management center generates a data migration instruction in response to the resource usage insufficient result; Based on the data migration instruction, data hotness analysis and dependency analysis are performed on the data shards stored in the resource insufficient node to obtain access frequency, correlation degree of the data shards and resource usage of other storage nodes; According to the access frequency, correlation degree of the data shards and resource usage of other storage nodes, the data shards are redistributed to other storage nodes according to a preset data migration strategy and a load balancing algorithm, and data index information of the cluster management center is synchronously updated.

Images