Distributed storage system with infinitely extensible capacity and performance and storage method thereof

Abstract

The invention relates to the technical field of distributed storage, and discloses a distributed storage system with infinitely extensible capacity and performance and a storage method thereof.The distributed storage system comprises a special storage device cluster and further comprises a mobile device node pool composed of mobile devices in a company internal local area network; the dynamic node management and self-adaptive expansion mechanism is located in the data scheduling and management module and is closely integrated with the data scheduling and management module; and a data transmission protocol and an encryption mechanism. Through the synergistic effect of the data caching module and the preloading engine, the system significantly reduces the direct access to the special storage device cluster, and optimizes the data flow path. The data scheduling and management module dynamically adjusts a cache strategy, so that common data can be quickly read from a cache, and meanwhile, a preloading engine loads possibly needed data in advance according to a historical access mode. According to the mechanism, the data access delay is reduced, and the response speed and the overall performance of the system are greatly improved especially in the scene of high-concurrency requests.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and more specifically to a distributed storage system with infinitely scalable capacity and performance and a storage method thereof. Background Art

[0002] With the acceleration of the information process, the data management challenges faced by enterprises are becoming increasingly complex. The generation and storage needs of large-scale data have become the core link in the daily operations of enterprises. Traditional distributed storage systems usually rely on high-performance storage devices, such as Storage Area Network (SAN) and Network Attached Storage (NAS) devices. These devices provide enterprises with high availability, high reliability and high performance data storage solutions through distributed file systems (such as Ceph, GlusterFS) and object storage systems (such as Amazon S3). However, as the scale of data continues to expand, these traditional storage systems have exposed some significant shortcomings when facing complex and changing data access requirements.

[0003] First, traditional distributed storage systems usually adopt a centralized storage architecture, which uses specific dedicated storage nodes to centrally manage and process data. Although this architecture can provide stable performance under high load conditions, its scalability is limited and the hardware cost is high, especially when the amount of enterprise data surges, requiring a large investment in dedicated hardware resources. In addition, this centralized storage architecture is prone to performance bottlenecks when faced with sudden data access peaks, resulting in increased data access delays and reduced system response speeds.

[0004] Secondly, with the rapid development of mobile Internet technology and the popularization of mobile office mode, more and more corporate employees have begun to rely on personal mobile devices (such as smartphones, tablets, and laptops) for daily office work; according to survey data, the frequency of use and data processing capabilities of mobile devices are increasing year by year, and the number and types of heterogeneous devices in the enterprise network environment are also expanding; these devices have certain computing power and storage capacity, and can access the enterprise's network and storage system through the enterprise local area network (LAN); however, existing distributed storage systems often ignore the potential of these heterogeneous mobile devices and do not fully utilize their storage and computing resources.

[0005] In addition, existing distributed storage systems still have shortcomings in optimizing data transmission performance. Traditional distributed storage systems usually rely on consistency protocols (such as Paxos and Raft) and replication mechanisms to ensure data reliability and consistency. However, while these mechanisms improve data reliability, they also increase system complexity and network communication overhead, especially when it comes to cross-regional and multi-data center data synchronization, which can easily lead to transmission delays and bandwidth consumption problems.

[0006] When processing lightweight data operations, traditional systems have low resource utilization efficiency; lightweight data operations usually involve small-scale data read and write requests with high frequency and require quick response; because traditional distributed storage systems are mainly designed for high concurrency and high throughput scenarios, when processing lightweight data, the mechanisms such as data redundancy, error correction and network topology optimization in their designs often lead to waste of resources and fail to fully tap the performance potential of storage devices and networks.

[0007] Based on the above technical background, there is a need for a method that can integrate mobile devices within the enterprise LAN with existing dedicated storage devices to form an efficient and flexible distributed storage network, which utilizes caching and preloading mechanisms, intelligent data scheduling algorithms, and multi-channel data transmission architecture to improve data access speed. Summary of the invention

[0008] In order to overcome the above-mentioned defects of the prior art, the present invention provides a distributed storage system and a storage method thereof with infinitely scalable capacity and performance, so as to solve the problems existing in the above-mentioned background technology.

[0009] The present invention provides the following technical solution: a distributed storage system with unlimited scalability in capacity and performance, including a dedicated storage device cluster, characterized in that: it also includes a mobile device node pool, which is composed of mobile devices in a company's internal local area network;

[0010] The data scheduling and management module is located on the core network node of the enterprise LAN;

[0011] Data caching and preloading mechanism, located inside the mobile device node pool;

[0012] Dual data channels, located inside the mobile device node pool;

[0013] Dynamic node management and adaptive expansion mechanism, located inside the data scheduling and management module, and tightly integrated with it;

[0014] Data transmission protocol and encryption mechanism.

[0015] Preferably, the dedicated storage device cluster includes a storage area network, which connects multiple storage devices and provides block-level data storage. The storage area network is deployed inside the data center and is connected via Fibre Channel or high-speed Ethernet. The dedicated storage device cluster includes a network-attached storage, which provides file storage services through a file-level protocol and supports shared file access within the enterprise. The network-attached storage is connected to a local area network, and multiple client devices access the network-attached storage simultaneously. The dedicated storage device cluster includes an object storage device.

[0016] Preferably, the mobile device node pool includes smart phones and tablet computers, which are connected to the local area network via a wireless network and participate in the storage and access of lightweight data; the mobile device node pool includes laptops and desktop computers, which are connected to the local area network via a wireless network and are allocated large-capacity storage space and data processing performance; the mobile device node pool includes wearable devices and Internet of Things devices.

[0017] Preferably, the data scheduling and management module includes a node management module, a data scheduling module, a load balancing module, and a security control module.

[0018] Preferably, the data caching and preloading mechanism includes a local caching module, which sets up a local cache area on each mobile device to store data; the data caching and preloading mechanism includes an intelligent preloading module, which loads frequently accessed data blocks to related mobile device nodes in advance.

[0019] Preferably, the dual data channels include a high-value data channel, which uses high-speed Ethernet and encrypted transmission protocols; the dual data channels include a lightweight data channel, which is connected via Wi-Fi or a wired network and can flexibly handle various data operation requirements.

[0020] Preferably, the dynamic node management and adaptive expansion mechanism includes real-time monitoring and automatic identification; dynamic adjustment and fault handling and adaptive expansion strategy.

[0021] Preferably, in the data transmission protocol and encryption mechanism, the UDP lightweight transmission protocol is adopted to reduce transmission overhead, and an efficient transport layer protocol is used during the data transmission process to ensure the integrity and transmission quality of the data packet; in the data transmission protocol and encryption mechanism, the SSL encryption protocol is adopted, and a lightweight encryption and verification mechanism is adopted.

[0022] How to use a distributed storage system with unlimited scalability in capacity and performance:

[0023] S1: External users or internal components of the system initiate data requests, and the data scheduling and management module initially processes them;

[0024] S2: The data scheduling and management module selects one or two appropriate dual data channels J for data transmission. After the data arrives at the dedicated storage device cluster, the system will shard and redundancy process the data and write it to the corresponding storage medium;

[0025] S3: Dynamic node management and adaptive expansion mechanism continuously monitors the status of the storage device cluster. Based on the monitoring feedback, the system automatically adjusts the node load.

[0026] S4: When the system receives a data read request, the data scheduling and management module will read the data according to the priority of the request, the data location and the network status;

[0027] S5: The system regularly analyzes the performance data of storage nodes and transmission channels, and optimizes the scheduling strategy based on this analysis.

[0028] Technical effects and advantages of the present invention:

[0029] 1. Through the synergy of the data cache module and the preloading engine, the system significantly reduces direct access to the dedicated storage device cluster and optimizes the data flow path. The data scheduling and management module dynamically adjusts the cache strategy so that commonly used data can be quickly read from the cache. At the same time, the preloading engine loads the data that may be needed in advance according to the historical access pattern. This mechanism reduces data access latency, especially in the scenario of high concurrent requests, greatly improving the response speed and overall performance of the system.

[0030] 2. Through the application of dynamic node management and adaptive expansion mechanism, the system can automatically increase or decrease nodes according to the real-time load situation, and dynamically adjust the cache strategy in cooperation with the cache manager to ensure the optimal configuration of system resources. When the load increases, the newly added nodes are quickly integrated into the system, and the preloading engine ensures that these nodes can obtain the necessary data in advance. This mechanism ensures that the system can still maintain efficient and stable operation in the face of load fluctuations, providing a high degree of flexibility and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a diagram of the specific implementation principle steps of the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. In addition, the forms of the various structures recorded in the following implementation modes are merely illustrative. The distributed storage system with infinitely expandable capacity and performance and the storage method thereof involved in the present invention are not limited to the various structures recorded in the following implementation modes. All other implementation modes obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0033] Reference Figure 1 The present invention provides a distributed storage system with infinitely expandable capacity and performance, including a dedicated storage device cluster, characterized in that it also includes a mobile device node pool, which is composed of mobile devices in the company's internal local area network; a data scheduling and management module, which is located on the core network node of the enterprise local area network; a data cache and preloading mechanism, which is located inside the mobile device node pool; a dual data channel, which is located inside the mobile device node pool; a dynamic node management and adaptive expansion mechanism, which is located inside the data scheduling and management module and is tightly integrated with it; and a data transmission protocol and encryption mechanism.

[0034] Specifically, the coordination of various components places more emphasis on dynamic and intelligent management, especially in data scheduling, node management and security protection. Compared with traditional distributed storage systems, this system can achieve more efficient resource utilization, more flexible expansion capabilities and more comprehensive security protection; for example, dynamic node management and adaptive expansion mechanisms can automatically adjust resource configuration according to real-time load, while the combination of dual data channels and edge devices further enhances the efficiency and security of data transmission; these characteristics enable the system to provide higher performance and reliability when facing complex application scenarios and changing needs.

[0035] Reference Figure 1 , the dedicated storage device cluster includes a storage area network, which connects multiple storage devices and provides block-level data storage. The storage area network is deployed inside the data center and is connected via Fibre Channel or high-speed Ethernet; the dedicated storage device cluster includes network-attached storage, which provides file storage services through file-level protocols and supports shared file access within the enterprise. The network-attached storage is connected to the local area network, and multiple client devices access the network-attached storage at the same time; the dedicated storage device cluster includes object storage devices.

[0036] Specifically, the dedicated storage device cluster consists of the following modules:

[0037] 1. Storage server: contains multiple storage media (such as SSD, HDD) for storing data;

[0038] 2. Data sharding and redundancy module: responsible for sharding data and generating redundancy to ensure data security and reliability;

[0039] 3. Cache module: used to speed up data reading and writing operations and reduce latency.

[0040] The operation principle of a dedicated storage device cluster is:

[0041] 1. Data writing: Receive write requests from the data scheduling and management module and store data slices in different storage media.

[0042] 2. Data reading: According to the read request, data is extracted from the corresponding storage medium, and the read operation is accelerated through the cache module.

[0043] 3. Data backup: Generate data backup regularly or according to policies and store them in different physical devices.

[0044] The functions of a dedicated storage device cluster are:

[0045] 1. Data storage: Provide high-capacity, low-latency data storage services.

[0046] 2. Data redundancy and recovery: Data redundancy improves data reliability and enables rapid data recovery when a storage device fails.

[0047] 3. Efficient data access: Optimize data read and write operations and support fast data retrieval and update.

[0048] Reference Figure 1 ,The mobile device node pool includes smart phones and tablets, which are connected to the LAN through a wireless network, and participate in the storage and access of lightweight data; ,the mobile device node pool includes laptops and desktop computers, which are connected to the LAN through a wireless network, and allocate large storage space and data processing performance; ,the mobile device node pool includes wearable devices and IoT devices.

[0049] Specifically, the mobile device node pool consists of the following modules:

[0050] 1. Mobile storage node: supports access to portable storage devices and provides flexible storage expansion.

[0051] 2. Computing nodes: include edge computing capabilities and support real-time data processing.

[0052] 3. Connection management module: manages the access, disconnection and connection status of devices.

[0053] The operating principle of the mobile device node pool is:

[0054] 1. Device access: The mobile device is connected to the system through a wireless or wired network and becomes a part of the system.

[0055] 2. Data exchange: The device can obtain data from or write data to a dedicated storage device cluster, or process data locally.

[0056] 3. Device disconnection: When the device completes its task or no longer needs to participate in data processing, it is safely disconnected.

[0057] The functions of the mobile device node pool are:

[0058] 1. Flexible storage expansion: supports flexible expansion of system storage and computing capabilities.

[0059] 2. Edge computing support: provides real-time data processing capabilities, suitable for application scenarios with low latency requirements.

[0060] 3. Mobility and convenience: The equipment is easy to access and remove, which improves the flexibility and scalability of the system.

[0061] Reference Figure 1 The data scheduling and management module includes a node management module, a data scheduling module, a load balancing module, and a security control module.

[0062] Specifically, the data scheduling and management module consists of the following modules:

[0063] 1. Scheduling engine: The core component responsible for determining data flow direction, data priority, and transmission path.

[0064] 2. Load balancer: used to monitor and allocate system resources to ensure load balance among nodes.

[0065] 3. Policy Manager: Dynamically adjust data scheduling policies based on preset policies or real-time analysis.

[0066] The operating principle of the data scheduling and management module is:

[0067] 1. Data reception: receiving data operation requests from user requests or internal systems.

[0068] 2. Policy decision: Determine the data transmission path and storage location based on the current system status and policy (such as minimum latency, maximum bandwidth utilization).

[0069] 3. Task allocation: Assign tasks such as data writing and reading to appropriate storage nodes or computing nodes.

[0070] The functions of the data scheduling and management module are:

[0071] 1. Data scheduling: manage the flow of data to ensure efficient transmission and storage of data within the system.

[0072] 2. Load balancing: prevent certain nodes in the system from being overloaded and optimize resource utilization efficiency.

[0073] 3. System monitoring and optimization: monitor system status in real time and adjust data flow and resource allocation as needed.

[0074] Reference Figure 1,The data caching and preloading mechanism includes a local caching module, which sets up a local cache area on each mobile device to store data; ,The data caching and preloading mechanism includes an intelligent preloading module, which loads frequently accessed data blocks to the relevant mobile device nodes in advance.

[0075] Specifically, the data caching and preloading mechanism consists of the following modules:

[0076] 1. Cache module: stores temporary data to speed up access to frequently used data.

[0077] 2. Preload engine: Predictively load data that may be accessed into the cache.

[0078] 3. Cache Manager: responsible for cache allocation, replacement strategy and data consistency management.

[0079] 4. Data access log: records data access history for optimizing caching and preloading strategies.

[0080] The operating principle of the data caching and preloading mechanism is:

[0081] 1. Data caching: When data is read from a storage device, it is first stored in the cache module; if the same data is requested again, the system will read the data directly from the cache without having to re-access the slower storage device. This greatly reduces read latency.

[0082] 2. Preloading mechanism: The preloading engine uses data access logs to analyze historical data access patterns and predict data that may be needed in the future, and loads this data into the cache in advance; these predictions are usually based on machine learning algorithms or access frequency statistics, such as preloading commonly used files or data blocks during peak hours.

[0083] 3. Cache management: The cache manager uses a variety of strategies to manage cache space, such as the least recently used (LRU) algorithm or adaptive caching strategy; the cache manager is also responsible for ensuring that the data in the cache is consistent with the data in the storage device, especially updating the cache in a timely manner when the data changes.

[0084] The functions of the data caching and preloading mechanism are:

[0085] 1. Accelerate data access: The cache mechanism significantly reduces the time for data reading and improves system response speed, especially in scenarios with frequently accessed data.

[0086] 2. Optimize system performance: The preloading mechanism reduces data access latency by predicting data demand and loading data in advance. This can significantly improve overall system performance, especially when processing a large number of concurrent requests.

[0087] 3. Save resources: By reducing the number of direct accesses to storage devices, the cache mechanism helps reduce the load and power consumption of storage devices, thereby extending the life of the device and reducing operating costs.

[0088] Reference Figure 1 The dual data channels include a high-value data channel, which uses high-speed Ethernet and encrypted transmission protocols; the dual data channels include a lightweight data channel, which is connected via Wi-Fi or wired network and can flexibly handle various data operation needs.

[0089] Specifically, the dual data channel consists of the following modules:

[0090] 1. High-value data channel: mainly used to transmit important data, ensuring high bandwidth and low latency.

[0091] 2. Lightweight data channel: used to transmit secondary or non-critical data and optimize network resource utilization.

[0092] The operating principle of dual data channels is:

[0093] 1. Channel selection: Choose to use high-value or lightweight data channels based on data importance, priority, and network conditions.

[0094] 2. Data transmission: Transmit data through the selected channel, monitor channel performance in real time and make dynamic adjustments.

[0095] 3. Load balancing: Dynamically adjust data traffic between two channels to avoid channel overload.

[0096] The functions of the dual data channels are:

[0097] 1. Optimize data transmission: Optimize data transmission speed and reliability through dual-channel design.

[0098] 2. Load balancing: Balance the load between different channels to improve the overall network resource utilization.

[0099] 3. Enhanced fault tolerance: When a channel fails, data can be transmitted through another channel, enhancing the system's fault tolerance.

[0100] Reference Figure 1 ,Dynamic node management and adaptive expansion mechanism include real-time monitoring and automatic identification; including dynamic adjustment and fault handling and adaptive expansion strategy.

[0101] Specifically, the dynamic node management and adaptive expansion mechanism consists of the following modules:

[0102] 1. Node monitoring module: real-time monitoring of the status and performance of each node in the system.

[0103] 2. Resource management module: dynamically adjust node resource configuration and load distribution.

[0104] 3. Automatic expansion module: automatically manages the increase and decrease of nodes and supports adaptive expansion of the system.

[0105] The operating principle of dynamic node management and adaptive expansion mechanism is:

[0106] 1. Node monitoring: Real-time monitoring of the status of each node, such as CPU, memory usage, network status, etc.

[0107] 2. Resource adjustment: Dynamically adjust node resource allocation based on monitoring data to balance system load.

[0108] 3. Automatic expansion: When the system load increases, node expansion is automatically triggered to add new storage or computing nodes.

[0109] The functions of dynamic node management and adaptive expansion mechanism are:

[0110] 1. Automated management: Reduce manual intervention and realize automated management and optimization of the system.

[0111] 2. Elastic expansion: Dynamically expand or reduce system resources according to demand to ensure high availability and flexibility of the system.

[0112] 3. Fault recovery: When a node fails, tasks are automatically reallocated to ensure stable operation of the system.

[0113] Reference Figure 1 In the data transmission protocol and encryption mechanism, the UDP lightweight transmission protocol is used to reduce transmission overhead, and an efficient transport layer protocol is used during the data transmission process to ensure the integrity and transmission quality of the data packet; in the data transmission protocol and encryption mechanism, the SSL encryption protocol is used, and a lightweight encryption and verification mechanism is adopted.

[0114] Specifically, the data transmission protocol and encryption mechanism consists of the following modules:

[0115] 1.Transmission protocol stack: supports multiple data transmission protocols (such as TCP / IP, UDP).

[0116] 2. Encryption module: supports multiple encryption algorithms and protocols (such as TLS, IPSec).

[0117] 3. Authentication and key management: Manage authentication and key distribution during data transmission.

[0118] The operating principle of the data transmission protocol and encryption mechanism is:

[0119] 1. Protocol selection: Select the appropriate transmission protocol based on the data type, transmission path, and security requirements.

[0120] 2. Data encryption: Before data transmission, the encryption module is used to encrypt the data to protect the confidentiality of the data.

[0121] 3. Data transmission: The encrypted data is transmitted to the target node through the selected transmission protocol.

[0122] 4. Data decryption: Decrypt the data at the target node and process it according to the request.

[0123] The functions of the data transmission protocol and encryption mechanism are:

[0124] 1. Secure data transmission: Ensure the confidentiality and integrity of data during transmission to prevent eavesdropping and tampering.

[0125] 2. Multi-protocol support: flexibly select transmission protocols according to needs to improve the adaptability and performance of the system.

[0126] 3. Encryption protection: protect the security of data during transmission through encryption mechanism.

[0127] When the above components complete their corresponding functions, they mainly include the following functions and the specific coordination and cooperation between them:

[0128] 1. Data flow and access optimization

[0129] In the entire system, the data cache module and preloading engine work closely with the data scheduling and management module to optimize the data flow path and access efficiency. When a request arrives at the system, the data scheduling and management module will first check whether the cache module has cached the data. If the data exists in the cache, it will read the data directly from the cache, reducing the access delay of the storage device.

[0130] At the same time, the preloading engine predicts the data that may be needed in the future by analyzing historical access patterns, and loads this data into the cache in advance. This preloading not only improves the access speed of data, but also reduces the access pressure of storage devices. The data scheduling and management module uses the information of these cached and preloaded data to optimize the data transmission path and further improve the overall performance of the system.

[0131] 2. Dynamic management and expansion of resources

[0132] The dynamic node management and adaptive expansion mechanism in the system automatically adjusts the number of nodes and resource configuration by monitoring the system load in real time. When the system load increases, new nodes will be automatically added. These new nodes will be quickly adjusted according to the current cache strategy and data preloading status to ensure that the new nodes can quickly integrate into the system and take on the load. At the same time, the cache manager will also adjust the cache allocation strategy accordingly to ensure that the system's cache hit rate remains at a high level, thereby maintaining the overall system performance.

[0133] 3. Data transmission efficiency and security

[0134] The dual data channel design allows the system to select the optimal data transmission channel based on the priority and importance of the data. The cache module and preloading engine give priority to high-speed channels when transmitting important preloaded data to ensure the timeliness and stability of the data. For general data, the system can select a suboptimal channel to optimize the overall utilization of network resources.

[0135] In addition, dual data channels can also work with edge devices and firewalls to ensure the security of data during transmission. The firewall monitors all cached and preloaded data passing through to prevent unauthorized data access and potential security threats. This security mechanism not only ensures data transmission efficiency, but also improves the overall security of the system.

[0136] 4. Edge computing and distributed processing

[0137] Edge devices play an important role in the system, especially in reducing latency and improving local data processing capabilities. The cache and preloading mechanism implements local storage and data processing on edge devices, significantly reducing the frequency of remote data access and network bandwidth usage. When there is a user request, the edge device first obtains data from the local cache without directly accessing the central storage device.

[0138] This distributed processing method not only improves the efficiency of data access, but also improves the stability of the entire system by reducing the load on the central device. In this environment, the mobile device node pool further enhances the effectiveness of edge computing. Through dynamic management, mobile devices can intelligently preload data based on user habits and network status to ensure a smooth user experience even when offline or the network is unstable.

[0139] 5. Overall coordination and optimization

[0140] All these mechanisms and modules are interdependent and work closely together to form a highly coordinated system. For example, the data scheduling and management module constantly interacts with the cache manager and preloading engine to adjust the data flow strategy and cache strategy. The dynamic node management mechanism cooperates with the cache module to ensure that the system can dynamically adjust according to load changes and maintain high performance. Edge devices, dual data channels, and firewalls work together to ensure efficient and secure data transmission.

[0141] During use, through the synergy of the data cache module and the preloading engine, the system significantly reduces direct access to the dedicated storage device cluster and optimizes the data flow path. The data scheduling and management module dynamically adjusts the cache strategy so that commonly used data can be quickly read from the cache. At the same time, the preloading engine loads the data that may be needed in advance based on historical access patterns. This mechanism reduces data access latency, especially in scenarios with high concurrent requests, greatly improving the system's response speed and overall performance.

[0142] Through the application of dynamic node management and adaptive expansion mechanism, the system can automatically increase or decrease nodes according to the real-time load situation, and dynamically adjust the cache strategy with the cache manager to ensure the optimal configuration of system resources. When the load increases, the new nodes are quickly integrated into the system, and the preloading engine ensures that these nodes can obtain the necessary data in advance. This mechanism ensures that the system can maintain efficient and stable operation in the face of load fluctuations, providing a high degree of elasticity and scalability.

[0143] Through the cooperation of dual data channels and edge devices, the data transmission path is optimized, and the firewall provides security during the transmission process. Critical data is transmitted through high-speed channels first, while secondary data is transmitted through suboptimal channels to ensure efficient use of resources. Edge devices process data locally, reducing dependence on central storage, while firewalls monitor data flow to prevent unauthorized access. This mechanism not only improves data transmission efficiency, but also enhances the overall security of the system.

[0144] Working principle of the present invention:

[0145] S1: Data sources and preliminary processing

[0146] a. Trigger event: An external user or internal component of the system initiates a data request (such as a data write or read request).

[0147] b. Preliminary processing: Data is first collected or generated by the mobile device node pool or other terminal devices, and is preliminarily processed by the data scheduling and management module, such as verifying the data format and content.

[0148] S2: Selection of transmission protocol and encryption mechanism

[0149] a. Protocol selection: The data scheduling and management module selects the appropriate transmission protocol (such as TCP / IP, UDP) based on the data type and destination.

[0150] b. Encryption processing: To ensure data security, select an appropriate encryption mechanism (such as TLS, IPSec) to encrypt data before transmission.

[0151] S3: Data transfer to storage device

[0152] a. Data transmission initiation: The data scheduling and management module selects one or two appropriate dual data channels and transmits data to the dedicated storage device cluster based on load and priority.

[0153] b. Channel optimization: During data transmission, the system monitors the performance of the two channels in real time. If bottlenecks or failures are encountered, the transmission strategy is dynamically adjusted to ensure that data arrives safely and quickly.

[0154] S4: Data is written to the storage cluster

[0155] a. Storage write: After the data reaches the dedicated storage device cluster, the system will shard and process the data redundantly, and write it to the corresponding storage medium (such as SSD or HDD).

[0156] b. Data backup: The system automatically generates backups based on policies and stores the backup data on different physical devices to improve data reliability.

[0157] S5: Real-time monitoring and node management

[0158] a. Monitoring and feedback: Dynamic node management and adaptive expansion mechanisms continuously monitor the status of storage device clusters, such as storage capacity, read and write performance, and device health.

[0159] b. Dynamic adjustment: Based on monitoring feedback, the system automatically adjusts the node load. If a node is overloaded or fails, the system will reallocate data tasks to other available nodes or trigger an adaptive expansion mechanism to add new nodes.

[0160] S6: Adaptive expansion

[0161] a. Expansion trigger: When the system detects that the load continues to increase over a period of time, it automatically expands the storage or computing nodes to ensure that the system continues to meet demand.

[0162] b. Extended execution: After a new node is added, the system automatically configures it and incorporates it into the existing data scheduling and management system without manual intervention.

[0163] S7: Data read request processing

[0164] a. Request reception: When the system receives a data read request, the data scheduling and management module will select the best storage node and transmission path based on the request priority, data location and network conditions.

[0165] b. Protocol and encryption reuse: When transmitting data back to users or applications, the transmission protocol and encryption mechanism are enabled again to ensure data security during the transmission process.

[0166] S8: Edge devices and firewall protection

[0167] a. Edge protection: When data is transmitted across network boundaries, edge devices and firewalls protect the data through encrypted channels (such as VPN) and perform protocol filtering to prevent unauthorized access.

[0168] b. Data integrity verification: Before the data reaches the terminal, the system uses hash verification and other methods to ensure that the data has not been tampered with.

[0169] S9: System self-optimization

[0170] a. Performance analysis: The system regularly analyzes the performance data of storage nodes and transmission channels, and optimizes the scheduling strategy based on this analysis to further improve the efficiency of data transmission and storage.

[0171] b. Resource recovery and reuse: For nodes that are no longer active, the system automatically recycles their resources and configures them for new tasks or storage needs.

[0172] S10: Security updates and policy adjustments

[0173] a. Firewall rule update: Automatically update firewall rules based on the latest security threat intelligence to protect the system from new attacks.

[0174] b. Update of encryption mechanism: Update the data encryption mechanism regularly according to changes in industry standards to ensure the long-term security of data transmission and storage.

[0175] Finally, a few points should be explained: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, which may refer to mechanical connection or electrical connection, or internal communication between two components, or direct connection. "upper", "lower", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may change;

[0176] Secondly: In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;

[0177] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed storage system with unlimited scalability in capacity and performance, comprising a dedicated storage device cluster, characterized in that: It also includes a mobile device node pool, which consists of mobile devices in the company's internal LAN; The data scheduling and management module is located on the core network node of the enterprise LAN; Data caching and preloading mechanism, located inside the mobile device node pool; Dual data channels, located inside the mobile device node pool; Dynamic node management and adaptive expansion mechanism, located inside the data scheduling and management module, and tightly integrated with it; Data transmission protocol and encryption mechanism.

2. The distributed storage system with infinitely scalable capacity and performance according to claim 1, characterized in that: The dedicated storage device cluster includes a storage area network, which connects multiple storage devices and provides block-level data storage. The storage area network is deployed inside the data center and connected via Fibre Channel or high-speed Ethernet. The dedicated storage device cluster includes network-attached storage, which provides file storage services through file-level protocols and supports shared file access within the enterprise. The network-attached storage is connected to the local area network, and multiple client devices access the network-attached storage at the same time. The dedicated storage device cluster includes object storage devices.

3. The distributed storage system with infinitely scalable capacity and performance according to claim 2, characterized in that: The mobile device node pool includes smart phones and tablet computers, which are connected to the local area network through a wireless network and participate in the storage and access of lightweight data; the mobile device node pool includes laptops and desktop computers, which are connected to the local area network through a wireless network and are allocated large-capacity storage space and data processing performance; the mobile device node pool includes wearable devices and Internet of Things devices.

4. The distributed storage system with infinitely scalable capacity and performance according to claim 3, characterized in that: The data scheduling and management module includes a node management module, a data scheduling module, a load balancing module, and a security control module.

5. The distributed storage system with infinitely scalable capacity and performance according to claim 4, characterized in that: The data cache and preloading mechanism includes a local cache module, which sets up a local cache area on each mobile device to store data; the data cache and preloading mechanism includes an intelligent preloading module, which loads frequently accessed data blocks to related mobile device nodes in advance.

6. The distributed storage system with infinitely scalable capacity and performance according to claim 5, characterized in that: The dual data channels include a high-value data channel, which uses high-speed Ethernet and encrypted transmission protocols; the dual data channels include a lightweight data channel, which is connected via Wi-Fi or a wired network and can flexibly handle various data operation requirements.

7. The distributed storage system with infinitely scalable capacity and performance according to claim 6, characterized in that: The dynamic node management and adaptive expansion mechanism includes real-time monitoring and automatic identification; Includes dynamic adjustment and fault handling and adaptive expansion strategies.

8. The distributed storage system with infinitely scalable capacity and performance according to claim 7, characterized in that: In the data transmission protocol and encryption mechanism, the UDP lightweight transmission protocol is used to reduce transmission overhead, and an efficient transport layer protocol is used during the data transmission process to ensure the integrity and transmission quality of the data packet; In the data transmission protocol and encryption mechanism, the SSL encryption protocol is adopted, and a lightweight encryption and verification mechanism is adopted.

9. The method for using the distributed storage system with infinitely scalable capacity and performance according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1: External users or internal components of the system initiate data requests, and the data scheduling and management module initially processes them; S2: The data scheduling and management module selects one or two appropriate dual data channels J for data transmission. After the data arrives at the dedicated storage device cluster, the system will shard and redundancy process the data and write it to the corresponding storage medium; S3: Dynamic node management and adaptive expansion mechanism continuously monitors the status of the storage device cluster. Based on the monitoring feedback, the system automatically adjusts the node load. S4: When the system receives a data read request, the data scheduling and management module will read the data according to the priority of the request, the data location and the network status; S5: The system regularly analyzes the performance data of storage nodes and transmission channels, and optimizes the scheduling strategy based on this analysis.

Images