Dynamic switching distributed file storage system based on elastic data network

Abstract

The invention discloses a dynamic switching distributed file storage system based on an elastic data network, and the system comprises a multi-network architecture module which is used for deploying a plurality of independent network channels, and each network channel independently transmits data; the client elastic data network access module is used for accessing and storing data according to the data network and the priority, and automatically switching and recovering when the network fails; the intelligent data transmission module is used for selecting an optimal data transmission path; the data recovery module is used for calling the intelligent data transmission module and selecting an optimal path according to the network use condition; the network fault automatic detection and recovery module is used for detecting and processing network channel or path abnormity, detecting master and salve when a fault occurs, acquiring network health state data, and switching master and slave according to the network health state data; the data security processing module is used for encryption and verification; and the cache optimization module is used for collecting data access frequency data and heat dynamic data.

Description

Technical Field

[0001] The present invention relates to the field of distributed file storage systems, and in particular to a dynamically switched distributed file storage system based on an elastic data network. Background Art

[0002] With the rapid development of information technology, the amount of data has exploded, and distributed file storage systems have emerged. It plays a key role in many fields such as cloud computing, big data analysis, and enterprise data management. It can store and manage massive amounts of data, support multi-user concurrent access, and provide high availability and scalability services. The needs of different industries for distributed file storage systems are becoming increasingly diverse. For example, Internet companies need to process a large amount of user data, financial institutions require high security and stability, and the scientific research field focuses on efficient data processing and analysis capabilities.

[0003] At present, the market for distributed file storage systems is highly competitive, with a variety of technologies and products coexisting. Some systems perform well in specific scenarios, but there are some common problems. Many systems lack flexibility when dealing with large-scale data and complex business scenarios, and it is difficult to quickly adapt to changes in demand. The performance optimization of some systems is limited, and delays and freezes are prone to occur when high concurrency and large data volumes are transmitted. The compatibility and interoperability between different systems also need to be improved, which brings difficulties to enterprise integration and data sharing.

[0004] Existing distributed file storage systems usually build a basic network architecture first, mostly in a single network mode, connecting storage nodes and servers through the network. When storing data, files are divided and distributed on multiple nodes according to preset rules, and metadata management nodes are responsible for recording information such as file location. When users access data, they make network requests, search for metadata on the server to determine the data location, and then obtain data from the corresponding node and return it to the user. In this process, the network configuration is relatively fixed, the data transmission path selection lacks flexibility, node load balancing relies on simple rules, and data security measures are relatively simple.

[0005] In terms of resource utilization, the existing technology's single network architecture is prone to bandwidth bottlenecks and cannot dynamically allocate resources, resulting in resource waste or overload. In terms of system reliability, there is a lack of effective fault detection and dynamic switching mechanisms, and single-point failures may cause service interruptions. In terms of data transmission performance, path selection is not intelligent enough, and the multi-network collaborative transmission capability is weak, affecting the transmission speed. Data security protection is insufficient, encryption and verification methods are limited, and it is vulnerable to attacks and data leakage. Cache management is inflexible and difficult to dynamically optimize based on data access. Load balancing is not effective, and uneven node load affects overall performance. Data lifecycle management is extensive, lacking fine classification and policy execution, affecting data storage efficiency and compliance. Summary of the invention

[0006] The present application provides a dynamically switching distributed file storage system based on an elastic data network, which is used to solve the technical problems of insufficient scalability and reliability and cluster bandwidth performance bottleneck in the prior art.

[0007] In view of the above problems, the present application provides a dynamically switching distributed file storage system based on an elastic data network, including: a multi-network architecture module, used to deploy multiple independent network channels, each network channel independently transmits data; a client elastic data network access module, used to access and store data according to the data network and priority, and automatically switch and recover when the network fails; an intelligent data transmission module, used to select the optimal data transmission path; a data recovery module, used to call the intelligent data transmission module, and select the best path according to network usage; an automatic detection and recovery network fault module, used to detect and process network channel or path abnormalities, detect master and salve in case of failure, obtain network health status data, and switch master and slave according to the network health status data; a data security processing module, used for encryption and verification; a cache optimization module, used to collect data access frequency data and heat dynamic data, and adjust the cache strategy according to the access frequency data and heat dynamic data; a load balancing module, which collects node load parameters and allocates data traffic according to the node load parameters; a data storage module, used to set data life cycle parameters, and execute corresponding storage and deletion strategies for corresponding data according to the data life cycle parameters.

[0008] In summary, the present invention mainly has the following beneficial effects:

[0009] 1. The present invention constructs multi-channels through multi-network architecture modules, and intelligent data transmission and other modules select routes according to network conditions, so as to realize multi-channel transmission and fully utilize bandwidth, dynamically adjust configuration, avoid resource waste, improve resource utilization, and make the system run efficiently.

[0010] 2. The present invention monitors faults through an automatic detection and recovery network fault module, detects node status switching between master and slave when a fault occurs, selects a backup link when a link fails, and migrates services when a node fails, thereby reducing the risk of single point failures and enhancing system reliability.

[0011] 3. The present invention uses an intelligent data transmission module to select the optimal path according to the file type and network, fully utilizes the network performance module to transmit data over multiple networks, selects a low-latency path for real-time files, and transmits over multiple networks simultaneously to increase the speed and ensure efficient transmission.

[0012] 4. The present invention encrypts data, verifies keys and ensures storage security through a data security processing module, implements AES encryption, regularly changes keys, verifies hash values, etc., prevents data leakage and tampering, and ensures data security and integrity.

[0013] 5. The present invention collects data access information through a cache optimization module to adjust the cache, eliminates data blocks according to heat values, monitors indicators such as hit rates, and adjusts strategies to improve cache utilization and accelerate data access.

[0014] 6. The present invention collects node load parameters and distributes traffic through a load balancing module, calculates weighted distribution requests based on CPU usage, avoids node overload, and improves the overall performance and stability of the system.

[0015] 7. The present invention sets parameters to manage data through the data storage module to classify and mark data according to importance, store or delete according to the policy, release space, and ensure orderly management and compliance of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A schematic diagram of cluster client elastic data network access of the present invention;

[0017] Figure 2 It is a schematic diagram of the elastic switching of the present invention. DETAILED DESCRIPTION

[0018] The present application provides a dynamically switching distributed file storage system based on an elastic data network, which is used to solve the technical problems of insufficient scalability and reliability and cluster bandwidth performance bottleneck in the prior art.

[0019] After introducing the basic principles of the present application, the technical solutions in the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described here. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0020] Embodiment 1

[0021] A dynamically switched distributed file storage system based on an elastic data network, characterized by comprising:

[0022] Multi-network architecture module, used to deploy multiple independent network channels, each of which transmits data independently;

[0023] It is mainly responsible for building the network infrastructure of the distributed file storage system, and realizing efficient and reliable data transmission through reasonable planning and configuration of multiple independent network channels. It contains the following submodules:

[0024] The network topology planning unit is used to plan the network topology structure according to the system scale and performance requirements, and obtain the number and layout parameters of independent network channels;

[0025] For large-scale enterprise-level distributed file storage systems with large amounts of data being read and written concurrently, high availability and high performance are required. They may be planned as a redundant star topology with multiple independent network channels. The core nodes are connected to multiple edge nodes through high-speed links to ensure the reliability and efficiency of data transmission.

[0026] A network device configuration unit, configured to configure a dedicated network device for each network channel according to the number and layout parameters of the independent network channels;

[0027] A network status acquisition unit is used to install a network sensor on each network channel according to the number and layout parameters of the independent network channels, collect real-time data of network delay, packet loss rate, bandwidth utilization, and network traffic, and the sensor collects and packages the sensor data set and transmits it to the management server;

[0028] The network management unit is used to configure the network management software according to the number of independent network channels and layout parameters, receive sensor data sets, display network status, and provide network configuration and fault diagnosis data.

[0029] The multi-network architecture module is the network foundation of the entire distributed file storage system. Its core function is to build and manage multiple independent network channels to ensure efficient data transmission in different network environments. Through reasonable planning of the network topology, the number and layout parameters of independent network channels are determined according to the system scale and performance requirements. Each network channel is equipped with a dedicated network device with redundant hardware, and network sensors are installed to collect real-time data and transmit it to the management server.

[0030] Finally, the network management software is used to display the network status and provide network configuration and fault diagnosis functions. The administrator uses the network management software to monitor the status of each channel in real time, adjust the network configuration according to the bandwidth utilization, and ensure the stable operation of the system.

[0031] The client elastic data network access module is used to access and store data according to the data network and priority, and automatically switch and recover in case of network failure;

[0032] The client elastic data network access module automatically switches and recovers when the network fails or data requirements change to ensure data continuity and availability. Specifically, the client elastic data network access module can select the optimal network channel for data transmission based on the real-time and integrity requirements of the data. When a network failure occurs, it can quickly detect the failure and automatically switch to other available network channels to ensure continuous data transmission. At the same time, the module also has the master-slave switching coordination capability, which can quickly update the connection information and switch to the new master node when the cluster module switches between master and slave to ensure that data access and storage are not affected.

[0033] The working principle of the client elastic data network access module is mainly based on the following core components and processes:

[0034] The network environment detection unit is used to detect the local network environment, obtain the list of available network interfaces, send ICMP Echo requests to the default gateway to test connectivity, record network delay and packet loss rate, and establish an initial connection with the cluster server to obtain cluster network information, including the list of supported data networks and their priorities;

[0035] The first connection unit is used by the client to try to connect according to the cluster network priority. When connecting, it sends a request containing its own network information and data transmission requirements to the server. The server selects the best path according to the internal network status of the cluster and returns it. The client establishes a connection accordingly. If it fails, the information is recorded and the fault processing is started;

[0036] The connection monitoring unit is used for the client to continuously monitor the network status during data transmission, periodically send detection packets to obtain real-time information, and calculate the real-time evaluation value through the real-time evaluation value calculation formula. If the network change does not meet the requirements, the transmission is suspended and the server is requested to re-select the route. The server recalculates the selection evaluation value and adopts the path with the maximum selection evaluation value. The client switches the path to continue transmission;

[0037] The fault handling unit is used to mark the faulty network, cache data, select a secondary priority network and notify the server when the client detects a network fault. The server detects the internal fault of the cluster and recovers. The client waits and tries to reconnect. If it fails, it tries other networks in order of priority. If all are unavailable, it is handled according to the preset strategy.

[0038] The master-slave switching coordination unit is used to receive the new master broadcast notification, update the connection information, switch to the new master, reconnect and resume the request when the cluster module switches from master to slave. If a connection failure occurs, it is handled according to the error handling strategy;

[0039] The real-time evaluation value calculation formula is as follows:

[0040]

[0041] Among them, R is the real-time evaluation value, B is the available bandwidth of the path, which means the bandwidth available for data transmission on the network path; D is the network delay; L is the packet loss rate;

[0042] Among them, the calculation formula for selecting the evaluation value is as follows:

[0043] P=(α*BD*1000-L*10000)*E

[0044] Among them, α is the network stability weight, which ranges from 0 to 1 according to the preset control value, indicating the stability of the network path. The more stable it is, the higher the weight. B is the available bandwidth of the path, indicating the bandwidth available for data transmission on the network path. D is the network delay. L is the packet loss rate. E is the urgency of data recovery, ranging from 1 to 10.

[0045] For example, in a large enterprise-level distributed file storage system, the client needs to transmit an important real-time data file to the server. In the initial stage, the client elastic data network access module obtains local network environment information through the network environment detection unit, and selects the optimal network channel for connection based on the cluster network information. During data transmission, the connection monitoring unit continuously monitors the network status and calculates the real-time evaluation value of the current network according to the real-time evaluation value calculation formula. If the network delay suddenly increases or the packet loss rate increases, causing the real-time evaluation value to decrease, the connection monitoring unit will immediately suspend the transmission and request the server to reselect the route. The server will recalculate the selection evaluation value based on the new network status, and select the path with the largest evaluation value as the new transmission path. The client will then switch the path and continue to transmit data to ensure that the real-time and integrity of the data are not affected.

[0046] At the same time, if a network failure occurs during network transmission, the fault handling unit will immediately mark the faulty network and cache the data. Then, it will select the next priority network and notify the server to recover from the failure. After the server completes the failure recovery, the client will try to reconnect and continue to transmit data. If all networks are unavailable, it will be processed according to the preset strategy. In this way, the client elastic data network access module can ensure efficient and stable transmission of data in the network, and automatically switch and recover when the network fails or the data demand changes.

[0047] Intelligent data transmission module, used to select the optimal data transmission path;

[0048] This module selects the optimal path for data transmission based on file type and network status to improve data transmission efficiency and quality. It contains the following submodules:

[0049] A sorting unit, used for analyzing file features and distinguishing file types according to the file features, wherein the file types include real-time files and integrity files;

[0050] A real-time file evaluation unit, configured to calculate a real-time evaluation value by a real-time evaluation value calculation formula when receiving data classified as a real-time file, and select a path with a maximum value of the real-time evaluation value;

[0051] An integrity file evaluation unit, configured to calculate a reliability evaluation value by a reliability evaluation value calculation formula when receiving data classified as an integrity file, and select a path with a maximum reliability evaluation value;

[0052] The reliability evaluation value calculation formula is as follows:

[0053] K=B*(1-L) 2

[0054] Among them, K is the selected evaluation value, B is the available path bandwidth, which means the bandwidth available for data transmission on the network path; L is the packet loss rate.

[0055] The intelligent data transmission module is a highly intelligent system component designed to select the optimal path for data transmission based on file type and network status, thereby improving the efficiency and quality of data transmission. This module not only has basic data transmission functions, but also integrates a variety of advanced sub-modules and technologies to meet data transmission needs in different scenarios.

[0056] Specifically, the intelligent data transmission module analyzes the file features through the sorting unit and distinguishes them according to the file type. For real-time files, the module will use the real-time evaluation value calculation formula to select the path with the largest real-time evaluation value for transmission to ensure the timely arrival of data. For integrity files, the module will use the reliability evaluation value calculation formula to select the path with the largest reliability evaluation value for transmission to ensure the integrity and accuracy of the data.

[0057] In addition, the intelligent data transmission module is also equipped with a data recovery module, which can select the best path according to the network status during data recovery operations to ensure that data can be recovered quickly and stably. At the same time, the automatic detection and recovery network fault module is responsible for real-time monitoring of the network and server node status, timely detection and processing of faults, and providing key detection data support during master-slave switching.

[0058] For example, in a large enterprise-level data transmission system, the intelligent data transmission module needs to transmit a mixed data packet containing real-time files and integrity files. During the transmission process, the sorting unit first analyzes the data packet to distinguish the real-time files from the integrity files.

[0059] For real-time files, the real-time file evaluation unit will calculate the real-time evaluation value of each path according to the current network status, and select the path with the largest real-time evaluation value for transmission. In this way, even in the case of network congestion or large delay, it can ensure that the real-time file can arrive in time.

[0060] For integrity files, the integrity file evaluation unit will calculate the reliability evaluation value of each path according to the reliability evaluation value calculation formula, and select the path with the largest reliability evaluation value for transmission. In this way, the integrity and accuracy of the integrity file can be ensured even in the case of unstable network or high packet loss rate.

[0061] At the same time, during the data transmission process, if the network fails or data is lost, the data recovery module will start immediately, call the relevant functions of the intelligent data transmission module, and select the best path for data recovery based on the current network usage. The automatic detection and recovery network fault module will monitor the network and server node status in real time, detect and handle faults in time, and ensure the continuity and stability of data transmission.

[0062] In the master-slave switching scenario, the master-slave switching detection unit will monitor the network status and health status of the master node and slave node in real time. Once a failure of the master node is detected, the master-slave switching process will be started immediately, and the best slave node will be selected for switching based on the network health status data set to ensure the continuity and reliability of data transmission.

[0063] The data recovery module is used to call the intelligent data transmission module and select the best path according to the network usage. The data recovery module is mainly responsible for selecting the best path according to the network status during the data recovery operation to ensure that the data can be recovered quickly and stably. It calls the relevant functions of the intelligent data transmission module and selects the best path according to the network usage.

[0064] Automatic detection and recovery of network faults module, used to detect and handle network channel or path anomalies, detect master and slave in case of faults, obtain network health status data, and switch between master and slave according to network health status data;

[0065] The automatic detection and recovery module is responsible for real-time monitoring of the status of the network and server nodes, timely detection and processing of network channel or path anomalies, and providing key detection data support during master-slave switching. This module integrates the following sub-modules to achieve comprehensive monitoring of the network environment and rapid response to faults. It contains the following sub-modules:

[0066] A fault detection unit is used to collect sensor monitoring hardware status and agent program monitoring software service status, and regularly package them into a fault detection data set and send them to the fault detection server;

[0067] The fault judgment unit is used to judge the fault of the fault detection server according to the collected fault detection data set. If the CPU usage rate, memory usage rate and network device temperature of the server node continue to exceed the manually set value, it is judged as a fault;

[0068] The fault recovery unit is used to select an available link from the pre-configured backup links to switch after detecting a fault. If it is a network link fault, it updates the routing table and network configuration. If it is a server node fault, it takes corresponding measures according to the type, migrates the service on the faulty node to other healthy nodes and notifies the administrator to replace the hardware. If it is a software fault, it restarts the service and analyzes the log. If it cannot be automatically recovered, it sends a detailed report to the administrator.

[0069] The master-slave switching detection unit is used to detect the network status and health status of the master and slave when the network link of the cluster module master / salve fails, obtain the network health status data set, and provide a basis for subsequent master-slave switching.

[0070] The fault detection unit continuously collects hardware and software status data and sends it to the fault detection server regularly. The fault detection server makes a judgment based on the received data set and the preset threshold, and triggers the fault recovery process if a fault exists. The fault recovery unit takes corresponding recovery measures according to the fault type, including network link switching, service migration, hardware replacement, etc. When the master-slave network link fails, the master-slave switching detection unit is responsible for detecting the network status and health status, and providing data support for subsequent master-slave switching.

[0071] For example, in a large data center, the CPU usage of the server node suddenly soared to 95%, while the memory usage also reached 90%, and the temperature of the network equipment also exceeded the normal working range. At this time, the fault detection unit will capture these abnormal data and package them into a fault detection data set and send them to the fault detection server. After receiving the data, the fault judgment unit quickly determines that the server node has a fault according to the preset threshold. After receiving the fault notification, the fault recovery unit first tries to recover by restarting the service, but finds it invalid. Then migrate the key services on the node to other healthy nodes, and notify the administrator to replace the faulty hardware as soon as possible. In this process, if the master-slave network link of the cluster module also fails, the master-slave switching detection unit will immediately detect this change and start to detect the network status and health status of the slave node. Once it is confirmed that the slave node is in good condition, it will automatically trigger the master-slave switching process to ensure the continuity of the service.

[0072] Data security processing module, used for encryption and verification;

[0073] The data security processing module encryption algorithm encrypts the data to ensure the security of the data during transmission and storage. At the same time, it also generates a random encryption key and stores it securely in the hardware security module. It contains the following submodules:

[0074] A data encryption unit, used to encrypt data using an AES algorithm and generate a random encryption key, wherein the key is stored in a hardware security module;

[0075] A hardware security module, used to store the key, which is periodically changed according to manually set sensitivity parameters;

[0076] The data verification unit is used for the client to send the key hash value to the server for record. After receiving the data, the server compares the hash value. If they are consistent, it will be decrypted. If they are inconsistent, it will be rejected and notified to resend. During the verification process, the verification results and related information are recorded;

[0077] The storage security unit is used to perform integrity verification and encrypted storage on OSS when data is stored in OSS, select storage location and redundancy method according to the file characteristics, regularly calculate hash value comparison, and restore data when problems are found.

[0078] Exemplarily, in order to ensure the security of files during transmission and storage, before the file is transmitted, the data encryption unit uses the AES algorithm to encrypt the file and generates a random encryption key. The key is securely stored in the hardware security module. The system sets a key replacement cycle based on the sensitivity of the file. The hardware security module automatically replaces the key according to the set cycle to ensure the security of the key. During the file transmission process, the data verification unit generates a hash value of the key and sends it to the partner's server for filing. When the partner's server receives the file, it compares the hash value to verify the integrity of the file. If the hash values ​​are consistent, the file is decrypted; if not, the file is rejected and the financial institution is notified to resend it.

[0079] During the file storage phase, the storage security unit verifies the integrity of the file and encrypts the storage. At the same time, it selects the best storage location and redundancy method based on the characteristics of the file. In addition, the unit will regularly calculate the hash value of the file and perform a comparison test to ensure the integrity and security of the file.

[0080] The cache optimization module is used to collect data access frequency data and heat dynamic data, and adjust the cache strategy according to the access frequency data and heat dynamic data;

[0081] The cache optimization module focuses on collecting data access frequency data and hot dynamic data, and intelligently adjusts the cache strategy based on this data to optimize cache performance and significantly improve data access speed, ensuring that cache resources are maximized. It includes the following sub-modules:

[0082] The data collection unit is used to collect data access frequency data and heat dynamic data to calculate the heat value:

[0083] The cache management unit is used to adjust the cache strategy based on data access frequency data and heat dynamic data. When the cache space is insufficient, the data block is eliminated according to the heat value. If the data is modified before elimination, it is marked as dirty data and written back to the backend storage in time. During the cache management process, the cache hit rate and cache utilization rate are monitored to evaluate the effectiveness of the cache strategy.

[0084] The cache adjustment unit is used to dynamically adjust the cache size according to the system memory situation; when M <M T <γ, adjust the cache by adjusting the formula;

[0085] The heat value calculation formula is as follows:

[0086] H=βnf

[0087] Among them, H is the heat value, n is the number of visits in a unit time interval, f is the visit frequency, and β is the heat adjustment coefficient, which is used to adjust the influence of the visit time interval and visit frequency on the heat value;

[0088] The adjustment formula is as follows:

[0089] C′=C+δ*(MM T *γ)

[0090] Among them, C' is the adjusted cache, C' is the current cache, M is the current available memory, and M T is the total system memory, δ is the adjustment step size, and γ is the memory utilization threshold.

[0091] The cache optimization module first uses the data collection unit to comprehensively collect the frequency information and heat dynamic data of data access. These data are important bases for the subsequent adjustment of the cache strategy. Based on the collected data, the module calculates the heat value of each data block. The heat value is a comprehensive indicator that reflects the access frequency and popularity of the data block. For example, when a user frequently browses a certain product on an e-commerce website, the data collection unit will record the number of visits and time intervals of this product to provide data support for the subsequent heat value calculation. Another example is that if a certain product is browsed many times in a short period of time, its heat value will be relatively high, making it more likely to be cached in memory.

[0092] Based on the heat value, the cache management unit will intelligently adjust the cache strategy, including the storage and elimination rules of cached data. When the cache space is insufficient, data blocks with lower heat values ​​will be eliminated first. The cache adjustment unit will dynamically adjust the cache size according to the current memory situation of the system to ensure that cache resources are fully utilized while avoiding waste of memory resources.

[0093] Exemplarily, a cache optimization module is used on an e-commerce website to optimize the cache of product data. When a user frequently browses a certain product, the data collection unit will record the number of visits and time intervals for this product. Based on these data, the heat value calculation unit will calculate the heat value of this product. If the heat value is high, the data block of this product will be cached in the memory first. As the user continues to access, the cache management unit will intelligently manage the cache data according to the heat value to ensure that the data of popular products is always cached in the memory. At the same time, if the system memory situation changes, the cache adjustment unit will dynamically adjust the cache size according to the adjustment formula to ensure that the cache resources are maximized.

[0094] The load balancing module collects node load parameters and distributes data traffic according to the node load parameters;

[0095] The load balancing module is responsible for intelligently distributing data traffic according to the load parameters of the nodes to ensure the stability and efficiency of the system. This module collects the load parameters of the nodes, calculates the load weights, and distributes the newly arrived data traffic according to these weights. It contains the following submodules:

[0096] The load data collection unit is used to deploy performance monitoring agents on nodes to collect data on CPU usage, memory usage, network bandwidth utilization, and disk I / O usage, and to smooth the data to reduce the impact of data fluctuations on load balancing decisions.

[0097] A load weight calculation unit, used for the load balancer to calculate the node load weight according to a formula;

[0098] Traffic distribution unit, used when new requests arrive, the load balancer uses weighted polling to distribute them to lightly loaded nodes according to weights, re-evaluates and adjusts weights regularly, and adjusts strategies in advance based on historical data prediction and analysis;

[0099] The load balancer is based on the following formula:

[0100] W=U CPU *W CPU +U MEM *W MEM +U NET *W NET +U DISK *W DISK

[0101] Among them, W CPU , W MEM , W NET , W DISK is the weight coefficient, the sum is 1; U CPU U is the CPU usage, which indicates the proportion of CPU currently being used; MEM U is the memory usage rate, which indicates the proportion of memory currently occupied; NET U is the network bandwidth utilization, which indicates the proportion of network bandwidth currently occupied; DISK Disk I / O usage, indicating the current busyness of disk I / O operations

[0102] The core of the load balancing module lies in the traffic distribution mechanism. It first deploys a performance monitoring agent on the node through the load data collection unit to collect the node's CPU usage, memory occupancy, network bandwidth utilization, and disk I / O usage parameters in real time. The load weight calculation unit calculates the load weight of each node based on these parameters and a specific formula. Finally, the traffic distribution unit uses a weighted polling method to distribute newly arrived requests to nodes with lighter loads based on these weights, and regularly re-evaluates and adjusts the weights to cope with changes in system load.

[0103] Exemplarily, a website uses a load balancing module to manage its backend server cluster. During peak shopping hours, a large number of users visit the website at the same time to perform shopping operations. At this time, the load data collection unit will collect key load parameters such as the CPU usage rate and memory occupancy rate of each server in real time. The load weight calculation unit will calculate the load weight of each server based on these parameters and a specific formula. Then, the traffic distribution unit will use a weighted round-robin method to distribute the newly arrived requests to the servers with lighter loads based on these weights.

[0104] As the shopping peak continues, the load on some servers may gradually increase. At this time, the traffic distribution unit will adjust the strategy in time to distribute more requests to other servers with lighter loads. At the same time, the load data collection unit will continue to collect new load parameters so that the load weight calculation unit can accurately calculate the latest load weight of each server.

[0105] In this way, e-commerce websites can ensure system stability and efficiency during peak shopping periods, providing users with a smooth shopping experience. At the same time, the load balancing module can also intelligently adjust strategies based on the actual situation of the system to adapt to the changing load requirements.

[0106] The data storage module is used to set data life cycle parameters and execute corresponding storage and deletion strategies for corresponding data according to the data life cycle parameters.

[0107] The data storage module allows users or system administrators to set lifecycle parameters for different types of data. These parameters define the data storage period, storage location, backup strategy, and deletion conditions. The module will then automatically execute the corresponding storage and deletion strategies based on these parameters to ensure effective management and compliance of data throughout its lifecycle.

[0108] For example, in a medical system, for sensitive patient data, the system administrator may set a shorter storage period and specify that the data should be stored in a secure cloud storage and backed up daily. For some non-sensitive data, such as patient appointment records, a longer storage period may be set and stored on a local disk.

[0109] Based on the set data lifecycle parameters, the data storage module automatically executes the corresponding storage strategy. Depending on the manual preset, this includes storing the data in a specified location, creating a backup, and applying any necessary encryption and compression techniques. For example, when the medical system receives new patient data, the data storage module automatically stores the data in a secure cloud storage according to the preset parameters and immediately creates a backup. If the data needs to be encrypted, the module will automatically apply the corresponding encryption algorithm.

[0110] When data reaches its storage period or meets other deletion criteria, the data storage module automatically executes the deletion policy. This may include deleting data from primary storage and backup, and ensuring that the deletion process complies with any relevant legal or compliance requirements.

[0111] Those skilled in the art will understand that the various digital numbers such as the first and second involved in this application are only for the convenience of description, and are not used to limit the scope of this application, nor do they indicate a sequence of precedence. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated with each other are in an "or" relationship. "At least one" refers to one or more. At least two refers to two or more. "At least one", "any one" or similar expressions refer to any combination of these items, including any combination of single items (individuals) or plural items (individuals). For example, at least one item (individual, kind) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0112] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0113] The steps of the method or algorithm described in this application can be directly embedded in hardware, software units executed by a processor, or a combination of the two. The software unit can be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM or any other form of storage medium in the art. Exemplarily, the storage medium can be connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Optionally, the storage medium can also be integrated into the processor. The processor and the storage medium can be arranged in an ASIC, and the ASIC can be arranged in a terminal. Optionally, the processor and the storage medium can also be arranged in different components in the terminal. These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0114] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present application and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application is intended to include these modifications and variations.

Claims

1. A dynamically switched distributed file storage system based on an elastic data network, characterized in that: include: Multi-network architecture module, used to deploy multiple independent network channels, each of which transmits data independently; The client elastic data network access module is used to access and store data according to the data network and priority, and automatically switch and recover in case of network failure; Intelligent data transmission module, used to select the optimal data transmission path; A data recovery module, used to call the intelligent data transmission module and select the best path according to network usage; Automatic detection and recovery of network faults module, used to detect and handle network channel or path anomalies, detect master and slave in case of faults, obtain network health status data, and switch between master and slave according to network health status data; Data security processing module, used for encryption and verification; The cache optimization module is used to collect data access frequency data and heat dynamic data, and adjust the cache strategy according to the access frequency data and heat dynamic data; The load balancing module collects node load parameters and distributes data traffic according to the node load parameters; The data storage module is used to set data life cycle parameters and execute corresponding storage and deletion strategies for corresponding data according to the data life cycle parameters.

2. The dynamically switching distributed file storage system based on elastic data network according to claim 1 is characterized in that: The multi-network architecture module further comprises: The network topology planning unit is used to plan the network topology structure according to the system scale and performance requirements, and obtain the number and layout parameters of independent network channels; A network device configuration unit, configured to configure a dedicated network device for each network channel according to the number and layout parameters of the independent network channels; A network status acquisition unit is used to install a network sensor on each network channel according to the number and layout parameters of the independent network channels, collect real-time data of network delay, packet loss rate, bandwidth utilization, and network traffic, and the sensor collects and packages the sensor data set and transmits it to the management server; The network management unit is used to configure the network management software according to the number of independent network channels and layout parameters, receive sensor data sets, display network status, and provide network configuration and fault diagnosis data.

3. The dynamically switching distributed file storage system based on elastic data network according to claim 1 is characterized in that: The client elastic data network access module further comprises: The network environment detection unit is used to detect the local network environment, obtain the list of available network interfaces, send ICMP Echo requests to the default gateway to test connectivity, record network delay and packet loss rate, and establish an initial connection with the cluster server to obtain cluster network information, including the list of supported data networks and their priorities; The first connection unit is used by the client to try to connect according to the cluster network priority. When connecting, it sends a request containing its own network information and data transmission requirements to the server. The server selects the best path according to the internal network status of the cluster and returns it. The client establishes a connection accordingly. If it fails, the information is recorded and the fault processing is started; The connection monitoring unit is used for the client to continuously monitor the network status during data transmission, periodically send detection packets to obtain real-time information, and calculate the real-time evaluation value through the real-time evaluation value calculation formula. If the network change does not meet the requirements, the transmission is suspended and the server is requested to re-select the route. The server recalculates the selection evaluation value and adopts the path with the maximum selection evaluation value. The client switches the path to continue transmission; The fault handling unit is used to mark the faulty network, cache data, select a secondary priority network and notify the server when the client detects a network fault. The server detects the internal fault of the cluster and recovers. The client waits and tries to reconnect. If it fails, it tries other networks in order of priority. If all are unavailable, it is handled according to the preset strategy. The master-slave switching coordination unit is used to receive the new master broadcast notification, update the connection information, switch to the new master, reconnect and resume the request when the cluster module switches from master to slave. If a connection failure occurs, it is handled according to the error handling strategy; The real-time evaluation value calculation formula is as follows: Among them, R is the real-time evaluation value, B is the available bandwidth of the path, which means the bandwidth available for data transmission on the network path; D is the network delay; L is the packet loss rate; Among them, the calculation formula for selecting the evaluation value is as follows: P=(α*BD*1000-L*10000)*E Among them, α is the network stability weight, which ranges from 0 to 1 according to the preset control value, indicating the stability of the network path. The more stable it is, the higher the weight. B is the available bandwidth of the path, indicating the bandwidth available for data transmission on the network path. D is the network delay. L is the packet loss rate. E is the urgency of data recovery, ranging from 1 to 10.

4. The dynamically switching distributed file storage system based on elastic data network according to claim 1 is characterized in that: The intelligent data transmission module further comprises: A sorting unit, used for analyzing file features and distinguishing file types according to the file features, wherein the file types include real-time files and integrity files; A real-time file evaluation unit, configured to calculate a real-time evaluation value by a real-time evaluation value calculation formula when receiving data classified as a real-time file, and select a path with a maximum value of the real-time evaluation value; An integrity file evaluation unit, configured to calculate a reliability evaluation value by a reliability evaluation value calculation formula when receiving data classified as an integrity file, and select a path with a maximum reliability evaluation value; The reliability evaluation value calculation formula is as follows: K=B*(1-L) 2 Among them, K is the selected evaluation value, B is the available path bandwidth, which means the bandwidth available for data transmission on the network path; L is the packet loss rate.

5. The dynamically switching distributed file storage system based on elastic data network according to claim 1 is characterized in that: The automatic detection and recovery network failure module further comprises: A fault detection unit is used to collect sensor monitoring hardware status and agent program monitoring software service status, and regularly package them into a fault detection data set and send them to the fault detection server; The fault judgment unit is used to judge the fault of the fault detection server according to the collected fault detection data set. If the CPU usage rate, memory usage rate and network device temperature of the server node continue to exceed the manually set value, it is judged as a fault; The fault recovery unit is used to select an available link from the pre-configured backup links to switch after detecting a fault. If it is a network link fault, it updates the routing table and network configuration. If it is a server node fault, it takes corresponding measures according to the type, migrates the service on the faulty node to other healthy nodes and notifies the administrator to replace the hardware. If it is a software fault, it restarts the service and analyzes the log. If it cannot be automatically recovered, it sends a detailed report to the administrator. The master-slave switching detection unit is used to detect the network status and health status of the master and slave when the network link of the cluster module master / salve fails, obtain the network health status data set, and provide a basis for subsequent master-slave switching.

6. The dynamically switching distributed file storage system based on elastic data network according to claim 4 is characterized in that: The data security processing module further comprises: A data encryption unit, used to encrypt data using an AES algorithm and generate a random encryption key, wherein the key is stored in a hardware security module; A hardware security module for storing the key, which is periodically changed according to manually set sensitivity parameters; The data verification unit is used for the client to send the key hash value to the server for record. After receiving the data, the server compares the hash value. If they are consistent, it will be decrypted. If they are inconsistent, it will be rejected and notified to resend. During the verification process, the verification results and related information are recorded; The storage security unit is used to perform integrity verification and encrypted storage on OSS when data is stored in OSS, select storage location and redundancy method according to the file characteristics, regularly calculate hash value comparison, and restore data when problems are found.

7. The dynamically switching distributed file storage system based on elastic data network according to claim 1 is characterized in that: The cache optimization module further comprises: The data collection unit is used to collect data access frequency data and heat dynamic data to calculate the heat value: The cache management unit is used to adjust the cache strategy based on data access frequency data and heat dynamic data. When the cache space is insufficient, the data block is eliminated according to the heat value. If the data is modified before elimination, it is marked as dirty data and written back to the backend storage in time. During the cache management process, the cache hit rate and cache utilization rate are monitored to evaluate the effectiveness of the cache strategy. The cache adjustment unit is used to dynamically adjust the cache size according to the system memory situation; when M <M T <γ, adjust the cache by adjusting the formula; The heat value calculation formula is as follows: H=βnf Among them, H is the heat value, n is the number of visits in a unit time interval, f is the visit frequency, and β is the heat adjustment coefficient, which is used to adjust the influence of the visit time interval and visit frequency on the heat value; The adjustment formula is as follows: C′=C+δ*(MM T *c) Among them, C' is the adjusted cache, C' is the current cache, M is the current available memory, and M T is the total system memory, δ is the adjustment step size, and γ is the memory utilization threshold.

8. The dynamically switching distributed file storage system based on elastic data network according to claim 1 is characterized in that: The load balancing module further comprises: The load data collection unit is used to deploy performance monitoring agents on nodes to collect data on CPU usage, memory usage, network bandwidth utilization, and disk I / O usage, and to smooth the data to reduce the impact of data fluctuations on load balancing decisions. A load weight calculation unit, used for the load balancer to calculate the node load weight according to a formula; Traffic distribution unit, used when new requests arrive, the load balancer uses weighted polling to distribute them to lightly loaded nodes according to weights, re-evaluates and adjusts weights regularly, and adjusts strategies in advance based on historical data prediction and analysis; The load balancer is based on the following formula: W=U CPU *W CPU +U MEM *W MEM +U NET *W NET +U DISK *W DISK Among them, W CPU , W MEM , W NET , W DISK is the weight coefficient, the sum is 1; U CPU U is the CPU usage rate, which indicates the proportion of CPU currently being used; MEM U is the memory usage rate, which indicates the proportion of memory currently occupied; NET U is the network bandwidth utilization, which indicates the proportion of network bandwidth currently occupied; DISK It is the disk usage, which indicates how busy the disk operation is.

Images