A method for storing and reading data in a computer system
Through data compression, encryption and decentralized storage, combined with node load analysis and time stamp recording, the data is realized hierarchical reading and intelligent overload processing, which solves the performance degradation and crash problems of traditional storage methods at high loads, and improves storage and reading efficiency and load balancing.
Patent Information
- Application Number
- CN202410559182.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-05-07
AI Technical Summary
Traditional storage and read methods degrade or crash when facing high loads, and overload processing often requires manual intervention, resulting in low efficiency and load balancing.
Through data compression, encryption and decentralized storage, combined with node load analysis and time stamp recording, hierarchical reading and intelligent overload processing of data are realized, and data rollback and incremental backup strategies are adopted to deal with system overload.
Improve the efficiency and load balancing of data storage and reading, ensure data security and system stability, and reduce resource waste and costs.
Smart Images

Figure CN118377431B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data storage control, and in particular to a computer system data storage and reading method. Background Art
[0002] With the increase in data volume and the requirement for data security, the Redundant Array of Independent Disks (RAID) technology was proposed. RAID improves data security and reliability by distributing data on multiple disks and using redundant data for verification. At the same time, solid-state drives (SSDs) began to enter the commercial market. Compared with traditional mechanical hard disks, SSDs have faster read and write speeds and lower access latency, and are more durable, bringing significant performance improvements to computer systems. With the development of the Internet and the advent of the big data era, cloud storage has become an important data storage method. Users can store data on remote servers and access and manage them through the network. Cloud storage not only provides a convenient way to back up and share data, but also reduces the data management costs of individuals and enterprises. Currently, non-volatile memory (NVM) technologies such as 3D XPoint and storage class memory (SCM) are beginning to emerge. These technologies combine the advantages of memory and storage, have access speeds close to memory and persistent storage characteristics, and are expected to further change the way data is stored and read. However, in the current traditional storage and reading, the load of storage nodes is faced with the high-speed operation of the system, causing some nodes to be overloaded while other nodes are idle. At the same time, performance degradation or crash problems occur when facing high loads. Overload processing is usually manually intervened, resulting in low efficiency and load balance of storage and reading. Summary of the invention
[0003] Based on this, it is necessary to provide a computer system data storage and reading method to solve at least one of the above technical problems.
[0004] To achieve the above object, a computer system data storage and reading method is provided, the method comprising the following steps:
[0005] Step S1: obtaining data to be stored in a computer; compressing the data to be stored in the computer to generate compressed data to be stored in the computer;
[0006] Step S2: Allocate server storage nodes to the compressed data to be stored in the computer to generate data storage server nodes; use the data storage server nodes to perform distributed storage monitoring on the compressed data to be stored in the computer to generate distributed storage node monitoring data; perform node load analysis on the distributed storage node monitoring data to generate node load analysis data;
[0007] Step S3: constructing a front-end and back-end storage information encryption mechanism for the distributed storage node monitoring data, generating a front-end and back-end storage information encryption mechanism; constructing a data storage index for the distributed storage node monitoring data using the front-end and back-end storage information encryption mechanism, generating a data storage hash index;
[0008] Step S4: Record the node timestamp of the distributed storage node monitoring data based on the data storage hash index to obtain the node processing timestamp; synchronize the data using the node processing timestamp to generate synchronized storage data; perform data hierarchical reading of the synchronized storage data using the node load analysis data to generate hierarchical reading data instructions;
[0009] Step S5: perform access frequency analysis on the hierarchical data read instructions to generate read access frequency data; perform system overload prediction on the read access frequency data to generate system overload prediction data; compare the system overload prediction data with a preset overload threshold, and when the system overload prediction data is greater than or equal to the preset overload threshold, perform data rollback on the system overload prediction data to generate rollback storage data; when the system overload prediction data is less than the preset overload threshold, perform incremental backup on the system overload prediction data to generate incremental backup storage data; perform system overload decision on the computer system based on the rollback storage data and the incremental backup storage data to realize the data storage and reading process of the computer system.
[0010] The present invention reduces the size and risk of data during transmission and storage by data compression and encryption. The data encryption mechanism and storage index construction provide a higher level of data security, reduce the possibility of data being illegally obtained or tampered with, and enhance the confidentiality and integrity of the data. The data is dispersed and stored in multiple server nodes, and load balancing analysis is performed to improve storage efficiency and performance. Node load analysis data helps balance the workload of each node, so that the system can more effectively manage data storage and access. Based on data storage hash index and node timestamp record, the synchronization and consistency of data between different nodes are ensured. The node processing timestamp is used to track and manage the update of data, ensuring the correctness and reliability of the data. Through access frequency analysis and system overload prediction, the system can predict the occurrence of overload in advance. When the system overload prediction data is greater than the preset threshold, a data rollback strategy is adopted to ensure the security and integrity of the data; when it is less than the threshold, an incremental backup is performed to improve the stability and fault tolerance of the system. This intelligent overload processing strategy enables the system to better cope with high load conditions and ensure the availability of data and the stability of the system. According to the node load analysis data, data is read in a hierarchical manner, which can optimize the utilization of resources and improve the efficiency of the system. At the same time, the incremental backup strategy enables the system to perform backup according to actual conditions, reducing unnecessary resource waste and lowering costs. Therefore, the present invention improves the efficiency and load balancing of storage and reading by performing data security, storage efficiency, data synchronization, and system overload processing on stored data.
[0011] The beneficial effect of the present invention is that storage space can be saved by compressing data, and decentralized storage can improve data redundancy and reliability and prevent data loss. Monitoring data storage nodes can understand the system status in real time, and load analysis helps to optimize resource allocation and predict system load conditions. Encryption and index construction improve data security and retrieval efficiency, protect sensitive information and improve data access speed. Data synchronization ensures data consistency, while hierarchical reading can selectively obtain data according to demand priority to improve data access efficiency. By analyzing access frequency and predicting system load, measures can be taken in time to prevent system overload and ensure stable operation of the system. Data rollback and incremental backup provide flexible data recovery solutions to ensure data security and reduce the impact of system failures. Based on the analysis of rollback data and backup data, the system can make appropriate overload decisions to ensure the stability and performance of the system under high load. Therefore, the present invention improves the efficiency and load balancing of storage and reading by treating the stored data for data security, storage efficiency, data synchronization, and system overload processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 A schematic diagram of a process flow of a method for storing and reading data in a computer system;
[0013] Figure 2 for Figure 1 Detailed implementation steps of step S2 in the flowchart;
[0014] Figure 3 for Figure 1 Detailed implementation steps of step S3 in FIG.
[0015] Figure 4 for Figure 1 Detailed implementation steps of step S4 in FIG.
[0016] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0017] The following is a clear and complete description of the technical method of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.
[0018] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.
[0019] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.
[0020] To achieve this, please refer to Figures 1 to 4 , a computer system data storage and reading method, the method comprising the following steps:
[0021] Step S1: obtaining data to be stored in a computer; compressing the data to be stored in the computer to generate compressed data to be stored in the computer;
[0022] Step S2: Allocate server storage nodes to the compressed data to be stored in the computer to generate data storage server nodes; use the data storage server nodes to perform distributed storage monitoring on the compressed data to be stored in the computer to generate distributed storage node monitoring data; perform node load analysis on the distributed storage node monitoring data to generate node load analysis data;
[0023] Step S3: constructing a front-end and back-end storage information encryption mechanism for the distributed storage node monitoring data, generating a front-end and back-end storage information encryption mechanism; constructing a data storage index for the distributed storage node monitoring data using the front-end and back-end storage information encryption mechanism, generating a data storage hash index;
[0024] Step S4: Record the node timestamp of the distributed storage node monitoring data based on the data storage hash index to obtain the node processing timestamp; synchronize the data using the node processing timestamp to generate synchronized storage data; perform data hierarchical reading of the synchronized storage data using the node load analysis data to generate hierarchical reading data instructions;
[0025] Step S5: perform access frequency analysis on the hierarchical data read instructions to generate read access frequency data; perform system overload prediction on the read access frequency data to generate system overload prediction data; compare the system overload prediction data with a preset overload threshold, and when the system overload prediction data is greater than or equal to the preset overload threshold, perform data rollback on the system overload prediction data to generate rollback storage data; when the system overload prediction data is less than the preset overload threshold, perform incremental backup on the system overload prediction data to generate incremental backup storage data; perform system overload decision on the computer system based on the rollback storage data and the incremental backup storage data to realize the data storage and reading process of the computer system.
[0026] The present invention reduces the size and risk of data during transmission and storage by data compression and encryption. The data encryption mechanism and storage index construction provide a higher level of data security, reduce the possibility of data being illegally obtained or tampered with, and enhance the confidentiality and integrity of the data. The data is dispersed and stored in multiple server nodes, and load balancing analysis is performed to improve storage efficiency and performance. Node load analysis data helps balance the workload of each node, so that the system can more effectively manage data storage and access. Based on data storage hash index and node timestamp record, the synchronization and consistency of data between different nodes are ensured. The node processing timestamp is used to track and manage the update of data, ensuring the correctness and reliability of the data. Through access frequency analysis and system overload prediction, the system can predict the occurrence of overload in advance. When the system overload prediction data is greater than the preset threshold, a data rollback strategy is adopted to ensure the security and integrity of the data; when it is less than the threshold, an incremental backup is performed to improve the stability and fault tolerance of the system. This intelligent overload processing strategy enables the system to better cope with high load conditions and ensure the availability of data and the stability of the system. According to the node load analysis data, data is read in a hierarchical manner, which can optimize the utilization of resources and improve the efficiency of the system. At the same time, the incremental backup strategy enables the system to perform backup according to actual conditions, reducing unnecessary resource waste and lowering costs. Therefore, the present invention improves the efficiency and load balancing of storage and reading by performing data security, storage efficiency, data synchronization, and system overload processing on stored data.
[0027] In the embodiment of the present invention, reference Figure 1 The above is a flowchart of a method for storing and reading data in a computer system according to the present invention. In this example, the method for storing and reading data in a computer system comprises the following steps:
[0028] Step S1: obtaining data to be stored in a computer; compressing the data to be stored in the computer to generate compressed data to be stored in the computer;
[0029] In an embodiment of the present invention, by using the file system interface provided by the operating system, the specified folder or file path is traversed. The content of the file is read and loaded into the memory, and is prepared for subsequent compression processing. A query operation is performed on the database to retrieve the data to be stored. The query result set is obtained and converted into a data structure in the program. Real-time sensor data is obtained through a sensor interface or a sensor device. The sensor data is converted into a format that can be processed by a computer program. An appropriate compression algorithm, such as Gzip, Deflate, LZ77, etc., is selected according to the data type and requirements. For example, a dictionary-based compression algorithm can be used for text data, and a lossless or lossy image compression algorithm can be used for image data. The data is converted into the format required by the compression algorithm. Character encoding is performed on text data, and bit encoding is performed on binary data. The selected compression algorithm is applied to compress the data. The data block or the entire data stream is compressed to generate compressed data. A tag or header information is added to the compressed data so that the data type and compression algorithm can be correctly identified during decompression. Such a tag may include information such as a compression algorithm identifier and the size of the original data. The compression parameters are adjusted according to the actual situation to achieve the best compression ratio and compression speed. Some compression algorithms have adjustable parameters, such as the compression level of Deflate.
[0030] Step S2: Allocate server storage nodes to the compressed data to be stored in the computer to generate data storage server nodes; use the data storage server nodes to perform distributed storage monitoring on the compressed data to be stored in the computer to generate distributed storage node monitoring data; perform node load analysis on the distributed storage node monitoring data to generate node load analysis data;
[0031] In an embodiment of the present invention, a suitable number and type of data storage server nodes are selected according to storage requirements and system architecture. These nodes can be physical servers or virtualized storage nodes. The compressed data blocks are allocated to different server nodes using the allocation algorithm of the distributed storage system. The allocation algorithm can be a consistent hashing algorithm based on a hash function or other data sharding algorithm. The storage node information to which each compressed data block is allocated is recorded for subsequent data access and management. This information can be stored in a database or other data storage center. The data storage server node receives the allocated compressed data block and stores it according to the allocated location. To ensure the correctness and integrity of data storage, a data checksum check mechanism can be used. The status of the data storage server node is monitored, including storage capacity, operating status, network connection, etc. Monitoring tools or custom scripts can be used for real-time monitoring. Appropriate access rights and authentication mechanisms are set to ensure that only authorized users or systems can access the stored data. This can include access control lists (ACLs), key management, etc. Monitoring data of each storage node is collected, including storage capacity utilization, number of read and write requests, data transmission rate, etc. These data can be obtained through monitoring agents, log files, or other monitoring tools. Use load analysis algorithms to process and analyze monitoring data to evaluate the load of nodes, involving data statistics, smoothing, trend analysis, etc. Based on the results of load analysis, evaluate the load level of each storage node to determine whether load balancing or adjustment is required. Load evaluation can be based on threshold comparison, historical data comparison, etc.
[0032] Step S3: constructing a front-end and back-end storage information encryption mechanism for the distributed storage node monitoring data, generating a front-end and back-end storage information encryption mechanism; constructing a data storage index for the distributed storage node monitoring data using the front-end and back-end storage information encryption mechanism, generating a data storage hash index;
[0033] In the embodiment of the present invention, by selecting a suitable encryption algorithm, such as AES (Advanced Encryption Standard), RSA (asymmetric encryption), HMAC (hash message authentication code), etc., the selection is made according to the sensitivity of the data and the performance requirements. Generate and manage the keys required for encryption. The key can be a symmetric key or an asymmetric key. Symmetric keys are used to encrypt and decrypt data, and asymmetric keys are used for digital signatures and key exchange. The monitoring data is encrypted to generate encrypted monitoring data. An encryption channel is established between the front-end storage node and the back-end receiving node to ensure the security of the data during transmission. The logic of the encryptor and decryptor is implemented for the encryption and decryption operations of the data. The encryptor is used to encrypt the data at the front end, and the decryptor is used to decrypt the data at the back end. Design a suitable data storage index structure, such as a hash table, a tree structure (B-tree, B+ tree, etc.), an inverted index, etc. Ensure that the index structure can quickly locate and retrieve data. The encrypted monitoring data is stored in the corresponding data storage node. The mapping relationship between the index items in the index structure and the actual data blocks needs to be encrypted. The index items are encrypted to protect the security of the index data. Index encryption can use the same encryption algorithm and key management mechanism. Build a data storage hash index based on the encrypted monitoring data and encrypted index items to ensure the consistency and accuracy of the index for subsequent data retrieval and access.
[0034] Step S4: Record the node timestamp of the distributed storage node monitoring data based on the data storage hash index to obtain the node processing timestamp; synchronize the data using the node processing timestamp to generate synchronized storage data; perform data hierarchical reading of the synchronized storage data using the node load analysis data to generate hierarchical reading data instructions;
[0035] In an embodiment of the present invention, by adding a timestamp to each data block of monitoring data, the time when the data is generated or updated is recorded. The timestamp can be the exact time when the data is generated, such as a UNIX timestamp, or a relative time, such as the time interval after the data block is generated. The monitoring data block with the timestamp is stored on the corresponding storage node. Ensure the consistency of the data and the timestamp for subsequent data synchronization and hierarchical reading. Each storage node compares the timestamp of its own data block regularly or in real time to determine whether the data needs to be updated or synchronized. If the node finds that the data timestamp of other nodes is newer, a data synchronization operation is required. Using a synchronization algorithm, the new data block is copied from one node to other nodes to maintain data consistency. The synchronization algorithm can be version-based replication, incremental synchronization, or other distributed data synchronization algorithms. According to the load analysis data of the node, the stored data is divided into different load levels, such as high load, medium load, low load, etc. This can be divided based on indicators such as the load level of the node, network bandwidth, CPU usage, etc. Different data access instructions are generated according to the load level to achieve data access optimization between different nodes. High-load nodes can give priority to processing high-priority access requests, and medium- and low-load nodes can process medium- and low-priority access requests.
[0036] Step S5: perform access frequency analysis on the hierarchical data read instructions to generate read access frequency data; perform system overload prediction on the read access frequency data to generate system overload prediction data; compare the system overload prediction data with a preset overload threshold, and when the system overload prediction data is greater than or equal to the preset overload threshold, perform data rollback on the system overload prediction data to generate rollback storage data; when the system overload prediction data is less than the preset overload threshold, perform incremental backup on the system overload prediction data to generate incremental backup storage data; perform system overload decision on the computer system based on the rollback storage data and the incremental backup storage data to realize the data storage and reading process of the computer system.
[0037] In an embodiment of the present invention, data access instructions of various levels are recorded, including access requests of high priority, medium priority and low priority. The recorded data includes information such as request time, request node, access level, etc. The frequency of data access instructions of each level is counted to obtain access frequency data. Statistics can be performed according to a certain time window, such as access frequency per hour or per day. An overload prediction model is constructed based on historical access frequency data. Commonly used models include time series analysis, regression analysis, neural network, etc. The model can be constructed separately according to different data access levels, or multiple levels can be comprehensively considered. The trained overload prediction model is used to predict new access frequency data to obtain system overload prediction data. The prediction result can be the load degree of the system within a certain time period, such as the predicted number of requests. A preset overload threshold is set according to system performance and load conditions. This threshold can be the access frequency of a certain level, or it can be an indicator such as system resource utilization. The system overload prediction data is compared with the preset overload threshold. If the predicted data is greater than or equal to the threshold, the system is about to be overloaded. Overload processing can be done in two ways: data rollback and incremental backup. If the system overload prediction data is greater than or equal to the threshold, the system overload prediction data is rolled back to generate rollback storage data. If the system overload prediction data is less than the threshold, the system overload prediction data is incrementally backed up to generate incremental backup storage data. Based on the generated rollback storage data and incremental backup storage data, the current system status, that is, whether an overload has occurred, is determined. Based on the data rollback and incremental backup situations, the integrity and availability of the system data are determined. If the system is overloaded, the system response measures are determined based on the rollback storage data and incremental backup storage data. If the system rollback storage data is available and the data is complete, the system can be restored to the rollback storage state. If the system data backup is complete, the incremental backup data can be restored. If both data are unavailable, other emergency measures are required.
[0038] Preferably, step S1 comprises the following steps:
[0039] Step S11: Obtaining data to be stored in the computer;
[0040] Step S12: Cleaning the data to be stored in the computer to generate cleaned data to be stored in the computer;
[0041] Step S13: performing data format conversion on the cleaned data to be stored in the computer to generate format conversion data to be stored in the computer;
[0042] Step S14: performing data capacity detection on the data to be stored in the computer format conversion to generate the data capacity to be stored in the computer; and compressing the data to be stored in the computer format conversion according to the data capacity to be stored in the computer to generate compressed data to be stored in the computer.
[0043] The present invention can remove errors, duplications or incomplete parts in the data to be stored through data cleaning, thereby improving the quality and accuracy of the data and ensuring that the stored data is reliable. Data format conversion converts data into a unified format, making it easier to manage, process and analyze. This helps to ensure the compatibility and consistency of data between different systems and applications. Data capacity detection and compression can help optimize the utilization of storage resources. By detecting data capacity and compressing as needed, the use of storage space can be reduced and costs can be saved when storing large amounts of data. The storage requirements of compressed data mean that less data needs to be transmitted and processed, thereby increasing the speed of reading and writing data. This is especially important for applications that need to frequently access stored data. By reducing the volume of data, the cost of data transmission can be reduced, especially in network transmission or cloud storage. This is an important consideration for large-scale data processing and storage solutions.
[0044] In an embodiment of the present invention, data is collected from a network, a database, or other sources as required, for example, by using Web Scraping tools (such as Beautiful Soup, Scrapy), database queries (such as SQL queries), etc. Missing values are filled or deleted using technical means, such as mean filling, median filling, deleting rows or columns containing missing values, etc. Outliers are identified and processed, such as using a box plot or Z-score method to detect outliers, and then processed according to the specific circumstances, such as replacing them with acceptable values or deleting them. Duplicate data rows are detected and deleted. Data is converted to a unified format, such as a date and time format, a unified unit, etc. Data is converted from one type to another, such as converting text data to numeric data. Date and time formats are converted as required, such as from a string to a date and time object, or vice versa. Data is encoded and converted, such as converting UTF-8 encoding to other encoding methods. According to the requirements of the target system, the data structure is adjusted, such as converting data from a wide table to a long table, or vice versa. Calculate the total amount of data, including the number of rows and columns, and calculate the storage requirements of the data based on the data type and size, such as storage space calculation for text data, numerical data, image data, etc. Compress the data using lossless compression algorithms such as GZIP, ZIP, etc.
[0045] Preferably, step S2 comprises the following steps:
[0046] Step S21: Slice the compressed data to be stored in the computer to generate sliced data blocks to be stored in the computer;
[0047] Step S22: performing a cyclic redundancy check on the shard data block to be stored in the computer to generate a shard redundancy check code to be stored;
[0048] Step S23: Allocating server nodes to the shard data blocks to be stored and the shard redundancy check codes to be stored on the computer based on the distributed storage system to generate data storage server nodes;
[0049] Step S24: using the data storage server node to perform data distributed storage monitoring on the computer's to-be-stored shard data blocks and to-be-stored shard redundant check codes, and generate distributed storage node monitoring data; performing node load analysis on the distributed storage node monitoring data, and generating node load analysis data.
[0050] The present invention can improve storage and transmission efficiency by dividing data into small blocks, especially when processing large data sets. The fragmented data blocks can be managed and allocated more flexibly, so that the reading and writing of data can be more parallel and efficient. CRC can be used to detect errors and damages in the transmission or storage process of data. By calculating and appending the check code, the integrity of the data can be verified when the data is read, thereby improving the reliability and stability of the data. The server node allocation based on the distributed storage system can improve the availability and fault tolerance of the data. The data and the check code are distributed to multiple nodes, and the data can still be recovered or accessible even if some nodes fail. By monitoring the storage nodes, the status and performance of the storage system can be understood in real time. Node load analysis can help optimize the distribution and access strategy of data, thereby improving the performance and scalability of the entire system.
[0051] As an example of the present invention, refer to Figure 2 As shown, in this example, step S2 includes:
[0052] Step S21: Slice the compressed data to be stored in the computer to generate sliced data blocks to be stored in the computer;
[0053] In an embodiment of the present invention, by determining the size of each data block, it is usually selected according to system limitations or data processing efficiency, for example, the size of each data block is 1MB, 10MB, etc. The way of selecting the sharding can be by row, by column, by time period, etc., which is determined according to the characteristics of the data and storage requirements. Ensure that the data blocks after sharding can still maintain the association between the data, which is convenient for subsequent processing and recovery. If it is large-scale data, you can use a distributed file system such as Hadoop's HDFS or AWS S3. These systems have built-in data sharding functions and use data processing frameworks such as Apache Spark and Apache Flink. They provide distributed computing capabilities and data sharding support. According to the data sharding strategy, write an algorithm to divide the compressed data into blocks according to the set rules to generate sharded data blocks.
[0054] Step S22: performing a cyclic redundancy check on the shard data block to be stored in the computer to generate a shard redundancy check code to be stored;
[0055] In the embodiment of the present invention, a CRC polynomial is selected, which is usually fixed. For example, the polynomial of CRC-32 is 0x04C11DB7. Each data block is padded to adapt to the calculation of CRC. A common method is to add a certain number of 0s at the end of the data block so that the length of the data block meets the requirements of the CRC polynomial. CRC calculation is performed on the padded data block to obtain a check value, called a CRC check code. The calculated CRC check code is stored together with the original data block so that it can be checked during subsequent reading.
[0056] Step S23: Allocating server nodes to the shard data blocks to be stored and the shard redundancy check codes to be stored on the computer based on the distributed storage system to generate data storage server nodes;
[0057] In an embodiment of the present invention, the previously generated data blocks with CRC check codes are fragmented according to the rules of the distributed storage system, usually according to the data block size or other rules. The fragmented data blocks will be stored on different storage nodes. Using the redundant check code method, an additional redundant check code data will be generated and stored on the node according to the rules of the distributed storage system. The redundant check code can be a Parity check, a Reed-Solomon code, etc. According to the strategy of the distributed storage system, the fragmented data blocks and redundant check codes are allocated to different storage nodes. Distributed storage systems usually consider factors such as node load balancing, data block location and replication strategy. Ensure that each data block has a sufficient number of copies or redundant check codes to cope with node failures or data corruption. According to the configuration of the distributed storage system, parameters such as the number of copies and the number of redundant check codes can be set.
[0058] Step S24: using the data storage server node to perform data distributed storage monitoring on the computer's to-be-stored shard data blocks and to-be-stored shard redundant check codes, and generate distributed storage node monitoring data; performing node load analysis on the distributed storage node monitoring data, and generating node load analysis data.
[0059] In an embodiment of the present invention, a monitoring agent is deployed on a data storage server node to monitor the storage of sharded data blocks and redundant check codes. Through the monitoring program, the storage space utilization, data reading and writing speed and other indicators on each server node are regularly checked. The monitoring data is aggregated and stored in a centralized monitoring database for subsequent analysis and processing. Information such as the storage capacity, read and write request volume, and response time of each node is collected. The collected data is analyzed using an algorithm, such as threshold-based load warnings, average load calculations, and historical load trend analysis. Based on historical data and models, the load conditions of the nodes in the future are predicted to provide a reference for load balancing. The load indicators of the nodes, such as the load balancing index and load prediction data, are calculated according to the load analysis algorithm, and the generated node load analysis data is stored in the corresponding database or data warehouse.
[0060] Preferably, step S24 includes the following steps:
[0061] Step S241: Based on the consistent hashing algorithm, the computer's to-be-stored shard data blocks and the to-be-stored shard redundancy check codes are distributed to the data storage server nodes for data reception, thereby generating local storage data blocks;
[0062] Step S242: registering data nodes according to the locally stored data blocks, generating data storage server nodes, wherein the data storage server nodes include front-end storage nodes and back-end receiving nodes;
[0063] Step S243: using the data storage server node to replicate the local storage data block to generate a distributed data copy; performing data distribution storage update on the distributed data copy to obtain a dispersed storage location table;
[0064] Step S244: monitoring data access requests for local storage data blocks to generate access request data; collecting monitoring information of data storage server nodes where data are located in the distributed storage location table based on the access request data to obtain distributed storage node monitoring data;
[0065] Step S245: Perform node load analysis on the distributed storage node monitoring data to generate node load analysis data.
[0066] The present invention can evenly distribute data to different data storage server nodes through a consistent hashing algorithm, thereby achieving data load balancing and improving system performance and scalability. Data node registration and server node generation will ensure that each node in the system is correctly identified and configured, which is conducive to subsequent management and monitoring work. The replication and distribution storage update of data copies ensures redundant backup and update synchronization of data in the system, and enhances the fault tolerance and data reliability of the system. By monitoring data access requests and collecting storage node information, access hotspots and storage node load conditions in the system can be discovered in a timely manner, providing a data basis for subsequent load analysis. Node load analysis can deeply understand the load conditions and performance bottlenecks of each storage node in the system, help optimize the system's resource allocation and load balancing strategy, and improve the overall performance and stability of the system.
[0067] In an embodiment of the present invention, the hash value of the sharded data block to be stored and the redundant check code is calculated. The hash value is mapped to the storage node on the consistent hash ring. The data block and the check code are transmitted to the corresponding storage node. After receiving the data block and the check code, the storage node generates a local storage data block. Ensure the integrity and consistency of the data block. The storage node registers its own information with the registration service, including node ID, IP address, storage capacity, etc. The registration service maintains the status and availability information of the storage node. Based on the registered storage node information, the system generates a complete data storage server node. The front-end storage node is used to receive and process data access requests, and the back-end receiving node is used to receive a copy of the data block. The local storage data block is copied to generate a distributed data copy. Ensure the consistency and reliability of the copy, for example, using a data synchronization algorithm (such as Raft, Paxos). Distribute the copy of the data block to other storage nodes, and determine the storage location of the copy according to the consistent hash algorithm. When the data block is updated, a storage update operation is performed to ensure the consistency of all replica data. Record the distributed storage location of each data block to form a decentralized storage location table. The table includes information such as the data block ID and the storage node ID.
[0068] Preferably, step S3 comprises the following steps:
[0069] Step S31: screening the front-end and back-end storage data of the front-end storage nodes and the back-end receiving nodes according to the monitoring data of the distributed storage nodes to obtain the front-end storage data and the back-end storage data;
[0070] Step S32: encrypting the front-end storage data to generate front-end encrypted storage data; generating an encryption key for the front-end encrypted storage data using a pseudo-random number generator to obtain the front-end encrypted storage data key;
[0071] Step S33: receiving the front-end encrypted storage data based on the back-end receiving node to generate the back-end received storage encrypted data; decrypting the back-end received storage encrypted data using the front-end encrypted storage data key to generate the front-end and back-end storage information encryption mechanism;
[0072] Step S34: Utilize the front-end and back-end storage information encryption mechanism to extract ciphertext features from the distributed storage node monitoring data to generate a decrypted storage feature vector; construct a data storage index for the decrypted storage feature vector to generate a data storage hash index.
[0073] The present invention can distinguish front-end storage data from back-end storage data by screening the monitoring data of distributed storage nodes, thereby realizing data classification and management. The efficiency and accuracy of data storage are improved, and different types of data are processed and stored in a targeted manner. Encrypting the front-end storage data can protect the security and privacy of the data and prevent unauthorized access and leakage. The overall security of the system is improved, meeting the requirements of data protection and privacy confidentiality, and reducing the risk of data leakage. Based on the front-end and back-end storage information encryption mechanism, the security of data during transmission and storage is ensured to prevent data from being tampered with or stolen. A complete data encryption protection system is established to improve the system's control and protection capabilities for data and enhance data security and integrity. By decrypting the stored feature vectors and constructing hash indexes, fast retrieval and index management of encrypted data are realized. The efficiency of data retrieval and access is improved, the time cost of system data processing is reduced, and the performance of data management and query operations is optimized.
[0074] As an example of the present invention, refer to Figure 3 As shown, in this example, step S3 includes:
[0075] Step S31: screening the front-end and back-end storage data of the front-end storage nodes and the back-end receiving nodes according to the monitoring data of the distributed storage nodes to obtain the front-end storage data and the back-end storage data;
[0076] In an embodiment of the present invention, the load information of each node, such as storage capacity utilization rate, read / write request processing speed, etc., is obtained from the monitoring data of the distributed storage nodes. A load balancing algorithm, such as weighted polling, minimum number of connections, etc., is used to select a suitable front-end storage node according to the load of the node. For example, if the storage capacity of a node is close to saturation or the read / write request processing speed is slow, the weight of the node can be reduced to reduce the probability of receiving new data. At the same time, considering the distribution of data blocks, similar data blocks are stored on the same front-end storage node as much as possible to improve data access efficiency. According to the result of the load balancing algorithm, a front-end storage node suitable for storing data is screened. The data block is sent to the selected front-end storage node for storage. The number and distribution strategy of data copies are determined according to system requirements and available resources. For example, two copies of each data block can be selected, one in the local storage node and the other in the remote node. The load information of each node, including storage capacity utilization rate, etc., is obtained from the monitoring data of the distributed storage nodes. According to the data copy strategy and the node load, a back-end receiving node suitable for storing copy data is selected. Ensure that the selected node has sufficient storage capacity and processing power to receive and store copy data. Send a copy of the data block to the selected backend receiving node for storage.
[0077] Step S32: encrypting the front-end storage data to generate front-end encrypted storage data; generating an encryption key for the front-end encrypted storage data using a pseudo-random number generator to obtain the front-end encrypted storage data key;
[0078] In an embodiment of the present invention, a symmetric encryption algorithm and encryption mode suitable for the scenario are selected, such as the AES encryption algorithm and the CBC mode. The CBC mode is selected here because it has better security and can resist selective resistance attacks. A key needs to be generated for encrypting and decrypting data. This key is called an encryption key. Key generation can use a pseudo-random number generator to generate a secure random number as a key. The front-end storage data is encrypted using the selected symmetric encryption algorithm and the generated encryption key. Each data block is encrypted using the same key to ensure that the encryption results of the same data are consistent. A pseudo-random number generator is used to generate a random number of sufficient length and complexity as the front-end encrypted storage data key. This key is the key for encrypting and decrypting the front-end storage data. The generated front-end encrypted storage data key is securely stored in the system for subsequent decryption. The key can be encrypted and stored in a secure key management system to increase the security of the key.
[0079] Step S33: receiving the front-end encrypted storage data based on the back-end receiving node to generate the back-end received storage encrypted data; decrypting the back-end received storage encrypted data using the front-end encrypted storage data key to generate the front-end and back-end storage information encryption mechanism;
[0080] In an embodiment of the present invention, a back-end receiving node receives front-end encrypted storage data through a communication protocol. Ensure the integrity and security of the data, for example, use checksum, hash and other methods to verify the data. Store the received encrypted data on the storage medium of the back-end receiving node. At this time, the data is still in an encrypted state and needs to be decrypted before it can be read and used. Obtain the previously generated front-end encrypted storage data key from the secure storage. Use the front-end encrypted storage data key and the corresponding decryption algorithm to decrypt the encrypted data received and stored at the back-end. Ensure the integrity and correctness of the decrypted data. The decrypted data is the original front-end storage data. Combine the front-end storage data with the encrypted data received and stored at the back-end to form a front-end and back-end storage information encryption mechanism. This mechanism ensures the security and privacy protection of the data during the storage process.
[0081] Step S34: Utilize the front-end and back-end storage information encryption mechanism to extract ciphertext features from the distributed storage node monitoring data to generate a decrypted storage feature vector; construct a data storage index for the decrypted storage feature vector to generate a data storage hash index.
[0082] In an embodiment of the present invention, feature extraction is performed on the monitoring data of the distributed storage nodes by using a feature extraction algorithm. A hash algorithm, such as SHA-256, can be used here to perform hash calculation on the monitoring data to obtain ciphertext features. Since the data is encrypted, the ciphertext features are obtained, and the plaintext information of the data will not be leaked. The ciphertext features extracted from the monitoring data of each distributed storage node are stored to form a decrypted storage feature vector. These ciphertext feature vectors will be used for subsequent data storage index construction. Features are obtained from the decrypted storage feature vector obtained in the previous step. The decrypted storage feature vector is mapped to the hash index space using a hash index construction algorithm (such as LSH). Similar features in the hash index space will be mapped to similar hash buckets to improve the query efficiency of similar data. Determine a suitable hash index data structure, such as a B-tree or a hash table, to support fast data query. The data storage hash index of each distributed storage node is stored in the system. In this way, a hash index for data storage is established, which can be used to quickly locate and query data.
[0083] Preferably, step S4 comprises the following steps:
[0084] Step S41: Perform distributed transaction processing on the distributed storage node monitoring data based on the data storage hash index to generate a distributed transaction processing log;
[0085] Step S42: Record the node timestamp of the data storage server node according to the distributed transaction processing log to obtain the node processing timestamp; synchronize the data of the data storage server node using the node processing timestamp to generate synchronized storage data;
[0086] Step S43: using the node load analysis data to perform hierarchical data reading on the synchronous storage data, and generating hierarchical data reading instructions.
[0087] The present invention ensures the consistency and reliability of data by performing distributed transaction processing based on data storage hash index, and records operation logs for subsequent tracking and recovery. Improve the reliability and stability of the system, ensure the atomicity and consistency of data operations, and reduce the risk of data errors and loss. Record node timestamps according to distributed transaction processing logs, and use timestamps to synchronize data on data storage server nodes to ensure data consistency between different nodes. Ensure data synchronization and updating in a distributed environment, improve the reliability and performance of the data storage system, and reduce the possibility of data conflicts and loss. Perform data hierarchical reading based on node load analysis data to achieve intelligent management and optimized reading of stored data. Improve the efficiency and response speed of data reading, adjust the data reading priority according to actual needs, optimize the utilization of system resources, and improve user experience.
[0088] As an example of the present invention, refer to Figure 4 As shown, in this example, step S4 includes:
[0089] Step S41: Perform distributed transaction processing on the distributed storage node monitoring data based on the data storage hash index to generate a distributed transaction processing log;
[0090] In the embodiment of the present invention, the transaction processing process starts when the system receives a transaction request. For each monitoring data to be processed, its position in the data storage hash index needs to be obtained. A distributed transaction protocol, such as the Two-Phase Commit (2PC) protocol, is used to ensure the atomicity and consistency of the transaction. The 2PC protocol is divided into a voting phase (Prepare) and an execution phase (Commit or Abort): Prepare phase: The coordinator sends a Prepare request to the participants, and the participants perform the preparatory work before the transaction and return the prepared transaction status (Commit or Abort) to the coordinator. Commit or Abort phase: The coordinator decides whether to commit the transaction based on the response of the participants. If all participants return Commit, the coordinator commits the transaction; otherwise, the coordinator rolls back the transaction. In the execution phase of the transaction, according to the decision of the coordinator, the monitoring data of each participating distributed storage node is processed accordingly, including operations such as update, delete, and insert. These operations ensure the consistency and atomicity of the data, that is, either all operations are successfully completed or none of the operations are executed. In the transaction processing process, a distributed transaction log is recorded, including information such as the start, commit, and rollback of the transaction. These logs are used for recovery and replay when a failure occurs. During the transaction processing process, key information such as the start, commit, and rollback of each transaction is recorded. This includes information such as the transaction identifier, operation type, operation object, and operation result. The generated distributed transaction processing logs are stored on a reliable storage medium. You can choose to store logs in the form of log files, database tables, etc.
[0091] Step S42: Record the node timestamp of the data storage server node according to the distributed transaction processing log to obtain the node processing timestamp; synchronize the data of the data storage server node using the node processing timestamp to generate synchronized storage data;
[0092] In an embodiment of the present invention, the execution order and node participation of each transaction are obtained by parsing the distributed transaction processing log. For each data storage server node, the timestamp of the node processing the transaction is recorded. The timestamp can be the start time, commit time or other time mark indicating the node processing order. According to the recorded node processing timestamp, the processing timestamps of all nodes are sorted to determine the execution order of the transaction. According to the execution order of the transaction, the data storage server nodes are synchronized. The following steps can be used for data synchronization: Incremental synchronization: only the changed part is transmitted to reduce the amount of transmitted data. Full synchronization: complete data synchronization is performed at specific time intervals or when an event is triggered. Checksum confirmation: checksum and other mechanisms are used during the synchronization process to ensure the correctness of the data. Conflict resolution: handle data conflicts that occur, such as adopting the latest modification, merge and other strategies. After completing data synchronization, synchronized storage data is generated to ensure that the data of all nodes remain consistent.
[0093] Step S43: using the node load analysis data to perform hierarchical data reading on the synchronous storage data, and generating hierarchical data reading instructions.
[0094] In an embodiment of the present invention, a data classification strategy is formulated based on the node load analysis data, and the strategy includes: based on the access frequency: frequently accessed data is marked as a high level, occasionally accessed data is marked as a medium level, and rarely accessed data is marked as a low level. Based on the importance of the data: important data is marked as a high level, and auxiliary data is marked as a low level. Based on the node load situation: node data with a higher load is marked as a low level, and node data with a lower load is marked as a high level. According to the data classification strategy, the classification standard of the data is determined, such as high, medium, and low levels. The characteristics and conditions of each level are defined for subsequent data reading operations. A hierarchical reading strategy is formulated based on the data classification results. The reading priority of each level of data is determined, such as high-level data is read first, and low-level data is read later. According to the hierarchical reading strategy, corresponding read instructions are generated for each data. Determine when and how to read high, medium, and low-level data: high-level data needs to be read in real time. Medium-level data can be read regularly according to demand. Low-level data can be read during low-peak demand or on demand.
[0095] Preferably, step S43 includes the following steps:
[0096] Step S431: using the node load analysis data to perform storage node load classification on the synchronous storage data, and generating storage load classification nodes, wherein the storage load classification nodes include high-load storage nodes, medium-load storage nodes and low-load storage nodes;
[0097] Step S432: using pessimistic locking to lock resources of high-load storage nodes, and generating high-load storage node adjustment data;
[0098] Step S433: performing optimistic concurrency control on the medium-load storage node through optimistic locking to generate medium-load storage node adjustment data;
[0099] Step S434: performing lock granularity control on the low-load storage node according to the distributed lock, and generating low-load storage node adjustment data;
[0100] Step S435: Based on the high-load storage node adjustment data, the medium-load storage node adjustment data and the low-load storage node adjustment data, the synchronous storage data is read in a hierarchical manner to generate a hierarchical data reading instruction, wherein the data classification includes assigning a high priority to the high-load storage node adjustment data, assigning a medium priority to the medium-load storage node adjustment data, and assigning a low priority to the low-load storage node adjustment data.
[0101] The present invention classifies the storage node load of synchronous storage data according to the node load analysis data, and divides it into high, medium and low load storage nodes. The load of the storage node is effectively balanced to avoid performance degradation or failure caused by overloading of some nodes, and the stability and reliability of the system are improved. Pessimistic locks are used to lock resources for high-load storage nodes, and optimistic locks are used to control concurrency for medium-load storage nodes. Through a reasonable lock mechanism, the consistency and integrity of data are ensured under high load conditions, while the concurrency performance of the system is improved under medium load conditions, and the data processing efficiency is improved. The lock granularity of low-load storage nodes is controlled according to distributed locks. Under low load conditions, appropriate lock granularity is used to avoid lock competition and performance loss, and improve the concurrency performance and resource utilization of the system. Data hierarchical reading is performed on the synchronous storage data based on the node adjustment data, and corresponding hierarchical reading data instructions are generated. The data reading priority is dynamically adjusted according to the node load conditions, the system's responsiveness and performance under different load conditions are improved, and the user experience is optimized.
[0102] In an embodiment of the present invention, a data hierarchical reading strategy is formulated by adjusting data of high-load storage nodes, adjusting data of medium-load storage nodes, and adjusting data of low-load storage nodes. According to storage requirements and node load conditions, the reading priority of high-, medium-, and low-load node data is determined. According to the hierarchical reading strategy, corresponding read instructions are generated for each data. According to data classification, it is determined when and how to read the data of high-, medium-, and low-load nodes: high-load node data needs to be read in real time, so a high priority is assigned. Medium-load node data can be read regularly according to demand and a medium priority is assigned. Low-load node data can be read during low-peak demand, or read on demand, and a low priority is assigned. Pessimistic locks are used to lock resources of high-load storage nodes. High-load storage node adjustment data is generated, such as data compression, data migration, etc. Optimistic locks are used to handle concurrent access to medium-load storage nodes. Medium-load storage node adjustment data is generated, such as data update, data migration, etc. Distributed locks are used to perform fine-grained concurrency control on low-load storage nodes. Low-load storage node adjustment data is generated, such as data backup, data cleaning, etc. A data tiered reading strategy is formulated based on the high-load storage node adjustment data, the medium-load storage node adjustment data, and the low-load storage node adjustment data.
[0103] Preferably, step S432 includes the following steps:
[0104] Step S4321: performing data read and write log analysis on the high-load storage node to generate high-load node access log data; performing concurrent request analysis on the high-load node access log data to generate high-load node concurrent request data;
[0105] Step S4322: performing data access heat analysis on the high-load storage node according to the high-load node access log data and the high-load node concurrent request data to generate access heat data;
[0106] Step S4323: Perform resource time lock control on the access heat data according to the pessimistic lock to generate high-load storage node adjustment data.
[0107] The present invention generates high-load node access log data by analyzing the data reading and writing logs of high-load storage nodes. Understanding the actual access situation of storage nodes can help discover potential performance bottlenecks and optimization space, and provide data support for subsequent optimization. Concurrent request analysis is performed on the high-load node access log data to generate high-load node concurrent request data. Understanding the concurrent access mode of the system under high load conditions is helpful to evaluate the carrying capacity and concurrent processing capacity of the system, and provide a reference for system performance optimization. According to the high-load node access log data and concurrent request data, data access heat analysis is performed on the high-load storage node to generate access heat data. Hotspot data on the storage node, that is, frequently accessed data, is identified to provide a basis for subsequent data storage and access strategy optimization, and improve data access efficiency. Resource time locking and regulation are performed on the access heat data according to the pessimistic lock to generate high-load storage node adjustment data. Through the regulation of the pessimistic lock, the access priority and response speed of the hotspot data are improved, data inconsistency or performance degradation caused by concurrent access is avoided, and the stability and reliability of the system are improved.
[0108] In an embodiment of the present invention, a data access log is recorded by a storage node, including information such as timestamps of read and write operations, data identifiers, and access types. These access log data record the data access situation of high-load storage nodes. The access log data is analyzed using a log analysis algorithm to extract key information such as data access frequency, read-write ratio, etc. High-load node access log data is generated, including data access frequency, hot data identifiers, etc. The access log data is analyzed using a concurrent request analysis algorithm to identify the situation of concurrent read and write requests. High-load node concurrent request data is generated, including the number of concurrent requests, requested data identifiers, etc. The access heat analysis algorithm is used to combine the high-load node access log data and concurrent request data to calculate the access heat of the data. Factors such as the read frequency, write frequency, and number of concurrent requests of the data are considered. Access heat data is generated according to the analysis results. The access heat data includes heat indicators of each data, such as access frequency, hot data identifiers, etc. The pessimistic lock mechanism is used to lock the resources of the high-load storage node for a time period. Ensure that only one transaction can access the data of the high-load storage node within a specified time period. According to the locked time period and access popularity data, high-load storage node adjustment data is generated. The adjustment data includes data migration, compression, backup and other operations to optimize data access of high-load nodes.
[0109] Preferably, step S5 comprises the following steps:
[0110] Step S51: performing access frequency analysis on the hierarchical data read instruction to generate read access frequency data;
[0111] Step S52: dividing the read access frequency data into data sets to generate a model training set and a model test set;
[0112] Step S53: Perform model training on the model training set using a convolutional neural network algorithm to generate a data reading overload training model; perform model testing on the data reading overload training model using a model testing set to generate a data reading overload prediction model;
[0113] Step S54: importing the read access frequency data into the data read overload prediction model to perform system overload prediction and generate system overload prediction data; comparing the system overload prediction data with a preset overload threshold, and when the system overload prediction data is greater than or equal to the preset overload threshold, rolling back the system overload prediction data and generating rollback storage data;
[0114] Step S55: when the system overload prediction data is less than a preset overload threshold, incremental backup is performed on the system overload prediction data to generate incremental backup storage data;
[0115] Step S56: Performing a system overload decision on the computer system according to the rollback storage data and the incremental backup storage data to implement the data storage and reading process of the computer system.
[0116] The present invention generates read access frequency data by performing access frequency analysis on hierarchical data read instructions. Understanding the frequency distribution of data read operations is helpful to determine which data is frequently accessed, thereby providing basic data for subsequent system overload prediction. Data sets are divided for read access frequency data to generate model training sets and model test sets. Separating training sets and test sets helps to evaluate the generalization ability and accuracy of the model and ensure the prediction performance of the model on unknown data. The model training set is trained using a convolutional neural network algorithm to generate a data read overload training model; the data read overload training model is tested using a model test set to generate a data read overload prediction model. A prediction model is established, and the accuracy and reliability of the model are verified through training and testing, so as to provide a reliable prediction model for subsequent system overload prediction. The read access frequency data is imported into the data read overload prediction model to perform system overload prediction and generate system overload prediction data; the system overload prediction data is compared with a preset overload threshold and corresponding data processing is performed. By predicting system overload through a prediction model, potential system overload risks can be discovered in advance, and corresponding data processing measures can be taken to ensure the stability and reliability of the system. Rollback storage or incremental backup is performed based on system overload prediction data, and system overload decisions are made. Reasonable system overload decisions are made based on prediction data, and system load is effectively managed through rollback storage or incremental backup to ensure the normal operation of the system and the integrity of data storage.
[0117] In an embodiment of the present invention, the recorded data instructions are analyzed by using tools or writing codes to identify common and uncommon data access patterns. Read access frequency data is generated for subsequent processing and analysis. The generated read access frequency data is divided into a model training set and a model test set. Random division or time division is usually adopted to ensure that the data distribution and characteristics of the training set and the test set are similar. The model training set is trained using algorithms such as convolutional neural networks (CNN) to establish a data read overload training model. The trained model is tested using a model test set to evaluate the performance and accuracy of the model and generate a data read overload prediction model. The read access frequency data is input into the data read overload prediction model to perform system overload prediction and generate system overload prediction data. The system overload prediction data is compared with a preset overload threshold. If the threshold is exceeded, a data rollback operation is performed. The data rollback operation can be to roll back the system state to an earlier time point or to migrate the data to a backup storage to reduce the system load. If the system overload prediction data does not exceed the preset overload threshold, an incremental backup operation is performed. Incremental backup can be to back up newly added data to an independent storage medium so that data can be restored or restored when needed. System overload decisions are made for the computer system based on the rollback storage data and incremental backup storage data. Decisions include adjusting system resource allocation, optimizing data storage strategies, or performing other management operations to achieve a balance and optimization of the data storage and reading process of the computer system.
[0118] Preferably, step S55 comprises the following steps:
[0119] Step S551: when the system overload prediction data is less than a preset overload threshold, a data full backup time analysis is performed based on the system overload prediction data to generate a full backup time;
[0120] Step S552: Identify data block changes according to the full backup time and generate changed data blocks;
[0121] Step S553: performing change frequency analysis on the changed data block to generate data block change frequency data;
[0122] Step S554: performing fast Fourier transformation on the data block change frequency data to generate data block change spectrum data; performing spectrum peak extraction on the data block change spectrum data to generate data block change spectrum peak data;
[0123] Step S555: locally backing up the changed data block based on the changed spectrum peak data of the data block to generate incremental backup storage data.
[0124] By analyzing the frequency and spectrum peak of the changed data blocks, the system can more accurately determine the data that needs to be backed up, reducing the amount of backup data and backup time. Incremental backup only backs up data blocks with high frequency of change, which can save storage space and is more efficient when restoring data. The backup strategy can minimize the resources and time required for backup while ensuring data integrity, thereby improving backup efficiency and system stability.
[0125] In an embodiment of the present invention, data is passed to an overload prediction model for analysis based on the real-time monitoring of the system load to determine whether the system is in an overload state. If the predicted data is less than the overload threshold, the system performs a full backup time analysis to determine the best backup time point. According to the full backup time point, the most recent backup and the current data state are compared, and the data blocks that have changed since the last backup are identified through a comparison algorithm or a hash algorithm, and a list of changed data blocks is generated. The identified changed data blocks are statistically analyzed to calculate the change frequency of each data block, that is, the number of times it has been modified within a period of time. The change frequency data of the changed data blocks is fast Fourier transformed, the time domain data is converted into frequency domain data, and spectrum peaks are extracted from the spectrum data, which represent data blocks with a higher change frequency. According to the data block change spectrum peak data, an incremental backup strategy is used to perform local backup of data blocks with a higher change frequency. This means that only data blocks with a high change frequency are backed up to generate incremental backup storage data.
[0126] The beneficial effect of the present invention is that storage space can be saved by compressing data, and decentralized storage can improve data redundancy and reliability and prevent data loss. Monitoring data storage nodes can understand the system status in real time, and load analysis helps to optimize resource allocation and predict system load conditions. Encryption and index construction improve data security and retrieval efficiency, protect sensitive information and improve data access speed. Data synchronization ensures data consistency, while hierarchical reading can selectively obtain data according to demand priority to improve data access efficiency. By analyzing access frequency and predicting system load, measures can be taken in time to prevent system overload and ensure stable operation of the system. Data rollback and incremental backup provide flexible data recovery solutions to ensure data security and reduce the impact of system failures. Based on the analysis of rollback data and backup data, the system can make appropriate overload decisions to ensure the stability and performance of the system under high load. Therefore, the present invention improves the efficiency and load balancing of storage and reading by treating the stored data for data security, storage efficiency, data synchronization, and system overload processing.
[0127] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.
[0128] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.
Claims
1. A method for storing and reading data in a computer system, characterized in that: The following steps are involved: Step S1: Obtaining data to be stored in the computer; Compressing the data to be stored in the computer to generate compressed data to be stored in the computer; Step S2: Allocate server storage nodes for the compressed data to be stored in the computer, and generate data storage server nodes; Using the data storage server node to perform distributed storage monitoring on the compressed data to be stored in the computer, and generating distributed storage node monitoring data; Perform node load analysis on distributed storage node monitoring data to generate node load analysis data; Step S2 includes the following steps: Step S21: Slice the compressed data to be stored in the computer to generate sliced data blocks to be stored in the computer; Step S22: performing a cyclic redundancy check on the shard data block to be stored in the computer to generate a shard redundancy check code to be stored; Step S23: Allocating server nodes to the shard data blocks to be stored and the shard redundancy check codes to be stored on the computer based on the distributed storage system to generate data storage server nodes; Step S24: using the data storage server node to perform data distributed storage monitoring on the computer to-be-stored shard data blocks and the to-be-stored shard redundancy check codes, and generating distributed storage node monitoring data; performing node load analysis on the distributed storage node monitoring data, and generating node load analysis data; Step S3: constructing a front-end and back-end storage information encryption mechanism for the distributed storage node monitoring data, generating a front-end and back-end storage information encryption mechanism; constructing a data storage index for the distributed storage node monitoring data using the front-end and back-end storage information encryption mechanism, generating a data storage hash index; Step S4: Record the node timestamp of the distributed storage node monitoring data based on the data storage hash index to obtain the node processing timestamp; synchronize the data using the node processing timestamp to generate synchronized storage data; perform data hierarchical reading of the synchronized storage data using the node load analysis data to generate hierarchical reading data instructions; Step S4 includes the following steps: Step S41: Perform distributed transaction processing on the distributed storage node monitoring data based on the data storage hash index to generate a distributed transaction processing log; Step S42: Record the node timestamp of the data storage server node according to the distributed transaction processing log to obtain the node processing timestamp; synchronize the data of the data storage server node using the node processing timestamp to generate synchronized storage data; Step S43: using the node load analysis data to perform hierarchical data reading on the synchronous storage data, and generating hierarchical data reading instructions; Step S43 includes the following steps: Step S431: using the node load analysis data to perform storage node load classification on the synchronous storage data, and generating storage load classification nodes, wherein the storage load classification nodes include high-load storage nodes, medium-load storage nodes and low-load storage nodes; Step S432: using pessimistic locking to lock resources of high-load storage nodes, and generating high-load storage node adjustment data; Step S433: performing optimistic concurrency control on the medium-load storage node through optimistic locking to generate medium-load storage node adjustment data; Step S434: performing lock granularity control on the low-load storage node according to the distributed lock, and generating low-load storage node adjustment data; Step S435: performing data hierarchical reading of the synchronous storage data based on the high-load storage node adjustment data, the medium-load storage node adjustment data, and the low-load storage node adjustment data, and generating a hierarchical reading data instruction, wherein the data hierarchical reading includes assigning a high priority to the high-load storage node adjustment data, assigning a medium priority to the medium-load storage node adjustment data, and assigning a low priority to the low-load storage node adjustment data; Step S5: perform access frequency analysis on the hierarchical data read instructions to generate read access frequency data; perform system overload prediction on the read access frequency data to generate system overload prediction data; compare the system overload prediction data with a preset overload threshold, and when the system overload prediction data is greater than or equal to the preset overload threshold, perform data rollback on the system overload prediction data to generate rollback storage data; when the system overload prediction data is less than the preset overload threshold, perform incremental backup on the system overload prediction data to generate incremental backup storage data; perform system overload decision on the computer system based on the rollback storage data and the incremental backup storage data to realize the data storage and reading process of the computer system.
2. The computer system data storage and reading method according to claim 1, characterized in that: Step S1 includes the following steps: Step S11: Obtaining data to be stored in the computer; Step S12: Cleaning the data to be stored in the computer to generate cleaned data to be stored in the computer; Step S13: performing data format conversion on the cleaned data to be stored in the computer to generate format-converted data to be stored in the computer; Step S14: performing data capacity detection on the data to be stored in the computer format conversion to generate the data capacity to be stored in the computer; and compressing the data to be stored in the computer format conversion according to the data capacity to be stored in the computer to generate compressed data to be stored in the computer.
3. The computer system data storage and reading method according to claim 1, characterized in that: Step S24 includes the following steps: Step S241: Based on the consistent hashing algorithm, the computer's to-be-stored shard data blocks and the to-be-stored shard redundancy check codes are distributed to the data storage server nodes for data reception, thereby generating local storage data blocks; Step S242: registering data nodes according to the locally stored data blocks, generating data storage server nodes, wherein the data storage server nodes include front-end storage nodes and back-end receiving nodes; Step S243: using the data storage server node to replicate the local storage data block to generate a distributed data copy; performing data distribution storage update on the distributed data copy to obtain a dispersed storage location table; Step S244: monitoring data access requests for local storage data blocks to generate access request data; collecting monitoring information of data storage server nodes where data are located in the distributed storage location table based on the access request data to obtain distributed storage node monitoring data; Step S245: Perform node load analysis on the distributed storage node monitoring data to generate node load analysis data.
4. The computer system data storage and reading method according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: screening the front-end and back-end storage data of the front-end storage nodes and the back-end receiving nodes according to the monitoring data of the distributed storage nodes to obtain the front-end storage data and the back-end storage data; Step S32: encrypting the front-end storage data to generate front-end encrypted storage data; generating an encryption key for the front-end encrypted storage data using a pseudo-random number generator to obtain the front-end encrypted storage data key; Step S33: receiving the front-end encrypted storage data based on the back-end receiving node to generate the back-end received storage encrypted data; decrypting the back-end received storage encrypted data using the front-end encrypted storage data key to generate the front-end and back-end storage information encryption mechanism; Step S34: Utilize the front-end and back-end storage information encryption mechanism to extract ciphertext features from the distributed storage node monitoring data to generate a decrypted storage feature vector; construct a data storage index for the decrypted storage feature vector to generate a data storage hash index.
5. The computer system data storage and reading method according to claim 4, characterized in that: Step S432 includes the following steps: Step S4321: performing data read and write log analysis on the high-load storage node to generate high-load node access log data; performing concurrent request analysis on the high-load node access log data to generate high-load node concurrent request data; Step S4322: performing data access heat analysis on the high-load storage node according to the high-load node access log data and the high-load node concurrent request data to generate access heat data; Step S4323: Perform resource time lock control on the access heat data according to the pessimistic lock to generate high-load storage node adjustment data.
6. The computer system data storage and reading method according to claim 1, characterized in that: Step S5 includes the following steps: Step S51: performing access frequency analysis on the hierarchical data read instruction to generate read access frequency data; Step S52: dividing the read access frequency data into data sets to generate a model training set and a model test set; Step S53: Perform model training on the model training set using a convolutional neural network algorithm to generate a data reading overload training model; perform model testing on the data reading overload training model using a model testing set to generate a data reading overload prediction model; Step S54: importing the read access frequency data into the data read overload prediction model to perform system overload prediction and generate system overload prediction data; comparing the system overload prediction data with a preset overload threshold, and when the system overload prediction data is greater than or equal to the preset overload threshold, rolling back the system overload prediction data and generating rollback storage data; Step S55: when the system overload prediction data is less than a preset overload threshold, incremental backup is performed on the system overload prediction data to generate incremental backup storage data; Step S56: Performing a system overload decision on the computer system according to the rollback storage data and the incremental backup storage data to implement the data storage and reading process of the computer system.
7. The computer system data storage and reading method according to claim 6, characterized in that: Step S55 includes the following steps: Step S551: when the system overload prediction data is less than a preset overload threshold, a data full backup time analysis is performed based on the system overload prediction data to generate a full backup time; Step S552: Identify data block changes according to the full backup time and generate changed data blocks; Step S553: performing change frequency analysis on the changed data block to generate data block change frequency data; Step S554: performing fast Fourier transformation on the data block change frequency data to generate data block change spectrum data; performing spectrum peak extraction on the data block change spectrum data to generate data block change spectrum peak data; Step S555: locally backing up the changed data block based on the changed spectrum peak data of the data block to generate incremental backup storage data.
Citation Information
Patent Citations
Hybrid distributed graph data storage and calculation method
CN117112692A
Distributed object read-write performance improving method based on multi-thread technology
CN117707771A
Data storage method and system
CN117827850A
Method, device and system for storing and reading data
WO2016180055A1
