Computer data writing system and method

Through the combination of cache module, redundant storage module, load balancing module and data persistence module, the performance bottlenecks, data loss risks and storage media life of computer data writing systems in high-load scenarios are solved, and efficient Reliable data writing and storage.

CN119045735AInactive Publication Date: 2024-11-29ZHIXINGZHE (SHENZHEN) COMPUTER COMMUNICATIONS CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411119484.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-29
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing computer data writing systems have slow data writing speed in high load or high concurrency scenarios, which poses a risk of data loss, and some storage media have limited lifespans, and long-term high-frequency writing may lead to shortening of lifespan.

Method used

Adopt cache modules, redundant storage modules, load balancing modules and data persistence modules, combining non-volatile memory, RAID technology, software-defined storage and machine learning algorithms to realize temporary data storage, redundant storage, load scheduling and fast data recover.

Benefits of technology

It improves data writing speed, enhances data security, optimizes load balancing, and ensures system reliability and equipment life extension.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119045735A_ABST
    Figure CN119045735A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of computer data writing, and discloses a computer data writing system and method.The computer data writing system comprises a high-speed cache module which temporarily stores data to be written and uses a nonvolatile memory as a high-speed cache layer; and the redundancy storage module is used for enhancing the security of the data through a redundancy storage strategy and realizing data redundancy by adopting an RAID (Redundant Array of Independent Disks) technology, and comprises a data replication mechanism in an RAID5 / 6 or distributed file system. Through temporary storage of the cache module, the data writing speed is increased, direct access to the main storage medium is reduced, the overall system performance is remarkably improved, an extra data protection layer is provided for an RAID technology or a data replication mechanism of the redundant storage module, and the risk of data loss is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer data writing, and in particular to a computer data writing system and method. Background Art

[0002] A computer data writing system is a set of mechanisms or processes used to store data into computer storage media. This typically involves transferring data from memory or an input source to a storage device. The system manages the organization of data, writing speed, integrity checking, and error handling to ensure that data is stored correctly and efficiently, and can be reliably read when needed.

[0003] Existing computer data writing systems face several shortcomings, including: 1. Performance bottleneck: Although SSDs are significantly better than traditional hard disks, data writing speed may still become a bottleneck in high-load or high-concurrency scenarios; 2. Data loss risk: Data may be lost or damaged during the writing process, especially in the event of a power outage or system crash; 3. Lifespan limitation: Some storage media, such as SSD, have a limited number of write times. Long-term and high-frequency writing may shorten their lifespan.

[0004] Therefore, it is necessary to design a computer data writing system and method to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a computer data writing system and method.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: A computer data writing system comprises the following modules: Cache module: temporarily stores data to be written, using non-volatile memory as a cache layer; Redundant storage module: Enhance data security through redundant storage strategies and use RAID technology to achieve data redundancy, including data replication mechanisms in RAID5 / 6 or distributed file systems; Load balancing module: dynamically distributes write tasks to different storage units to avoid overloading a single storage unit. It uses an intelligent scheduling algorithm based on software-defined storage and combines machine learning technology to predict and adjust load distribution. Data persistence module: uses logging and checkpoint technology to quickly restore data status after an abnormal system restart; Life management module: Uses a wear leveling algorithm to evenly distribute write operations.

[0007] As a preferred technical solution of the present invention, in the cache module, an LRU algorithm or an LFU algorithm is used to manage the data in the cache, and commonly used data is preloaded into the cache at startup to improve the initial access speed. In the event of a power outage or system crash, logging technology is used to ensure the persistence of data, and the data in the cache is regularly synchronized to the main storage device to reduce the amount of data that needs to be restored after an abnormal system restart. A multi-level cache architecture is used in combination with different types of storage media to meet different levels of needs, sensitive data in the cache is encrypted, and redundant storage strategies are implemented, including mirroring or RAID.

[0008] As a preferred technical solution of the present invention, in the redundant storage module, RAID5 or RAID6 is used to achieve data redundancy, and the data replication mechanism in the distributed file system is adopted to achieve data redundancy. Check codes are used to detect and correct data errors, and parity information is distributed on multiple disks. It supports replacing a disk failure, provides block storage, object storage and file system services, supports automatic repair and data replication, uses parity bits to detect data errors, and regularly creates data snapshots to restore to the state at a certain point in time when data is damaged or lost. Once a failed disk is detected, the system automatically marks it as offline and replaces it with a hot spare disk.

[0009] As a preferred technical solution of the present invention, in the load balancing module, write tasks are dynamically allocated to different storage units according to real-time load conditions, and software-defined storage technology is adopted to manage and allocate storage resources through the software layer. The load conditions of each storage unit are continuously monitored, including indicators such as write speed and latency. A supervised learning algorithm is used to train the model to predict load changes in different time periods, including random forests and gradient boosting trees.

[0010] As a preferred technical solution of the present invention, in the data persistence module, detailed logs of all data writing operations are recorded in real time, including transaction operations, modification records and system status. The log format is adopted to ensure that each write is first recorded in the log and then actually written to the storage medium. The complete data snapshot of the system is regularly saved, including the current data status and transaction progress. The checkpoint interval is set, and the data snapshot is automatically generated and saved in a stable storage area. In combination with log recording and checkpoint technology, an intelligent recovery strategy is formulated, including selecting the best recovery point and data reconstruction process. Through hash verification or data verification and technology, the consistency of the restored data state with the original state is checked.

[0011] A method for writing computer data comprises the following steps: Step 1, data reception and caching: After the data is received into the system, it is first written into the non-volatile memory in the cache module. The cache module temporarily stores the data to be written to reduce the direct access burden to the main storage medium and improve the writing speed; Step 2: Redundant storage and data protection: extract data from the cache module and send the data to the redundant storage module. The redundant storage module uses RAID technology or the data replication mechanism in the distributed file system to disperse and store the data in multiple storage units to ensure data redundancy and security. Step 3: Load balancing and scheduling: Before data is written to the redundant storage module, the load balancing module dynamically evaluates the load of each storage unit and uses an intelligent scheduling algorithm to assign data writing tasks to storage units with lighter loads to avoid overloading any single storage unit. Step 4: Data persistence processing: After the data is written, the data persistence module starts to record detailed logs of the write operation and creates checkpoints at appropriate times. Through logging and checkpoint technology, the system can quickly recover to the most recent consistent state after an abnormal restart. Step 5: Life management: During the data writing process, the life management module uses a wear leveling algorithm to ensure that the writing operation is evenly distributed to different areas of the storage medium, which helps to extend the service life of the storage device and reduce excessive wear in local areas; Step 6: Abnormal monitoring and recovery: The data persistence module continuously monitors the status of logs and checkpoints during system operation, promptly discovers and handles potential problems, and relies on log records and checkpoints to restore data when the system is abnormal or crashes, combined with data backup from redundant storage modules for comprehensive recovery.

[0012] The present invention has the following beneficial effects: 1. Improve performance: Through temporary storage in the cache module, data writing speed is accelerated, direct access to the main storage medium is reduced, and overall system performance is significantly improved.

[0013] 2. Enhanced data security: RAID technology or data replication mechanism of redundant storage modules provides an additional layer of data protection, reducing the risk of data loss.

[0014] 3. Optimize load balancing: The dynamic load balancing module balances writing tasks to avoid overloading of a single storage unit and improve the overall stability and response speed of the system.

[0015] 4. Fast data recovery: The logging and checkpoint technology of the data persistence module can quickly restore data when the system is abnormal, ensuring system reliability and data integrity.

[0016] 5. Extend the life of the device: The wear leveling algorithm of the life management module effectively disperses the write operation, reduces the local wear of the storage medium, and extends the service life of the device.

[0017] 6. Improve system stability: Through intelligent scheduling and continuous monitoring, the system can adjust and repair potential problems in a timely manner to maintain the stability and reliability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of the structure of a computer data writing system proposed by the present invention; Figure 2 The present invention is a flowchart of the steps of a method for writing computer data. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0020] Reference Figure 1-2 , a computer data writing system, comprising the following modules: Cache module: temporarily stores data to be written, uses non-volatile memory as a cache layer, and in the cache module, uses an LRU algorithm or an LFU algorithm to manage data in the cache, preloads commonly used data into the cache at startup, improves initial access speed, uses logging technology to ensure data persistence in the event of a power outage or system crash, regularly synchronizes data in the cache to the main storage device, reduces the amount of data that needs to be restored after an abnormal system restart, uses a multi-level cache architecture, and combines different types of storage media to meet different levels of needs, encrypts sensitive data in the cache, and implements redundant storage strategies, including mirroring or RAID; Redundant storage module: Enhance data security through redundant storage strategies, use RAID technology to achieve data redundancy, including RAID5 / 6 or data replication mechanism in distributed file systems. In the redundant storage module, RAID5 or RAID6 is used to achieve data redundancy, and the data replication mechanism in distributed file systems is used to achieve data redundancy. Checksums are used to detect and correct data errors, and parity information is distributed on multiple disks to support replacement of a disk failure. Block storage, object storage and file system services are provided, automatic repair and data replication are supported, parity bits are used to detect data errors, and data snapshots are created regularly to restore to a certain point in time when data is damaged or lost. Once a failed disk is detected, the system automatically marks it as offline and replaces it with a hot spare disk; Load balancing module: dynamically allocates write tasks to different storage units to avoid overloading of a single storage unit. It uses an intelligent scheduling algorithm based on software-defined storage and combines machine learning technology to predict and adjust load distribution. In the load balancing module, write tasks are dynamically allocated to different storage units according to real-time load conditions. It uses software-defined storage technology to manage and allocate storage resources through the software layer. It continuously monitors the load conditions of each storage unit, including indicators such as write speed and latency. It uses supervised learning algorithms to train models and predict load changes in different time periods, including random forests and gradient boosting trees. Data persistence module: adopts logging and checkpoint technology, through which data status can be quickly restored after abnormal system restart. In the data persistence module, detailed logs of all data write operations are recorded in real time, including transaction operations, modification records and system status. The log format is adopted to ensure that each write is first recorded in the log and then actually written to the storage medium. The complete data snapshot of the system is regularly saved, including the current data status and transaction progress, and the checkpoint interval is set to automatically generate data snapshots and save them in a stable storage area. In combination with logging and checkpoint technology, an intelligent recovery strategy is formulated, including the selection of the best recovery point and data reconstruction process. The consistency of the restored data status with the original status is checked through hash verification or data verification technology; Life management module: Uses a wear leveling algorithm to evenly distribute write operations.

[0021] The specific working principle of the present invention is as follows: The cache module is responsible for temporarily storing data to be written. It uses non-volatile memory as a cache layer to improve the speed of reading and writing data. The cache uses the LFU algorithm to manage data to optimize the utilization efficiency of the cache. When the system starts, it preloads commonly used data into the cache to improve the initial access speed. In order to maintain the persistence of data in the event of power outages or system crashes, the data to be written is recorded using logging technology, and the data in the cache is synchronized to the main storage device regularly to reduce the amount of data that needs to be restored after the system restarts abnormally. In addition, a multi-level cache architecture is combined with different types of storage media to meet different levels of needs, sensitive data in the cache is encrypted, and redundant storage strategies (such as mirroring or RAID) are implemented to enhance data security; The redundant storage module achieves data redundancy by adopting RAID technology (such as RAID5 or RAID6) or the data replication mechanism of the distributed file system. These technologies distribute parity information on multiple disks, not only supporting the replacement of failed disks, but also providing block storage, object storage and file system services, and supporting automatic repair and data replication. The system uses checksums to detect and correct data errors, and regularly creates data snapshots to ensure that the data can be restored to a certain point in time when it is damaged or lost. After the faulty disk is detected, it will be automatically marked as offline and replaced by a hot spare disk; The load balancing module is responsible for dynamically allocating write tasks to different storage units to avoid overloading a single storage unit. It uses the intelligent scheduling algorithm based on software-defined storage and combines machine learning techniques (such as random forest and gradient boosting tree) to predict and adjust load distribution, monitor the load of each storage unit in real time (including write speed, latency, etc.), and dynamically adjust the allocation of storage resources based on real-time data to improve the overall performance and responsiveness of the system. The data persistence module uses logging and checkpoint technology to ensure that data can be quickly restored after an abnormal system restart. It records all data write operations in detail, including transaction operations, modification records, and system status. Each write operation is first recorded in the log and then written to the storage medium. The system's complete data snapshots and transaction progress are saved regularly. The checkpoint interval is set to automatically generate data snapshots and save them in a stable storage area. In combination with logging and checkpoint technology, an intelligent recovery strategy is formulated, the best recovery point is selected, and data reconstruction is performed. The consistency of the restored data state with the original state is checked through hash verification or data verification technology. The life management module distributes write operations evenly to the storage medium through a wear leveling algorithm to reduce wear in specific areas and extend the life of the device. This approach ensures the efficiency, reliability, and durability of the storage system.

[0022] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A computer data writing system, It is characterized in that Includes the following modules: Cache module: temporarily stores data to be written, using non-volatile memory as a cache layer; Redundant storage module: Enhance data security through redundant storage strategies and use RAID technology to achieve data redundancy, including data replication mechanisms in RAID5 / 6 or distributed file systems; Load balancing module: dynamically distributes write tasks to different storage units to avoid overloading a single storage unit. It uses an intelligent scheduling algorithm based on software-defined storage and combines machine learning technology to predict and adjust load distribution. Data persistence module: uses logging and checkpointing technology to quickly restore data status after an abnormal system restart; Life management module: Uses a wear leveling algorithm to evenly distribute write operations.

2. A computer data writing system according to claim 1, It is characterized in that In the cache module, an LRU algorithm or an LFU algorithm is used to manage the data in the cache. Commonly used data is preloaded into the cache at startup to improve the initial access speed. In the event of a power outage or system crash, logging technology is used to ensure data persistence. The data in the cache is periodically synchronized to the main storage device to reduce the amount of data that needs to be restored after an abnormal system restart. A multi-level cache architecture is used in combination with different types of storage media to meet different levels of needs. Sensitive data in the cache is encrypted and a redundant storage strategy is implemented, including mirroring or RAID.

3. A computer data writing system according to claim 1, It is characterized in that In the redundant storage module, RAID5 or RAID6 is used to achieve data redundancy, and the data replication mechanism in the distributed file system is adopted to achieve data redundancy. Check codes are used to detect and correct data errors, and parity information is distributed on multiple disks. It supports replacing a disk failure, provides block storage, object storage and file system services, supports automatic repair and data replication, uses parity bits to detect data errors, and regularly creates data snapshots to restore to the state at a certain point in time when data is damaged or lost. Once a failed disk is detected, the system automatically marks it as offline and replaces it with a hot spare disk.

4. A computer data writing system according to claim 1, It is characterized in that In the load balancing module, write tasks are dynamically allocated to different storage units according to the real-time load conditions. Software-defined storage technology is adopted to manage and allocate storage resources through the software layer. The load conditions of each storage unit are continuously monitored, including indicators such as write speed and latency. A supervised learning algorithm is used to train the model to predict load changes in different time periods, including random forests and gradient boosting trees.

5. A computer data writing system according to claim 1, It is characterized in that In the data persistence module, detailed logs of all data writing operations are recorded in real time, including transaction operations, modification records and system status. The log format is used to ensure that each write is first recorded in the log and then actually written to the storage medium. The complete data snapshot of the system is regularly saved, including the current data status and transaction progress. The checkpoint interval is set, and the data snapshot is automatically generated and saved in a stable storage area. In combination with log recording and checkpoint technology, an intelligent recovery strategy is formulated, including the selection of the best recovery point and data reconstruction process. Through hash verification or data verification and technology, the consistency of the restored data state with the original state is checked.

6. A computer data writing method, based on the computer data writing system according to any one of claims 1 to 5, It is characterized in that The following steps are involved: Step 1, data reception and caching: After the data is received into the system, it is first written into the non-volatile memory in the cache module. The cache module temporarily stores the data to be written to reduce the direct access burden to the main storage medium and improve the writing speed; Step 2: Redundant storage and data protection: extract data from the cache module and send the data to the redundant storage module. The redundant storage module uses RAID technology or the data replication mechanism in the distributed file system to disperse and store the data in multiple storage units to ensure data redundancy and security. Step 3: Load balancing and scheduling: Before data is written to the redundant storage module, the load balancing module dynamically evaluates the load of each storage unit and uses an intelligent scheduling algorithm to assign data writing tasks to storage units with lighter loads to avoid overloading any single storage unit. Step 4: Data persistence processing: After the data is written, the data persistence module starts to record detailed logs of the write operation and creates checkpoints at appropriate times. Through logging and checkpoint technology, the system can quickly recover to the most recent consistent state after an abnormal restart. Step 5: Life management: During the data writing process, the life management module uses a wear leveling algorithm to ensure that the writing operation is evenly distributed to different areas of the storage medium, which helps to extend the service life of the storage device and reduce excessive wear in local areas; Step 6: Abnormal monitoring and recovery: The data persistence module continuously monitors the status of logs and checkpoints during system operation, promptly discovers and handles potential problems, and relies on log records and checkpoints to restore data when the system is abnormal or crashes, combined with data backup from redundant storage modules for comprehensive recovery.

Citation Information

Cited By

  • Software integration monitoring system based on modular architecture

    CN120336120A

  • A software integration level monitoring system based on a modular architecture

    CN120336120B