A distributed file system

Abstract

The present invention discloses a distributed file system, including a distributed database system, wherein the distributed database system includes a library and table module, and the library and table module is used to distribute database tables whose data volume exceeds a set threshold to multiple distributed database systems. The distributed database system of the distributed file system of the present invention can be used to solve the scalability problem of relational databases, and has the characteristics of being lightweight (stateless), flexible, stable, and efficient; the distributed database system is compatible with common database protocols and syntax, supports library and table sharding, smooth capacity expansion, service upgrade and downgrade, transparent read-write separation, and distributed transactions, and has the operation and maintenance management and control capabilities of the distributed database throughout its life cycle; the main application scenario of the distributed database system is in large-scale online data operations, and the operation efficiency is improved to the extreme through a split method that fits the business, effectively meeting the user's online business requirements for relational databases.

Description

Technical Field

[0001] The present invention particularly relates to a distributed file system. Background Art

[0002] The design of distributed file systems is based on the client / server model. A typical network may include multiple servers for multiple users to access. In addition, the peer-to-peer feature allows some systems to play the dual role of client and server. Computers manage and store data through file systems, and the amount of data that can be obtained is growing exponentially. The method of simply expanding the storage capacity of computer file systems by increasing the number of hard disks is not satisfactory in terms of capacity size, capacity growth rate, data backup, data security, etc.

[0003] Distributed file systems can effectively solve the problem of data storage and management: a file system fixed to a certain location can be expanded to any number of locations and multiple file systems, and numerous nodes form a file system network. Each node can be distributed in different locations, and communication and data transmission between nodes can be carried out through the network. When people use distributed file systems, they do not need to care about which node the data is stored on or from which node it is obtained. They only need to manage and store the data in the file system like using a local file system.

[0004] In order to overcome at least one of the defects of the existing distributed file system, such as database capacity bottleneck, expansion difficulty, and high cost, the present invention proposes an improved distributed file system. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed file system to address the deficiencies of the above-mentioned prior art.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A distributed file system is characterized by comprising a distributed database system, wherein the distributed database system comprises a library and table sharding module, and the library and table sharding module is used to distribute database tables whose data volume exceeds a set threshold into multiple distributed database systems.

[0008] Furthermore, the distributed database system includes a transparent read-write separation module, and the transparent read-write separation module is used to separate the read operation and the write operation for different distributed database systems.

[0009] Furthermore, the distributed database system includes a data storage smooth expansion module, and the data storage smooth expansion module is used to distribute the data tables to the nodes at the bottom layer when a new database node server is connected to the distributed database system.

[0010] Furthermore, the distributed database system includes a distributed operation and maintenance instruction set module, and the distributed operation and maintenance instruction set module is used to provide a distributed database operation and maintenance instruction set to achieve rapid discovery and location of problems.

[0011] Furthermore, the distributed database system has a globally unique digital sequence.

[0012] Furthermore, the distributed database system has a unified database account authority system.

[0013] Furthermore, the distributed database system adopts a distributed transaction suite.

[0014] Furthermore, the distributed database system includes a monitoring and alarm module, which is used for real-time monitoring and alarming of core resource indicators and database instance indicators.

[0015] Furthermore, the distributed database system supports one or more of a file channel, a database channel, a synchronous message channel, and an asynchronous message channel.

[0016] Furthermore, the distributed database system has a message queue, which is a professional message middleware of the distributed database system.

[0017] The distributed database system of the distributed file system of the present invention can be used to solve the scalability problem of relational databases, and has the characteristics of lightweight (stateless), flexibility, stability, and efficiency; the distributed database system is compatible with common database protocols and syntax, supports sub-library and sub-table, smooth capacity expansion, service upgrade and downgrade, transparent read-write separation, and distributed transactions, and has the operation and maintenance management capabilities of the entire life cycle of distributed databases; the main application scenario of the distributed database system is large-scale online data operations. By splitting in a way that fits the business, the operating efficiency is improved to the extreme, effectively meeting the user's online business requirements for relational databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a typical structural diagram of a distributed file system.

[0019] Figure 2 This is a functional architecture diagram of a distributed database system.

[0020] Figure 3 Data flow diagram for a distributed database system.

[0021] Figure 4 This is a schematic diagram of vertical data splitting.

[0022] Figure 5 Schematic diagram of horizontal data splitting.

[0023] Figure 6 This is a schematic diagram of transparent read-write separation.

[0024] Figure 7 This is a schematic diagram of smooth expansion of data storage.

[0025] Figure 8 A graphical example of distributed transactions.

[0026] Fig. 9 Another illustrated example of distributed transactions. DETAILED DESCRIPTION

[0027] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0028] The typical structure of a distributed file system is shown in Figure 1 The distributed database system of the distributed file system of the present invention can be used to solve the scalability problem of relational databases, and has the characteristics of lightweight (stateless), flexibility, stability, and efficiency; the distributed database system is compatible with common database protocols and syntax, supports sub-library and sub-table, smooth capacity expansion, service upgrade and downgrade, transparent read-write separation, and distributed transactions, and has the operation and maintenance management capabilities of the entire life cycle of distributed databases; the main application scenario of the distributed database system is large-scale online data operations. By splitting in a way that fits the business, the operating efficiency is improved to the extreme, effectively meeting the user's online business requirements for relational databases.

[0029] Distributed database systems mainly solve the following problems:

[0030] Single-machine database capacity bottleneck: As the amount of data and access volume grows, single-machine databases will encounter great challenges, and relying on hardware upgrades cannot completely solve the problem.

[0031] It is difficult to expand a single-machine database: Expanding the capacity of a traditional database often means service interruption, which makes it difficult to achieve zero or minimal business awareness.

[0032] Traditional databases are expensive to use: When business data and access volume increase to a certain amount, traditional databases need to rely on specific high-end storage and minicomputer equipment, and the cost curve rises rapidly.

[0033] The functional architecture of the distributed database system is shown in the figure below: Figure 2 shown.

[0034] Sub-library and table sharding: The main function is to distribute database tables with large data volumes into multiple databases. When access to this table occurs, each database will be responsible for a portion of the query, read, and write operations, thereby effectively dispersing the database access pressure. Therefore, the distributed database system includes a sub-library and table sharding module, which is used to distribute database tables with data volumes exceeding a set threshold into multiple distributed database systems.

[0035] Transparent read-write separation: The main function is to separate the operations of reading and writing the database into different database servers. For example, the first database server is a production server that provides external add, delete and modify services; the second database server mainly performs read operations. Therefore, the distributed database system includes a transparent read-write separation module, which is used to separate the read operations and write operations for different distributed database systems.

[0036] Data storage smooth expansion: The main function is to adopt a data smooth migration strategy to further distribute the data table to all nodes at the bottom layer when a new database node server is connected to the distributed database system. Therefore, the distributed database system includes a data storage smooth expansion module, which is used to distribute the data table to the nodes at the bottom layer when a new database node server is connected to the distributed database system.

[0037] Distributed operation and maintenance instruction set: The main function is to provide a distributed database operation and maintenance instruction set to achieve rapid discovery and location of problems. Therefore, the distributed database system includes a distributed operation and maintenance instruction set module, which is used to provide a distributed database operation and maintenance instruction set to achieve rapid discovery and location of problems.

[0038] Globally unique digital sequence: The main function is to meet the special requirements for primary keys or unique keys when using a distributed database. Therefore, the distributed database system has a globally unique digital sequence.

[0039] Unified database account permission system: The main function is that the distributed database system ensures the management of user permissions by establishing user tables and permission tables. At the same time, a series of intermediate tables are established within the system to solve the one-to-many and many-to-many relationships between users and roles. Therefore, the distributed database system has a unified database account permission system.

[0040] Distributed transaction: The main function is that the distributed database system adopts a distributed transaction suite. The distributed transaction suite is a high-performance, highly reliable, and easy-to-access distributed transaction middleware used to solve data consistency problems in a distributed environment. Therefore, the distributed database system adopts a distributed transaction suite.

[0041] Monitoring and alarm: The main function is that the distributed database system supports real-time monitoring and alarming of core resource indicators and database instance indicators. Therefore, the distributed database system includes a monitoring and alarm module, which is used for real-time monitoring and alarming of core resource indicators and database instance indicators.

[0042] Message channel: The main function is that the channel is divided into four modes, including file channel, database channel, synchronous message channel, and asynchronous message channel. File channel is to transmit images, texts and other files through file stream; database channel is the connection mode of various databases: including ODBC database interface, JDBC database interface, OLE DB database interface, ADO database interface, ADO.NET database interface, etc. Synchronous message channel means that the information on the synchronous channel contains the information necessary for the mobile station to calibrate the time, and also contains network air interface correction, system data, and paging channel data rate information; asynchronous message channel means that the sender can start sending characters at any time and in frames. Therefore, the distributed database system supports one or more of file channel, database channel, synchronous message channel, and asynchronous message channel.

[0043] SQL routing: The main function is that in the sharding mode, the distributed database system will distribute SQL statements to the underlying sharding tables for execution based on the split key (i.e. split field) and SQL semantics. After the execution is completed, the distributed database system will merge the data obtained from each sharding table and return it to the user.

[0044] Message queue: The main function is to be a professional message middleware for distributed database systems. It is also an important message component in the platform. It is a proven message delivery backbone that provides all-round and multi-purpose data transmission. It mainly solves problems such as application coupling, asynchronous messages, and traffic clipping, and realizes high-performance, high-availability, scalable and eventually consistent architecture. Therefore, the distributed database system has a message queue, which is a professional message middleware for distributed database systems.

[0045] The data flow diagram of the distributed database system is as follows Figure 3 shown.

[0046] For the construction of the entire distributed database system, there are two types of business systems that need to be connected, namely the existing systems of various committees, offices, departments, and bureaus and the newly built systems. The existing systems and newly built systems transmit data through four major channels, namely synchronous data channel, asynchronous data channel, database channel and file channel.

[0047] A data collection process is carried out for each channel, that is, relevant data is collected according to the needs of the system itself and the needs of users. Then each channel will initiate a request for data and locate the route, and then index it through the distributed database search engine to provide information retrieval functions.

[0048] When entering the distributed database search engine, if data is needed, the SQL statement will be distributed through SQL, and the queried data will first be stored in a cache. After caching, the data will be placed in the message queue, and then the message queue will use the push function to push the queried data to the system that requires the data.

[0049] After the push, the data is sent to the platform through a business continuity process component, which includes an interface for calling the data. Through the data interface, data push is provided to other applications or services.

[0050] 1. Sub-library and sub-table

[0051] The distributed database system horizontally splits the data tables with large data volumes into each distributed database system database node at the back end. These split databases are called shards, and the tables in the shards are called shard tables. At the same time, the distributed database system can generate splitting rules according to the splitting key, and distribute the database tables with large data volumes to multiple database nodes. When access to this table occurs, each database node will be responsible for part of the query operation, thereby effectively dispersing the database access pressure and improving the operating efficiency of the entire database. When expanding the system, you only need to increase the number of shards horizontally and migrate related data to increase the overall capacity of the distributed database.

[0052] The split key is the sub-database and sub-table field, so it is divided into sub-database key and sub-table key.

[0053] The split key currently only supports a single field. The distributed database system splits the data horizontally into each sub-database in the backend according to the value of the sub-database key. Data with the same key value will be located in the same distributed database system database node. The sub-table key means that each logical table can define its own sub-table key, and data with the same key value will be located in the same distributed data table.

[0054] The table sharding method can be divided into the same database sharding and different database sharding. The same database sharding means that all the shards are in one database. Since the table names in the database cannot be repeated, the data table names need to be changed to different names. On the other hand, for different database sharding, since the shards are in different databases, the same table name can be used at this time. The table sharding function can solve the performance bottleneck problems such as CPU, memory, file IO, network IO, etc., and it is relatively simple to handle.

[0055] Database sharding is a data splitting method that combines vertical and horizontal methods to distribute data services and data storage to multiple database servers.

[0056] like Figure 4 As shown in the figure, vertical data splitting means dividing data according to functional modules and placing them in different databases and servers. However, with this method, as the access pressure increases, the read and write operations continue to increase, and the pressure on the database is absolutely increasing, sometimes it may approach the limit, and the amount of data is also growing rapidly. At this time, according to the vertical data splitting method, you only need to consider separating the read and write operations and placing different data in different databases according to the business.

[0057] like Figure 5 As shown in the figure, horizontal splitting means that when a single machine bottleneck is encountered after vertical splitting, horizontal splitting can be used. The difference from vertical splitting is that horizontal splitting splits the same table into different databases.

[0058] Compared with vertical splitting, horizontal splitting does not classify the data in the table, but distributes it to multiple different databases according to certain rules of a certain field, and each table contains a part of the data. In simple terms, we can understand the horizontal splitting of data as the splitting according to the data rows, that is, some rows in the table are split into one database, and some other rows are split into other databases.

[0059] Horizontal splitting does not have the performance bottleneck of single-database big data and high concurrency, and is transparent to applications, so applications can access it without much modification. Splitting according to reasonable splitting rules basically avoids cross-database join operations, which improves the stability and load capacity of the system.

[0060] 2. Transparent read-write separation

[0061] Simply put, read-write separation means that the operations of reading and writing to the database are separated and corresponded to different database servers. This can effectively reduce the pressure on the database and the pressure on IO. The master database provides write operations, and the slave database provides read operations. In fact, in many systems, read operations are the main operations. When the master database performs a write operation, the data must be synchronized to the slave database, so as to effectively ensure the integrity of the database.

[0062] The schematic diagram of transparent read-write separation is as follows Figure 6 shown.

[0063] After the master database is synchronized to the slave database, the slave database is generally composed of multiple databases, so as to achieve the purpose of reducing pressure. Read operations should be assigned to servers according to the pressure of the server, rather than simply randomly assigned.

[0064] The storage and access of massive data can be improved by performing read-write separation on the database to improve data processing capabilities. The read-write separation solution is characterized by the database generating multiple copies, and the database write operations are concentrated on one database, while some read operations can be decomposed to other databases. In this way, as long as the cost of data replication is paid, the database processing pressure can be decomposed to multiple databases, thereby greatly improving data processing capabilities.

[0065] The basic principle of read-write separation is to let the master database handle transactional add, modify, and delete operations (INSERT, UPDATE, DELETE), while the slave database handles SELECT query operations. Database replication is used to synchronize changes caused by transactional operations to the slave databases in the cluster.

[0066] For this type of database where the read rate is greater than the write rate and the increase in data volume is not very obvious, it is recommended to adopt the read-write separation + cache mode. Therefore, the first thing that read-write separation solves is the splitting of frequently changing data, such as: user login time, record user login IP. This type of information can be separated and recorded in the cache of the persistence class to increase the retrieval speed. At the same time, different partitions can be separated to different disks to balance I / 0 and improve the performance of the entire system. In addition, it also has great advantages in data maintenance. When dumping historical data, you only need to back up the data that needs to be dumped and dump it to the tape. There is no need to back up the entire table.

[0067] 3. Smooth expansion of data storage

[0068] Smooth expansion is a horizontal expansion method that smoothly migrates sub-databases to newly added underlying storage. In practice, it increases the number of instances to increase the overall data storage capacity, migrates sub-databases to newly added instances, and reduces the processing pressure of a single instance.

[0069] This situation is used when the underlying storage corresponding to the logical library has reached a physical bottleneck and needs to be expanded horizontally. For example, if the disk margin is close to 30%, it can be improved through smooth expansion.

[0070] After adding a distributed database system database node, you need to migrate the sub-database. The migration task will not change the data in the original database and will not affect the business. It only synchronizes the data in the database to be migrated to the newly added distributed database system database node. Before switching, you can roll back and abandon this smooth expansion operation.

[0071] See the schematic diagram of smooth expansion of data storage for Figure 7 .

[0072] After the migration is complete, you can also switch tasks. The switching task will switch the read and write traffic to the newly added distributed database system instance. The whole process will be completed within a few minutes. During the switching process, except for one or two flash outages, the service will not be affected. If you do not want to affect the business, you can also set it to switch during the business downturn.

[0073] After the switch is completed, users can also migrate the migrated sub-databases on the original distributed database system instance. The cleanup task is also an asynchronous task and can be viewed in the task schedule.

[0074] At present, smooth expansion is achieved by moving databases. If the expansion reaches a certain level and a sub-database exceeds the capacity of a single distributed database system database node and cannot be further smoothly expanded, the number of sub-databases can be increased and the capacity can be expanded again. At this time, the data will be re-calculated and redistributed by HASH.

[0075] 4. Distributed operation and maintenance instruction set

[0076] The distributed database system provides a distributed database operation and maintenance instruction set to achieve rapid discovery and location of problems, and can monitor various website performance indicators at any time.

[0077] At the same time, the system can provide about 23 optimization rules, including reducing the number of HTTP requests, weighing the number of DNS lookups, reducing the size of cookies, using CDN, etc. Through these rules, developers and managers can make corresponding optimizations.

[0078] 5. Globally unique digital sequence

[0079] In a distributed architecture, generating a unique serial number is a common problem when designing a system, especially when using sharding tables in a database. When the database is divided into several sharding tables, how to quickly get a unique serial number is a common problem.

[0080] Distributed database systems support distributed globally unique numeric sequences to meet special business requirements for primary keys or unique keys when using distributed databases. The main goal is to generate globally unique and ordered increasing numeric sequences, which are often used to generate values ​​of primary key columns, unique index columns, etc. The generated values ​​are natural number sequences, but are not guaranteed to be continuous and monotonically increasing. Custom step sizes, maximum values, and cyclic and non-cyclic utilization are supported.

[0081] 6. Unified database account permission system

[0082] The distributed database system supports a unified data account and permission system to ensure the security of account operations.

[0083] The distributed database system ensures the management of user permissions by establishing user tables and permission tables. At the same time, a series of intermediate tables are established within the system to resolve the one-to-many and many-to-many relationships between users and roles.

[0084] 7. Distributed Transactions

[0085] The distributed database system uses a distributed transaction suite, which is a high-performance, highly reliable, and easily accessible distributed transaction middleware used to solve data consistency issues in a distributed environment.

[0086] Data consistency includes strong consistency, weak consistency, and eventual consistency.

[0087] Strong consistency means that after the update operation is completed, any subsequent processes or accesses will return the latest updated value. This is the most user-friendly, that is, the user is guaranteed to read what he wrote last time. According to the CAP theory, this implementation requires sacrificing availability.

[0088] Weak consistency means that the system does not guarantee that subsequent processes or threads will return the latest updated value. After the data is successfully written, the system does not promise that the latest written value can be read immediately, nor does it promise a specific time before it can be read.

[0089] Eventual consistency refers to a specific form of weak consistency. The system guarantees that if there are no subsequent updates, the system will eventually return the value of the last update operation. If there are no failures, the duration of the inconsistency window is mainly affected by communication delays, system load, and the number of replicas.

[0090] Distributed database systems solve the scalability problem of single-machine relational databases by splitting data horizontally through sharding. However, after the original single database and single table are split, the data of the single table is dispersed into tables of multiple databases. The original changes to multiple rows of data in a single table may become changes to data in multiple databases and tables, that is, single-machine local transactions become distributed transactions. Adding a distributed transaction suite to the distributed database system realizes the atomicity of multi-database transaction operations and solves the problem of cross-database transactions in distributed databases.

[0091] Distributed transactions are transactions in which the participants, servers supporting transactions, resource servers, and transaction managers are located on different nodes of different distributed systems. Simply put, a large operation is composed of different small operations, which are distributed on different servers and belong to different applications. Distributed transactions need to ensure that these small operations are either all successful or all failed. In essence, distributed transactions are to ensure data consistency in different databases.

[0092] Distributed transactions can be applied to the database's sharding function. When sharding the database, data consistency must be guaranteed, so distributed transactions will be used.

[0093] See the example of a specific diagram of distributed transactions. Figure 8 .

[0094] On the other hand, distributed transactions can be applied at the SOA level. The so-called SOA level is the application of SOA, which is simply the service-oriented business. For example, a single machine originally supported the entire website, but now the entire website is disassembled and separated into A center, B center, and C center. For A center, there is a dedicated database to store A's information, B center also has a dedicated database to store B's information, and C center also has a dedicated database to store C's information. At this time, if you want to operate A and C at the same time, then the A database and C database will be involved. In order to ensure data consistency, distributed transactions are required.

[0095] For another example of a specific diagram of distributed transactions, see Fig. 9 .

[0096] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, which all belong to the protection scope of the present invention.

Claims

1. A distributed file system, characterized in that: A distributed database system is included, wherein the distributed database system includes a library and table sharding module, and the library and table sharding module is used to distribute database tables whose data volume exceeds a set threshold to multiple distributed database systems; When access to this table occurs, each database node will be responsible for a portion of the query read and write operations; In the sharding mode, through SQL routing, the distributed database system will distribute SQL statements to the underlying stored shards for execution according to the split key and SQL semantics; after the execution, the distributed database system will merge the data obtained from each shard and return it to the user.

2. The distributed file system according to claim 1, characterized in that: The distributed database system comprises a transparent read-write separation module, and the transparent read-write separation module is used to separate the read operation and the write operation for different distributed database systems.

3. The distributed file system according to claim 1, characterized in that: The distributed database system comprises a data storage smooth expansion module, and the data storage smooth expansion module is used to distribute the data table to the nodes at the bottom layer when a new database node server is connected to the distributed database system.

4. The distributed file system according to claim 1, characterized in that: The distributed database system includes a distributed operation and maintenance instruction set module, which is used to provide a distributed database operation and maintenance instruction set to achieve rapid discovery and location of problems.

5. The distributed file system according to claim 1, wherein: The distributed database system has a globally unique digital sequence.

6. The distributed file system according to claim 1, characterized in that: The distributed database system has a unified database account authority system.

7. The distributed file system according to claim 1, characterized in that: The distributed database system adopts a distributed transaction suite.

8. The distributed file system according to claim 1, wherein: The distributed database system includes a monitoring and alarm module, which is used for real-time monitoring and alarming of core resource indicators and database instance indicators.

9. The distributed file system according to claim 1, characterized in that: The distributed database system supports one or more of a file channel, a database channel, a synchronous message channel, and an asynchronous message channel.

10. The distributed file system according to claim 1, characterized in that: The distributed database system has a message queue, which is a professional message middleware of the distributed database system.

Images