Distributed database construction method and system

Abstract

The invention relates to the technical field of database construction, and discloses a distributed database construction method and system.The distributed database construction system comprises a database demand analysis module, a database fragmentation system, a database analysis module and a database testing module.The specific demand of a distributed database is defined through the database demand module; according to the method, a database framework is set, spatial data is fragmented through a database fragmentation system according to a selected fragmentation strategy, and then a fault recovery system and a data consistency maintenance system are tested through a database analysis module and a database test module. Data in the database are maintained through the fault recovery system and the data consistency maintenance system, when the distributed database encounters various faults in the operation process and data damage is caused, fault nodes can be automatically detected and repaired, the data are recovered from backup, and protection of the system to the data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of database construction, and in particular to a distributed database construction method and system. Background Art

[0002] Distributed database refers to the use of computer high-speed networks to combine multiple physically dispersed databases into a logically unified large database. Its basic idea is to distribute the data in the original centralized database to multiple data storage nodes to increase the total amount of data storage and improve the efficiency of concurrent access.

[0003] When existing distributed databases encounter various failures, the internal data of the database is easily lost. After losing data, the existing distributed databases can only restore data from backups. When encountering failures, the backup data will also be damaged, resulting in poor data recovery effects. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a distributed database construction method and system, which solves the problems raised in the above background technology.

[0005] To solve the above technical problems, according to one aspect of the present invention, more specifically, a distributed database construction system is provided, including a database demand analysis module, a database sharding system, a database analysis module and a database testing module, characterized in that the database demand analysis module is used to clarify the specific requirements of the distributed database, and analyze the characteristics and access patterns of the demand data to provide a basis for subsequent architecture design and technology selection, the database sharding system is used to shard the spatial data according to the selected sharding strategy, and store each shard on a different node, the database analysis module is used to perform real-time monitoring and log analysis on the operating status of the database, and the database testing module is used to test the function, performance and stability of the database system.

[0006] Furthermore, the following steps are included:

[0007] S1. Database requirements analysis: clarify the data volume, data type, access frequency, consistency requirements, availability requirements, and fault recovery strategies required by the database, and analyze the characteristics and access patterns of the data based on the requirements;

[0008] S2. Set up the database architecture: Set up the database architecture of the sharded cluster mode, and perform distributed transformation on the STR algorithm to complete distributed index construction and data insertion;

[0009] S2.1. Set up a PostgreSQL database management system within the sharded cluster database architecture and configure the corresponding hardware resources;

[0010] S2.2, setting up hard disk drives, columnar storage formats, and data distribution strategies within the sharded cluster database architecture;

[0011] S2.3. Design metadata management mechanisms for managing data models and index information;

[0012] S3, combine genetic algorithm to optimize parallel computing framework task allocation, complete through population initialization, fitness function design, genetic operation execution, task allocation plan determination

[0013] S4, data sharding: according to the characteristics of the data and business requirements, the spatial data is sharded according to the selected sharding strategy, and each shard is stored on a different node;

[0014] S5. Database analysis and testing: Real-time monitoring and log analysis of the database's operating status, and testing of the database system's fault recovery system and data consistency maintenance system;

[0015] S5.1. Test passed: Deploy the database system to the production environment and conduct continuous monitoring and maintenance;

[0016] S5.2. Test failed: Maintain the fault recovery system and data consistency maintenance system of the database system, and test the database analysis module and database test module again after maintenance.

[0017] Furthermore, in S4, according to the characteristics of the data and business requirements, the database sharding system stores a large amount of data in multiple nodes through horizontal sharding and vertical sharding;

[0018] Horizontal sharding: divide the global relationship into several non-overlapping subsets by row according to certain conditions, and each subset is a fragment of the relationship;

[0019] Vertical sharding: Divide the attribute set of each global relationship into several subsets, perform projection operations on the subsets, and each projection is a vertical shard.

[0020] Furthermore, in S5, the fault recovery system of the database test is to ensure the stability and reliability of the system. When the distributed database encounters various faults during operation, resulting in data damage, it can automatically detect and repair the faulty nodes and restore the data from the backup.

[0021] Furthermore, in S5, the data consistency maintenance system of the database test is to improve the availability and fault tolerance of the distributed database. When a node fails, data can be restored from other nodes. At the same time, the consistency of the internal data of the database is guaranteed by the consistency maintenance system.

[0022] Furthermore, in S2, the distributed transformation of the STR algorithm to complete the distributed index construction includes the following steps:

[0023] S1, Minimum Bounding Rectangle (MBR)

[0024] MBR=(x 1 ,y 1 , x 2 ,y 2 )

[0025] S2. Spatial range area: Assuming the spatial object is "O", the calculation formula for the spatial range area of ​​spatial object O is:

[0026] S=(x 2 -x 1 )(y 2 -y 1 )

[0027] In the formula, S is the area of ​​the spatial object O, x 1 ,y 1 , x 2 ,y 2 They are nodes in the spatial object respectively;

[0028] S3. Spatial overlap calculation

[0029]

[0030] In the formula, R space is the spatial overlap, X overlap is the overlap length in the X direction, Y overpal is the overlapping length in the Y direction; S N is the spatial range area of ​​node N; where:

[0031] X overlap =max(0,min(x 2 , x n 2)-max(x 1 , x n 1))

[0032] Y overlap =max(0,min(y 2 ,y n 2)-max(y 1 ,y n 1));

[0033] S4. Time overlap calculation

[0034]

[0035] In the formula, Rtime is the time overlap; T overlap is the overlapping time interval of node N, T N is the length of the time interval of node N;

[0036] S5. Complete data insertion based on STR

[0037] Select nodes with large spatial overlap and temporal overlap and not full for insertion. If multiple nodes meet the conditions, further screening is performed according to certain strategies.

[0038] When the selected node is full, a node split operation is required, and the split strategy used is linear split;

[0039] First, calculate the median of the MBR of all spatial objects in the node in the x and y directions, denoted as x median and median , and then divide the spatial object in the node into two parts: one part is the x of MBR 2 Less than x median The other part is the x of MBR 1 Greater than or equal to x median The spatial object is divided similarly in the y direction, so that the original node is split into two new nodes, and their MBR and time interval information are adjusted.

[0040] Furthermore, in step S3, optimizing parallel computing framework task allocation in combination with genetic algorithm includes the following steps:

[0041] S1. Define the fitness function

[0042]

[0043] In the formula, U is the resource utilization, t is the task execution time, and f 1 (x) and f 2 (x) are the adaptability function, the adaptability function f 1 (x) The larger the fitness value, the shorter the maximum execution time of all tasks. The fitness function f 2 (x) The larger the fitness value, the higher the resource utilization rate;

[0044] S2, Task Individual Coding: The individual coding adopts integer coding, where one individual represents the number of the computing node to which the task is assigned;

[0045] S3. Optimizing adaptive functions

[0046] For two individuals X a =(X a 1, X a2, ..., X a 2N) and X b =(X b 1, X b 2, ..., X b N), randomly select an intersection point k (1<=k<=N-1).

[0047] After crossover, we get a new individual X a’ and X b’ :

[0048] X a =(X a 1, X a 2, ..., X a k,X b (k+1), X b (k+2),...,X b n)

[0049] X b =(X b 1, X b 2, ..., X b k,X a (k+1), X a (k+2),...,X a n)

[0050] Based on the single-point crossover formula, the adaptive function results of various combinations are statistically analyzed to continuously optimize the self-use function.

[0051] The beneficial effects of a distributed database construction method and system of the present invention are:

[0052] The present invention clarifies the specific requirements of the distributed database through a database requirement module, and analyzes the characteristics and access modes of the data according to the requirements, so as to set the database architecture, and slices the spatial data according to the selected sharding strategy through a database sharding system, and stores each shard on a different node. After that, the fault recovery system and the data consistency maintenance system are tested through a database analysis module and a database test module. After the test passes, the database system can be deployed to a production environment, and continuously monitored and maintained. In this database system, the data inside the database is maintained through a fault recovery system and a data consistency maintenance system. When the distributed database encounters various faults during operation and the data is damaged, the faulty node can be automatically detected and repaired, and the data can be restored from the backup, thereby increasing the system's protection of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0054] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0056] like Figure 1 As shown, according to one aspect of the present invention, a distributed database construction system is provided, including a database requirement analysis module, a database sharding system, a database analysis module and a database testing module, characterized in that the database requirement analysis module is used to clarify the specific requirements of the distributed database, and according to the characteristics and access patterns of the requirement analysis data, provide a basis for subsequent architecture design and technology selection, the database sharding system is used to shard the spatial data according to the selected sharding strategy, and store each shard on a different node, the database analysis module is used to perform real-time monitoring and log analysis on the operating status of the database, and the database testing module is used to test the function, performance and stability of the database system.

[0057] In this embodiment, the following steps are included:

[0058] S1. Database requirements analysis: clarify the data volume, data type, access frequency, consistency requirements, availability requirements, and fault recovery strategies required by the database, and analyze the characteristics and access patterns of the data based on the requirements;

[0059] S2. Set up the database architecture: Set up the database architecture of the sharded cluster mode, and perform distributed transformation on the STR algorithm to complete distributed index construction and data insertion;

[0060] S2.1. Set up a PostgreSQL database management system within the sharded cluster database architecture and configure the corresponding hardware resources;

[0061] S2.2, setting up hard disk drives, columnar storage formats, and data distribution strategies within the sharded cluster database architecture;

[0062] S2.3. Design metadata management mechanisms for managing data models and index information;

[0063] S3, combine genetic algorithm to optimize parallel computing framework task allocation, complete through population initialization, fitness function design, genetic operation execution, task allocation plan determination

[0064] S4, data sharding: according to the characteristics of the data and business requirements, the spatial data is sharded according to the selected sharding strategy, and each shard is stored on a different node;

[0065] S5. Database analysis and testing: Real-time monitoring and log analysis of the database's operating status, and testing of the database system's fault recovery system and data consistency maintenance system;

[0066] S5.1. Test passed: Deploy the database system to the production environment and conduct continuous monitoring and maintenance;

[0067] S5.2. Test failed: Maintain the fault recovery system and data consistency maintenance system of the database system, and test the database analysis module and database test module again after maintenance.

[0068] In this embodiment, in S4, according to the characteristics of the data and business requirements, the database sharding system stores a large amount of data in multiple nodes through horizontal sharding and vertical sharding;

[0069] Horizontal sharding: divide the global relationship into several non-overlapping subsets by row according to certain conditions, and each subset is a fragment of the relationship;

[0070] Vertical sharding: Divide the attribute set of each global relationship into several subsets, perform projection operations on the subsets, and each projection is a vertical shard.

[0071] In this embodiment, in S5, the fault recovery system of the database test is to ensure the stability and reliability of the system. When the distributed database encounters various faults during operation, resulting in data damage, it can automatically detect and repair the faulty nodes and restore the data from the backup.

[0072] In this embodiment, in S5, the data consistency maintenance system of the database test is to improve the availability and fault tolerance of the distributed database. When a node fails, data can be restored from other nodes. At the same time, the consistency of the internal data of the database is guaranteed by the consistency maintenance system.

[0073] In this embodiment, in S2, the distributed transformation of the STR algorithm to complete the distributed index construction includes the following steps:

[0074] S1, Minimum Bounding Rectangle (MBR)

[0075] MBR=(x 1 ,y 1 ,x 2 ,y 2 )

[0076] S2. Spatial range area: Assuming the spatial object is "O", the calculation formula for the spatial range area of ​​spatial object O is:

[0077] S=(x 2 -x 1 )(y 2 -y 1 )

[0078] In the formula, S is the area of ​​the spatial object O, x 1 ,y 1 ,x 2 ,y 2 They are nodes in the spatial object respectively;

[0079] S3. Spatial overlap calculation

[0080]

[0081] In the formula, R space is the spatial overlap, X overlap is the overlap length in the X direction, Y overpal is the overlapping length in the Y direction; S N is the spatial range area of ​​node N; where:

[0082] X overlap =max(0,min(x 2 , x n 2)-max(x 1 , x n 1))

[0083] Y overlap =max(0,min(y 2 ,y n 2)-max(y 1 ,y n 1));

[0084] S4. Time overlap calculation

[0085]

[0086] In the formula, R time is the time overlap; T overlap is the overlapping time interval of node N, T N is the length of the time interval of node N;

[0087] S5. Complete data insertion based on STR

[0088] Select nodes with large spatial overlap and temporal overlap and not full for insertion. If multiple nodes meet the conditions, further screening is performed according to certain strategies.

[0089] When the selected node is full, a node split operation is required, and the split strategy used is linear split;

[0090] First, calculate the median of the MBR of all spatial objects in the node in the x and y directions, denoted as x median and median , and then divide the spatial object in the node into two parts: one part is the x of MBR 2 Less than x median The other part is the x of MBR 1 Greater than or equal to x median The spatial object is divided similarly in the y direction, so that the original node is split into two new nodes, and their MBR and time interval information are adjusted.

[0091] In this embodiment, in step S3, optimizing parallel computing framework task allocation in combination with a genetic algorithm comprises the following steps:

[0092] S1. Define the fitness function

[0093]

[0094]

[0095] In the formula, U is the resource utilization, t is the task execution time, and f 1 (x) and f 2 (x) are the adaptability function, the adaptability function f 1 (x) The larger the fitness value, the shorter the maximum execution time of all tasks. The fitness function f 2 (x) The larger the fitness value, the higher the resource utilization rate;

[0096] S2, Task Individual Coding: The individual coding adopts integer coding, where one individual represents the number of the computing node to which the task is assigned;

[0097] S3. Optimizing adaptive functions

[0098] For two individuals X a =(X a 1, X a 2, ..., X a 2N) and X b =(Xb 1 , Xb 2 , ..., X b N), randomly select an intersection point k (1<=k<=N-1).

[0099] After crossover, we get a new individual X a’ and X b’ :

[0100] X a =(X a 1, Xa 2, ..., X a k,X b (k+1), X b (k+2),...,X b n)

[0101] X b =(X b 1, X b 2, ..., X b k,X a (k+1), X a (k+2),...,X a n)

[0102] Based on the single-point crossover formula, the adaptive function results of various combinations are statistically analyzed to continuously optimize the self-use function.

[0103] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention also fall within the protection scope of the present invention.

Claims

1. A distributed database construction system, comprising a database demand analysis module, a database sharding system, a database analysis module and a database testing module, characterized in that: The database requirement analysis module is used to clarify the specific requirements of the distributed database, and analyze the characteristics and access patterns of the data according to the requirements to provide a basis for subsequent architecture design and technology selection. The database sharding system is used to shard the spatial data according to the selected sharding strategy, and store each shard on a different node. The database analysis module is used to monitor the operating status of the database in real time and perform log analysis. The database testing module is used to test the function, performance and stability of the database system.

2. A distributed database construction method according to claim 1, characterized in that: The following steps are involved: S1. Database requirements analysis: clarify the data volume, data type, access frequency, consistency requirements, availability requirements, and fault recovery strategy required by the database, and analyze the characteristics and access patterns of the data based on the requirements; S2. Set up the database architecture: Set up the database architecture of the sharded cluster mode, and perform distributed transformation on the STR algorithm to complete distributed index construction and data insertion; S2.

1. Set up a PostgreSQL database management system within the sharded cluster database architecture and configure the corresponding hardware resources; S2.2, setting up hard disk drives, columnar storage formats, and data distribution strategies within the sharded cluster database architecture; S2.

3. Design metadata management mechanisms for managing data models and index information; S3, combine genetic algorithm to optimize parallel computing framework task allocation, complete through population initialization, fitness function design, genetic operation execution, task allocation plan determination S4, data sharding: according to the characteristics of the data and business requirements, the spatial data is sharded according to the selected sharding strategy, and each shard is stored on a different node; S5. Database analysis and testing: Real-time monitoring and log analysis of the database's operating status, and testing of the database system's fault recovery system and data consistency maintenance system; S5.

1. Test passed: Deploy the database system to the production environment and conduct continuous monitoring and maintenance; S5.

2. Test failed: Maintain the database system's fault recovery system and data consistency maintenance system, and test the database analysis module and database test module again after maintenance.

3. A distributed database construction method according to claim 2, characterized in that: In S4, according to the characteristics of the data and business requirements, the database sharding system stores a large amount of data in multiple nodes through horizontal sharding and vertical sharding; Horizontal sharding: divide the global relationship into several non-overlapping subsets by row according to certain conditions, and each subset is a fragment of the relationship; Vertical sharding: Divide the attribute set of each global relationship into several subsets, perform projection operations on the subsets, and each projection is a vertical shard.

4. A distributed database construction method according to claim 2, characterized in that: In S5, the fault recovery system of the database test is to ensure the stability and reliability of the system. When the distributed database encounters various faults during operation, resulting in data damage, it can automatically detect and repair the faulty nodes and restore the data from the backup.

5. A distributed database construction method according to claim 1, characterized in that: In S5, the data consistency maintenance system of the database test is to improve the availability and fault tolerance of the distributed database. When a node fails, data can be restored from other nodes. At the same time, the consistency of the internal data of the database is guaranteed by the consistency maintenance system.

6. A distributed database construction method according to claim 1, characterized in that: In S2, the distributed transformation of the STR algorithm to complete the distributed index construction includes the following steps: S1, Minimum Bounding Rectangle (MBR) MBR=(x1,y1,x2,y2) S2. Spatial range area: Assuming the spatial object is "O", the calculation formula for the spatial range area of ​​spatial object O is: S=(x2-x1)(y2-y1) In the formula, S is the range area of ​​the spatial object O, and x1, y1, x2, y2 are the nodes in the spatial object respectively; S3. Spatial overlap calculation In the formula, R space is the spatial overlap, X overlap is the overlap length in the X direction, Y overpal is the overlapping length in the Y direction; S N is the spatial range area of ​​node N; where: X overlap =max(0,min(x2,x n 2)-max(x1,x n 1)) AND overlap =max(0,min(y2,y n 2)-max(y1,y n 1)); S4. Time overlap calculation In the formula, R time is the time overlap; T overlap is the overlapping time interval of node N, T N is the length of the time interval of node N; S5. Complete data insertion based on STR Select nodes with large spatial overlap and temporal overlap and not full for insertion. If multiple nodes meet the conditions, further screening is performed according to certain strategies. When the selected node is full, a node split operation is required, and the split strategy used is linear split; First, calculate the median of the MBR of all spatial objects in the node in the x and y directions, denoted as x median and median , and then divide the spatial object in the node into two parts: one part is the MBR x2 is less than x median The other part is the space object of MBR x1 is greater than or equal to x median The spatial object is divided similarly in the y direction, so that the original node is split into two new nodes, and their MBR and time zones are adjusted. Time information.

7. A distributed database construction method according to claim 1, characterized in that: In step S3, optimizing parallel computing framework task allocation in combination with genetic algorithm includes the following steps: S1. Define the fitness function In the formula, U is the resource utilization rate, t is the task execution time, f1(x) and f2(x) are the adaptability functions, respectively. The larger the fitness value of the adaptability function f1(x), the shorter the maximum execution time of all tasks. The larger the fitness value of the adaptability function f2(x), the higher the resource utilization rate. S2, Task Individual Coding: The individual coding adopts integer coding, where one individual represents the number of the computing node to which the task is assigned; S3. Optimizing adaptive functions For two individuals X a =(X a 1,X a 2,...,X a 2N) and X b =(X b 1,X b 2,...,X b N), randomly select an intersection point k (1<=k<=N-1). After crossover, we get a new individual X a’ and X b’ : X a =(X a 1,X a 2,...,X a k,X b (k+1),X b (k+2),...,X b n) X b =(X b 1,X b 2,...,X b k,X a (k+1),X a (k+2),...,X a n) Based on the single-point crossover formula, the adaptive function results of various combinations are statistically analyzed to continuously optimize the self-use function.

Images