Database backup server and system based on HDFS technology

Abstract

The present invention provides a server and system for database backup based on HDFS technology, including: a main database server for managing a database; a slave database server for backing up the database in the main database server, including: a segmentation module for segmenting the backed-up database into a plurality of sub-databases; a plurality of generation modules, each generation module corresponding to managing a sub-database, and generating a database hash table for the corresponding sub-database, the database hash table is generated when the segmentation module segments the sub-database, and is synchronously updated when the sub-database is updated; a processing module for uniformly managing the database hash table, and formulating a partitioning strategy based on the data in the sub-database, notifying the generation module corresponding to the sub-database, so that the generation module further subdivides the database hash table corresponding to the sub-database into sub-database hash tables based on the partitioning strategy. The system balances high-load access and improves data processing capabilities.

Description

Technical Field

[0001] The present invention relates to the field of servers, and in particular to a server and system for database backup based on HDFS technology. Background Art

[0002] At present, large systems have complex applications that require the use of databases. As the core of information systems, databases play an important role. When faced with a large number of accesses, the bottleneck of the database will soon appear, and some important data have high requirements for the reliability of the database. Therefore, database backup to meet the needs of a large number of accesses has become the goal and direction of research. Summary of the invention

[0003] In view of this, the present invention provides a server and system for database backup based on HDFS technology to solve the problem of low database reliability and meet a large number of access requirements.

[0004] Some embodiments of the present application provide a server and system for database backup based on HDFS technology. The present application is introduced from multiple aspects below, and the embodiments and beneficial effects of the following aspects can be referenced to each other.

[0005] In a first aspect, the present invention provides a database backup system based on HDFS technology, comprising:

[0006] The main database server is used to manage the database;

[0007] The slave database server is used to back up the database in the master database server, including:

[0008] A splitting module, the splitting module is used to split the backed up database into a plurality of sub-databases;

[0009] A plurality of generating modules, each of which manages one of the sub-databases and generates a database hash table for the corresponding sub-database, wherein the database hash table is generated when the sub-database is segmented by the segmentation module and is synchronously updated when the sub-database is updated;

[0010] The processing module is used to uniformly manage the database hash table, formulate a partitioning strategy based on the data in the sub-database, and notify the generation module corresponding to the sub-database so that the generation module further subdivides the database hash table corresponding to the sub-database into sub-database hash tables based on the partitioning strategy.

[0011] According to an embodiment of the present invention, the processing module formulates a partitioning strategy based on the data in the sub-database, including:

[0012] The processing module divides users of different levels based on the user data in the sub-database, and screens out user IDs that meet the level number.

[0013] According to an embodiment of the first aspect of the present invention, the generating module further subdivides the database hash table corresponding to the sub-database into sub-database hash tables based on the partitioning strategy, including:

[0014] The generating module generates a user table according to the user IDs that are screened out and meet the level number.

[0015] According to an embodiment of the first aspect of the present invention, the slave database server further includes:

[0016] A compression module is used to compress the database in the main database server after backing it up.

[0017] According to an embodiment of the first aspect of the present invention, the slave database server adopts a PCIe series cache system.

[0018] According to an embodiment of the first aspect of the present invention, the master database server is connected to the slave database server via a heartbeat line, and the processing module is further used to monitor the master database server. When a failure of the master database server is detected, the slave database server is put into use as a backup server.

[0019] According to an embodiment of the first aspect of the present invention, the failure includes: one or more of a power failure of the primary database server, a main component failure, and a boot disk failure.

[0020] In a second aspect, the present application also discloses a database backup server based on HDFS technology, comprising:

[0021] A splitting module, the splitting module is used to split the backed up database into a plurality of sub-databases;

[0022] A plurality of generating modules, each of which manages one of the sub-databases and generates a database hash table for the corresponding sub-database, wherein the database hash table is generated when the sub-database is segmented by the segmentation module and is synchronously updated when the sub-database is updated;

[0023] The processing module is used to uniformly manage the database hash table, formulate a partitioning strategy based on the data in the sub-database, and notify the generation module corresponding to the sub-database so that the generation module further subdivides the database hash table corresponding to the sub-database into sub-database hash tables based on the partitioning strategy.

[0024] According to an embodiment of the second aspect of the present invention, the processing module formulates a partitioning strategy based on the data in the sub-database, including:

[0025] The processing module divides users of different levels based on the user data in the sub-database, and screens out user IDs that meet the level number.

[0026] According to an embodiment of the second aspect of the present invention, the generating module further subdivides the database hash table corresponding to the sub-database into sub-database hash tables based on the partitioning strategy, including:

[0027] The generation module generates a user table according to the user IDs that meet the level number. In a second aspect, the present application also provides an electronic device, including: a processor; and an all-flash array high-performance storage system, wherein the all-flash array high-performance storage system is a system of the first aspect embodiment.

[0028] The above technical solution of the present invention has at least one of the following beneficial effects:

[0029] The database backup server based on HDFS technology according to the embodiment of the present application can ensure the security and reliability of data, and can balance the high-load access volume of the main database, facilitate query, and increase access speed, thereby improving data processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of the structure of a database backup system based on HDFS technology according to an embodiment of the present invention;

[0031] Figure 2 A flowchart of a server backup process according to an embodiment of the present invention;

[0032] Figure 3 A server for database backup based on HDFS technology according to an embodiment of the present invention;

[0033] Figure 4 The figure is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The database backup system of the present application is described below in conjunction with the accompanying drawings.

[0036] refer to Figure 1 , Figure 1 The structural diagram of the database backup system is shown. The system includes: a main database server 110 and a slave database server 120. The main database server 110 is used to maintain the database. For example, the database can be edited, modified, added, and deleted in the main database server 110. The database can be one or more of a file information database, a picture database, etc. The slave database server 120 is used to back up the database in the main database server 110. When the main database server 110 fails, for example, the access volume is too high, and the load on the main database exceeds 80% of the load that can be carried, at this time, the slave database server 120 can be put into use as a backup server, thereby ensuring the security and reliability of the data, and balancing the problem of high-load access to the main database.

[0037] In one embodiment of the present invention, a real-time disaster response solution using a heartbeat line is used between the master database server and the slave database server. The slave database server monitors the master database server in real time. When the master database server fails, for example, the power supply of the master database server fails, the main component fails, the boot disk fails, etc. fails, the slave database server is used as a backup server to replace the master database, thereby effectively avoiding data loss and ensuring data security.

[0038] The backup process of the server for database backup based on HDFS technology in an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0039] refer to Figure 2 , Figure 2 FIG. 1 shows a flow chart of the server backup process. Figure 2 As shown, the flow chart includes:

[0040] S210, obtaining a database to be backed up. For example, the slave database server obtains the database to be backed up from the master database server and backs up the database.

[0041] S220, the backed up database is divided into a plurality of sub-databases. For example, the database may be divided according to the amount of data, or divided according to detailed categories, for example, according to picture classification, according to file classification, etc.

[0042] S230, generating a database hash table for the corresponding sub-database. The specific database hash table can be generated when the server divides the sub-database, and is synchronously updated when the sub-database is updated, so that each sub-database can have a database hash table. When the server receives a query task, it can find the corresponding data address in time through the database hash table, thereby improving the query efficiency and facilitating the unified management of the database.

[0043] S240, further subdividing the database hash table corresponding to the sub-database into sub-database hash tables based on the division strategy. For example, the server divides users into different levels based on the user data in the sub-database, for example, according to the user usage, or the priority of the user determined by the operator based on the payment situation, and screens out the user IDs that meet the number of levels. Based on the screened user IDs that meet the number of levels, the database hash table is further subdivided into multiple sub-database hash tables, such as user tables. This further refines the data in the database, facilitates unified management of the data, and realizes the usage experience of different users.

[0044] refer to Figure 3 , Figure 3 The structure diagram of a server for database backup based on HDFS technology is shown. The server 100 includes: a segmentation module 310 , several generation modules 320 and a processing module 330 .

[0045] The segmentation module 310 is used to segment the backed-up database into several sub-databases, each generation module 320 manages a sub-database and generates a database hash table for the corresponding sub-database. The database hash table is generated when the segmentation module 310 segments the sub-database and is updated synchronously when the sub-database is updated. The processing module 330 is used to uniformly manage the database hash table and formulate a partitioning strategy based on the data in the sub-database, and notify the generation module corresponding to the sub-database so that the generation module 310 further subdivides the database hash table corresponding to the sub-database into sub-database hash tables based on the partitioning strategy.

[0046] The server 100 according to the embodiment of the present application can be used as a backup to ensure data security, and the database is uniformly classified and managed, which is more conducive to querying and accessing data and avoiding overload of access.

[0047] In one embodiment of the present invention, the processing module 330 divides users into different levels based on the user data in the sub-database, and screens out user IDs that meet the level number.

[0048] In one embodiment of the present invention, the generating module 320 generates a user table based on the user IDs that are screened out and meet the level number.

[0049] In one embodiment of the present invention, the server 100 further includes a compression module 340, and the compression module 340 is used to compress the database in the main database server after backing it up.

[0050] In one embodiment of the present invention, the server may further include a communication module for communicating with the Web terminal, so as to enable the user to establish and expand the database through the Web terminal, so as to facilitate use.

[0051] The server 100 for database backup based on HDFS technology according to the embodiment of the present application ensures data security and reliability, and can balance the high-load access volume of the main database, facilitate query, increase access speed, and thus improve data processing performance.

[0052] refer to Figure 4 , Figure 4 1 is a block diagram of a device 1200 according to one embodiment of the present application. The device 1200 may include one or more processors 1201 coupled to a controller hub 1203. For at least one embodiment, the controller hub 1203 communicates with the processor 1201 via a multi-drop bus such as a front side bus (FSB), a point-to-point interface such as a Quick Path Interconnect (QPI), or a similar connection 1206. The processor 1201 executes instructions that control general types of data processing operations. In one embodiment, the controller hub 1203 includes, but is not limited to, a graphics memory controller hub (GMCH) (not shown) and an input / output hub (IOH) (which may be on a separate chip) (not shown), wherein the GMCH includes a memory and a graphics controller and is coupled to the IOH.

[0053] The device 1200 may also include a coprocessor 1202 and a memory 1204 coupled to the controller hub 1203. Alternatively, one or both of the memory and the GMCH may be integrated within the processor (as described in the present application), with the memory 1204 and the coprocessor 1202 directly coupled to the processor 1201 and the controller hub 1203, with the controller hub 1203 and the IOH being in a single chip. The memory 1204 may be, for example, a dynamic random access memory (DRAM), a phase change memory (PCM), or a combination of the two. In one embodiment, the coprocessor 1202 is a special-purpose processor, such as, for example, a high-throughput MIC processor (Many Integerated Core, MIC), a network or communication processor, a compression engine, a graphics processor, a general-purpose graphics processor (General Purpose Computing on GPU, GPGPU), or an embedded processor, etc. The optional properties of the coprocessor 1202 are indicated by dashed lines in Figure 4 middle.

[0054] The memory 1204, as a computer-readable storage medium, may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. For example, the memory 1204 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as one or more hard disk drives (HDD(s)), one or more compact disc (CD) drives, and / or one or more digital versatile disc (DVD) drives.

[0055] In one embodiment, the device 1200 may further include a network interface (Network Interface Controller, NIC) 1206. The network interface 1206 may include a transceiver for providing a radio interface for the device 1200 to communicate with any other suitable device (such as a front-end module, an antenna, etc.). In various embodiments, the network interface 1206 may be integrated with other components of the device 1200. The network interface 1206 may implement the functions of the communication unit in the above-mentioned embodiments.

[0056] The device 1200 may further include input / output (I / O) devices 1205. The I / O 1205 may include: a user interface designed to enable a user to interact with the device 1200; a peripheral component interface designed to enable peripheral components to interact with the device 1200; and / or sensors designed to determine environmental conditions and / or location information related to the device 1200.

[0057] It is worth noting that Figure 4 This is for illustrative purposes only. Figure 4 It is shown that the device 1200 includes multiple components such as a processor 1201, a controller hub 1203, and a memory 1204. However, in actual applications, the device using the methods of the present application may only include a part of the components of the device 1200, for example, it may only include the processor 1201 and the NIC 1206. Figure 4 The properties of optional devices are shown with dotted lines.

[0058] According to some embodiments of the present application, the memory 1204 as a computer-readable storage medium stores instructions, which, when executed on a computer, enable the system 1200 to perform the above-mentioned embodiments. Figure 2 The method shown can be specifically referred to the method in the above embodiment, which will not be described in detail here.

[0059] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0060] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0061] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0062] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0063] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0064] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0065] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0066] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A database backup system based on HDFS technology, characterized in that: include: The main database server is used to manage the database; A slave database server, used for backing up the database in the master database server, wherein the slave database server adopts a PCIe series high-speed cache system; The slave database server comprises: A segmentation module, the segmentation module is used to segment the backed-up database into a plurality of sub-databases, the segmentation module performs segmentation according to the amount of data or according to detailed categories; A plurality of generating modules, each of which manages one of the sub-databases and generates a database hash table for the corresponding sub-database, wherein the database hash table is generated when the sub-database is segmented by the segmentation module and is synchronously updated when the sub-database is updated; a processing module, for uniformly managing the database hash table, and formulating a partitioning strategy based on the data in the sub-database, wherein the partitioning strategy includes the processing module partitioning users of different levels based on the user data in the sub-database, and screening out user IDs that meet the number of levels, notifying a generation module corresponding to the sub-database so that the generation module further subdivides the database hash table corresponding to the sub-database into a plurality of sub-database hash tables based on the partitioning strategy, and the generation module generates a user table according to the screened out user IDs that meet the number of levels; A compression module, the compression module is used to compress the database in the main database server after backing it up; The master database server is connected to the slave database server via a heartbeat line. The processing module is also used to monitor the master database server. When a failure of the master database server is detected, the slave database server is put into use as a backup server.

2. The system according to claim 1, characterized in that The failure includes: one or more of the following: the load of the main database server exceeds a preset value, power failure, main component failure, and boot disk failure.

3. A database backup server based on HDFS technology, characterized in that: The database backup system applied to the HDFS technology described in claim 1 or claim 2, wherein the server comprises: A segmentation module, the segmentation module is used to segment the backed-up database into a plurality of sub-databases, the segmentation module performs segmentation according to the amount of data or according to detailed categories; A plurality of generating modules, each of which manages one of the sub-databases and generates a database hash table for the corresponding sub-database, wherein the database hash table is generated when the sub-database is segmented by the segmentation module and is synchronously updated when the sub-database is updated; A processing module is used to uniformly manage the database hash table and formulate a partitioning strategy based on the data in the sub-database, wherein the partitioning strategy includes the processing module partitioning users of different levels based on the user data in the sub-database, and screening out user IDs that meet the number of levels, notifying the generation module corresponding to the sub-database so that the generation module further subdivides the database hash table corresponding to the sub-database into multiple sub-database hash tables based on the partitioning strategy, and the generation module generates a user table based on the screened out user IDs that meet the number of levels.

Images