Database system, data processing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data storage, and discloses a database system, a data processing method, device and equipment and a storage medium, and the system comprises a data write-in operation record storage component which is used for storing data write-in operation records; each operation record corresponds to one random identifier; extending database service; the extended database service on the master node is also used for obtaining the latest N operation records from the slave node after the slave node is recovered from the fault, and matching the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the master node according to the sequence of time from near to far; and if the random identifiers are matched with the random identifiers of N continuous first operation records in the master node, sending a second operation record after the N first operation records in the master node to the slave node, so that the slave node performs data writing according to the second operation record. The reliability and the write-in performance of the database system can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a database system, a data processing method, a device, a equipment and a storage medium. Background Art

[0002] The main focus of database fault handling is how to maintain data consistency and persistence. In related technologies, databases usually use WAL (Write-Ahead Logging) technology to ensure data consistency and persistence. Before each data is written, the change record is first written to the log file. Then the data is written to the memory (or directly to the disk). In this way, even if the system fails during the writing process, the data can be restored through the log file. However, with this processing method, if the log file itself is lost or damaged during the failure, data recovery cannot be performed, and the reliability is low. Summary of the invention

[0003] In view of this, the present invention provides a database system, a data processing method, an apparatus, a device and a storage medium to solve the problem of low reliability of the database system.

[0004] In a first aspect, the present invention provides a database system, wherein each node of the database system comprises:

[0005] Extended database software development kit, which is used to provide application programming interfaces for database systems;

[0006] Election service, used to designate the master node in the database system;

[0007] The extended database service is used to read and write data and operation records on the node; the extended database service on the master node is also used to receive data written by the user and send it to all slave nodes;

[0008] A data storage component for storing data;

[0009] A data writing operation record storage component is used to store data writing operation records; each of the operation records corresponds to a random identifier;

[0010] The extended database service on the master node is also used to obtain the latest N operation records from the data write operation record storage component of the slave node after the slave node recovers from a failure, and match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data write operation record storage component of the master node in order from recent to far in time. If the random identifiers match the N consecutive first operation records in the data write operation record storage component of the master node, the data of the master node is written into the operation record storage component, and the second operation record after the N first operation records is sent to the slave node, so that the slave node writes data according to the second operation record.

[0011] In an optional implementation, the operation record of the data writing operation record storage component is stored according to a plurality of pairs of index files and data files;

[0012] The index file is a file used to store metadata of the operation record, and the data file is a file used to store specific operation record data content;

[0013] The index file and the data file both store the random identifier of the operation record, and the random identifier of the same operation record is consistent.

[0014] In an optional implementation, the data structure of the index file includes: an index file integrity flag, a version number, and metadata of the operation record;

[0015] The metadata of the operation record includes: the offset of the operation record in the data file, the total size of the operation record and the random identifier.

[0016] In an optional implementation, the data structure of the data file includes: a data file integrity mark, a version number and the operation record;

[0017] The data structure of the operation record includes: a start synchronization identifier, the random identifier, a sequence number of the operation record, a data length, data content of the operation record, an end synchronization identifier and a check code;

[0018] The data content of the operation record includes: data writing time, data key length, data key content, data writing type, data value length, data value content, an identifier indicating whether there is an old value, the length of the old value and the content of the old value.

[0019] In an optional implementation, the index file and the data file have the same file name;

[0020] The file name includes the start sequence number, the number of operation records and the file creation time;

[0021] The starting sequence number is a value that gradually increases from 1. The value of the starting sequence number of a file is equal to the starting sequence number of the previous file plus the number of operation records. The number of operation records in each file is the same, and the starting sequence number is the same as the sequence number of the first operation record in the data file.

[0022] In a second aspect, the present invention provides a method for processing database data, which is applied to any database system in the first aspect, and the method comprises:

[0023] After the slave node fails and recovers, the extended database service of the master node obtains the latest N operation records from the data writing operation record storage component of the slave node;

[0024] Match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data writing operation record storage component of the master node in order of time from recent to far;

[0025] If the random identifier matches N consecutive first operation records in the data write operation record storage component of the master node, the data of the master node is written into the operation record storage component, and the second operation record after the N first operation records is sent to the slave node, so that the slave node writes data according to the second operation record.

[0026] In an optional implementation, the data of each node in the database system is written into an operation record storage component, and the operation record is stored according to a plurality of pairs of index files and data files; the index file is a file used to store metadata of the operation record, and the data file is a file used to store specific operation record data content; the random identifier of the operation record is stored in both the index file and the data file, and the random identifier of the same operation record is consistent;

[0027] Before acquiring the latest N operation records from the data writing operation record storage component of the slave node, the method further includes:

[0028] The extended database service of the slave node that fails obtains the latest generated target index file and target data file from the data writing operation record storage component after the slave node is started;

[0029] Determining whether the target index file and the target data file are closed normally according to the index file integrity mark in the target index file and the data file integrity mark in the target data file;

[0030] If the target index file and the target data file are not closed normally, deleting the target index file in the data writing operation record storage component and generating a new index file;

[0031] The content in the target data file is traversed, and index entries are generated and written into the new index file.

[0032] In a third aspect, the present invention provides a database data processing device, applied to any database system in the first aspect, the device comprising:

[0033] An acquisition module, used for acquiring the latest N operation records from the data writing operation record storage component of the slave node after the slave node fails and recovers;

[0034] A matching module, used to match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data writing operation record storage component of the master node in order of time from recent to far;

[0035] A sending module is used to write the data of the master node into the operation record storage component if the random identifier matches N consecutive first operation records in the data write operation record storage component of the master node, and send the second operation record after the N first operation records to the slave node, so that the slave node writes data according to the second operation record.

[0036] In a fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the database data processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0037] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the database data processing method of the first aspect or any corresponding embodiment thereof.

[0038] In a sixth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the database data processing method of the first aspect or any corresponding embodiment thereof.

[0039] For a database system, effective data synchronization between nodes (including normal data storage and data recovery operations) is the key to the implementation of the database system, and plays a vital role in achieving data reliability and efficient access. The present invention stores data write operation records in each node in the database system, and provides a specific data recovery method. In particular, when searching for data synchronization points, there is no need to compare the entire operation record, but to compare by random identifiers of the operation record, which can further improve the efficiency of data recovery, so that the node can efficiently recover the data storage during the failure, thereby improving the reliability of the database system.

[0040] In addition, the database system provided by the embodiment of the present invention also has high availability. Specifically, in the master-slave mode, the system is available as long as the master node survives. When the master node fails, the slave node can also provide data reading functions. In the cluster mode, as long as more than half of the nodes in the system survive, the entire system is effective. The database system provided by the embodiment of the present invention is also easy to expand, because for each node in the system, it is itself a KV database system, which supports all functional features of the database system and can handle all services. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a schematic diagram of the architecture of a database system according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of file names and data structures of index files and data files according to an embodiment of the present invention;

[0044] Figure 3 is a flowchart of a method for processing data in a database according to an embodiment of the present invention;

[0045] Figure 4 It is a schematic diagram of a process of a master node performing data recovery on a fault recovery slave node according to an embodiment of the present invention;

[0046] Figure 5 is a flow chart of a self-check recovery operation performed by a failed slave node after startup according to an embodiment of the present invention;

[0047] Figure 6is a structural block diagram of a data processing device for a database according to an embodiment of the present invention;

[0048] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] In this embodiment, a database system is provided. Figure 1 Schematic diagram of the architecture of a database system according to an embodiment of the present invention. Figure 1 As shown, each node of the database system includes:

[0051] The extended database software development kit ExDBSDK is used to provide an application programming interface (API) for the database system, simplify and encapsulate the database usage functions, and finally provide static / dynamic link libraries for database users (Clients) to use;

[0052] ElectServer, the election service, is used to specify the master node in the database system;

[0053] The extended database service ExDBServer is used to read and write data and operation records on this node; the extended database service on the master node is also used to receive data written by the user and send it to all slave nodes;

[0054] The data storage component DataStorage, i.e. the storage engine, is used to store data. The data storage component DataStorage is where the database system actually stores data, and the ExDBServer on this node performs data read and write operations. Under normal circumstances, the DataStorage data on all nodes in the database system is the same. Under abnormal circumstances, the data stored in the DataStorage on the master node is the latest and most complete.

[0055] The data writing operation record storage component ActionLog, i.e., the data writing operation record storage engine, is used to store data writing operation records, which can also be referred to as operation records, including the writing operations of setting and deleting and the written data content, which is the basis for the database system to efficiently perform data recovery; each of the operation records corresponds to a random identifier, which can be a random number;

[0056] The extended database service on the master node is also used to obtain the latest N operation records from the data write operation record storage component of the slave node after the slave node recovers from a failure, and match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data write operation record storage component of the master node in order from recent to far in time. If the random identifiers match the N consecutive first operation records in the data write operation record storage component of the master node, the data of the master node is written into the operation record storage component, and the second operation record after the N first operation records is sent to the slave node, so that the slave node writes data according to the second operation record.

[0057] The database system in the embodiment of the present invention can be a KV database, which is a non-relational database. It stores data in the form of key-value pairs, where the key is unique and the value can be any data type. The data writing of the KV database includes setting and deletion. The setting is divided into creation and update. When performing a setting operation, if the Key does not exist, it is a creation operation. If the Key exists, it is an update operation, and the new Value will replace the old Value as a whole.

[0058] In the embodiment of the present invention, the node is also a node server, and each node server runs the above five service modules. For each single node server, it is a relatively independent database system that supports all functional features of the database and can process any operation of the database user (Client).

[0059] The above database system supports single-node, cluster, and master-slave operation modes, and can be built according to business needs. In both cluster and master-slave operation modes, data write operations are performed by the master service storing the data in the local node and sending it to the slave node for storage and backup, that is, data writing is dominated by the master node. In the master-slave mode, the master node only needs to write successfully to consider the write operation successful. In the cluster mode, the master node sends data to the slave nodes concurrently for writing. As long as it detects that half of the slave nodes except itself have written successfully, it considers the write operation successful. This method greatly improves the data writing performance.

[0060] Specifically, the master node in the system is specified according to the different operating mode configurations of the database system. A single-node database system does not need to be specified, and the master node of the active-standby mode database system is specified by the configuration. The election service ElectServer specifies the extended database service ExDBServer of the corresponding node as the main service of the system based on the read configuration parameters. The cluster mode database system ElectServer selects a node ExDBServer as the main service of the system through an election algorithm.

[0061] In addition, the operation records stored in the data writing operation record storage component ActionLog are the latest and valid records of the data stored in DataStorage, and the invalid old operation records stored will be detected regularly and deleted in real time. That is, the embodiment of the present invention can dynamically delete invalid old operation records in real time, and only retain a few recently valid operation records, so the space occupation can be ignored, and there is no need to reserve a certain storage space for the storage of operation records. Alternatively, the data writing operation record can automatically delete the oldest operation record according to the number of operation records and the size of the disk space.

[0062] In some specific implementations, the operation records of the data writing operation record storage component are stored according to a plurality of pairs of index files *.bidx and data files *.blog; the size of the final full *.bidx file can be 128KB by default, and the size of the full *.blog file can be 4M by default, and the file size can be set according to actual conditions;

[0063] The index file is a file used to store metadata of the operation record, and the data file is a file used to store specific operation record data content;

[0064] The index file and the data file both store the random identifier of the operation record, and the random identifier of the same operation record is consistent.

[0065] For a database system, effective data synchronization between nodes (including normal data storage and data recovery operations) is the key to the implementation of the database system, and plays a vital role in achieving data reliability and efficient access. The present invention stores data write operation records in each node in the database system, and provides a specific data recovery method. In particular, when searching for data synchronization points, there is no need to compare the entire operation record, but to compare by random identifiers of the operation record, which can further improve the efficiency of data recovery, so that the node can efficiently recover the data storage during the failure, thereby improving the reliability of the database system.

[0066] In addition, the database system provided by the embodiment of the present invention also has high availability. Specifically, in the master-slave mode, the system is available as long as the master node survives. When the master node fails, the slave node can also provide data reading functions. In the cluster mode, as long as more than half of the nodes in the system survive, the entire system is effective. The database system provided by the embodiment of the present invention is also easy to expand, because for each node in the system, it is itself a KV database system, which supports all functional features of the database system and can handle all services.

[0067] In some specific implementations, such as Figure 2 As shown, the data structure of the index file includes: an index file integrity flag fourcc, a version number version, and metadata idxItem of the operation record. If the index file is full and closed normally, the index file integrity flag fourcc is BLOG_IDX_FCC; if the index file is not full but closed abnormally during the writing process, the index file integrity flag fourcc is 0. The version number version is used for compatibility and upgrade of subsequent versions.

[0068] The metadata idxItem of the operation record includes: the offset offset of the operation record in the data file, the total size size of the operation record and the random identifier randTag.

[0069] In some specific implementations, the data structure of the data file includes: a data file integrity flag fourcc, a version number, and the operation record logItem. If the data file is full and closed normally, the data file integrity flag fourcc is BLOG_IDX_FCC; if the data file is not full but closed abnormally during the writing process, the data file integrity flag fourcc is 0. The version number version is used for compatibility and upgrade of subsequent versions.

[0070] The data structure of the operation record logItem includes: a starting synchronization identifier bfourcc, whose value is BLOG_DATA_FCC_B; the random identifier randtag, which is used to delete the cyclic redundancy check (CRC) and the operation record sequence positioning match; the serial number serinum of the operation record, the serial number of the first operation record is the same as the starting serial number of the file name; the data length datalen; the data content data of the operation record; the end synchronization identifier efourcc, whose value is BLOG_DATA_FCC_E; the check code sumcrc, which is the sum code of all other field data except this field, used to check the operation record data.

[0071] The data content of the operation record includes: data writing time ltWrote, length keyLen of the data key (KEY), content key of the data key (KEY), type of data writing isSet (whether setting or deletion), length valLen of the data value (VALUE) (if it is a deletion operation, the value is 0), content value of the data value (VALUE) (if it is a deletion operation, the value is empty), an identifier existOld indicating whether there is an old value (old VALUE) (if the data is modified, the old VALUE will exist), length oldLen of the old value (old VALUE) (the new data value is 0) and content oldValue of the old value (old VALUE) (the new data value is empty).

[0072] In some specific implementations, the index file *.bidx and the data file *.blog are one-to-one, and the file names of the index file and the data file are the same.

[0073] The file name includes a starting sequence number, the number of operation records, and the file creation time; the format of the file name may be, for example, "starting sequence number% number of operation records% file creation time". The number of operation records is used to indicate how many operation records are stored in the current file. The file creation time indicates the time when the file was created, and the files can be sorted according to the time and the adjacent files before and after a file can be quickly found.

[0074] The starting sequence number is a value that increases gradually from 1. The starting sequence number of a file (index file *.bidx, data file *.blog) is equal to the starting sequence number of the previous file plus the number of operation records. For example Figure 2 In the example, the starting sequence number 000,0000,0000,0045,5020%10567%2024-07-05,03-03-44.902.bidx file is the sum of the starting sequence number 000,0000,0000,0044,4453%10567%2024-07-05,01-41-06.336.bidx of the previous file 000,0000,0000,0044,4453 and the number of operation records 10567. This naming method is very helpful for data validity detection and fast positioning of the starting point of data recovery.

[0075] The number of operation records in each file is the same, and the starting sequence number is the same as the sequence number of the first operation record in the data file.

[0076] In addition, the data write operation record storage component ActionLog in the embodiment of the present invention is also used to store the number of failures of each node of the database system. After a node fails once, the number of failures corresponding to the node is added by 1 and sent to the master node, and synchronized to the slave node (which may not include the failed slave node) by the master node extended database service ExDBServer. When the master node needs to be re-determined, the election service ElectServer determines the master node according to the number of failures of each node, and generally selects the node with the least number of failures as the master node. The backup master node can also be selected according to the number of failures of the node. When the master node fails, the backup master node serves as the master node, and generally selects the node with the second least number of failures as the backup master node.

[0077] According to an embodiment of the present invention, an embodiment of a method for processing data in a database is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of executable computer instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0078] In this embodiment, a method for processing database data is provided, which can be used in any of the above-mentioned database systems. Figure 3 is a flow chart of a method for processing data in a database according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0079] Step S301, after the slave node fails and recovers, the extended database service ExDBServer of the master node obtains the latest (that is, the latest) N operation records from the data write operation record storage component of the slave node. Specifically, the extended database service ExDBServer of the master node can read the *.bidx file being written by the failed slave node in reverse order to obtain the latest (that is, the latest) N operation records. Wherein, N is an integer greater than or equal to 1, for example, it can be 8.

[0080] Step S302, the random identifiers (i.e., random numbers) corresponding to the N operation records are matched with the random identifiers corresponding to the operation records stored in the data write operation record storage component of the master node in order from recent to far in time. This step is to find the data synchronization point by comparing the random numbers of the operation records recorded in the *.bidx file of the slave node with the corresponding *.bidx file on the master node. This step is also specifically performed by the extended database service ExDBServer of the master node.

[0081] Step S303, if it matches the random identifier of N consecutive first operation records in the data write operation record storage component of the master node, the data of the master node is written into the operation record storage component, and the second operation record after the N first operation records is sent to the slave node, so that the slave node writes data according to the second operation record. In this embodiment, if N consecutive randTags are matched, it will be considered that the synchronization point is found. When the master node finds the data synchronization point, it will read the operation record of the *.blog file from the synchronization point, and then perform data recovery operations on the failed slave node according to the normal data writing process. After catching up with the normal write operation record of the master node, the master node will write data according to the normal data writing process. This step is also specifically performed by the extended database service ExDBServer of the master node.

[0082] The data writing in the database system of the embodiment of the present invention is all led by the master node, including the normal data storage process and the data recovery process after the failed node is restored.

[0083] In addition, after the failed slave node starts and completes the self-check recovery operation, it will first register with the master node. The master node will execute the above data recovery process only after receiving the registration information of the slave node.

[0084] In a specific embodiment, Figure 4 As shown in the figure, the process of data recovery by the master node for the failure recovery slave node is as follows:

[0085] 1) After the fault recovery node (slave node) completes self-check recovery, it initiates registration with the master node;

[0086] 2) After receiving the registration information from the slave node, the master node obtains the *.bidx file being written by the registered node and reads the operation records stored in the *.bidx file in reverse order;

[0087] 3) The master node compares the random number of the operation record information returned by the registered node (i.e. the random identifier mentioned above) with the random number of the operation record stored in the corresponding *.bidx file on its own node to find the data synchronization point;

[0088] 4) If the data synchronization point is found, execute step 5); if no data synchronization point is found, continue to execute step 3);

[0089] 5) The master node reads the operation record information of the corresponding *.blog file from the data synchronization point, and then writes the data to the registration node according to the data writing process;

[0090] 6) After receiving the data writing information, the registered node will store it in DataStorage and ActionLog according to the normal data writing process and data operation recording process respectively;

[0091] 7) The master node determines whether the data synchronization recovery is completed. If it is completed, it executes step 8). If it is not completed, it continues to execute step 5);

[0092] 8) The master node completes the data synchronization recovery operation, and subsequently writes data to the registered node according to the normal data writing operation;

[0093] 9) Repeat step 8) repeatedly.

[0094] In some optional specific implementations, the data of each node in the database system is written into an operation record storage component, and the operation record is stored according to a plurality of pairs of index files and data files; the index file is a file used to store metadata of the operation record, and the data file is a file used to store specific operation record data content; the index file and the data file both store random identifiers of the operation records, and the random identifiers of the same operation record are consistent;

[0095] Before acquiring the latest N operation records from the data writing operation record storage component of the slave node, the method further includes:

[0096] The extended database service of the slave node that fails obtains the latest generated target index file and target data file from the data writing operation record storage component after the slave node is started;

[0097] Determining whether the target index file and the target data file are closed normally according to the index file integrity mark in the target index file and the data file integrity mark in the target data file;

[0098] If the target index file and the target data file are not closed normally, deleting the target index file in the data writing operation record storage component and generating a new index file;

[0099] The content in the target data file is traversed, and index entries are generated and written into the new index file.

[0100] In the embodiment of the present invention, the index file *.bidx and the data file *.blog used to store operation records in the ActionLog storage component are the basis for the database system to recover data. In order to ensure that an unexpected node failure does not affect the correctness of data recovery, when each node is started, the index file *.bidx and the data file *.blog generated latest in the ActionLog storage component of this node must be detected and processed for data recovery (i.e., self-check and recovery operation).

[0101] In the embodiment of the present invention, the data recovery operation of fault recovery includes the self-check recovery operation of the node itself after startup, which can greatly improve the reliability of data.

[0102] In a specific implementation, Figure 5 As shown, the self-check recovery operation process performed by the failed slave node after startup can be specifically as follows:

[0103] 1) When the ExDBServer service starts, it obtains the latest generated *bidx and *blog files from the ActionLog storage component, and reads the value of the completion flag fourcc in *bidx and *blog to determine whether the file is closed normally;

[0104] 2) When the value of fourcc is normal closure, execute step 5). When the value of fourcc is abnormal closure, delete the *.bidx file and create a new *.bidx file corresponding to the *.blog file.

[0105] 3) Traverse and read *.blog files to generate index entries and write the index entries into *.bidx files;

[0106] 4) Check if the *.blog file has been read. If it has been read, go to the next step. If it has not been read, go to step 3.

[0107] 5) Determine whether the current *.blog file is full, if it is full, execute step 6), if not, execute step 7);

[0108] 6) Close the *.bidx and *.blog files currently being written, and create new *.bidx and *.blog files;

[0109] 7) Continue to receive new data and write it into *.blog files and *.bidx files;

[0110] 8) Loop through operations 5), 6), and 7).

[0111] In the present embodiment, a data processing device for a database is also provided, which is used to implement the above method embodiment and preferred implementation mode, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0112] This embodiment provides a data processing device for a database, such as Figure 6 As shown, applied to any one of the database systems in the above embodiments, the data processing device of the database includes:

[0113] The acquisition module 601 is used to acquire the latest N operation records from the data writing operation record storage component of the slave node after the slave node recovers from the failure;

[0114] A matching module 602 is used to match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data writing operation record storage component of the master node in order of time from recent to far;

[0115] The sending module 603 is used to write the data of the master node into the operation record storage component if the random identifier matches N consecutive first operation records in the data write operation record storage component of the master node, and send the second operation record after the N first operation records to the slave node, so that the slave node writes data according to the second operation record.

[0116] In some optional implementations, the data of each node in the database system is written into an operation record storage component, and the operation record is stored according to a plurality of pairs of index files and data files; the index file is a file used to store metadata of the operation record, and the data file is a file used to store specific operation record data content; the random identifier of the operation record is stored in both the index file and the data file, and the random identifier of the same operation record is consistent;

[0117] The data processing device of the above database also includes:

[0118] A file acquisition module, used for acquiring the latest generated target index file and target data file from the data writing operation record storage component after the slave node is started;

[0119] A determination module, used to determine whether the target index file and the target data file are closed normally according to the index file integrity mark in the target index file and the data file integrity mark in the target data file;

[0120] A first generating module, configured to delete the target index file in the data writing operation record storage component and generate a new index file if the target index file and the target data file are not closed normally;

[0121] The second generating module is used to traverse the content in the target data file and generate index entries to write into the new index file.

[0122] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0123] The data processing device of the database in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0124] The embodiment of the present invention also provides a computer device having the above Figure 6 A data processing device for the database shown.

[0125] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device is a server of a database system, and the computer device includes: one or more processors 10, a memory 20, and an interface for connecting various components, including a high-speed interface and a low-speed interface. Each component utilizes different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0126] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0127] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0128] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0129] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0130] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0131] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0132] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0133] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A database system, characterized in that: Each node of the database system includes: Extended database software development kit, which is used to provide application program interfaces for database systems; Election service, used to designate the master node in the database system; The extended database service is used to read and write data and operation records on the node; the extended database service on the master node is also used to receive data written by the user and send it to all slave nodes; A data storage component for storing data; A data writing operation record storage component is used to store data writing operation records; each of the operation records corresponds to a random identifier; The extended database service on the master node is also used to obtain the latest N operation records from the data write operation record storage component of the slave node after the slave node recovers from a failure, and match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data write operation record storage component of the master node in order from recent to far in time. If the random identifiers match the N consecutive first operation records in the data write operation record storage component of the master node, the data of the master node is written into the operation record storage component, and the second operation record after the N first operation records is sent to the slave node, so that the slave node writes data according to the second operation record.

2. The database system according to claim 1, characterized in that: The operation records of the data writing operation record storage component are stored according to a plurality of pairs of index files and data files; The index file is a file used to store metadata of the operation record, and the data file is a file used to store specific operation record data content; The index file and the data file both store the random identifier of the operation record, and the random identifier of the same operation record is consistent.

3. The database system according to claim 2, characterized in that: The data structure of the index file includes: an index file integrity mark, a version number and metadata of the operation record; The metadata of the operation record includes: the offset of the operation record in the data file, the total size of the operation record and the random identifier.

4. The database system according to claim 2, characterized in that: The data structure of the data file includes: a data file integrity mark, a version number and the operation record; The data structure of the operation record includes: a start synchronization identifier, the random identifier, a sequence number of the operation record, a data length, data content of the operation record, an end synchronization identifier and a check code; The data content of the operation record includes: data writing time, data key length, data key content, data writing type, data value length, data value content, an identifier indicating whether there is an old value, the length of the old value and the content of the old value.

5. The database system according to claim 2, characterized in that: The index file and the data file have the same file name; The file name includes the start sequence number, the number of operation records and the file creation time; The starting sequence number is a value that gradually increases from 1. The value of the starting sequence number of a file is equal to the starting sequence number of the previous file plus the number of operation records. The number of operation records in each file is the same, and the starting sequence number is the same as the sequence number of the first operation record in the data file.

6. A method for processing data in a database, characterized in that: The method applied to the database system according to any one of claims 1 to 5 comprises: After the slave node fails and recovers, the extended database service of the master node obtains the latest N operation records from the data writing operation record storage component of the slave node; Match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data writing operation record storage component of the master node in order of time from recent to far; If the random identifier matches N consecutive first operation records in the data write operation record storage component of the master node, the data of the master node is written into the operation record storage component, and the second operation record after the N first operation records is sent to the slave node, so that the slave node writes data according to the second operation record.

7. The method according to claim 6, characterized in that The data of each node in the database system is written into an operation record storage component, and the operation record is stored according to a plurality of pairs of index files and data files; the index file is a file used to store metadata of the operation record, and the data file is a file used to store specific operation record data content; the random identifier of the operation record is stored in both the index file and the data file, and the random identifier of the same operation record is consistent; Before acquiring the latest N operation records from the data writing operation record storage component of the slave node, the method further includes: The extended database service of the slave node that fails obtains the latest generated target index file and target data file from the data writing operation record storage component after the slave node is started; Determining whether the target index file and the target data file are closed normally according to the index file integrity mark in the target index file and the data file integrity mark in the target data file; If the target index file and the target data file are not closed normally, deleting the target index file in the data writing operation record storage component and generating a new index file; The content in the target data file is traversed, and index entries are generated and written into the new index file.

8. A data processing device for a database, characterized in that: The database system applied to any one of claims 1 to 5, wherein the device comprises: An acquisition module, used for acquiring the latest N operation records from the data writing operation record storage component of the slave node after the slave node fails and recovers; A matching module, used to match the random identifiers corresponding to the N operation records with the random identifiers corresponding to the operation records stored in the data writing operation record storage component of the master node in order of time from recent to far; A sending module is used to write the data of the master node into the operation record storage component if the random identifier matches N consecutive first operation records in the data write operation record storage component of the master node, and send the second operation record after the N first operation records to the slave node, so that the slave node writes data according to the second operation record.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the database data processing method according to any one of claims 6 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the database data processing method according to any one of claims 6 to 7.

Images