Data processing method and device, computer device and storage medium

Abstract

The application relates to a data processing method and device, computer equipment and a storage medium, the method comprising: a computer equipment calling a computing service node to receive a database operation request, if the database operation request is a data write request, constructing a pre-write log based on the data write request; calling a storage service node to divide the pre-write log, obtaining a plurality of pre-write log segments, and performing storage processing on the plurality of pre-write log segments. In the scheme, the storage service and the computing service in the database are isolated, the coupling of the storage service and the computing service is reduced, the influence on the performance of the database is reduced during the data read / write process, in addition, only the constructed pre-write log is transmitted during the data write process of the database, and no data is transmitted, so that the data transmission amount is greatly reduced, the pre-write log is divided and stored during the storage of the pre-write log, the data rapid recovery capability is improved, and the write performance of the database is optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a data processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the rapid development of the Internet, the amount of data generated grows in a geometric pattern. The phenomenon of large data volume and multiple data types requires higher and higher performance of databases.

[0003] The data writing and data storage of the traditional database are coupled in the database engine, and in the process of executing the data writing, the traditional database needs to perform synchronous data processing on data files, log files, redo logs and the like.

[0004] The above-mentioned traditional database data processing process involves a large amount of data, which leads to an increase in the time consumption of the database in the data processing process and a low data processing efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a data processing method, device, computer equipment and storage medium capable of improving the data processing efficiency of the database in view of the above technical problems.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] receiving a database operation request by calling a computing service node, and if the database operation request is a data writing request, constructing a pre-write log based on the data writing request;

[0008] dividing the pre-write log by calling a storage service node to obtain a plurality of pre-write log segments, and performing storage processing on the plurality of pre-write log segments.

[0009] In the embodiment, the storage service and the computing service in the database are isolated, the coupling between the storage service and the computing service is reduced, the influence on the performance of the database in the data reading / writing process is reduced, in addition, in the present solution, only the constructed pre-write log is transmitted in the data writing process of the database, the data itself is not transmitted, the amount of transmitted data is greatly reduced, and in the process of storing the pre-write log, the pre-write log is divided and stored, the ability of fast recovery of data is improved, and the write performance of the database is optimized.

[0010] In one of the optional embodiments, the storage service node comprises a plurality of storage nodes; dividing the pre-write log by calling the storage service node to obtain a plurality of pre-write log segments, and performing storage processing on the plurality of pre-write log segments, comprising:

[0011] According to the correspondence between the data written in the pre-write log and the storage nodes, the pre-write log is divided to obtain the pre-write log segments corresponding to each storage node;

[0012] Each pre-write log segment is stored in the corresponding storage node.

[0013] In the embodiment, the wal is divided and stored according to the data division rule, reducing the amount of data written to the storage node.

[0014] In one of the optional embodiments, the storage node includes a pre-write log segment file, a data file and a data cache; the storage processing of the plurality of pre-write log segments includes:

[0015] According to the preset sorting rule, each pre-write log segment is sequentially stored in the pre-write log segment file corresponding to each storage node;

[0016] According to the preset first operation frequency, the corresponding pre-write log segment is read from each pre-write log segment file, and the pre-write log segment is replayed to the data file and the data cache of each storage node.

[0017] In the embodiment, the storage service node can realize the replay of the wal segment in the data file and the data cache based on the wal segment in the local wal segment, without crossing the node, avoiding network overhead, and improving the efficiency of the wal segment replay.

[0018] In one of the optional embodiments, according to the preset first operation frequency, the corresponding pre-write log segment is read from each pre-write log segment file, and the pre-write log segment is replayed to the data file and the data cache of each storage node, including:

[0019] For each storage node, according to the first operation frequency, the first sequence number of the first pre-write log segment in the pre-write log segment file, the second sequence number of the second pre-write log segment in the data file, and the third sequence number of the third pre-write log segment in the data cache are obtained.

[0020] According to the first sequence number, the second sequence number and the third sequence number, the first pre-write log segment is replayed to the data file and the data cache of each storage node.

[0021] In the embodiment, the storage service node can realize the replay of the latest wal segment in the data file and the data cache based on the sequence number of the wal segment in the local wal segment, realize the consistency of the data, and without crossing the node, also improve the efficiency of the wal segment replay.

[0022] In one of the optional embodiments, the first pre-write log segment is replayed into the data file and the data cache of each storage node according to the first sequence number, the second sequence number and the third sequence number, including:

[0023] In the case that the first sequence number is greater than the second sequence number, the first pre-write log segment file is replayed into the data file;

[0024] In the case that the first sequence number is greater than the third sequence number, the first pre-write log segment file is replayed into the data cache.

[0025] In this embodiment, through the wal segment replay operation on the data file and the data cache, it can be ensured that the wal segment in the data cache is the latest data, and the consistency and effectiveness of the data are ensured.

[0026] In one of the optional embodiments, the method further includes:

[0027] According to the preset number of copies, the pre-write log segment in the current storage node is synchronized to other storage nodes.

[0028] In this embodiment, after receiving the synchronization response from the node, the response of successful data writing can be returned to the computing service node, without waiting for the storage service node to complete the wal persistence, wal replay and other processing, thereby improving the write performance of the database.

[0029] In one of the optional embodiments, the method further includes:

[0030] The effective number of copies of the pre-write log segment of the current storage node is obtained;

[0031] In the case that the effective number of copies is less than the preset number of copies, the required number of copies is calculated based on the effective number of copies and the preset number of copies;

[0032] The other storage nodes with the required number of copies are determined as candidate storage nodes, and the pre-write log segment of the current storage node is synchronized to the candidate storage nodes.

[0033] In this embodiment, the storage service node can effectively manage the number of copies in time, realize the effective copies of the pre-write log segment of each storage node, and improve the efficiency of data recovery.

[0034] In one of the optional embodiments, the method further includes,

[0035] In the case that the database operation request is a data read request, the storage service node is called to obtain the data page corresponding to the data identifier carried in the data read request according to the data identifier;

[0036] And the data page is returned to the computing service node.

[0037] In the embodiment, the storage service node can return the corresponding data page obtained from the storage service node to the computing service node based on the data identifier, thereby improving the data reading efficiency.

[0038] In a second aspect, the present application further provides a data processing apparatus. The apparatus comprises:

[0039] The computing service module is configured to receive a database operation request, and in a case where the database operation request is a data write request, construct a pre-write log based on the data write request.

[0040] The storage service module is configured to divide the pre-write log to obtain a plurality of pre-write log segments, and perform storage processing on the plurality of pre-write log segments.

[0041] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method provided in the first aspect when executing the computer program.

[0042] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method provided in the first aspect.

[0043] In a fifth aspect, the present application further provides a computer program product. The computer program product comprises a computer program, and the computer program is executed by a processor to implement the method provided in the first aspect.

[0044] The above data processing method, apparatus, computer device and storage medium, the computer device calls a computing service node to receive a database operation request, and if the database operation request is a data write request, constructs a pre-write log based on the data write request; calls a storage service node to divide the pre-write log to obtain a plurality of pre-write log segments, and performs storage processing on the plurality of pre-write log segments. In the present scheme, the storage service and the computing service in the database are isolated, the coupling between the storage service and the computing service is reduced, the influence on the performance of the database is reduced during the data reading / writing process, in addition, in the present scheme, only the constructed pre-write log is transmitted during the data writing process of the database, the data itself is not transmitted, the amount of transmitted data is greatly reduced, and in the process of storing the pre-write log, the pre-write log is divided and stored, the data rapid recovery capability is improved, and the write performance of the database is optimized. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 An application environment diagram of the data processing method in an embodiment;

[0046] Figure 2 a structure diagram of a database system in an embodiment;

[0047] Figure 3 a flow diagram of a data processing method in an embodiment;

[0048] Figure 4 a flow diagram of a data processing method in an embodiment;

[0049] Figure 5 a flow diagram of a data processing method in an embodiment;

[0050] Figure 6 a flow diagram of a data processing method in an embodiment;

[0051] Figure 7 a flow diagram of a data processing method in an embodiment;

[0052] Figure 8 a flow diagram of a data processing method in an embodiment;

[0053] Figure 9 a flow diagram of a data processing method in an embodiment;

[0054] Figure 10 a structure block diagram of a data processing apparatus in an embodiment. DETAILED DESCRIPTION

[0055] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0056] The data processing method provided by the embodiments of the present application can be applied in an application environment as shown in Figure 1 In an embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in Figure 1As shown in the figure. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with external terminals. Wireless mode can be achieved through WIFI, mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by the processor to implement a data processing method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad provided on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0057] Those skilled in the art can understand that, Figure 1 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0058] A database system is deployed in the computer device. The database system provided in the embodiment includes a computing service node and a storage service node. For details, please refer to Figure 2 As shown in the figure, the computing service node receives a SQL request of a database write operation, constructs a write ahead log (Wal) based on the data carried in the SQL request, executes a computing task and performs transaction management. The storage service node includes a storage driver service node and a storage node.

[0059] The storage driver service node receives the Wal, writes each Wal segment into the corresponding storage node after dividing and processing the Wal, and controls the synchronization of the wal between the driver service nodes. The storage driver service node is also used to maintain the number of copies of the wal file and the data file of the storage node. In addition, when receiving a read request of the database, the storage driver service node reads a data page from the storage node and returns it to the computing service node. When receiving a read data request, the storage driver service node judges whether the data in the cache is the latest according to the transaction log sequence number lsn of the wal. If it is the latest, it is returned directly, otherwise the wal is replayed to update the cache and the data file. The storage driver service node is also used to persist the wal to the Wal data file of the storage node, and replay the wal to update the data file regularly.

[0060] Exemplarily, the storage driver service node comprises one master node (i.e. read-write node) and multiple read nodes; the storage driver service comprises a pre-write log processing module Wal Processor, a read module Read Block, and a replication management module Replication Manager.

[0061] The Wal Processor performs wal synchronization, wal division, wal persistence, and wal replay of the master node and the read nodes.

[0062] Optionally, the wal synchronization: after the master node receives the wal, a synchronization message is generated, and the synchronization message is synchronized to all read nodes; when more than half of the read nodes return a synchronization success response to the master node, the master node returns a data write success response to the computing service node. In this case, even if there is a failure of an individual read node, the consistency of the wal can be guaranteed. Optionally, the data synchronized between the master node and the read nodes includes some attribute information in addition to the wal, and the attribute information includes data definition language (DDL) update, wal segment division information, wal minimum log sequence number (LogSequence Number, lsn), and maximum lsn of the persisted data. Based on this, the master node can return a data write success response to the computing service node after receiving the synchronization response from the read node, without waiting for the storage service node to complete the wal persistence, wal replay, and the like, thereby improving the write performance of the database.

[0063] The wal division: the master node divides the wal according to a data division rule to obtain multiple wal segments, stores the multiple wal segments in the corresponding storage nodes, and sorts the wals according to the lsns in the wal data file. Based on this, the master node can complete data persistence by replaying the local wal, without crossing nodes, thereby avoiding network overhead.

[0064] The wal persistence: the master node persists the sorted wal segments in the wal data file. Taking a case of three replicas as an example, the write success in two nodes can return; the wal data file (wal segment) organizes the replicas in a small storage block manner, such as 2 GB for one segment file, to take one segment file as a fault recovery unit, to reduce the average recovery time, and to realize high availability and fault self-healing of the storage. The storage driver node writes the wal segments belonging to the storage node 1, the storage node 2, and the storage node 3 into the wal segment 11, the wal segment 21, and the wal segment 31, respectively, and the storage nodes synchronize the wal segments in a pipeline manner to realize the replica synchronization of the wal segments in the respective storage nodes.

[0065] wal replay: periodically replaying the wal segment in the wal segment to the data file to complete the data landing, during the replay, comparing the lsn of the wal segment in the wal segment, the lsn of the wal segment in the data file and the lsn of the wal segment in the data cache, if the lsn of the wal segment in the wal segment is greater than the lsn of the wal segment in the data cache and the data file, then the replay operation is performed, that is, the wal segment in the wal segment is updated to the data file and the data cache, if the lsn of the wal segment in the data cache and the data file is equal to the lsn of the wal segment in the wal segment, then nothing is done. In this way, the data cache is updated after replaying to keep the wal segment in the data cache always up-to-date.

[0066] Read Block reads data in data page units and returns to the computing service. When reading data, according to the current transaction id carried in the operation request, the lsn of the wal segment in the data cache and the tid of the cached data, it is judged whether the target wal segment in the data cache is the latest, if so, the wal segment is directly obtained from the data cache; if not, the replay wal segment operation is triggered to update the wal segment in the data cache and the data file, and then the wal segment is read from the data cache and returned to the computing service node. When replaying the wal segment, it is judged whether the latest data of the current transaction is persisted in the wal data file, if so, the wal segment is first persisted to the wal data file, and then the replay operation of the wal segment is performed; if not, the operation of the wal segment is directly replayed. In this way, it can be guaranteed that no matter when the data is read, the latest data of the current transaction can be read to ensure data consistency.

[0067] Replication Manager is responsible for managing the replicas of the wal segment and the data file. Assuming that the number of system replicas is set to 3, when the number of valid replicas of a certain file is less than 3, the replication is performed from other valid replicas to meet the requirement of 3 replicas.

[0068] The following embodiments provide a data processing method based on a database system to further illustrate the data processing process.

[0069] In one embodiment, as shown in Figure 3 , a data processing method is provided, which is applied to a computer device in Figure 1 for illustration, including the following steps:

[0070] Step 201, calling a computing service node to receive a database operation request, if the database operation request is a data write request, constructing a pre-write log based on the data write request.

[0071] Optionally, the database operation request comprises a database-based data write request and a database-based data read request. The data write request can carry to-be-written data, and the computing service node can construct a corresponding pre-write log (wal) based on the to-be-written data. The data read request can carry to-be-read data identification, and the computing service node can obtain a corresponding data page from the storage service node according to the to-be-read data identification.

[0072] In this embodiment, the computer device invokes the computing service node to receive a database operation request sql request. In the case of a data write request, the computer device invokes the computing service node to obtain to-be-written data carried in the data write request, and constructs a corresponding pre-write log (wal) based on the to-be-written data. Optionally, the pre-write log (wal) comprises the to-be-written data and other parameter information, for example, attribute information such as a log sequence number (lsn) of the wal and operation time.

[0073] Optionally, in the case of a data read request, the storage service node obtains a data page corresponding to the data identification according to the data identification carried in the data read request, and returns the data page to the computing service node.

[0074] In this embodiment, the computer device invokes the computing service node to receive a database operation request sql request. In the case of a data read request, the computer device invokes the computing service node to obtain a data page corresponding to the data identification from the storage service node according to the data identification carried in the data read request, and returns the data page to the computing service node.

[0075] In this embodiment, the computer device invokes the computing service node to receive a database operation request sql request. In the case of a data read request, the computer device invokes the computing service node to obtain a data page corresponding to the data identification from the storage service node according to the data identification carried in the data read request, and returns the data page to the computing service node.

[0076] Optionally, after the computer device invokes the computing service node to construct the wal, the computer device can transmit the constructed wal to the storage service node, and the storage service node performs storage processing based on the received wal.

[0077] In this embodiment, the storage service node can perform division processing on the received wal. Optionally, the storage service node can divide the wal according to a preset division rule. The division rule can include dividing based on the data amount of the wal. Alternatively, the division rule can be based on the correspondence between the data in the wal and the storage node. Alternatively, the division rule can also be based on the processing time period of the data in the wal. After performing division processing on the wal, the storage service node obtains multiple wal segments, and stores each wal segment in the corresponding storage node. Optionally, the storage service node can perform storage processing according to the time sequence of the wal segment, or the storage service node can also sort the wal segment according to the lsn in ascending or descending order for storage processing.

[0078] In the above data processing method, the computer device calls the computing service node to receive a database operation request. If the database operation request is a data write request, a write-ahead log is constructed based on the data write request. The storage service node is called to divide the write-ahead log to obtain multiple write-ahead log segments, and the multiple write-ahead log segments are stored. In this scheme, the storage service and the computing service in the database are isolated, reducing the coupling between the storage service and the computing service. During data read / write, the impact on the performance of the database is reduced. In addition, in this scheme, only the constructed write-ahead log is transmitted during data writing of the database, and the data itself is not transmitted, greatly reducing the amount of data transmission. In the process of storing the write-ahead log, the write-ahead log is divided and stored, improving the ability of rapid recovery of data, and optimizing the write performance of the database.

[0079] The storage service node can include multiple storage nodes. In one of the optional embodiments, the storage service node includes multiple storage nodes. As shown in Figure 4 The storage service node is called to divide the write-ahead log to obtain multiple write-ahead log segments, and the multiple write-ahead log segments are stored, including:

[0080] In step 301, the write-ahead log is divided according to the correspondence between the write data in the write-ahead log and the storage node to obtain write-ahead log segments corresponding to each storage node.

[0081] In this embodiment, the write-ahead log wal can store the correspondence between the write data and the storage node. For example, the storage node corresponding to the write data A is node1, and the storage node corresponding to the write data B is node2. The storage service node divides the write data A and the write data B in the wal based on the correspondence to obtain the wal segment 1 corresponding to the write data A and the wal segment 2 corresponding to the write data B.

[0082] At step 302, each pre-written log segment is stored in the corresponding storage node.

[0083] In this embodiment, the storage service node writes wal segment 1 into the corresponding storage node node 1 and writes wal segment 2 into the corresponding storage node node 2 based on the correspondence.

[0084] In this embodiment, the wal is divided and stored according to the data division rule, the amount of data written into the storage node is reduced, and the local wal replay operation can be implemented based on each storage node, that is, the wal segment can be directly written into the data file without crossing the node, avoiding network overhead.

[0085] In the process of storing the wal segment, the computer device can implement the persistence processing and wal replay processing of the wal. In one of the optional embodiments, the storage node includes a pre-written log segment file, a data file and a data cache; as shown in Figure 5 The storage processing of the plurality of pre-written log segments includes:

[0086] At step 401, according to a preset sorting rule, each pre-written log segment is sequentially stored in the pre-written log segment file corresponding to each storage node.

[0087] In this embodiment, the preset sorting rule can be a sorting rule based on the lsn of the wal segment, which can be descending order of lsn or ascending order of lsn. For example, the storage service node writes the wal segment into the corresponding storage node in ascending order of the lsn of the wal segment, and optionally, the wal segment is persisted to the pre-written log segment file wal segment of the storage node.

[0088] Optionally, the process of persisting the wal segment to the storage node by the storage service node further includes synchronizing the pre-written log segment in the current storage node to other storage nodes according to a preset number of copies.

[0089] Optionally, the preset number of copies can be determined according to the number of actual database storage nodes, the number of wal segments and other parameters. For example, the number of copies here can be 3, that is, successfully writing the wal segment of the current storage node into other two storage nodes is regarded as synchronization success. For reference Figure 6As shown, the sorted wal segments in the current storage node node1 are persisted into wal segment 11, and the wal segment 11 is synchronized to node2 and node3. The wal segments of each storage node are stored in the form of small storage blocks, such as 2 GB of a segment file, and a segment file is taken as a fault recovery unit to reduce the average recovery time. For example, in a gigabit network environment, the recovery time of a segment file is less than 2 seconds, so as to realize high availability and fault self-healing of storage. Each storage node can have multiple segment files, for reference Figure 6 As shown, after the synchronization of node1, node2 and node3, the wal segment 11, wal segment 21 and wal segment 31 are included in node1. Optionally, if the wal segment 11 of the storage node node1 is full, the wal segments are continuously written into the wal segment 12 according to the order of the wal segments.

[0090] After synchronizing the wal segments, the storage service node can obtain the number of valid copies of each wal segment. If the number of valid copies decreases, it means that there may be an abnormal or fault condition of the storage node. In one of the optional embodiments, as shown in the following table, the storage service node can obtain the number of valid copies of each wal segment. Figure 7 As shown, the method further includes:

[0091] In step 501, the number of valid copies of the pre-write log segment of the current storage node is obtained.

[0092] In this embodiment, the copy management module in the storage service node can obtain the number of valid copies of the wal segment of the current storage node, that is, the number of other storage nodes storing the wal segment of the current storage node.

[0093] In step 502, if the number of valid copies is less than the preset number of copies, the required number of copies is calculated based on the number of valid copies and the preset number of copies.

[0094] In this embodiment, for example, the preset number of copies can be 3, and if the number of valid copies is 2, which is less than the preset number of copies, there may be an abnormal or fault condition of the storage node. In this case, the copy management module can calculate the number of copies to be created, that is, the required number of copies. In this embodiment, the number of copies to be created is 1.

[0095] In step 503, the other storage nodes of the required number of copies are determined as candidate storage nodes, and the pre-write log segment of the current storage node is synchronized to the candidate storage nodes.

[0096] In the embodiment, when the replica management module determines that the required number of replicas is 1, one other storage node with a normal working state is obtained, and the wal segment of the current storage node is synchronized to the other storage node. Optionally, the synchronization process includes synchronization of the wal segment of the current storage node and synchronization of the data file of the current storage node.

[0097] At step 402, according to a preset first operation frequency, the corresponding prewrite log segment is read from each prewrite log segment file, and the prewrite log segment is replayed to the data file and the data cache of each storage node.

[0098] Optionally, the preset first operation frequency can be once per 1 s. The first operation frequency can also be synchronized with the frequency of wal segment persistence of the storage service node, that is, after the storage service node completes the wal segment persistence operation, the storage service node performs the replay operation of the wal segment.

[0099] In the embodiment, the storage service node persists the wal segment to the wal segment of the storage node, and the storage service node replays the wal segment to the data file and the data cache of the storage node based on the wal segment in the wal segment.

[0100] In the embodiment, the storage service node can replay the wal segment in the data file and the data cache based on the wal segment in the local wal segment, without crossing nodes, avoiding network overhead, and improving the efficiency of wal segment replay.

[0101] Further, in order to ensure the effectiveness of the wal segment in the data file and the data cache, in one of the optional embodiments, as shown in Figure 8 According to a preset first operation frequency, the corresponding prewrite log segment is read from each prewrite log segment file, and the prewrite log segment is replayed to the data file and the data cache of each storage node, including:

[0102] At step 601, for each storage node, according to a first operation frequency, a first sequence number of a first prewrite log segment in a prewrite log segment file, a second sequence number of a second prewrite log segment in a data file, and a third sequence number of a third prewrite log segment in a data cache are obtained.

[0103] In the embodiment, the storage service node acquires a first LSN of the first pre-write log segment, a second LSN of the second pre-write log segment, and a third LSN of the third pre-write log segment. It should be noted that the LSN is a sequence number composed of numbers, and the larger the number is, the later the time when the wal segment corresponding to the LSN is generated, that is, the data version of the wal segment corresponding to the LSN is the latest. The storage service node can acquire the LSN corresponding to the wal segment stored in the wal segment, the data file, and the data cache respectively.

[0104] In step 602, the first pre-write log segment is replayed into the data file and the data cache of each storage node according to the first sequence number, the second sequence number, and the third sequence number.

[0105] In the embodiment, the wal segment in the wal segment is replayed into the data file and the data cache according to the first LSN, the second LSN, and the third LSN.

[0106] Optionally, the first pre-write log segment is replayed into the data file and the data cache of each storage node according to the first sequence number, the second sequence number, and the third sequence number, including the following two cases:

[0107] In the case where the first sequence number is greater than the second sequence number, the first pre-write log segment file is replayed into the data file.

[0108] In the embodiment, the first LSN is greater than the second LSN, which indicates that the wal segment in the wal segment has been updated, but the wal segment in the data file is still old data. At this time, the storage service node can replay the wal segment in the wal segment into the data file.

[0109] In the case where the first sequence number is greater than the third sequence number, the first pre-write log segment file is replayed into the data cache.

[0110] Similarly to the replay into the data file, the first LSN is greater than the third LSN, which indicates that the wal segment in the wal segment has been updated, but the wal segment in the data cache is still old data. At this time, the storage service node can replay the wal segment in the wal segment into the data cache, so that the wal segment in the data cache is always the latest valid data.

[0111] Further, in the data reading process, when the storage service node obtains the data page corresponding to the data identifier from the data cache, it can determine from the data cache whether the wal segment corresponding to the data identifier is the latest data, i.e., whether the lsn of the wal segment in the data cache is the maximum lsn. If yes, the wal segment and the data page corresponding to the data identifier are directly obtained from the data cache. If not, it needs to determine whether the data page corresponding to the data identifier has performed wal persistence processing. If yes, replay the operation in the wal segment to the data file and the data cache. If not, perform wal persistence processing based on the wal segment corresponding to the data identifier, and then replay the operation in the wal segment to the data file and the data cache, so as to read the wal segment and the data page corresponding to the data identifier from the data cache. The embodiment is not limited in this regard.

[0112] In the embodiment, through the wal segment replay operation of the data file and the data cache, it can be ensured that the wal segment in the data cache is the latest data, and the consistency and effectiveness of the data are guaranteed.

[0113] In order to better illustrate the above method, as shown in the figure, Figure 9 The embodiment provides a data processing method, which specifically comprises:

[0114] S101, calling the computing service node to receive a database operation request;

[0115] S102, in the case of a data write request, calling the computing service node to construct a pre-write log based on the data write request;

[0116] S103, calling the storage service node to divide the pre-write log according to the correspondence between the write data in the pre-write log and the storage node, to obtain a pre-write log segment corresponding to each storage node;

[0117] S104, calling the storage service node to store each pre-write log segment in the pre-write log segment file corresponding to each storage node according to a preset sorting rule;

[0118] S105, calling the storage service node to read the corresponding pre-write log segment from each pre-write log segment file according to a preset first operation frequency;

[0119] S106, obtaining the first sequence number of the first pre-write log segment in the pre-write log segment file, the second sequence number of the second pre-write log segment in the data file, and the third sequence number of the third pre-write log segment in the data cache;

[0120] S107, in a case where the first sequence number is greater than the second sequence number, calling the storage service node to rewrite the first prewrite log segment into the data file;

[0121] S108, in a case where the first sequence number is greater than the third sequence number, calling the storage service node to rewrite the first prewrite log segment into the data cache;

[0122] S109, according to the preset number of copies, calling the storage service node to synchronize the prewrite log segment in the current storage node to other storage nodes;

[0123] S110, calling the storage service node to obtain the number of valid copies of the prewrite log segment in the current storage node;

[0124] S111, in a case where the number of valid copies is less than the preset number of copies, calling the storage service node to calculate the required number of copies based on the number of valid copies and the preset number of copies;

[0125] S112, calling the storage service node to determine other storage nodes with the required number of copies as candidate storage nodes, and synchronizing the prewrite log segment of the current storage node to the candidate storage nodes;

[0126] S113, in a case where the database operation request is a data read request, calling the storage service node to obtain a data page corresponding to a data identifier carried in the data read request according to the data identifier, and returning the data page to the computing service node.

[0127] In the embodiment, the storage service and the computing service in the database are isolated, the coupling between the storage service and the computing service is reduced, the influence on the performance of the database is reduced in the data read / write process, in addition, in the scheme, only the constructed prewrite log is transmitted in the data write process of the database, and the data itself is not transmitted, so that the amount of transmitted data is greatly reduced, the network IO is reduced, and the write performance of the database is improved; the read / write node prewrite log cache is synchronized to return the result, so that the prewrite log is not needed to be persisted, and the write performance of the database is improved; the prewrite log is stored according to the data division rule, the storage node local prewrite log replay can write the data file, the cross-node is not needed, and the network overhead is avoided; in the data read process, part of the prewrite log is replayed, the latest data is read, and the data consistency is ensured.

[0128] The data processing method provided in the above embodiment has similar implementation principles and technical effects to the above method embodiments, and details are not repeated here.

[0129] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps.

[0130] Based on the same inventive concept, the embodiments of the present application also provide a data processing apparatus for implementing the above-mentioned data processing method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so the specific limitations in one or more data processing apparatus embodiments provided below can refer to the limitations of the data processing method described above, which will not be repeated here.

[0131] In one embodiment, as shown in Figure 10 a data processing apparatus is provided, comprising:

[0132] The computing service module 01 is configured to receive a database operation request, and in the case that the database operation request is a data write request, construct a pre-write log based on the data write request.

[0133] The storage service module 02 is configured to divide the pre-write log to obtain a plurality of pre-write log segments, and perform storage processing on the plurality of pre-write log segments.

[0134] In one of the optional embodiments, the storage service node includes a plurality of storage nodes; the storage service module 02 is configured to divide the pre-write log according to the correspondence between the write data in the pre-write log and the storage nodes, to obtain pre-write log segments corresponding to each storage node; and store each pre-write log segment into the corresponding storage node.

[0135] In one of the optional embodiments, the storage node includes a pre-write log segment file, a data file and a data cache; the storage service module 02 is configured to store each pre-write log segment into the pre-write log segment file corresponding to each storage node according to a preset sorting rule; and read the corresponding pre-write log segment from each pre-write log segment file according to a preset first operation frequency, and replay the pre-write log segment into the data file and the data cache of each storage node.

[0136] In one optional embodiment, the storage service module 02 is further configured to, for each storage node, obtain, according to a first operation frequency, a first sequence number of a first write-ahead log fragment in a write-ahead log fragment file, a second sequence number of a second write-ahead log fragment in a data file, and a third sequence number of a write-ahead log fragment in a third data cache; and replay the first write-ahead log fragment to the data file and data cache of each storage node according to the first sequence number, the second sequence number, and the third sequence number.

[0137] In one optional embodiment, the storage service module 02 is configured to replay the first write-ahead log fragment file to the data file if the first sequence number is greater than the second sequence number; and to replay the first write-ahead log fragment file to the data cache if the first sequence number is greater than the third sequence number.

[0138] In one optional embodiment, the storage service module 02 is further configured to synchronize the write-ahead log fragments in the current storage node to other storage nodes according to a preset number of replicas.

[0139] In one optional embodiment, the storage service module 02 is further configured to obtain the number of valid replicas of the write-ahead log fragments of the current storage node; if the number of valid replicas is less than the preset number of replicas, calculate the required number of replicas based on the number of valid replicas and the preset number of replicas; determine other storage nodes that require the required number of replicas as candidate storage nodes, and synchronize the write-ahead log fragments of the current storage node to the candidate storage nodes.

[0140] In one optional embodiment, the storage service module 01 is further configured to, when the database operation request is a data read request, obtain the data page corresponding to the data identifier carried in the data read request, and return the data page to the computing service node.

[0141] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0142] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0143] The compute service node receives database operation requests. If the database operation request is a data write request, a write-ahead log is built based on the data write request.

[0144] The prewrite log is divided by calling the storage service node to obtain a plurality of prewrite log segments, and the plurality of prewrite log segments are stored.

[0145] The computer device provided in the above embodiment has similar implementation principles and technical effects to the above method embodiments, and thus detailed description is omitted here.

[0146] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the following steps:

[0147] The database operation request is received by calling the computing service node. If the database operation request is a data write request, a prewrite log is constructed based on the data write request.

[0148] The prewrite log is divided by calling the storage service node to obtain a plurality of prewrite log segments, and the plurality of prewrite log segments are stored.

[0149] The computer readable storage medium provided in the above embodiment has similar implementation principles and technical effects to the above method embodiments, and thus detailed description is omitted here.

[0150] In one embodiment, a computer program product is provided, and the computer program product includes a computer program. The computer program is executed by a processor to implement the following steps:

[0151] The database operation request is received by calling the computing service node. If the database operation request is a data write request, a prewrite log is constructed based on the data write request.

[0152] The prewrite log is divided by calling the storage service node to obtain a plurality of prewrite log segments, and the plurality of prewrite log segments are stored.

[0153] The computer program product provided in the above embodiment has similar implementation principles and technical effects to the above method embodiments, and thus detailed description is omitted here.

[0154] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0155] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0156] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0157] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data processing method, characterized by, The method includes: The computing service node is invoked to receive database operation requests. If the database operation request is a data write request, a write-ahead log is constructed based on the data write request. The storage service node is invoked to divide the write-ahead log into multiple write-ahead log segments, and the multiple write-ahead log segments are stored. The calling storage service node divides the write-ahead log into multiple write-ahead log segments, and performs storage processing on the multiple write-ahead log segments, including: Based on the correspondence between the written data in the write-ahead log and the storage nodes, the write-ahead log is divided to obtain write-ahead log segments corresponding to multiple storage nodes in the storage service nodes. Each of the aforementioned write-ahead log fragments is stored in its corresponding storage node; According to the preset sorting rules, each of the write-ahead log segments is stored sequentially in the write-ahead log segment file corresponding to each of the storage nodes; According to a preset first operation frequency, the corresponding write-ahead log fragments are read from each of the write-ahead log fragment files, and the write-ahead log fragments are replayed into the data files and data caches of each of the storage nodes.

2. The method of claim 1, wherein, The step of reading corresponding write-ahead log segments from each of the pre-written log segment files according to a preset first operation frequency, and replaying the pre-written log segments to the data files and data caches of each of the storage nodes, includes: For each of the storage nodes, according to the first operation frequency, obtain the first sequence number of the first write-ahead log fragment in the write-ahead log fragment file, the second sequence number of the second write-ahead log fragment in the data file, and the third sequence number of the write-ahead log fragment in the third data cache; Based on the first sequence number, the second sequence number, and the third sequence number, the first write-ahead log fragment is replayed to the data files and data caches of each of the storage nodes.

3. The method of claim 2, wherein, The step of replaying the write-ahead log fragment to the data files and data caches of each of the storage nodes according to the first sequence number, the second sequence number, and the third sequence number includes: If the first sequence number is greater than the second sequence number, the first write-ahead log fragment file is replayed into the data file; If the first sequence number is greater than the third sequence number, the first write-ahead log fragment file is replayed into the data cache.

4. The method of claim 1, wherein, The method further includes: Based on the preset number of replicas, the write-ahead log fragments in the current storage node are synchronized to other storage nodes.

5. The method of claim 4, wherein, The method further includes: Obtain the number of valid replicas of the write-ahead log fragment of the current storage node; If the number of valid copies is less than the preset number of copies, the required number of copies is calculated based on the number of valid copies and the preset number of copies. Other storage nodes that are identified as the required number of replicas are selected as candidate storage nodes, and the write-ahead log fragments of the current storage node are synchronized to the candidate storage nodes.

6. A data processing apparatus for implementing the data processing method according to claim 1, characterized in that, The device includes: A computing service module is configured to receive a database operation request, and in a case where the database operation request is a data write request, construct a pre-write log based on the data write request. A storage service module is configured to divide the pre-write log to obtain a plurality of pre-write log segments, and perform storage processing on the plurality of pre-write log segments. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 5.

Images