Data processing method and related device
By executing the WAL writing operation of business data and the writing operation of index data in parallel in Phoenix, the problem that the update process of indexed data takes too long is solved, and more efficient writing performance is achieved.
Patent Information
- Application Number
- CN202310893279.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-19
- Publication Date
- 2025-01-21
Smart Images
Figure CN119336767A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of databases, and in particular to a data processing method and related devices. Background Art
[0002] Hadoop Database (HBase) is a highly reliable, high-performance, column-oriented and scalable distributed storage system. HBase provides a primary key index based on dictionary sorting. Non-primary key queries require full table scans, which are slow and have poor performance. In order to make data queries more efficient and adapt to more business scenarios, it is necessary to build secondary indexes for HBase. Secondary indexes can provide the ability to index based on specific column values, so that full table scans are not required for non-primary key queries.
[0003] Phoenix, a query tool, provides global secondary indexes for HBase. Phoenix is built on the SQL layer on HBase and completes the secondary index by creating two HBase tables, namely the data table and the index table. When retrieving data, the secondary index is first queried from the index table based on the index field, and then the primary key is quickly determined based on the secondary index. Then, the query is performed using the primary key to obtain business data from the data table.
[0004] Phoenix's data modification operations (insert, update, delete, etc.) will cause the index to be updated. In order to ensure the strong consistency of data before and after the update, Phoenix will perform multiple writes. Specifically, after the regional server (RS) where the data table is located receives the data modification request sent by the user, it sends a remote procedure call (RPC) request to the RS where the index table is located. The RS where the index table is located inserts the new index data and sets the status of the index data to unverified. After inserting the index data, the RS where the data table is located writes the business data, and then sends an RPC request to the RS where the index table is located again to update the status of the index data to verified. The status of the index data can be used to record whether the index is valid, and the invalid index can be corrected through read repair, thereby ensuring the strong consistency of the data.
[0005] However, the update process of indexed data in Phoenix is equivalent to performing three data write operations (including two index data writes and one business data write), which takes about three times as long as the update process of non-indexed data. It takes too long and has poor write performance. Summary of the invention
[0006] The present application provides a data processing method and related devices, which can improve the writing speed of indexed data.
[0007] A first aspect of the present application provides a data processing method, which is applied to a first regional server, and the method includes: the first regional server sends a first instruction to a second regional server, the first instruction is used to instruct the second regional server to write first index data, and the status bit of the first index data is unverified; the first regional server writes first business data into a write-ahead log based on the first instruction, and the first index data is used to index the first business data; the first regional server receives a response message returned by the second regional server after writing the first index data; after the first business data is written into the write-ahead log, the first regional server writes the first business data into a first cache module based on the response message, and the first cache module belongs to the first regional server; after the first business data is written into the first cache module, the first regional server sends a second instruction to the second regional server, the second instruction is used to instruct the second regional server to write second index data, and the second index data is obtained after the status bit of the first index data is updated to verified.
[0008] In the first aspect of the present application, the first instruction triggers the operation of the second regional server writing the first index data, and the operation of the first regional server writing the first business data into the WAL, so that the WAL writing operation of the first business data and the writing operation of the first index data are executed in parallel, saving the writing time of the indexed data and shortening the writing time.
[0009] In a possible implementation manner of the first aspect, the first instruction and the second instruction are remote procedure call instructions.
[0010] The first instruction and the second instruction may be remote procedure call (RPC) instructions, and the first regional server calls the relevant process of the second regional server through the RPC instruction, so that the second regional server writes the first index data into the index table. Of course, the first instruction may also be other similar calling instructions, such as remote method invocation (RMI) instructions, etc., which are not specifically limited here. In addition, the first instruction may not be a calling request, and there is no need for the first regional server to call the second regional server. The first instruction carries the parameters of the first index data, and the second regional server actively writes after receiving the first instruction.
[0011] In this possible implementation, the specific forms of the first instruction and the second instruction are limited, thereby improving the feasibility of the solution.
[0012] In a possible implementation manner of the first aspect, the above step: the first region server writes the first business data into a write-ahead log based on the first instruction includes: the first region server sets a status bit of the first business data to unverified, and obtains second business data; the first region server writes the second business data into the write-ahead log; the above step: after the first region server sends the second instruction to the second region server, the method further includes: the first region server updates the status bit of the second business data to verified, and obtains third business data; the first region server writes the third business data into the write-ahead log.
[0013] When the first regional server is restarted, the first business data stored in the first cache module will be lost, and the first business data in the WAL of the first regional server will not be affected. Since the WAL write operation of the first business data and the write operation of the first index data are performed in parallel, the first index data may not have been written yet (it is unclear which step is executed). At this time, it is necessary to query the index data (it can be the first index data or the second index data). If the index data exists, it means that the first business data is valid, and the memstore write operation of the first business data continues. If the index data does not exist, it means that the first business data is invalid, and the first business data in the WAL is deleted. This process needs to be performed every time the first regional server is restarted, which is time-consuming.
[0014] In view of this, a status bit is set for the first business data written into the WAL, and the status bit can be used to determine whether the business data is valid. Specifically, the first regional server first sets the status bit of the first business data to unverified, obtains the second business data, and then writes the second business data into the WAL. After the first business data is written into the first cache module, the status bit of the second business data is updated to verified, the third business data is obtained, and the third business data is written into the WAL of the first regional server to complete the update of the business data in the WAL.
[0015] Because the second regional server needs to write the first index data before the first business data is written into the first cache module, and the third business data is written after the memstore write operation of the first business data. Therefore, when the status bit of the second business data is updated to verified (i.e., the third business data), the business data is confirmed to be valid. When the first regional server is restarted, there is no need to check whether the business data is valid by querying the index data, thus avoiding the time-consuming repeated query of the index data.
[0016] It is understandable that the update of the second business data status bit (obtaining the third business data) and the writing of the third business data do not affect the overall reading and writing process of the business data. Before this step, the first business data has been written into the first cache module, and the second regional server has also written the second index data. The business data can be read normally without waiting for this step to be completed. Therefore, the writing time of this step is not perceived in the overall process and is not considered in the writing time.
[0017] In a possible implementation manner of the first aspect, the method further includes: the first area server repairing the second service data based on the status bit of the second service data.
[0018] If the status bit of the business data is verified (i.e., the third business data has been written to WAL), it means that the business data is valid and no state repair is required. If the first regional server is restarted before the WAL write operation of the third business data, the status bit of the business data is unverified (i.e., the second business data), and the repair process can be performed at this time. Specifically, first check whether the first index data exists. If the first index data exists, update the status bit of the second business data to verified. If the first index data does not exist, delete the second business data.
[0019] In this possible implementation, the second business data can be repaired according to the status bit of the second business data, and the result of the repair is to delete the second business data or update the status bit of the second business data to verified. Therefore, the repair process only needs to be executed once, avoiding the time-consuming process of repeatedly querying the index data to detect whether the business data is valid.
[0020] In a possible implementation manner of the first aspect, the second instruction is used to instruct the second region server to write the second index data into a second cache module, and the second cache module belongs to the second region server.
[0021] The second instruction may be to instruct the second region server to write only the second index data into the second cache module of the second region server, without writing the second index data into the WAL of the second region server. The specific operation may be to set the WAL parameter of the second index data carried in the second instruction to be invalid (disable), so that only the second index data is written into the second cache module.
[0022] When the second regional server is restarted, the data in the second cache module will be lost, but the data in the WAL will not be affected. That is, the second index data will be lost, but the first index data will still exist (because the second index data is only written to the second cache module, while the first index data is written to the WAL). At this time, the second index data can be repaired through the read repair process, so that the overall process is not affected.
[0023] By eliminating the WAL write operation for the second index data, the data writing time can be further saved and the writing time can be shortened. Although the index data needs to be repaired through the read repair process when the second region server is restarted, the frequency of restarting the second region server is low, and the writing time saved in the overall process is still considerable.
[0024] In a possible implementation manner of the first aspect, the method further includes: the first region server repairing the first index data based on a status bit of the first index data.
[0025] The status bit of the first index data is used to record whether the index data is valid, that is, to record whether the index data and the corresponding business data have been successfully written. The first index data can be repaired according to the status bit of the first index data. Specifically, when the client wants to read the business data, it first queries the index data. If the status bit of the index data is verified (that is, the status bit of the first index data has been updated and the second index data has been written), the index data is a valid index.
[0026] If the status bit of the index data is unverified (that is, the index data is still the first index data, and the status bit is not updated), it means that there is a failed operation during the data writing process. The first regional server performs a full table scan on the data table. If the first business data cannot be found (that is, the writing of the first business data fails), the second regional server is instructed to delete the first index data. If the first business data can be queried, the status bit of the first index data is changed to verified, the second index data is obtained, and the second regional server is instructed to write the second index data. In this way, the read repair of the first index data is completed, ensuring the strong consistency of the index data and the business data. After the first index data is repaired, there is no need to perform a full table scan when reading the first business data subsequently.
[0027] In this possible implementation, strong consistency of data can be ensured by indexing the status bit of the data.
[0028] The second aspect of the present application provides a regional server, including a sending unit, a writing unit and a receiving unit. The sending unit is used to send a first instruction to a second regional server, the first instruction is used to instruct the second regional server to write first index data, and the status bit of the first index data is unverified; the writing unit is used to write the first business data into a write-ahead log based on the first instruction, and the first index data is used to index the first business data; the receiving unit is used to receive a response message returned by the second regional server after writing the first index data; the writing unit is also used to write the first business data into a first cache module based on the response message after the first business data is written into the write-ahead log, and the first cache module belongs to the regional server; the sending unit is also used to send a second instruction to the second regional server after the first business data is written into the first cache module, and the second instruction is used to instruct the second regional server to write second index data, and the second index data is obtained after the status bit of the first index data is updated to verified.
[0029] In a possible implementation manner of the second aspect, the first instruction and the second instruction are remote procedure call instructions.
[0030] In a possible implementation of the second aspect, the writing unit is specifically used to set the status bit of the first business data to unverified, to obtain the second business data; and to write the second business data into the write-ahead log. The regional server also includes an updating unit, which is used to update the status bit of the second business data to verified, to obtain the third business data. The writing unit is also used to write the third business data into the write-ahead log.
[0031] In a possible implementation manner of the second aspect, the regional server further includes a repairing unit, configured to repair the second service data based on a status bit of the second service data.
[0032] In a possible implementation manner of the second aspect, the second instruction is used to instruct the second region server to write the second index data into a second cache module, and the second cache module belongs to the second region server.
[0033] In a possible implementation manner of the second aspect, the repair unit is further configured to repair the first index data based on a status bit of the first index data.
[0034] The regional server provided in the second aspect of the present application is used to execute the method described in the first aspect or any possible implementation manner of the first aspect.
[0035] The third aspect of the present application provides a regional server, comprising a processor and a memory, wherein the memory is used to store instructions, and the processor is used to obtain the instructions stored in the memory to execute the method described in the first aspect or any possible implementation of the first aspect.
[0036] A fourth aspect of the present application provides a computer-readable storage medium, which includes instructions. When the instructions are executed on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.
[0037] A fifth aspect of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method described in the first aspect or any possible implementation of the first aspect.
[0038] In a sixth aspect, the present application provides a chip system, which includes at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via lines, and the at least one processor is used to run computer programs or instructions to execute the method described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 A schematic diagram of the architecture of a database system used in an embodiment of the present application;
[0040] Figure 2 A schematic diagram of an embodiment of a data processing method provided in an embodiment of the present application;
[0041] Figure 3 A schematic diagram of another embodiment of the data processing method provided in the embodiment of the present application;
[0042] Figure 4 A flowchart of WAL status repair in an embodiment of the present application;
[0043] Figure 5 A schematic diagram of another embodiment of the data processing method provided in the embodiment of the present application;
[0044] Figure 6 A schematic diagram of the structure of a regional server provided in an embodiment of the present application;
[0045] Figure 7 Another structural diagram of the regional server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0046] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0047] The terms "system" and "network" in the specification and claims of the present application and the above-mentioned drawings can be used interchangeably. Unless otherwise specified, ordinal numbers such as "first" and "second" are used to distinguish multiple objects and are not used to limit the order, timing, priority or importance of multiple objects. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0048] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of the present application are first introduced below.
[0049] 1.HBase
[0050] Hadoop Database (HBase) is a distributed, column-oriented open source database. Unlike general relational databases, HBase is a database suitable for storing unstructured data and is specifically designed for fast random reading and writing of large-scale data.
[0051] 2. RS
[0052] Region Server (RS) is a service running on each working node of the HBase cluster. It is used to maintain the status of the Region, provide management and services for the Region, upload the load information of the Region, and participate in distributed coordination management.
[0053] 3.RPC
[0054] Remote Procedure Call (RPC) is a computer communication protocol that allows you to call remote services just like calling local services. The caller sends a call message with process parameters to the callee, and then gets the result through the information returned by the callee.
[0055] 4.WAL
[0056] Write Ahead Log (WAL) provides a highly concurrent and persistent log storage and playback mechanism. Before each business data write operation (PUT / DELETE) is executed, it will be recorded in WAL. When the server restarts and causes data loss, the data can be restored through the operations recorded in WAL.
[0057] 5. memstore
[0058] The cache module (memstore) serves as the write cache of HBase, which stores the most recent data update. When reading data, the memstore is checked first. According to the principle of locality, the newly written data is more likely to be accessed. When certain conditions are met, the memstore will perform a flush and write the data to the file system. A data submission is to write the data to WAL first, and then write it to memstore after the write is successful.
[0059] The embodiment of the present application provides a data processing method, which can improve the writing speed of indexed data while ensuring strong data consistency. The embodiment of the present application also provides a corresponding device, a computer-readable storage medium, and a computer program product, etc. The following are respectively described.
[0060] See below Figure 1 , Figure 1 A schematic diagram of the architecture of a database system used in an embodiment of the present application.
[0061] like Figure 1 As shown, the database system includes a client, a first regional server and a second regional server. The first regional server stores a data table and is responsible for processing the client's data write request, read request and delete request. Specifically, the first regional server stores the business data requested to be written by the client in the data table, or reads the business data from the data table and deletes the business data. The second regional server stores an index table, which provides secondary index data for indexing the business data stored in the data table so as to quickly find the data required by the client.
[0062] At present, when processing the client's write request, in order to ensure strong consistency of data, the second regional server will write index data twice. Specifically, the index data is written once before the business data is written, and the status bit of the index data written this time is unverified. After the business data is written, the index data is written again, and the status bit of the index data written this time is verified, and the old index data is deleted. In other words, three serial write operations are performed in the process of writing index data once, that is, the index data is written first, and then the business data is written after the writing is completed, and the index data is written again after the business data is written. The data write operation includes two steps, that is, writing to WAL first, and then writing to memstore after the writing is completed. This leads to the problem that writing index data takes too long and the write performance is low.
[0063] In view of this, an embodiment of the present application provides a data processing method, which utilizes the read invisibility of WAL to perform the WAL write operation of business data in parallel when executing the first write of index data, thereby saving data writing time and improving data writing performance.
[0064] The data processing method provided in the embodiment of the present application can be applied to HBase, and can also be applied to databases such as Lindorm. In the embodiment of the present application, HBase is used as an example for explanation, and the specific application scenario is not limited.
[0065] Let's combine Figure 2 This section introduces the overall data writing process. Figure 2 As shown, the data writing process includes writing index data and writing business data, wherein the first regional server is responsible for storing business data, the second regional server is responsible for storing new index data, and the third regional server is the server where the old index data is located. Figure 2 The writing process shown includes steps 201 to 209 .
[0066] 201. A client initiates a data write request to a first regional server.
[0067] 202. The first regional server queries historical data.
[0068] Using the coprocessor mechanism of HBase, a global index service coprocessor (GIS coprocessor) module is constructed in the first regional server, which is responsible for intercepting and processing the user's write request and instructing the second regional server to write index data. Among them, instructing the second regional server to write index data can be instructing the second regional server to write new index data, or it can be instructing the second regional server to update the old index data. Specifically, the GIS coprocessor module queries the historical data and checks whether there is relevant data in the historical data. If not, step 203 is executed to instruct the second regional server to write new index data. If so, step 204 is executed to instruct the second regional server to update the index data. After updating the index data, the old index data also needs to be deleted.
[0069] 203. The first region server instructs the second region server to write new index data.
[0070] When there is no relevant data in the historical data, the GIS coprocessor module on the first regional server sends a request to write new index data to the second regional server, instructing the second regional server to write new index data. At this time, the status bit of the index data is unverified.
[0071] 204. The first regional server instructs the second regional server to update index data.
[0072] When relevant data exists in the historical data, the GIS coprocessor module sends a request to update the index data to the second regional server, instructing the second regional server to update the index data, and the status bit of the updated index data is unverified.
[0073] 205. The first region server instructs the third region server to change the status bit of the old index data.
[0074] After the index data is updated, the GIS coprocessor module instructs the third regional server to change the status bit of the old index data to unverified.
[0075] 206. The first regional server writes the service data.
[0076] 207. The first region server instructs the second region server to update the status bit of the index data.
[0077] After the first regional server writes the business data, it indicates that the newly written index data is valid, and the GIScoprocessor module sends a request to write the index data to the second regional server again, in order to update the status bit of the new index data to verified. For the convenience of description, step 207 corresponds to two situations: the second regional server writes new index data (i.e., step 203) and updates index data (i.e., step 204).
[0078] 208. The first region server instructs the third region server to delete old index data.
[0079] In the case where the second regional server updates the index data, after the second regional server updates the status bit of the index data to verified, the GIS coprocessor module instructs the third regional server to delete the old index data.
[0080] 209. The first regional server replies a response message to the client, informing the client that the data has been successfully written.
[0081] It should be noted that, for the sake of brevity, the above description omits the description of the second and third regional servers replying to the first regional server. After the first regional server sends a request, the second and third regional servers will reply with response messages accordingly, and the first regional server will execute the next step after receiving the response message. In addition, the writing operations of index data and business data include two steps: writing to WAL and writing to memstore. Figure 2 The illustrated process does not describe in detail the writing operations of index data and business data.
[0082] The embodiment of the present application mainly improves the step of writing business data to the first regional server, so that the operation of the first regional server writing business data to the WAL is performed in parallel with the first writing of index data by the second regional server, thereby saving data writing time.
[0083] See below Figure 3 , is a schematic diagram of an embodiment of the data processing method provided in the embodiment of the present application. Figure 3 As shown, this embodiment includes steps 301 to 306.
[0084] 301. A first regional server sends a first instruction to a second regional server, where the first instruction is used to instruct the second regional server to write first index data, and a status bit of the first index data is unverified.
[0085] After receiving the data write request sent by the client, the first regional server sends a first instruction to the second regional server, where the first instruction is used to instruct the second regional server to write the first index data and set the status bit of the first index data to unverified. The process of the second regional server writing the first index data is to first write the first index data into the WAL of the second regional server, and then write the first index data into the second cache module of the second regional server.
[0086] The first instruction may be an RPC instruction, and the first regional server sends an RPC request to the second regional server, calling a related process of the second regional server, so that the second regional server writes the first index data into the index table and sets the status bit of the first index data to unverified. Of course, the first instruction may also be other similar calling instructions, such as RMI instructions, etc., which are not specifically limited here.
[0087] In addition, the first instruction may not be a call request, and the first regional server does not need to call the second regional server. The first instruction carries the parameters of the first index data, and the second regional server actively writes after receiving the first instruction and sets the status bit of the first index data to unverified.
[0088] The first instruction instructing the second regional server to write the first index data may be to instruct the second regional server to write new index data or to update old index data, which is not limited here. It should be noted that if the old index data is updated, the old index data needs to be deleted after the index data is updated.
[0089] 302. The first regional server writes first business data into a write-ahead log based on a first instruction, and the first index data is used to index the business data.
[0090] The first index data corresponds to the business data, and provides a secondary index for the business data, so as to quickly query the primary key of the business data and then query the corresponding business data.
[0091] Taking advantage of the fact that WAL data is invisible to readers when being written, the first regional server triggers the WAL write operation of the first business data after sending the first instruction. That is to say, the first regional server writes the first business data into the WAL of the first regional server while instructing the second regional server to write the first index data. The first instruction is the triggering condition for these two steps, so that the write operation of the first index data and the WAL write operation of the first business data can be executed in parallel, saving the data writing time. In addition, since the writing of the first index data includes two operations, WAL writing and memstore writing, the writing time of the first index data can usually cover the time of writing the first business data into WAL, which is equivalent to reducing one step in terms of time.
[0092] 303. After writing the first index data, the second region server returns a response message to the first region server.
[0093] After writing the first index data, the second region server sends a response message to the first region server to inform the first region server that the first index data has been written.
[0094] 304. After the first service data is written into the write-ahead log, the first regional server writes the first service data into a first cache module based on a response message, where the first cache module belongs to the first regional server.
[0095] After writing the first business data into WAL and receiving the response message returned by the second regional server, the first regional server writes the first business data into the first cache module of the first regional server. Before writing the first business data into the first cache module, it is necessary to ensure that the first business data has been written into WAL. WAL can provide atomicity and persistence, so that when the data in the first cache module is lost due to unexpected conditions such as the restart of the first regional server, the first business data can be restored through WAL. In addition, before writing the first business data into the first cache module, it is also necessary to ensure that the corresponding first index data has been written into the second regional server. In this way, when the user needs to read the first business data after it is written into the first cache module, the secondary index can be provided through the first index data.
[0096] It can be understood that the order of writing the first business data into the WAL and receiving the response message is not limited here, that is, the first business data can be written first, or the response message can be received first.
[0097] 305. After the first service data is written into the first cache module, the first regional server sends a second instruction to the second regional server, where the second instruction is used to instruct the second regional server to write second index data, where the second index data is obtained by changing the status bit of the first index data to verified.
[0098] After the first business data is written into the first cache module, it indicates that the first index data is valid. At this time, the first index data needs to be updated, and the status bit of the first index data is changed to verified to obtain the second index data. The first regional server sends a second instruction to the second regional server, and the second instruction instructs the second regional server to write the second index data, thereby completing the update of the index data. Writing the second index data also includes two operations: writing the second index data into the WAL of the second regional server and writing into the second cache module of the second regional server. The second instruction is similar to the first instruction and will not be repeated here.
[0099] 306. The first regional server repairs the first index data.
[0100] The status bit of the first index data is used to record whether the index data is valid, that is, to record whether the index data and the corresponding business data have been successfully written. The first index data can be repaired according to the status bit of the first index data. Specifically, when the client wants to read the business data, it first queries the index data. If the status bit of the index data is verified (that is, the status bit of the first index data has been updated and the second index data has been written), the index data is a valid index.
[0101] If the status bit of the index data is unverified (that is, the index data is still the first index data, and the status bit is not updated), it means that there is a failed operation during the data writing process. The first regional server performs a full table scan on the data table. If the first business data cannot be found (that is, the writing of the first business data fails), the second regional server is instructed to delete the first index data. If the first business data can be queried, the status bit of the first index data is changed to verified, the second index data is obtained, and the second regional server is instructed to write the second index data. In this way, the read repair of the first index data is completed, ensuring the strong consistency of the index data and the business data. After the first index data is repaired, there is no need to perform a full table scan when reading the first business data subsequently.
[0102] In this embodiment, the read invisibility of WAL is utilized to execute the WAL write operation of the first business data and the write operation of the first index data in parallel, which saves the writing time of the indexed data and shortens the writing time compared with the serial execution. In addition, writing WAL occupies most of the time in the data writing operation, while writing memstore is only a lightweight operation, so the time saved is considerable, which can effectively improve the writing performance.
[0103] In order to further save writing time, the second instruction in the above step 305 may be to instruct the second regional server to only write the second index data into the second cache module of the second regional server, without writing the second index data into the WAL of the second regional server. The specific operation may be to set the WAL parameter of the second index data carried in the second instruction to be invalid (disable), for example, when the second instruction is an RPC instruction, the WAL parameter of the second index data is set to disable in the RPC instruction, so that only the second index data is written into the second cache module.
[0104] The second regional server sometimes needs to be restarted during operation. For example, when it encounters a fault and crashes, the data in the second cache module will be lost after the restart, while the data in the WAL will not be affected. That is to say, the second index data will be lost, while the first index data still exists (because the second index data is only written to the second cache module, while the first index data is written to the WAL). At this time, if the business data reading process is performed, the index data queried is the first index data, and the status bit of the first index data is unverified. Then, the first index data is repaired according to the read repair process described in the above step 306. Specifically, if the first business data cannot be found (writing fails), the first index data is deleted. If the first business data can be found, the status bit of the first index data is changed to verified, and the second index data is obtained. The second regional server then writes the second index data into the second cache module.
[0105] By eliminating the WAL write operation for the second index data, the writing time can be further shortened. Although the index data needs to be repaired through the read repair process when the second region server is restarted, the frequency of restarting the second region server is low, and the writing time saved in the overall process is still considerable.
[0106] When the first regional server is restarted, the first business data stored in the first cache module will also be lost, and the first business data in the WAL of the first regional server will not be affected. Since the WAL write operation of the first business data and the write operation of the first index data are performed in parallel, the first index data may not have been written yet (it is unclear which step is executed). At this time, it is necessary to query the index data (it can be the first index data or the second index data). If the index data exists, it means that the first business data is valid, and the memstore write operation of the first business data continues. If the index data does not exist, it means that the first business data is invalid, and the first business data in the WAL is deleted. This process needs to be performed every time the first regional server is restarted, which is time-consuming.
[0107] In view of this, in the above step 302, the first regional server writes the first business data into the write-ahead log based on the first instruction, which may be to first set the status bit of the first business data to unverified, obtain the second business data, and then write the second business data into the WAL. After the first business data is written into the first cache module, the status bit of the second business data is updated to verified, the third business data is obtained, and the third business data is written into the WAL of the first regional server to complete the update of the business data in the WAL.
[0108] Because the second regional server needs to write the first index data before the first business data is written to the first cache module, and the writing of the third business data is after the memstore write operation of the first business data. Therefore, when the first regional server is restarted, it can be known whether the index data exists based on the status bit of the business data in the WAL. Specifically, if the status bit of the business data is verified (that is, the third business data has been written to the WAL), it means that the first index data has been written, and there is no need to query the index data, which saves the time required for this operation. When the first regional server is restarted before the WAL write operation of the third business data, the status bit of the business data is unverified (that is, the second business data), which can be repaired through the WAL status repair operation. The following is combined with Figure 4 Describe the repair process. Figure 4 As shown, the repair process includes steps 401 to 406.
[0109] 401. When the first regional server is restarted, the business data in the WAL is loaded first.
[0110] 402. Check the status bit of the business data to determine whether the status bit is unverified. If the status bit of the business data is verified (i.e., the third business data has been written into the WAL), execute step 407. If the status bit of the business data is unverified (i.e., the third business data has not been written into the WAL), execute step 403.
[0111] 403 , query the first index data to determine whether the first index data exists. If not, execute step 404 ; if so, execute step 405 .
[0112] 404. When the first index data does not exist, delete the second business data in the WAL and end the repair process.
[0113] 405. When the first index data exists, update the status bit of the second business data to verified, obtain the third business data, and write the third business data into WAL.
[0114] 406. Perform data recovery according to the third business data in the WAL, write the third business data into the first cache module, and end the repair process.
[0115] That is to say, after the status bit is added, the second business data can be repaired according to the status bit of the second business data. If the status bit of the second business data has been updated to verified, the business data is confirmed to be valid. If the status bit of the second business data is still unverified, check whether the first index data exists. If the first index data exists, update the status bit of the second business data to verified; if the first index data does not exist, delete the second business data.
[0116] After the status bit of the second service data is updated to verified, there is no need to query the index data when the first regional server is restarted, thereby avoiding the time-consuming process of querying the index data to detect whether the service data is valid each time the first regional server is restarted.
[0117] It is understandable that the update of the second business data status bit (obtaining the third business data) and the writing of the third business data do not affect the overall read and write process of the business data (except for the WAL state repair process when the first regional server is restarted). Before this step, the first business data has been written into the first cache module, and the second regional server has also written the second index data. The business data can be read normally without waiting for this step to be completed. Therefore, the writing time of this step is not perceived in the overall process and is not considered in the writing time.
[0118] In combination with the above-mentioned two situations in which the second index data is only written into the second cache module and the status bit is added to the business data, when the regional server (including the first regional server and the second regional server) is restarted, the processing methods corresponding to different steps are shown in Table 1:
[0119]
[0120] Table 1
[0121] As can be seen from Table 1, when the regional server is restarted, the business data is corrected through the above WAL state repair process, and the index data is corrected through the above read repair process.
[0122] In summary, the following Figure 5 Another embodiment of the data processing method provided in the embodiment of the present application is described. Figure 5 As shown, this embodiment includes steps 501 to 510.
[0123] 501. A client sends a first service data write request to a first regional server.
[0124] 502. In response to the write request, the first regional server sends a first instruction to the second regional server, instructing the second regional server to write the first index data, and the status bit of the first index data is unverified. Figure 3 Step 301 in the illustrated embodiment is similar and will not be described in detail here.
[0125] 503. The first regional server triggers a WAL write operation of the first business data based on the first instruction. Specifically, the state bit of the first business data is set to unverified, the second business data is obtained, and the second business data is written into the WAL of the first regional server.
[0126] 504. The second regional server writes the first index data into the WAL of the second regional server and the second cache module based on the first instruction. It can be understood that step 503 and step 504 are performed in parallel.
[0127] 505. After writing the first index data, the second region server sends a response message to the first region server, informing the first region server that the first index data has been written.
[0128] 506. After writing the second service data into the WAL and receiving a response message sent by the second regional server, the first regional server writes the first service data into a first cache module of the first regional server.
[0129] 507. The first regional server sends a second instruction to the second regional server, instructing the second regional server to write second index data into the second cache module, where the second index data is obtained by updating the status bit of the first index data to verified.
[0130] 508. The second regional server writes the second index data into the second cache module based on the second instruction.
[0131] 509. After writing into the second cache module, the second regional server replies a response message to the first regional server, informing the first regional server that the second index data has been written. After receiving the response, the first regional server sends a response message to the client, informing the client that the writing process of the first service data has been completed.
[0132] 510. The first region server asynchronously updates the status bit of the second service data, obtains the third service data, and writes the third service data into the WAL of the first region server.
[0133] In this embodiment, the WAL write operation of the first business data is performed in parallel with the write operation of the first index data, which saves the data writing time. In addition, the second index data is only written to the second cache module, eliminating the WAL write operation of the second index data, further shortening the writing time. In addition, when the regional server is restarted, the status bit of the business data can be used to determine whether the business data is valid. After the status bit of the business data is updated to verified, there is no need to query the index data when the regional server is restarted later, avoiding the time-consuming process of the first regional server to query the index data to detect whether the business data is valid each time it is restarted, further saving the writing time.
[0134] Therefore, this embodiment is equivalent to executing a complete index data write operation (including WAL write and memstore write, covering the WAL write time of business data) and two memstore write operations. Writing to memstore is a lightweight operation, so the end-to-end write time of the overall process is close to the write time of one data, and close to the write time of non-indexed data. It optimizes the write performance of indexed data while ensuring strong data consistency.
[0135] The above describes the embodiments of the present application from the perspective of the method. The following describes the relevant devices in the embodiments of the present application from the perspective of specific device implementation.
[0136] See also Figure 6 , an embodiment of the present application provides a schematic diagram of a regional server 600 , wherein the regional server 600 includes a sending unit 601 , a writing unit 602 and a receiving unit 603 .
[0137] The sending unit 601 is used to send a first instruction to the second region server, where the first instruction is used to instruct the second region server to write first index data, and the status bit of the first index data is unverified.
[0138] The writing unit 602 is used to write the first business data into the write-ahead log based on the first instruction, and the first index data is used to index the first business data.
[0139] The receiving unit 603 is configured to receive a response message returned by the second region server after writing the first index data.
[0140] The writing unit 602 is further configured to write the first business data into a first cache module based on a response message after the first business data is written into the write-ahead log, and the first cache module belongs to the regional server 600 .
[0141] The sending unit 601 is further used to send a second instruction to the second regional server after the first service data is written into the first cache module, the second instruction is used to instruct the second regional server to write second index data, and the second index data is obtained after the status bit of the first index data is updated to verified.
[0142] Optionally, the first instruction and the second instruction are remote procedure call instructions.
[0143] Optionally, the write unit 601 is specifically used to set the status bit of the first business data to unverified, obtain the second business data, and write the second business data into the write-ahead log. The regional server 600 also includes an update unit 604, which is used to update the status bit of the second business data to verified, and obtain the third business data. The write unit 601 is also used to write the third business data into the write-ahead log.
[0144] Optionally, the regional server 600 further includes a repairing unit 605, configured to repair the second service data based on the status bit of the second service data.
[0145] Optionally, the second instruction is used to instruct the second regional server to write the second index data into a second cache module, and the second cache module belongs to the second regional server.
[0146] Optionally, the repair unit 605 is further configured to repair the first index data based on a status bit of the first index data.
[0147] Each unit in the regional server 600 performs the above Figures 3 to 5 The operation of the first area server in the illustrated embodiment will not be described in detail herein.
[0148] See below Figure 7 , is a possible structural diagram of a regional server 700 provided in an embodiment of the present application, including a processor 701, a communication interface 702, a memory 703 and a bus 704. The processor 701, the communication interface 702 and the memory 703 are interconnected via the bus 704. In an embodiment of the present application, the processor 701 is used to control and manage the actions of the regional server. For example, the processor 701 is used to execute Figure 3 The steps performed by the first regional server in the method embodiment are shown. The communication interface 702 is used to support the first regional server to communicate. The memory 703 is used to store the program code and data of the first regional server.
[0149] Among them, the processor 701 can be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the contents disclosed in this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and so on. The bus 704 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0150] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium includes instructions, and when the instructions are executed on a computer, the computer executes the aforementioned Figures 3 to 5 The method in the illustrated embodiment.
[0151] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the aforementioned Figures 3 to 5 The method in the illustrated embodiment.
[0152] The embodiment of the present application also provides a chip system, the chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to execute the aforementioned Figures 3 to 5 The method in the illustrated embodiment.
[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0154] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
Claims
1. A data processing method, characterized in that: The method is applied to a first region server, and the method comprises: Sending a first instruction to the second region server, wherein the first instruction is used to instruct the second region server to write first index data, wherein a status bit of the first index data is unverified; Writing first service data into a write-ahead log based on the first instruction, wherein the first index data is used to index the first service data; receiving a response message returned by the second region server after writing the first index data; After the first service data is written into the write-ahead log, writing the first service data into a first cache module based on the response message, the first cache module belonging to the first regional server; After the first service data is written into the first cache module, a second instruction is sent to the second regional server, where the second instruction is used to instruct the second regional server to write second index data, where the second index data is obtained by updating the status bit of the first index data to verified.
2. The method according to claim 1, characterized in that The first instruction and the second instruction are remote procedure call instructions.
3. The method according to claim 1 or 2, characterized in that: The step of writing the first service data into a write-ahead log based on the first instruction includes: Setting the status bit of the first service data to unverified to obtain second service data; Writing the second service data into the write-ahead log; After sending the second instruction to the second regional server, the method further includes: Update the status bit of the second service data to verified, and obtain third service data; The third service data is written into the write-ahead log.
4. The method according to claim 3, characterized in that The method further comprises: The second service data is repaired based on the status bit of the second service data.
5. The method according to any one of claims 1 to 4, characterized in that: The second instruction is used to instruct the second regional server to write the second index data into a second cache module, and the second cache module belongs to the second regional server.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The first index data is repaired based on the status bit of the first index data.
7. A regional server, characterized in that: include: a sending unit, configured to send a first instruction to the second region server, wherein the first instruction is used to instruct the second region server to write first index data, wherein a status bit of the first index data is unverified; a writing unit, configured to write first service data into a write-ahead log based on the first instruction, wherein the first index data is used to index the first service data; a receiving unit, configured to receive a response message returned by the second region server after writing the first index data; The writing unit is further configured to write the first business data into a first cache module based on the response message after the first business data is written into the write-ahead log, the first cache module belonging to the regional server; The sending unit is further used to send a second instruction to the second regional server after the first service data is written into the first cache module, wherein the second instruction is used to instruct the second regional server to write second index data, and the second index data is obtained after the status bit of the first index data is updated to verified.
8. The regional server according to claim 7, characterized in that: The first instruction and the second instruction are remote procedure call instructions.
9. The regional server according to claim 7 or 8, characterized in that: The writing unit is specifically used for: Setting the status bit of the first service data to unverified to obtain second service data; Writing the second service data into the write-ahead log; The regional server also includes: An updating unit, configured to update the status bit of the second service data to verified, to obtain third service data; The writing unit is also used for: The third service data is written into the write-ahead log.
10. The regional server according to claim 9, characterized in that: The regional server also includes: A repair unit is used to repair the second service data based on the status bit of the second service data.
11. The regional server according to any one of claims 7 to 10, characterized in that: The second instruction is used to instruct the second regional server to write the second index data into a second cache module, and the second cache module belongs to the second regional server.
12. The regional server according to any one of claims 7 to 11, characterized in that: The repair unit is also used for: The first index data is repaired based on the status bit of the first index data.
13. A regional server, characterized in that: include: Processor and memory; The memory is used to store instructions; The processor is configured to execute instructions stored in the memory to implement the method according to any one of claims 1 to 6.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 6 is implemented.
15. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.